sequel-hexspace 1.0.0 → 2.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b2949511f39f19c48b5120965d8b236fe93215c07ee9cf987d7b80f81d649b81
-  data.tar.gz: 68b65e86ccc1ad3cc1fef989eb1d79b7927f96c4f4b5a60abb9699c7b425f4a4
+  metadata.gz: 8ce1fd4d7fed71e9678ed12c286b3a1b1e0bff80d52ef2d78c222a52e7a09713
+  data.tar.gz: 9ae18e5f352c95e8390d6d26ea09d217f63dad5c105a2bd30b2290c87483d566
 SHA512:
-  metadata.gz: 021db16089f80aee9a8398b54747265e6eedb59713b7fac36316d4623a324ce96036e532df0ac75171ec47a04ade882ee168c5c773e5487a7591567bfaab707c
-  data.tar.gz: 2767b9661167bc70fb775513f4abcb0465a77cc19c16f9bdb94972acd30896a310c3daeb6d8a34cfdd089ccf6c8bd6558bc459a9ae2eb3415c7cfd35b5d7273a
+  metadata.gz: f35426bc639c4dcf948f425629ef4ec895cc0c0334106c60461b18acb28ee4cf7b0067f6f36196a57d6ba8535476b018b6c1b9b210f1f5e06c2af397cb2760ce
+  data.tar.gz: d308e6d823bf60ddacfc2578e3b09a1892e376863ec5101573f4cc6a7c3fd7b496362288c63a0e8ec5ec595f95a981713fe6c4fca5588b9872ce95152d44d8d9

data/CHANGELOG.md

@@ -0,0 +1,70 @@
+# Changelog
+
+## [2.0.1] - 2026-10-07
+
+### Fixed
+
+- Emulated `DELETE` and `UPDATE` no longer drop the original table. Spark has
+  neither statement, so the adapter rebuilds the table under a temporary name and
+  swaps it in. The swap used to drop the original and rename the copy into place.
+  A process killed between those two steps lost the table, with its rows left
+  under a generated name. The swap is now an `INSERT OVERWRITE`, so the original
+  table always exists.
+- A temporary table left behind by an interrupted delete or update no longer
+  breaks every later delete or update on that table with
+  `TABLE_OR_VIEW_ALREADY_EXISTS`. The adapter now tolerates it and cleans it up.
+- The temporary table for a schema-qualified table (`sch.items`) is now created
+  in that schema. It used to be created as a single dotted identifier in the
+  default database.
+- `Sequel.date_add` / `Sequel.date_sub` now honour the `date_arithmetic`
+  extension's cast: the `:cast` option, or the generic timestamp type by
+  default. Both were silently ignored.
+
+### Changed
+
+- Require `thrift >= 0.23` (was `>= 0.18`), and delete the `Thrift::Bytes`
+  backport that older versions needed. The upper bound stays `< 0.25`.
+- Bound `hexspace` to `>= 0.2.1, < 0.4` (was `>= 0.2.1`). hexspace is a 0.x gem,
+  so a new minor version may break its API. 0.3.0 is its newest release, so the
+  bound excludes nothing that exists.
+- Internal simplifications with no change in behaviour.
+
+## [2.0.0] - 2026-10-01
+
+### Breaking changes
+
+- Declare `required_ruby_version >= 3.3`. Ruby 3.2 reached end of life in March
+  2026 and is no longer tested. Installing on an older Ruby now fails with a
+  clear message rather than breaking at runtime. This is the only breaking
+  change, and it is what makes this release a major one.
+
+### Note on the gap since 1.0.0
+
+1.0.0 (April 2024) was the previous published release. There was never a 1.0.1
+release and never a v1.0.1 tag — 1.0.1 was only a gemspec version bump, at
+`ce8b23e`, never tagged and never published. Everything below had therefore been
+unavailable to users until this release, which is the first to carry any of it.
+The list is reconstructed from git history.
+
+### Added
+
+- Shared Hexspace mock adapter support, so dependent projects can test without a
+  live Spark server.
+- `EXTERNAL`, `LOCATION` and `LIKE` support in `CREATE TABLE`.
+- Always-on SimpleCov coverage and RuboCop enforcement in the development setup.
+
+### Fixed
+
+- Support thrift 0.24 by restoring `Thrift::Client#handle_exception` and
+  `#reply_seqid`, which 0.24.0 removed while hexspace's generated client still
+  calls them. Without this every Spark connection failed with a
+  `Sequel::DatabaseConnectionError`. thrift is constrained to `>= 0.18, < 0.25`.
+- Silence a frozen-string-literal warning from thrift 0.22.0 on Ruby 3.4+.
+- `Database#database_type` returns `:spark` rather than `:hexspace`.
+- Ignore `CASCADE` in `drop_table_sql`, which Spark SQL does not accept.
+- Route prepared `DELETE` statements through the emulated delete path.
+- Date and timestamp casting and literalisation, including Ruby 4.0
+  compatibility.
+
+[2.0.0]: https://github.com/outcomesinsights/sequel-hexspace/compare/v1.0.0...v2.0.0
+[2.0.1]: https://github.com/outcomesinsights/sequel-hexspace/compare/v2.0.0...v2.0.1

data/LICENSE

@@ -1,4 +1,5 @@
 Copyright (c) 2023 Jeremy Evans
+Copyright (c) 2026 Outcomes Insights, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to

data/README

@@ -1,3 +1,92 @@
 This is a hexspace adapter for Sequel, designed to be used with Spark (not
 Hive). You can use the hexspace:// protocol in the Sequel connection URL
 to use this adapter.
+
+== Emulated DELETE and UPDATE
+
+Spark has no DELETE and no UPDATE, so Sequel::Dataset#delete and
+Sequel::Dataset#update are emulated. Both run as:
+
+1. CREATE TABLE <table>__sequel_delete_emulate AS SELECT <the rows the
+   operation should leave behind>
+2. INSERT OVERWRITE TABLE <table> SELECT ... FROM
+   <table>__sequel_delete_emulate
+3. check that <table> now holds as many rows as the temp table did, then DROP
+   the temp table
+
+The suffix goes on the table's own name, so the temp table is always in the
+same schema -- and catalog -- as your table: deleting from schema.items builds
+schema.items__sequel_delete_emulate there. Before 2026-10-02 the name was
+assembled by stripping the quoting out of the literalized target, which
+collapsed `schema`.`items` into the single identifier
+"schema.items__sequel_delete_emulate", and Spark rejects that outright ("not a
+valid name for tables/databases. Valid names only contain alphabet characters,
+numbers and _"). So an emulated DELETE or UPDATE against a schema-qualified
+dataset used to fail at step 1, without touching your table. If you are
+upgrading past that there is nothing to clean up: the temp table could never be
+created, so there is none to find.
+
+This is not a transaction, and this adapter has none to offer you:
+Database#transaction just yields. The adapter's own source says this path is
+"designed to minimize the changes to the tests, and is not recommended for
+production use". The rest of this section is what that disclaimer means in
+practice, so that you can decide rather than discover.
+
+=== What is guaranteed
+
+* Your table is never dropped. There is no window in which reading it fails
+  with table-or-view-not-found. (Before 2026-10-02 the swap dropped your table
+  and renamed the temp table into its place, and a process that died in between
+  left the table GONE.)
+
+* The rows the operation intends to keep exist under
+  <table>__sequel_delete_emulate, in your table's own schema, from step 1 until
+  step 3. If the process dies before step 3, or if the row-count check in step
+  3 disagrees, that table is left in place on purpose -- it is the recovery
+  copy, and it is the only name your data may be sitting under. Nothing in your
+  own source will name it, so if an emulated delete or update died, look there
+  first.
+
+* That name is derived from your table's name and nothing else. It carries no
+  timestamp, process id or random suffix, and that is deliberate: the deaths
+  recovery exists for -- a SIGKILL, a lost connection -- raise nothing, so a
+  generated name would appear in no error, in no log, and nowhere in your
+  source, and the rows would be unfindable by anyone who had not been watching.
+  What that choice costs is the concurrency limitation below.
+
+* Step 3 does not drop the temp table on the strength of step 2 merely
+  returning. Until the row count has been checked, the temp table is treated as
+  the only intact copy.
+
+=== What is NOT guaranteed
+
+* Atomicity. INSERT OVERWRITE is unambiguously atomic only on an ACID table
+  such as Delta. This adapter emits USING <format> only when you pass :using to
+  create_table, and the temp table above passes none, so unless your metastore
+  defaults to an ACID format both tables are not ACID. Against a default Spark
+  thriftserver (Provider hive, TextInputFormat, LazySimpleSerDe) the write was
+  measured to stage and then commit -- a job that failed after writing 150,000
+  of 200,000 rows left the table holding its full pre-operation contents, and
+  SIGKILLing the client at seven points across the statement never left the
+  table absent, empty or partial. What is left is the commit step: replacing a
+  multi-file Hive table in place is not one atomic filesystem operation, so a
+  death of the SERVER, or a storage failure, inside that step can leave the
+  table PRESENT but empty or partial. A plain SELECT then returns the wrong
+  rows SILENTLY rather than failing loudly. That window has not been
+  reproduced and is not quantified; it is accepted, not closed.
+
+* Re-running is not free. Step 1 is preceded by a DROP TABLE IF EXISTS of
+  <table>__sequel_delete_emulate, because Spark's CREATE TABLE is not
+  idempotent and a stranded temp table otherwise makes every later delete or
+  update on the same table fail. That means re-running a delete or update that
+  died DESTROYS the recovery copy. Recover your data first, then re-run.
+
+* Concurrency. The temp table name is derived from the target table name and is
+  not made unique per operation, so two emulated deletes or updates against the
+  same table at the same time will fight over it. Serialize them. This is not
+  fixed, and it is not going to be: a unique name is what would fix it, and the
+  recovery guarantee above is why that is not on offer.
+
+If any of that is unacceptable for your workload, do not use emulated DELETE or
+UPDATE -- build the replacement table yourself, under a name you control, and
+swap it in on terms you have chosen.

data/lib/sequel/adapters/hexspace/thrift_compat.rb

@@ -0,0 +1,54 @@
+# rubocop:disable all
+# Restore Thrift::Client#handle_exception and #reply_seqid for thrift >= 0.24.
+#
+# hexspace's generated Thrift client (lib/hexspace/tcli_service.rb) calls both,
+# 21 times each. thrift 0.24.0 removed them, folding their work into the new
+# #validate_message_begin -- so on thrift 0.24 every Spark connection dies in
+# recv_OpenSession with NoMethodError, surfacing as
+# Sequel::DatabaseConnectionError.
+#
+# Upstream will not fix this for us: hexspace's generated client is untouched
+# since 2023-05-08, the latest release (0.3.0, Apr 2025) predates thrift 0.24,
+# and no issue or PR there mentions thrift. Regenerating that client against
+# 0.24 would drop support for thrift < 0.24, which is a maintainer's decision
+# rather than ours to force.
+#
+# Both methods are restored verbatim from thrift 0.23.0 and are implemented
+# purely in terms of primitives thrift 0.24 still provides and still uses in
+# validate_message_begin. The dequeue accounting is unchanged: exactly one
+# dequeue_pending_seqid per call on both the reply and the exception path.
+#
+# Note this keeps 0.23 semantics -- it does NOT add 0.24's stricter checks for
+# invalid message type and wrong method name. Remove once hexspace regenerates
+# its client against thrift >= 0.24.
+if defined?(Thrift::Client)
+  # Both method bodies below are copied verbatim from thrift 0.23.0 and
+  # deliberately not restyled: keeping them byte-comparable with upstream is
+  # what makes them auditable. That is enforced by the `rubocop:disable all` on
+  # this file's first line, which rubocop honours however it is invoked -- a
+  # .rubocop.yml Exclude is not, because rubocop ignores Exclude for a path named
+  # on the command line and treefmt names them. This file holds the shim and
+  # nothing else so the rest of the adapter stays linted.
+  #
+  # reply_seqid also cannot be renamed to a predicate -- hexspace's generated
+  # client calls it by that exact name.
+  module Thrift
+    module Client
+      unless method_defined?(:handle_exception) || private_method_defined?(:handle_exception)
+        def handle_exception(mtype)
+          if mtype == MessageTypes::EXCEPTION
+            dequeue_pending_seqid
+            raise_application_exception
+          end
+        end
+      end
+
+      unless method_defined?(:reply_seqid) || private_method_defined?(:reply_seqid)
+        def reply_seqid(rseqid)
+          expected_seqid = dequeue_pending_seqid
+          !expected_seqid.nil? && rseqid == expected_seqid
+        end
+      end
+    end
+  end
+end

data/lib/sequel/adapters/hexspace.rb

@@ -1,10 +1,11 @@
-require 'hexspace'
-require_relative 'shared/spark'
+require_relative "shared/hexspace"
+require "hexspace"
+require_relative "hexspace/thrift_compat"
 
 module Sequel
   module Hexspace
     class Database < Sequel::Database
-      include Spark::DatabaseMethods
+      include DatabaseMethods
 
       set_adapter_scheme :hexspace
 
@@ -13,7 +14,7 @@ module Sequel
       def connect(server)
         opts = server_opts(server)
         opts[:username] = opts[:user]
-        opts.select!{|k,v| v.to_s != '' && ALLOWED_CLIENT_KEYWORDS.include?(k)}
+        opts.select! { |k, v| v.to_s != "" && ALLOWED_CLIENT_KEYWORDS.include?(k) }
         ::Hexspace::Client.new(**opts)
       end
 
@@ -27,9 +28,20 @@ module Sequel
         conn.instance_variable_set(:@sequel_invalid, true)
       end
 
-      def execute(sql, opts=OPTS)
+      # Hexspace returns timestamp strings in UTC without an explicit offset.
+      # When no database timezone is configured, Sequel treats naive strings as
+      # local time, which shifts CURRENT_TIMESTAMP by the local UTC offset.
+      def to_application_timestamp(value)
+        if value.is_a?(String) && timezone.nil? && value !~ /(?:Z|[+-]\d{2}(?::?\d{2})?)\z/
+          Sequel.convert_timestamp(value, :utc)
+        else
+          super
+        end
+      end
+
+      def execute(sql, opts = OPTS)
         synchronize(opts[:server]) do |conn|
-          res = log_connection_yield(sql, conn){conn.execute(sql, result_object: true)}
+          res = log_connection_yield(sql, conn) { conn.execute(sql, result_object: true) }
         rescue => e
           raise_error(e)
         else
@@ -37,7 +49,7 @@ module Sequel
         end
       end
 
-      def execute_insert(sql, opts=OPTS)
+      def execute_insert(sql, opts = OPTS)
         execute(sql, opts)
 
         # Return nil instead of empty array.
@@ -51,7 +63,7 @@ module Sequel
     end
 
     class Dataset < Sequel::Dataset
-      include Spark::DatasetMethods
+      include DatasetMethods
 
       def fetch_rows(sql)
         execute(sql) do |result|
@@ -62,26 +74,20 @@ module Sequel
           types = result.column_types
           column_info = columns.map.with_index do |name, i|
             conversion_proc = case types[i]
-            when 'binary'
+            when "binary"
               Sequel.method(:blob)
-            when 'timestamp'
+            when "timestamp"
               db.method(:to_application_timestamp)
             end
 
-            [i, name, conversion_proc]
+            [ i, name, conversion_proc ]
           end
 
           result.rows.each do |row|
             h = {}
             column_info.each do |i, name, conversion_proc|
               value = row[i]
-              h[name] = if value.nil?
-                nil
-              elsif conversion_proc
-                conversion_proc.call(value)
-              else
-                value
-              end
+              h[name] = conversion_proc && !value.nil? ? conversion_proc.call(value) : value
             end
             yield h
           end

data/lib/sequel/adapters/shared/hexspace.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require_relative "spark"
+
+module Sequel
+  module Hexspace
+    DatabaseMethods = Sequel::Spark::DatabaseMethods
+    DatasetMethods = Sequel::Spark::DatasetMethods
+  end
+
+  Sequel::Database.set_shared_adapter_scheme(:hexspace, Sequel::Hexspace)
+end

data/lib/sequel/adapters/shared/spark.rb

@@ -1,6 +1,6 @@
 # frozen-string-literal: true
 
-require 'sequel/adapters/utils/unmodified_identifiers'
+require "sequel/adapters/utils/unmodified_identifiers"
 
 module Sequel
   module Spark
@@ -9,7 +9,7 @@ module Sequel
     module DatabaseMethods
       include UnmodifiedIdentifiers::DatabaseMethods
 
-      def create_schema(schema_name, opts=OPTS)
+      def create_schema(schema_name, opts = OPTS)
         run(create_schema_sql(schema_name, opts))
       end
 
@@ -17,7 +17,7 @@ module Sequel
         :spark
       end
 
-      def drop_schema(schema_name, opts=OPTS)
+      def drop_schema(schema_name, opts = OPTS)
         run(drop_schema_sql(schema_name, opts))
       end
 
@@ -26,28 +26,28 @@ module Sequel
       def serial_primary_key_options
         # We could raise an exception here instead of just
         # ignoring the primary key setting.
-        {:type=>Integer}
+        { type: Integer }
       end
- 
+
       def supports_create_table_if_not_exists?
         true
       end
 
-      def tables(opts=OPTS)
+      def tables(opts = OPTS)
         _mangle_tables(_tables("TABLES", :tableName, opts) - _views(opts), opts)
       end
 
       # Spark does not support transactions.
-      def transaction(opts=nil)
+      def transaction(_opts = nil)
         yield
       end
 
       # Use an inline VALUES table.
       def values(v)
-        @default_dataset.clone(:values=>v)
+        @default_dataset.clone(values: v)
       end
 
-      def views(opts=OPTS)
+      def views(opts = OPTS)
         _mangle_tables(_views(opts), opts)
       end
 
@@ -56,10 +56,10 @@ module Sequel
       def _tables(type, column, opts)
         sql = String.new
         sql << "SHOW " << type
-        if schema = opts[:schema]
+        if (schema = opts[:schema])
           sql << " IN " << literal(schema)
         end
-        if like = opts[:like]
+        if (like = opts[:like])
           sql << " LIKE " << literal(like)
         end
 
@@ -68,7 +68,7 @@ module Sequel
         # Always internally qualify, so that if a table name in a schema
         # has the same name as a temporary view, it will not exclude
         # the table name.
-        ds.map([:namespace, column]).map do |ns, name|
+        ds.map([ :namespace, column ]).map do |ns, name|
           if ns && !ns.empty?
             Sequel::SQL::QualifiedIdentifier.new(ns, name)
           else
@@ -85,39 +85,54 @@ module Sequel
         if opts[:qualify]
           tables
         else
-          tables.map{|t| t.is_a?(Sequel::SQL::QualifiedIdentifier) ? t.column.to_sym : t}
+          tables.map { |t| t.is_a?(Sequel::SQL::QualifiedIdentifier) ? t.column.to_sym : t }
         end
       end
 
       def create_schema_sql(schema_name, opts)
         sql = String.new
-        sql << 'CREATE SCHEMA '
-        sql << 'IF NOT EXISTS ' if opts[:if_not_exists]
+        sql << "CREATE SCHEMA "
+        sql << "IF NOT EXISTS " if opts[:if_not_exists]
         sql << literal(schema_name)
 
-        if comment = opts[:comment]
-          sql << ' COMMENT '
+        if (comment = opts[:comment])
+          sql << " COMMENT "
           sql << literal(comment)
         end
 
-        if location = opts[:location]
-          sql << ' LOCATION '
+        if (location = opts[:location])
+          sql << " LOCATION "
           sql << literal(location)
         end
 
-        if properties = opts[:properties]
-          sql << ' WITH DBPROPERTIES ('
+        if (properties = opts[:properties])
+          sql << " WITH DBPROPERTIES ("
           properties.each do |k, v|
             sql << literal(k.to_s) << "=" << literal(v.to_s)
           end
-          sql << ')'
+          sql << ")"
         end
 
         sql
       end
 
+      def create_table_prefix_sql(name, options)
+        sql = String.new
+        sql << "CREATE "
+        sql << "EXTERNAL " if options[:external]
+        sql << "TEMPORARY " if options[:temp]
+        sql << "TABLE "
+        sql << "IF NOT EXISTS " if options[:if_not_exists]
+        sql << quote_schema_table(name)
+        sql
+      end
+
       def create_table_sql(name, generator, options)
-        _append_table_view_options_sql(super, options)
+        if options[:like] || (options[:using] && generator.columns.empty?)
+          _append_table_view_options_sql(create_table_prefix_sql(name, options), options)
+        else
+          _append_table_view_options_sql(super, options)
+        end
       end
 
       def create_table_as_sql(name, sql, options)
@@ -131,7 +146,7 @@ module Sequel
         end
 
         sql = String.new
-        sql << create_view_sql_append_columns("CREATE #{'OR REPLACE 'if options[:replace]}#{'TEMPORARY ' if options[:temp]}VIEW#{' IF NOT EXISTS' if options[:if_not_exists]} #{quote_schema_table(name)}", options[:columns])
+        sql << create_view_sql_append_columns("CREATE #{'OR REPLACE ' if options[:replace]}#{'TEMPORARY ' if options[:temp]}VIEW#{' IF NOT EXISTS' if options[:if_not_exists]} #{quote_schema_table(name)}", options[:columns])
 
         if source
           source = source.sql if source.is_a?(Dataset)
@@ -142,10 +157,18 @@ module Sequel
       end
 
       def _append_table_view_options_sql(sql, options)
+        if (like = options[:like])
+          sql << " LIKE " << literal(like)
+        end
+
         if options[:using]
           sql << " USING " << options[:using].to_s
         end
 
+        if (location = options[:location])
+          sql << " LOCATION " << literal(location)
+        end
+
         if options[:partitioned_by]
           sql << " PARTITIONED BY "
           _append_column_list_sql(sql, options[:partitioned_by])
@@ -164,36 +187,42 @@ module Sequel
         end
 
         if options[:options]
-          sql << ' OPTIONS ('
+          sql << " OPTIONS ("
           options[:options].each do |k, v|
             sql << literal(k.to_s) << "=" << literal(v.to_s)
           end
-          sql << ')'
+          sql << ")"
         end
 
         sql
       end
 
       def _append_column_list_sql(sql, columns)
-        sql << '(' 
+        sql << "("
         schema_utility_dataset.send(:identifier_list_append, sql, Array(columns))
-        sql << ')'
+        sql << ")"
       end
 
       def drop_schema_sql(schema_name, opts)
         sql = String.new
-        sql << 'DROP SCHEMA '
-        sql << 'IF EXISTS ' if opts[:if_exists]
+        sql << "DROP SCHEMA "
+        sql << "IF EXISTS " if opts[:if_exists]
         sql << literal(schema_name)
-        sql << ' CASCADE' if opts[:cascade]
+        sql << " CASCADE" if opts[:cascade]
         sql
       end
 
+      # Spark SQL does not support CASCADE on DROP TABLE.
+      # Support :purge option, ignore :cascade.
+      def drop_table_sql(name, options)
+        "DROP TABLE#{' IF EXISTS' if options[:if_exists]} #{quote_schema_table(name)}#{' PURGE' if options[:purge]}"
+      end
+
       def schema_parse_table(table, opts)
         m = output_identifier_meth(opts[:dataset])
         im = input_identifier_meth(opts[:dataset])
         metadata_dataset.with_sql("DESCRIBE #{"#{im.call(opts[:schema])}." if opts[:schema]}#{im.call(table)}").map do |row|
-          [m.call(row[:col_name]), {:db_type=>row[:data_type], :type=>schema_column_type(row[:data_type])}]
+          [ m.call(row[:col_name]), { db_type: row[:data_type], type: schema_column_type(row[:data_type]) } ]
         end
       end
 
@@ -201,36 +230,55 @@ module Sequel
         true
       end
 
-      def type_literal_generic_file(column)
-        'binary'
+      def type_literal_generic_file(_column)
+        "binary"
       end
 
-      def type_literal_generic_float(column)
-        'float'
+      def type_literal_generic_float(_column)
+        "float"
       end
 
-      def type_literal_generic_string(column)
-        'string'
+      def type_literal_generic_string(_column)
+        "string"
       end
     end
 
     module DatasetMethods
       include UnmodifiedIdentifiers::DatasetMethods
 
-      Dataset.def_sql_method(self, :select, [['if opts[:values]', %w'values'], ['else', %w'with select distinct columns from join where group having compounds order limit']])
+      Dataset.def_sql_method(self, :select, [ [ "if opts[:values]", %w[values] ], [ "else", %w[with select distinct columns from join where group having compounds order limit] ] ])
 
+      # Handle the date_arithmetic extension's DateAdd expressions using
+      # Spark's make_ym_interval/make_dt_interval functions.
+      #
+      # The extension lets a caller name the result type with the :cast
+      # option (DateAdd#cast_type), defaulting to the generic timestamp type,
+      # and the result is always cast to it here.
+      #
+      # The cast is unconditional because Spark's interval arithmetic does
+      # not settle on one result type -- it depends on both the input type
+      # and which interval function is used. Measured against a live server
+      # on the 3.5.0 that ci.yml pins: DATE + make_dt_interval and TIMESTAMP
+      # + either function yield a TIMESTAMP, but DATE + make_ym_interval
+      # yields a DATE. So there is no subset of cases in which the cast can
+      # be skipped and still be honoured, and an interval of all zeroes adds
+      # nothing to cast in the first place.
+      #
+      # Spark's CAST is narrower than other databases': types it does not
+      # have (:timestamptz) or cannot convert to (:interval) are rejected by
+      # the server with UNSUPPORTED_DATATYPE or DATATYPE_MISMATCH, raising
+      # Sequel::DatabaseError. That is deliberately not pre-screened here --
+      # an allowlist of castable types in the adapter would go stale against
+      # the server, and the server's own message names the offending type.
       def date_add_sql_append(sql, da)
         expr = da.expr
-        cast_type = da.cast_type || Time
 
         h = Hash.new(0)
         da.interval.each do |k, v|
           h[k] = v || 0
         end
 
-        if h[:weeks]
-          h[:days] += h[:weeks] * 7
-        end
+        h[:days] += h[:weeks] * 7
 
         if h[:years] != 0 || h[:months] != 0
           expr = Sequel.+(expr, Sequel.function(:make_ym_interval, h[:years], h[:months]))
@@ -240,15 +288,28 @@ module Sequel
           expr = Sequel.+(expr, Sequel.function(:make_dt_interval, h[:days], h[:hours], h[:minutes], h[:seconds]))
         end
 
-        literal_append(sql, expr)
+        literal_append(sql, Sequel.cast(expr, da.cast_type || Time))
+      end
+
+      # Route prepared statement / bound variable deletes through the
+      # emulated delete path, since Spark does not support native DELETE.
+      def call(type, bind_variables = OPTS, *values, &)
+        if type == :delete
+          ps = to_prepared_statement(type, values, extend: send(:bound_variable_modules))
+          ps.bind(bind_variables).delete
+        else
+          super
+        end
       end
 
       # Emulate delete by selecting all rows except the ones being deleted
-      # into a new table, drop the current table, and rename the new
-      # table to the current table name.
+      # into a new table, then overwriting the current table with that new
+      # table's contents.
       #
       # This is designed to minimize the changes to the tests, and is
-      # not recommended for production use.
+      # not recommended for production use. It is not atomic and it has a
+      # residual failure window even after sequel-hexspace-rho hardened it --
+      # see _with_temp_table and the README section on emulated DELETE/UPDATE.
       def delete
         _with_temp_table
       end
@@ -259,21 +320,184 @@ module Sequel
         updated_vals = columns.values
 
         _with_temp_table do |tmp_name|
-          db.from(tmp_name).insert([*updated_cols, *other_cols], select(*updated_vals, *other_cols))
+          db.from(tmp_name).insert([ *updated_cols, *other_cols ], select(*updated_vals, *other_cols))
         end
       end
 
+      # Build `<table>__sequel_delete_emulate` -- in the target's own schema,
+      # see _temp_table_name -- holding the rows the operation should leave
+      # behind, let the caller adjust it, then INSERT OVERWRITE those rows back
+      # into the original table.
+      #
+      # The original table object is never dropped. That is deliberate: until
+      # 2026-10-02 this method dropped the original and renamed the temp table
+      # into its place, so a process that died in between left the named table
+      # GONE, with its rows sitting under a runtime-generated name that appears
+      # nowhere in the caller's source. See sequel-hexspace-rho.
+      #
+      # WHAT THIS STILL DOES NOT GUARANTEE -- read this before relying on it.
+      # INSERT OVERWRITE is unambiguously atomic only on an ACID table. This
+      # adapter emits USING <format> only when a caller passes :using, and the
+      # create_table below passes none, so the temp table -- and typically the
+      # caller's own table -- is whatever the metastore defaults to. Measured
+      # against this gem's test server on 2026-10-02: Provider `hive`,
+      # TextInputFormat, LazySimpleSerDe. Not ACID.
+      #
+      # It nonetheless behaved better than "not ACID" implies, and the
+      # measurements are recorded here because they bound the risk
+      # that is left:
+      #
+      #   * A job failure part way through the write does NOT truncate the
+      #     target. raise_error fired after 150k of 200k rows and the table was
+      #     left holding its FULL pre-operation contents, so the old files are
+      #     not removed up front -- the write stages and then commits.
+      #   * SIGKILLing the client at seven points across the statement never
+      #     left the table absent, empty or partial. The thriftserver carries
+      #     the statement on server-side after the client vanishes, so the table
+      #     held either its pre- or its post-operation rows every time. The same
+      #     harness against the old drop/rename sequence destroyed the table on
+      #     the first try (400,000 rows stranded under the temp name), so it was
+      #     landing in the window.
+      #
+      # What is left is the commit itself: replacing a multi-file Hive table in
+      # place is not one atomic filesystem operation, so a death of the SERVER
+      # -- not the client -- inside that commit, or a storage failure there, can
+      # still leave the table PRESENT but empty or partial, which a plain SELECT
+      # reports silently rather than loudly. That window was NOT reproduced: it
+      # needs the thriftserver killed, and this gem's test server is shared.
+      # Treat it as unquantified, not as absent.
+      #
+      # The trade is therefore deliberate and it is not free. A silently wrong
+      # read is in some ways worse than a loud table-not-found. What buys it is
+      # recoverability: an absent table is unrecoverable by anyone reading their
+      # own source, because the name holding the data appears nowhere in it,
+      # whereas the rows to keep are recoverable here at every instant -- see
+      # the ensure below. The same residual window is written down in the
+      # README, because a consumer cannot see this comment.
+      #
+      # The temp table is dropped in exactly two states, both of which make it a
+      # discardable copy:
+      #
+      #   * the overwrite never started, so the original still holds its
+      #     pre-operation contents; or
+      #   * the overwrite finished and its row count was verified against the
+      #     temp table, so the original holds the intended contents.
+      #
+      # In any other state -- a raise inside the overwrite, or a row count that
+      # disagrees with the temp table -- the temp table holds the only intact
+      # copy of the rows to keep and is left alone. That is sequel-hexspace-c5q's
+      # rule, carried over from the drop/rename sequence it was written for.
+      #
+      # Both the leading drop and the ensure exist because this sequence is not
+      # atomic and Spark's CREATE TABLE is not idempotent, so a process that
+      # dies part way through strands the temp table and every later delete or
+      # update on the same table then fails with TABLE_OR_VIEW_ALREADY_EXISTS --
+      # an error that names a table nothing in the caller's code refers to.
+      # This gem's own suite was poisoned that way on 2026-10-01: one aborted
+      # run left `a__sequel_delete_emulate` behind and the next run reported 51
+      # errors in tests that never touch table `a`.
+      #
+      # The ensure handles a raise or an interrupt, which still unwinds; the
+      # leading drop handles the rest (SIGKILL, a lost connection, a crash),
+      # where nothing of ours gets to run at all. Each names exactly the one
+      # table this method creates and owns -- neither enumerates the server,
+      # which may be shared with tables belonging to nobody here.
       private def _with_temp_table
         n = count
         table_name = first_source_table
-        tmp_name = literal(table_name).gsub('`', '') + "__sequel_delete_emulate"
-        db.create_table(tmp_name, :as=>select_all.invert)
-        yield tmp_name if defined?(yield)
-        db.drop_table(table_name)
-        db.rename_table(tmp_name, table_name)
+        tmp_name = _temp_table_name(table_name)
+        db.drop_table?(tmp_name)
+        db.create_table(tmp_name, as: select_all.invert)
+        overwrite_started = false
+        overwrite_verified = false
+        begin
+          yield tmp_name if defined?(yield)
+
+          # Counted before the overwrite rather than after, so that what the
+          # overwrite is checked against is the intended content itself and not
+          # a number derived from the same statement being verified.
+          expected = db.from(tmp_name).count
+
+          overwrite_started = true
+          _overwrite_from_temp_table(table_name, tmp_name)
+
+          actual = db.from(table_name).count
+          unless actual == expected
+            raise Sequel::Error, "emulated delete/update of #{literal(table_name)} left #{actual} rows, " \
+                                 "expected #{expected}; the rows to keep remain in #{literal(tmp_name)}, " \
+                                 "which has deliberately not been dropped"
+          end
+
+          overwrite_verified = true
+        ensure
+          db.drop_table?(tmp_name) if overwrite_verified || !overwrite_started
+        end
         n
       end
 
+      # The recovery table's name: the target's own name with
+      # `__sequel_delete_emulate` appended, IN THE TARGET'S OWN SCHEMA. The
+      # suffix goes on the LAST component only, so any qualifier -- a schema,
+      # or a catalog and a schema -- survives intact.
+      #
+      # Until 2026-10-02 this was `literal(table_name).gsub("`", "") +
+      # "__sequel_delete_emulate"`, which stripped the backticks that were
+      # keeping a qualified name's parts apart: `sch`.`items` collapsed to the
+      # single String "sch.items__sequel_delete_emulate". Sequel does not split
+      # a String on the dot -- Dataset#schema_and_table returns [nil, the whole
+      # string] for one -- so that went to the server as ONE identifier
+      # containing a dot, naming the default database rather than `sch`.
+      #
+      # That did not merely misplace the recovery copy, it broke the operation
+      # outright, which is worth recording because the bug report assumed
+      # otherwise. Spark refuses the name at CREATE TABLE:
+      #
+      #   `sch.items__sequel_delete_emulate` is not a valid name for
+      #   tables/databases. Valid names only contain alphabet characters,
+      #   numbers and _.
+      #
+      # Measured against this gem's test server on 2026-10-02. So emulated
+      # DELETE and UPDATE against a schema-qualified dataset never worked at
+      # all, and no such temp table can ever have existed anywhere for a
+      # consumer to depend on. Appending to the last component is therefore a
+      # strict widening of the `<table>__sequel_delete_emulate` convention the
+      # README documents, not a change to it -- the unqualified case below is
+      # byte-for-byte what it was.
+      #
+      # A UNIQUE per-operation suffix was considered and rejected, even though
+      # it would close the concurrent-operation collision the README documents
+      # as a hazard. It would put the rows to keep under a runtime-generated
+      # name that appears nowhere in the caller's source, and that is the exact
+      # property sequel-hexspace-rho removed from this method. It is also
+      # unrecoverable in the cases recovery exists for: a SIGKILL or a lost
+      # connection raises nothing, so there is no error message for a generated
+      # name to be reported in, and the leading drop_table? above cannot find
+      # it on the next run either. A predictable name is what makes the README's
+      # "look there first" instruction executable. Concurrency stays the
+      # caller's problem, documented rather than solved.
+      private def _temp_table_name(table_name)
+        case table_name
+        when SQL::QualifiedIdentifier
+          SQL::QualifiedIdentifier.new(table_name.table, "#{table_name.column}__sequel_delete_emulate")
+        else
+          literal(table_name).gsub("`", "") + "__sequel_delete_emulate"
+        end
+      end
+
+      # Replace the whole contents of +table_name+ with the rows in +tmp_name+,
+      # keeping the original table object. The column list is taken from the
+      # TARGET, because INSERT OVERWRITE matches columns by position: naming them
+      # explicitly means a column order difference between the two tables would
+      # be a visible error rather than silently transposed data.
+      #
+      # Extracted so a test can stand in for it, which is the only way to pin
+      # the "temp table survives an unfinished overwrite" rule without killing a
+      # process in the middle of a statement.
+      private def _overwrite_from_temp_table(table_name, tmp_name)
+        rows = db.from(tmp_name).select(*db.from(table_name).columns)
+        db.run("INSERT OVERWRITE TABLE #{literal(table_name)} #{rows.sql}")
+      end
+
       protected def compound_clone(type, dataset, opts)
         dataset = dataset.from_self if dataset.opts[:with]
         super
@@ -303,19 +527,37 @@ module Sequel
       end
 
       def quoted_identifier_append(sql, name)
-        sql << '`' << name.to_s.gsub('`', '``') << '`'
+        sql << "`" << name.to_s.gsub("`", "``") << "`"
       end
 
       def requires_sql_standard_datetimes?
         true
       end
 
-      def insert_supports_empty_values? 
+      def insert_supports_empty_values?
         false
       end
 
       def literal_blob_append(sql, v)
-        sql << "to_binary('" << [v].pack("m*").gsub("\n", "") << "', 'base64')"
+        sql << "to_binary('" << [ v ].pack("m0") << "', 'base64')"
+      end
+
+      # Spark requires DATE 'YYYY-MM-DD' syntax instead of plain quoted strings
+      def literal_date_append(sql, v)
+        sql << "DATE "
+        super
+      end
+
+      # Spark requires TIMESTAMP 'YYYY-MM-DD HH:MM:SS' syntax instead of plain quoted strings
+      def literal_time_append(sql, v)
+        sql << "TIMESTAMP "
+        super
+      end
+
+      # Spark requires TIMESTAMP 'YYYY-MM-DD HH:MM:SS' syntax instead of plain quoted strings
+      def literal_datetime_append(sql, v)
+        sql << "TIMESTAMP "
+        super
       end
 
       def literal_false
@@ -330,7 +572,7 @@ module Sequel
         "true"
       end
 
-      def supports_cte?(type=:select)
+      def supports_cte?(type = :select)
         type == :select
       end
 
@@ -360,16 +602,16 @@ module Sequel
 
       # Handle forward references in existing CTEs in the dataset by inserting this
       # dataset before any dataset that would reference it.
-      def with(name, dataset, opts=OPTS)
-        opts = Hash[opts].merge!(:name=>name, :dataset=>dataset).freeze
+      def with(name, dataset, opts = OPTS)
+        opts = Hash[opts].merge!(name: name, dataset: dataset).freeze
         references = ReferenceExtractor.references(dataset)
 
-        if with = @opts[:with]
+        if (with = @opts[:with])
           with = with.dup
           existing_references = @opts[:with_references]
 
-          if referencing_dataset = existing_references[literal(name)]
-            unless i = with.find_index{|o| o[:dataset].equal?(referencing_dataset)}
+          if (referencing_dataset = existing_references[literal(name)])
+            unless (i = with.find_index { |o| o[:dataset].equal?(referencing_dataset) })
               raise Sequel::Error, "internal error finding referencing dataset"
             end
 
@@ -389,32 +631,38 @@ module Sequel
           # Assume we will insert the dataset at the end, so existing references have priority
           references = references.merge(existing_references)
         else
-          with = [opts]
+          with = [ opts ]
         end
 
-        clone(:with=>with.freeze, :with_references=>references.freeze)
+        clone(with: with.freeze, with_references: references.freeze)
       end
 
       private def select_values_sql(sql)
-        sql << 'VALUES '
+        sql << "VALUES "
         expression_list_append(sql, opts[:values])
       end
     end
 
     # ReferenceExtractor extracts references from datasets that will be used as CTEs.
     class ReferenceExtractor < ASTTransformer
-      TABLE_IDENTIFIER_KEYS = [:from, :join].freeze
-      COLUMN_IDENTIFIER_KEYS = [:select, :where, :having, :order, :group, :compounds].freeze
+      TABLE_IDENTIFIER_KEYS = [ :from, :join ].freeze
+      COLUMN_IDENTIFIER_KEYS = [ :select, :where, :having, :order, :group, :compounds ].freeze
 
       # Returns a hash of literal string identifier keys referenced by the given
       # dataset with the given dataset as the value for each key.
       def self.references(dataset)
-        new(dataset).tap{|ext| ext.transform(dataset)}.references
+        new(dataset).tap { |ext| ext.transform(dataset) }.references
       end
 
       attr_reader :references
 
       def initialize(dataset)
+        # `super()` with explicit empty parens, NOT bare `super`. ASTTransformer
+        # defines no #initialize, so the inherited one is Object's, which takes
+        # no arguments -- bare `super` would forward `dataset` to it and raise
+        # ArgumentError on every instantiation. The call is a no-op today and is
+        # here so the chain still works if ASTTransformer ever gains state.
+        super()
         @dataset = dataset
         @references = {}
       end
@@ -446,10 +694,10 @@ module Sequel
         case o
         when Sequel::Dataset
           # Special case FROM/JOIN, because identifiers inside refer to tables and not columns
-          TABLE_IDENTIFIER_KEYS.each{|k| o.opts[k]&.each{|jc| table_identifier_extract(jc)}}
+          TABLE_IDENTIFIER_KEYS.each { |k| o.opts[k]&.each { |jc| table_identifier_extract(jc) } }
 
           # Look in other keys that may have qualified references or subqueries
-          COLUMN_IDENTIFIER_KEYS.each{|k| v(o.opts[k])}
+          COLUMN_IDENTIFIER_KEYS.each { |k| v(o.opts[k]) }
         when SQL::QualifiedIdentifier
           # If a qualified identifier has a qualified identifier as a key,
           # such as schema.table.column, ignore it, because CTE identifiers shouldn't

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sequel-hexspace
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 2.0.1
 platform: ruby
 authors:
 - Jeremy Evans
-autorequire: 
+- Ryan Duryea
 bindir: bin
 cert_chain: []
-date: 2024-04-03 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel
@@ -31,6 +31,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.2.1
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '0.4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -38,81 +41,113 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.2.1
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '0.4'
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: thrift
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
+        version: '0.23'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '0.25'
+  type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.23'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '0.25'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.7'
+        version: '6.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.7'
+        version: '6.0'
 - !ruby/object:Gem::Dependency
   name: minitest-hooks
   requirement: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 1.5.4
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 1.5.4
 - !ruby/object:Gem::Dependency
   name: minitest-global_expectations
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 description: |
   This is a hexspace adapter for Sequel, designed to be used with Spark (not
   Hive). You can use the hexspace:// protocol in the Sequel connection URL
   to use this adapter.
-email: code@jeremyevans.net
+email: aguynamedryan@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files:
 - LICENSE
 files:
+- CHANGELOG.md
 - LICENSE
 - README
 - lib/sequel/adapters/hexspace.rb
+- lib/sequel/adapters/hexspace/thrift_compat.rb
+- lib/sequel/adapters/shared/hexspace.rb
 - lib/sequel/adapters/shared/spark.rb
-homepage: https://github.com/jeremyevans/sequel-hexspace.git
+homepage: https://github.com/outcomesinsights/sequel-hexspace
 licenses:
 - MIT
-metadata: {}
-post_install_message: 
+metadata:
+  rubygems_mfa_required: 'true'
 rdoc_options:
 - "--quiet"
 - "--line-numbers"
@@ -127,15 +162,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '3.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.6
-signing_key: 
+rubygems_version: 4.0.20
 specification_version: 4
 summary: Sequel adapter for hexspace driver and Apache Spark database
 test_files: []