sequelizer 0.4.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9cda95a3cd5386a1d2e0ce1402793044ba78a07f457a265efe7a45e930095cb0
-  data.tar.gz: 5cbfe2a32b9b65f9922d47fa3a9e894e2335874dfed53fd972bc9d9483ee9dd5
+  metadata.gz: 50510a32c25ed7c0218c60a7ba9d615a2ccf909685cc719dc1110a3fe43a5baf
+  data.tar.gz: a11d4852a6f18ffa2fca2fb5a9a75b67fd6c605d8d38321cb2b7a042e7ed8427
 SHA512:
-  metadata.gz: 3657c7dfd5f9a040cf0b38caa910c1e5526ff874367fc261eaa2da3c3cebfaba9ffc9ca511e183466b10d019d822cfecd9e4b12e3fcbe07e59fc8075608a98de
-  data.tar.gz: 29dd4df1852d028fea492621110a08a65cf186fe672a18645c6f4832aa89c5803c514db68bcd125dc8341e892d24273e0d11eee9f16c0f154cc10c59a967a02a
+  metadata.gz: 4f859dc2afb3bf840d8f2a53215ba7c2437a511eafdd23467cdab868092c45a3e4a8772159ca9098f1b808eda15041d357b05d7fd2f0e7c0f4acc7ec82c68df5
+  data.tar.gz: 69bb86b1fc5965eef2fa4a5449618050cb984d285c93d99fa2e56269c9d2967573f2b8b918c0a992880f7278a0c55bdd48c165a2e744ccf814f356a8565befdb

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.4.1] - 2026-10-29
+
+### Added
+
+- **sequelizer:** Add Sequelizer.dataset_db, a read-only DuckDB dataset connection ([acf8ac8](https://github.com/outcomesinsights/sequelizer/commit/acf8ac8f130d3db87dd3e99374d1d44de4a214f4))
+
 ## [0.4.0] - 2026-10-25
 
 ### Breaking changes
@@ -173,3 +179,4 @@ All notable changes to this project will be documented in this file.
 - The project itself
 
 [0.4.0]: https://github.com/outcomesinsights/sequelizer/compare/v0.3.0...v0.4.0
+[0.4.1]: https://github.com/outcomesinsights/sequelizer/compare/v0.4.0...v0.4.1

data/README.md

@@ -1,4 +1,4 @@
-# Sequelizer [![Build Status](https://travis-ci.org/outcomesinsights/sequelizer.svg?branch=master)](https://travis-ci.org/outcomesinsights/sequelizer)
+# Sequelizer [![CI](https://github.com/outcomesinsights/sequelizer/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/outcomesinsights/sequelizer/actions/workflows/test.yml)
 
 I was tired of writing the code to bootstrap a connection to my databases.
 
@@ -78,6 +78,56 @@ For PostgreSQL, `search_path`, `schema_search_path` and `schema` are three names
 
 Take a look at the examples directory for a few ways you can specify your database configuration options. Copy a `sequelizer*.yml` example to `config/sequelizer.yml` to use it.
 
+## Read-only DuckDB dataset connections
+
+`Sequelizer.dataset_db` returns a read-only connection to one DuckDB dataset. It needs the `sequel-duckdb` and `duckdb` gems.
+
+```ruby
+db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf",
+                           memory_limit: "51GB", temp_directory: "/scratch/duckdb",
+                           max_temp_directory_size: "500GB")
+db[:patients].count
+```
+
+- `url:` is `duckdb://<directory>` or `duckdb://<file>`. For a directory, the dataset file is the one `*.duckdb` file in `<directory>/<schema>/`. A URL that names an existing DuckDB file uses that file.
+- `schema:` is the schema in the file that holds the dataset's tables. A table name without a schema reads that schema.
+- The connection is an in-memory DuckDB database that attaches the dataset file read-only, so a write to a dataset table fails. A `CREATE` without a catalog goes to the in-memory database.
+- `memory_limit:`, `temp_directory:`, `max_temp_directory_size:` and `threads:` are DuckDB settings. A setting that you do not pass keeps DuckDB's default. DuckDB's default `memory_limit` is 80% of the memory that the container may use, for each DuckDB database in the process.
+- `readonly:` must be true. `readonly: false` raises an `ArgumentError`, because read-write connections are not supported yet.
+- Each process caches one connection for each (`url`, `schema`), and every thread gets the same one. A second call for a cached (`url`, `schema`) with other DuckDB settings raises `Sequelizer::DatasetSettingsConflictError`.
+- The .env file, environment variables and YAML files are not read.
+
+## Loading Sequel extensions
+
+An option named `extension_<name>` loads the Sequel extension `<name>` on each connection that Sequelizer makes. The option's value does not matter. For example, this line in .env loads `funky`:
+
+```
+SEQUELIZER_EXTENSION_FUNKY=1
+```
+
+In config/sequelizer.yml, write `extension_funky: 1`. You can also load an extension yourself with `db.extension :funky`.
+
+An option named `<database type>_db_opt_<setting>` becomes a `SET <setting>=<value>` statement on each new connection. For example, `postgres_db_opt_work_mem: 256MB` runs `SET work_mem=256MB`.
+
+## Sequel extensions
+
+Sequelizer ships these Sequel extensions. You can load each one on any Sequel connection, not only on one that Sequelizer made.
+
+- **cold_col**: gives the columns of a dataset without a query to the database. It reads table schemas from a YAML file (`db.load_schema`), from `db.add_table_schema`, and from the tables and views that the session creates.
+- **db_opts**: applies the `<database type>_db_opt_<setting>` options above. Sequelizer loads it on every connection.
+- **funky**: functions that are written differently on Spark and on DuckDB, through `db.funky`. It supports only those two databases, and it raises an error on any other.
+  - `db.funky.hash(*parts)` returns a signed 64-bit integer that hashes the parts. The default strategy, `:md5`, gives the same id on DuckDB and on Spark when each part turns into the same string on both. That is true for strings, integers, booleans, decimals, dates and timestamps without a time zone. It is false for doubles of 10,000,000 or more, or under 0.001, in absolute value, and for DuckDB's TIMESTAMPTZ. Cast such a part to a decimal or a timestamp first.
+  - `db.funky.hash(*parts, strategy: :legacy)` gives the ids that sequelizer 0.3 and earlier built. Use it only to reproduce those ids, because on DuckDB it gives duplicate ids. An unknown strategy raises `Sequel::Funky::UnknownHashStrategyError`.
+  - `str_to_date(value, format)`, `collect_list(column)` and `make_json_column(dataset, key_column, value_column)` write the same operation for each database.
+- **make_readyable**: `db.make_ready(use_schema:, search_path:, only:, except:)` prepares a Spark or DuckDB database for queries. It creates temporary views for the tables in a list of schemas, in order of precedence, and for external Parquet, ORC or other files given as `Pathname`s.
+- **more_sql**: `db.create_schema(name, if_not_exists: true)`.
+- **platform**: `db.platform` answers what the database supports and prefers (`supports?(:cte)`, `prefers?(:parquet)`, `[:interim_schema]`), and translates functions such as `date_diff(from, to)` and `days_between(from, to)`. The answers come from the CSV files in config/platforms/.
+- **settable**: `db.set(timezone: "UTC")` runs `SET timezone=UTC`. A value is written into the statement as it is. Wrap a value in `Sequel.expr` when the database needs a quoted literal. `db.set_sql` returns the statements and runs nothing.
+- **smart_select_remove**: replaces Sequel's `select_remove`. It finds the columns from the dataset's own SELECT list, or through `from_self` chains, so it does not need to query the database. It raises `Sequel::SmartSelectRemove::NoColumnsError` when it cannot find the columns, and `Sequel::SmartSelectRemove::AllColumnsRemovedError` when every column would be removed. In both cases Sequel's own `select_remove` would produce `SELECT *`, which still contains the removed columns.
+- **sql_recorder**: `db.sql_recorder` is an array of every SQL statement sent to the database. Call `db.sql_recorder.clear` to empty it.
+- **unionize**: `db.unionize(datasets, chunk_size: 100, all: true)` combines many datasets with UNION, one chunk at a time, for databases that cannot run one very large UNION. Each chunk goes into a temporary view on Spark and a temporary table on DuckDB. It supports only those two databases.
+- **usable**: `db.use(:my_schema)` runs a USE statement that switches to that schema.
+
 ## Frustrations
 
 I can't seem to figure out a way to avoid having to specify the database gem in the a user's bundler file. If anyone has ideas on how to automagically load the correct database gem based on the database options fed to Sequelizer, please let me know (#1)!

data/lib/sequel/extensions/funky.rb

@@ -28,7 +28,17 @@ module Sequel
       # strategy: :md5 (the default) casts each part to a string, writes a NULL
       # part as chr(30), joins the parts with chr(31), and reads the first 16
       # hex characters of their md5 as a signed integer. It gives the same id
-      # on DuckDB and on Spark.
+      # on DuckDB and on Spark when each part turns into the same string on
+      # both. Measured on DuckDB 1.5 and Spark 3.5, that is true for strings,
+      # integers, booleans, decimals, dates, timestamps without a time zone,
+      # and doubles that are zero or from 0.001 up to (not including)
+      # 10,000,000 in absolute value. It is false for other doubles, which
+      # Spark writes in E notation (1.0E7) and DuckDB does not (10000000.0),
+      # and for DuckDB's TIMESTAMPTZ, whose text ends in a UTC offset.
+      # Before you pass such a part, cast a TIMESTAMPTZ to a timestamp, and
+      # cast a double to a decimal. The decimal gives the same text for
+      # 123456789.5, but not for 1e20 at DECIMAL(38,6), where DuckDB keeps
+      # the double's rounding error.
       #
       # strategy: :legacy is each database's form before :md5 existed:
       # hash(concat(...)) on DuckDB and xxhash64(...) on Spark. Use it only to

data/lib/sequelizer/duckdb_dataset.rb

@@ -0,0 +1,157 @@
+require "pathname"
+require_relative "connection_maker"
+
+module Sequelizer
+  # Raised when a dataset URL is not a duckdb:// URL.
+  class UnsupportedDatasetURLError < ArgumentError
+    def initialize(url)
+      super("dataset_db supports only duckdb:// URLs, not #{url.inspect}")
+    end
+  end
+
+  # Raised when a dataset connection is asked to write.
+  class ReadWriteDatasetError < ArgumentError
+    def initialize
+      super("dataset_db supports only readonly: true; read-write dataset connections are not supported yet")
+    end
+  end
+
+  # Raised when a directory URL has no dataset directory for the schema, or
+  # when that directory does not hold exactly one DuckDB file.
+  class MissingDatasetError < ArgumentError
+    def initialize(directory, schema, files)
+      super("#{directory}/#{schema} must hold exactly one *.duckdb file to be the dataset #{schema}, " \
+            "but it holds #{files.size}#{": #{files.join(", ")}" unless files.empty?}")
+    end
+  end
+
+  # Raised when dataset_db is called again for a cached (url, schema) with
+  # other DuckDB settings than the cached connection uses.
+  class DatasetSettingsConflictError < ArgumentError
+    def initialize(url, schema, held, requested)
+      super("the dataset connection for url #{url.inspect}, schema #{schema.inspect} is already open " \
+            "in this process with the DuckDB settings #{held.inspect}, not the requested settings " \
+            "#{requested.inspect}. The first dataset_db call for a (url, schema) fixes its settings.")
+    end
+  end
+
+  # = DuckDBDataset
+  #
+  # A read-only connection to one dataset in DuckDB. The connection is an
+  # in-memory DuckDB database, which attaches the dataset's file read-only
+  # as the catalog +dataset+. The search path is <tt>memory.main</tt>, then
+  # <tt>dataset.<schema></tt>, so a table name without a schema reads the
+  # dataset, and a CREATE without a catalog goes to the in-memory database.
+  #
+  # The URL is <tt>duckdb://<path></tt>. When +path+ is a file, it is the
+  # dataset file. Otherwise +path+ is a directory of datasets, and the dataset
+  # file is the one <tt>*.duckdb</tt> file in <tt><path>/<schema></tt>.
+  #
+  # Use Sequelizer.dataset_db, which caches one connection per (url, schema).
+  #
+  # @example
+  #   DuckDBDataset.new(url: "duckdb:///data/datasets", schema: "synpuf").connection
+  class DuckDBDataset
+    URL_PREFIX = "duckdb://".freeze
+    MEMORY_URL = "duckdb::memory:".freeze
+
+    # The dataset file is attached under this fixed catalog name. A catalog
+    # named like its file can equal the schema inside it (seer_x.duckdb holds
+    # the schema seer_x), and DuckDB then stops with "Ambiguous reference to
+    # catalog or schema".
+    DATASET_CATALOG = "dataset".freeze
+
+    # The catalog of the in-memory database. It stays first on the search
+    # path: the first entry is the default database, which DuckDB cannot
+    # detach, and a read-only default database makes every CREATE without a
+    # catalog fail.
+    MEMORY_CATALOG = "memory".freeze
+
+    # The DuckDB settings that dataset_db accepts.
+    SETTINGS = %i[memory_limit temp_directory max_temp_directory_size threads].freeze
+
+    # Whether +url+ names a directory of datasets: a duckdb:// URL whose path
+    # is not an existing file.
+    #
+    # @param url [String] a database URL
+    # @return [Boolean]
+    def self.directory_url?(url)
+      url.to_s.start_with?(URL_PREFIX) && !Pathname(url.to_s.delete_prefix(URL_PREFIX)).file?
+    end
+
+    # The settings Hash in the one form that dataset_db compares: String
+    # names and String values, without the settings that are nil.
+    #
+    # @param settings [Hash{Symbol => Object}] DuckDB settings
+    # @return [Hash{String => String}]
+    def self.normalize_settings(settings)
+      settings.compact.to_h { |name, value| [ name.to_s, value.to_s ] }
+    end
+
+    # @!attribute [r] url
+    #   @return [String] the duckdb:// URL of the dataset file or directory
+    # @!attribute [r] schema
+    #   @return [String] the dataset's schema name
+    # @!attribute [r] settings
+    #   @return [Hash{String => String}] the DuckDB settings
+    attr_reader :url, :schema, :settings
+
+    # @param url [String] a duckdb:// URL of a dataset file or of a directory
+    #   of datasets
+    # @param schema [String, Symbol] the schema that holds the dataset's tables
+    # @param settings [Hash{String => String}] DuckDB settings, already
+    #   normalized (see normalize_settings)
+    # @raise [UnsupportedDatasetURLError] if +url+ is not a duckdb:// URL
+    def initialize(url:, schema:, settings: {})
+      raise UnsupportedDatasetURLError, url unless url.to_s.start_with?(URL_PREFIX)
+
+      @url = url.to_s
+      @schema = schema.to_s
+      @settings = settings
+    end
+
+    # The DuckDB file that holds the dataset.
+    #
+    # @return [Pathname] the absolute path of the dataset file
+    # @raise [MissingDatasetError] if a directory URL has no single dataset
+    #   file for the schema
+    def dataset_file
+      path = Pathname(url.delete_prefix(URL_PREFIX)).expand_path
+      return path unless self.class.directory_url?(url)
+
+      dir = path + schema
+      files = dir.directory? ? dir.glob("*.duckdb").sort : []
+      raise MissingDatasetError.new(path, schema, files) unless files.one?
+
+      files.first
+    end
+
+    # Opens the connection. Every new pool connection attaches the dataset
+    # file read-only and sets its search path, in :after_connect. DuckDB keeps
+    # an ATTACH for the whole database until its last connection closes, but
+    # the search path is per connection, so both run on each connection.
+    # :after_connect gets the driver's connection, which runs only SQL
+    # strings, so it runs the SQL of sequel-duckdb's attach, not db.attach.
+    #
+    # The configuration sources (.env, environment variables and the YAML
+    # files) are ignored: the URL and the settings passed here decide the
+    # connection.
+    #
+    # @return [Sequel::Database] the connection, already tested
+    # @raise [MissingDatasetError] if the dataset file cannot be found
+    # @raise [Sequel::DatabaseConnectionError] if DuckDB cannot attach the file
+    def connection
+      file = dataset_file
+      sqls = nil
+      after_connect = ->(conn) { sqls.each { |sql| conn.query(sql) } }
+      db = ConnectionMaker.new(url: MEMORY_URL, config: settings, after_connect: after_connect,
+                               test: false, ignore_yaml: true, ignore_env: true).connection
+      sqls = [
+        db.attach_sql(file.to_s, as: DATASET_CATALOG, read_only: true, if_not_exists: true),
+        "SET search_path = #{db.literal("#{MEMORY_CATALOG}.main,#{DATASET_CATALOG}.#{schema}")}"
+      ]
+      db.test_connection
+      db
+    end
+  end
+end

data/lib/sequelizer/version.rb

@@ -1,4 +1,4 @@
 module Sequelizer
   # Version for the gem
-  VERSION = "0.4.0".freeze
+  VERSION = "0.4.1".freeze
 end

data/lib/sequelizer.rb

@@ -1,5 +1,6 @@
 require_relative "sequelizer/version"
 require_relative "sequelizer/connection_maker"
+require_relative "sequelizer/duckdb_dataset"
 require_relative "sequel/extensions/db_opts"
 require_relative "sequel/extensions/settable"
 
@@ -48,6 +49,68 @@ module Sequelizer
     Options.new.to_hash
   end
 
+  @dataset_dbs = {}
+  @dataset_dbs_mutex = Mutex.new
+
+  # Returns a read-only connection to one DuckDB dataset, cached per process
+  # by (url, schema). Every thread that asks for the same (url, schema) gets
+  # the same Sequel::Database.
+  #
+  # The connection is an in-memory DuckDB database that attaches the dataset
+  # file read-only, so a write to a dataset table fails. Its search path puts
+  # the dataset's schema after the in-memory database, so a table name without
+  # a schema reads the dataset (<tt>db[:patients]</tt>), and a CREATE without
+  # a catalog goes to the in-memory database. See DuckDBDataset.
+  #
+  # The DuckDB settings apply to the in-memory database when it opens. A
+  # setting that is nil keeps DuckDB's default. DuckDB's default memory_limit
+  # is 80% of the memory that the container may use, for each DuckDB database
+  # in the process, so pass memory_limit where more than one runs.
+  #
+  # The .env file, environment variables and YAML files are not read.
+  #
+  # @param url [String] <tt>duckdb://<directory></tt>, a directory that holds
+  #   one directory for each dataset, <tt><directory>/<schema>/</tt>, with
+  #   exactly one <tt>*.duckdb</tt> file in it; or <tt>duckdb://<file></tt>,
+  #   an existing DuckDB file, which is attached directly
+  # @param schema [String, Symbol] the schema in the DuckDB file that holds the
+  #   dataset's tables
+  # @param readonly [Boolean] must be true; read-write connections are not
+  #   supported yet
+  # @param memory_limit [String, nil] DuckDB's memory_limit, for example "51GB"
+  # @param temp_directory [String, nil] DuckDB's temp_directory, where it
+  #   spills data that does not fit in memory
+  # @param max_temp_directory_size [String, nil] DuckDB's
+  #   max_temp_directory_size, for example "500GB"
+  # @param threads [Integer, String, nil] DuckDB's threads
+  # @return [Sequel::Database] the cached read-only connection
+  # @raise [ReadWriteDatasetError] if +readonly+ is false
+  # @raise [UnsupportedDatasetURLError] if +url+ is not a duckdb:// URL
+  # @raise [MissingDatasetError] if a directory URL has no single DuckDB file
+  #   for +schema+
+  # @raise [DatasetSettingsConflictError] if (url, schema) is cached with other
+  #   DuckDB settings
+  #
+  # @example
+  #   db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf",
+  #                              memory_limit: "51GB", temp_directory: "/scratch/duckdb",
+  #                              max_temp_directory_size: "500GB")
+  #   db[:patients].count
+  def self.dataset_db(url:, schema:, readonly: true, memory_limit: nil, temp_directory: nil,
+                      max_temp_directory_size: nil, threads: nil)
+    raise ReadWriteDatasetError unless readonly
+
+    settings = DuckDBDataset.normalize_settings(memory_limit:, temp_directory:, max_temp_directory_size:, threads:)
+    key = [ url.to_s, schema.to_s ]
+
+    @dataset_dbs_mutex.synchronize do
+      cached = @dataset_dbs[key] ||= { db: DuckDBDataset.new(url:, schema:, settings:).connection, settings: }
+      raise DatasetSettingsConflictError.new(*key, cached[:settings], settings) unless cached[:settings] == settings
+
+      cached[:db]
+    end
+  end
+
   # Instantiates and memoizes a database connection. The +db+ method instantiates
   # the connection on the first call and then memoizes itself so only a single
   # connection is used on repeated calls.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sequelizer
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.1
 platform: ruby
 authors:
 - Ryan Duryea
@@ -285,6 +285,7 @@ files:
 - lib/sequelizer.rb
 - lib/sequelizer/cli.rb
 - lib/sequelizer/connection_maker.rb
+- lib/sequelizer/duckdb_dataset.rb
 - lib/sequelizer/env_config.rb
 - lib/sequelizer/options.rb
 - lib/sequelizer/version.rb