sequelizer 0.4.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 50510a32c25ed7c0218c60a7ba9d615a2ccf909685cc719dc1110a3fe43a5baf
-  data.tar.gz: a11d4852a6f18ffa2fca2fb5a9a75b67fd6c605d8d38321cb2b7a042e7ed8427
+  metadata.gz: 99fc6e668db9f60787a98b20cb018fbfe3efe4281a77be3717a418e91d213193
+  data.tar.gz: df0939af163eff7f74e4cd19593b7e4a29cadac5800dcf821e9d727cc663730c
 SHA512:
-  metadata.gz: 4f859dc2afb3bf840d8f2a53215ba7c2437a511eafdd23467cdab868092c45a3e4a8772159ca9098f1b808eda15041d357b05d7fd2f0e7c0f4acc7ec82c68df5
-  data.tar.gz: 69bb86b1fc5965eef2fa4a5449618050cb984d285c93d99fa2e56269c9d2967573f2b8b918c0a992880f7278a0c55bdd48c165a2e744ccf814f356a8565befdb
+  metadata.gz: 9f090d45429c11b44d00e3b13cd15ee7c96756dc4492d20d113992270d01619611568f49e3271226ef4823bd704d8cb4da570b423dde2f18351d78c982b58fe2
+  data.tar.gz: 38f2a9007b8829bce8b8ac4a7e5c8d08d862562639a1c8f1fe0926ce523f77c31cad5656853181e74c7ad07c8d6683f4ebfa38807f869f30cdaebd4db29877b2

data/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.5.0] - 2026-11-02
+
+### Breaking changes
+
+- **dataset_db:** Attach every DuckDB dataset to one database per process ([4c81a9f](https://github.com/outcomesinsights/sequelizer/commit/4c81a9fb557674ab0b539798185ec3fee9f27385)): Sequelizer::DuckDBDataset::DATASET_CATALOG and the catalog name "dataset" are gone; use Sequelizer.dataset_catalog(schema). dataset_db returns one Sequel::Database for every dataset, and an unqualified table name reads a dataset only inside with_dataset. The DuckDB settings are per process, not per dataset. DuckDBDataset#connection and DuckDBDataset.normalize_settings are removed.
+
+### Added
+
+- **dataset_db:** Add search_path_prefix to with_dataset ([4cda3f4](https://github.com/outcomesinsights/sequelizer/commit/4cda3f4e3fd0e56426ecf3bc4497f3580eae8ed4))
+- **dataset_db:** Add Sequelizer.close_dataset_dbs ([a309e22](https://github.com/outcomesinsights/sequelizer/commit/a309e22fa48d5492f1f3c2193f7723bb02ffa53b))
+
+### Fixed
+
+- **dataset_db:** Key the connection cache on the resolved dataset file ([50f6242](https://github.com/outcomesinsights/sequelizer/commit/50f6242a07017f4401a789d78a961213e193726c))
+- **dataset_db:** Say when the URL's path is neither a file nor a directory ([5182707](https://github.com/outcomesinsights/sequelizer/commit/51827072d29181859216cbf89d1f6121ab5630da))
+
 ## [0.4.1] - 2026-10-29
 
 ### Added
@@ -180,3 +196,4 @@ All notable changes to this project will be documented in this file.
 
 [0.4.0]: https://github.com/outcomesinsights/sequelizer/compare/v0.3.0...v0.4.0
 [0.4.1]: https://github.com/outcomesinsights/sequelizer/compare/v0.4.0...v0.4.1
+[0.5.0]: https://github.com/outcomesinsights/sequelizer/compare/v0.4.1...v0.5.0

data/README.md

@@ -78,23 +78,47 @@ For PostgreSQL, `search_path`, `schema_search_path` and `schema` are three names
 
 Take a look at the examples directory for a few ways you can specify your database configuration options. Copy a `sequelizer*.yml` example to `config/sequelizer.yml` to use it.
 
-## Read-only DuckDB dataset connections
+## Read-only DuckDB datasets
 
-`Sequelizer.dataset_db` returns a read-only connection to one DuckDB dataset. It needs the `sequel-duckdb` and `duckdb` gems.
+Sequelizer attaches DuckDB datasets read-only to one in-memory DuckDB database for each process. It needs the `sequel-duckdb` and `duckdb` gems.
 
 ```ruby
-db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf",
-                           memory_limit: "51GB", temp_directory: "/scratch/duckdb",
-                           max_temp_directory_size: "500GB")
-db[:patients].count
+opts = { memory_limit: "51GB", temp_directory: "/scratch/duckdb", max_temp_directory_size: "500GB",
+         max_connections: 5 }
+
+# Name a dataset's tables with its catalog and schema anywhere:
+db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf", **opts)
+db[Sequel[Sequelizer.dataset_catalog(:synpuf)][:synpuf][:patients]].count
+
+# Or read them without a schema, as ConceptQL's SQL does, inside with_dataset:
+Sequelizer.with_dataset(url: "duckdb:///data/datasets", schema: "synpuf", **opts) do |db|
+  db[:patients].count
+end
+```
+
+- `url:` is `duckdb://<directory>` or `duckdb://<file>`, with the scheme in any case. For a directory, the dataset file is the one `*.duckdb` file in `<directory>/<schema>/`. A URL that names an existing DuckDB file uses that file.
+- `schema:` is the schema in the file that holds the dataset's tables.
+- `dataset_db` attaches the dataset file at the first call for it and returns the process's one `Sequel::Database`. Every dataset uses the same one. The file is attached under the catalog `dataset_<schema>`, which `Sequelizer.dataset_catalog(schema)` returns as a Symbol. A write to a dataset table fails.
+- Outside `with_dataset`, name a table with all three parts: catalog, schema and table. DuckDB does not find `synpuf.patients` or `patients` there, so a query that misses `with_dataset` fails with an error instead of reading another dataset.
+- `with_dataset` runs the block on one connection whose search path is `memory.main`, then the dataset's schema. When the block ends, the search path is what it was before. A lazy dataset that runs after the block, or a query in a thread that the block starts, uses no dataset path and fails.
+- A `CREATE` without a catalog goes to the in-memory database. Inside `with_dataset`, a table created there hides the dataset table of the same name. The in-memory database is shared by every dataset and thread in the process.
+- `with_dataset(search_path_prefix:)` puts other search path entries first, for example a read-write catalog that you attached to the dataset database. It takes one entry or a list. An entry is a catalog, which DuckDB reads as `<catalog>.main`, or `catalog.schema`. A `CREATE` without a catalog then goes to the first entry. DuckDB raises a Catalog Error before the block runs if an entry is not attached. An empty entry, or one with a comma, raises `Sequelizer::SearchPathEntryError`.
+
+```ruby
+db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf")
+db.attach("/scratch/working.duckdb", as: "working")
+Sequelizer.with_dataset(url: "duckdb:///data/datasets", schema: "synpuf", search_path_prefix: "working.main") do |d|
+  d.create_table(:results) { Integer :person_id } # creates working.main.results
+end
 ```
 
-- `url:` is `duckdb://<directory>` or `duckdb://<file>`. For a directory, the dataset file is the one `*.duckdb` file in `<directory>/<schema>/`. A URL that names an existing DuckDB file uses that file.
-- `schema:` is the schema in the file that holds the dataset's tables. A table name without a schema reads that schema.
-- The connection is an in-memory DuckDB database that attaches the dataset file read-only, so a write to a dataset table fails. A `CREATE` without a catalog goes to the in-memory database.
-- `memory_limit:`, `temp_directory:`, `max_temp_directory_size:` and `threads:` are DuckDB settings. A setting that you do not pass keeps DuckDB's default. DuckDB's default `memory_limit` is 80% of the memory that the container may use, for each DuckDB database in the process.
+- A schema name can come from only one file in a process. A second file with the same schema raises `Sequelizer::DatasetSchemaConflictError`. Two URLs for one file (`duckdb:///data/datasets`, `duckdb:///data/datasets/` and the file's own URL) are one dataset.
+- `memory_limit:`, `temp_directory:`, `max_temp_directory_size:` and `threads:` are DuckDB settings of the one database, so the limits are for all the datasets together. `max_connections:` is the size of its connection pool (Sequel's default is 4). The first call fixes all of them, and a later call with other values raises `Sequelizer::DatasetSettingsConflictError`. A setting that you do not pass keeps DuckDB's default. DuckDB's default `memory_limit` is 80% of the memory that the container may use.
+- Two processes must not share one `temp_directory`. DuckDB gives the spill files the same names in every database, so the processes corrupt each other's spills.
 - `readonly:` must be true. `readonly: false` raises an `ArgumentError`, because read-write connections are not supported yet.
-- Each process caches one connection for each (`url`, `schema`), and every thread gets the same one. A second call for a cached (`url`, `schema`) with other DuckDB settings raises `Sequelizer::DatasetSettingsConflictError`.
+- A call that fails, for example because a file is not a DuckDB file, attaches nothing. When no dataset is attached, the database closes too, so a later call starts clean.
+- An attached file stays attached, so a file that is replaced on disk is not read, even after `disconnect`. `Sequelizer.close_dataset_dbs` detaches every file and closes the database, so the next call opens the files that are on disk then. Call it when no thread uses the database, for example between tests or after a dataset file is replaced. Do not use a database that `dataset_db` returned before the call.
+- A forked child process opens its own database.
 - The .env file, environment variables and YAML files are not read.
 
 ## Loading Sequel extensions

data/lib/sequelizer/duckdb_dataset.rb

@@ -1,5 +1,4 @@
 require "pathname"
-require_relative "connection_maker"
 
 module Sequelizer
   # Raised when a dataset URL is not a duckdb:// URL.
@@ -16,59 +15,68 @@ module Sequelizer
     end
   end
 
-  # Raised when a directory URL has no dataset directory for the schema, or
-  # when that directory does not hold exactly one DuckDB file.
+  # Raised when the URL's path is neither a file nor a directory, when a
+  # directory URL has no dataset directory for the schema, or when that
+  # directory does not hold exactly one DuckDB file.
   class MissingDatasetError < ArgumentError
-    def initialize(directory, schema, files)
-      super("#{directory}/#{schema} must hold exactly one *.duckdb file to be the dataset #{schema}, " \
-            "but it holds #{files.size}#{": #{files.join(", ")}" unless files.empty?}")
+    # @param path [Pathname] the path in the URL
+    # @param schema [String] the dataset's schema
+    # @param files [Array<Pathname>, nil] the DuckDB files in
+    #   <tt><path>/<schema></tt>, or nil when +path+ is neither a file nor a
+    #   directory
+    def initialize(path, schema, files)
+      if files.nil?
+        super("#{path} is neither a DuckDB file nor a directory, so it cannot hold the dataset #{schema}")
+      else
+        super("#{path}/#{schema} must hold exactly one *.duckdb file to be the dataset #{schema}, " \
+              "but it holds #{files.size}#{": #{files.join(", ")}" unless files.empty?}")
+      end
     end
   end
 
-  # Raised when dataset_db is called again for a cached (url, schema) with
-  # other DuckDB settings than the cached connection uses.
+  # Raised when a dataset_db call asks for other DuckDB settings or another
+  # max_connections than the process's dataset database has.
   class DatasetSettingsConflictError < ArgumentError
-    def initialize(url, schema, held, requested)
-      super("the dataset connection for url #{url.inspect}, schema #{schema.inspect} is already open " \
-            "in this process with the DuckDB settings #{held.inspect}, not the requested settings " \
-            "#{requested.inspect}. The first dataset_db call for a (url, schema) fixes its settings.")
+    def initialize(held, requested)
+      super("the dataset database is already open in this process with #{held.inspect}, not the " \
+            "requested #{requested.inspect}. The first dataset_db call fixes them until " \
+            "Sequelizer.close_dataset_dbs.")
+    end
+  end
+
+  # Raised when a with_dataset search_path_prefix entry is empty or holds a
+  # comma. DuckDB separates search path entries with commas, so it would
+  # read such an entry as two entries.
+  class SearchPathEntryError < ArgumentError
+    def initialize(entries)
+      super("each search_path_prefix entry must be one catalog or catalog.schema with no comma, " \
+            "but #{entries.inspect} is not")
+    end
+  end
+
+  # Raised when a dataset's schema is already attached from another file.
+  # The catalog name comes from the schema, so two files with one schema
+  # would need one catalog.
+  class DatasetSchemaConflictError < ArgumentError
+    def initialize(schema, attached_file, requested_file)
+      super("the schema #{schema.inspect} is already attached from #{attached_file}, so " \
+            "#{requested_file} cannot be attached for it in this process")
     end
   end
 
   # = DuckDBDataset
   #
-  # A read-only connection to one dataset in DuckDB. The connection is an
-  # in-memory DuckDB database, which attaches the dataset's file read-only
-  # as the catalog +dataset+. The search path is <tt>memory.main</tt>, then
-  # <tt>dataset.<schema></tt>, so a table name without a schema reads the
-  # dataset, and a CREATE without a catalog goes to the in-memory database.
-  #
-  # The URL is <tt>duckdb://<path></tt>. When +path+ is a file, it is the
+  # The dataset that a duckdb:// URL and a schema name identify. The URL is
+  # <tt>duckdb://<path></tt>, in any case. When +path+ is a file, it is the
   # dataset file. Otherwise +path+ is a directory of datasets, and the dataset
   # file is the one <tt>*.duckdb</tt> file in <tt><path>/<schema></tt>.
   #
-  # Use Sequelizer.dataset_db, which caches one connection per (url, schema).
+  # Sequelizer.dataset_db attaches it; see DuckDBDatasets.
   #
   # @example
-  #   DuckDBDataset.new(url: "duckdb:///data/datasets", schema: "synpuf").connection
+  #   DuckDBDataset.new(url: "duckdb:///data/datasets", schema: "synpuf").dataset_file
   class DuckDBDataset
-    URL_PREFIX = "duckdb://".freeze
-    MEMORY_URL = "duckdb::memory:".freeze
-
-    # The dataset file is attached under this fixed catalog name. A catalog
-    # named like its file can equal the schema inside it (seer_x.duckdb holds
-    # the schema seer_x), and DuckDB then stops with "Ambiguous reference to
-    # catalog or schema".
-    DATASET_CATALOG = "dataset".freeze
-
-    # The catalog of the in-memory database. It stays first on the search
-    # path: the first entry is the default database, which DuckDB cannot
-    # detach, and a read-only default database makes every CREATE without a
-    # catalog fail.
-    MEMORY_CATALOG = "memory".freeze
-
-    # The DuckDB settings that dataset_db accepts.
-    SETTINGS = %i[memory_limit temp_directory max_temp_directory_size threads].freeze
+    URL_PREFIX = %r{\Aduckdb://}i
 
     # Whether +url+ names a directory of datasets: a duckdb:// URL whose path
     # is not an existing file.
@@ -76,48 +84,43 @@ module Sequelizer
     # @param url [String] a database URL
     # @return [Boolean]
     def self.directory_url?(url)
-      url.to_s.start_with?(URL_PREFIX) && !Pathname(url.to_s.delete_prefix(URL_PREFIX)).file?
-    end
-
-    # The settings Hash in the one form that dataset_db compares: String
-    # names and String values, without the settings that are nil.
-    #
-    # @param settings [Hash{Symbol => Object}] DuckDB settings
-    # @return [Hash{String => String}]
-    def self.normalize_settings(settings)
-      settings.compact.to_h { |name, value| [ name.to_s, value.to_s ] }
+      url.to_s.match?(URL_PREFIX) && !Pathname(url.to_s.sub(URL_PREFIX, "")).file?
     end
 
     # @!attribute [r] url
     #   @return [String] the duckdb:// URL of the dataset file or directory
     # @!attribute [r] schema
     #   @return [String] the dataset's schema name
-    # @!attribute [r] settings
-    #   @return [Hash{String => String}] the DuckDB settings
-    attr_reader :url, :schema, :settings
+    attr_reader :url, :schema
 
     # @param url [String] a duckdb:// URL of a dataset file or of a directory
     #   of datasets
     # @param schema [String, Symbol] the schema that holds the dataset's tables
-    # @param settings [Hash{String => String}] DuckDB settings, already
-    #   normalized (see normalize_settings)
     # @raise [UnsupportedDatasetURLError] if +url+ is not a duckdb:// URL
-    def initialize(url:, schema:, settings: {})
-      raise UnsupportedDatasetURLError, url unless url.to_s.start_with?(URL_PREFIX)
+    def initialize(url:, schema:)
+      raise UnsupportedDatasetURLError, url unless url.to_s.match?(URL_PREFIX)
 
       @url = url.to_s
       @schema = schema.to_s
-      @settings = settings
     end
 
-    # The DuckDB file that holds the dataset.
+    # The DuckDB file that holds the dataset. It is found once, on the first
+    # call.
     #
-    # @return [Pathname] the absolute path of the dataset file
-    # @raise [MissingDatasetError] if a directory URL has no single dataset
-    #   file for the schema
+    # @return [Pathname] the real path of the dataset file
+    # @raise [MissingDatasetError] if the URL's path is neither a file nor a
+    #   directory, or if a directory URL has no single dataset file for the
+    #   schema
     def dataset_file
-      path = Pathname(url.delete_prefix(URL_PREFIX)).expand_path
-      return path unless self.class.directory_url?(url)
+      @dataset_file ||= find_dataset_file.realpath
+    end
+
+    private
+
+    def find_dataset_file
+      path = Pathname(url.sub(URL_PREFIX, "")).expand_path
+      return path if path.file?
+      raise MissingDatasetError.new(path, schema, nil) unless path.directory?
 
       dir = path + schema
       files = dir.directory? ? dir.glob("*.duckdb").sort : []
@@ -125,33 +128,5 @@ module Sequelizer
 
       files.first
     end
-
-    # Opens the connection. Every new pool connection attaches the dataset
-    # file read-only and sets its search path, in :after_connect. DuckDB keeps
-    # an ATTACH for the whole database until its last connection closes, but
-    # the search path is per connection, so both run on each connection.
-    # :after_connect gets the driver's connection, which runs only SQL
-    # strings, so it runs the SQL of sequel-duckdb's attach, not db.attach.
-    #
-    # The configuration sources (.env, environment variables and the YAML
-    # files) are ignored: the URL and the settings passed here decide the
-    # connection.
-    #
-    # @return [Sequel::Database] the connection, already tested
-    # @raise [MissingDatasetError] if the dataset file cannot be found
-    # @raise [Sequel::DatabaseConnectionError] if DuckDB cannot attach the file
-    def connection
-      file = dataset_file
-      sqls = nil
-      after_connect = ->(conn) { sqls.each { |sql| conn.query(sql) } }
-      db = ConnectionMaker.new(url: MEMORY_URL, config: settings, after_connect: after_connect,
-                               test: false, ignore_yaml: true, ignore_env: true).connection
-      sqls = [
-        db.attach_sql(file.to_s, as: DATASET_CATALOG, read_only: true, if_not_exists: true),
-        "SET search_path = #{db.literal("#{MEMORY_CATALOG}.main,#{DATASET_CATALOG}.#{schema}")}"
-      ]
-      db.test_connection
-      db
-    end
   end
 end

data/lib/sequelizer/duckdb_datasets.rb

@@ -0,0 +1,171 @@
+require_relative "connection_maker"
+require_relative "duckdb_dataset"
+
+module Sequelizer
+  # = DuckDBDatasets
+  #
+  # The read-only DuckDB datasets of one process. The process has one
+  # in-memory DuckDB database, so memory_limit, temp_directory,
+  # max_temp_directory_size and threads apply once, and the datasets use one
+  # set of spill files. Each dataset file is attached READ_ONLY once, under
+  # the catalog <tt>dataset_<schema></tt>.
+  #
+  # A table named with its catalog, schema and table
+  # (<tt>Sequel[:dataset_synpuf][:synpuf][:patients]</tt>) reads the dataset
+  # on any connection. A name with only a schema and a table, or with only a
+  # table, reads a dataset only inside with_dataset, which sets the
+  # connection's search path for the block.
+  #
+  # The attached files stay attached until close, so a dataset file that is
+  # replaced on disk is not read until then.
+  #
+  # Use Sequelizer.dataset_db, Sequelizer.with_dataset and
+  # Sequelizer.close_dataset_dbs.
+  class DuckDBDatasets
+    MEMORY_URL = "duckdb::memory:".freeze
+
+    # The schema of the in-memory database. It comes first on a dataset's
+    # search path: the first entry is where a CREATE without a catalog goes,
+    # and a read-only dataset there makes every such CREATE fail.
+    MEMORY_SCHEMA = "memory.main".freeze
+
+    # The catalog that the dataset with +schema+ is attached as.
+    #
+    # @param schema [String, Symbol] the dataset's schema
+    # @return [Symbol] <tt>:dataset_<schema></tt>
+    def self.catalog_for(schema) = :"dataset_#{schema}"
+
+    def initialize
+      @mutex = Mutex.new
+      reset
+    end
+
+    # Attaches the dataset, if it is not attached yet.
+    #
+    # @param url [String] a duckdb:// URL (see DuckDBDataset)
+    # @param schema [String, Symbol] the dataset's schema
+    # @param options [Hash] <tt>{ settings: Hash{String => String},
+    #   max_connections: Integer or nil }</tt>
+    # @return [Array(Sequel::Database, String)] the process's dataset database
+    #   and the dataset's catalog
+    # @raise [DatasetSettingsConflictError] if +options+ differ from the open
+    #   database's
+    # @raise [DatasetSchemaConflictError] if the schema is attached from
+    #   another file
+    def attach(url:, schema:, options:)
+      dataset = DuckDBDataset.new(url:, schema:)
+      file = dataset.dataset_file.to_s
+
+      @mutex.synchronize do
+        forget_parent_process
+        raise DatasetSettingsConflictError.new(@options, options) if @db && @options != options
+
+        begin
+          @db ||= connect(options)
+          @options = options
+          [ @db, attach_file(file, dataset.schema) ]
+        rescue StandardError
+          close_db if @schemas.empty?
+          raise
+        end
+      end
+    end
+
+    # Attaches the dataset, holds one connection for the block, and sets the
+    # connection's search path to the +search_path_prefix+ entries, then
+    # <tt>memory.main,<catalog>.<schema></tt>, so a table name without a
+    # schema reads the dataset. At the end of the block, the search path is
+    # what it was before.
+    #
+    # @param search_path_prefix [Array<String, Symbol>] DuckDB search path
+    #   entries to put first
+    # @yieldparam db [Sequel::Database] the process's dataset database
+    # @return [Object] the block's value
+    # @raise [SearchPathEntryError] if an entry is empty or holds a comma
+    def with_dataset(url:, schema:, options:, search_path_prefix: [])
+      prefix = search_path_prefix.map(&:to_s)
+      bad = prefix.select { |entry| entry.empty? || entry.include?(",") }
+      raise SearchPathEntryError, bad unless bad.empty?
+
+      db, catalog = attach(url:, schema:, options:)
+      path = [ *prefix, MEMORY_SCHEMA, "#{catalog}.#{schema}" ].join(",")
+
+      db.synchronize do
+        previous = db.get(Sequel.function(:current_setting, "search_path"))
+        db.run("SET search_path = #{db.literal(path)}")
+        begin
+          yield db
+        ensure
+          db.run("SET search_path = #{db.literal(previous)}")
+        end
+      end
+    end
+
+    # Detaches every dataset file and disconnects the database. The next
+    # attach opens a new database.
+    #
+    # @return [void]
+    def close
+      @mutex.synchronize do
+        forget_parent_process
+        close_db
+      end
+    end
+
+    private
+
+    def reset
+      @pid = Process.pid
+      @db = nil
+      @options = nil
+      @catalogs = {}
+      @schemas = {}
+    end
+
+    # A forked child inherits the parent's database, but its DuckDB instance
+    # belongs to the parent. The child opens its own.
+    def forget_parent_process
+      reset unless @pid == Process.pid
+    end
+
+    # The catalog of +file+, which this attaches the first time. The catalog
+    # name has a prefix, because a catalog named like a schema in its file
+    # makes DuckDB stop with "Ambiguous reference to catalog or schema".
+    def attach_file(file, schema)
+      if (held = @schemas[schema])
+        raise DatasetSchemaConflictError.new(schema, held[:file], file) unless held[:file] == file
+
+        return held[:catalog]
+      end
+
+      catalog = @catalogs[file]
+      unless catalog
+        catalog = self.class.catalog_for(schema).to_s
+        @db.attach(file, as: catalog, read_only: true)
+        @catalogs[file] = catalog
+      end
+      @schemas[schema] = { file:, catalog: }
+      catalog
+    end
+
+    def close_db
+      if @db
+        @catalogs.each_value { |catalog| @db.detach(catalog, if_exists: true) }
+        @db.disconnect
+      end
+      reset
+    end
+
+    # The in-memory database. The configuration sources (.env, environment
+    # variables and the YAML files) are ignored, and Sequel::DATABASES does
+    # not hold it, so a closed database can be garbage collected.
+    def connect(options)
+      opts = { url: MEMORY_URL, config: options[:settings], test: false, keep_reference: false,
+               ignore_yaml: true, ignore_env: true }
+      opts[:max_connections] = options[:max_connections] if options[:max_connections]
+      db = ConnectionMaker.new(opts).connection
+      db.test_connection
+      db
+    end
+  end
+end

data/lib/sequelizer/version.rb

@@ -1,4 +1,4 @@
 module Sequelizer
   # Version for the gem
-  VERSION = "0.4.1".freeze
+  VERSION = "0.5.0".freeze
 end

data/lib/sequelizer.rb

@@ -1,6 +1,6 @@
 require_relative "sequelizer/version"
 require_relative "sequelizer/connection_maker"
-require_relative "sequelizer/duckdb_dataset"
+require_relative "sequelizer/duckdb_datasets"
 require_relative "sequel/extensions/db_opts"
 require_relative "sequel/extensions/settable"
 
@@ -49,30 +49,43 @@ module Sequelizer
     Options.new.to_hash
   end
 
-  @dataset_dbs = {}
-  @dataset_dbs_mutex = Mutex.new
+  @duckdb_datasets = DuckDBDatasets.new
 
-  # Returns a read-only connection to one DuckDB dataset, cached per process
-  # by (url, schema). Every thread that asks for the same (url, schema) gets
-  # the same Sequel::Database.
+  # Attaches one DuckDB dataset read-only and returns the process's dataset
+  # database: one in-memory DuckDB database that every dataset in the process
+  # shares. The dataset file is attached at the first call for it, under the
+  # catalog <tt>dataset_<schema></tt>, and stays attached until
+  # close_dataset_dbs. Every call returns the same Sequel::Database.
   #
-  # The connection is an in-memory DuckDB database that attaches the dataset
-  # file read-only, so a write to a dataset table fails. Its search path puts
-  # the dataset's schema after the in-memory database, so a table name without
-  # a schema reads the dataset (<tt>db[:patients]</tt>), and a CREATE without
-  # a catalog goes to the in-memory database. See DuckDBDataset.
+  # Name a dataset's tables with the catalog, the schema and the table:
+  # <tt>Sequel[Sequelizer.dataset_catalog(:synpuf)][:synpuf][:patients]</tt>.
+  # A name with only a schema and a table, or with only a table, reads a
+  # dataset only inside with_dataset. Outside it, DuckDB raises a Catalog
+  # Error. A write to a dataset table fails.
   #
-  # The DuckDB settings apply to the in-memory database when it opens. A
-  # setting that is nil keeps DuckDB's default. DuckDB's default memory_limit
-  # is 80% of the memory that the container may use, for each DuckDB database
-  # in the process, so pass memory_limit where more than one runs.
+  # The DuckDB settings and +max_connections+ apply once, to the one
+  # database, so memory_limit and max_temp_directory_size are limits for all
+  # the datasets together. The first call fixes them until close_dataset_dbs.
+  # A setting that is nil keeps DuckDB's default. DuckDB's default
+  # memory_limit is 80% of the memory that the container may use, for each
+  # DuckDB database. Two processes must not share one temp_directory: DuckDB
+  # gives the spill files the same names in every database.
+  #
+  # Each schema can come from one file in a process, because the catalog name
+  # comes from the schema. The dataset file is the real path of the file that +url+
+  # resolves to, so two URLs for one file are one dataset.
+  #
+  # A file that is replaced on disk after it is attached is not read, even
+  # after +disconnect+. Call close_dataset_dbs, or restart the process, to
+  # read the new file.
   #
   # The .env file, environment variables and YAML files are not read.
   #
   # @param url [String] <tt>duckdb://<directory></tt>, a directory that holds
   #   one directory for each dataset, <tt><directory>/<schema>/</tt>, with
   #   exactly one <tt>*.duckdb</tt> file in it; or <tt>duckdb://<file></tt>,
-  #   an existing DuckDB file, which is attached directly
+  #   an existing DuckDB file, which is attached directly. The scheme can be
+  #   in any case.
   # @param schema [String, Symbol] the schema in the DuckDB file that holds the
   #   dataset's tables
   # @param readonly [Boolean] must be true; read-write connections are not
@@ -83,33 +96,103 @@ module Sequelizer
   # @param max_temp_directory_size [String, nil] DuckDB's
   #   max_temp_directory_size, for example "500GB"
   # @param threads [Integer, String, nil] DuckDB's threads
-  # @return [Sequel::Database] the cached read-only connection
+  # @param max_connections [Integer, nil] the size of the database's
+  #   connection pool, for example the number of threads that use it; nil
+  #   keeps Sequel's default of 4
+  # @return [Sequel::Database] the process's dataset database
   # @raise [ReadWriteDatasetError] if +readonly+ is false
   # @raise [UnsupportedDatasetURLError] if +url+ is not a duckdb:// URL
-  # @raise [MissingDatasetError] if a directory URL has no single DuckDB file
-  #   for +schema+
-  # @raise [DatasetSettingsConflictError] if (url, schema) is cached with other
-  #   DuckDB settings
+  # @raise [MissingDatasetError] if the URL's path is neither a file nor a
+  #   directory, or if a directory URL has no single DuckDB file for +schema+
+  # @raise [DatasetSettingsConflictError] if the database is open with other
+  #   DuckDB settings or another max_connections
+  # @raise [DatasetSchemaConflictError] if +schema+ is attached from another
+  #   file
   #
   # @example
   #   db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf",
   #                              memory_limit: "51GB", temp_directory: "/scratch/duckdb",
-  #                              max_temp_directory_size: "500GB")
-  #   db[:patients].count
-  def self.dataset_db(url:, schema:, readonly: true, memory_limit: nil, temp_directory: nil,
-                      max_temp_directory_size: nil, threads: nil)
-    raise ReadWriteDatasetError unless readonly
+  #                              max_temp_directory_size: "500GB", max_connections: 5)
+  #   db[Sequel[Sequelizer.dataset_catalog(:synpuf)][:synpuf][:patients]].count
+  def self.dataset_db(url:, schema:, **options)
+    @duckdb_datasets.attach(url:, schema:, options: dataset_options(**options)).first
+  end
+
+  # Attaches the dataset like dataset_db, and runs the block on one
+  # connection whose search path is <tt>memory.main</tt>, then the dataset's
+  # schema. A table name without a schema reads the dataset in the block,
+  # as ConceptQL's SQL needs (<tt>db[:patients]</tt>). The search path is
+  # restored when the block ends, so a query outside the block, or a lazy
+  # dataset that runs after it, finds no dataset table and fails.
+  #
+  # A CREATE without a catalog goes to the in-memory database, which comes
+  # first on the search path, so a created table hides the dataset table of
+  # the same name in the block. The in-memory database is shared by every
+  # dataset and thread in the process.
+  #
+  # +search_path_prefix+ puts other DuckDB search path entries first, for
+  # example a read-write catalog that the caller attached to the dataset
+  # database. A CREATE without a catalog then goes to the first entry. An
+  # entry is a catalog (DuckDB reads +working+ as <tt>working.main</tt>) or
+  # <tt>catalog.schema</tt>. DuckDB raises a Catalog Error, before the block
+  # runs, for an entry that is not attached.
+  #
+  # @param (see dataset_db)
+  # @param search_path_prefix [String, Symbol, Array<String, Symbol>] the
+  #   search path entries that come before <tt>memory.main</tt> and the
+  #   dataset, in order
+  # @yieldparam db [Sequel::Database] the process's dataset database
+  # @return [Object] the block's value
+  # @raise [SearchPathEntryError] if a search_path_prefix entry is empty or
+  #   holds a comma
+  #
+  # @example
+  #   Sequelizer.with_dataset(url: "duckdb:///data/datasets", schema: "synpuf") do |db|
+  #     db[:patients].count
+  #   end
+  #
+  # @example A read-write catalog first
+  #   db = Sequelizer.dataset_db(url: "duckdb:///data/datasets", schema: "synpuf")
+  #   db.attach("/scratch/working.duckdb", as: "working")
+  #   Sequelizer.with_dataset(url: "duckdb:///data/datasets", schema: "synpuf",
+  #                           search_path_prefix: "working.main") do |d|
+  #     d.create_table(:results) { Integer :person_id } # working.main.results
+  #   end
+  def self.with_dataset(url:, schema:, search_path_prefix: [], **options, &)
+    @duckdb_datasets.with_dataset(url:, schema:, options: dataset_options(**options),
+                                  search_path_prefix: Array(search_path_prefix), &)
+  end
 
-    settings = DuckDBDataset.normalize_settings(memory_limit:, temp_directory:, max_temp_directory_size:, threads:)
-    key = [ url.to_s, schema.to_s ]
+  # The catalog that the dataset with +schema+ is attached as, for a table
+  # name outside with_dataset:
+  # <tt>Sequel[Sequelizer.dataset_catalog(:synpuf)][:synpuf][:patients]</tt>.
+  #
+  # @param schema [String, Symbol] the dataset's schema
+  # @return [Symbol] <tt>:dataset_<schema></tt>
+  def self.dataset_catalog(schema) = DuckDBDatasets.catalog_for(schema)
 
-    @dataset_dbs_mutex.synchronize do
-      cached = @dataset_dbs[key] ||= { db: DuckDBDataset.new(url:, schema:, settings:).connection, settings: }
-      raise DatasetSettingsConflictError.new(*key, cached[:settings], settings) unless cached[:settings] == settings
+  # Detaches every dataset file and closes the process's dataset database,
+  # so the next dataset_db call opens a new database with the files that are
+  # on disk then.
+  #
+  # Call it when no thread uses the dataset database, for example between
+  # tests, at shutdown, or after a dataset file is replaced. Do not use a
+  # Sequel::Database that dataset_db returned before the call.
+  #
+  # @return [void]
+  def self.close_dataset_dbs
+    @duckdb_datasets.close
+  end
+
+  def self.dataset_options(readonly: true, memory_limit: nil, temp_directory: nil, max_temp_directory_size: nil,
+                           threads: nil, max_connections: nil)
+    raise ReadWriteDatasetError unless readonly
 
-      cached[:db]
-    end
+    settings = { memory_limit:, temp_directory:, max_temp_directory_size:, threads: }
+               .compact.to_h { |name, value| [ name.to_s, value.to_s ] }
+    { settings:, max_connections: }
   end
+  private_class_method :dataset_options
 
   # Instantiates and memoizes a database connection. The +db+ method instantiates
   # the connection on the first call and then memoizes itself so only a single

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sequelizer
 version: !ruby/object:Gem::Version
-  version: 0.4.1
+  version: 0.5.0
 platform: ruby
 authors:
 - Ryan Duryea
@@ -286,6 +286,7 @@ files:
 - lib/sequelizer/cli.rb
 - lib/sequelizer/connection_maker.rb
 - lib/sequelizer/duckdb_dataset.rb
+- lib/sequelizer/duckdb_datasets.rb
 - lib/sequelizer/env_config.rb
 - lib/sequelizer/options.rb
 - lib/sequelizer/version.rb