dwh 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d91ce7dc2866f584744c0636918a6e4d5a2d189d3356b08fcee2b60ee40b5317
-  data.tar.gz: 51e0c508c1a77e5302493868a3e493547ec00fc022195a1b31f581b49abb843b
+  metadata.gz: ec16e1365da69bdc1778ff349775274d8b3a6d85978a9a69be0d6aa0da44d866
+  data.tar.gz: 4fc4ed0d62f7e657f80779265fec1945317cbdbd127e43c654916f2f0c157f88
 SHA512:
-  metadata.gz: 1ac97b11067710faab814ff16f4e17e46ffa70acc7281e3c45c86e32861feb5eb7207d9f372f29849bc7453f891241c5715624024fc82f5d660d098d812990e5
-  data.tar.gz: 2aa0820ef26779facc52dc41797403930ebb39186fc0eb81146b310395e8a0052c2d23c1468ee3991ec901204cd9d4f1ce079cf1d86d2c69c7a6616a07b2093e
+  metadata.gz: '008e23f7a38eae74baffc20f04b14daf6344dc5e3b9ae8616926fa74fe4bab393b4a21134755dcfc289e999ec967c2c5a2baa98044834ce1f415d6d179046ab4'
+  data.tar.gz: 2febcb2fb010b298d70b2e010dc1d2be942f63bacc4ef7e57eaf63e5fb616c27472037687ad144c706478ce80be66a7445f9479d9e774f78683ca9d6a9dc3732

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 ## [Unreleased]
 
+## [0.6.0] - 2026-09-28
+
+### Added
+
+- Google BigQuery adapter (`:bigquery`) with dedicated settings and unit/system test coverage. Authenticates with a service-account keyfile or Application Default Credentials; requires the `google-cloud-bigquery` gem.
+
+## [0.5.1] - 2026-08-03
+
+### Fixed
+
+- **Factory#shutdown**: `case pool.class` never matched (every call fell into `else`); undefined `c` NameError when closing live connections; symbol branch called `delete` on the pool instead of the map. Shutdown now matches on the argument, removes the map entry before closing, and tolerates unknown names.
+- **Factory#pool**: check-then-set race could orphan a pool under concurrent first-use; creation is now mutex-guarded.
+- **Factory#start_reaper**: no longer checks out a connection just to log stats (which created connections and reset idle clocks); Idle/Available labels use `pool.idle` / `pool.available`; iterates a snapshot; rescues `PoolShuttingDownError` per pool so a retired pool cannot kill the reaper thread.
+- **DuckDb#close**: use `@connection&.disconnect` so closing an already-closed adapter does not open a new connection.
+- **DuckDb.close_all**: close then clear instead of deleting while iterating (which could skip entries).
+
 ## [0.5.0] - 2026-06-19
 
 ### Added

data/README.md

@@ -33,11 +33,12 @@ The adapter only has 5 core methods (6 including the connection method).  A YAML
 - **PostgreSQL** - Full-featured RDBMS with advanced SQL support
 - **MySQL** - Popular open-source database
 - **SQL Server** - Microsoft's enterprise database
+- **ClickHouse** - High performance analytical database
+- **Databricks** - Lakehouse SQL warehouse
+- **Google BigQuery** - Google Cloud serverless warehouse
 
 ## Integrations Coming Soon
 
-- **ClickHouse** - High performance analytical db
-- **Databricks** - Big data compute engine
 - **MotherDuck** - Hosted DuckDB service
 
 ## Quick Start
@@ -121,6 +122,21 @@ Run tests on  druid:
 bundle exec rake test:system:druid 
 ```
 
+Cloud warehouse tests (`test/system/cloud_*_test.rb`) run against real accounts and skip
+unless configured through environment variables, so no credentials live in the repo:
+
+| Adapter | Variables |
+|---|---|
+| Athena | `ATHENA_S3_OUTPUT`, optional `ATHENA_REGION`, `ATHENA_DATABASE`; AWS credentials from the environment |
+| BigQuery | `BIGQUERY_PROJECT`, optional `BIGQUERY_DATASET`; `GOOGLE_APPLICATION_CREDENTIALS` or gcloud ADC. Creates its own fixture tables |
+| Databricks | `DATABRICKS_HOST`, `DATABRICKS_WAREHOUSE`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET` |
+| Redshift | `REDSHIFT_HOST`, `REDSHIFT_PASSWORD`, optional `REDSHIFT_USER`, `REDSHIFT_DATABASE`, `REDSHIFT_PORT` |
+| Snowflake | `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_PAT`, optional `SNOWFLAKE_DATABASE`; key-pair test needs `SNOWFLAKE_USER`, `SNOWFLAKE_PRIVATE_KEY` |
+
+```bash
+BIGQUERY_PROJECT=my-project BUNDLE_WITH=development bundle exec ruby -Itest test/system/cloud_bigquery_test.rb
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt.
@@ -137,4 +153,4 @@ This project is available as open source under the terms of the MIT License.
 
 ## Version
 
-Current version: 0.1.0
+Current version: 0.6.0

data/docs/guides/adapters.md

@@ -697,6 +697,22 @@ athena = DWH.create(:athena, {
 
 See full list of config options here: [athena-api](https://docs.aws.amazon.com/sdk-for-ruby/v2/api/Aws/Athena/Client.html#initialize-instance_method)
 
+## Google BigQuery Adapter
+
+Requires the `google-cloud-bigquery` gem (`gem install google-cloud-bigquery`, pure Ruby).
+
+```ruby
+bq = DWH.create(:bigquery, {
+    project_id: 'my-gcp-project',
+    dataset: 'analytics',                        # default dataset for unqualified table names
+    keyfile: '/path/to/service-account.json',   # optional; omit to use Application Default Credentials
+    query_timeout: 300                           # optional, seconds
+})
+```
+
+Without `keyfile`, credentials resolve via `GOOGLE_APPLICATION_CREDENTIALS` or `gcloud auth application-default login`.
+Extra client options can be passed with `extra_connection_params` (see `Google::Cloud::Bigquery.new`).
+
 ## Configuration Validation
 
 DWH validates configuration parameters at creation time:

data/lib/dwh/adapters/big_query.rb

@@ -0,0 +1,173 @@
+require 'csv'
+require 'timeout'
+
+module DWH
+  module Adapters
+    # Google BigQuery adapter. Requires the google-cloud-bigquery gem, which is
+    # loaded lazily on first connection so the adapter can be created without it.
+    #
+    # @example Service account keyfile
+    #   DWH.create(:bigquery, {
+    #     project_id: 'my-gcp-project',
+    #     dataset: 'analytics',
+    #     keyfile: '/path/to/service-account.json'
+    #   })
+    #
+    # @example Application Default Credentials (gcloud auth application-default login)
+    #   DWH.create(:bigquery, { project_id: 'my-gcp-project', dataset: 'analytics' })
+    class BigQuery < Adapter
+      config :project_id, String, required: true, message: 'GCP project id'
+      config :dataset, String, required: true, message: 'default dataset for unqualified table names'
+      config :keyfile, String, required: false, default: nil,
+                               message: 'path to service-account JSON; omit to use Application Default Credentials'
+      config :query_timeout, Integer, required: false, default: 300, message: 'query timeout in seconds'
+
+      # The schema API reports legacy type names; map the ones whose
+      # normalized type would otherwise be wrong (INTEGER is 64-bit in BigQuery).
+      TYPE_ALIASES = { 'INTEGER' => 'INT64', 'FLOAT' => 'FLOAT64', 'BOOLEAN' => 'BOOL', 'RECORD' => 'STRUCT' }.freeze
+
+      # (see Adapter#connection)
+      def connection
+        return @connection if @connection
+
+        require 'google/cloud/bigquery'
+        opts = { project_id: config[:project_id] }
+        opts[:credentials] = config[:keyfile] unless config[:keyfile].to_s.empty?
+        @connection = Google::Cloud::Bigquery.new(**opts, **extra_connection_params)
+      rescue LoadError
+        raise ConfigError, <<~MSG
+          BigQuery adapter requires the 'google-cloud-bigquery' gem.
+
+          Install with: gem install google-cloud-bigquery
+
+          No system libraries required (pure Ruby).
+        MSG
+      rescue StandardError => e
+        raise ConfigError, "Failed to connect to BigQuery: #{e.message}"
+      end
+
+      # (see Adapter#test_connection)
+      def test_connection(raise_exception: false)
+        raise ConnectionError, "Dataset '#{config[:dataset]}' not found" unless connection.dataset(config[:dataset])
+
+        true
+      rescue StandardError => e
+        raise ConnectionError, "BigQuery connection test failed: #{e.message}" if raise_exception
+
+        false
+      end
+
+      # (see Adapter#tables)
+      def tables(**qualifiers)
+        dataset_for(qualifiers).tables.all.map(&:table_id)
+      end
+
+      # (see Adapter#stats)
+      def stats(table, date_column: nil, **qualifiers)
+        sql = 'SELECT COUNT(*) AS row_count'
+        sql += ", MIN(#{date_column}) AS date_start, MAX(#{date_column}) AS date_end" if date_column
+        row = execute("#{sql} FROM `#{dataset_name(qualifiers)}.#{table}`", format: :object).first || {}
+
+        TableStats.new(row_count: row['row_count'], date_start: row['date_start'], date_end: row['date_end'])
+      end
+
+      # (see Adapter#metadata)
+      # Uses the table schema API instead of INFORMATION_SCHEMA: one call, and
+      # precision/scale/length come back as attributes rather than parsed from strings.
+      def metadata(table, **qualifiers)
+        db_table = Table.new table, schema: dataset_name(qualifiers)
+        bq_table = dataset_for(qualifiers).table(db_table.physical_name)
+        raise ExecutionError, "Table '#{db_table.schema}.#{db_table.physical_name}' not found" unless bq_table
+
+        bq_table.schema.fields.each do |field|
+          db_table << Column.new(
+            name: field.name,
+            data_type: TYPE_ALIASES.fetch(field.type, field.type),
+            precision: field.precision || 0,
+            scale: field.scale || 0,
+            max_char_length: field.max_length
+          )
+        end
+
+        db_table
+      end
+
+      # (see Adapter#execute)
+      def execute(sql, format: :array, retries: 0)
+        result = with_debug(sql) { with_retry(retries) { run_query(sql) } }
+
+        format = format.downcase if format.is_a?(String)
+        case format.to_sym
+        when :array then result[:rows]
+        when :object then result[:rows].map { |row| Hash[result[:headers].zip(row)] }
+        when :csv then rows_to_csv(result[:headers], result[:rows])
+        when :native then result
+        else raise UnsupportedCapability, "Unsupported format: #{format} for BigQuery adapter"
+        end
+      end
+
+      # (see Adapter#execute_stream)
+      def execute_stream(sql, io, stats: nil, retries: 0)
+        with_debug(sql) do
+          with_retry(retries) do
+            data = query_data(sql)
+            io.write(CSV.generate_line(headers_of(data)))
+            data.all.each do |row|
+              values = row.values
+              stats << values unless stats.nil?
+              io.write(CSV.generate_line(values))
+            end
+          end
+        end
+
+        io.rewind
+        io
+      end
+
+      # (see Adapter#stream)
+      def stream(sql, &block)
+        with_debug(sql) { query_data(sql).all.each { |row| block.call(row.values) } }
+      end
+
+      private
+
+      def dataset_name(qualifiers)
+        qualifiers[:dataset] || qualifiers[:schema] || config[:dataset]
+      end
+
+      def dataset_for(qualifiers)
+        name = dataset_name(qualifiers)
+        connection.dataset(name) || raise(ExecutionError, "Dataset '#{name}' not found")
+      end
+
+      # Runs the query and returns the gem's paged Data object. Rows are hashes
+      # keyed by column name in column order, so row.values matches data.fields.
+      def query_data(sql)
+        Timeout.timeout(config[:query_timeout]) do
+          connection.query(sql, dataset: config[:dataset], project: config[:project_id])
+        end
+      rescue DWHError
+        raise
+      rescue StandardError => e
+        raise ExecutionError, "BigQuery query failed: #{e.message}"
+      end
+
+      def run_query(sql)
+        data = query_data(sql)
+        { headers: headers_of(data), rows: data.all.map(&:values) }
+      end
+
+      # DDL/DML statements return no schema.
+      def headers_of(data)
+        data.schema&.fields&.map(&:name) || []
+      end
+
+      def rows_to_csv(headers, rows)
+        CSV.generate do |csv|
+          csv << headers
+          rows.each { |row| csv << row }
+        end
+      end
+    end
+  end
+end

data/lib/dwh/adapters/duck_db.rb

@@ -56,10 +56,9 @@ module DWH
       # we open one instance but many connections. Use this
       # method to close them all.
       def self.close_all
-        databases.each do |key, db|
-          db.close
-          databases.delete(key)
-        end
+        # Iterate values then clear — mutating the hash while each-ing skips entries.
+        databases.each_value(&:close)
+        databases.clear
       end
 
       # This disconnects the current connection but
@@ -68,7 +67,9 @@ module DWH
       #
       # (see Adapter#close)
       def close
-        connection.disconnect
+        # Use @connection directly: #connection opens a new handle when nil,
+        # which would re-open just to close.
+        @connection&.disconnect
         @connection = nil
       end
 

data/lib/dwh/column.rb

@@ -36,7 +36,7 @@ module DWH
       inner = unwrap_type(data_type)
 
       case inner
-      when /binary/, 'image'
+      when /binary/, 'image', 'bytes'
         'binary'
       when /varchar/, 'string', /text/, /char/, /fixedstring/
         'string'
@@ -50,7 +50,7 @@ module DWH
       when 'bigint', 'bit_int', 'big_integer', /^int64$/, /^int128$/, /^int256$/,
            /^uint64$/, /^uint128$/, /^uint256$/
         'bigint'
-      when 'decimal', 'double', 'float', 'real', 'dec', 'numeric', 'money',
+      when 'decimal', 'double', 'float', 'real', 'dec', 'numeric', 'bignumeric', 'money',
            /^float32$/, /^float64$/, /^decimal/
         'decimal'
       when 'boolean', 'bit', 'bool'

data/lib/dwh/factory.rb

@@ -46,6 +46,11 @@ module DWH
       @pools ||= {}
     end
 
+    # Mutex guarding pool creation / map mutation.
+    def pool_mutex
+      @pool_mutex ||= Mutex.new
+    end
+
     # The canonical way of creating an adapter instance
     # in DWH.
     # @param adapter_name [String, Symbol]
@@ -64,62 +69,80 @@ module DWH
     # Create a pool of connections for a given name and adapter.
     # Returns existing pool if it was already created.
     #
-    # @param name [String] custom name for your pool
+    # @param name [String, Symbol] custom name for your pool (stored as String)
     # @param adapter_name [String, Symbol]
     # @param config [Hash] connection options
     # @param timeout [Integer] pool checkout time out
     # @param size [Integer] size of the pool
     def pool(name, adapter_name, config, timeout: 5, size: 10)
-      if pools.key?(name)
-        pools[name]
-      else
-        pools[name] = ConnectionPool.new(size: size, timeout: timeout) do
-          create(adapter_name, config)
+      name = name.to_s
+      pool_mutex.synchronize do
+        if pools.key?(name)
+          pools[name]
+        else
+          pools[name] = ConnectionPool.new(size: size, timeout: timeout) do
+            create(adapter_name, config)
+          end
         end
       end
     end
 
     # Shutdown a specific pool or all pools
-    # @param pool [String, ConnectionPool, nil] pool or name of pool
+    # @param pool [String, Symbol, ConnectionPool, nil] pool or name of pool
     #   or nil to shut everything down
     def shutdown(pool = nil)
-      case pool.class
-      when String
-        pools[pool].shutdown { it.close }
-        pools.delete(pool)
-      when Symbol
-        pools[pool.to_s].shutdown { it.close }
-        pools[pool.to_s].delete
-      when ConnectionPool
-        pool.shutdown { it.close }
-        pools.delete(pools.key(pool))
-      else
-        pools.each_value do |val|
-          val.shutdown { c.close }
+      # Mutate the map under the same mutex as pool creation so a concurrent
+      # create cannot be orphaned by @pools = {} / delete racing it.
+      # Close outside the lock — ConnectionPool#shutdown can wait on check-in.
+      to_close = pool_mutex.synchronize do
+        case pool
+        when String, Symbol
+          # Delete first so a raising close cannot leave a dead pool in the map.
+          removed = pools.delete(pool.to_s)
+          removed ? [removed] : []
+        when ConnectionPool
+          key = pools.key(pool)
+          pools.delete(key) if key
+          [pool]
+        else
+          closing = pools.values
+          @pools = {}
+          closing
         end
-        @pools = {}
       end
+      to_close.each { |p| p.shutdown { it.close } }
     end
 
     # Start reaper that will periodically clean up
     # unused or idle connections.
     # @param frequency [Integer] defaults to 300 seconds
+    # @return [Thread] the reaper thread
     def start_reaper(frequency = 300)
       logger.info 'Starting DB Adapter reaper process'
       Thread.new do
         loop do
-          pools.each do |name, pool|
-            logger.info "DB POOL FOR #{name} STATS:"
-            pool.with do
-              logger.info "\tSize:      #{pool.size}"
-              logger.info "\tIdle:      #{pool.available}"
-              logger.info "\tAvailable: #{pool.available}"
-            end
-            pool.reap(frequency) { it.close }
-          end
+          reaper_tick(frequency)
           sleep frequency
         end
       end
     end
+
+    # One reaper cycle: log pool stats (without checking out) and reap idle connections.
+    # Safe to call from tests; rescues per-pool so a shut-down pool cannot kill the loop.
+    # @param frequency [Integer] idle threshold passed to ConnectionPool#reap
+    def reaper_tick(frequency = 300)
+      # Snapshot so concurrent shutdown deletions do not mutate while we iterate.
+      pools.to_a.each do |name, pool|
+        logger.info "DB POOL FOR #{name} STATS:"
+        logger.info "\tSize:      #{pool.size}"
+        logger.info "\tIdle:      #{pool.idle}"
+        logger.info "\tAvailable: #{pool.available}"
+        pool.reap(frequency) { it.close }
+      rescue ConnectionPool::PoolShuttingDownError => e
+        logger.info "Skipping reaper for pool #{name}: #{e.class}"
+      rescue StandardError => e
+        logger.error "Reaper error for pool #{name}: #{e.class}: #{e.message}"
+      end
+    end
   end
 end

data/lib/dwh/settings/bigquery.yml

@@ -0,0 +1,92 @@
+# Google BigQuery adapter settings
+# Only overrides that differ from base.yml are listed here.
+
+# BigQuery uses backticks for identifier quoting.
+quote: "`@exp`"
+
+# FORMAT_DATE uses strftime elements. base.yml's %W/%M mean
+# week-number/minute in BigQuery, so use the standard name codes.
+day_name_format: "%A"
+abbreviated_day_name_format: "%a"
+month_name_format: "%B"
+abbreviated_month_name_format: "%b"
+
+# Current time functions require parentheses.
+current_date:      "CURRENT_DATE()"
+current_time:      "CURRENT_TIME()"
+current_timestamp: "CURRENT_TIMESTAMP()"
+
+# Date literals
+date_literal:      "DATE '@val'"
+date_time_literal: "TIMESTAMP '@val'"
+
+# BigQuery puts the expression first and takes the unit as a bare keyword.
+truncate_date:   "DATE_TRUNC(@exp, @unit)"
+date_add:        "DATE_ADD(@exp, INTERVAL @val @unit)"
+date_diff:       "DATE_DIFF(@end_exp, @start_exp, @unit)"
+date_format_sql: "FORMAT_DATE('@format', @exp)"
+
+extract_year:         "EXTRACT(YEAR FROM @exp)"
+extract_month:        "EXTRACT(MONTH FROM @exp)"
+extract_quarter:      "EXTRACT(QUARTER FROM @exp)"
+extract_day_of_year:  "EXTRACT(DAYOFYEAR FROM @exp)"
+extract_day_of_month: "EXTRACT(DAY FROM @exp)"
+extract_day_of_week:  "EXTRACT(DAYOFWEEK FROM @exp)"
+extract_week_of_year: "EXTRACT(WEEK FROM @exp)"
+extract_hour:         "EXTRACT(HOUR FROM @exp)"
+extract_minute:       "EXTRACT(MINUTE FROM @exp)"
+extract_year_month:   "CAST(FORMAT_DATE('%Y%m', @exp) AS INT64)"
+
+# DATE_TRUNC(x, WEEK) is Sunday-based in BigQuery; the explicit forms
+# below are used whenever the requested week_start_day differs.
+default_week_start_day: "sunday"
+sunday_week_start_day:  "DATE_TRUNC(@exp, WEEK(SUNDAY))"
+monday_week_start_day:  "DATE_TRUNC(@exp, WEEK(MONDAY))"
+
+# Null handling
+if_null: "IFNULL(@exp, @when_null)"
+
+# Array operations via UNNEST
+array_in_list:      "EXISTS(SELECT 1 FROM UNNEST(@exp) AS x WHERE x IN (@list))"
+array_exclude_list: "NOT EXISTS(SELECT 1 FROM UNNEST(@exp) AS x WHERE x IN (@list))"
+array_unnest_join:  "CROSS JOIN UNNEST(@exp) AS @alias"
+
+# Capabilities
+# Temp tables only live inside a script/session; use CTEs instead.
+supports_temp_tables: false
+temp_table_type: "cte"
+
+# BigQuery-specific reserved keywords not in the standard baseline.
+extra_reserved_keywords:
+  - struct
+  - unnest
+  - qualify
+  - tablesample
+  - window
+  - ignore
+  - respect
+  - int64
+  - float64
+  - bytes
+  - bignumeric
+  - bool
+  - datetime
+  - geography
+  - dayofweek
+  - dayofyear
+  - isoweek
+  - isoyear
+
+# BigQuery-specific aggregate functions not in the standard baseline.
+extra_aggregate_functions:
+  - any_value
+  - countif
+  - logical_and
+  - logical_or
+  - array_concat_agg
+  - approx_quantiles
+  - approx_top_count
+  - approx_top_sum
+  - bit_xor
+  - max_by
+  - min_by

data/lib/dwh/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DWH
-  VERSION = '0.5.0'
+  VERSION = '0.6.0'
 end

data/lib/dwh.rb

@@ -21,6 +21,7 @@ require_relative 'dwh/adapters/athena'
 require_relative 'dwh/adapters/redshift'
 require_relative 'dwh/adapters/databricks'
 require_relative 'dwh/adapters/click_house'
+require_relative 'dwh/adapters/big_query'
 
 # DWH encapsulates the full functionality of this gem.
 #
@@ -54,6 +55,7 @@ module DWH
   register(:redshift, Adapters::Redshift)
   register(:databricks, Adapters::Databricks)
   register(:clickhouse, Adapters::ClickHouse)
+  register(:bigquery, Adapters::BigQuery)
 
   # The raw base.yml settings, loaded once. This is the single source of
   # truth for the standard, warehouse-agnostic dialect baseline.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dwh
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.6.0
 platform: ruby
 authors:
 - Ajo Abraham
@@ -154,6 +154,7 @@ files:
 - lib/dwh.rb
 - lib/dwh/adapters.rb
 - lib/dwh/adapters/athena.rb
+- lib/dwh/adapters/big_query.rb
 - lib/dwh/adapters/click_house.rb
 - lib/dwh/adapters/databricks.rb
 - lib/dwh/adapters/druid.rb
@@ -181,6 +182,7 @@ files:
 - lib/dwh/settings.rb
 - lib/dwh/settings/athena.yml
 - lib/dwh/settings/base.yml
+- lib/dwh/settings/bigquery.yml
 - lib/dwh/settings/clickhouse.yml
 - lib/dwh/settings/databricks.yml
 - lib/dwh/settings/druid.yml