sequel-bigquery 0.4.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3e8459abb689482e387bc4b0cd7dac2c582a40a9c449d5cd5bad5c8da57fffdb
-  data.tar.gz: 4409fa1b5704c03afcd068915311c4944bdaec1326a21ba6df3ed47d9ea118a2
+  metadata.gz: 6c15547bd0bbe53a384a987286cda32f9e1ad624c2c9e5339ac412322e810653
+  data.tar.gz: 26d87c1ff3d3aecf9d5401de88ad9048a239dd65bc3a682cfbb9d9d726675b9f
 SHA512:
-  metadata.gz: 8573f4d5d1f46d63fd062d97efa1bf19ffc7d2e9a3dd24433cdd079554380f7639e2d767657285a9cc1eea829a300c08aa40cfd3a7e24eb9f0f229c0f13e0e8c
-  data.tar.gz: 51371f99aba0a7799a0c8c34f9d3541c86c76113d4acda78badb420375abded01cdd456f184753793505be70f92eba1d9f0a99f35abdd1f766806023f3860979
+  metadata.gz: daa23d28fc8359540d61e57a5f4087f76da68045e6535d864f711c51d8fd0517bf9d74e332ea9cd3a5399ece489beb06df08dfaf409caf8877a1cf6aaa8140a3
+  data.tar.gz: eb73f36dbe0317d42e5b1fd9a774dd7d5440d36a8ddebd33c5683a67723efaf5538d4b915a4b518e49a6165854b42152762d57934a57dd050efee0f735563c4f

data/README.md

@@ -4,11 +4,22 @@
 
 A Sequel adapter for [Google's BigQuery](https://cloud.google.com/bigquery).
 
+This gem was created in order to manage schema migrations of a BigQuery dataset at GreenSync. At the time of writing, we couldn't find any good tools in any language to manage changes to the schema as a set of migrations.
+
+Beyond migrations, I'm unsure how useful this gem is. I haven't yet tested what the performance would be for data interactions vs. directly using the `google-cloud-bigquery` gem's native facilities. If you're inserting a bunch of data, it's probably a better idea to use an [inserter from that gem](https://googleapis.dev/ruby/google-cloud-bigquery/latest/Google/Cloud/Bigquery/Dataset.html#insert_async-instance_method) rather than going through SQL.
+
 ## Contents
 
 <!-- MarkdownTOC autolink=true -->
 
 - [Intro](#intro)
+- [Quirks](#quirks)
+  - [Creating tables with column defaults](#creating-tables-with-column-defaults)
+  - [Transactions](#transactions)
+  - [Update statements without `WHERE`](#update-statements-without-where)
+  - [Combining statements](#combining-statements)
+  - [Alter table](#alter-table)
+  - [Column recreation](#column-recreation)
 - [Installation](#installation)
 - [Usage](#usage)
 - [Contributing](#contributing)
@@ -25,12 +36,12 @@ A Sequel adapter for [Google's BigQuery](https://cloud.google.com/bigquery).
 Features:
 
 - Connecting
-- Migrating
-- Table creation, with automatic removal of defaults from statements (since BigQuery doesn't support it)
+- Migrating (see quirks)
+- Table creation (see quirks)
 - Inserting rows
-- Updating rows, with automatic addition of `where 1 = 1` to statements (since BigQuery requires a `where` clause)
+- Updating rows (see quirks)
 - Querying
-- Transactions (buffered since BigQuery only supports them when you execute the whole transaction at once)
+- Transactions (see quirks)
 - Table partitioning
 - Ruby types:
   + String
@@ -42,6 +53,34 @@ Features:
   + BigDecimal
 - Selecting the BigQuery server location
 
+## Quirks
+
+### Creating tables with column defaults
+
+BigQuery doesn't support defaults on columns. As a workaround, all defaults are automatically removed from statements (crudely).
+
+### Transactions
+
+BigQuery doesn't support transactions where the statements are executed individually. It does support them if entire transaction SQL is sent all at once though. As a workaround, buffering of statements within a transaction has been implemented. However, the impact of this is that no results can be returned within a transaction.
+
+### Update statements without `WHERE`
+
+BigQuery requires all `UPDATE` statement to have a `WHERE` clause. As a workaround, statements which lack one have `where 1 = 1` appended automatically (crudely).
+
+### Combining statements
+
+When combining multiple statements into one query (with `;`), and the final statement is not a `SELECT`, the `google-cloud-bigquery` gem has a [bug](https://github.com/googleapis/google-cloud-ruby/issues/9617) which causes an exception. Note that all the statements have been executed when this happens. A workaround is to append `; SELECT 1`.
+
+### Alter table
+
+BigQuery [rate-limits alter table statements](https://cloud.google.com/bigquery/quotas#dataset_limits) to 10 per second. This is mitigated somewhat by Sequel combining `ALTER TABLE` statements whenever possible, and BigQuery having extremely high latency (\~2 seconds per query); but you may still run into this limitation.
+
+We've also noticed a bug with `google-cloud-bigquery` where an `ALTER TABLE` statement resulted in a `NoMethodError` on nil for `fields` within `from_gapi_json`. We're not yet sure what caused this.
+
+### Column recreation
+
+Be careful when deleting a column which you might want to re-add. BigQuery reserves the name of a deleted column for up to the time travel duration - which is [*seven days*](https://cloud.google.com/bigquery/docs/time-travel). Re-creating the entire dataset is a painful workaround.
+
 ## Installation
 
 Add it to the `Gemfile` of your project:
@@ -68,6 +107,7 @@ Connect to BigQuery:
 
 ```
 require 'sequel-bigquery'
+require 'logger'
 
 db = Sequel.connect(
   adapter: :bigquery,
@@ -80,6 +120,8 @@ db = Sequel.connect(
 
 And use Sequel like normal.
 
+Note that it is important to supply a logger that will at least output warning messages so you know when your queries are being modifed or buffered, which may be unexpected behaviour.
+
 ## Contributing
 
 Pull requests welcome! =)

data/lib/sequel-bigquery.rb

@@ -4,6 +4,7 @@ require 'delegate'
 require 'time'
 
 require 'google/cloud/bigquery'
+require 'amazing_print'
 require 'paint'
 require 'sequel'
 
@@ -16,39 +17,34 @@ module Sequel
     class Database < Sequel::Database # rubocop:disable Metrics/ClassLength
       set_adapter_scheme :bigquery
 
-      def initialize(*args, **kawrgs)
-        puts '.new'
-        @orig_opts = kawrgs.fetch(:orig_opts)
+      def initialize(*args, **kwargs)
+        @bigquery_config = kwargs.fetch(:orig_opts)
         @sql_buffer = []
         @sql_buffering = false
         super
       end
 
       def connect(*_args)
-        puts '#connect'
-        config = @orig_opts.dup
-        config.delete(:adapter)
-        config.delete(:logger)
-        location = config.delete(:location)
-        bq_dataset_name = config.delete(:dataset) || config.delete(:database)
-        @bigquery = Google::Cloud::Bigquery.new(config)
+        log_each(:debug, '#connect')
+        get_or_create_bigquery_dataset
+          .tap { log_each(:debug, '#connect end') }
+      end
+
+      def bigquery
         # ObjectSpace.each_object(HTTPClient).each { |c| c.debug_dev = STDOUT }
-        @bigquery.dataset(bq_dataset_name) || begin
-          @loggers[0].debug('BigQuery dataset %s does not exist; creating it' % bq_dataset_name)
-          @bigquery.create_dataset(bq_dataset_name, location: location)
-        end
-          .tap { puts '#connect end' }
+        @bigquery ||= Google::Cloud::Bigquery.new(google_cloud_bigquery_gem_config)
       end
 
       def disconnect_connection(_c)
-        puts '#disconnect_connection'
+        log_each(:debug, '#disconnect_connection')
         # c.disconnect
       end
 
       def drop_datasets(*dataset_names_to_drop)
         dataset_names_to_drop.each do |dataset_name_to_drop|
-          puts "Dropping dataset #{dataset_name_to_drop.inspect}"
-          dataset_to_drop = @bigquery.dataset(dataset_name_to_drop)
+          log_each(:debug, "Dropping dataset #{dataset_name_to_drop.inspect}")
+          dataset_to_drop = bigquery.dataset(dataset_name_to_drop)
+          next unless dataset_to_drop
           dataset_to_drop.tables.each(&:delete)
           dataset_to_drop.delete
         end
@@ -56,7 +52,7 @@ module Sequel
       alias drop_dataset drop_datasets
 
       def execute(sql, opts = OPTS) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
-        puts '#execute'
+        log_each(:debug, '#execute')
         log_query(sql)
 
         # require 'pry'; binding.pry if sql =~ /CREATE TABLE IF NOT EXISTS/i
@@ -82,21 +78,31 @@ module Sequel
           warn("Warning: Will now execute entire buffered transaction:\n" + @sql_buffer.join("\n"))
         end
 
+        sql_to_execute = @sql_buffer.any? ? @sql_buffer.join("\n") : sql
+
         synchronize(opts[:server]) do |conn|
           results = log_connection_yield(sql, conn) do
-            sql_to_execute = @sql_buffer.any? ? @sql_buffer.join("\n") : sql
             conn.query(sql_to_execute)
           end
-          require 'amazing_print'
-          ap results
+          log_each(:debug, results.awesome_inspect)
           if block_given?
             yield results
           else
             results
           end
-        # TODO
-        # rescue ::ODBC::Error, ArgumentError => e
-        rescue Google::Cloud::InvalidArgumentError, ArgumentError => e
+        rescue Google::Cloud::InvalidArgumentError, Google::Cloud::PermissionDeniedError => e
+          if e.message.include?('too many table update operations for this table')
+            warn('Triggered rate limit of table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas')
+            if retryable_query?(sql_to_execute)
+              warn('Detected retryable query - re-running query after a 1 second sleep')
+              sleep 1
+              retry
+            else
+              log_each(:error, "Query not detected as retryable; can't automatically recover from being rate-limited")
+            end
+          end
+          raise_error(e)
+        rescue ArgumentError => e
           raise_error(e)
         end # rubocop:disable Style/MultilineBlockChain
           .tap do
@@ -123,6 +129,33 @@ module Sequel
 
       private
 
+      attr_reader :bigquery_config
+
+      def google_cloud_bigquery_gem_config
+        bigquery_config.dup.tap do |config|
+          %i[
+            adapter
+            database
+            dataset
+            location
+            logger
+          ].each do |option|
+            config.delete(option)
+          end
+        end
+      end
+
+      def get_or_create_bigquery_dataset # rubocop:disable Naming/AccessorMethodName
+        bigquery.dataset(bigquery_dataset_name) || begin
+          log_each(:debug, 'BigQuery dataset %s does not exist; creating it' % bigquery_dataset_name)
+          bigquery.create_dataset(bigquery_dataset_name, location: bigquery_config[:location])
+        end
+      end
+
+      def bigquery_dataset_name
+        bigquery_config[:dataset] || bigquery_config[:database] || (raise ArgumentError, 'BigQuery dataset must be specified')
+      end
+
       def connection_execute_method
         :query
       end
@@ -137,9 +170,9 @@ module Sequel
       end
 
       def schema_parse_table(_table_name, _opts)
-        logger.debug(Paint['schema_parse_table', :red, :bold])
+        log_each(:debug, Paint['schema_parse_table', :red, :bold])
         # require 'pry'; binding.pry
-        @bigquery.datasets.map do |dataset|
+        bigquery.datasets.map do |dataset|
           [
             dataset.dataset_id,
             {},
@@ -154,13 +187,12 @@ module Sequel
 
       # Padded to horizontally align with post-execution log message which includes the execution time
       def log_query(sql)
-        pad = '                                                                '
-        puts Paint[pad + sql, :cyan, :bold]
-        # @loggers[0]&.debug('            ' + sql)
+        pad = ' ' * 12
+        log_each(:debug, Paint[pad + sql, :cyan, :bold])
       end
 
       def warn(msg)
-        @loggers[0].warn(Paint[msg, '#FFA500', :bold])
+        log_each(:warn, Paint[msg, '#FFA500', :bold])
       end
 
       def warn_default_removal(sql)
@@ -190,11 +222,27 @@ module Sequel
 
         sql
       end
+
+      def supports_combining_alter_table_ops?
+        true
+      end
+
+      def retryable_query?(sql)
+        single_statement_query?(sql) && alter_table_query?(sql)
+      end
+
+      def single_statement_query?(sql)
+        !sql.rstrip.chomp(';').include?(';')
+      end
+
+      def alter_table_query?(sql)
+        sql.match?(/\Aalter table /i)
+      end
     end
 
     class Dataset < Sequel::Dataset
       def fetch_rows(sql, &block)
-        puts '#fetch_rows'
+        db.send(:log_each, :debug, '#fetch_rows')
 
         execute(sql) do |bq_result|
           self.columns = bq_result.fields.map { |field| field.name.to_sym }
@@ -220,7 +268,7 @@ module Sequel
 
       # Like MySQL, BigQuery uses the nonstandard ` (backtick) for quoting identifiers.
       def quoted_identifier_append(sql, c)
-        sql << '`%s`' % c
+        sql << ('`%s`' % c)
       end
 
       def input_identifier(v)

data/lib/sequel_bigquery/version.rb

@@ -2,6 +2,6 @@
 
 module Sequel
   module Bigquery
-    VERSION = '0.4.0'
+    VERSION = '0.6.0'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sequel-bigquery
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.6.0
 platform: ruby
 authors:
 - Brendan Weibrecht
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-10-28 00:00:00.000000000 Z
+date: 2021-11-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: amazing_print
@@ -98,7 +98,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.16
+rubygems_version: 3.1.6
 signing_key:
 specification_version: 4
 summary: A Sequel adapter for Google's BigQuery