google-cloud-bigquery 1.27.0 → 1.32.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 334d37be375f513da083f7338e1324413409a6c30554456c08290ed1337c2701
-  data.tar.gz: f21c7793b5a9f82d0c3840aa5915fd10ef43644ccc1d9ea996a63860fa8b16ed
+  metadata.gz: 36bd68946313d76f828e1fdbfb66606e7495ff876e25da7925772ad324ea8da3
+  data.tar.gz: b75c3a1b4d585c64c048bb073d0e3c0ab0231a666e8df6d350b98a0b282150d4
 SHA512:
-  metadata.gz: 187d0549552613c40f85001fdf82197db01af5b5e285dfd6f7615a64c62ed74807c0ea31ac560d2a1516dae0c91f14fcc08b143882fcf8437b8d7c964f256348
-  data.tar.gz: e6b80ed9ccff8ffb2f8537a009a6d41a7930ee080f50800001b0afa9f7706a85af58180d091051f34a2499d66ca15f5c084a74cbbff244d8c22d266fc62713db
+  metadata.gz: 823a1a0dd34a02c1278df3620270225fd9d592e0577f6812ad9df48722a2132c05e4799e5acd7aafc1973874ab60cd6f6c1a90b48382e1a55a52e8a0a38a8cb4
+  data.tar.gz: 9f3ab6425237255aa466637fe62911ed6253a13033205929e3e43d5948a0ad30943dc5deffcae74329dc94c782358305eb545767a9a5352e135ea2b38d3705a5

data/CHANGELOG.md

@@ -1,5 +1,63 @@
 # Release History
 
+### 1.32.0 / 2021-06-21
+
+#### Features
+
+* Add support for Parquet options
+  * feat(bigquery): Add Bigquery::External::ParquetSource
+  * Add Parquet options to LoadJob
+    * Add LoadJob#parquet_options?
+    * Add LoadJob#parquet_enable_list_inference?
+    * Add LoadJob#parquet_enum_as_string?
+    * Add LoadJob::Updater#parquet_enable_list_inference=
+    * Add LoadJob::Updater#parquet_enum_as_string=
+
+#### Bug Fixes
+
+* Expand googleauth dependency to support future 1.x versions
+
+### 1.31.0 / 2021-04-28
+
+#### Features
+
+* Add support for mutable clustering configuration
+  * Add Table#clustering_fields=
+
+### 1.30.0 / 2021-04-20
+
+#### Features
+
+* Add support for BIGNUMERIC data type
+  * Add support for BIGNUMERIC data insert.
+  * Add support for BIGNUMERIC query params.
+  * Add #bignumeric to Schema, Table, LoadJob::Updater and Field.
+  * Update types tables in docs.
+
+#### Bug Fixes
+
+* Fix Table#time_partitioning_expiration=
+  * Accept nil argument.
+
+### 1.29.0 / 2021-03-10
+
+#### Features
+
+* Drop support for Ruby 2.4 and add support for Ruby 3.0
+
+### 1.28.0 / 2021-03-09
+
+#### Features
+
+* Add Materialized View support
+  * Add Dataset#create_materialized_view
+  * Add Table#materialized_view?
+  * Add Table#enable_refresh?
+  * Add Table#enable_refresh=
+  * Add Table#last_refresh_time
+  * Add Table#refresh_interval_ms
+  * Add Table#refresh_interval_ms=
+
 ### 1.27.0 / 2021-02-10
 
 #### Features

data/CONTRIBUTING.md

@@ -24,7 +24,7 @@ be able to accept your pull requests.
 In order to use the google-cloud-bigquery console and run the project's tests,
 there is a small amount of setup:
 
-1. Install Ruby. google-cloud-bigquery requires Ruby 2.4+. You may choose to
+1. Install Ruby. google-cloud-bigquery requires Ruby 2.5+. You may choose to
    manage your Ruby and gem installations with [RVM](https://rvm.io/),
    [rbenv](https://github.com/rbenv/rbenv), or
    [chruby](https://github.com/postmodern/chruby).
@@ -119,15 +119,14 @@ If you alter an example's title, you may encounter breaking tests.
 ### BigQuery Acceptance Tests
 
 The BigQuery acceptance tests interact with the live service API. Follow the
-instructions in the {file:AUTHENTICATION.md Authentication guide} for enabling
+instructions in the {file:AUTHENTICATION.md Authentication Guide} for enabling
 the BigQuery API. Occasionally, some API features may not yet be generally
 available, making it difficult for some contributors to successfully run the
 entire acceptance test suite. However, please ensure that you do successfully
 run acceptance tests for any code areas covered by your pull request.
 
 To run the acceptance tests, first create and configure a project in the Google
-Developers Console, as described in the {file:AUTHENTICATION.md Authentication
-guide}. Be sure to download the JSON KEY file. Make note of the PROJECT_ID and
+Developers Console, as described in the {file:AUTHENTICATION.md Authentication Guide}. Be sure to download the JSON KEY file. Make note of the PROJECT_ID and
 the KEYFILE location on your system.
 
 Before you can run the BigQuery acceptance tests, you must first create indexes

data/LOGGING.md

@@ -4,7 +4,7 @@ To enable logging for this library, set the logger for the underlying [Google
 API
 Client](https://github.com/google/google-api-ruby-client/blob/master/README.md#logging)
 library. The logger that you set may be a Ruby stdlib
-[`Logger`](https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html) as
+[`Logger`](https://ruby-doc.org/stdlib/libdoc/logger/rdoc/Logger.html) as
 shown below, or a
 [`Google::Cloud::Logging::Logger`](https://googleapis.dev/ruby/google-cloud-logging/latest)
 that will write logs to [Stackdriver

data/OVERVIEW.md

@@ -155,20 +155,21 @@ more complex types such as `ARRAY` and `STRUCT`.
 
 The BigQuery data types are converted to and from Ruby types as follows:
 
-| BigQuery    | Ruby           | Notes  |
-|-------------|----------------|---|
-| `BOOL`      | `true`/`false` | |
-| `INT64`     | `Integer`      | |
-| `FLOAT64`   | `Float`        | |
-| `NUMERIC`   | `BigDecimal`   | Will be rounded to 9 decimal places |
-| `STRING`    | `String`       | |
-| `DATETIME`  | `DateTime`     | `DATETIME` does not support time zone. |
-| `DATE`      | `Date`         | |
-| `TIMESTAMP` | `Time`         | |
-| `TIME`      | `Google::Cloud::BigQuery::Time` | |
-| `BYTES`     | `File`, `IO`, `StringIO`, or similar | |
-| `ARRAY`     | `Array` | Nested arrays and `nil` values are not supported. |
-| `STRUCT`    | `Hash`         | Hash keys may be strings or symbols. |
+| BigQuery     | Ruby                                 | Notes                                                        |
+|------------- |--------------------------------------|--------------------------------------------------------------|
+| `BOOL`       | `true`/`false`                       |                                                              |
+| `INT64`      | `Integer`                            |                                                              |
+| `FLOAT64`    | `Float`                              |                                                              |
+| `NUMERIC`    | `BigDecimal`                         | `BigDecimal` values will be rounded to scale 9.              |
+| `BIGNUMERIC` | converted to `BigDecimal`            | Pass data as `String` and map query param values in `types`. |
+| `STRING`     | `String`                             |                                                              |
+| `DATETIME`   | `DateTime`                           | `DATETIME` does not support time zone.                       |
+| `DATE`       | `Date`                               |                                                              |
+| `TIMESTAMP`  | `Time`                               |                                                              |
+| `TIME`       | `Google::Cloud::BigQuery::Time`      |                                                              |
+| `BYTES`      | `File`, `IO`, `StringIO`, or similar |                                                              |
+| `ARRAY`      | `Array`                              | Nested arrays, `nil` values are not supported.               |
+| `STRUCT`     | `Hash`                               | Hash keys may be strings or symbols.                         |
 
 See [Data
 Types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types)

data/lib/google/cloud/bigquery/convert.rb

@@ -23,30 +23,28 @@ require "date"
 module Google
   module Cloud
     module Bigquery
-      # rubocop:disable Metrics/ModuleLength
-
       ##
       # @private
       #
-      # Internal conversion of raw data values to/from Bigquery values
+      # Internal conversion of raw data values to/from BigQuery values
+      #
+      #   | BigQuery     | Ruby                                 | Notes                                              |
+      #   |--------------|--------------------------------------|----------------------------------------------------|
+      #   | `BOOL`       | `true`/`false`                       |                                                    |
+      #   | `INT64`      | `Integer`                            |                                                    |
+      #   | `FLOAT64`    | `Float`                              |                                                    |
+      #   | `NUMERIC`    | `BigDecimal`                         | `BigDecimal` values will be rounded to scale 9.    |
+      #   | `BIGNUMERIC` | converted to `BigDecimal`            | Pass data as `String`; map query params in `types`.|
+      #   | `STRING`     | `String`                             |                                                    |
+      #   | `DATETIME`   | `DateTime`                           | `DATETIME` does not support time zone.             |
+      #   | `DATE`       | `Date`                               |                                                    |
+      #   | `TIMESTAMP`  | `Time`                               |                                                    |
+      #   | `TIME`       | `Google::Cloud::BigQuery::Time`      |                                                    |
+      #   | `BYTES`      | `File`, `IO`, `StringIO`, or similar |                                                    |
+      #   | `ARRAY`      | `Array`                              | Nested arrays, `nil` values are not supported.     |
+      #   | `STRUCT`     | `Hash`                               | Hash keys may be strings or symbols.               |
       #
-      # | BigQuery    | Ruby           | Notes  |
-      # |-------------|----------------|---|
-      # | `BOOL`      | `true`/`false` | |
-      # | `INT64`     | `Integer`      | |
-      # | `FLOAT64`   | `Float`        | |
-      # | `NUMERIC`   | `BigDecimal`   | Will be rounded to 9 decimal places |
-      # | `STRING`    | `String`       | |
-      # | `DATETIME`  | `DateTime`  | `DATETIME` does not support time zone. |
-      # | `DATE`      | `Date`         | |
-      # | `TIMESTAMP` | `Time`         | |
-      # | `TIME`      | `Google::Cloud::BigQuery::Time` | |
-      # | `BYTES`     | `File`, `IO`, `StringIO`, or similar | |
-      # | `ARRAY` | `Array` | Nested arrays, `nil` values are not supported. |
-      # | `STRUCT`    | `Hash`        | Hash keys may be strings or symbols. |
       module Convert
-        ##
-        # @private
         def self.format_rows rows, fields
           Array(rows).map do |row|
             # convert TableRow to hash to handle nested TableCell values
@@ -54,8 +52,6 @@ module Google
           end
         end
 
-        ##
-        # @private
         def self.format_row row, fields
           row_pairs = fields.zip(row[:f]).map do |f, v|
             [f.name.to_sym, format_value(v, f)]
@@ -92,6 +88,8 @@ module Google
             end
           elsif field.type == "NUMERIC"
             BigDecimal value[:v]
+          elsif field.type == "BIGNUMERIC"
+            BigDecimal value[:v]
           elsif field.type == "BOOLEAN"
             (value[:v] == "true" ? true : (value[:v] == "false" ? false : nil))
           elsif field.type == "BYTES"
@@ -109,30 +107,21 @@ module Google
           end
         end
 
-        ##
-        # @private
-        def self.to_query_param param, type = nil
-          type ||= default_query_param_type_for param
-
-          Google::Apis::BigqueryV2::QueryParameter.new(
-            parameter_type:  to_query_param_type(type),
-            parameter_value: to_query_param_value(param)
-          )
-        end
+        # rubocop:enable all
 
-        ##
-        # @private
-        def self.to_query_param_value value
+        def self.to_query_param_value value, type = nil
           return Google::Apis::BigqueryV2::QueryParameterValue.new value: nil if value.nil?
 
-          json_value = to_json_value value
+          json_value = to_json_value value, type
 
-          if Array === json_value
-            array_values = json_value.map { |v| to_query_param_value v }
+          case json_value
+          when Array
+            type = extract_array_type type
+            array_values = json_value.map { |v| to_query_param_value v, type }
             Google::Apis::BigqueryV2::QueryParameterValue.new array_values: array_values
-          elsif Hash === json_value
-            struct_pairs = json_value.map do |key, value|
-              [String(key), to_query_param_value(value)]
+          when Hash
+            struct_pairs = json_value.map do |k, v|
+              [String(k), to_query_param_value(v, type)]
             end
             struct_values = Hash[struct_pairs]
             Google::Apis::BigqueryV2::QueryParameterValue.new struct_values: struct_values
@@ -143,12 +132,13 @@ module Google
         end
 
         def self.to_query_param_type type
-          if Array === type
+          case type
+          when Array
             Google::Apis::BigqueryV2::QueryParameterType.new(
               type: "ARRAY".freeze,
               array_type: to_query_param_type(type.first)
             )
-          elsif Hash === type
+          when Hash
             Google::Apis::BigqueryV2::QueryParameterType.new(
               type: "STRUCT".freeze,
               struct_types: type.map do |key, val|
@@ -159,10 +149,14 @@ module Google
               end
             )
           else
-            Google::Apis::BigqueryV2::QueryParameterType.new(type: type.to_s.freeze)
+            Google::Apis::BigqueryV2::QueryParameterType.new type: type.to_s.freeze
           end
         end
 
+        # rubocop:disable Lint/DuplicateBranch
+        # rubocop:disable Metrics/CyclomaticComplexity
+        # rubocop:disable Style/GuardClause
+
         def self.default_query_param_type_for param
           raise ArgumentError, "nil params are not supported, must assign optional type" if param.nil?
 
@@ -202,9 +196,9 @@ module Google
             end
             [non_nil_values.first]
           when Hash
-            Hash[param.map do |key, value|
-              [key, default_query_param_type_for(value)]
-            end]
+            param.transform_values do |value|
+              default_query_param_type_for value
+            end
           else
             if param.respond_to?(:read) && param.respond_to?(:rewind)
               :BYTES
@@ -214,9 +208,11 @@ module Google
           end
         end
 
-        ##
-        # @private
-        def self.to_json_value value
+        # rubocop:enable Lint/DuplicateBranch
+        # rubocop:enable Metrics/CyclomaticComplexity
+        # rubocop:enable Style/GuardClause
+
+        def self.to_json_value value, type = nil
           if DateTime === value
             value.strftime "%Y-%m-%d %H:%M:%S.%6N"
           elsif Date === value
@@ -226,30 +222,46 @@ module Google
           elsif Bigquery::Time === value
             value.value
           elsif BigDecimal === value
-            # Round to precision of 9
-            value.finite? ? value.round(9).to_s("F") : value.to_s
+            if value.finite?
+              # Round to precision of 9 unless explicit `BIGNUMERIC`
+              bigdecimal = type == :BIGNUMERIC ? value : value.round(9)
+              bigdecimal.to_s "F"
+            else
+              value.to_s
+            end
           elsif value.respond_to?(:read) && value.respond_to?(:rewind)
             value.rewind
-            Base64.strict_encode64(value.read.force_encoding("ASCII-8BIT"))
+            Base64.strict_encode64 value.read.force_encoding("ASCII-8BIT")
           elsif Array === value
-            value.map { |v| to_json_value v }
+            type = extract_array_type type
+            value.map { |x| to_json_value x, type }
           elsif Hash === value
-            Hash[value.map { |k, v| [k.to_s, to_json_value(v)] }]
+            Hash[value.map { |k, v| [k.to_s, to_json_value(v, type)] }]
           else
             value
           end
         end
 
-        # rubocop:enable all
+        def self.to_query_param param, type = nil
+          type ||= default_query_param_type_for param
 
-        ##
-        # @private
-        def self.to_json_rows rows
-          rows.map { |row| to_json_row row }
+          Google::Apis::BigqueryV2::QueryParameter.new(
+            parameter_type:  to_query_param_type(type),
+            parameter_value: to_query_param_value(param, type)
+          )
         end
 
         ##
-        # @private
+        # Lists are specified by providing the type code in an array. For example, an array of integers are specified as
+        # `[:INT64]`. Extracts the symbol.
+        def self.extract_array_type type
+          return nil if type.nil?
+          unless type.is_a?(Array) && type.count == 1 && type.first.is_a?(Symbol)
+            raise ArgumentError, "types Array #{type.inspect} should include only a single symbol element."
+          end
+          type.first
+        end
+
         def self.to_json_row row
           Hash[row.map { |k, v| [k.to_s, to_json_value(v)] }]
         end
@@ -261,8 +273,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts create disposition strings to API values.
         #
         # @return [String] API representation of create disposition.
@@ -281,8 +291,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts write disposition strings to API values.
         #
         # @return [String] API representation of write disposition.
@@ -303,8 +311,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts source format strings to API values.
         #
         # @return [String] API representation of source format.
@@ -327,8 +333,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts file paths into source format by extension.
         #
         # @return [String] API representation of source format.
@@ -339,8 +343,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts file path into source format by extension.
         #
         # @return [String] API representation of source format.
@@ -355,8 +357,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts a primitive time value in milliseconds to a Ruby Time object.
         #
         # @return [Time, nil] The Ruby Time object, or nil if the given argument
@@ -367,8 +367,6 @@ module Google
         end
 
         ##
-        # @private
-        #
         # Converts a Ruby Time object to a primitive time value in milliseconds.
         #
         # @return [Integer, nil] The primitive time value in milliseconds, or
@@ -378,8 +376,6 @@ module Google
           (time_obj.to_i * 1000) + (time_obj.nsec / 1_000_000)
         end
       end
-
-      # rubocop:enable Metrics/ModuleLength
     end
   end
 end

data/lib/google/cloud/bigquery/copy_job.rb

@@ -144,6 +144,7 @@ module Google
           ##
           # @private Create an Updater object.
           def initialize gapi
+            super()
             @gapi = gapi
           end