google-cloud-pubsub 2.1.1 → 2.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 450987f2e0c301c5b43af3211dda2c6698dfa76a29076c47dc9a9690698fa9df
-  data.tar.gz: 84f19d7d3aacaa290e7364558ebd29795f3fce76783254df8d2df43b653e02f8
+  metadata.gz: f6636ac65229c0dc438c63fa9aebda474396701364bfbea7e5da460ceea50a26
+  data.tar.gz: a0c0cafbbaf4083c1141d851dc6cecb8cb70bc01df940a139fe21d057aa2fb97
 SHA512:
-  metadata.gz: 7b10cd6edeb5cf4d352028f58a233eb8f56f4de079ed4f2b344cb930750efe16758c736bb160635a8e019083ddf04fe01255175f18467d35a46966e9d1c19787
-  data.tar.gz: b67f8da4a9c0b73fa065a9364773dbaaa2682d993163f9bac62fab849b00178aaeab258cc0d4962e74106535be540ec2e42cfb6b888d973b4b97c06efb217f32
+  metadata.gz: a187e9935196b66d4be65c7cf7b34148de36197f9e2c5e9144ccc9f78652370d7a441b33e14a002133b05aa6565cc38998912277e8896d81178bf5712ce9df8c
+  data.tar.gz: ee801f9d5959f5e2a38ccf13beaac930ca4629c81a268b627a8bacc7ad9ce430581fd690100fa5ef57b4fa5e20e3c5f40d72e905ce2535d978398f03112a48eb

data/CHANGELOG.md

@@ -1,5 +1,93 @@
 # Release History
 
+### 2.7.0 / 2021-06-15
+
+#### Features
+
+* Add Publisher Flow Control
+  * Add flow_control to async options in Project#create_topic and Project#topic
+  * Add FlowControlLimitError
+
+#### Bug Fixes
+
+* Fix Project#schema and #schemas to return full resource
+  * Include schema definition in default return values.
+  * Fix Schema#definition to return nil instead of empty string when not present.
+
+### 2.6.1 / 2021-04-28
+
+#### Bug Fixes
+
+* Add final flush of pending requests to Subscriber#wait!
+  * fix(pubsub): Add final flush of pending requests to Subscriber#wait!
+
+### 2.6.0 / 2021-04-19
+
+#### Features
+
+* Add ordering_key to Topic#publish
+  * Add ordering_key to BatchPublisher#publish
+
+#### Documentation
+
+* The immediate: false option is recommended to avoid adverse impacts on the performance of pull operations ([#11153](https://www.github.com/googleapis/google-cloud-ruby/issues/11153))
+* Update Subscription#pull docs and samples to recommend immediate: false
+
+### 2.5.0 / 2021-04-01
+
+#### Features
+
+* Add Schema support
+  * Add Schema
+  * Add Project#create_schema
+  * Add Project#schema
+  * Add Project#schemas (Schema::List)
+  * Add Project#valid_schema?
+  * Add schema options to Project#create_topic
+  * Add Topic#schema_name
+  * Add Topic#message_encoding
+  * Add Topic#message_encoding_binary?
+  * Add Topic#message_encoding_json?
+
+### 2.4.0 / 2021-03-10
+
+#### Features
+
+* Drop support for Ruby 2.4 and add support for Ruby 3.0
+
+### 2.3.2 / 2021-02-08
+
+#### Bug Fixes
+
+* Fix project option in Project#topic and Project#subscription
+  * Ensure that project option is used when skip_lookup is false.
+  * Improve documentation of topic_name, subscription_name and snapshot_name.
+
+### 2.3.1 / 2021-01-13
+
+#### Bug Fixes
+
+* Update Subscription#retry_policy=
+  * Remove conditional RPC to fetch full resource before update.
+
+### 2.3.0 / 2020-11-18
+
+#### Features
+
+* Add inventory.use_legacy_flow_control to listen options
+  * Add inventory.use_legacy_flow_control to Subscription#listen options
+  * Add Subscriber#use_legacy_flow_control?
+
+#### Documentation
+
+* Remove EXPERIMENTAL label from RetryPolicy docs
+
+### 2.2.0 / 2020-11-11
+
+#### Features
+
+* Add Subscription#remove_dead_letter_policy
+
 ### 2.1.1 / 2020-10-26
 
 #### Documentation

data/CONTRIBUTING.md

@@ -24,7 +24,7 @@ be able to accept your pull requests.
 In order to use the google-cloud-pubsub console and run the project's tests,
 there is a small amount of setup:
 
-1. Install Ruby. google-cloud-pubsub requires Ruby 2.4+. You may choose to
+1. Install Ruby. google-cloud-pubsub requires Ruby 2.5+. You may choose to
    manage your Ruby and gem installations with [RVM](https://rvm.io/),
    [rbenv](https://github.com/rbenv/rbenv), or
    [chruby](https://github.com/postmodern/chruby).
@@ -45,7 +45,7 @@ there is a small amount of setup:
 
    ```sh
    $ cd google-cloud-pubsub/
-   $ bundle exec rake bundleupdate
+   $ bundle install
    ```
 
 ## Console
@@ -119,15 +119,14 @@ If you alter an example's title, you may encounter breaking tests.
 ### Pub/Sub Acceptance Tests
 
 The Pub/Sub acceptance tests interact with the live service API. Follow the
-instructions in the {file:AUTHENTICATION.md Authentication guide} for enabling
+instructions in the {file:AUTHENTICATION.md Authentication Guide} for enabling
 the Pub/Sub API. Occasionally, some API features may not yet be generally
 available, making it difficult for some contributors to successfully run the
 entire acceptance test suite. However, please ensure that you do successfully
 run acceptance tests for any code areas covered by your pull request.
 
 To run the acceptance tests, first create and configure a project in the Google
-Developers Console, as described in the {file:AUTHENTICATION.md Authentication
-guide}. Be sure to download the JSON KEY file. Make note of the PROJECT_ID and
+Developers Console, as described in the {file:AUTHENTICATION.md Authentication Guide}. Be sure to download the JSON KEY file. Make note of the PROJECT_ID and
 the KEYFILE location on your system.
 
 Before you can run the Pub/Sub acceptance tests, you must first create indexes

data/LOGGING.md

@@ -3,7 +3,7 @@
 To enable logging for this library, set the logger for the underlying
 [gRPC](https://github.com/grpc/grpc/tree/master/src/ruby) library. The logger
 that you set may be a Ruby stdlib
-[`Logger`](https://ruby-doc.org/stdlib-2.5.0/libdoc/logger/rdoc/Logger.html) as
+[`Logger`](https://ruby-doc.org/stdlib/libdoc/logger/rdoc/Logger.html) as
 shown below, or a
 [`Google::Cloud::Logging::Logger`](https://googleapis.dev/ruby/google-cloud-logging/latest)
 that will write logs to [Stackdriver

data/OVERVIEW.md

@@ -205,13 +205,16 @@ sleep
 Messages also can be pulled directly in a one-time operation. (See
 {Google::Cloud::PubSub::Subscription#pull Subscription#pull})
 
+The `immediate: false` option is recommended to avoid adverse impacts on the
+performance of pull operations.
+
 ```ruby
 require "google/cloud/pubsub"
 
 pubsub = Google::Cloud::PubSub.new
 
 sub = pubsub.subscription "my-topic-sub"
-received_messages = sub.pull
+received_messages = sub.pull immediate: false
 ```
 
 A maximum number of messages to pull can be specified:
@@ -222,7 +225,7 @@ require "google/cloud/pubsub"
 pubsub = Google::Cloud::PubSub.new
 
 sub = pubsub.subscription "my-topic-sub"
-received_messages = sub.pull max: 10
+received_messages = sub.pull immediate: false, max: 10
 ```
 
 ## Acknowledging a Message
@@ -263,7 +266,7 @@ require "google/cloud/pubsub"
 pubsub = Google::Cloud::PubSub.new
 
 sub = pubsub.subscription "my-topic-sub"
-received_messages = sub.pull
+received_messages = sub.pull immediate: false
 sub.acknowledge received_messages
 ```
 
@@ -328,7 +331,7 @@ require "google/cloud/pubsub"
 pubsub = Google::Cloud::PubSub.new
 
 sub = pubsub.subscription "my-topic-sub"
-received_messages = sub.pull
+received_messages = sub.pull immediate: false
 sub.modify_ack_deadline 120, received_messages
 ```
 
@@ -474,7 +477,7 @@ sub = pubsub.subscription "my-topic-sub"
 
 snapshot = sub.create_snapshot
 
-received_messages = sub.pull
+received_messages = sub.pull immediate: false
 sub.acknowledge received_messages
 
 sub.seek snapshot
@@ -521,5 +524,5 @@ sub.topic.name #=> "projects/other-project-id/topics/other-topic"
 ## Additional information
 
 Google Cloud Pub/Sub can be configured to use an emulator or to enable gRPC's
-logging. To learn more, see the {file:EMULATOR.md Emulator guide} and
+logging. To learn more, see the {file:EMULATOR.md Emulator guide}} and
 {file:LOGGING.md Logging guide}.

data/lib/google/cloud/pubsub.rb

@@ -33,8 +33,6 @@ module Google
     # See {file:OVERVIEW.md Google Cloud Pub/Sub Overview}.
     #
     module PubSub
-      # rubocop:disable Metrics/AbcSize
-
       ##
       # Creates a new object for connecting to the Pub/Sub service.
       # Each call creates a new connection.
@@ -110,8 +108,6 @@ module Google
         PubSub::Project.new service
       end
 
-      # rubocop:enable Metrics/AbcSize
-
       ##
       # Configure the Google Cloud PubSub library.
       #

data/lib/google/cloud/pubsub/async_publisher.rb

@@ -16,6 +16,7 @@
 require "monitor"
 require "concurrent"
 require "google/cloud/pubsub/errors"
+require "google/cloud/pubsub/flow_controller"
 require "google/cloud/pubsub/async_publisher/batch"
 require "google/cloud/pubsub/publish_result"
 require "google/cloud/pubsub/service"
@@ -44,8 +45,8 @@ module Google
       #
       #   topic.async_publisher.stop!
       #
-      # @attr_reader [String] topic_name The name of the topic the messages are published to. In the form of
-      #   "/projects/project-identifier/topics/topic-name".
+      # @attr_reader [String] topic_name The name of the topic the messages are published to. The value is a
+      #   fully-qualified topic name in the form `projects/{project_id}/topics/{topic_id}`.
       # @attr_reader [Integer] max_bytes The maximum size of messages to be collected before the batch is published.
       #   Default is 1,000,000 (1MB).
       # @attr_reader [Integer] max_messages The maximum number of messages to be collected before the batch is
@@ -59,16 +60,27 @@ module Google
       class AsyncPublisher
         include MonitorMixin
 
-        attr_reader :topic_name, :max_bytes, :max_messages, :interval,
-                    :publish_threads, :callback_threads
+        attr_reader :topic_name
+        attr_reader :max_bytes
+        attr_reader :max_messages
+        attr_reader :interval
+        attr_reader :publish_threads
+        attr_reader :callback_threads
+        attr_reader :flow_control
         ##
         # @private Implementation accessors
         attr_reader :service, :batch, :publish_thread_pool,
-                    :callback_thread_pool
+                    :callback_thread_pool, :flow_controller
 
         ##
         # @private Create a new instance of the object.
-        def initialize topic_name, service, max_bytes: 1_000_000, max_messages: 100, interval: 0.01, threads: {}
+        def initialize topic_name,
+                       service,
+                       max_bytes: 1_000_000,
+                       max_messages: 100,
+                       interval: 0.01,
+                       threads: {},
+                       flow_control: {}
           # init MonitorMixin
           super()
           @topic_name = service.topic_path topic_name
@@ -79,6 +91,10 @@ module Google
           @interval         = interval
           @publish_threads  = (threads[:publish] || 2).to_i
           @callback_threads = (threads[:callback] || 4).to_i
+          @flow_control = {
+            message_limit: 10 * @max_messages,
+            byte_limit: 10 * @max_bytes
+          }.merge(flow_control).freeze
 
           @published_at = nil
           @publish_thread_pool = Concurrent::ThreadPoolExecutor.new max_threads: @publish_threads
@@ -87,7 +103,7 @@ module Google
           @ordered = false
           @batches = {}
           @cond = new_cond
-
+          @flow_controller = FlowController.new(**@flow_control)
           @thread = Thread.new { run_background }
         end
 
@@ -117,13 +133,22 @@ module Google
         #
         def publish data = nil, attributes = nil, ordering_key: nil, **extra_attrs, &callback
           msg = Convert.pubsub_message data, attributes, ordering_key, extra_attrs
+          begin
+            @flow_controller.acquire msg.to_proto.bytesize
+          rescue FlowControlLimitError => e
+            stop_publish ordering_key, e if ordering_key
+            raise
+          end
 
           synchronize do
             raise AsyncPublisherStopped if @stopped
             raise OrderedMessagesDisabled if !@ordered && !msg.ordering_key.empty? # default is empty string
 
             batch = resolve_batch_for_message msg
-            raise OrderingKeyError, batch.ordering_key if batch.canceled?
+            if batch.canceled?
+              @flow_controller.release msg.to_proto.bytesize
+              raise OrderingKeyError, batch.ordering_key
+            end
             batch_action = batch.add msg, callback
             if batch_action == :full
               publish_batches!
@@ -301,9 +326,24 @@ module Google
           @batches[ordering_key]
         end
 
+        def stop_publish ordering_key, err
+          synchronize do
+            batch = resolve_batch_for_ordering_key ordering_key
+            return if batch.nil?
+            items = batch.cancel!
+            items.each do |item|
+              @flow_controller.release item.bytesize
+              next unless item.callback
+
+              publish_result = PublishResult.from_error item.msg, err
+              execute_callback_async item.callback, publish_result
+            end
+          end
+        end
+
         def publish_batches! stop: nil
           @batches.reject! { |_ordering_key, batch| batch.empty? }
-          @batches.values.each do |batch|
+          @batches.each_value do |batch|
             ready = batch.publish! stop: stop
             publish_batch_async @topic_name, batch if ready
           end
@@ -321,7 +361,6 @@ module Google
         end
 
         # rubocop:disable Metrics/AbcSize
-        # rubocop:disable Metrics/MethodLength
 
         def publish_batch_sync topic_name, batch
           # The only batch methods that are safe to call from the loop are
@@ -333,6 +372,7 @@ module Google
             unless items.empty?
               grpc = @service.publish topic_name, items.map(&:msg)
               items.zip Array(grpc.message_ids) do |item, id|
+                @flow_controller.release item.bytesize
                 next unless item.callback
 
                 item.msg.message_id = id
@@ -359,6 +399,7 @@ module Google
           end
 
           items.each do |item|
+            @flow_controller.release item.bytesize
             next unless item.callback
 
             publish_result = PublishResult.from_error item.msg, e
@@ -370,7 +411,6 @@ module Google
         end
 
         # rubocop:enable Metrics/AbcSize
-        # rubocop:enable Metrics/MethodLength
 
         PUBLISH_RETRY_ERRORS = [
           GRPC::Cancelled, GRPC::DeadlineExceeded, GRPC::Internal,

data/lib/google/cloud/pubsub/async_publisher/batch.rb

@@ -25,7 +25,8 @@ module Google
         class Batch
           include MonitorMixin
 
-          attr_reader :items, :ordering_key
+          attr_reader :items
+          attr_reader :ordering_key
 
           def initialize publisher, ordering_key
             # init MonitorMixin
@@ -128,8 +129,8 @@ module Google
           end
 
           ##
-          # Rebalances the batch by moving any queued items that will fit into
-          # the active item list.
+          # Fills the batch by sequentially moving the queued items that will
+          # fit into the active item list.
           #
           # This method is only intended to be used by the active publishing
           # job.
@@ -151,8 +152,6 @@ module Google
             end
           end
 
-          # rubocop:disable Metrics/MethodLength
-
           ##
           # Resets the batch after a successful publish. This clears the active
           # item list and moves the queued items that will fit into the active
@@ -201,8 +200,6 @@ module Google
             true
           end
 
-          # rubocop:enable Metrics/MethodLength
-
           ##
           # Cancel the batch and hault futher batches until resumed. See
           # {#resume!} and {#canceled?}.

data/lib/google/cloud/pubsub/batch_publisher.rb

@@ -13,6 +13,8 @@
 # limitations under the License.
 
 
+require "google/cloud/pubsub/convert"
+
 module Google
   module Cloud
     module PubSub
@@ -26,11 +28,12 @@ module Google
       #   pubsub = Google::Cloud::PubSub.new
       #
       #   topic = pubsub.topic "my-topic"
-      #   msgs = topic.publish do |t|
-      #     t.publish "task 1 completed", foo: :bar
-      #     t.publish "task 2 completed", foo: :baz
-      #     t.publish "task 3 completed", foo: :bif
+      #   msgs = topic.publish do |batch_publisher|
+      #     batch_publisher.publish "task 1 completed", foo: :bar
+      #     batch_publisher.publish "task 2 completed", foo: :baz
+      #     batch_publisher.publish "task 3 completed", foo: :bif
       #   end
+      #
       class BatchPublisher
         ##
         # @private The messages to publish
@@ -38,20 +41,40 @@ module Google
 
         ##
         # @private Create a new instance of the object.
-        def initialize data = nil, attributes = {}
+        def initialize data, attributes, ordering_key, extra_attrs
           @messages = []
           @mode = :batch
           return if data.nil?
           @mode = :single
-          publish data, attributes
+          publish data, attributes, ordering_key: ordering_key, **extra_attrs
         end
 
         ##
         # Add a message to the batch to be published to the topic.
         # All messages added to the batch will be published at once.
         # See {Google::Cloud::PubSub::Topic#publish}
-        def publish data, attributes = {}
-          @messages << create_pubsub_message(data, attributes)
+        #
+        # @param [String, File] data The message payload. This will be converted
+        #   to bytes encoded as ASCII-8BIT.
+        # @param [Hash] attributes Optional attributes for the message.
+        # @param [String] ordering_key Identifies related messages for which
+        #   publish order should be respected.
+        #
+        # @example Multiple messages can be sent at the same time using a block:
+        #   require "google/cloud/pubsub"
+        #
+        #   pubsub = Google::Cloud::PubSub.new
+        #
+        #   topic = pubsub.topic "my-topic"
+        #   msgs = topic.publish do |batch_publisher|
+        #     batch_publisher.publish "task 1 completed", foo: :bar
+        #     batch_publisher.publish "task 2 completed", foo: :baz
+        #     batch_publisher.publish "task 3 completed", foo: :bif
+        #   end
+        #
+        def publish data, attributes = nil, ordering_key: nil, **extra_attrs
+          msg = Convert.pubsub_message data, attributes, ordering_key, extra_attrs
+          @messages << msg
         end
 
         ##
@@ -69,28 +92,6 @@ module Google
             msgs
           end
         end
-
-        protected
-
-        def create_pubsub_message data, attributes
-          attributes ||= {}
-          if data.is_a?(::Hash) && attributes.empty?
-            attributes = data
-            data = nil
-          end
-          # Convert IO-ish objects to strings
-          if data.respond_to?(:read) && data.respond_to?(:rewind)
-            data.rewind
-            data = data.read
-          end
-          # Convert data to encoded byte array to match the protobuf defn
-          data_bytes = String(data).dup.force_encoding(Encoding::ASCII_8BIT).freeze
-
-          # Convert attributes to strings to match the protobuf definition
-          attributes = Hash[attributes.map { |k, v| [String(k), String(v)] }]
-
-          Google::Cloud::PubSub::V1::PubsubMessage.new data: data_bytes, attributes: attributes
-        end
       end
     end