google-cloud-pubsub 1.0.2 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8d3c2b3976af69e59bdb1ea9653f42164db682ce28a929b5325d760c46bab4aa
-  data.tar.gz: 0f922862ba91d39e1ba8f55a66d2a6699392157ed850e129ed127dc9f63ac1d5
+  metadata.gz: 60202a1d96c472dc3bb05992db2ed19da1e34077ceeb5e0a066126a69d569f30
+  data.tar.gz: b671179cf63643ec45598fe63b29440504f2d24ce9f2d8495ec0e16e66838809
 SHA512:
-  metadata.gz: d33fdcdb157fa0795e6fc28517ccdd26b8b03b3380dd96bd6f931063eae3ac30a5c4d3b5e9dafd32bbc93ee76e4db3544611c4b1ac9a63ad6068422c9bab466e
-  data.tar.gz: b8218aceaa79d35777e540a43764e7e49474e5dc393c508384f12e4dff09e7d5e1470a8d5a9f2ed6bf96c17d54c63be8489aefbf32babccfeb57a15035867521
+  metadata.gz: 01ab09622f6773bfe6a0a5b96171eebb348c71dffd13b06d48eb1754d7cfde3464f81d51051f4a60d5d72c7e39b1b9e746b8417f0d9b86b50dd54280978f3ca4
+  data.tar.gz: c0eeb49f436cd8e385ce16b37f5a93445dd29417729624edd572e5a5bc2454c8b3e4504edba9cd73ffe227be64b4b68bfcadceb6e65983b12dfccb222a2879e2

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Release History
 
+### 1.1.0 / 2019-10-23
+
+#### Features
+
+* Add support for Ordering Keys
+  * Google Cloud Pub/Sub ordering keys provide the ability to ensure related
+    messages are sent to subscribers in the order in which they were published.
+    The service guarantees that, for a given ordering key and publisher, messages
+    are sent to subscribers in the order in which they were published.
+  * Note: At the time of this release, ordering keys are not yet publicly enabled
+    and requires special project enablements.
+  * Add Google::Cloud::PubSub::Topic#enable_message_ordering! method.
+  * Add Google::Cloud::PubSub::Topic#message_ordering? method.
+  * Add ordering_key argument to Google::Cloud::PubSub::Topic#publish_async method.
+  * Add Google::Cloud::PubSub::Topic#resume_publish method.
+  * Add message_ordering argument to Google::Cloud::PubSub::Topic#subscribe method.
+  * Add Google::Cloud::PubSub::Subscription#message_ordering? method.
+* Update Ruby dependency to minimum of 2.4.
+
 ### 1.0.2 / 2019-10-10
 
 #### Bug Fixes

data/CONTRIBUTING.md

@@ -24,7 +24,7 @@ be able to accept your pull requests.
 In order to use the google-cloud-pubsub console and run the project's tests,
 there is a small amount of setup:
 
-1. Install Ruby. google-cloud-pubsub requires Ruby 2.3+. You may choose to
+1. Install Ruby. google-cloud-pubsub requires Ruby 2.4+. You may choose to
    manage your Ruby and gem installations with [RVM](https://rvm.io/),
    [rbenv](https://github.com/rbenv/rbenv), or
    [chruby](https://github.com/postmodern/chruby).

data/OVERVIEW.md

@@ -161,7 +161,7 @@ msgs = topic.publish do |batch|
 end
 ```
 
-## Receiving messages
+## Receiving Messages
 
 Messages can be streamed from a subscription with a subscriber object that is
 created using `listen`. (See {Google::Cloud::PubSub::Subscription#listen
@@ -316,6 +316,88 @@ received_messages = sub.pull
 sub.modify_ack_deadline 120, received_messages
 ```
 
+## Using Ordering Keys
+
+Google Cloud Pub/Sub ordering keys provide the ability to ensure related
+messages are sent to subscribers in the order in which they were published.
+Messages can be tagged with an ordering key, a string that identifies related
+messages for which publish order should be respected. The service guarantees
+that, for a given ordering key and publisher, messages are sent to subscribers
+in the order in which they were published. Ordering does not require sacrificing
+high throughput or scalability, as the service automatically distributes
+messages for different ordering keys across subscribers.
+
+Note: At the time of this release, ordering keys are not yet publicly enabled
+and requires special project enablements.
+
+### Publishing Ordered Messages
+
+To use ordering keys when publishing messages, a call to
+{Google::Cloud::PubSub::Topic#enable_message_ordering!
+Topic#enable_message_ordering!} must be made and the `ordering_key` argument
+must be provided when calling {Google::Cloud::PubSub::Topic#publish_async
+Topic#publish_async}.
+
+```ruby
+require "google/cloud/pubsub"
+
+pubsub = Google::Cloud::PubSub.new
+
+topic = pubsub.topic "my-ordered-topic"
+
+# Ensure that message ordering is enabled.
+topic.enable_message_ordering!
+
+# Publish an ordered message with an ordering key.
+topic.publish_async "task completed",
+                    ordering_key: "task-key"
+
+# Shut down the publisher when ready to stop publishing messages.
+topic.async_publisher.stop.wait!
+```
+
+### Handling errors with Ordered Keys
+
+Ordered messages that fail to publish to the Pub/Sub API due to error will put
+the `ordering_key` in a failed state, and future calls to
+{Google::Cloud::PubSub::Topic#publish_async Topic#publish_async} with the
+`ordering_key` will raise {Google::Cloud::PubSub::OrderingKeyError
+OrderingKeyError}. To allow future messages with the `ordering_key` to be
+published, the `ordering_key` must be passed to
+{Google::Cloud::PubSub::Topic#resume_publish Topic#resume_publish}.
+
+### Receiving Ordered Messages
+
+To use ordering keys when subscribing to messages, the subscription must be
+created with message ordering enabled (See
+{Google::Cloud::PubSub::Topic#subscribe Topic#subscribe} and
+{Google::Cloud::PubSub::Subscription#message_ordering?
+Subscription#message_ordering?}) before calling
+{Google::Cloud::PubSub::Subscription#listen Subscription#listen}. When enabled,
+the subscriber will deliver messages with the same `ordering_key` in the order
+they were published.
+
+```ruby
+require "google/cloud/pubsub"
+
+pubsub = Google::Cloud::PubSub.new
+
+sub = pubsub.subscription "my-ordered-topic-sub"
+sub.message_ordering? #=> true
+
+subscriber = sub.listen do |received_message|
+  # Messsages with the same ordering_key are received
+  # in the order in which they were published.
+  received_message.acknowledge!
+end
+
+# Start background threads that will call block passed to listen.
+subscriber.start
+
+# Shut down the subscriber when ready to stop receiving messages.
+subscriber.stop.wait!
+```
+
 ## Minimizing API calls before receiving and acknowledging messages
 
 A subscription object can be created without making any API calls by providing

data/lib/google-cloud-pubsub.rb

@@ -64,9 +64,7 @@ module Google
     #
     def pubsub scope: nil, timeout: nil, client_config: nil
       timeout ||= @timeout
-      Google::Cloud.pubsub @project, @keyfile, scope:         scope,
-                                               timeout:       timeout,
-                                               client_config: client_config
+      Google::Cloud.pubsub @project, @keyfile, scope: scope, timeout: timeout, client_config: client_config
     end
 
     ##
@@ -108,10 +106,8 @@ module Google
     def self.pubsub project_id = nil, credentials = nil, scope: nil,
                     timeout: nil, client_config: nil
       require "google/cloud/pubsub"
-      Google::Cloud::PubSub.new project_id: project_id,
-                                credentials: credentials,
-                                scope: scope, timeout: timeout,
-                                client_config: client_config
+      Google::Cloud::PubSub.new project_id: project_id, credentials: credentials,
+                                scope: scope, timeout: timeout, client_config: client_config
     end
   end
 end
@@ -123,8 +119,7 @@ Google::Cloud.configure.add_config! :pubsub do |config|
   end
   default_creds = Google::Cloud::Config.deferred do
     Google::Cloud::Config.credentials_from_env(
-      "PUBSUB_CREDENTIALS", "PUBSUB_CREDENTIALS_JSON",
-      "PUBSUB_KEYFILE", "PUBSUB_KEYFILE_JSON"
+      "PUBSUB_CREDENTIALS", "PUBSUB_CREDENTIALS_JSON", "PUBSUB_KEYFILE", "PUBSUB_KEYFILE_JSON"
     )
   end
   default_emulator = Google::Cloud::Config.deferred do
@@ -133,15 +128,12 @@ Google::Cloud.configure.add_config! :pubsub do |config|
 
   config.add_field! :project_id, default_project, match: String, allow_nil: true
   config.add_alias! :project, :project_id
-  config.add_field! :credentials, default_creds,
-                    match:     [String, Hash, Google::Auth::Credentials],
-                    allow_nil: true
+  config.add_field! :credentials, default_creds, match: [String, Hash, Google::Auth::Credentials], allow_nil: true
   config.add_alias! :keyfile, :credentials
   config.add_field! :scope, nil, match: [String, Array]
   config.add_field! :timeout, nil, match: Integer
   config.add_field! :client_config, nil, match: Hash
-  config.add_field! :emulator_host, default_emulator,
-                    match: String, allow_nil: true
+  config.add_field! :emulator_host, default_emulator, match: String, allow_nil: true
   config.add_field! :on_error, nil, match: Proc
   config.add_field! :endpoint, nil, match: String
 end

data/lib/google/cloud/pubsub.rb

@@ -77,9 +77,8 @@ module Google
       #   topic = pubsub.topic "my-topic"
       #   topic.publish "task completed"
       #
-      def self.new project_id: nil, credentials: nil, scope: nil, timeout: nil,
-                   client_config: nil, endpoint: nil, emulator_host: nil,
-                   project: nil, keyfile: nil
+      def self.new project_id: nil, credentials: nil, scope: nil, timeout: nil, client_config: nil, endpoint: nil,
+                   emulator_host: nil, project: nil, keyfile: nil
         project_id    ||= (project || default_project_id)
         scope         ||= configure.scope
         timeout       ||= configure.timeout
@@ -93,9 +92,7 @@ module Google
 
           return PubSub::Project.new(
             PubSub::Service.new(
-              project_id, :this_channel_is_insecure,
-              host: emulator_host, timeout: timeout,
-              client_config: client_config
+              project_id, :this_channel_is_insecure, host: emulator_host, timeout: timeout, client_config: client_config
             )
           )
         end
@@ -105,17 +102,13 @@ module Google
           credentials = PubSub::Credentials.new credentials, scope: scope
         end
 
-        if credentials.respond_to? :project_id
-          project_id ||= credentials.project_id
-        end
+        project_id ||= credentials.project_id if credentials.respond_to? :project_id
         project_id = project_id.to_s # Always cast to a string
         raise ArgumentError, "project_id is missing" if project_id.empty?
 
         PubSub::Project.new(
           PubSub::Service.new(
-            project_id, credentials, timeout:       timeout,
-                                     host:          endpoint,
-                                     client_config: client_config
+            project_id, credentials, timeout: timeout, host: endpoint, client_config: client_config
           )
         )
       end

data/lib/google/cloud/pubsub/async_publisher.rb

@@ -15,8 +15,11 @@
 
 require "monitor"
 require "concurrent"
+require "google/cloud/pubsub/errors"
+require "google/cloud/pubsub/async_publisher/batch"
 require "google/cloud/pubsub/publish_result"
 require "google/cloud/pubsub/service"
+require "google/cloud/pubsub/convert"
 
 module Google
   module Cloud
@@ -68,8 +71,7 @@ module Google
 
         ##
         # @private Create a new instance of the object.
-        def initialize topic_name, service, max_bytes: 10000000,
-                       max_messages: 1000, interval: 0.25, threads: {}
+        def initialize topic_name, service, max_bytes: 10_000_000, max_messages: 1000, interval: 0.25, threads: {}
           @topic_name = service.topic_path topic_name
           @service    = service
 
@@ -79,6 +81,14 @@ module Google
           @publish_threads  = (threads[:publish] || 4).to_i
           @callback_threads = (threads[:callback] || 8).to_i
 
+          @published_at = nil
+          @publish_thread_pool = Concurrent::ThreadPoolExecutor.new max_threads: @publish_threads
+          @callback_thread_pool = Concurrent::ThreadPoolExecutor.new max_threads: @callback_threads
+          @thread = Thread.new { run_background }
+
+          @ordered = false
+          @batches = {}
+
           @cond = new_cond
 
           # init MonitorMixin
@@ -89,29 +99,45 @@ module Google
         # Add a message to the async publisher to be published to the topic.
         # Messages will be collected in batches and published together.
         # See {Google::Cloud::PubSub::Topic#publish_async}
-        def publish data = nil, attributes = {}, &block
-          msg = create_pubsub_message data, attributes
+        #
+        # @param [String, File] data The message payload. This will be converted
+        #   to bytes encoded as ASCII-8BIT.
+        # @param [Hash] attributes Optional attributes for the message.
+        # @param [String] ordering_key Identifies related messages for which
+        #   publish order should be respected.
+        # @yield [result] the callback for when the message has been published
+        # @yieldparam [PublishResult] result the result of the asynchronous
+        #   publish
+        # @raise [Google::Cloud::PubSub::AsyncPublisherStopped] when the
+        #   publisher is stopped. (See {#stop} and {#stopped?}.)
+        # @raise [Google::Cloud::PubSub::OrderedMessagesDisabled] when
+        #   publishing a message with an `ordering_key` but ordered messages are
+        #   not enabled. (See {#message_ordering?} and
+        #   {#enable_message_ordering!}.)
+        # @raise [Google::Cloud::PubSub::OrderingKeyError] when publishing a
+        #   message with an `ordering_key` that has already failed when
+        #   publishing. Use {#resume_publish} to allow this `ordering_key` to be
+        #   published again.
+        #
+        def publish data = nil, attributes = nil, ordering_key: nil, **extra_attrs, &callback
+          msg = Convert.pubsub_message data, attributes, ordering_key, extra_attrs
 
           synchronize do
-            raise "Can't publish when stopped." if @stopped
-
-            if @batch.nil?
-              @batch ||= Batch.new self
-              @batch.add msg, block
-            else
-              unless @batch.try_add msg, block
-                publish_batch!
-                @batch = Batch.new self
-                @batch.add msg, block
-              end
+            raise AsyncPublisherStopped if @stopped
+            raise OrderedMessagesDisabled if !@ordered && !msg.ordering_key.empty? # default is empty string
+
+            batch = resolve_batch_for_message msg
+            raise OrderingKeyError, batch.ordering_key if batch.canceled?
+            batch_action = batch.add msg, callback
+            if batch_action == :full
+              publish_batches!
+            elsif @published_at.nil?
+              # Set initial time to now to start the background counter
+              @published_at = Time.now
             end
-
-            init_resources!
-
-            publish_batch! if @batch.ready?
-
             @cond.signal
           end
+
           nil
         end
 
@@ -127,9 +153,9 @@ module Google
             break if @stopped
 
             @stopped = true
-            publish_batch!
+            publish_batches! stop: true
             @cond.signal
-            @publish_thread_pool.shutdown if @publish_thread_pool
+            @publish_thread_pool.shutdown
           end
 
           self
@@ -149,14 +175,10 @@ module Google
         # @return [AsyncPublisher] returns self so calls can be chained.
         def wait! timeout = nil
           synchronize do
-            if @publish_thread_pool
-              @publish_thread_pool.wait_for_termination timeout
-            end
+            @publish_thread_pool.wait_for_termination timeout
 
-            if @callback_thread_pool
-              @callback_thread_pool.shutdown
-              @callback_thread_pool.wait_for_termination timeout
-            end
+            @callback_thread_pool.shutdown
+            @callback_thread_pool.wait_for_termination timeout
           end
 
           self
@@ -185,7 +207,7 @@ module Google
         # @return [AsyncPublisher] returns self so calls can be chained.
         def flush
           synchronize do
-            publish_batch!
+            publish_batches!
             @cond.signal
           end
 
@@ -208,33 +230,63 @@ module Google
           synchronize { @stopped }
         end
 
-        protected
+        ##
+        # Enables message ordering for messages with ordering keys. When
+        # enabled, messages published with the same `ordering_key` will be
+        # delivered in the order they were published.
+        #
+        # See {#message_ordering?}. See {Topic#publish_async},
+        # {Subscription#listen}, and {Message#ordering_key}.
+        #
+        def enable_message_ordering!
+          synchronize { @ordered = true }
+        end
 
-        # rubocop:disable Naming/MemoizedInstanceVariableName
+        ##
+        # Whether message ordering for messages with ordering keys has been
+        # enabled. When enabled, messages published with the same `ordering_key`
+        # will be delivered in the order they were published. When disabled,
+        # messages may be delivered in any order.
+        #
+        # See {#enable_message_ordering!}. See {Topic#publish_async},
+        # {Subscription#listen}, and {Message#ordering_key}.
+        #
+        # @return [Boolean]
+        #
+        def message_ordering?
+          synchronize { @ordered }
+        end
 
-        def init_resources!
-          @first_published_at   ||= Time.now
-          @publish_thread_pool  ||= Concurrent::ThreadPoolExecutor.new \
-            max_threads: @publish_threads
-          @callback_thread_pool ||= Concurrent::ThreadPoolExecutor.new \
-            max_threads: @callback_threads
-          @thread ||= Thread.new { run_background }
+        ##
+        # Resume publishing ordered messages for the provided ordering key.
+        #
+        # @param [String] ordering_key Identifies related messages for which
+        #   publish order should be respected.
+        #
+        # @return [boolean] `true` when resumed, `false` otherwise.
+        #
+        def resume_publish ordering_key
+          synchronize do
+            batch = resolve_batch_for_ordering_key ordering_key
+            return if batch.nil?
+            batch.resume!
+          end
         end
 
-        # rubocop:enable Naming/MemoizedInstanceVariableName
+        protected
 
         def run_background
           synchronize do
             until @stopped
-              if @batch.nil?
+              if @published_at.nil?
                 @cond.wait
                 next
               end
 
-              time_since_first_publish = Time.now - @first_published_at
+              time_since_first_publish = Time.now - @published_at
               if time_since_first_publish > @interval
-                # interval met, publish the batch...
-                publish_batch!
+                # interval met, flush the batches...
+                publish_batches!
                 @cond.wait
               else
                 # still waiting for the interval to publish the batch...
@@ -245,40 +297,93 @@ module Google
           end
         end
 
-        def publish_batch!
-          return unless @batch
+        def resolve_batch_for_message msg
+          @batches[msg.ordering_key] ||= Batch.new self, msg.ordering_key
+        end
 
-          publish_batch_async @topic_name, @batch
-          @batch = nil
-          @first_published_at = nil
+        def resolve_batch_for_ordering_key ordering_key
+          @batches[ordering_key]
+        end
+
+        def publish_batches! stop: nil
+          @batches.reject! { |_ordering_key, batch| batch.empty? }
+          @batches.values.each do |batch|
+            ready = batch.publish! stop: stop
+            publish_batch_async @topic_name, batch if ready
+          end
+          # Set published_at to nil to wait indefinitely
+          @published_at = nil
         end
 
         def publish_batch_async topic_name, batch
+          # TODO: raise unless @publish_thread_pool.running?
           return unless @publish_thread_pool.running?
 
           Concurrent::Promises.future_on(
             @publish_thread_pool, topic_name, batch
-          ) do |t_name, btch|
-            publish_batch_sync t_name, btch
-          end
+          ) { |t, b| publish_batch_sync t, b }
         end
 
+        # rubocop:disable Metrics/AbcSize
+        # rubocop:disable Metrics/MethodLength
+
         def publish_batch_sync topic_name, batch
-          grpc = @service.publish topic_name, batch.messages
-          batch.items.zip Array(grpc.message_ids) do |item, id|
-            next unless item.callback
+          # The only batch methods that are safe to call from the loop are
+          # rebalance! and reset! because they are the only methods that are
+          # synchronized.
+          loop do
+            items = batch.rebalance!
+
+            unless items.empty?
+              grpc = @service.publish topic_name, items.map(&:msg)
+              items.zip Array(grpc.message_ids) do |item, id|
+                next unless item.callback
+
+                item.msg.message_id = id
+                publish_result = PublishResult.from_grpc item.msg
+                execute_callback_async item.callback, publish_result
+              end
+            end
 
-            item.msg.message_id = id
-            publish_result = PublishResult.from_grpc item.msg
-            execute_callback_async item.callback, publish_result
+            break unless batch.reset!
           end
         rescue StandardError => e
-          batch.items.each do |item|
+          items = batch.items
+
+          unless batch.ordering_key.empty?
+            retry if publish_batch_error_retryable? e
+            # Cancel the batch if the error is not to be retried.
+            begin
+              raise OrderingKeyError, batch.ordering_key
+            rescue OrderingKeyError => e
+              # The existing e variable is not set to OrderingKeyError
+              # Get all unsent messages for the callback
+              items = batch.cancel!
+            end
+          end
+
+          items.each do |item|
             next unless item.callback
 
             publish_result = PublishResult.from_error item.msg, e
             execute_callback_async item.callback, publish_result
           end
+
+          # publish will retry indefinitely, as long as there are unsent items.
+          retry if batch.reset!
+        end
+
+        # rubocop:enable Metrics/AbcSize
+        # rubocop:enable Metrics/MethodLength
+
+        PUBLISH_RETRY_ERRORS = [
+          GRPC::Cancelled, GRPC::DeadlineExceeded, GRPC::Internal,
+          GRPC::ResourceExhausted, GRPC::Unauthenticated, GRPC::Unavailable,
+          GRPC::Core::CallError
+        ].freeze
+
+        def publish_batch_error_retryable? error
+          PUBLISH_RETRY_ERRORS.any? { |klass| error.is_a? klass }
         end
 
         def execute_callback_async callback, publish_result
@@ -290,79 +395,6 @@ module Google
             cback.call p_result
           end
         end
-
-        def create_pubsub_message data, attributes
-          attributes ||= {}
-          if data.is_a?(::Hash) && attributes.empty?
-            attributes = data
-            data = nil
-          end
-          # Convert IO-ish objects to strings
-          if data.respond_to?(:read) && data.respond_to?(:rewind)
-            data.rewind
-            data = data.read
-          end
-          # Convert data to encoded byte array to match the protobuf defn
-          data_bytes = \
-            String(data).dup.force_encoding(Encoding::ASCII_8BIT).freeze
-
-          # Convert attributes to strings to match the protobuf definition
-          attributes = Hash[attributes.map { |k, v| [String(k), String(v)] }]
-
-          Google::Cloud::PubSub::V1::PubsubMessage.new data:       data_bytes,
-                                                       attributes: attributes
-        end
-
-        ##
-        # @private
-        class Batch
-          attr_reader :messages, :callbacks
-
-          def initialize publisher
-            @publisher = publisher
-            @messages = []
-            @callbacks = []
-            @total_message_bytes = publisher.topic_name.bytesize + 2
-          end
-
-          def add msg, callback
-            @messages << msg
-            @callbacks << callback
-            @total_message_bytes += msg.to_proto.bytesize + 2
-          end
-
-          def try_add msg, callback
-            new_message_count = total_message_count + 1
-            new_message_bytes = total_message_bytes + msg.to_proto.bytesize + 2
-            if new_message_count > @publisher.max_messages ||
-               new_message_bytes >= @publisher.max_bytes
-              return false
-            end
-            add msg, callback
-            true
-          end
-
-          def ready?
-            total_message_count >= @publisher.max_messages ||
-              total_message_bytes >= @publisher.max_bytes
-          end
-
-          def total_message_count
-            @messages.count
-          end
-
-          def total_message_bytes
-            @total_message_bytes
-          end
-
-          def items
-            @messages.zip(@callbacks).map do |msg, callback|
-              Item.new msg, callback
-            end
-          end
-
-          Item = Struct.new :msg, :callback
-        end
       end
     end