pubsub_client 1.2.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f957900bdf24e83c57067bb4e1d21c0010bf0e7d6aacc45da639c2250c7fa1b4
-  data.tar.gz: 036d770cee6fa93d718bb5aa91236825eb8bfe510293a0e352c611c46645bed0
+  metadata.gz: 4efa5b8b5abeff4f0dd13ee6d945c8f733b0b3b46dac6528fcf4bd527a5f08a5
+  data.tar.gz: c69aa4f023b2af7fbfe44c7021650026b7a31d2e49f5e543741b213fbb53d005
 SHA512:
-  metadata.gz: 6f52651a2b7b2f7279266a96850a84e4ca3639d454ce5f6fa437276b545c8c79e5d7fa99cddd760f9b0987c18739003eca20efbeadebe17e9fa794c042f8c3b5
-  data.tar.gz: 8157eaf01bef5317607e77b687f107fa3579a1a4b251bc8e6d4922a600d177274436a9302130642de2f6f8d86f9f79ca6f42bed5b3a973b92f52874194e68d92
+  metadata.gz: c4d38c468281279004e4504fe4a01b31652937b57270b0d63ce5930da8fefc82ec5d1ad2c12f5d0921001e12c460d0fe9e808c5fdaf1c790f47d4556ee7ed4b7
+  data.tar.gz: a7586ca0100862cb137627e110ffcae41e46d775645ef88622c6e594746eac636f7e7be67cb68ebffebfd0c899a5e7bd9bd79281e72a2ed3534fd3b49c7817df

data/CHANGELOG.txt

@@ -1,3 +1,32 @@
+Version 2.0.0 2021-03-30
+------------------------
+172dc19 Add support for multiple clients
+3525193 Update subscriber example to include name of subscription instead of …
+
+Version 1.5.0 2020-11-05
+------------------------
+6cab366 Do not require GAC when stubbed
+dab08dd Update `Subscriber#on_error` documentation
+
+Version 1.4.1 2020-10-26
+------------------------
+d28d99f Pass published attributes all the way through the client
+
+Version 1.4.0 2020-10-26
+------------------------
+60b8403 Publish attributes
+
+Version 1.3.0 2020-10-23
+------------------------
+e9e87f7 This adds the ability to subscribe to a Pub/Sub topic. Of note, we expose two additional configuration params:
+- `concurrency`: the number of threads the subscriber will run to process messages (defaults to 8 threads)
+-  `auto_ack`: flag to auto ack messages (default is `true` and _will_ ack messages)
+
+These changes come with a handful of useful checks:
+- ensures credentials are configured prior to subscribing
+- raises an error if the target subscription does not exist
+- raises an error if attempting to subscribe to a topic that has already been subscribed to
+
 Version 1.2.0 2020-09-25
 ------------------------
 aede3ab Raise custom error if GOOGLE_APPLICATION_CREDENTIALS not set

data/README.md

@@ -22,21 +22,25 @@ Or install it yourself as:
 
 In order to use this gem, the environment variable `GOOGLE_APPLICATION_CREDENTIALS` must be set and point to the credentials JSON file.
 
-If there are environments where setting up credentials is too burdensome and/or publishing messages is not desired, `PubsubClient` can be stubbed out with `PubsubClient.stub!`, e.g.
+If there are environments where setting up credentials is too burdensome and/or publishing messages is not desired, `PubsubClient` can be stubbed out with `.stub!`, e.g.
 
 ```ruby
+@client = PubsubClient.new
+
 if test_env?
-  PubsubClient.stub!
+   @client.stub!
 end
 ```
 
 ## Usage
 
-To publish a message to Pub/Sub, call `PubsubClient.publish(message, 'the-topic')`. This method takes any serializable object as an argument and yields a result object to a block. The `result` object has a method `#succeeded?` that returns `true` if the message was successfully published, otherwise `false`. In the latter case, there is a method `#error` that returns the error.
+### Publishing
+
+To publish a message to Pub/Sub, call `@client.publish(message, 'the-topic')`. This method takes any serializable object as an argument and yields a result object to a block. The `result` object has a method `#succeeded?` that returns `true` if the message was successfully published, otherwise `false`. In the latter case, there is a method `#error` that returns the error.
 
-### Example
+#### Example
 ```ruby
-PubsubClient.publish(message, 'some-topic') do |result|
+@client.publish(message, 'some-topic') do |result|
   if result.succeeded?
     puts 'yay!'
   else
@@ -45,14 +49,54 @@ PubsubClient.publish(message, 'some-topic') do |result|
 end
 ```
 
+### Subscribing
+
+To subscribe to a topic, a client must first get a handle to the subscriber object. After doing so, a call to `subscriber.listener` will yield two arguments: the data (most clients will only need this) and the full Pub/Sub message (for anything more robust). Documentation for the full message can be found [here](https://googleapis.dev/ruby/google-cloud-pubsub/latest/Google/Cloud/PubSub/ReceivedMessage.html).
+
+Optionally, a client can choose to handle exceptions raised by the subscriber. If a client chooses to do so, the listener **must** be configured before `on_error` since the latter needs a handler to the underlying listener. Additionally, exceptions will only be raised when the work inside the block is happening on the same thread. For instance, if the block enqueues a background worker and that worker raises an error, it will not be handled by the `on_error` block.
+
+#### Example
+```ruby
+subscriber = @client.subscriber('some-subscription')
+
+subscriber.listener(concurrency: 4, auto_ack: false) do |data, received_message|
+  # Most clients will only need the first yielded arg.
+  # It is the same as calling received_message.data
+end
+
+# Optional
+subscriber.on_error do |ex|
+  # Do something with the exception.
+end
+
+subscriber.subscribe # This will sleep
+```
+
+By default, the underlying subscriber will use a concurrency of `8` threads and will acknowledge all messages. If these defaults are acceptable to the client, no arguments need to be passed in the call to `listener`.
+```ruby
+subscriber.listener do |data, received_message|
+  # Do something
+end
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To install this gem onto your local machine, run `bundle exec rake install`.
 
 ## Contributing
 
+To contribute, open a pull request against `main`. Note that once your changes have been made, you should _not_ manually modify the `version.rb` or `CHANGELOG` as these will get updated automatically as part of the release process.
+
+To release a new version, after you have merged your changes into `main`:
+1. Run the `gem-release` script. This can be found [here](https://github.com/apartmentlist/scripts/blob/main/bin/gem-release).
+    ```
+    path/to/gem-release [major/minor/patch] "Short message with changes"
+    ```
+    Note that the `Short message with changes` is what gets reflected in the releases of the repo.
+1. Run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). This step will update the `version.rb` and `CHANGELOG` files.
+
 Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/pubsub_client. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/pubsub_client/blob/master/CODE_OF_CONDUCT.md).
 
 

data/lib/pubsub_client.rb

@@ -1,26 +1,50 @@
+# frozen_string_literal: true
+
 require 'pubsub_client/version'
 require 'pubsub_client/null_publisher_factory'
+require 'pubsub_client/null_subscriber_factory'
 require 'pubsub_client/publisher_factory'
+require 'pubsub_client/subscriber_factory'
 
-module PubsubClient
+class PubsubClient
   Error = Class.new(StandardError)
   ConfigurationError = Class.new(Error)
   CredentialsError = Class.new(Error)
   InvalidTopicError = Class.new(Error)
+  InvalidSubscriptionError = Class.new(Error)
 
-  class << self
-    def stub!
-      raise ConfigurationError, 'PubsubClient is already configured' if @publisher_factory
-      @publisher_factory = NullPublisherFactory.new
-    end
+  def stub!
+    raise ConfigurationError, 'PubsubClient is already configured' if @publisher_factory || @subscriber_factory
+
+    @publisher_factory = NullPublisherFactory.new
+    @subscriber_factory = NullSubscriberFactory.new
+    @stubbed = true
+  end
+
+  # @param subscription [String] - The name of the subscription to subscribe to.
+  def subscriber(subscription)
+    ensure_credentials!
+
+    @subscriber_factory ||= SubscriberFactory.new
+    @subscriber_factory.build(subscription)
+  end
+
+  def publish(message, topic, attributes = {}, &block)
+    ensure_credentials!
+
+    @publisher_factory ||= PublisherFactory.new
+    @publisher_factory.build(topic).publish(message, attributes, &block)
+  end
+
+  private
+
+  attr_reader :stubbed, :publisher_factory, :subscriber_factory
 
-    def publish(message, topic, &block)
-      unless ENV['GOOGLE_APPLICATION_CREDENTIALS']
-        raise CredentialsError, 'GOOGLE_APPLICATION_CREDENTIALS must be set'
-      end
+  def ensure_credentials!
+    return if defined?(stubbed) && stubbed
 
-      @publisher_factory ||= PublisherFactory.new
-      @publisher_factory.build(topic).publish(message, &block)
+    unless ENV['GOOGLE_APPLICATION_CREDENTIALS']
+      raise CredentialsError, 'GOOGLE_APPLICATION_CREDENTIALS must be set'
     end
   end
 end

data/lib/pubsub_client/null_publisher.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-module PubsubClient
+class PubsubClient
   # A null object to act as a publisher when clients are in dev or test
   class NullPublisher
     # This is required so that this publisher maintains the same contract

data/lib/pubsub_client/null_publisher_factory.rb

@@ -2,7 +2,7 @@
 
 require_relative 'null_publisher'
 
-module PubsubClient
+class PubsubClient
   # A null object to act as a publisher factory when clients are in dev or test
   class NullPublisherFactory
     def build(*)

data/lib/pubsub_client/null_subscriber.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+class PubsubClient
+  # A null object to act as a subscriber when clients are in dev or test
+  class NullSubscriber
+    # This adds a subset of the available methods on the
+    # Google::Cloud::PubSub::ReceivedMessage, which is what
+    # gets yielded by the subscription when configuring the listener.
+    # For a list of methods, see the following link:
+    # https://googleapis.dev/ruby/google-cloud-pubsub/latest/Google/Cloud/PubSub/ReceivedMessage.html
+    NullResult = Struct.new(:acknowledge!) do
+      def data
+        '{"key":"value"}'
+      end
+
+      def published_at
+        Time.now
+      end
+    end
+
+    def listener(*, &block)
+      res = NullResult.new
+      yield res.data, res
+    end
+
+    def subscribe
+      # no-op
+    end
+
+    def on_error(&block)
+      # no-op
+    end
+  end
+end

data/lib/pubsub_client/null_subscriber_factory.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require_relative 'null_subscriber'
+
+class PubsubClient
+  # A null object to act as a subscriber factory when clients are in dev or test
+  class NullSubscriberFactory
+    def build(*)
+      NullSubscriber.new
+    end
+  end
+end

data/lib/pubsub_client/publisher.rb

@@ -2,14 +2,15 @@
 
 require 'google/cloud/pubsub'
 
-module PubsubClient
+class PubsubClient
   class Publisher
+    # @param topic [Google::Cloud::PubSub::Topic]
     def initialize(topic)
       @topic = topic
     end
 
-    def publish(message, &block)
-      topic.publish_async(message, &block)
+    def publish(message, attributes = {}, &block)
+      topic.publish_async(message, attributes, &block)
     end
 
     def flush

data/lib/pubsub_client/publisher_factory.rb

@@ -2,7 +2,7 @@
 
 require_relative 'publisher'
 
-module PubsubClient
+class PubsubClient
   # Build and memoize the Publisher, accounting for GRPC's requirements around forking.
   class PublisherFactory
     def initialize
@@ -10,6 +10,8 @@ module PubsubClient
       @publishers = {}
     end
 
+    # @param topic_name [String]
+    # @return [Publisher]
     def build(topic_name)
       # GRPC fails when attempting to use a connection created in a process that gets
       # forked with the message

data/lib/pubsub_client/subscriber.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require 'google/cloud/pubsub'
+
+class PubsubClient
+  class Subscriber
+    DEFAULT_CONCURRENCY = 8
+
+    # @param subscription [Google::Cloud::PubSub::Subscription]
+    def initialize(subscription)
+      @subscription = subscription
+    end
+
+    # @param concurrency [Integer] - The number of threads to run the subscriber with. Default is 8.
+    # @param auto_ack [Boolean] - Flag to acknowledge the Pub/Sub message. A message must be acked
+    # to remove it from the topic. Default is `true`.
+    #
+    # @return [Google::Cloud::PubSub::Subscriber]
+    def listener(concurrency: DEFAULT_CONCURRENCY, auto_ack: true, &block)
+      @listener ||= begin
+        @subscription.listen(threads: { callback: concurrency }) do |received_message|
+          yield received_message.data, received_message
+          received_message.acknowledge! if auto_ack
+        end
+      end
+    end
+
+    def subscribe
+      raise ConfigurationError, 'A listener must be configured' unless @listener
+
+      begin
+        @listener.start
+
+        sleep
+      rescue SignalException
+        @listener.stop.wait!
+      end
+    end
+
+    def on_error(&block)
+      raise ConfigurationError, 'A listener must be configured' unless @listener
+
+      @listener.on_error do |exception|
+        yield exception
+      end
+    end
+  end
+end

data/lib/pubsub_client/subscriber_factory.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require_relative 'subscriber'
+
+class PubsubClient
+  class SubscriberFactory
+    def initialize
+      @subscribers = {}
+    end
+
+    # @param subscription_name [String]
+    # @retrun [Subscriber]
+    def build(subscription_name)
+      if @subscribers.key?(subscription_name)
+        raise ConfigurationError, "PubsubClient already subscribed to #{subscription_name}"
+      end
+
+      @subscribers[subscription_name] = build_subscriber(subscription_name)
+    end
+
+    private
+
+    def build_subscriber(subscription_name)
+      pubsub = Google::Cloud::PubSub.new
+      subscription = pubsub.subscription(subscription_name)
+      raise InvalidSubscriptionError, "The subscription #{subscription_name} does not exist" unless subscription
+      Subscriber.new(subscription)
+    end
+  end
+end

data/lib/pubsub_client/version.rb

@@ -1,3 +1,3 @@
-module PubsubClient
-  VERSION = '1.2.0'
+class PubsubClient
+  VERSION = '2.0.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pubsub_client
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 2.0.0
 platform: ruby
 authors:
 - Apartment List Platforms
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-09-29 00:00:00.000000000 Z
+date: 2021-03-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -88,8 +88,12 @@ files:
 - lib/pubsub_client.rb
 - lib/pubsub_client/null_publisher.rb
 - lib/pubsub_client/null_publisher_factory.rb
+- lib/pubsub_client/null_subscriber.rb
+- lib/pubsub_client/null_subscriber_factory.rb
 - lib/pubsub_client/publisher.rb
 - lib/pubsub_client/publisher_factory.rb
+- lib/pubsub_client/subscriber.rb
+- lib/pubsub_client/subscriber_factory.rb
 - lib/pubsub_client/version.rb
 - pubsub_client.gemspec
 homepage: https://github.com/apartmentlist/pubsub_client