pubsub_client 0.1.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2508e7f65206f72e0092d048d71416a77bf1391fcda8a13e43fbd50c2a4ae561
-  data.tar.gz: 7d1f9e9b1e0f9e6af43d732da015a08cb04cc7983237fd6ed4e5a24cd94f58c9
+  metadata.gz: fcd27cf1ed4e02b4b8bae0e9b2f5227d970f158196b68d903cbfa2b031c2b1eb
+  data.tar.gz: 415e17dfb4465a44ad9ffe3aeabe8b116a8f47b79f77e3a408d67da8c44409fa
 SHA512:
-  metadata.gz: 016b31843467a3de64bf9331650ce2ca371fd11c3ebe19e6c2b99e905407ad745e793c56ec08831e20e173f45e4cc13f89eb36a07f5eae33669fc4488c2acbc0
-  data.tar.gz: d07b99e406fe71164285e94624b53c0355ad8c9ac9c2fd2cbc2738f38fd19a3b1baef6845053000bf20cd21aa4ca4cc723c1d1e006da4b2a9c15859aad9145bb
+  metadata.gz: bacbf023017f2ba125de0be70c2040e697c5c7927f24569f6e5af043d78eefc0eee813af9bfaeadbd4fde3213554bbe135ef011a3343408552b69ea217aa4bac
+  data.tar.gz: 9a84a7a5c32ba17ea5523e3b0831935dc162d89a0cc7dcd89d1a3863951796dee63fa59639d583de72965d75e42047ca0ec275b1dde3541ae4b40d818e1e8359

data/CHANGELOG.txt

@@ -0,0 +1,31 @@
+Version 1.4.0 2020-10-26
+------------------------
+60b8403 Publish attributes
+
+Version 1.3.0 2020-10-23
+------------------------
+e9e87f7 This adds the ability to subscribe to a Pub/Sub topic. Of note, we expose two additional configuration params:
+- `concurrency`: the number of threads the subscriber will run to process messages (defaults to 8 threads)
+-  `auto_ack`: flag to auto ack messages (default is `true` and _will_ ack messages)
+
+These changes come with a handful of useful checks:
+- ensures credentials are configured prior to subscribing
+- raises an error if the target subscription does not exist
+- raises an error if attempting to subscribe to a topic that has already been subscribed to
+
+Version 1.2.0 2020-09-25
+------------------------
+aede3ab Raise custom error if GOOGLE_APPLICATION_CREDENTIALS not set
+
+Version 1.1.0 2020-09-25
+------------------------
+d3f8ed1 Throw error if topic does not exist
+a95430f Bump major version and update change log
+
+Version 1.0.0 2020-09-24
+------------------------
+1d7dc1c Allow publishing to any topic
+
+Version 0.1.0 2020-08-25
+------------------------
+92a2e7a Initial release

data/README.md

@@ -20,30 +20,25 @@ Or install it yourself as:
 
 ## Configuration
 
-In order to use this gem, the environment variable `GOOGLE_APPLICATION_CREDENTIALS` must be set and point to the credentials JSON file. Additionally, here are configuration settings that may need to be set:
-- `topic_name` (required unless stubbed) - name of the Google Cloud Pub/Sub topic to publish messages to.
+In order to use this gem, the environment variable `GOOGLE_APPLICATION_CREDENTIALS` must be set and point to the credentials JSON file.
 
-If there are environments where setting up credentials is too burdensome and/or publishing messages is not desired, `PubsubClient` can be stubbed out with `PubsubClient.stub!`
-
-E.g.
+If there are environments where setting up credentials is too burdensome and/or publishing messages is not desired, `PubsubClient` can be stubbed out with `PubsubClient.stub!`, e.g.
 
 ```ruby
 if test_env?
   PubsubClient.stub!
-else
-  PubsubClient.configure do |config|
-    config.topic_name = 'some-topic'
-  end
 end
 ```
 
 ## Usage
 
-To publish a message to Pub/Sub, call `PubsubClient.publish(message)`. This method takes any serializable object as an argument and yields a result object to a block. The `result` object has a method `#succeeded?` that returns `true` if the message was successfully published, otherwise `false`. In the latter case, there is a method `#error` that returns the error.
+### Publishing
+
+To publish a message to Pub/Sub, call `PubsubClient.publish(message, 'the-topic')`. This method takes any serializable object as an argument and yields a result object to a block. The `result` object has a method `#succeeded?` that returns `true` if the message was successfully published, otherwise `false`. In the latter case, there is a method `#error` that returns the error.
 
-### Example
+#### Example
 ```ruby
-PubsubClient.publish(message) do |result|
+PubsubClient.publish(message, 'some-topic') do |result|
   if result.succeeded?
     puts 'yay!'
   else
@@ -52,14 +47,54 @@ PubsubClient.publish(message) do |result|
 end
 ```
 
+### Subscribing
+
+To subscribe to a topic, a client must first get a handle to the subscriber object. After doing so, a call to `subscriber.listener` will yield two arguments: the data (most clients will only need this) and the full Pub/Sub message (for anything more robust). Documentation for the full message can be found [here](https://googleapis.dev/ruby/google-cloud-pubsub/latest/Google/Cloud/PubSub/ReceivedMessage.html).
+
+Optionally, a client can choose to handle exceptions raised by the subscriber. If a client chooses to do so, the listener **must** be configured before `on_error` since the latter needs a handler to the underlying listener.
+
+#### Example
+```ruby
+subscriber = PubsubClient.subscriber('some-topic')
+
+subscriber.listener(concurrency: 4, auto_ack: false) do |data, received_message|
+  # Most clients will only need the first yielded arg.
+  # It is the same as calling received_message.data
+end
+
+# Optional
+subscriber.on_error do |ex|
+  # Do something with the exception.
+end
+
+subscriber.subscribe # This will sleep
+```
+
+By default, the underlying subscriber will use a concurrency of `8` threads and will acknowledge all messages. If these defaults are acceptable to the client, no arguments need to be passed in the call to `listener`.
+```ruby
+subscriber.listener do |data, received_message|
+  # Do something
+end
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To install this gem onto your local machine, run `bundle exec rake install`.
 
 ## Contributing
 
+To contribute, open a pull request against `main`. Note that once your changes have been made, you should _not_ manually modify the `version.rb` or `CHANGELOG` as these will get updated automatically as part of the release process.
+
+To release a new version, after you have merged your changes into `main`:
+1. Run the `gem-release` script. This can be found [here](https://github.com/apartmentlist/scripts/blob/main/bin/gem-release).
+    ```
+    path/to/gem-release [major/minor/patch] "Short message with changes"
+    ```
+    Note that the `Short message with changes` is what gets reflected in the releases of the repo.
+1. Run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). This step will update the `version.rb` and `CHANGELOG` files.
+
 Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/pubsub_client. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/pubsub_client/blob/master/CODE_OF_CONDUCT.md).
 
 

data/lib/pubsub_client.rb

@@ -1,43 +1,46 @@
 require 'pubsub_client/version'
 require 'pubsub_client/null_publisher_factory'
+require 'pubsub_client/null_subscriber_factory'
 require 'pubsub_client/publisher_factory'
+require 'pubsub_client/subscriber_factory'
 
 module PubsubClient
   Error = Class.new(StandardError)
-  CredentialsError = Class.new(Error)
   ConfigurationError = Class.new(Error)
-
-  Config = Struct.new(:topic_name)
+  CredentialsError = Class.new(Error)
+  InvalidTopicError = Class.new(Error)
+  InvalidSubscriptionError = Class.new(Error)
 
   class << self
-    def configure(&block)
-      raise ConfigurationError, 'PubsubClient is already configured' if @publisher_factory
+    def stub!
+      raise ConfigurationError, 'PubsubClient is already configured' if @publisher_factory || @subscriber_factory
+      @publisher_factory = NullPublisherFactory.new
+      @subscriber_factory = NullSubscriberFactory.new
+    end
 
-      unless ENV['GOOGLE_APPLICATION_CREDENTIALS']
-        raise CredentialsError, 'GOOGLE_APPLICATION_CREDENTIALS must be set.'
-      end
+    # @param message [String] The message to publish.
+    # @param topic [String] The name of the topic to publish to.
+    def publish(message, topic, &block)
+      ensure_credentials!
 
-      config = Config.new
-      yield config
+      @publisher_factory ||= PublisherFactory.new
+      @publisher_factory.build(topic).publish(message, &block)
+    end
 
-      unless config.topic_name
-        raise ConfigurationError, 'The topic_name must be configured.'
-      end
+    # @param subscription [String] - The name of the topic to subscribe to.
+    def subscriber(subscription)
+      ensure_credentials!
 
-      @publisher_factory = PublisherFactory.new(config.topic_name)
+      @subscriber_factory ||= SubscriberFactory.new
+      @subscriber_factory.build(subscription)
     end
 
-    def stub!
-      raise ConfigurationError, 'PubsubClient is already configured' if @publisher_factory
-      @publisher_factory = NullPublisherFactory.new
-    end
+    private
 
-    def publish(message, &block)
-      unless @publisher_factory
-        raise ConfigurationError, 'PubsubClient must be configured or stubbed'
+    def ensure_credentials!
+      unless ENV['GOOGLE_APPLICATION_CREDENTIALS']
+        raise CredentialsError, 'GOOGLE_APPLICATION_CREDENTIALS must be set'
       end
-
-      @publisher_factory.build.publish(message, &block)
     end
   end
 end

data/lib/pubsub_client/null_publisher_factory.rb

@@ -5,7 +5,7 @@ require_relative 'null_publisher'
 module PubsubClient
   # A null object to act as a publisher factory when clients are in dev or test
   class NullPublisherFactory
-    def build
+    def build(*)
       NullPublisher.new
     end
   end

data/lib/pubsub_client/null_subscriber.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module PubsubClient
+  # A null object to act as a subscriber when clients are in dev or test
+  class NullSubscriber
+    # This adds a subset of the available methods on the
+    # Google::Cloud::PubSub::ReceivedMessage, which is what
+    # gets yielded by the subscription when configuring the listener.
+    # For a list of methods, see the following link:
+    # https://googleapis.dev/ruby/google-cloud-pubsub/latest/Google/Cloud/PubSub/ReceivedMessage.html
+    NullResult = Struct.new(:acknowledge!) do
+      def data
+        '{"key":"value"}'
+      end
+
+      def published_at
+        Time.now
+      end
+    end
+
+    def listener(*, &block)
+      res = NullResult.new
+      yield res.data, res
+    end
+
+    def subscribe
+      # no-op
+    end
+
+    def on_error(&block)
+      # no-op
+    end
+  end
+end

data/lib/pubsub_client/null_subscriber_factory.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require_relative 'null_subscriber'
+
+module PubsubClient
+  # A null object to act as a subscriber factory when clients are in dev or test
+  class NullSubscriberFactory
+    def build(*)
+      NullSubscriber.new
+    end
+  end
+end

data/lib/pubsub_client/publisher.rb

@@ -4,12 +4,13 @@ require 'google/cloud/pubsub'
 
 module PubsubClient
   class Publisher
+    # @param topic [Google::Cloud::PubSub::Topic]
     def initialize(topic)
       @topic = topic
     end
 
-    def publish(message, &block)
-      topic.publish_async(message, &block)
+    def publish(message, attributes = {}, &block)
+      topic.publish_async(message, attributes, &block)
     end
 
     def flush

data/lib/pubsub_client/publisher_factory.rb

@@ -5,12 +5,14 @@ require_relative 'publisher'
 module PubsubClient
   # Build and memoize the Publisher, accounting for GRPC's requirements around forking.
   class PublisherFactory
-    def initialize(topic_name)
-      @topic_name = topic_name
+    def initialize
       @mutex = Mutex.new
+      @publishers = {}
     end
 
-    def build
+    # @param topic_name [String]
+    # @return [Publisher]
+    def build(topic_name)
       # GRPC fails when attempting to use a connection created in a process that gets
       # forked with the message
       #
@@ -20,27 +22,28 @@ module PubsubClient
       # PubSub.
       #
       # To prevent incurring overhead, memoize the publisher per process.
-      return @publisher if @publisher_pid == current_pid
+      return publishers[topic_name].publisher if publishers[topic_name]&.pid == current_pid
 
       # We are in a multi-threaded world and need to be careful not to build the publisher
       # in multiple threads. Lock the mutex so that only one thread can enter this block
       # at a time.
-      mutex.synchronize do
+      @mutex.synchronize do
         # It's possible two threads made it to this point, but since we have a lock we
         # know that one will have built the publisher before the second is able to enter.
         # If we detect that case, then bail out so as to not rebuild the publisher.
-        unless @publisher_pid == current_pid
-          @publisher = build_publisher
-          @publisher_pid = Process.pid
+        unless publishers[topic_name]&.pid == current_pid
+          publishers[topic_name] = Memo.new(build_publisher(topic_name), Process.pid)
         end
       end
 
-      @publisher
+      publishers[topic_name].publisher
     end
 
     private
 
-    attr_reader :mutex, :topic_name
+    attr_accessor :publishers
+
+    Memo = Struct.new(:publisher, :pid)
 
     # Used for testing to simulate when a process is forked. In those cases,
     # this helps us test that the `.build` method creates different publishers.
@@ -48,9 +51,10 @@ module PubsubClient
       Process.pid
     end
 
-    def build_publisher
+    def build_publisher(topic_name)
       pubsub = Google::Cloud::PubSub.new
       topic = pubsub.topic(topic_name)
+      raise InvalidTopicError, "The topic #{topic_name} does not exist" unless topic
       publisher = Publisher.new(topic)
 
       at_exit { publisher.flush }

data/lib/pubsub_client/subscriber.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require 'google/cloud/pubsub'
+
+module PubsubClient
+  class Subscriber
+    DEFAULT_CONCURRENCY = 8
+
+    # @param subscription [Google::Cloud::PubSub::Subscription]
+    def initialize(subscription)
+      @subscription = subscription
+    end
+
+    # @param concurrency [Integer] - The number of threads to run the subscriber with. Default is 8.
+    # @param auto_ack [Boolean] - Flag to acknowledge the Pub/Sub message. A message must be acked
+    # to remove it from the topic. Default is `true`.
+    #
+    # @return [Google::Cloud::PubSub::Subscriber]
+    def listener(concurrency: DEFAULT_CONCURRENCY, auto_ack: true, &block)
+      @listener ||= begin
+        @subscription.listen(threads: { callback: concurrency }) do |received_message|
+          yield received_message.data, received_message
+          received_message.acknowledge! if auto_ack
+        end
+      end
+    end
+
+    def subscribe
+      raise ConfigurationError, 'A listener must be configured' unless @listener
+
+      begin
+        @listener.start
+
+        sleep
+      rescue SignalException
+        @listener.stop.wait!
+      end
+    end
+
+    def on_error(&block)
+      raise ConfigurationError, 'A listener must be configured' unless @listener
+
+      @listener.on_error do |exception|
+        yield exception
+      end
+    end
+  end
+end

data/lib/pubsub_client/subscriber_factory.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require_relative 'subscriber'
+
+module PubsubClient
+  class SubscriberFactory
+    def initialize
+      @subscribers = {}
+    end
+
+    # @param subscription_name [String]
+    # @retrun [Subscriber]
+    def build(subscription_name)
+      if @subscribers.key?(subscription_name)
+        raise ConfigurationError, "PubsubClient already subscribed to #{subscription_name}"
+      end
+
+      @subscribers[subscription_name] = build_subscriber(subscription_name)
+    end
+
+    private
+
+    def build_subscriber(subscription_name)
+      pubsub = Google::Cloud::PubSub.new
+      subscription = pubsub.subscription(subscription_name)
+      raise InvalidSubscriptionError, "The subscription #{subscription_name} does not exist" unless subscription
+      Subscriber.new(subscription)
+    end
+  end
+end

data/lib/pubsub_client/version.rb

@@ -1,3 +1,3 @@
 module PubsubClient
-  VERSION = '0.1.0'
+  VERSION = '1.4.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pubsub_client
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 1.4.0
 platform: ruby
 authors:
 - Apartment List Platforms
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-08-25 00:00:00.000000000 Z
+date: 2020-10-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -76,6 +76,7 @@ files:
 - ".ci"
 - ".gitignore"
 - ".rspec"
+- CHANGELOG.txt
 - CODEOWNERS
 - CODE_OF_CONDUCT.md
 - Gemfile
@@ -87,8 +88,12 @@ files:
 - lib/pubsub_client.rb
 - lib/pubsub_client/null_publisher.rb
 - lib/pubsub_client/null_publisher_factory.rb
+- lib/pubsub_client/null_subscriber.rb
+- lib/pubsub_client/null_subscriber_factory.rb
 - lib/pubsub_client/publisher.rb
 - lib/pubsub_client/publisher_factory.rb
+- lib/pubsub_client/subscriber.rb
+- lib/pubsub_client/subscriber_factory.rb
 - lib/pubsub_client/version.rb
 - pubsub_client.gemspec
 homepage: https://github.com/apartmentlist/pubsub_client
@@ -113,7 +118,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
+rubygems_version: 3.1.4
 signing_key:
 specification_version: 4
 summary: Google Pub/Sub Wrapper