megaphone-client 1.0.1 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 3194e82bf4582a1956b969f23902751161badf46
-  data.tar.gz: 3e604bdf98d9440d551cead5ce2be555f9ced59c
+  metadata.gz: 15b75d53f469b5b1eb81174951801eb6b131957d
+  data.tar.gz: 72d170f64c4a6eb00f009411e21700ce238eee82
 SHA512:
-  metadata.gz: d68dd2886287ff4feeb390bbcffc0b0218769af1e5cbf385a412e72156e75cb3a8ccbe1dd8c287e21f90b9da933ec181a760f9d6c42fc9ca084ccf33eedd7364
-  data.tar.gz: 4b5261feb237c02f1c894bf8141effadf68d4a0ec1e2d5230ff287552ab19f0a43928c364cfe4e7081be84e0289a7290e0c91a4ed544a8315c2587719801654e
+  metadata.gz: 5510da93aa27aaa55aa0bfb6b4ea68cb11678f9c7424d89cd309cad42e87e220dc62b4a0bd2ba7d5925fe233a34f8fe07c8c2cfd427c8e97645b4f9508fdffcd
+  data.tar.gz: 2e6f71ac8caa1f13ab0bbf81351d6baf4056322d4e5cd4ffb85e8e3ca86a56d5ab8ba9a901eecc27a22a38d525d2b94bc3d7d219f9308285ae370d81059d7eba

data/CHANGELOG.md

@@ -5,11 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and
 this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] - 2018-01-30
+
+### Added
+
+* Add new `MegaphoneMessageDelayWarning` exception for transient errors
+* Add `close` method to allow connection to Fluentd to be closed
+* Support `buffer_overflow` callback to notify when messages have been lost ([documentation](../README.md#buffer-overflow-callback-handler))
+
+### Fixed
+
+* Update Fluentd Gem from 0.7.1 to 0.7.2
+* Do not dump entire payload into exceptions; just stream/topic IDs
+
 ## [1.0.1] - 2017-09-18
 
 ### Fixed
 
-- Output format to match the [stream-schema v2][stream-schema-v2].
+* Output format to match the [stream-schema v2][stream-schema-v2].
 
   [stream-schema-v2]: https://github.com/redbubble/megaphone-event-type-registry/blob/master/stream-schema-2.0.0.json
 
@@ -17,39 +30,40 @@ this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html
 
 ### Changed
 
-- The `Megaphone::Client` API is now stable!
+* The `Megaphone::Client` API is now stable!
 
 ## [0.3.0] - 2017-08-17
 
 ### Changed
 
-- Signature of the `Megaphone::Client` constructor : now takes a host and a port as optional arguments and doesn't take the logger as an argument anymore.
+* Signature of the `Megaphone::Client` constructor : now takes a host and a port as optional arguments and doesn't take the logger as an argument anymore.
 
 ## [0.2.0] - 2017-08-14
 
 ### Changed
 
-- Signature of the `publish!` method: now takes a schema and a partition key as arguments
+* Signature of the `publish!` method: now takes a schema and a partition key as arguments
 
 ## [0.1.2] - 2017-08-04
 
 ### Fixed
 
-- Removing unnecessary files from gemspec
+* Removing unnecessary files from gemspec
 
 ## [0.1.1] - 2017-08-03
 
 ### Fixed
 
-- Including files in gemspec
+* Including files in gemspec
 
 ## 0.1.0 - 2017-08-03 [YANKED]
 
 ### Added
 
-- Initial implementation of the `Megaphone::Client.publish!` method
+* Initial implementation of the `Megaphone::Client.publish!` method
 
-  [1.0.1]: https://github.com/redbubble/megaphone-client-ruby/compare/v1.0.1...v1.0.1
+  [1.1.0]: https://github.com/redbubble/megaphone-client-ruby/compare/v1.0.1...v1.1.0
+  [1.0.1]: https://github.com/redbubble/megaphone-client-ruby/compare/v1.0.0...v1.0.1
   [1.0.0]: https://github.com/redbubble/megaphone-client-ruby/compare/v0.3.0...v1.0.0
   [0.3.0]: https://github.com/redbubble/megaphone-client-ruby/compare/v0.2.0...v0.3.0
   [0.2.0]: https://github.com/redbubble/megaphone-client-ruby/compare/v0.1.2...v0.2.0

data/README.md

@@ -1,5 +1,4 @@
-Megaphone::Client
-=================
+# Megaphone::Client
 
 [![Build Status](https://badge.buildkite.com/9f4fdb370f5f295ee6bf3d68937b1be2d7cf9bf65b2c7b4213.svg?branch=master)](https://buildkite.com/redbubble/megaphone-client-ruby)
 [![Gem Version](https://badge.fury.io/rb/megaphone-client.svg)](https://badge.fury.io/rb/megaphone-client)
@@ -7,8 +6,7 @@ Megaphone::Client
 
 Send events to [Megaphone](https://github.com/redbubble/Megaphone).
 
-Getting Started
----------------
+## Getting Started
 
 Add the gem to your `Gemfile`:
 
@@ -18,18 +16,17 @@ Add the gem to your `Gemfile`:
 gem 'megaphone-client', '~> 1.0' # see semver.org
 ```
 
-Usage
------
+## Usage
 
 In order to be as unobstrusive as possible, this client will append events to local files (e.g. `./work-updates.stream`) unless:
 
-- the `MEGAPHONE_FLUENT_HOST` and `MEGAPHONE_FLUENT_PORT` environment variables are set.
-- **or** the Fluentd host and port values are passed as arguments to the client's constructor
+* the `MEGAPHONE_FLUENT_HOST` and `MEGAPHONE_FLUENT_PORT` environment variables are set.
+* **or** the Fluentd host and port values are passed as arguments to the client's constructor
 
 That behaviour ensures that unless you want to send events to the Megaphone [streams][stream], you do not need to [start Fluentd][megaphone-fluentd] at all.
 
-  [stream]: https://github.com/redbubble/megaphone#stream
-  [megaphone-fluentd]: https://github.com/redbubble/megaphone-fluentd-container
+[stream]: https://github.com/redbubble/megaphone#stream
+[megaphone-fluentd]: https://github.com/redbubble/megaphone-fluentd-container
 
 ### Publishing events
 
@@ -54,19 +51,113 @@ payload = { url: 'https://www.redbubble.com/people/wytrab8/works/26039653-toadal
 
 # Publish your event
 client.publish!(topic, subtopic, schema, partition_key, payload)
+
+# Note: the client will close the connection to Fluentd on exit, if you need to do it before that (unlikely), you can use Megaphone::Client#close method.
+
+# See below for error handling instructions and examples.
 ```
 
-Credits
--------
+## Error Handling
+
+### Exceptions the client will raise
+
+`publish!` can raise two exceptions if the underlying Fluentd
+client library throws an error.
+
+The most common is `MegaphoneMessageDelayWarning` which indicates
+a transient failure occurred, but the message will probably be resent
+later. See section on _[Internal buffering](#internal-buffering-upon-error)_ below for more details.
+
+The second exception is `MegaphoneUnavailableError`, which is thrown
+for all other errors. Note that these _may or may not_ also buffer
+the message for later transmission. Unfortunately the underlying
+client library does not make the distinction.
+
+### Internal buffering upon error
+
+Note that, in some cases, the client library will buffer failed messages
+and attempt to resend them later. It will resend buffered messages in
+the following cases:
+
+* another message is sent
+* the `close` method is called
+* at exit (via an `at_exit` handler)
+
+Applications that require fast handling of events should read the section
+below on handling time-sensitive events and errors.
+
+Because of this buffering, users of the library should not treat exceptions
+as a definite reason to resend a message, as this may result in multiple
+messages being eventually dispatched. However without in-depth knowledge
+it can be hard to tell which exceptions are recoverable, and which indicate
+some kind of catastrophic failure.
+
+This author has confirmed that both connection failure and `ECONNRESET`
+errors will result in buffering of messages.
+
+If the internal buffer fills up, the buffer overflow handler will be called.
+
+At exit, or when `client.close` is called, the buffer will be flushed.
+If it cannot be flushed to the daemon, then the buffer overflow
+handler will be called.
+
+### Buffer overflow callback handler
+
+Passing a lambda to the `:overflow_handler` will enable your application
+to receive notifications of messages being lost. They can be lost in at
+least two ways:
+
+* Fluentd daemon has gone away, and internal client-side buffer exceeded
+* process is shutting down, and fluent client library was unable to flush buffers
+  to the daemon.
+
+Production applications [MUST][rfc2119] handle this case, and raise an alert, if their
+messages are considered important.
+
+[rfc2119]: https://tools.ietf.org/html/rfc2119
+
+```ruby
+# Example usage:
+my_handler = -> (*) {
+  Rollbar.error("Megaphone/fluent messages lost due to buffer overflow")
+}
+
+logger = Megaphone::Client.new({
+  origin: "my-app",
+  overflow_handler: my_handler
+})
+
+begin
+  logger.publish!(... event ...)
+rescue Megaphone::Client::MegaphoneUnavailableError => e
+  Rollbar.warning("Megaphone client error", e)
+rescue Megaphone::Client::MegaphoneMessageDelayWarning => e
+  Rollbar.info("Megaphone transient message delay", e)
+end
+```
+
+### Handling time-sensitive events and errors
+
+As long as an application is either short lived, or frequently sending messages,
+then it will probably be fine with the usual behaviour, which is to flush
+buffers at next send, or flush at exit.
+
+Applications with time-sensitive, infrequent events, will need to find a
+different strategy if errors have been raised during previous message publishing.
+
+Unfortunately there is no means to do this with the underlying Ruby client
+for Fluentd. It would require patches to the upstream code to expose a flush
+method.
+
+## Credits
 
 [![](doc/redbubble.png)][redbubble]
 
 Megaphone::Client is maintained and funded by [Redbubble][redbubble].
 
-  [redbubble]: https://www.redbubble.com
+[redbubble]: https://www.redbubble.com
 
-License
--------
+## License
 
     Megaphone::Client
     Copyright (C) 2017 Redbubble

data/lib/megaphone/client.rb

@@ -8,18 +8,45 @@ module Megaphone
     attr_reader :logger, :origin
     private :logger, :origin
 
+    # Main entry point for apps using this library.
+    # Will default to environment for host and port settings, if not passed.
+    # Note that a missing callback_handler will result in a default handler
+    # being assigned if the FluentLogger is used.
     def initialize(config)
       @origin = config.fetch(:origin)
       host = config.fetch(:host, ENV['MEGAPHONE_FLUENT_HOST'])
       port = config.fetch(:port, ENV['MEGAPHONE_FLUENT_PORT'])
-      @logger = Megaphone::Client::Logger.create(host, port)
+      overflow_handler = config.fetch(:overflow_handler, nil)
+      @logger = Megaphone::Client::Logger.create(host, port, overflow_handler)
     end
 
     def publish!(topic, subtopic, schema, partition_key, payload)
       event = Event.new(topic, subtopic, origin, schema, partition_key, payload)
       unless logger.post(topic, event.to_hash)
-        raise MegaphoneUnavailableError.new(logger.last_error.message, event)
+        if transient_error?(logger.last_error)
+          raise MegaphoneMessageDelayWarning.new(logger.last_error.message, event)
+        else
+          raise MegaphoneUnavailableError.new(logger.last_error.message, event)
+        end
       end
     end
+
+    def close
+      logger.close
+    end
+
+    private
+
+    def transient_error?(err)
+      err_msg = err.message
+      if err_msg.include?("Connection reset by peer")
+        true
+      elsif err_msg.include?("Broken pipe")
+        true
+      else
+        false
+      end
+    end
+
   end
 end

data/lib/megaphone/client/errors.rb

@@ -9,7 +9,20 @@ module Megaphone
       end
 
       def to_s
-        "#{@msg} - The following event was not published: #{@event.to_s}"
+        "#{@msg} - An event could not be immediately published, but may be retried: #{@event.stream_id}"
+      end
+    end
+
+    class MegaphoneMessageDelayWarning < StandardError
+
+      def initialize(msg, event)
+        super(msg)
+        @msg = msg
+        @event = event
+      end
+
+      def to_s
+        "#{@msg} - Event delayed and will be reattempted: #{@event.stream_id}"
       end
     end
   end

data/lib/megaphone/client/event.rb

@@ -12,6 +12,10 @@ module Megaphone
         @payload = payload
       end
 
+      def stream_id
+        "#{@topic}.#{@subtopic}"
+      end
+
       def to_hash
         {
           schema: @schema,

data/lib/megaphone/client/fluent_logger.rb

@@ -6,12 +6,26 @@ module Megaphone
     class FluentLogger
       extend Forwardable
 
-      def_delegators :@logger, :post, :last_error
+      def_delegators :@logger, :post, :last_error, :close
 
-      def initialize(host, port)
+      def initialize(host, port, overflow_handler = nil)
+        overflow_handler ||= default_overflow_handler
         @logger = Fluent::Logger::FluentLogger.new('megaphone',
                                                    host: host,
-                                                   port: port)
+                                                   port: port,
+                                                   buffer_overflow_handler: overflow_handler
+                                                 )
+      end
+
+      private
+      # A default overflow handler that just prints a warning message.
+      # Production applications should be passing in their own handlers,
+      # which should be alerting monitoring systems!
+      def default_overflow_handler
+        $stderr.puts("Megaphone::Client::FluentLogger - Production apps MUST override buffer overflow handler!")
+        -> (*) {
+          $stderr.puts("Buffer overflow in Megaphone/fluent logger - messages lost")
+        }
       end
 
     end

data/lib/megaphone/client/logger.rb

@@ -4,10 +4,11 @@ require 'megaphone/client/fluent_logger'
 module Megaphone
   class Client
     class Logger
-      def self.create(host = nil, port = nil)
+
+      def self.create(host = nil, port = nil, overflow_handler = nil)
         if !port.nil? && !port.empty? &&
            !host.nil? && !host.empty?
-          return FluentLogger.new(host, port)
+          return FluentLogger.new(host, port, overflow_handler)
         end
         FileLogger.new
       end

data/lib/megaphone/client/version.rb

@@ -1,5 +1,5 @@
 module Megaphone
   class Client
-    VERSION = "1.0.1"
+    VERSION = "1.1.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: megaphone-client
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.1.0
 platform: ruby
 authors:
 - Redbubble
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-09-18 00:00:00.000000000 Z
+date: 2018-01-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluent-logger
@@ -94,7 +94,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 2.4.5.1
 signing_key: 
 specification_version: 4
 summary: Send events to Megaphone.