logstash-logger 0.17.0 → 0.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1de31545fcd13dd8e1d6a5c3bcd4d69b29e4579a
-  data.tar.gz: c45861aa001389601e4ef8f4352879d58d0758fb
+  metadata.gz: 416ec549b79d37b497160c4adda407080ec37350
+  data.tar.gz: 394cdb7be907401581ec546ae9579256253b4fe2
 SHA512:
-  metadata.gz: 308d6259229cc7aa957cdf5390ac3f7f244b61f6eb0c181498dce44eec7dad74326efb213561824e98349d120c5f39dbe4ef0a3a32919bea292b622a44643833
-  data.tar.gz: 8a746761ac4c20717bb73f21117569c038438433c6fab5200175e6d0e9b71a752e7f7cecf3e44148f7e1962e344948b97485ff6be53548bbc8bf6d3376bb1a8a
+  metadata.gz: 8a1e557351c9d65ef6a471024dfd5e86cabe1d2efac52ae55b02810cfea26dbacac6105e462304edc43b486bba62892c90acd8c08c5edaa6bde797b86a575524
+  data.tar.gz: 0ab2f410b37fb0737afe6f20d70ad1fad7ce3cff71c4a95bda2a1f8b5e4d2555f3eed66a9a68b52471192d406d1d637001dc3b57e236543c6492858f56e460b1

data/CHANGELOG.md

@@ -1,3 +1,13 @@
+## 0.18.0
+
+This release removes the dependency on `stud` and vendors in a forked version
+of `Stud::Buffer`. This improves the buffering behavior of LogStashLogger by
+flushing all log messages in a background thread by default. This eliminates
+blocking behavior and exceptions bubbling up to the main process.
+
+- Fixes `Attempt to unlock a mutex which is not locked (ThreadError)`.
+  [#88](https://github.com/dwbutler/logstash-logger/issues/88)
+
 ## 0.17.0
 - Support for logger silencing. [#87](https://github.com/dwbutler/logstash-logger/pull/87)
 - Fixes Rails 5 support. [#86](https://github.com/dwbutler/logstash-logger/issues/86)

data/README.md

@@ -241,19 +241,20 @@ This configuration would result in the following output.
 ## Buffering / Automatic Retries
 
 For devices that establish a connection to a remote service, log messages are buffered internally
-and automatically re-sent if there is a connection problem.
+and flushed in a background thread. If there is a connection problem, the
+messages are held in the buffer and automatically resent until it is successful.
 Outputs that support batch writing (Redis and Kafka) will write log messages in bulk from the
-buffer. This functionality is implemented using
+buffer. This functionality is implemented using a fork of
 [Stud::Buffer](https://github.com/jordansissel/ruby-stud/blob/master/lib/stud/buffer.rb).
 You can configure its behavior by passing the following options to LogStashLogger:
 
 * :buffer_max_items - Max number of items to buffer before flushing. Defaults to 50.
 * :buffer_max_interval - Max number of seconds to wait between flushes. Defaults to 5.
 * :drop_messages_on_flush_error - Drop messages when there is a flush error. Defaults to false.
-* :drop_messages_on_full_buffer - Drop messages when the buffer is full.
-  Defaults to true.
+* :drop_messages_on_full_buffer - Drop messages when the buffer is full. Defaults to true.
+* :sync - Flush buffer every time a message is received (blocking). Defaults to false.
 
-You can turn buffering off by setting `buffer_max_items` to `1` or `sync` to `true`.
+You can turn buffering off by setting `sync = true`.
 
 Please be aware of the following caveats to this behavior:
 
@@ -266,19 +267,39 @@ Please be aware of the following caveats to this behavior:
    immediately. In my testing, it took Ruby about 4 seconds to notice the receiving end was down
    and start raising exceptions. Since logstash listeners over TCP/UDP do not acknowledge received
    messages, it's not possible to know which log messages to re-send.
- * If your output source is unavailable long enough, the buffer will fill up
-   and messages will be dropped.
- * If your application suddenly terminates (for example, by SIGKILL or a power outage), the whole
-   buffer will be lost.
+
+## Full Buffer
 
 By default, messages are discarded when the buffer gets full. This can happen
-if the output source is down for too long.
+if the output source is down for too long or log messages are being received
+too quickly. If your application suddenly terminates (for example, by SIGKILL or a power outage),
+the whole buffer will be lost.
+
 You can make message loss less likely by increasing `buffer_max_items`
-(so that more events can be held in the buffer), and increasing `buffer_max_interval` (to wait
-longer between flushes). This will increase memory pressure on your application as log messages
+(so that more events can be held in the buffer), and decreasing `buffer_max_interval` (to wait
+less time between flushes). This will increase memory pressure on your application as log messages
 accumulate in the buffer, so make sure you have allocated enough memory to your process.
 
-## Error handling
+If you don't want to lose messages when the buffer gets full, you can set
+`drop_messages_on_full_buffer = false`. Note that if the buffer gets full, any
+incoming log message will block, which could be undesirable.
+
+## Sync Mode
+
+All logger outputs support a `sync` setting. This is analogous to the "sync mode" setting on Ruby IO
+objects. When set to `true`, output is immediately flushed and is not buffered internally. Normally,
+for devices that connect to a remote service, buffering is a good thing because
+it improves performance and reduces the likelihood of errors affecting the program. For these devices,
+`sync` defaults to `false`, and it is recommended to leave the default value.
+You may want to turn sync mode on for testing, for example if you want to see
+log messages immediately after they are written.
+
+It is recommended to turn sync mode on for file and Unix socket outputs. This
+ensures that log messages from different threads or proceses are written correctly on separate lines.
+
+See [#44](https://github.com/dwbutler/logstash-logger/issues/44) for more details.
+
+## Error Handling
 
 If an exception occurs while writing a message to the device, the exception is
 logged using an internal logger. By default, this logs to $stderr. You can

data/lib/logstash-logger/buffer.rb

@@ -0,0 +1,302 @@
+# Forked from https://github.com/jordansissel/ruby-stud/blob/master/lib/stud/buffer.rb
+
+module LogStashLogger
+
+  # @author {Alex Dean}[http://github.com/alexdean]
+  #
+  # Implements a generic framework for accepting events which are later flushed
+  # in batches. Flushing occurs whenever +:max_items+ or +:max_interval+ (seconds)
+  # has been reached.
+  #
+  # Including class must implement +flush+, which will be called with all
+  # accumulated items either when the output buffer fills (+:max_items+) or
+  # when a fixed amount of time (+:max_interval+) passes.
+  #
+  # == batch_receive and flush
+  # General receive/flush can be implemented in one of two ways.
+  #
+  # === batch_receive(event) / flush(events)
+  # +flush+ will receive an array of events which were passed to +buffer_receive+.
+  #
+  #   batch_receive('one')
+  #   batch_receive('two')
+  #
+  # will cause a flush invocation like
+  #
+  #   flush(['one', 'two'])
+  #
+  # === batch_receive(event, group) / flush(events, group)
+  # flush() will receive an array of events, plus a grouping key.
+  #
+  #   batch_receive('one',   :server => 'a')
+  #   batch_receive('two',   :server => 'b')
+  #   batch_receive('three', :server => 'a')
+  #   batch_receive('four',  :server => 'b')
+  #
+  # will result in the following flush calls
+  #
+  #   flush(['one', 'three'], {:server => 'a'})
+  #   flush(['two', 'four'],  {:server => 'b'})
+  #
+  # Grouping keys can be anything which are valid Hash keys. (They don't have to
+  # be hashes themselves.) Strings or Fixnums work fine. Use anything which you'd
+  # like to receive in your +flush+ method to help enable different handling for
+  # various groups of events.
+  #
+  # == on_flush_error
+  # Including class may implement +on_flush_error+, which will be called with an
+  # Exception instance whenever buffer_flush encounters an error.
+  #
+  # * +buffer_flush+ will automatically re-try failed flushes, so +on_flush_error+
+  #   should not try to implement retry behavior.
+  # * Exceptions occurring within +on_flush_error+ are not handled by
+  #   +buffer_flush+.
+  #
+  # == on_full_buffer_receive
+  # Including class may implement +on_full_buffer_receive+, which will be called
+  # whenever +buffer_receive+ is called while the buffer is full.
+  #
+  # +on_full_buffer_receive+ will receive a Hash like <code>{:pending => 30,
+  # :outgoing => 20}</code> which describes the internal state of the module at
+  # the moment.
+  #
+  # == final flush
+  # Including class should call <code>buffer_flush(:final => true)</code>
+  # during a teardown/shutdown routine (after the last call to buffer_receive)
+  # to ensure that all accumulated messages are flushed.
+  module Buffer
+
+    public
+    # Initialize the buffer.
+    #
+    # Call directly from your constructor if you wish to set some non-default
+    # options. Otherwise buffer_initialize will be called automatically during the
+    # first buffer_receive call.
+    #
+    # Options:
+    # * :max_items, Max number of items to buffer before flushing. Default 50.
+    # * :max_interval, Max number of seconds to wait between flushes. Default 5.
+    # * :logger, A logger to write log messages to. No default. Optional.
+    # * :autoflush, Whether to immediately flush all inbound messages. Default true.
+    # * :drop_messages_on_flush_error, Whether to drop messages when there is a flush error. Default false.
+    # * :drop_messages_on_full_buffer, Whether to drop messages when the buffer is full. Default false.
+    #
+    # @param [Hash] options
+    def buffer_initialize(options={})
+      if ! self.class.method_defined?(:flush)
+        raise ArgumentError, "Any class including Stud::Buffer must define a flush() method."
+      end
+
+      @buffer_config = {
+        :max_items => options[:max_items] || 50,
+        :max_interval => options[:max_interval] || 5,
+        :logger => options[:logger] || nil,
+        :autoflush => options.fetch(:autoflush, true),
+        :has_on_flush_error => self.class.method_defined?(:on_flush_error),
+        :has_on_full_buffer_receive => self.class.method_defined?(:on_full_buffer_receive),
+        :drop_messages_on_flush_error => options.fetch(:drop_messages_on_flush_error, false),
+        :drop_messages_on_full_buffer => options.fetch(:drop_messages_on_full_buffer, false)
+      }
+
+      reset_buffer
+    end
+
+    def reset_buffer
+      @buffer_state = {
+        # items accepted from including class
+        :pending_items => {},
+        :pending_count => 0,
+
+        # guard access to pending_items & pending_count
+        :pending_mutex => Mutex.new,
+
+        # items which are currently being flushed
+        :outgoing_items => {},
+        :outgoing_count => 0,
+
+        # ensure only 1 flush is operating at once
+        :flush_mutex => Mutex.new,
+
+        # data for timed flushes
+        :last_flush => Time.now,
+        :timer => Thread.new do
+          loop do
+            sleep(@buffer_config[:max_interval])
+            begin
+              buffer_flush(:force => true)
+            rescue
+            end
+          end
+        end
+      }
+
+      # events we've accumulated
+      buffer_clear_pending
+    end
+
+    # Determine if +:max_items+ has been reached.
+    #
+    # buffer_receive calls will block while <code>buffer_full? == true</code>.
+    #
+    # @return [bool] Is the buffer full?
+    def buffer_full?
+      @buffer_state[:pending_count] + @buffer_state[:outgoing_count] >= @buffer_config[:max_items]
+    end
+
+    # Save an event for later delivery
+    #
+    # Events are grouped by the (optional) group parameter you provide.
+    # Groups of events, plus the group name, are later passed to +flush+.
+    #
+    # This call will block if +:max_items+ has been reached.
+    #
+    # @see Stud::Buffer The overview has more information on grouping and flushing.
+    #
+    # @param event An item to buffer for flushing later.
+    # @param group Optional grouping key. All events with the same key will be
+    #              passed to +flush+ together, along with the grouping key itself.
+    def buffer_receive(event, group=nil)
+      buffer_initialize if ! @buffer_state
+
+      # block if we've accumulated too many events
+      while buffer_full? do
+        on_full_buffer_receive(
+          :pending => @buffer_state[:pending_count],
+          :outgoing => @buffer_state[:outgoing_count]
+        ) if @buffer_config[:has_on_full_buffer_receive]
+
+        if @buffer_config[:drop_messages_on_full_buffer]
+          reset_buffer
+        else
+          sleep 0.1
+        end
+      end
+
+      @buffer_state[:pending_mutex].synchronize do
+        @buffer_state[:pending_items][group] << event
+        @buffer_state[:pending_count] += 1
+      end
+
+      if @buffer_config[:autoflush]
+        buffer_flush(force: true)
+      end
+    end
+
+    # Try to flush events.
+    #
+    # Returns immediately if flushing is not necessary/possible at the moment:
+    # * :max_items have not been accumulated
+    # * :max_interval seconds have not elapased since the last flush
+    # * another flush is in progress
+    #
+    # <code>buffer_flush(:force => true)</code> will cause a flush to occur even
+    # if +:max_items+ or +:max_interval+ have not been reached. A forced flush
+    # will still return immediately (without flushing) if another flush is
+    # currently in progress.
+    #
+    # <code>buffer_flush(:final => true)</code> is identical to <code>buffer_flush(:force => true)</code>,
+    # except that if another flush is already in progress, <code>buffer_flush(:final => true)</code>
+    # will block/wait for the other flush to finish before proceeding.
+    #
+    # @param [Hash] options Optional. May be <code>{:force => true}</code> or <code>{:final => true}</code>.
+    # @return [Fixnum] The number of items successfully passed to +flush+.
+    def buffer_flush(options={})
+      force = options[:force] || options[:final]
+      final = options[:final]
+
+      # final flush will wait for lock, so we are sure to flush out all buffered events
+      if options[:final]
+        @buffer_state[:flush_mutex].lock
+      elsif ! @buffer_state[:flush_mutex].try_lock # failed to get lock, another flush already in progress
+        return 0
+      end
+
+      items_flushed = 0
+
+      begin
+        time_since_last_flush = (Time.now - @buffer_state[:last_flush])
+
+        return 0 if @buffer_state[:pending_count] == 0
+        return 0 if (!force) &&
+           (@buffer_state[:pending_count] < @buffer_config[:max_items]) &&
+           (time_since_last_flush < @buffer_config[:max_interval])
+
+        @buffer_state[:pending_mutex].synchronize do
+          @buffer_state[:outgoing_items] = @buffer_state[:pending_items]
+          @buffer_state[:outgoing_count] = @buffer_state[:pending_count]
+          buffer_clear_pending
+        end
+
+        @buffer_config[:logger].debug("Flushing output",
+          :outgoing_count => @buffer_state[:outgoing_count],
+          :time_since_last_flush => time_since_last_flush,
+          :outgoing_events => @buffer_state[:outgoing_items],
+          :batch_timeout => @buffer_config[:max_interval],
+          :force => force,
+          :final => final
+        ) if @buffer_config[:logger]
+
+        @buffer_state[:outgoing_items].each do |group, events|
+          begin
+            if group.nil?
+              flush(events,final)
+            else
+              flush(events, group, final)
+            end
+
+            @buffer_state[:outgoing_items].delete(group)
+            events_size = events.size
+            @buffer_state[:outgoing_count] -= events_size
+            items_flushed += events_size
+            @buffer_state[:last_flush] = Time.now
+
+          rescue => e
+
+            @buffer_config[:logger].warn("Failed to flush outgoing items",
+              :outgoing_count => @buffer_state[:outgoing_count],
+              :exception => e.class.name,
+              :backtrace => e.backtrace
+            ) if @buffer_config[:logger]
+
+            if @buffer_config[:has_on_flush_error]
+              on_flush_error e
+            end
+
+            if @buffer_config[:drop_messages_on_flush_error]
+              reset_buffer
+            else
+              cancel_flush
+            end
+
+          end
+        end
+
+      ensure
+        @buffer_state[:flush_mutex].unlock
+      end
+
+      return items_flushed
+    end
+
+    private
+    def buffer_clear_pending
+      @buffer_state[:pending_items] = Hash.new { |h, k| h[k] = [] }
+      @buffer_state[:pending_count] = 0
+    end
+
+    def buffer_clear_outgoing
+      @buffer_state[:outgoing_items] = Hash.new { |h, k| h[k] = [] }
+      @buffer_state[:outgoing_count] = 0
+    end
+
+    def cancel_flush
+      @buffer_state[:pending_mutex].synchronize do
+        @buffer_state[:outgoing_items].each do |group, items|
+          @buffer_state[:pending_items][group].concat items
+        end
+        @buffer_state[:pending_count] += @buffer_state[:outgoing_count]
+      end
+      buffer_clear_outgoing
+    end
+  end
+end

data/lib/logstash-logger/device/connectable.rb

@@ -1,9 +1,9 @@
-require 'stud/buffer'
+require 'logstash-logger/buffer'
 
 module LogStashLogger
   module Device
     class Connectable < Base
-      include Stud::Buffer
+      include LogStashLogger::Buffer
 
       def initialize(opts = {})
         super
@@ -19,7 +19,7 @@ module LogStashLogger
         @buffer_group = nil
         @buffer_max_items = opts[:batch_events] || opts[:buffer_max_items]
         @buffer_max_interval = opts[:batch_timeout] || opts[:buffer_max_interval]
-        @drop_messages_on_flush_error = 
+        @drop_messages_on_flush_error =
           if opts.key?(:drop_messages_on_flush_error)
             opts.delete(:drop_messages_on_flush_error)
           else
@@ -33,13 +33,17 @@ module LogStashLogger
             true
           end
 
-        reset_buffer
+        buffer_initialize(
+          max_items: @buffer_max_items,
+          max_interval: @buffer_max_interval,
+          autoflush: @sync,
+          drop_messages_on_flush_error: @drop_messages_on_flush_error,
+          drop_messages_on_full_buffer: @drop_messages_on_full_buffer
+        )
       end
 
       def write(message)
         buffer_receive message, @buffer_group
-        buffer_flush(force: true) if @sync
-      rescue
       end
 
       def flush(*args)
@@ -49,23 +53,10 @@ module LogStashLogger
           messages, group = *args
           write_batch(messages, group)
         end
-      rescue
-        if @drop_messages_on_flush_error
-          reset_buffer
-        else
-          cancel_flush
-        end
-        raise
-      end
-
-      def on_flush_error(e)
-        raise e
       end
 
-      def on_full_buffer_receive(args)
-        if @drop_messages_on_full_buffer
-          reset_buffer
-        end
+      def on_full_buffer_receive(data)
+        log_warning("Buffer Full - #{data}")
       end
 
       def close(opts = {})
@@ -117,37 +108,6 @@ module LogStashLogger
         close(flush: false)
         raise
       end
-
-      private
-
-      def reset_buffer
-        buffer_initialize max_items: @buffer_max_items, max_interval: @buffer_max_interval
-        @buffer_state[:timer] = Thread.new do
-          loop do
-            sleep(@buffer_config[:max_interval])
-            begin
-              buffer_flush(:force => true)
-            rescue
-            end
-          end
-        end
-      end
-
-      def buffer_clear_outgoing
-        @buffer_state[:outgoing_items] = Hash.new { |h, k| h[k] = [] }
-        @buffer_state[:outgoing_count] = 0
-      end
-
-      def cancel_flush
-        @buffer_state[:flush_mutex].lock rescue false
-        @buffer_state[:outgoing_items].each do |group, items|
-          @buffer_state[:pending_items][group].concat items
-        end
-        @buffer_state[:pending_count] += @buffer_state[:outgoing_count]
-        buffer_clear_outgoing
-      ensure
-        @buffer_state[:flush_mutex].unlock rescue false
-      end
     end
   end
 end

data/lib/logstash-logger/version.rb

@@ -1,3 +1,3 @@
 module LogStashLogger
-  VERSION = "0.17.0"
+  VERSION = "0.18.0"
 end

data/logstash-logger.gemspec

@@ -19,7 +19,6 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
 
   gem.add_runtime_dependency 'logstash-event', '~> 1.2'
-  gem.add_runtime_dependency 'stud'
 
   gem.add_development_dependency 'rails'
   if RUBY_VERSION < '2'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-logger
 version: !ruby/object:Gem::Version
-  version: 0.17.0
+  version: 0.18.0
 platform: ruby
 authors:
 - David Butler
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-07-24 00:00:00.000000000 Z
+date: 2016-07-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-event
@@ -24,20 +24,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.2'
-- !ruby/object:Gem::Dependency
-  name: stud
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: rails
   requirement: !ruby/object:Gem::Requirement
@@ -172,6 +158,7 @@ files:
 - gemfiles/rails_4.2.gemfile
 - gemfiles/rails_5.0.gemfile
 - lib/logstash-logger.rb
+- lib/logstash-logger/buffer.rb
 - lib/logstash-logger/configuration.rb
 - lib/logstash-logger/device.rb
 - lib/logstash-logger/device/balancer.rb