logstash-logger 0.26.1 → 1.1.0

data/docs/troubleshooting.md

@@ -0,0 +1,106 @@
+# Troubleshooting
+
+## Logstash never receives any logs
+
+If you are using a device backed by a Ruby IO object (such as a file, UDP socket, or TCP socket), please be aware that Ruby
+keeps its own internal buffer. Despite the fact that LogStashLogger buffers
+messages and flushes them periodically, the data written to the IO object can
+be buffered by Ruby internally indefinitely, and may not even write until the
+program terminates. If this bothers you or you need to see log messages
+immediately, your only recourse is to set the `sync: true` option.
+
+## JSON::GeneratorError
+
+Your application is probably attempting to log data that is not encoded in a valid way. When this happens, Ruby's
+standard JSON library will raise an exception. You may be able to overcome this by swapping out a different JSON encoder
+such as Oj. Use the [oj_mimic_json](https://github.com/ohler55/oj_mimic_json) gem to use Oj for JSON generation.
+
+## Records tagged `_logstashlogger_format_failure`
+
+When a log call cannot be turned into a record, for example because a logged
+value cannot be serialized to JSON, LogStashLogger writes a placeholder record
+instead. It is framed like any other record, so the records around it are not
+affected. The placeholder has the message `Failed to format log event`, the
+tag `_logstashlogger_format_failure`, the logger's time and severity, and the
+error class in an `error` field. The error itself is reported to the error
+logger. Before 1.1.0 a bare `Failed to format log event` string was written
+without a line ending, which could merge with the next record.
+
+## The `cee_syslog` header shows the local hostname
+
+The `cee_syslog` formatter writes a `hostname progname:@cee:` header before
+each record. The hostname is always this machine's hostname, and control
+characters in the header are replaced with spaces. A `host` field in the logged
+content, or one set in `customize_event`, appears only in the JSON body.
+
+## No logs getting sent on Heroku
+
+Heroku recommends installing the [rails_12factor](https://github.com/heroku/rails_12factor) so that logs get sent to STDOUT.
+Unfortunately, this overrides LogStashLogger, preventing logs from being sent to their configured destination. The solution
+is to remove `rails_12factor` from your Gemfile.
+
+## Logging eventually stops in production
+
+This is most likely not a problem with LogStashLogger, but rather a different gem changing the log level of `Rails.logger`.
+This is especially likely if you're using a threaded server such as Puma, since gems often change the log level of
+`Rails.logger` in a non thread-safe way. See [#17](https://github.com/dwbutler/logstash-logger/issues/17) for more information.
+
+## Sometimes two lines of JSON log messages get sent as one message
+
+If you're using UDP output and writing to a logstash listener, you are most likely encountering a bug in the UDP implementation
+of the logstash listener. There is no known fix at this time. See [#43](https://github.com/dwbutler/logstash-logger/issues/43)
+for more information.
+
+## Errno::EMSGSIZE - Message too long
+
+A known drawback of using TCP or UDP is the 65535 byte limit on total message size. To workaround
+this issue, you will have to truncate the message by setting the max message size:
+
+```ruby
+LogStashLogger.configure do |config|
+  config.max_message_size = 2000
+end
+```
+
+This will truncate only the `message` field of the LogStash Event. So make sure
+you set the max message size significantly less than 65535 bytes to make room
+for other fields.
+
+# Breaking changes
+
+## Version 1.1+
+
+MRI Ruby < 3.3 and Rails < 8.0 are no longer supported, since they have been EOL'ed. If you are on Ruby 3.2 or Rails 7.2, you will need to use 1.0.x.
+
+## Version 0.27+
+
+MRI Ruby < 3.2 is no longer supported, since it has been EOL'ed. If you are on an older version of Ruby, you will need to use 0.26 or below.
+
+## Version 0.25+
+
+Rails 3.2, MRI Ruby < 2.2, and JRuby 1.7 are no longer supported, since they have been
+EOL'ed. If you are on an older version of Ruby, you will need to use 0.24 or below.
+
+## Version 0.5+
+
+* The `source` event key has been replaced with `host` to better match the latest logstash.
+* The `(host, port, type)` constructor has been deprecated in favor of an options hash constructor.
+
+## Version 0.4+
+
+`LogStash::Event` uses the v1 format starting version 1.2+. If you're using the v1, you'll need to install
+LogStashLogger version 0.4+. This is not backwards compatible with the old `LogStash::Event` v1.1.5, which uses
+the v0 format.
+
+## Version 0.3+
+
+Earlier versions of this gem (<= 0.2.1) only implemented a TCP connection.
+Newer versions (>= 0.3) also implement UDP, and use that as the new default.
+Please be aware if you are using the default constructor and still require TCP, you should add an additional argument:
+
+```ruby
+# Now defaults to UDP instead of TCP
+logger = LogStashLogger.new('localhost', 5228)
+# Explicitly specify TCP instead of UDP
+logger = LogStashLogger.new('localhost', 5228, :tcp)
+```

data/docs/usage.md

@@ -0,0 +1,148 @@
+# Usage Examples
+
+## Basic Usage
+
+```ruby
+require 'logstash-logger'
+
+# Defaults to UDP on 0.0.0.0
+logger = LogStashLogger.new(port: 5228)
+
+# Specify host and type (UDP or TCP) explicitly
+udp_logger = LogStashLogger.new(type: :udp, host: 'localhost', port: 5228)
+tcp_logger = LogStashLogger.new(type: :tcp, host: 'localhost', port: 5229)
+
+# Other types of loggers
+file_logger = LogStashLogger.new(type: :file, path: 'log/development.log', sync: true)
+unix_logger = LogStashLogger.new(type: :unix, path: '/tmp/sock')
+syslog_logger = LogStashLogger.new(type: :syslog)
+redis_logger = LogStashLogger.new(type: :redis)
+kafka_logger = LogStashLogger.new(type: :kafka)
+stdout_logger = LogStashLogger.new(type: :stdout)
+stderr_logger = LogStashLogger.new(type: :stderr)
+io_logger = LogStashLogger.new(type: :io, io: io)
+```
+
+## Using Formatters
+
+```ruby
+# Use a different formatter
+cee_logger = LogStashLogger.new(
+  type: :tcp,
+  host: 'logsene-receiver-syslog.sematext.com',
+  port: 514,
+  formatter: :cee_syslog
+)
+
+custom_formatted_logger = LogStashLogger.new(
+  type: :redis,
+  formatter: MyCustomFormatter
+)
+
+lambda_formatted_logger = LogStashLogger.new(
+  type: :stdout,
+  formatter: ->(severity, time, progname, msg) { "[#{progname}] #{msg}" }
+)
+
+ruby_default_formatter_logger = LogStashLogger.new(
+  type: :file,
+  path: 'log/development.log',
+  formatter: ::Logger::Formatter
+)
+```
+
+## Multiple Outputs
+
+```ruby
+# Send messages to multiple outputs. Each output will have the same format.
+# Syslog cannot be an output because it requires a separate logger.
+multi_delegating_logger = LogStashLogger.new(
+  type: :multi_delegator,
+  outputs: [
+    { type: :file, path: 'log/development.log' },
+    { type: :udp, host: 'localhost', port: 5228 }
+  ])
+
+# Balance messages between several outputs.
+# Works the same as multi delegator, but randomly chooses an output to send each message.
+balancer_logger = LogStashLogger.new(
+  type: :balancer,
+  outputs: [
+    { type: :udp, host: 'host1', port: 5228 },
+    { type: :udp, host: 'host2', port: 5228 }
+  ])
+
+# Send messages to multiple loggers.
+# Use this if you need to send different formats to different outputs.
+# If you need to log to syslog, you must use this.
+multi_logger = LogStashLogger.new(
+  type: :multi_logger,
+  outputs: [
+    { type: :file, path: 'log/development.log', formatter: ::Logger::Formatter },
+    { type: :tcp, host: 'localhost', port: 5228, formatter: :json }
+  ])
+```
+
+## Logging Messages
+
+```ruby
+# The following messages are written to UDP port 5228:
+
+logger.info 'test'
+# {"message":"test","@timestamp":"2014-05-22T09:37:19.204-07:00","@version":"1","severity":"INFO","host":"[hostname]"}
+
+logger.error '{"message": "error"}'
+# {"message":"error","@timestamp":"2014-05-22T10:10:55.877-07:00","@version":"1","severity":"ERROR","host":"[hostname]"}
+
+logger.debug message: 'test', foo: 'bar'
+# {"message":"test","foo":"bar","@timestamp":"2014-05-22T09:43:24.004-07:00","@version":"1","severity":"DEBUG","host":"[hostname]"}
+
+logger.warn LogStash::Event.new(message: 'test', foo: 'bar')
+# {"message":"test","foo":"bar","@timestamp":"2014-05-22T16:44:37.364Z","@version":"1","severity":"WARN","host":"[hostname]"}
+
+# Tagged logging
+logger.tagged('foo') { logger.fatal('bar') }
+# {"message":"bar","@timestamp":"2014-05-26T20:35:14.685-07:00","@version":"1","severity":"FATAL","host":"[hostname]","tags":["foo"]}
+```
+
+## URI Configuration
+
+You can use a URI to configure your logstash logger instead of a hash. This is useful in environments
+such as Heroku where you may want to read configuration values from the environment. The URI scheme
+is `type://host:port/path?key=value`. Some sample URI configurations are given below.
+
+```
+udp://localhost:5228
+tcp://localhost:5229
+unix:///tmp/socket
+file:///path/to/file
+redis://localhost:6379
+kafka://localhost:9092
+stdout:/
+stderr:/
+```
+
+Pass the URI into your logstash logger like so:
+
+```ruby
+# Read the URI from an environment variable
+logger = LogStashLogger.new(uri: ENV['LOGSTASH_URI'])
+```
+
+## What type of logger should I use?
+
+It depends on your specific needs, but most applications should use the default (UDP). Here are the advantages and
+disadvantages of each type:
+
+* UDP is faster than TCP because it's asynchronous (fire-and-forget). However, this means that log messages could get dropped.
+  This is okay for many applications.
+* TCP verifies that every message has been received via two-way communication. It also supports SSL for secure transmission
+  of log messages over a network. This could slow your app down to a crawl if the TCP listener is under heavy load.
+* A file is simple to use, but you will have to worry about log rotation and running out of disk space.
+* Writing to a Unix socket is faster than writing to a TCP or UDP port, but only works locally.
+* Writing to Redis is good for distributed setups that generate tons of logs. However, you will have another moving part and
+  have to worry about Redis running out of memory.
+* Writing to stdout is only recommended for debugging purposes.
+
+For a more detailed discussion of UDP vs TCP, I recommend reading this article:
+[UDP vs. TCP](http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/)

data/gemfiles/rails_8.0.gemfile

@@ -0,0 +1,10 @@
+# This file was generated by Appraisal
+
+source "https://rubygems.org"
+
+gem "simplecov", ">= 0.22", group: :test, require: false
+gem "simplecov-cobertura", group: :test, require: false
+gem "rails", "~> 8.0.0"
+gem "json", "< 3"
+
+gemspec path: "../"

data/gemfiles/rails_8.1.gemfile

@@ -0,0 +1,9 @@
+# This file was generated by Appraisal
+
+source "https://rubygems.org"
+
+gem "simplecov", ">= 0.22", group: :test, require: false
+gem "simplecov-cobertura", group: :test, require: false
+gem "rails", "~> 8.1.0"
+
+gemspec path: "../"

data/lib/logstash-logger/buffer.rb

@@ -66,7 +66,6 @@ module LogStashLogger
   # to ensure that all accumulated messages are flushed.
   module Buffer
 
-    public
     # Initialize the buffer.
     #
     # Call directly from your constructor if you wish to set some non-default
@@ -181,6 +180,16 @@ module LogStashLogger
       end
     end
 
+    # Put items back into the pending queue for the next flush, without
+    # applying the full-buffer policy. Safe to call from inside a flush.
+    def buffer_requeue(items, group=nil)
+      return if items.empty?
+      @buffer_state[:pending_mutex].synchronize do
+        @buffer_state[:pending_items][group].concat(items)
+        @buffer_state[:pending_count] += items.size
+      end
+    end
+
     # Try to flush events.
     #
     # Returns immediately if flushing is not necessary/possible at the moment:
@@ -306,11 +315,12 @@ module LogStashLogger
         end
     end
 
+    # Never kills the timer from inside itself: called on the timer thread,
+    # this keeps the running timer, and reset_buffer reuses it.
     def reset_flush_timer_thread
-      unless @flush_timer_thread.nil?
-        @flush_timer_thread.kill
-        @flush_timer_thread = nil
-      end
+      return if @flush_timer_thread.nil? || @flush_timer_thread == Thread.current
+      @flush_timer_thread.kill
+      @flush_timer_thread = nil
     end
 
     def buffer_clear_pending

data/lib/logstash-logger/configuration.rb

@@ -13,12 +13,11 @@ module LogStashLogger
     attr_accessor :max_message_size
     attr_accessor :default_error_logger
 
-    def initialize(*args)
+    def initialize(*_args)
       @customize_event_block = nil
       @default_error_logger = Logger.new($stderr)
 
       yield self if block_given?
-      self
     end
 
     def customize_event(&block)

data/lib/logstash-logger/device/aws_stream.rb

@@ -18,6 +18,7 @@ module LogStashLogger
       end
 
       attr_accessor :aws_region, :stream
+      attr_reader :max_record_bytes
 
       def initialize(opts)
         super
@@ -25,6 +26,22 @@ module LogStashLogger
         @secret_access_key = opts[:aws_secret_access_key]
         @aws_region = opts[:aws_region]
         @stream = opts[:stream] || DEFAULT_STREAM
+        # The largest record the stream accepts. Kinesis streams can be
+        # configured above the 1 MiB default, up to 10 MiB.
+        @max_record_bytes = Integer(opts[:aws_max_record_bytes] || self.class.max_record_bytes)
+      end
+
+      # Service limits, defined by each subclass
+      def self.max_records
+        fail NotImplementedError
+      end
+
+      def self.max_record_bytes
+        fail NotImplementedError
+      end
+
+      def self.max_request_bytes
+        fail NotImplementedError
       end
 
       def transform_message(message)
@@ -55,27 +72,32 @@ module LogStashLogger
         yield
       rescue => e
         log_error(e)
-        log_warning("giving up")
         close(flush: false)
+        raise
       end
 
-      def write_batch(messages, group = nil)
-        records = messages.map{ |m| transform_message(m) }
-
-        with_connection do
-          resp = put_records(records)
+      def write_batch(messages, _group = nil)
+        records = []
+        messages.each do |message|
+          record = transform_message(message.to_s)
+          size = record_size(record)
+          if size > max_record_bytes
+            log_warning("dropping a record of #{size} bytes: over the #{max_record_bytes}-byte limit of #{self.class.stream_class.name}")
+          else
+            records << record
+          end
+        end
 
-          # Put any failed records back into the buffer
-          if !is_successful_response(resp)
-            get_response_records(resp).each_with_index do |record, index|
-              if self.class.recoverable_error_codes.include?(record.error_code)
-                log_warning("Failed to post record using #{self.class.stream_class.name} with error: #{record.error_code} #{record.error_message}")
-                log_warning("Retrying")
-                write(records[index][:data])
-              elsif !record.error_code.nil? && record.error_code != ''
-                log_error("Failed to post record using #{self.class.stream_class.name} with error: #{record.error_code} #{record.error_message}")
-              end
-            end
+        slices = request_slices(records)
+        slices.each_with_index do |slice, index|
+          begin
+            with_connection { send_slice(slice) }
+          rescue StandardError
+            # Already logged by with_connection. Only records that were not
+            # sent go back; dropped oversized records never do.
+            unsent = slices[index..-1].flatten(1).map { |r| r[:data] }
+            give_back(unsent)
+            return
           end
         end
       end
@@ -88,6 +110,63 @@ module LogStashLogger
         @io = nil
       end
 
+      private
+
+      def send_slice(records)
+        resp = put_records(records)
+        return if is_successful_response(resp)
+
+        retries = []
+        retry_codes = []
+        get_response_records(resp).each_with_index do |record, index|
+          if self.class.recoverable_error_codes.include?(record.error_code)
+            retries << records[index][:data]
+            retry_codes << record.error_code
+          elsif !record.error_code.nil? && record.error_code != ''
+            log_warning("dropping a record that #{self.class.stream_class.name} rejected: #{record.error_code} #{record.error_message}")
+          end
+        end
+        return if retries.empty?
+
+        log_warning("#{retries.size} records will be retried on the next flush (#{retry_codes.tally.map { |c, n| "#{c}: #{n}" }.join(', ')})")
+        # This runs inside a flush, so requeue without the full-buffer check,
+        # which would count the batch being flushed against the limit.
+        buffer_requeue(retries, @buffer_group)
+      end
+
+      def record_size(record)
+        record[:data].bytesize + record.fetch(:partition_key, '').bytesize
+      end
+
+      # Groups records into requests that fit the service's record-count and
+      # request-size limits.
+      def request_slices(records)
+        slices = []
+        bytes = 0
+        records.each do |record|
+          size = record_size(record)
+          if slices.empty? || slices.last.size >= self.class.max_records || bytes + size > self.class.max_request_bytes
+            slices << []
+            bytes = 0
+          end
+          slices.last << record
+          bytes += size
+        end
+        slices
+      end
+
+      # Records from a request that failed: retried on the next flush, or
+      # dropped when drop_messages_on_flush_error is set.
+      def give_back(messages)
+        return if messages.empty?
+        if @drop_messages_on_flush_error
+          log_warning("dropping #{messages.size} records after a failed request")
+        else
+          log_warning("#{messages.size} records will be retried on the next flush")
+          buffer_requeue(messages, @buffer_group)
+        end
+      end
+
     end
   end
 end

data/lib/logstash-logger/device/base.rb

@@ -29,7 +29,7 @@ module LogStashLogger
         end
       end
 
-      def write_batch(messages, group = nil)
+      def write_batch(messages, _group = nil)
         messages.each do |message|
           write_one(message)
         end
@@ -43,7 +43,7 @@ module LogStashLogger
         close
       end
 
-      def close(opts = {})
+      def close(_opts = {})
         close!
       rescue => e
         log_error(e)

data/lib/logstash-logger/device/connectable.rb

@@ -10,17 +10,9 @@ module LogStashLogger
       def initialize(opts = {})
         super
 
-        if opts[:batch_events]
-          warn "The :batch_events option is deprecated. Please use :buffer_max_items instead"
-        end
-
-        if opts[:batch_timeout]
-          warn "The :batch_timeout option is deprecated. Please use :buffer_max_interval instead"
-        end
-
         @buffer_group = nil
-        @buffer_max_items = opts[:batch_events] || opts[:buffer_max_items]
-        @buffer_max_interval = opts[:batch_timeout] || opts[:buffer_max_interval]
+        @buffer_max_items = opts[:buffer_max_items]
+        @buffer_max_interval = opts[:buffer_max_interval]
         @drop_messages_on_flush_error =
           if opts.key?(:drop_messages_on_flush_error)
             opts.delete(:drop_messages_on_flush_error)
@@ -44,6 +36,11 @@ module LogStashLogger
 
         @buffer_logger = opts[:buffer_logger]
 
+        # Optional cap on the size of a single formatted record. Off by default.
+        # Deleted from opts so outputs that forward their options (Redis) never
+        # receive it.
+        @max_record_bytes = parse_max_record_bytes(opts.delete(:max_record_bytes))
+
         buffer_initialize(
           max_items: @buffer_max_items,
           max_interval: @buffer_max_interval,
@@ -56,7 +53,15 @@ module LogStashLogger
       end
 
       def write(message)
-        buffer_receive(message, @buffer_group) unless message.nil?
+        return if message.nil?
+        if @max_record_bytes
+          size = message.to_s.bytesize
+          if size > @max_record_bytes
+            log_warning("dropping a record of #{size} bytes: over max_record_bytes (#{@max_record_bytes})")
+            return
+          end
+        end
+        buffer_receive(message, @buffer_group)
       end
 
       def flush(*args)
@@ -117,8 +122,21 @@ module LogStashLogger
         connect
       end
 
+      private
+
+      def parse_max_record_bytes(value)
+        return nil if value.nil? || value.to_s.strip.empty?
+        bytes = Integer(value)
+        raise ArgumentError, "max_record_bytes must be a positive number of bytes: #{value.inspect}" unless bytes > 0
+        bytes
+      rescue TypeError
+        raise ArgumentError, "max_record_bytes must be a positive number of bytes: #{value.inspect}"
+      end
+
+      public
+
       # Ensure the block is executed with a valid connection
-      def with_connection(&block)
+      def with_connection()
         connect unless connected?
         yield
       rescue => e

data/lib/logstash-logger/device/file.rb

@@ -1,11 +1,14 @@
 require 'fileutils'
-
 module LogStashLogger
   module Device
     class File < Base
       def initialize(opts)
         super
         @path = opts[:path] || fail(ArgumentError, "Path is required")
+        @shift_age = opts[:shift_age]
+        @shift_size = opts[:shift_size]
+        @shift_period_suffix = opts[:shift_period_suffix]
+        @use_log_device = opts.key?(:shift_age) || opts.key?(:shift_size) || opts.key?(:shift_period_suffix)
         open
       end
 
@@ -14,9 +17,23 @@ module LogStashLogger
           ::FileUtils.mkdir_p ::File.dirname @path
         end
 
-        @io = ::File.open @path, ::File::WRONLY | ::File::APPEND | ::File::CREAT
-        @io.binmode
-        @io.sync = self.sync
+        if @use_log_device
+          require 'logger'
+          log_device_options = { binmode: true }
+          log_device_options[:shift_age] = @shift_age unless @shift_age.nil?
+          log_device_options[:shift_size] = @shift_size unless @shift_size.nil?
+          log_device_options[:shift_period_suffix] = @shift_period_suffix unless @shift_period_suffix.nil?
+          @io = ::Logger::LogDevice.new(@path, **log_device_options)
+          @io.dev.sync = self.sync unless self.sync.nil?
+        else
+          @io = ::File.open @path, ::File::WRONLY | ::File::APPEND | ::File::CREAT
+          @io.binmode
+          @io.sync = self.sync unless self.sync.nil?
+        end
+      end
+
+      def to_io
+        @io.respond_to?(:dev) ? @io.dev : @io
       end
     end
   end

data/lib/logstash-logger/device/firehose.rb

@@ -16,6 +16,12 @@ module LogStashLogger
         "ServiceUnavailableException"
       ].freeze
 
+      # https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html
+      # 1,000 KiB per record and 4 MB per request; these cannot be changed.
+      def self.max_records; 500; end
+      def self.max_record_bytes; 1_024_000; end
+      def self.max_request_bytes; 4_000_000; end
+
       def transform_message(message)
         {
           data: message