logstash-logger 0.26.1 → 1.0.0

data/docs/troubleshooting.md

@@ -0,0 +1,84 @@
+# Troubleshooting
+
+## Logstash never receives any logs
+
+If you are using a device backed by a Ruby IO object (such as a file, UDP socket, or TCP socket), please be aware that Ruby
+keeps its own internal buffer. Despite the fact that LogStashLogger buffers
+messages and flushes them periodically, the data written to the IO object can
+be buffered by Ruby internally indefinitely, and may not even write until the
+program terminates. If this bothers you or you need to see log messages
+immediately, your only recourse is to set the `sync: true` option.
+
+## JSON::GeneratorError
+
+Your application is probably attempting to log data that is not encoded in a valid way. When this happens, Ruby's
+standard JSON library will raise an exception. You may be able to overcome this by swapping out a different JSON encoder
+such as Oj. Use the [oj_mimic_json](https://github.com/ohler55/oj_mimic_json) gem to use Oj for JSON generation.
+
+## No logs getting sent on Heroku
+
+Heroku recommends installing the [rails_12factor](https://github.com/heroku/rails_12factor) so that logs get sent to STDOUT.
+Unfortunately, this overrides LogStashLogger, preventing logs from being sent to their configured destination. The solution
+is to remove `rails_12factor` from your Gemfile.
+
+## Logging eventually stops in production
+
+This is most likely not a problem with LogStashLogger, but rather a different gem changing the log level of `Rails.logger`.
+This is especially likely if you're using a threaded server such as Puma, since gems often change the log level of
+`Rails.logger` in a non thread-safe way. See [#17](https://github.com/dwbutler/logstash-logger/issues/17) for more information.
+
+## Sometimes two lines of JSON log messages get sent as one message
+
+If you're using UDP output and writing to a logstash listener, you are most likely encountering a bug in the UDP implementation
+of the logstash listener. There is no known fix at this time. See [#43](https://github.com/dwbutler/logstash-logger/issues/43)
+for more information.
+
+## Errno::EMSGSIZE - Message too long
+
+A known drawback of using TCP or UDP is the 65535 byte limit on total message size. To workaround
+this issue, you will have to truncate the message by setting the max message size:
+
+```ruby
+LogStashLogger.configure do |config|
+  config.max_message_size = 2000
+end
+```
+
+This will truncate only the `message` field of the LogStash Event. So make sure
+you set the max message size significantly less than 65535 bytes to make room
+for other fields.
+
+# Breaking changes
+
+## Version 0.27+
+
+MRI Ruby < 3.2 is no longer supported, since it has been EOL'ed. If you are on an older version of Ruby, you will need to use 0.26 or below.
+
+## Version 0.25+
+
+Rails 3.2, MRI Ruby < 2.2, and JRuby 1.7 are no longer supported, since they have been
+EOL'ed. If you are on an older version of Ruby, you will need to use 0.24 or below.
+
+## Version 0.5+
+
+* The `source` event key has been replaced with `host` to better match the latest logstash.
+* The `(host, port, type)` constructor has been deprecated in favor of an options hash constructor.
+
+## Version 0.4+
+
+`LogStash::Event` uses the v1 format starting version 1.2+. If you're using the v1, you'll need to install
+LogStashLogger version 0.4+. This is not backwards compatible with the old `LogStash::Event` v1.1.5, which uses
+the v0 format.
+
+## Version 0.3+
+
+Earlier versions of this gem (<= 0.2.1) only implemented a TCP connection.
+Newer versions (>= 0.3) also implement UDP, and use that as the new default.
+Please be aware if you are using the default constructor and still require TCP, you should add an additional argument:
+
+```ruby
+# Now defaults to UDP instead of TCP
+logger = LogStashLogger.new('localhost', 5228)
+# Explicitly specify TCP instead of UDP
+logger = LogStashLogger.new('localhost', 5228, :tcp)
+```

data/docs/usage.md

@@ -0,0 +1,148 @@
+# Usage Examples
+
+## Basic Usage
+
+```ruby
+require 'logstash-logger'
+
+# Defaults to UDP on 0.0.0.0
+logger = LogStashLogger.new(port: 5228)
+
+# Specify host and type (UDP or TCP) explicitly
+udp_logger = LogStashLogger.new(type: :udp, host: 'localhost', port: 5228)
+tcp_logger = LogStashLogger.new(type: :tcp, host: 'localhost', port: 5229)
+
+# Other types of loggers
+file_logger = LogStashLogger.new(type: :file, path: 'log/development.log', sync: true)
+unix_logger = LogStashLogger.new(type: :unix, path: '/tmp/sock')
+syslog_logger = LogStashLogger.new(type: :syslog)
+redis_logger = LogStashLogger.new(type: :redis)
+kafka_logger = LogStashLogger.new(type: :kafka)
+stdout_logger = LogStashLogger.new(type: :stdout)
+stderr_logger = LogStashLogger.new(type: :stderr)
+io_logger = LogStashLogger.new(type: :io, io: io)
+```
+
+## Using Formatters
+
+```ruby
+# Use a different formatter
+cee_logger = LogStashLogger.new(
+  type: :tcp,
+  host: 'logsene-receiver-syslog.sematext.com',
+  port: 514,
+  formatter: :cee_syslog
+)
+
+custom_formatted_logger = LogStashLogger.new(
+  type: :redis,
+  formatter: MyCustomFormatter
+)
+
+lambda_formatted_logger = LogStashLogger.new(
+  type: :stdout,
+  formatter: ->(severity, time, progname, msg) { "[#{progname}] #{msg}" }
+)
+
+ruby_default_formatter_logger = LogStashLogger.new(
+  type: :file,
+  path: 'log/development.log',
+  formatter: ::Logger::Formatter
+)
+```
+
+## Multiple Outputs
+
+```ruby
+# Send messages to multiple outputs. Each output will have the same format.
+# Syslog cannot be an output because it requires a separate logger.
+multi_delegating_logger = LogStashLogger.new(
+  type: :multi_delegator,
+  outputs: [
+    { type: :file, path: 'log/development.log' },
+    { type: :udp, host: 'localhost', port: 5228 }
+  ])
+
+# Balance messages between several outputs.
+# Works the same as multi delegator, but randomly chooses an output to send each message.
+balancer_logger = LogStashLogger.new(
+  type: :balancer,
+  outputs: [
+    { type: :udp, host: 'host1', port: 5228 },
+    { type: :udp, host: 'host2', port: 5228 }
+  ])
+
+# Send messages to multiple loggers.
+# Use this if you need to send different formats to different outputs.
+# If you need to log to syslog, you must use this.
+multi_logger = LogStashLogger.new(
+  type: :multi_logger,
+  outputs: [
+    { type: :file, path: 'log/development.log', formatter: ::Logger::Formatter },
+    { type: :tcp, host: 'localhost', port: 5228, formatter: :json }
+  ])
+```
+
+## Logging Messages
+
+```ruby
+# The following messages are written to UDP port 5228:
+
+logger.info 'test'
+# {"message":"test","@timestamp":"2014-05-22T09:37:19.204-07:00","@version":"1","severity":"INFO","host":"[hostname]"}
+
+logger.error '{"message": "error"}'
+# {"message":"error","@timestamp":"2014-05-22T10:10:55.877-07:00","@version":"1","severity":"ERROR","host":"[hostname]"}
+
+logger.debug message: 'test', foo: 'bar'
+# {"message":"test","foo":"bar","@timestamp":"2014-05-22T09:43:24.004-07:00","@version":"1","severity":"DEBUG","host":"[hostname]"}
+
+logger.warn LogStash::Event.new(message: 'test', foo: 'bar')
+# {"message":"test","foo":"bar","@timestamp":"2014-05-22T16:44:37.364Z","@version":"1","severity":"WARN","host":"[hostname]"}
+
+# Tagged logging
+logger.tagged('foo') { logger.fatal('bar') }
+# {"message":"bar","@timestamp":"2014-05-26T20:35:14.685-07:00","@version":"1","severity":"FATAL","host":"[hostname]","tags":["foo"]}
+```
+
+## URI Configuration
+
+You can use a URI to configure your logstash logger instead of a hash. This is useful in environments
+such as Heroku where you may want to read configuration values from the environment. The URI scheme
+is `type://host:port/path?key=value`. Some sample URI configurations are given below.
+
+```
+udp://localhost:5228
+tcp://localhost:5229
+unix:///tmp/socket
+file:///path/to/file
+redis://localhost:6379
+kafka://localhost:9092
+stdout:/
+stderr:/
+```
+
+Pass the URI into your logstash logger like so:
+
+```ruby
+# Read the URI from an environment variable
+logger = LogStashLogger.new(uri: ENV['LOGSTASH_URI'])
+```
+
+## What type of logger should I use?
+
+It depends on your specific needs, but most applications should use the default (UDP). Here are the advantages and
+disadvantages of each type:
+
+* UDP is faster than TCP because it's asynchronous (fire-and-forget). However, this means that log messages could get dropped.
+  This is okay for many applications.
+* TCP verifies that every message has been received via two-way communication. It also supports SSL for secure transmission
+  of log messages over a network. This could slow your app down to a crawl if the TCP listener is under heavy load.
+* A file is simple to use, but you will have to worry about log rotation and running out of disk space.
+* Writing to a Unix socket is faster than writing to a TCP or UDP port, but only works locally.
+* Writing to Redis is good for distributed setups that generate tons of logs. However, you will have another moving part and
+  have to worry about Redis running out of memory.
+* Writing to stdout is only recommended for debugging purposes.
+
+For a more detailed discussion of UDP vs TCP, I recommend reading this article:
+[UDP vs. TCP](http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/)

data/gemfiles/{rails_4.2.gemfile → rails_7.2.gemfile}

@@ -2,8 +2,7 @@
 
 source "https://rubygems.org"
 
-gem "codecov", require: false, group: :test
 gem "codeclimate-test-reporter", group: :test, require: nil
-gem "rails", "~> 4.2.0"
+gem "rails", "~> 7.2.0"
 
 gemspec path: "../"

data/gemfiles/{rails_5.0.gemfile → rails_8.0.gemfile}

@@ -2,8 +2,7 @@
 
 source "https://rubygems.org"
 
-gem "codecov", require: false, group: :test
 gem "codeclimate-test-reporter", group: :test, require: nil
-gem "rails", "~> 5.0.0"
+gem "rails", "~> 8.0.0"
 
 gemspec path: "../"

data/gemfiles/{rails_4.0.gemfile → rails_8.1.gemfile}

@@ -2,8 +2,7 @@
 
 source "https://rubygems.org"
 
-gem "codecov", require: false, group: :test
 gem "codeclimate-test-reporter", group: :test, require: nil
-gem "rails", "~> 4.0.0"
+gem "rails", "~> 8.1.0"
 
 gemspec path: "../"

data/lib/logstash-logger/buffer.rb

@@ -66,7 +66,6 @@ module LogStashLogger
   # to ensure that all accumulated messages are flushed.
   module Buffer
 
-    public
     # Initialize the buffer.
     #
     # Call directly from your constructor if you wish to set some non-default

data/lib/logstash-logger/configuration.rb

@@ -13,12 +13,11 @@ module LogStashLogger
     attr_accessor :max_message_size
     attr_accessor :default_error_logger
 
-    def initialize(*args)
+    def initialize(*_args)
       @customize_event_block = nil
       @default_error_logger = Logger.new($stderr)
 
       yield self if block_given?
-      self
     end
 
     def customize_event(&block)

data/lib/logstash-logger/device/aws_stream.rb

@@ -59,7 +59,7 @@ module LogStashLogger
         close(flush: false)
       end
 
-      def write_batch(messages, group = nil)
+      def write_batch(messages, _group = nil)
         records = messages.map{ |m| transform_message(m) }
 
         with_connection do

data/lib/logstash-logger/device/base.rb

@@ -29,7 +29,7 @@ module LogStashLogger
         end
       end
 
-      def write_batch(messages, group = nil)
+      def write_batch(messages, _group = nil)
         messages.each do |message|
           write_one(message)
         end
@@ -43,7 +43,7 @@ module LogStashLogger
         close
       end
 
-      def close(opts = {})
+      def close(_opts = {})
         close!
       rescue => e
         log_error(e)

data/lib/logstash-logger/device/connectable.rb

@@ -10,17 +10,9 @@ module LogStashLogger
       def initialize(opts = {})
         super
 
-        if opts[:batch_events]
-          warn "The :batch_events option is deprecated. Please use :buffer_max_items instead"
-        end
-
-        if opts[:batch_timeout]
-          warn "The :batch_timeout option is deprecated. Please use :buffer_max_interval instead"
-        end
-
         @buffer_group = nil
-        @buffer_max_items = opts[:batch_events] || opts[:buffer_max_items]
-        @buffer_max_interval = opts[:batch_timeout] || opts[:buffer_max_interval]
+        @buffer_max_items = opts[:buffer_max_items]
+        @buffer_max_interval = opts[:buffer_max_interval]
         @drop_messages_on_flush_error =
           if opts.key?(:drop_messages_on_flush_error)
             opts.delete(:drop_messages_on_flush_error)
@@ -118,7 +110,7 @@ module LogStashLogger
       end
 
       # Ensure the block is executed with a valid connection
-      def with_connection(&block)
+      def with_connection()
         connect unless connected?
         yield
       rescue => e

data/lib/logstash-logger/device/file.rb

@@ -1,11 +1,14 @@
 require 'fileutils'
-
 module LogStashLogger
   module Device
     class File < Base
       def initialize(opts)
         super
         @path = opts[:path] || fail(ArgumentError, "Path is required")
+        @shift_age = opts[:shift_age]
+        @shift_size = opts[:shift_size]
+        @shift_period_suffix = opts[:shift_period_suffix]
+        @use_log_device = opts.key?(:shift_age) || opts.key?(:shift_size) || opts.key?(:shift_period_suffix)
         open
       end
 
@@ -14,9 +17,23 @@ module LogStashLogger
           ::FileUtils.mkdir_p ::File.dirname @path
         end
 
-        @io = ::File.open @path, ::File::WRONLY | ::File::APPEND | ::File::CREAT
-        @io.binmode
-        @io.sync = self.sync
+        if @use_log_device
+          require 'logger'
+          log_device_options = { binmode: true }
+          log_device_options[:shift_age] = @shift_age unless @shift_age.nil?
+          log_device_options[:shift_size] = @shift_size unless @shift_size.nil?
+          log_device_options[:shift_period_suffix] = @shift_period_suffix unless @shift_period_suffix.nil?
+          @io = ::Logger::LogDevice.new(@path, **log_device_options)
+          @io.dev.sync = self.sync unless self.sync.nil?
+        else
+          @io = ::File.open @path, ::File::WRONLY | ::File::APPEND | ::File::CREAT
+          @io.binmode
+          @io.sync = self.sync unless self.sync.nil?
+        end
+      end
+
+      def to_io
+        @io.respond_to?(:dev) ? @io.dev : @io
       end
     end
   end

data/lib/logstash-logger/device/http.rb

@@ -0,0 +1,33 @@
+require 'uri'
+require 'net/http'
+
+module LogStashLogger
+  module Device
+
+    # Rudimentary write to Logstash HTTP Input relying on buffering
+    # rather than persistent HTTP connections for efficiency.
+    class HTTP < Connectable
+
+      def initialize(opts)
+        super
+        @url = URI(opts[:url])
+      end
+
+      def connect
+        # no-op
+      end
+
+      def write_one(message)
+        write_batch([message])
+      end
+
+      def write_batch(messages, _group = nil)
+        # Logstash HTTP input expects JSON array instead of lines of JSON
+        body = "[#{messages.join(',')}]"
+        resp = Net::HTTP.post @url, body, {"Content-Type" => "application/json"}
+        raise resp.message if Net::HTTPError === resp
+      end
+
+    end
+  end
+end

data/lib/logstash-logger/device/kafka.rb

@@ -1,56 +1,173 @@
-require 'poseidon'
-
 module LogStashLogger
   module Device
     class Kafka < Connectable
+      class TLSConfiguration
+        attr_reader :ssl_ca_cert, :ssl_client_cert, :ssl_client_cert_key
+
+        def initialize(opts = {})
+          @ssl_ca_cert = opts[:ssl_ca_cert]
+          @ssl_client_cert = opts[:ssl_client_cert]
+          @ssl_client_cert_key = opts[:ssl_client_cert_key]
+        end
+
+        def cert_bundle
+          @cert_bundle ||= all_cert_params? ? cert_params_as_hash : {}
+        end
+
+        def valid?
+          all_cert_params? || no_cert_params?
+        end
 
-      DEFAULT_HOST = 'localhost'
-      DEFAULT_PORT = 9092
-      DEFAULT_TOPIC = 'logstash'
-      DEFAULT_PRODUCER = 'logstash-logger'
-      DEFAULT_BACKOFF = 1
-
-      attr_accessor :hosts, :topic, :producer, :backoff
-
-      def initialize(opts)
-        super
-        host = opts[:host] || DEFAULT_HOST
-        port = opts[:port] || DEFAULT_PORT
-        @hosts = opts[:hosts] || host.split(',').map { |h| "#{h}:#{port}" }
-        @topic = opts[:path] || DEFAULT_TOPIC
-        @producer = opts[:producer] || DEFAULT_PRODUCER
-        @backoff = opts[:backoff] || DEFAULT_BACKOFF
+        def invalid?
+          !valid?
+        end
+
+        private
+
+        def cert_params_as_hash
+          { ssl_ca_cert: @ssl_ca_cert,
+            ssl_client_cert: @ssl_client_cert,
+            ssl_client_cert_key: @ssl_client_cert_key,
+          }
+        end
+
+        def all_cert_params?
+          cert_params_as_hash.values.compact.length == valid_cert_params_length
+        end
+
+        def no_cert_params?
+          cert_params_as_hash.values.compact.empty?
+        end
+
+        def valid_cert_params_length
+          cert_params_as_hash.keys.length
+        end
+      end
+
+      attr_reader :topic, :brokers, :cert_bundle, :kafka_tls_configurator,
+        :client_id
+
+      def initialize(opts = {}, kafka_tls_configurator = TLSConfiguration)
+        require 'ruby-kafka'
+        super(opts)
+
+        opts = normalize_uri_options(opts)
+        @client_id = opts[:client_id]
+        @topic = normalize_topic(opts[:topic])
         @buffer_group = @topic
+        @kafka_tls_configurator = kafka_tls_configurator
+        @brokers = make_brokers_array(opts[:brokers])
+        raise_no_brokers_set! if @brokers.empty?
+        make_cert_bundle(opts)
+      end
+
+      def connection
+        @io ||= ::Kafka.new(**kafka_client_connection_hash)
       end
 
       def connect
-        @io = ::Poseidon::Producer.new(@hosts, @producer)
+        @io = connection
       end
 
-      def with_connection
-        connect unless connected?
-        yield
-      rescue ::Poseidon::Errors::ChecksumError, Poseidon::Errors::UnableToFetchMetadata => e
-        log_error(e)
-        log_warning("reconnect/retry")
-        sleep backoff if backoff
-        reconnect
-        retry
-      rescue => e
-        log_error(e)
-        log_warning("giving up")
-        close(flush: false)
+      def write_one(message, topic=nil)
+        topic ||= @topic
+        write_messages_to_broker_and_deliver do |producer|
+          producer.produce(message, topic: topic)
+        end
       end
 
-      def write_batch(messages, topic = nil)
+      def write_batch(messages, topic=nil)
         topic ||= @topic
+        write_messages_to_broker_and_deliver do |producer|
+          messages.each {|msg| producer.produce(msg, topic: topic) }
+        end
+      end
+
+      private
+
+      def write_messages_to_broker_and_deliver(&block)
         with_connection do
-          @io.send_messages messages.map { |message| Poseidon::MessageToSend.new(topic, message) }
+          kproducer = producer
+          block.call(kproducer) if block_given?
+          kproducer.deliver_messages
+        end
+      end
+
+      def close!
+        begin
+          if @producer
+            if @producer.respond_to?(:shutdown)
+              @producer.shutdown
+            elsif @producer.respond_to?(:close)
+              @producer.close
+            end
+          end
+        ensure
+          @producer = nil
+          super
+        end
+      end
+
+      def kafka_client_connection_hash
+        { seed_brokers: @brokers,
+          client_id: @client_id,
+        }.merge(@cert_bundle)
+      end
+
+      def producer
+        @producer ||= (connected? ? @io : connection).producer
+      end
+
+      def raise_no_topic_set!
+        fail ArgumentError, "a topic must be configured"
+      end
+
+      def raise_no_brokers_set!
+        fail ArgumentError, "brokers must be configured"
+      end
+
+      def normalize_topic(topic)
+        normalized = topic.to_s.strip
+        return raise_no_topic_set! if normalized.empty?
+        normalized
+      end
+
+      def make_brokers_array(opt)
+        brokers =
+          case opt
+          when Array
+            opt.flatten
+          when String
+            opt.split(/\s+/)
+          else
+            []
+          end
+        brokers.compact.map(&:to_s).reject(&:empty?)
+      end
+
+      def normalize_uri_options(opts)
+        normalized = opts.dup
+        if normalized[:brokers].nil?
+          if normalized[:hosts]
+            normalized[:brokers] = normalized[:hosts]
+          elsif normalized[:host]
+            broker = normalized[:host]
+            broker = "#{broker}:#{normalized[:port]}" if normalized[:port]
+            normalized[:brokers] = broker
+          end
+        end
+        if normalized[:topic].nil? && normalized[:path]
+          normalized[:topic] = normalized[:path].to_s.sub(%r{\A/}, '')
         end
+        normalized
       end
 
-      def write_one(message, topic = nil)
-        write_batch([message], topic)
+      def make_cert_bundle(opts)
+        tls_conf = kafka_tls_configurator.new(opts)
+        if tls_conf.invalid?
+          fail ArgumentError, "all ssl parameters (ssl_ca_cert, ssl_client_cert and ssl_client_cert_key) are required or do use any of them to not use TLS"
+        end
+        @cert_bundle ||= tls_conf.cert_bundle
       end
     end
   end

data/lib/logstash-logger/device/redis.rb

@@ -13,6 +13,7 @@ module LogStashLogger
         @buffer_group = @list
 
         normalize_path(opts)
+        delete_unknown_keywords(opts)
 
         @redis_options = opts
       end
@@ -59,11 +60,17 @@ module LogStashLogger
       def normalize_path(opts)
         path = opts.fetch(:path, nil)
         if path
-          opts[:db] = path.gsub("/", "").to_i unless path.empty?
+          opts[:db] = path.delete("/").to_i unless path.empty?
           opts.delete(:path)
         end
       end
 
+      def delete_unknown_keywords(opts)
+        # due to restrictions in redis client version >= 5
+        # Continous error is being logged for unknown keywords, at present there is only one i.e. :sync
+        # to prevent that adding following line :)
+        opts.delete(:sync)
+      end
     end
   end
 end

data/lib/logstash-logger/device/tcp.rb

@@ -12,10 +12,6 @@ module LogStashLogger
         @ssl_context = opts[:ssl_context]
         @use_ssl = !!(@ssl_certificate || opts[:ssl_context])
         @use_ssl = opts[:ssl_enable] if opts.has_key? :ssl_enable
-        if opts.has_key?(:use_ssl)
-          @use_ssl = opts[:use_ssl]
-          warn "[LogStashLogger] The use_ssl option is deprecated. Use ssl_enable instead."
-        end
         @verify_hostname = opts.fetch(:verify_hostname, true)
       end
 
@@ -41,7 +37,7 @@ module LogStashLogger
           ssl_io
         else
           tcp_io
-        end
+                end
       end
 
       protected