logstash-logger 0.26.1 → 1.0.0

data/docs/buffering.md

@@ -0,0 +1,70 @@
+# Buffering and Automatic Retries
+
+For devices that establish a connection to a remote service, log messages are buffered internally
+and flushed in a background thread. If there is a connection problem, the
+messages are held in the buffer and automatically resent until it is successful.
+Outputs that support batch writing (Redis and Kafka) will write log messages in bulk from the
+buffer. This functionality is implemented using a fork of
+[Stud::Buffer](https://github.com/jordansissel/ruby-stud/blob/master/lib/stud/buffer.rb).
+
+## Configuration Options
+
+You can configure buffering behavior by passing the following options to LogStashLogger:
+
+* `:buffer_max_items` - Max number of items to buffer before flushing. Defaults to 50.
+* `:buffer_max_interval` - Max number of seconds to wait between flushes. Defaults to 5.
+* `:drop_messages_on_flush_error` - Drop messages when there is a flush error. Defaults to false.
+* `:drop_messages_on_full_buffer` - Drop messages when the buffer is full. Defaults to true.
+* `:sync` - Flush buffer every time a message is received (blocking). Defaults to false.
+* `:buffer_flush_at_exit` - Flush messages when exiting the program. Defaults to true.
+* `:buffer_logger` - Logger to write buffer debug/error messages to. Defaults to none.
+
+You can turn buffering off by setting `sync = true`.
+
+## Caveats
+
+Please be aware of the following caveats to this behavior:
+
+* It's possible for duplicate log messages to be sent when retrying. For outputs like Redis and
+  Kafka that write in batches, the whole batch could get re-sent. If this is a problem, you
+  can add a UUID field to each event to uniquely identify it. You can either do this
+  in a `customize_event` block, or by using logstash's
+  [UUID filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html).
+* It's still possible to lose log messages. Ruby won't detect a TCP/UDP connection problem
+  immediately. In my testing, it took Ruby about 4 seconds to notice the receiving end was down
+  and start raising exceptions. Since logstash listeners over TCP/UDP do not acknowledge received
+  messages, it's not possible to know which log messages to re-send.
+* When `sync` is turned off, Ruby may buffer data internally before writing to
+  the IO device. This is why you may not see messages written immediately to a
+  UDP or TCP socket, even though LogStashLogger's buffer is periodically flushed.
+
+## Full Buffer
+
+By default, messages are discarded when the buffer gets full. This can happen
+if the output source is down for too long or log messages are being received
+too quickly. If your application suddenly terminates (for example, by SIGKILL or a power outage),
+the whole buffer will be lost.
+
+You can make message loss less likely by increasing `buffer_max_items`
+(so that more events can be held in the buffer), and decreasing `buffer_max_interval` (to wait
+less time between flushes). This will increase memory pressure on your application as log messages
+accumulate in the buffer, so make sure you have allocated enough memory to your process.
+
+If you don't want to lose messages when the buffer gets full, you can set
+`drop_messages_on_full_buffer = false`. Note that if the buffer gets full, any
+incoming log message will block, which could be undesirable.
+
+## Sync Mode
+
+All logger outputs support a `sync` setting. This is analogous to the "sync mode" setting on Ruby IO
+objects. When set to `true`, output is immediately flushed and is not buffered internally. Normally,
+for devices that connect to a remote service, buffering is a good thing because
+it improves performance and reduces the likelihood of errors affecting the program. For these devices,
+`sync` defaults to `false`, and it is recommended to leave the default value.
+You may want to turn sync mode on for testing, for example if you want to see
+log messages immediately after they are written.
+
+It is recommended to turn sync mode on for file and Unix socket outputs. This
+ensures that log messages from different threads or processes are written correctly on separate lines.
+
+See [#44](https://github.com/dwbutler/logstash-logger/issues/44) for more details.

data/docs/customization.md

@@ -0,0 +1,86 @@
+# Customization
+
+## Custom Log Fields
+
+`LogStashLogger` by default will log a JSON object with the format below.
+
+```json
+{
+  "@timestamp":"2015-01-29T10:43:32.196-05:00",
+  "@version":"1",
+  "message":"Some Message",
+  "severity":"INFO",
+  "host":"hostname"
+}
+```
+
+Some applications may need to attach additional metadata to each message.
+The `LogStash::Event` can be manipulated directly by specifying a `customize_event` block in the `LogStashLogger` configuration. (Note: this is run after the event has been generated, not before.)
+
+```ruby
+config = LogStashLogger.configure do |config|
+  config.customize_event do |event|
+    event["other_field"] = "some_other_value"
+  end
+end
+```
+
+This configuration would result in the following output.
+
+```json
+{
+    "@timestamp": "2015-01-29T10:43:32.196-05:00",
+    "@version": "1",
+    "message": "Some Message",
+    "severity": "INFO",
+    "host": "hostname",
+    "other_field": "some_other_value"
+}
+```
+
+This block has full access to the event, so you can remove fields, modify
+existing fields, etc. For example, to remove the default timestamp:
+
+```ruby
+config = LogStashLogger.configure do |config|
+  config.customize_event do |event|
+    event.remove('@timestamp')
+  end
+end
+```
+
+You can also customize events on a per-logger basis by passing a callable object
+(lambda or proc) to the `customize_event` option when creating a logger:
+
+```ruby
+LogStashLogger.new(customize_event: ->(event){ event['other_field'] = 'other_field' })
+```
+
+## Logger Silencing
+
+LogStashLogger provides support for Rails-style logger silencing. The
+implementation was extracted from Rails, but has no dependencies, so it can be
+used outside of a Rails app. The interface is the same as in Rails:
+
+```ruby
+logger.silence(temporary_level) do
+  ...
+end
+```
+
+## Custom Logger Class
+
+By default, LogStashLogger creates a logger that extends Ruby's built in `Logger` class.
+If you require a different logger implementation, you can use a different class
+by passing in the class with the `logger_class` option.
+
+Note that for syslog, the `Syslog::Logger` class is required and cannot be
+changed.
+
+## Error Handling
+
+If an exception occurs while writing a message to the device, the exception is
+logged using an internal logger. By default, this logs to $stderr. You can
+change the error logger by setting `LogStashLogger.configuration.default_error_logger`, or by passing
+your own logger object in the `:error_logger` configuration key when
+instantiating a LogStashLogger.

data/docs/outputs.md

@@ -0,0 +1,42 @@
+# Output Configuration
+
+## Logstash Listener Configuration
+
+In order for logstash to correctly receive and parse the events, you will need to
+configure and run a listener that uses the `json_lines` codec. For example, to receive
+events over UDP on port 5228:
+
+```ruby
+input {
+  udp {
+    host => "0.0.0.0"
+    port => 5228
+    codec => json_lines
+  }
+}
+```
+
+File and Redis inputs should use the `json` codec instead. For more information
+read the [Logstash docs](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html).
+
+See the [samples](https://github.com/dwbutler/logstash-logger/tree/master/samples) directory for more configuration samples.
+
+## HTTP
+
+Supports rudimentary writes (buffered, non-persistent connections) to the [Logstash HTTP Input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html):
+
+```ruby
+input {
+  http {
+    port => 8080
+  }
+}
+```
+
+```ruby
+LogStashLogger.new \
+  type: :http,
+  url: 'http://localhost:8080'
+```
+
+Note the parameter is `url` and not `uri`. Relies on [Net:HTTP](https://ruby-doc.org/stdlib-2.7.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-HTTPS) to auto-detect SSL usage from the scheme.

data/docs/rails.md

@@ -0,0 +1,344 @@
+# Rails Integration
+
+Supports Rails 7.2, 8.0, and 8.1.
+
+By default, every Rails log message will be written to logstash in `LogStash::Event` JSON format.
+
+For minimal, more-structured logstash events, try one of the following gems:
+
+* [lograge](https://github.com/roidrage/lograge)
+* [yarder](https://github.com/rurounijones/yarder)
+
+Currently these gems output a JSON string, which LogStashLogger then parses.
+Future versions of these gems could potentially have deeper integration with LogStashLogger
+(e.g. by directly writing `LogStash::Event` objects).
+
+## Rails Configuration
+
+Add the following to your `config/environments/production.rb`:
+
+### Common Options
+
+```ruby
+# Optional, Rails sets the default to :info
+config.log_level = :debug
+
+# Optional, defaults to true in development and false in production
+config.autoflush_log = false
+
+# Optional, use a URI to configure
+config.logstash.uri = ENV['LOGSTASH_URI']
+
+# Optional. Defaults to :json_lines. If there are multiple outputs,
+# they will all share the same formatter.
+config.logstash.formatter = :json_lines
+
+# Optional, the logger to log writing errors to. Defaults to logging to $stderr
+config.logstash.error_logger = Logger.new($stderr)
+
+# Optional, max number of items to buffer before flushing. Defaults to 50
+config.logstash.buffer_max_items = 50
+
+# Optional, max number of seconds to wait between flushes. Defaults to 5
+config.logstash.buffer_max_interval = 5
+
+# Optional, drop message when a connection error occurs. Defaults to false
+config.logstash.drop_messages_on_flush_error = false
+
+# Optional, drop messages when the buffer is full. Defaults to true
+config.logstash.drop_messages_on_full_buffer = true
+```
+
+### UDP
+
+```ruby
+# Optional, defaults to '0.0.0.0'
+config.logstash.host = 'localhost'
+
+# Optional, defaults to :udp.
+config.logstash.type = :udp
+
+# Required, the port to connect to
+config.logstash.port = 5228
+```
+
+### TCP
+
+```ruby
+# Optional, defaults to '0.0.0.0'
+config.logstash.host = 'localhost'
+
+# Required, the port to connect to
+config.logstash.port = 5228
+
+# Required
+config.logstash.type = :tcp
+
+# Optional, enables SSL
+config.logstash.ssl_enable = true
+```
+
+### Unix Socket
+
+```ruby
+# Required
+config.logstash.type = :unix
+
+# Required
+config.logstash.path = '/tmp/sock'
+```
+
+### Syslog
+
+`Syslog::Logger` is built into the standard library for Ruby 3.2+.
+
+```ruby
+# Required
+config.logstash.type = :syslog
+
+# Optional. Defaults to 'ruby'
+config.logstash.program_name = 'MyApp'
+
+# Optional default facility level. Only works in Ruby 2+
+config.logstash.facility = Syslog::LOG_LOCAL0
+```
+
+### Redis
+
+Add the redis gem to your Gemfile:
+
+    gem 'redis'
+
+```ruby
+# Required
+config.logstash.type = :redis
+
+# Optional, will default to the 'logstash' list
+config.logstash.list = 'logstash'
+
+# All other options are passed in to the Redis client
+# Supported options include host, port, path, password, url
+# Example:
+
+# Optional, Redis will default to localhost
+config.logstash.host = 'localhost'
+
+# Optional, Redis will default to port 6379
+config.logstash.port = 6379
+```
+
+### Kafka
+
+Add the ruby-kafka gem to your Gemfile:
+
+    gem 'ruby-kafka'
+
+```ruby
+# Required
+config.logstash.type = :kafka
+
+# Required
+config.logstash.topic = 'logstash-topic'
+
+# Required, can be in one of two formats:
+# String format (splits on single space):
+config.logstash.brokers = 'localhost:9092 some-other-host.net:9300'
+# Array format
+config.logstash.brokers = %w(localhost:9092 some-other-host.net:9300)
+
+# Optional, defaults to 'ruby-kafka'
+config.logstash.client_id = 'logstash-client-alpha'
+
+# Optional, transmit over TLS
+# NOTE: either 0 or all 3 ssl_parameters must be provided for a
+# successful connection. An exception will be raised if 1 or 2 params
+# are povided
+config.logstash.ssl_ca_cert: ENV['CLOUDKAFKA_CA']
+config.logstash.ssl_client_cert: ENV['CLOUDKAFKA_CERT']
+config.logstash.ssl_client_cert_key: ENV['CLOUDKAFKA_PRIVATE_KEY']
+```
+
+### Kinesis
+
+Add the aws-sdk gem to your Gemfile:
+
+    # aws-sdk >= 3.0
+    gem 'aws-sdk-kinesis'
+
+    # aws-sdk < 3.0
+    gem 'aws-sdk'
+
+```ruby
+# Required
+config.logstash.type = :kinesis
+
+# Optional, will default to the 'logstash' stream
+config.logstash.stream = 'my-stream-name'
+
+# Optional, will default to 'us-east-1'
+config.logstash.aws_region = 'us-west-2'
+
+# Optional, will default to the AWS_ACCESS_KEY_ID environment variable
+config.logstash.aws_access_key_id = 'ASKASKHLD12341'
+
+# Optional, will default to the AWS_SECRET_ACCESS_KEY environment variable
+config.logstash.aws_secret_access_key = 'ASKASKHLD1234123412341234'
+```
+
+### Firehose
+
+Add the aws-sdk gem to your Gemfile:
+
+    # aws-sdk >= 3.0
+    gem 'aws-sdk-firehose'
+
+    # aws-sdk < 3.0
+    gem 'aws-sdk'
+
+```ruby
+# Required
+config.logstash.type = :firehose
+
+# Optional, will default to the 'logstash' delivery stream
+config.logstash.stream = 'my-stream-name'
+
+# Optional, will default to AWS default region config chain
+config.logstash.aws_region = 'us-west-2'
+
+# Optional, will default to AWS default credential provider chain
+config.logstash.aws_access_key_id = 'ASKASKHLD12341'
+
+# Optional, will default to AWS default credential provider chain
+config.logstash.aws_secret_access_key = 'ASKASKHLD1234123412341234'
+```
+
+### File
+
+```ruby
+# Required
+config.logstash.type = :file
+
+# Optional, defaults to Rails log path
+config.logstash.path = 'log/production.log'
+
+# Optional, enable log rotation
+config.logstash.shift_age = 7
+config.logstash.shift_size = 10 * 1024 * 1024
+
+# Optional, time-based rotation
+config.logstash.shift_age = 'daily' # or 'weekly'/'monthly'
+config.logstash.shift_period_suffix = '%Y-%m-%d'
+```
+
+### IO
+
+```ruby
+# Required
+config.logstash.type = :io
+
+# Required
+config.logstash.io = io
+```
+
+### Multi Delegator
+
+```ruby
+# Required
+config.logstash.type = :multi_delegator
+
+# Required
+config.logstash.outputs = [
+  {
+    type: :file,
+    path: 'log/production.log'
+  },
+  {
+    type: :udp,
+    port: 5228,
+    host: 'localhost'
+  }
+]
+```
+
+### Multi Logger
+
+```ruby
+# Required
+config.logstash.type = :multi_logger
+
+# Required. Each logger may have its own formatter.
+config.logstash.outputs = [
+  {
+    type: :file,
+    path: 'log/production.log',
+    formatter: ::Logger::Formatter
+  },
+  {
+    type: :udp,
+    port: 5228,
+    host: 'localhost'
+  }
+]
+```
+
+## Logging HTTP request data
+
+In web applications, you can log data from HTTP requests (such as headers) using the
+[RequestStore](https://github.com/steveklabnik/request_store) middleware. The following
+example assumes Rails.
+
+```ruby
+# in Gemfile
+gem 'request_store'
+```
+
+```ruby
+# in application.rb
+LogStashLogger.configure do |config|
+  config.customize_event do |event|
+    event["session_id"] = RequestStore.store[:load_balancer_session_id]
+  end
+end
+```
+
+```ruby
+# in app/controllers/application_controller.rb
+before_filter :track_load_balancer_session_id
+
+def track_load_balancer_session_id
+  RequestStore.store[:load_balancer_session_id] = request.headers["X-LOADBALANCER-SESSIONID"]
+end
+```
+
+## Cleaning up resources when forking
+
+If your application forks (as is common with many web servers) you will need to
+manage cleaning up resources on your LogStashLogger instances. The instance method
+`#reset` is available for this purpose. Here is sample configuration for
+several common web servers used with Rails:
+
+### Passenger
+
+```ruby
+::PhusionPassenger.on_event(:starting_worker_process) do |forked|
+  Rails.logger.reset
+end
+```
+
+### Puma
+
+```ruby
+# In config/puma.rb
+on_worker_boot do
+  Rails.logger.reset
+end
+```
+
+### Unicorn
+
+```ruby
+# In config/unicorn.rb
+after_fork do |server, worker|
+  Rails.logger.reset
+end
+```

data/docs/ssl.md

@@ -0,0 +1,90 @@
+# SSL Configuration
+
+If you are using TCP then there is the option of adding an SSL certificate to the options hash on initialize.
+
+```ruby
+LogStashLogger.new(type: :tcp, port: 5228, ssl_certificate: "/path/to/certificate.crt")
+```
+
+The SSL certificate and key can be generated using
+
+    openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash.key -out logstash.crt
+
+You can also enable SSL without a certificate:
+
+```ruby
+LogStashLogger.new(type: :tcp, port: 5228, ssl_enable: true)
+```
+
+Specify an SSL context to have more control over the behavior. For example,
+set the verify mode:
+
+```ruby
+ctx = OpenSSL::SSL::SSLContext.new
+ctx.set_params(verify_mode: OpenSSL::SSL::VERIFY_NONE)
+LogStashLogger.new(type: :tcp, port: 5228, ssl_context: ctx)
+```
+
+## Logstash Configuration for SSL
+
+The following Logstash configuration is required for SSL:
+
+```ruby
+input {
+  tcp {
+    host => "0.0.0.0"
+    port => 5228
+    codec => json_lines
+    ssl_enable => true
+    ssl_cert => "/path/to/certificate.crt"
+    ssl_key => "/path/to/key.key"
+  }
+}
+```
+
+## Hostname Verification
+
+Hostname verification is enabled by default. Without further configuration,
+the hostname supplied to `:host` will be used to verify the server's certificate
+identity.
+
+If you don't pass an `:ssl_context` or pass a falsey value to the
+`:verify_hostname` option, hostname verification will not occur.
+
+### Examples
+
+**Verify the hostname from the `:host` option**
+
+```ruby
+ctx = OpenSSL::SSL::SSLContext.new
+ctx.cert = '/path/to/cert.pem'
+ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER
+
+LogStashLogger.new \
+  type: :tcp,
+  host: 'logstash.example.com'
+  port: 5228,
+  ssl_context: ctx
+```
+
+**Verify a hostname different from the `:host` option**
+
+```ruby
+LogStashLogger.new \
+  type: :tcp,
+  host: '1.2.3.4'
+  port: 5228,
+  ssl_context: ctx,
+  verify_hostname: 'server.example.com'
+```
+
+**Explicitly disable hostname verification**
+
+```ruby
+LogStashLogger.new \
+  type: :tcp,
+  host: '1.2.3.4'
+  port: 5228,
+  ssl_context: ctx,
+  verify_hostname: false
+```