logstash-logger 0.26.1 → 1.1.0

data/docs/buffering.md

@@ -0,0 +1,86 @@
+# Buffering and Automatic Retries
+
+For devices that establish a connection to a remote service, log messages are buffered internally
+and flushed in a background thread. If there is a connection problem, the
+messages are held in the buffer and automatically resent until it is successful.
+Outputs that support batch writing (Redis and Kafka) will write log messages in bulk from the
+buffer. This functionality is implemented using a fork of
+[Stud::Buffer](https://github.com/jordansissel/ruby-stud/blob/master/lib/stud/buffer.rb).
+
+## Configuration Options
+
+You can configure buffering behavior by passing the following options to LogStashLogger:
+
+* `:buffer_max_items` - Max number of items to buffer before flushing. Defaults to 50.
+* `:buffer_max_interval` - Max number of seconds to wait between flushes. Defaults to 5.
+* `:drop_messages_on_flush_error` - Drop messages when there is a flush error. Defaults to false.
+* `:drop_messages_on_full_buffer` - Drop messages when the buffer is full. Defaults to true.
+* `:sync` - Flush buffer every time a message is received (blocking). Defaults to false.
+* `:buffer_flush_at_exit` - Flush messages when exiting the program. Defaults to true.
+* `:buffer_logger` - Logger to write buffer debug/error messages to. Defaults to none.
+* `:max_record_bytes` - Drop any single formatted record larger than this many bytes, with a warning that gives its size. Off by default.
+
+You can turn buffering off by setting `sync = true`.
+
+## Caveats
+
+Please be aware of the following caveats to this behavior:
+
+* It's possible for duplicate log messages to be sent when retrying. For outputs like Redis and
+  Kafka that write in batches, the whole batch could get re-sent. If this is a problem, you
+  can add a UUID field to each event to uniquely identify it. You can either do this
+  in a `customize_event` block, or by using logstash's
+  [UUID filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html).
+* It's still possible to lose log messages. Ruby won't detect a TCP/UDP connection problem
+  immediately. In my testing, it took Ruby about 4 seconds to notice the receiving end was down
+  and start raising exceptions. Since logstash listeners over TCP/UDP do not acknowledge received
+  messages, it's not possible to know which log messages to re-send.
+* When `sync` is turned off, Ruby may buffer data internally before writing to
+  the IO device. This is why you may not see messages written immediately to a
+  UDP or TCP socket, even though LogStashLogger's buffer is periodically flushed.
+
+## Full Buffer
+
+By default, messages are discarded when the buffer gets full. This can happen
+if the output source is down for too long or log messages are being received
+too quickly. If your application suddenly terminates (for example, by SIGKILL or a power outage),
+the whole buffer will be lost.
+
+You can make message loss less likely by increasing `buffer_max_items`
+(so that more events can be held in the buffer), and decreasing `buffer_max_interval` (to wait
+less time between flushes). This will increase memory pressure on your application as log messages
+accumulate in the buffer, so make sure you have allocated enough memory to your process.
+
+If you don't want to lose messages when the buffer gets full, you can set
+`drop_messages_on_full_buffer = false`. Note that if the buffer gets full, any
+incoming log message will block, which could be undesirable.
+
+## Records that can never be delivered
+
+Some outputs can reject a record for a reason that will not go away on retry, such as its size.
+Instead of retrying the whole batch until the buffer fills, these outputs drop only the rejected
+records, and log their size and the reason, never their content:
+
+* Kafka: records the broker or client rejects with `MessageSizeTooLarge`, `RecordListTooLarge`
+  or `BufferOverflow`. The rest of the batch is sent one record at a time.
+* Kinesis and Firehose: records over the service's per-record limit (Kinesis 1 MiB by default,
+  Firehose 1,000 KiB). Batches are split to fit each service's request limits. If your Kinesis
+  stream is configured for larger records, raise the limit with `aws_max_record_bytes`, for
+  example `aws_max_record_bytes: 10 * 1024 * 1024`.
+* HTTP: records the collector rejects with 400, 413, 415 or 422. The rest of the batch is sent
+  one record at a time. Other error statuses, such as 401, 429 or 5xx, are retried.
+
+## Sync Mode
+
+All logger outputs support a `sync` setting. This is analogous to the "sync mode" setting on Ruby IO
+objects. When set to `true`, output is immediately flushed and is not buffered internally. Normally,
+for devices that connect to a remote service, buffering is a good thing because
+it improves performance and reduces the likelihood of errors affecting the program. For these devices,
+`sync` defaults to `false`, and it is recommended to leave the default value.
+You may want to turn sync mode on for testing, for example if you want to see
+log messages immediately after they are written.
+
+It is recommended to turn sync mode on for file and Unix socket outputs. This
+ensures that log messages from different threads or processes are written correctly on separate lines.
+
+See [#44](https://github.com/dwbutler/logstash-logger/issues/44) for more details.

data/docs/customization.md

@@ -0,0 +1,86 @@
+# Customization
+
+## Custom Log Fields
+
+`LogStashLogger` by default will log a JSON object with the format below.
+
+```json
+{
+  "@timestamp":"2015-01-29T10:43:32.196-05:00",
+  "@version":"1",
+  "message":"Some Message",
+  "severity":"INFO",
+  "host":"hostname"
+}
+```
+
+Some applications may need to attach additional metadata to each message.
+The `LogStash::Event` can be manipulated directly by specifying a `customize_event` block in the `LogStashLogger` configuration. (Note: this is run after the event has been generated, not before.)
+
+```ruby
+config = LogStashLogger.configure do |config|
+  config.customize_event do |event|
+    event["other_field"] = "some_other_value"
+  end
+end
+```
+
+This configuration would result in the following output.
+
+```json
+{
+    "@timestamp": "2015-01-29T10:43:32.196-05:00",
+    "@version": "1",
+    "message": "Some Message",
+    "severity": "INFO",
+    "host": "hostname",
+    "other_field": "some_other_value"
+}
+```
+
+This block has full access to the event, so you can remove fields, modify
+existing fields, etc. For example, to remove the default timestamp:
+
+```ruby
+config = LogStashLogger.configure do |config|
+  config.customize_event do |event|
+    event.remove('@timestamp')
+  end
+end
+```
+
+You can also customize events on a per-logger basis by passing a callable object
+(lambda or proc) to the `customize_event` option when creating a logger:
+
+```ruby
+LogStashLogger.new(customize_event: ->(event){ event['other_field'] = 'other_field' })
+```
+
+## Logger Silencing
+
+LogStashLogger provides support for Rails-style logger silencing. The
+implementation was extracted from Rails, but has no dependencies, so it can be
+used outside of a Rails app. The interface is the same as in Rails:
+
+```ruby
+logger.silence(temporary_level) do
+  ...
+end
+```
+
+## Custom Logger Class
+
+By default, LogStashLogger creates a logger that extends Ruby's built in `Logger` class.
+If you require a different logger implementation, you can use a different class
+by passing in the class with the `logger_class` option.
+
+Note that for syslog, the `Syslog::Logger` class is required and cannot be
+changed.
+
+## Error Handling
+
+If an exception occurs while writing a message to the device, the exception is
+logged using an internal logger. By default, this logs to $stderr. You can
+change the error logger by setting `LogStashLogger.configuration.default_error_logger`, or by passing
+your own logger object in the `:error_logger` configuration key when
+instantiating a LogStashLogger.

data/docs/outputs.md

@@ -0,0 +1,65 @@
+# Output Configuration
+
+## Logstash Listener Configuration
+
+In order for logstash to correctly receive and parse the events, you will need to
+configure and run a listener that uses the `json_lines` codec. For example, to receive
+events over UDP on port 5228:
+
+```ruby
+input {
+  udp {
+    host => "0.0.0.0"
+    port => 5228
+    codec => json_lines
+  }
+}
+```
+
+File and Redis inputs should use the `json` codec instead. For more information
+read the [Logstash docs](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html).
+
+See the [samples](https://github.com/dwbutler/logstash-logger/tree/master/samples) directory for more configuration samples.
+
+## HTTP
+
+Supports rudimentary writes (buffered, non-persistent connections) to the [Logstash HTTP Input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html):
+
+```ruby
+input {
+  http {
+    port => 8080
+  }
+}
+```
+
+```ruby
+LogStashLogger.new \
+  type: :http,
+  url: 'http://localhost:8080'
+```
+
+A batch that gets an error status is retried on the next flush. If the collector rejects the content
+(400, 413, 415 or 422), the records are sent one at a time and only the rejected ones are dropped.
+See [Records that can never be delivered](buffering.md#records-that-can-never-be-delivered).
+
+Note the parameter is `url` and not `uri`. Relies on [Net:HTTP](https://ruby-doc.org/stdlib-2.7.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-HTTPS) to auto-detect SSL usage from the scheme.
+
+By default, each batch is sent as a JSON array, which is what the Logstash HTTP Input expects. For endpoints that expect newline-delimited JSON (NDJSON), with one event per line, set `format: :ndjson`:
+
+```ruby
+LogStashLogger.new \
+  type: :http,
+  url: 'https://example.com:8071/?token=TOKEN',
+  format: :ndjson
+```
+
+The `format` option accepts `:json_array` (the default) or `:ndjson`. With `:ndjson` the request is sent with `Content-Type: application/x-ndjson`; with `:json_array` it uses `application/json`. If your endpoint expects a different header, set it explicitly with `content_type`:
+
+```ruby
+LogStashLogger.new \
+  type: :http,
+  url: 'https://example.com:8071/?token=TOKEN',
+  format: :ndjson,
+  content_type: 'application/json'
+```

data/docs/rails.md

@@ -0,0 +1,349 @@
+# Rails Integration
+
+Supports Rails 8.0 and 8.1.
+
+By default, every Rails log message will be written to logstash in `LogStash::Event` JSON format.
+
+For minimal, more-structured logstash events, try one of the following gems:
+
+* [lograge](https://github.com/roidrage/lograge)
+* [yarder](https://github.com/rurounijones/yarder)
+
+Currently these gems output a JSON string, which LogStashLogger then parses.
+Future versions of these gems could potentially have deeper integration with LogStashLogger
+(e.g. by directly writing `LogStash::Event` objects).
+
+## Rails Configuration
+
+Add the following to your `config/environments/production.rb`:
+
+### Common Options
+
+```ruby
+# Optional, Rails sets the default to :info
+config.log_level = :debug
+
+# Optional, defaults to true in development and false in production
+config.autoflush_log = false
+
+# Optional, use a URI to configure
+config.logstash.uri = ENV['LOGSTASH_URI']
+
+# Optional. Defaults to :json_lines. If there are multiple outputs,
+# they will all share the same formatter.
+config.logstash.formatter = :json_lines
+
+# Optional, the logger to log writing errors to. Defaults to logging to $stderr
+config.logstash.error_logger = Logger.new($stderr)
+
+# Optional, max number of items to buffer before flushing. Defaults to 50
+config.logstash.buffer_max_items = 50
+
+# Optional, max number of seconds to wait between flushes. Defaults to 5
+config.logstash.buffer_max_interval = 5
+
+# Optional, drop message when a connection error occurs. Defaults to false
+config.logstash.drop_messages_on_flush_error = false
+
+# Optional, drop messages when the buffer is full. Defaults to true
+config.logstash.drop_messages_on_full_buffer = true
+```
+
+### UDP
+
+```ruby
+# Optional, defaults to '0.0.0.0'
+config.logstash.host = 'localhost'
+
+# Optional, defaults to :udp.
+config.logstash.type = :udp
+
+# Required, the port to connect to
+config.logstash.port = 5228
+```
+
+### TCP
+
+```ruby
+# Optional, defaults to '0.0.0.0'
+config.logstash.host = 'localhost'
+
+# Required, the port to connect to
+config.logstash.port = 5228
+
+# Required
+config.logstash.type = :tcp
+
+# Optional, enables SSL and verifies the server certificate against the system trust store
+config.logstash.ssl_enable = true
+
+# Optional, trusts this certificate (for a self-signed or private-CA server) and enables SSL
+# config.logstash.ssl_certificate = '/path/to/certificate.crt'
+```
+
+See [SSL Configuration](ssl.md) for other options.
+
+### Unix Socket
+
+```ruby
+# Required
+config.logstash.type = :unix
+
+# Required
+config.logstash.path = '/tmp/sock'
+```
+
+### Syslog
+
+`Syslog::Logger` is built into the standard library for Ruby 3.2+.
+
+```ruby
+# Required
+config.logstash.type = :syslog
+
+# Optional. Defaults to 'ruby'
+config.logstash.program_name = 'MyApp'
+
+# Optional default facility level. Only works in Ruby 2+
+config.logstash.facility = Syslog::LOG_LOCAL0
+```
+
+### Redis
+
+Add the redis gem to your Gemfile:
+
+    gem 'redis'
+
+```ruby
+# Required
+config.logstash.type = :redis
+
+# Optional, will default to the 'logstash' list
+config.logstash.list = 'logstash'
+
+# All other options are passed in to the Redis client
+# Supported options include host, port, path, password, url
+# Example:
+
+# Optional, Redis will default to localhost
+config.logstash.host = 'localhost'
+
+# Optional, Redis will default to port 6379
+config.logstash.port = 6379
+```
+
+### Kafka
+
+Add the ruby-kafka gem to your Gemfile:
+
+    gem 'ruby-kafka'
+
+```ruby
+# Required
+config.logstash.type = :kafka
+
+# Required
+config.logstash.topic = 'logstash-topic'
+
+# Required, can be in one of two formats:
+# String format (splits on single space):
+config.logstash.brokers = 'localhost:9092 some-other-host.net:9300'
+# Array format
+config.logstash.brokers = %w(localhost:9092 some-other-host.net:9300)
+
+# Optional, defaults to 'ruby-kafka'
+config.logstash.client_id = 'logstash-client-alpha'
+
+# Optional, transmit over TLS
+# NOTE: either 0 or all 3 ssl_parameters must be provided for a
+# successful connection. An exception will be raised if 1 or 2 params
+# are povided
+config.logstash.ssl_ca_cert: ENV['CLOUDKAFKA_CA']
+config.logstash.ssl_client_cert: ENV['CLOUDKAFKA_CERT']
+config.logstash.ssl_client_cert_key: ENV['CLOUDKAFKA_PRIVATE_KEY']
+```
+
+### Kinesis
+
+Add the aws-sdk gem to your Gemfile:
+
+    # aws-sdk >= 3.0
+    gem 'aws-sdk-kinesis'
+
+    # aws-sdk < 3.0
+    gem 'aws-sdk'
+
+```ruby
+# Required
+config.logstash.type = :kinesis
+
+# Optional, will default to the 'logstash' stream
+config.logstash.stream = 'my-stream-name'
+
+# Optional, will default to 'us-east-1'
+config.logstash.aws_region = 'us-west-2'
+
+# Optional, will default to the AWS_ACCESS_KEY_ID environment variable
+config.logstash.aws_access_key_id = 'ASKASKHLD12341'
+
+# Optional, will default to the AWS_SECRET_ACCESS_KEY environment variable
+config.logstash.aws_secret_access_key = 'ASKASKHLD1234123412341234'
+```
+
+### Firehose
+
+Add the aws-sdk gem to your Gemfile:
+
+    # aws-sdk >= 3.0
+    gem 'aws-sdk-firehose'
+
+    # aws-sdk < 3.0
+    gem 'aws-sdk'
+
+```ruby
+# Required
+config.logstash.type = :firehose
+
+# Optional, will default to the 'logstash' delivery stream
+config.logstash.stream = 'my-stream-name'
+
+# Optional, will default to AWS default region config chain
+config.logstash.aws_region = 'us-west-2'
+
+# Optional, will default to AWS default credential provider chain
+config.logstash.aws_access_key_id = 'ASKASKHLD12341'
+
+# Optional, will default to AWS default credential provider chain
+config.logstash.aws_secret_access_key = 'ASKASKHLD1234123412341234'
+```
+
+### File
+
+```ruby
+# Required
+config.logstash.type = :file
+
+# Optional, defaults to Rails log path
+config.logstash.path = 'log/production.log'
+
+# Optional, enable log rotation
+config.logstash.shift_age = 7
+config.logstash.shift_size = 10 * 1024 * 1024
+
+# Optional, time-based rotation
+config.logstash.shift_age = 'daily' # or 'weekly'/'monthly'
+config.logstash.shift_period_suffix = '%Y-%m-%d'
+```
+
+### IO
+
+```ruby
+# Required
+config.logstash.type = :io
+
+# Required
+config.logstash.io = io
+```
+
+### Multi Delegator
+
+```ruby
+# Required
+config.logstash.type = :multi_delegator
+
+# Required
+config.logstash.outputs = [
+  {
+    type: :file,
+    path: 'log/production.log'
+  },
+  {
+    type: :udp,
+    port: 5228,
+    host: 'localhost'
+  }
+]
+```
+
+### Multi Logger
+
+```ruby
+# Required
+config.logstash.type = :multi_logger
+
+# Required. Each logger may have its own formatter.
+config.logstash.outputs = [
+  {
+    type: :file,
+    path: 'log/production.log',
+    formatter: ::Logger::Formatter
+  },
+  {
+    type: :udp,
+    port: 5228,
+    host: 'localhost'
+  }
+]
+```
+
+## Logging HTTP request data
+
+In web applications, you can log data from HTTP requests (such as headers) using the
+[RequestStore](https://github.com/steveklabnik/request_store) middleware. The following
+example assumes Rails.
+
+```ruby
+# in Gemfile
+gem 'request_store'
+```
+
+```ruby
+# in application.rb
+LogStashLogger.configure do |config|
+  config.customize_event do |event|
+    event["session_id"] = RequestStore.store[:load_balancer_session_id]
+  end
+end
+```
+
+```ruby
+# in app/controllers/application_controller.rb
+before_filter :track_load_balancer_session_id
+
+def track_load_balancer_session_id
+  RequestStore.store[:load_balancer_session_id] = request.headers["X-LOADBALANCER-SESSIONID"]
+end
+```
+
+## Cleaning up resources when forking
+
+If your application forks (as is common with many web servers) you will need to
+manage cleaning up resources on your LogStashLogger instances. The instance method
+`#reset` is available for this purpose. Here is sample configuration for
+several common web servers used with Rails:
+
+### Passenger
+
+```ruby
+::PhusionPassenger.on_event(:starting_worker_process) do |forked|
+  Rails.logger.reset
+end
+```
+
+### Puma
+
+```ruby
+# In config/puma.rb
+on_worker_boot do
+  Rails.logger.reset
+end
+```
+
+### Unicorn
+
+```ruby
+# In config/unicorn.rb
+after_fork do |server, worker|
+  Rails.logger.reset
+end
+```

data/docs/ssl.md

@@ -0,0 +1,102 @@
+# SSL Configuration
+
+The TCP output can connect to Logstash over TLS. The server certificate is always
+verified, and so is the hostname, unless you opt out explicitly.
+
+Enable TLS and verify the server against the system trust store:
+
+```ruby
+LogStashLogger.new(type: :tcp, host: 'logstash.example.com', port: 5228, ssl_enable: true)
+```
+
+If your Logstash server uses a self-signed certificate or a private CA, pass that
+certificate (or CA bundle) as `ssl_certificate`. It is trusted in addition to the
+system trust store, and TLS is enabled implicitly:
+
+```ruby
+LogStashLogger.new(type: :tcp, host: 'logstash.example.com', port: 5228, ssl_certificate: "/path/to/certificate.crt")
+```
+
+The file is checked when the logger is created, and an `ArgumentError` is raised
+if it is missing or is not a certificate.
+
+A self-signed certificate and key for the Logstash server can be generated with
+
+    openssl req -x509 -batch -nodes -newkey rsa:2048 -subj "/CN=logstash.example.com" \
+      -addext "subjectAltName=DNS:logstash.example.com" -keyout logstash.key -out logstash.crt
+
+Use the name your clients connect to in the certificate.
+
+Specify an SSL context for full control over the connection. For example, to
+trust a private CA:
+
+```ruby
+ctx = OpenSSL::SSL::SSLContext.new
+ctx.set_params(verify_mode: OpenSSL::SSL::VERIFY_PEER, ca_file: '/path/to/ca.pem')
+LogStashLogger.new(type: :tcp, host: 'logstash.example.com', port: 5228, ssl_context: ctx)
+```
+
+### Turning verification off
+
+Verification can only be turned off with an explicit context. This exposes the
+log stream to anyone who can intercept the connection, so use it only for local
+testing:
+
+```ruby
+ctx = OpenSSL::SSL::SSLContext.new
+ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
+LogStashLogger.new(type: :tcp, host: 'localhost', port: 5228, ssl_context: ctx, verify_hostname: false)
+```
+
+Before 1.1.0, `ssl_enable: true` without an `ssl_context` did not verify the
+server certificate. If you relied on that, either pass `ssl_certificate` with the
+server's certificate, or use the explicit context above.
+
+## Logstash Configuration for SSL
+
+The following Logstash configuration is required for SSL:
+
+```ruby
+input {
+  tcp {
+    host => "0.0.0.0"
+    port => 5228
+    codec => json_lines
+    ssl_enable => true
+    ssl_cert => "/path/to/certificate.crt"
+    ssl_key => "/path/to/key.key"
+  }
+}
+```
+
+## Hostname Verification
+
+Hostname verification is enabled by default whenever TLS is used. The hostname
+supplied to `:host` is checked against the server's certificate, and it is also
+sent as the TLS server name (SNI) unless it is an IP address.
+
+**Verify a hostname different from the `:host` option**
+
+This is useful when connecting by IP address:
+
+```ruby
+LogStashLogger.new \
+  type: :tcp,
+  host: '1.2.3.4',
+  port: 5228,
+  ssl_certificate: '/path/to/certificate.crt',
+  verify_hostname: 'logstash.example.com'
+```
+
+**Disable hostname verification**
+
+The certificate chain is still verified:
+
+```ruby
+LogStashLogger.new \
+  type: :tcp,
+  host: '1.2.3.4',
+  port: 5228,
+  ssl_certificate: '/path/to/certificate.crt',
+  verify_hostname: false
+```