semantic_logger 5.1.0 → 5.2.0

data/docs/config.md

@@ -1,9 +1,12 @@
 ---
 layout: default
+title: Configuration
+description: >-
+  Configure Semantic Logger once at startup: global settings, per-appender
+  levels and formats, filtering, and writing your own formatters and
+  appenders.
 ---
 
-## Configuration
-{:.no_toc}
 
 **Contents**
 
@@ -270,8 +273,11 @@ filter: ->(log) { log.message !~ /\Aheartbeat/ }
 ### Example: rewriting a message inside a filter
 
 A filter can also **modify** the log event before it is written, as long as it still returns `true`
-so the (now edited) entry is kept. This is handy for redacting sensitive data. Resque, for example,
-logs the entire job payload, which may contain private information:
+so the (now edited) entry is kept. This is handy for redacting sensitive data in a logger's filter,
+as below, or in `on_log` (see [Redacting sensitive data](security.html#redacting-sensitive-data)).
+Do not redact in an appender's filter: every appender writes the same log event, so the appenders
+added before it have already written the data. Resque, for example, logs the entire job payload,
+which may contain private information:
 
 ```ruby
 Resque.logger.filter = ->(log) do
@@ -339,6 +345,7 @@ Available directives:
 | `%{payload}`           | Payload rendered as a string.                            |
 | `%{exception_class}`   | Class of the logged exception, e.g. `RuntimeError`.      |
 | `%{exception_message}` | Message of the logged exception.                         |
+| `%{exception_details}` | Detail Ruby adds to the exception's message, such as did_you_mean suggestions. See [Exception details](#exception-details). |
 | `%{backtrace}`         | Backtrace of the logged exception.                       |
 | `%{duration}`          | Human readable duration, e.g. `1.2ms`.                   |
 | `%{duration_ms}`       | Duration in milliseconds (numeric).                      |
@@ -467,9 +474,11 @@ characters. They are the recommended choice when forwarding logs that may contai
 a centralized logging system.
 
 For the text formatters, enable the `escape_control_chars` option to replace control characters in
-untrusted log data (the message, tags, named tags, and exception message) with a printable, escaped
-form. For example a newline is written as `\n` and the ANSI escape as `\e`. The option is **disabled
-by default** to preserve the existing human readable output:
+untrusted log data (the logger name, thread name, message, payload, tags, named tags, the message,
+details, and backtrace lines of the logged exception, and the message and backtrace lines of each
+exception that caused it) with a printable, escaped form. For example a newline is written as `\n`
+and the ANSI escape as `\e`. The option is **disabled by default** to preserve the existing human
+readable output:
 
 ```ruby
 # Text appender that escapes control characters in untrusted data:
@@ -481,11 +490,67 @@ SemanticLogger.add_appender(io: $stdout, formatter: {color: {escape_control_char
 ```
 
 The option only escapes the control characters in the logged data; it does not touch the formatter's
-own decoration, so the `:color` formatter keeps emitting its color codes. Multi-line exception
-backtraces are also preserved, since they are generated by Semantic Logger rather than supplied as
-log data. The pattern formatter supports the same option (`{pattern: {pattern: "...",
-escape_control_chars: true}}`), and the syslog, TCP, and UDP appenders enable it where appropriate;
-see [Appenders](appenders.html).
+own decoration, so the `:color` formatter keeps emitting its color codes. It does render the payload
+without Amazing Print's colors, and on a single line, since the colors and line breaks Amazing Print
+adds cannot be told apart from those in the data. The text formatters still lay out an exception and
+its backtrace over several lines, since that layout is generated by Semantic Logger rather than
+supplied as log data. The `:syslog` formatter, which escapes control characters by
+default, is the exception: it also escapes the newlines between an exception and its backtrace lines,
+so that each entry is a single line. The pattern formatter supports the same option (`{pattern:
+{pattern: "...", escape_control_chars: true}}`), and the syslog, TCP, and UDP appenders enable it
+where appropriate; see [Appenders](appenders.html).
+
+### Exception details
+
+Since Ruby 3.2, Ruby keeps some of the most useful information about an exception out of its
+`message`, and adds it in `Exception#detailed_message` instead: did_you_mean suggestions, the
+error_highlight snippet that points at the failing call, and any detail an exception class adds by
+overriding `#detailed_message`. Semantic Logger logs that detail, without repeating the class name
+that Ruby also adds, on the line after the exception message:
+
+~~~
+2026-10-25 12:00:00.000000 E [12345:1234 orders.rb:12] Orders -- Failed to ship -- Exception: NoMethodError: undefined method 'upcas' for an instance of String
+Did you mean?  upcase
+               upcase!
+orders.rb:12:in 'Orders#ship'
+...
+~~~
+
+The `log_exception_details` formatter option controls it:
+
+| Value | Logs |
+|-------|------|
+| `true` | The detail, without the error_highlight source snippet. |
+| `:with_source` | The detail, including the error_highlight source snippet. It is about 100 times slower, because error_highlight re-parses the source file of the failing line, and it copies source code into the log. |
+| `false` | No detail. The exception message is unchanged either way. |
+
+It defaults to `true` for the `:default`, `:color`, and `:pattern` formatters, and to `false` for
+every other formatter, including `:one_line` and the syslog formatters, so their output does not
+change. The syslog appender also turns it off for the formatter it uses with a local syslog daemon.
+When enabled, `:one_line` joins the detail onto the same line, after ` -- `. With the `:json`
+formatter and the others built on `:raw`, it adds an `exception.details` field (and `details`
+within each `cause`), which `:logfmt` writes as `exception_details`:
+
+```ruby
+# Include the error_highlight source snippet in development:
+SemanticLogger.add_appender(io: $stdout, formatter: {color: {log_exception_details: :with_source}})
+
+# Add exception.details to JSON output:
+SemanticLogger.add_appender(file_name: "production.json", formatter: {json: {log_exception_details: true}})
+```
+
+The pattern formatter's `%{exception_details}` directive interpolates the detail.
+
+Notes:
+
+* On Ruby 3.2, a `NameError` or `NoMethodError` calls `inspect` on its receiver every time its
+  message is built, and logging the detail builds it twice more. If the receivers of logged
+  exceptions are slow to inspect, set `log_exception_details: false`. Ruby 3.3 and later do not
+  inspect the receiver.
+* An exception class that overrides `#detailed_message` without accepting extra keyword arguments
+  (`def detailed_message(highlight: false)`) cannot be asked to leave out error_highlight's source
+  snippet. When error_highlight decorates that exception (a `NameError`, `TypeError`, or
+  `ArgumentError`), its detail is only logged with `:with_source`.
 
 ---
 
@@ -522,7 +587,7 @@ class SimpleAppender < SemanticLogger::Subscriber
     p log
 
     # Display the formatted output
-    puts formatter.call(log, self)
+    puts format_log(log)
   end
 
   # Optional
@@ -537,6 +602,43 @@ class SimpleAppender < SemanticLogger::Subscriber
 end
 ~~~
 
+When the appender runs asynchronously (`async: true` or batching), an exception raised by `#log`
+is retried after a back-off, on the basis that the destination may be briefly unavailable. An
+exception caused by the content of one message cannot succeed on retry, so that message is discarded
+instead. `#message_error?(exception)` decides which is which. Format each message with
+`format_log(log)` (or a batch with `format_batch(logs)`) rather than `formatter.call`: any error the
+formatter raises is then raised as a `SemanticLogger::Subscriber::FormatError`, which
+`#message_error?` recognizes, since a formatter can only fail because of the content of the message.
+By default it also recognizes the errors raised when serializing a value that cannot be written
+(`JSON::GeneratorError`, `JSON::NestingError`, `EncodingError`, and `FloatDomainError`). Override it
+to recognize errors your appender's client raises for content it cannot write:
+
+~~~ruby
+def message_error?(exception)
+  exception.is_a?(MyClient::InvalidDocument) || super
+end
+~~~
+
+A `SystemStackError` is handled separately: the message is discarded, and since a thread must not
+overflow its stack twice (on macOS that crashes Ruby), the worker carries on in a new thread. Do not
+rescue it in `#log` or `#batch`.
+
+An appender that supports batching implements `#batch(logs)`. When `#batch` raises a message error,
+the whole batch is discarded by default, since it may already have written some of the messages.
+If `#batch` writes nothing unless every message can be written, for example because it formats
+them all into one request before sending it, return `true` from `#batch_all_or_nothing?`. The batch
+is then split up and retried, so that only the offending message is discarded:
+
+~~~ruby
+def batch(logs)
+  client.send_all(logs.map { |log| format_log(log) })
+end
+
+def batch_all_or_nothing?
+  true
+end
+~~~
+
 Register the appender by passing an instance to `add_appender`:
 
 ~~~ruby

data/docs/index.md

@@ -1,11 +1,9 @@
 ---
 layout: default
+heading: What is Semantic Logger?
+mermaid: true
 ---
 
-![Semantic Logger](images/hero.jpg){:.hero-image}
-
-## What is Semantic Logger?
-{:.no_toc}
 
 **Contents**
 
@@ -177,7 +175,35 @@ Every logger forwards its log events to a single, shared background thread throu
 queue. That thread writes each event to every registered appender (destination) in turn. Because
 the writing happens off to the side, the thread that called `logger.info` returns immediately.
 
-![Log message flow diagram](images/log_event_flow.png "Flow Diagram")
+~~~mermaid
+flowchart LR
+  subgraph callers["Application threads"]
+    direction TB
+    A["OrderProcessor<br/>logger.info"]
+    B["PaymentGateway<br/>logger.measure_info"]
+    C["Rails<br/>logger.error"]
+  end
+
+  Q["In-memory queue<br/>max_queue_size: 10,000"]
+  P(["Processor<br/>one background thread"])
+
+  subgraph appenders["Appenders"]
+    direction TB
+    D["Screen"]
+    E["File"]
+    F["Elasticsearch"]
+    G["Sentry"]
+  end
+
+  A -->|"enqueue, return"| Q
+  B --> Q
+  C --> Q
+  Q --> P
+  P --> D
+  P --> E
+  P --> F
+  P --> G
+~~~
 
 This design is why Semantic Logger is both fast and thread safe: log calls from hundreds of
 concurrent threads simply enqueue events, and each appender writes them out sequentially in the

data/docs/log.md

@@ -1,9 +1,12 @@
 ---
 layout: default
+title: Log Event
+description: >-
+  The Log event object that carries every entry through the pipeline: which
+  fields are always present, its attributes and helper methods, and how to
+  inspect one.
 ---
 
-## Log Event
-{:.no_toc}
 
 **Contents**
 
@@ -22,23 +25,27 @@ You meet the Log event whenever you customize Semantic Logger:
 * A **formatter** receives it and turns it into the final output (text, JSON, ...). See [Custom formatters](config.html#custom-formatters).
 * A custom **appender**'s `#log(log)` method receives it and writes it to a destination.
 
-In every one of those places you are reading (and may modify) the same object documented here.
+In every one of those places you are reading the same object documented here.
 
 ### The event is mutable
 
 The Log event is a plain mutable Ruby object: every attribute below has both a reader and a
-writer. Filters and formatters are allowed to change it before it is written, for example to
-redact a message or enrich the payload:
+writer. All appenders share one event instance per log call, so change it only before it is handed
+to them: in an [`on_log`](config.html#capturing-context-with-on_log) subscriber, which runs for
+every entry, or in the `filter` of a logger instance, which runs for that logger's entries. For
+example, to add context to every entry:
 
 ```ruby
-filter: ->(log) do
-  log.message = log.message.gsub(/\d{16}/, "[REDACTED]")
-  true # keep the (now edited) entry
+SemanticLogger.on_log do |log|
+  log.set_context(:request_id, Thread.current[:request_id])
 end
 ```
 
-Because all appenders share one event instance per log call, keep any mutation deliberate: a
-change made in a filter is visible to every appender that runs afterwards.
+To redact sensitive data, see [Redacting sensitive data](security.html#redacting-sensitive-data).
+
+Do not change the event in an appender's `filter` or in a formatter: the appenders that ran before
+it have already written the event, and one that runs asynchronously may be reading it on its own
+thread at the same time.
 
 ### Which fields are always present
 
@@ -87,7 +94,8 @@ These read-only helpers derive convenient values from the attributes above:
 |`level_to_s`|`String`|The level as a single upper-case character, for example `"I"` for `:info`.|
 |`cleansed_message`|`String`|The message with Rails/ANSI color escape codes stripped. Used to keep terminal codes out of structured output.|
 |`each_exception`|Enumerator|Iterates the exception and its nested `cause` chain, yielding `(exception, depth)`.|
-|`backtrace_to_s`|`String`|The exception backtrace as a string, including every exception in the `cause` chain.|
+|`backtrace_to_s`|`String`|The exception backtrace as a string, including every exception in the `cause` chain. With a block, each cause's message and each backtrace line is replaced by the value the block returns, for example to escape it.|
+|`exception_details(exception, source: false)`|`String` or `nil`|The detail Ruby's `Exception#detailed_message` adds to the message of `exception` (this event's exception or one of its causes), such as did_you_mean suggestions, without the class name. `source: true` also includes error_highlight's source snippet. `nil` when there is none. Cached on the event.|
 |`file_name_and_line(short_name = false)`|`[String, Integer]` or `nil`|The file name and line number from the event's backtrace or exception. Pass `true` for just the base file name.|
 |`to_h`|`Hash`|The event as a plain Hash (via the Raw formatter), including `host`, `application`, and `environment`. Handy for inspecting an event or building a custom formatter.|
 |`set_context(key, value)`|`Hash`|Lazily initializes `context` and stores a key/value pair on it.|

data/docs/metrics.md

@@ -1,9 +1,12 @@
 ---
 layout: default
+title: Metrics
+description: >-
+  Emit named numeric metrics alongside log entries: counters, durations,
+  breaking a metric down with dimensions, and sending them on to a metrics
+  backend.
 ---
 
-## Metrics
-{:.no_toc}
 
 **Contents**
 

data/docs/operations.md

@@ -1,9 +1,12 @@
 ---
 layout: default
+title: Operations
+description: >-
+  Running Semantic Logger in production: process forking, log rotation, tuning
+  the background pipeline, Linux signals, and shipping to a centralized
+  system.
 ---
 
-## Operations
-{:.no_toc}
 
 **Contents**
 
@@ -111,19 +114,13 @@ By default the queue is capped (`max_queue_size`, default `10,000`). When it fil
 because an appender cannot keep up), `logger.info` **blocks** until there is room, guaranteeing no
 message is lost at the cost of briefly slowing the application.
 
-When availability matters more than complete logs, set `non_blocking: true` so that messages are
-**dropped** instead of blocking once the queue is full:
+There are two places a slow appender can hold up the application, and a `non_blocking` setting for
+each.
 
-~~~ruby
-SemanticLogger.add_appender(
-  file_name:    "production.log",
-  async:        true,
-  non_blocking: true
-)
-~~~
-
-Dropped messages are counted and reported at most once every `dropped_message_report_seconds`
-(default `30`) so they do not go unnoticed:
+**A slow appender that runs on its own thread.** An appender added with `async: true` (or a batching
+appender) has a second queue of its own, between the main queue and that appender. Set
+`non_blocking: true` on it so that, once its own queue is full, messages for that appender are
+**dropped** instead of holding up the main thread:
 
 ~~~ruby
 SemanticLogger.add_appender(
@@ -134,6 +131,30 @@ SemanticLogger.add_appender(
 )
 ~~~
 
+The main thread never waits on that appender, so the main queue keeps draining and every other
+appender still receives every message. Prefer this when you know which appender is slow, such as one
+that writes to a remote service. Dropped messages are counted and reported to the internal logger at
+most once every `dropped_message_report_seconds` (default `30`), and appear as `dropped` for that
+appender in `SemanticLogger.stats[:appenders]`.
+
+**Everything else.** The main queue still fills if an appender that is not async slows down (for
+example `io: $stdout` writing to a pipe that is not being read), or if the main thread cannot keep up
+with the volume of messages. Set `SemanticLogger.non_blocking = true` so that messages are
+**dropped** instead of blocking `logger.info` once the main queue is full:
+
+~~~ruby
+SemanticLogger.non_blocking = true
+~~~
+
+This can be set at any time, before or after appenders are added. Dropped messages are reported to
+the internal logger at most once every 30 seconds, and the count is available from
+`SemanticLogger.stats[:dropped]`. It is not supported in synchronous mode (`SemanticLogger.sync!`),
+which has no queue.
+
+Either setting keeps logging calls from blocking; neither makes `SemanticLogger.flush` return sooner.
+A flush, including the one that runs automatically when the process exits, still waits for every
+message already on the queue to be written.
+
 `non_blocking` applies only to a capped queue. An uncapped queue (`max_queue_size: -1`) never blocks
 and never drops, but can grow without bound.
 
@@ -144,8 +165,26 @@ a transient failure (such as a brief network blip to a remote appender) does not
 logging. Each restart sleeps with an increasing back-off (1 second, then 2, ...), reset as soon as a
 message is processed successfully.
 
+A message that fails because of its own content, rather than the destination, is not retried: for
+example a value the formatter cannot serialize (a `JSON::GeneratorError`, `JSON::NestingError`,
+encoding error, or `FloatDomainError`). That one message is discarded, and the thread carries on
+without backing off. When batching with an appender that sends the whole batch in a single request,
+such as the HTTP, Loki, Splunk HTTP, Elasticsearch, and OpenSearch appenders, only that message is
+discarded and the rest of the batch is still written. With an appender that may have written part
+of the batch before failing, the whole batch is discarded, so that no message is written twice.
+
+The first discarded message is reported to the internal logger straight away, with its error. After
+that, the number discarded is reported, with the latest error, at most once every
+`dropped_message_report_seconds` (default `30`). The total appears as `failed` for that appender in
+`SemanticLogger.stats[:appenders]`.
+
+A `JSON::ParserError`, such as from a destination that answered with an HTML error page, is a
+problem with the destination, so it is retried like any other error. The appender decides which
+errors are caused by message content; see [Custom appenders](config.html#custom-appenders).
+
 After `async_max_retries` (default `100`) consecutive failed restarts the worker thread gives up,
-rather than spinning forever on a persistent failure:
+rather than spinning forever on a persistent failure. From then on new log messages for that
+appender are dropped once its queue is full, instead of blocking the application:
 
 ~~~ruby
 SemanticLogger.add_appender(
@@ -159,6 +198,12 @@ SemanticLogger.add_appender(
 Set `async_max_retries: -1` to retry indefinitely. The back-off still applies and still resets after
 a successful message.
 
+A flush or close is never left waiting on a worker thread that cannot answer it. When a batch cannot
+be written, a flush or close taken off the queue with it still runs before the thread backs off; a
+close then stops the thread instead of restarting it. If the worker thread ends without answering,
+for example once it gives up, `SemanticLogger.flush` (including the flush at exit) stops waiting on
+that appender.
+
 ### Give a slow appender its own thread
 
 If one destination is slow, such as a remote HTTP service, run just that appender on its own thread
@@ -200,12 +245,13 @@ SemanticLogger.stats
 #          capped:         true,
 #          max_queue_size: 10_000,
 #          processed:      1_529,
-#          dropped:        0 }
+#          dropped:        0,
+#          failed:         0 }  # discarded because their content could not be written
 #      ]
 #    }
 ~~~
 
-The `processed` and `dropped` counters are cumulative since process startup. Reading `stats` is
+The `processed`, `dropped`, and `failed` counters are cumulative since process startup. Reading `stats` is
 thread-safe and adds no locking to the logging hot path.
 
 Semantic Logger also warns when an entry has waited on the queue too long. Tune the threshold and how

data/docs/rails.md

@@ -1,9 +1,12 @@
 ---
 layout: default
+title: Rails
+description: >-
+  Using Semantic Logger in Rails through the rails_semantic_logger gem:
+  configuring appenders, tuning what Rails logs, common recipes, and
+  upgrading.
 ---
 
-## Rails
-{:.no_toc}
 
 **Contents**
 
@@ -275,7 +278,7 @@ you can write color to the screen and JSON to a file at the same time.
 | `:color` | Plain text with color (uses Amazing Print for the payload when installed). |
 | `:json` | One JSON object per entry. |
 | `:logfmt` | `key=value` logfmt output. |
-| `:one_line` | Each entry reduced to a single line. |
+| `:one_line` | Each entry reduced to a single line: line breaks are removed from every field (or escaped with `escape_control_chars: true`) and an exception shows only its class and message, without the backtrace. |
 | A class instance | Any instance of a class derived from `SemanticLogger::Formatters::Base`. |
 | A `Proc` | Called with the `log` entry; returns the formatted output. |
 
@@ -677,6 +680,21 @@ If you use a forking server (Puma, Unicorn) or fork worker processes, see
 [Process Forking](operations.html#process-forking). With Semantic Logger v5 appenders are reopened automatically after
 a fork, so the manual `after_fork { SemanticLogger.reopen }` hook is usually no longer needed.
 
+### Drop messages instead of blocking
+
+By default, if log messages arrive faster than the appenders can write them, calls to `logger.info`
+wait once the in-memory queue is full, so requests, jobs, and shutdown hooks that log can stall behind
+a slow destination. When staying responsive matters more than keeping every message, have them dropped
+instead, in `config/application.rb` or `config/environments/production.rb`:
+
+~~~ruby
+config.semantic_logger.non_blocking = true
+~~~
+
+This needs Semantic Logger v5.2 or later. See
+[Drop messages instead of blocking](operations.html#drop-messages-instead-of-blocking) for how it
+relates to the per-appender `non_blocking:` option and how dropped messages are reported.
+
 ### Log rotation
 
 Because the log file is held open between writes, rotate it with a **copy-truncate** strategy rather
@@ -712,6 +730,39 @@ After they initialize, Rails Semantic Logger replaces the loggers of these libra
 
 ---
 
+## Upgrading to v5.1
+
+v5.1 requires Semantic Logger v5.1 and needs no configuration changes. Two behaviors differ.
+
+### A log file that cannot be opened now fails at boot
+
+Semantic Logger v5.1 opens the file appender's log file when the appender is created, rather than on
+the first write. A missing directory, a bad path, or insufficient permissions therefore raises
+during boot, where Rails Semantic Logger catches it and falls back to logging to standard error at
+the `:warn` level, with a message explaining the problem.
+
+Previously that failure surfaced asynchronously on the appender thread after the application had
+already booted, so the fallback never ran and the app kept running against an appender that silently
+dropped every message. If you have been running with an unwritable log path without noticing, this
+release is when you will find out.
+
+As a side effect, the log file is created as soon as the appender is added, even if nothing has been
+logged yet. This matches the standard Ruby and Rails loggers.
+
+### Sidekiq v4, v5, and v6 are no longer supported
+
+Sidekiq v7 and v8 are supported and tested. The patches for older versions have been removed, along
+with the pre-7.1.6 error handler branches. Sidekiq v7 and v8 both predate this gem's Rails 7.2 and
+Ruby 3.2 minimums, so upgrading Sidekiq is the path forward.
+
+On Sidekiq 8 two of its own settings are now honored: `logged_job_attributes` adds job attributes to
+the logging context, and `skip_default_job_logging` suppresses the `Start #perform` /
+`Completed #perform` messages (an alternative to
+`RailsSemanticLogger::Sidekiq::JobLogger.perform_messages = false`). See
+[Sidekiq](#sidekiq) for the logging configuration.
+
+---
+
 ## Migrating from v4 to v5
 
 ### Ruby and Rails minimums
@@ -814,3 +865,26 @@ and will be **removed in v6**. Each is replaced by the [appenders block](#config
 | `config.rails_semantic_logger.filter` | `filter:` on the appender |
 | `config.rails_semantic_logger.console_logger` | Declare (or omit) an `add_console` appender |
 | `config.rails_semantic_logger.add_file_appender` | Declare appenders in the block (doing so already replaces the default file appender) |
+
+### Silencing the warnings until you migrate
+
+The gem registers its deprecator with the application as
+`Rails.application.deprecators[:rails_semantic_logger]`, so the standard Rails deprecation settings
+govern these warnings:
+
+~~~ruby
+# Hide them until you have migrated
+config.active_support.report_deprecations = false
+
+# Or silence only this gem's warnings, leaving the rest of the application's in place
+Rails.application.deprecators[:rails_semantic_logger].silenced = true
+
+# Or make them fatal, e.g. in config/environments/test.rb, so CI catches them
+config.active_support.deprecation = :raise
+~~~
+
+The warnings are recorded when the option is set and emitted once Rails has applied the settings
+above. Rails reads `config/environments/*.rb` well before it applies deprecation configuration, so
+warning from the option setter itself would land where none of these settings could reach it. Each
+option is reported once per boot, attributed to the place it was set. Options set later, from
+`config/initializers/*` or at runtime, warn immediately instead.