semantic_logger 5.0.0 → 5.2.0

data/docs/upgrading.md

@@ -0,0 +1,697 @@
+---
+layout: default
+title: Upgrading
+description: >-
+  Version-by-version upgrade notes for Semantic Logger, from v4.0 through the
+  v5.2 security fixes, remote syslog facility fix, and deprecation of the Splunk SDK appender.
+---
+
+
+**Contents**
+
+* TOC
+{:toc}
+
+### Upgrading to Semantic Logger v5.2
+
+v5.2 sends the configured facility to remote syslog, deprecates the `:splunk` appender, stops
+replacing Ruby's `Thread#name`, and fixes several security issues. None of the security fixes change
+the public API, but a few change the output, which matters if you parse the logs or alert on them,
+and the file appender's `permissions:` option can now make `add_appender` raise for a log file that
+worked in v5.1.
+
+#### Set the Sentry user and fingerprint with named tags
+
+The `:sentry_ruby` appender takes the Sentry user (`user_id`, `username`, `user_email`, `ip_address`,
+and `user`) and `fingerprint` from named tags or the payload. Taking them from the payload is
+deprecated and will be **removed in v6**, since the payload often holds untrusted data, such as a
+request body, which could then set the user an error is reported for. It emits a deprecation warning
+the first time it happens. Until then, the named tags win over the payload, which no longer replaces
+a user field they set, and only a String, Symbol, or Integer is taken from the payload for a user
+field. Set them with named tags instead, which can now also hold the `fingerprint`:
+
+~~~ruby
+# Before:
+logger.error("Payment failed", user_id: user.id, fingerprint: ["payment-failed"])
+
+# After:
+SemanticLogger.tagged(user_id: user.id, fingerprint: ["payment-failed"]) do
+  logger.error("Payment failed")
+end
+~~~
+
+#### Replace the `:splunk` appender with `:splunk_http`
+
+The `:splunk` appender depends on the `splunk-sdk-ruby` gem, which was last released in 2015 and
+does not verify the server's TLS certificate. It now emits a deprecation warning and will be
+**removed in v6**. Send logs to the Splunk HTTP Event Collector instead:
+
+~~~ruby
+# Before:
+SemanticLogger.add_appender(appender: :splunk, index: "main")
+
+# After:
+SemanticLogger.add_appender(
+  appender: :splunk_http,
+  url:      "https://splunk.example.com:8088/services/collector/event",
+  token:    "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
+  index:    "main"
+)
+~~~
+
+#### Remote syslog sends the configured facility
+
+When logging to a remote syslog server over TCP or UDP, earlier versions sent the wrong facility.
+The `facility:` option is documented as one of Ruby's `::Syslog::LOG_*` facility constants, but the
+value was sent as if it were already the syslog protocol's facility number. Those constants are the
+facility number shifted left by 3 bits, so the default `::Syslog::LOG_USER` (8) went out as facility
+8, which is `uucp`, instead of facility 1, `user`. For example a `:info` message, at the default
+`LOG_NOTICE` severity, was sent as `<69>` rather than `<13>`. Logging to a local syslog daemon was
+not affected.
+
+v5.2 sends the facility the constant names. If your syslog server filters or routes on facility,
+for example an rsyslog rule for `uucp.*`, change it to the facility you configured: `user` unless
+you set one.
+
+The `facility:` option now only accepts the `::Syslog::LOG_*` facility constants, and raises
+`ArgumentError` for other values when the appender is created. Previously a constant other than
+`::Syslog::LOG_KERN`, `::Syslog::LOG_USER`, or `::Syslog::LOG_MAIL` raised `ArgumentError` while
+logging. If you passed a bare facility number to get around that, switch to the constant. Check
+this even if nothing raises: a bare 0, 8, or 16 is now read as `::Syslog::LOG_KERN`,
+`::Syslog::LOG_USER`, or `::Syslog::LOG_MAIL`, so `16`, which used to send `local0`, now sends
+`mail`:
+
+~~~ruby
+# Before (16 was sent as facility local0, and is now read as ::Syslog::LOG_MAIL):
+SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", facility: 16)
+
+# After:
+SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", facility: ::Syslog::LOG_LOCAL0)
+~~~
+
+This applies to the `:syslog` and `:syslog_cee` formatters. An application that logs only to a remote
+server, and so does not need the `syslog` gem, can use the same constants from
+`SemanticLogger::Formatters::Syslog`, for example `SemanticLogger::Formatters::Syslog::LOG_LOCAL0`.
+
+#### The file appender's `permissions:` refuses a log file it cannot secure
+
+With `permissions:`, the file appender only writes to a regular file owned by the process, with a
+single hard link, that the file name names directly rather than through a symbolic link.
+`SemanticLogger.add_appender` now raises `IOError`, giving the reason, for any other log file,
+which v5.1 wrote to, changing the permissions of whatever a link pointed at. So a configuration
+that worked in v5.1 can now fail at boot, for example `permissions:` on a log file that is a
+symbolic link to `/dev/stdout`, or on a log file that another user created. With `permissions:`,
+a log file that cannot be opened also raises `IOError`, instead of the operating system's error,
+such as `Errno::ENOENT`, which is its `cause`. A log file that fails the check, or to open, when it
+is reopened later, on a rotation or after a fork, is reported to the internal logger and not
+written to until it is fixed, instead of raising from the logging thread.
+
+If you set `permissions:` on such a log file, either fix the file or remove `permissions:`. To log
+to standard output, use `io: $stdout`:
+
+~~~ruby
+# Before, with log/production.log a symbolic link to /dev/stdout:
+SemanticLogger.add_appender(file_name: "log/production.log", permissions: 0o640)
+
+# After:
+SemanticLogger.add_appender(io: $stdout)
+~~~
+
+#### Ruby's own `Thread#name` is no longer replaced
+
+Earlier versions replaced `Thread#name` and `Thread#name=` with Semantic Logger's own versions,
+which were written before Ruby had them and kept the name away from Ruby itself. v5.2 leaves Ruby's
+own in place, so the name now also reaches the operating system, profilers, and `Thread#inspect`.
+On MRI, log entries do not change: a thread that has no name is still logged with its object id.
+
+Code anywhere in the application that calls `Thread#name` or `Thread#name=` now gets Ruby's
+behaviour:
+
+* `Thread#name` returns `nil` for a thread that has no name, where it returned the thread's object
+  id as a String (on JRuby, its JVM thread name). The main thread has no name on MRI; JRuby names it
+  `main`.
+* `Thread#name=` is Ruby's built-in implementation again, with Ruby's standard behaviour, instead of
+  Semantic Logger's version, which accepted any value and converted it with `to_s`. Ruby's version
+  raises `TypeError` for a value that is not a String or `nil`, such as a Symbol or an Integer. It
+  raises `ArgumentError` for a String that contains a NUL byte (`"\0"`), or whose encoding is not
+  ASCII-compatible, such as UTF-16. Setting `nil` leaves the thread with no name, where it set the
+  name to `""`.
+
+~~~ruby
+# Before:
+Thread.current.name = worker_number
+Thread.current.name.start_with?("worker")
+
+# After:
+Thread.current.name = "worker-#{worker_number}"
+Thread.current.name&.start_with?("worker")
+~~~
+
+Tests that expect `thread_name: Thread.current.name` from the
+[test helpers](testing.html) no longer work on a thread that has no name, such as the main thread
+on MRI: `Thread.current.name` is now `nil`, while the log entry still holds the thread's object id.
+The RSpec matchers `be_a_semantic_logger_event` and `log_semantic_logger_event` then fail. Worse,
+`assert_semantic_logger_event` in Minitest takes `nil` to mean "do not check", so the test still
+passes but no longer checks the thread name. Expect the object id when the thread has no name:
+
+~~~ruby
+# Before:
+assert_semantic_logger_event(event, thread_name: Thread.current.name)
+
+# After:
+assert_semantic_logger_event(event, thread_name: Thread.current.name || Thread.current.object_id.to_s)
+~~~
+
+Semantic Logger's own threads are now named after their appender, such as `SL File`, with
+`SL Appenders` for the main one, instead of `SemanticLogger::Appender::File` and
+`SemanticLogger::Appenders`. Linux keeps only the first 15 bytes of a thread name, so the longer
+names all looked the same in tools that list threads, such as `top -H`. The new names also appear
+in the thread column of the messages Semantic Logger logs about its own problems, such as an
+appender that cannot write.
+
+On JRuby:
+
+* A Ruby thread that has no name is logged with its object id, instead of its JVM thread name, such
+  as `Ruby-0-Thread-1: app/worker.rb:12`. Name the thread to give it a readable name in the logs. A
+  thread created by Java code, such as a servlet container's request thread, is still logged with
+  its JVM thread name, which now also stays current when Java code renames the thread.
+* Naming a Ruby thread sets its JVM thread name in JRuby's own format, such as
+  `Ruby-0-Thread-2@My Worker: app/worker.rb:12`, instead of exactly `My Worker`. Update any JVM
+  monitoring that matches thread names exactly.
+
+#### Output changes
+
+* `:logfmt`: spaces, control characters, invisible format characters (such as a zero width space),
+  `=`, `"`, and Unicode line separators in keys are replaced with `_`, and invalid UTF-8 bytes are
+  dropped. A tag such as `second breakfast` is now emitted as `second_breakfast=true`. A tag,
+  named tag, or payload key named like one of the entry's own fields, which are the time key
+  (`timestamp` by default), `level`, `name`, `message`, `duration`, `duration_ms`, `tag`, and the
+  exception fields, is emitted with its source as a prefix, such as `payload.name="shrek"` or
+  `tags.message=true`, instead of replacing the field, and so is a key that starts with `tags.`,
+  `named_tags.`, or `payload.`. A logged exception's class and message are emitted as
+  `exception_name` and `exception_message` (followed by `stack_trace` and `cause` as before),
+  instead of replacing the entry's `name` and `message`, and the detail that
+  `log_exception_details: true` adds as `exception_details`. Other keys that are the same once
+  encoded, such as `:user` and `"user"`, are emitted once, with the value from the later source:
+  the payload over named tags, and named tags over tags. A payload that is not a Hash is emitted
+  as a single `payload` field. A nested value (a Hash, Array, `Struct`,
+  `Data`, `Set`, or any other value whose JSON is an object or an array, such as the exception
+  backtrace) is emitted as quoted JSON, `user="{\"id\":7}"`, instead of its `#inspect` text.
+* `:syslog`: an entry with an exception stays on one line; the newlines between the exception and
+  its backtrace are escaped as `\n`.
+* `:default`, `:color`, `:pattern`, and `:syslog`: with `escape_control_chars`, the logger name,
+  thread name, file name, payload, the messages of the exceptions that caused the logged exception,
+  and the backtrace lines are escaped too. Invalid UTF-8 bytes are dropped instead of the entry failing to
+  format. With escaping on, `:color` renders the payload plain and on one line, since Amazing
+  Print's colors and line breaks cannot be told apart from control characters in the data.
+* `:one_line`: line breaks are removed from every field (name, thread name, file name, tags, named
+  tags, payload, message, and exception), carriage returns as well as newlines, and
+  `escape_control_chars` is honoured.
+* JSON based formatters: `Infinity` and `NaN` payload values are serialized as strings, and
+  structures nested deeper than 32 levels are replaced with a placeholder such as
+  `#<Hash nested too deeply>`, instead of the whole entry being dropped. A `Struct`, `Data`, or
+  `OpenStruct` value is written as an object of its members, and a `Set` as an array, as Rails
+  already does, instead of its `#inspect` text. An exception in the payload is written as its
+  message, and any other object whose JSON fails because of invalid UTF-8 as its `#to_s`, with
+  the invalid bytes dropped. The `:elasticsearch` and `:opensearch` appenders write their bulk lines the same way.
+* JSON based formatters and `:one_line`: invalid UTF-8 bytes in the message are dropped, and a
+  message in another encoding is converted to UTF-8, instead of the entry failing to format.
+* `:loki`: a nested Hash, Array, `Struct`, or `Set` in the payload is written once as JSON, keeping
+  the JSON types of its values, so `{number: 1}` is written as `{"number":1}` rather than
+  `{"number":"1"}`, and `[1, 2, 3]` as `[1,2,3]`. A payload that is not a Hash is written as a
+  single `payload` field.
+* `:open_telemetry`: a `Struct`, `Set`, or other structure in the payload, or in an Array in it, is
+  written as JSON instead of its `#inspect`. A payload that is not a Hash is submitted as a single
+  `payload` attribute.
+* Hashes logged as the payload, for example `logger.info("Webhook", params)`: a key that is also
+  a log argument (`duration`, `metric_amount`, `backtrace`, `log_exception`, ...) is only used as
+  that argument when its value is valid for it. Otherwise it stays in the payload, where it was
+  previously consumed or made the call raise. A `nil` value is ignored. When an explicit
+  `payload:` is also given, the other keys win over its values of the same name, a String key such
+  as `"account_id"` having the same name as `:account_id`.
+* Hashes returned from a logging block, for example `logger.info("Webhook") { {user: "bob"} }`, are
+  checked the same way. A log argument from the block, such as `message`, `duration`, `exception`,
+  or `backtrace`, is only used when the log entry does not have it yet, and is otherwise kept in the
+  payload. A backtrace captured because the level is at or above `backtrace_level` counts, so the
+  block's `backtrace` no longer replaces it. The block's keys no longer replace values the log call
+  supplied under the same name, whether as a String or a Symbol, and win over the values of the
+  block's own `payload` with the same name.
+* The HTTP appender and the appenders built on it (`:elasticsearch_http`, `:splunk_http`, `:loki`,
+  `:signalfx`): the url in the error logged for a rejected request shows only the scheme, host,
+  and port, and `#inspect` shows only the object address and that url.
+* The `:statsd` metrics appender sends a counter as a single `metric:amount|c` packet.
+
+#### Other behaviour changes
+
+* An async or batching appender discards a log message that it cannot format, instead of
+  retrying it. Discarded messages are reported to the internal logger and counted in the new
+  `failed` key of the appender's stats. Every error raised by a built-in appender's formatter
+  counts, as does a message too large for a UDP datagram, a MongoDB document that BSON cannot
+  serialize, and a message GELF cannot send. A custom appender can format through
+  `format_log(log)` (or `format_batch(logs)`) instead of `formatter.call` to get the same
+  treatment, or override `#message_error?` for errors its own client raises.
+* The text formatters write a payload nested more than 32 levels deep as the JSON formatters
+  write it, cut off with their placeholder, such as `"#<Hash nested too deeply>"`.
+* A log message that overflows the stack while it is written is discarded, and the worker
+  thread carries on in a new thread, instead of the worker stopping for good.
+* `#inspect` on every appender shows only its class, object address, and what it says is safe to
+  show, such as `url="https://es.example.com:9200"`, `queue_name="logs"`, or `group="/my/app"`,
+  instead of its instance variables. A custom appender can show more by overriding
+  `#inspect_details`. An appender added with `async: true` or `batch: true` shows as
+  `#<SemanticLogger::Appender::Async:0x... batch=true appender=#<...>>`.
+* `:cloudwatch_logs` sends the buffered messages in as many requests as the PutLogEvents limits
+  need, oldest first, cuts short a message larger than 1 MB, and reports to the internal logger a
+  request that fails, and the messages that AWS rejects as too new, too old, or expired. It formats each message when it is logged, writes invalid UTF-8 as valid
+  UTF-8, and discards a message that its formatter cannot write, or writes as something other than
+  a String with content, instead of failing the whole request.
+* `:sentry_ruby` leaves the log entry's payload and named tags unchanged, so every appender after
+  it, and your own Hash, keep the keys it reads. A user field, or `user` Hash, in the payload no
+  longer replaces one from the named tags. A payload user field that is not a String, Symbol, or
+  Integer, a `fingerprint` that is not an Array of Strings, Symbols, or numbers, and a payload that
+  is not a Hash, are sent as data instead.
+
+#### Stricter `logger_measure_method` arguments
+
+`logger_measure_method` now raises `ArgumentError` for a method name that is not a valid Ruby
+method name, a `min_duration` that is not a finite number, and an invalid `log_exception` or
+level. Numeric Strings, such as `min_duration: "100"`, are still accepted.
+
+### Upgrading to Semantic Logger v5.1
+
+v5.1 deprecates several legacy appenders and one appender option, and changes when the file appender
+opens its log file.
+
+The deprecated appenders and option all still work in v5.1, but each now emits a Ruby deprecation
+warning and will be **removed in v6**. The warnings use `category: :deprecated`, so they are silent
+by default and appear when deprecation warnings are enabled (for example `ruby -W:deprecated`, or
+Rails development mode). Switch away now to be ready for v6.
+
+#### Replace the `:sentry` appender with `:sentry_ruby`
+
+The `:sentry` appender depends on the end-of-life `sentry-raven` gem. Use the `:sentry_ruby`
+appender, backed by the maintained `sentry-ruby` gem.
+
+In your `Gemfile`:
+
+~~~ruby
+# Remove:
+gem "sentry-raven"
+# Add:
+gem "sentry-ruby"
+~~~
+
+When adding the appender:
+
+~~~ruby
+# Before:
+SemanticLogger.add_appender(appender: :sentry)
+
+# After:
+SemanticLogger.add_appender(appender: :sentry_ruby)
+~~~
+
+#### Replace the `:new_relic` appender with `:new_relic_logs`
+
+The `:new_relic` appender reports `:error` and `:fatal` entries to New Relic as **error events** via
+`newrelic_rpm`. It is superseded by `:new_relic_logs`, which forwards log entries to New Relic Logs:
+
+~~~ruby
+# Before:
+SemanticLogger.add_appender(appender: :new_relic)
+
+# After:
+SemanticLogger.add_appender(appender: :new_relic_logs)
+~~~
+
+Note that the behaviour differs. `:new_relic` created New Relic *error events* (and defaulted to the
+`:error` level), whereas `:new_relic_logs` forwards *log messages* to New Relic's log ingestion. Set
+`level:` to match what you want forwarded, and update any New Relic dashboards or alerts that relied
+on the old error events. Both appenders require the `newrelic_rpm` gem.
+
+#### Remove the `type:` option from the Elasticsearch and OpenSearch appenders
+
+Document `_type` has been unused since Elasticsearch 7, so the `type:` option (on `:elasticsearch`,
+`:elasticsearch_http`, and `:opensearch`) is now ignored on all server versions: the bulk appenders
+no longer send a `_type`, and the `:elasticsearch_http` appender posts to the typeless `_doc`
+endpoint. Passing `type:` logs a deprecation warning. Remove it:
+
+~~~ruby
+# Before:
+SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200", type: "log")
+
+# After:
+SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200")
+~~~
+
+#### The file appender opens its log file when it is created
+
+The file appender now opens its log file at creation rather than on the first write, matching the
+other appenders. A bad path or insufficient permissions therefore raises immediately from
+`SemanticLogger.add_appender`, instead of surfacing later on the appender thread while log messages
+are lost. This restores the pre-4.9 behavior.
+
+As a result the log file is created as soon as the appender is added, even if nothing has been
+logged yet, the same as the standard Ruby and Rails loggers. If you add appenders inside a `rescue`
+or a conditional that assumed creation could never fail, review it: the failure now happens earlier
+and in the calling thread.
+
+#### Rails: log file failures are caught at boot
+
+If you use [rails_semantic_logger](rails.html), the companion gem's v5.1 release pairs with the
+eager open above: a log file that cannot be opened is now caught during boot and the app falls back
+to standard error at the `:warn` level, rather than booting against a silently broken appender. v5.1
+also drops support for Sidekiq v4, v5, and v6. See
+[Upgrading to v5.1](rails.html#upgrading-to-v51) for details.
+
+### Upgrading to Semantic Logger v5.0
+
+#### Minimum Ruby version is now 3.2
+
+Semantic Logger v5 requires Ruby 3.2 or later. Earlier Ruby versions are end-of-life and are no
+longer supported or tested.
+
+If you are on an older Ruby, upgrade Ruby before upgrading Semantic Logger, or stay on the v4.x
+series.
+
+#### `SemanticLogger::Appender::AsyncBatch` has been removed
+
+The internal asynchronous proxy classes have been consolidated. Batch processing now runs through
+the same `SemanticLogger::Appender::Async` proxy (backed by an internal `QueueProcessor`), so the
+separate `SemanticLogger::Appender::AsyncBatch` class no longer exists.
+
+This only affects code that referenced the class directly, which is uncommon since appenders are
+added through `SemanticLogger.add_appender`. The `batch:`, `batch_size:`, and `batch_seconds:`
+options are unchanged:
+
+~~~ruby
+SemanticLogger.add_appender(appender: :http, url: "https://example.com/log", batch: true)
+~~~
+
+What changed:
+
+- `SemanticLogger.add_appender(..., batch: true)` now returns a `SemanticLogger::Appender::Async`
+  (with `#batch?` returning `true`) instead of a `SemanticLogger::Appender::AsyncBatch`.
+- Referencing the constant `SemanticLogger::Appender::AsyncBatch` now raises `NameError`.
+
+If you have a custom appender or test asserting on the proxy class, change
+`instance_of?`/`is_a?(SemanticLogger::Appender::AsyncBatch)` checks to
+`SemanticLogger::Appender::Async` and, if needed, check `appender.batch?`.
+
+#### Appenders are reopened automatically after fork
+
+Previously, applications that fork (Puma, Unicorn, Spring, Resque, `Process.daemon`, etc.) had to
+call `SemanticLogger.reopen` themselves in an after-fork hook, otherwise the child shared the
+parent's file handles and background thread.
+
+In v5 this happens automatically: a `Process._fork` / `Process.daemon` hook calls
+`SemanticLogger.reopen` in the child process. The call is guarded to run once per process.
+
+For most applications you can now **remove** the manual reopen hook, for example:
+
+~~~ruby
+# No longer needed in v5 — handled automatically:
+before_fork { SemanticLogger.flush }
+after_fork  { SemanticLogger.reopen }
+~~~
+
+If you have a reason to manage this yourself (for example you reopen at a very specific point in
+your boot sequence), disable the automatic behaviour:
+
+~~~ruby
+SemanticLogger.reopen_on_fork = false
+~~~
+
+#### Recommended: enable logger caching
+
+By default `SemanticLogger[SomeClass]` returns a **new** `Logger` instance on every call. This means
+that if one part of the code obtains a logger and later changes its `level` (or filter), other
+holders of "the same" logger do not see the change, because they hold different instances.
+
+v5 adds opt-in caching so that a `Class` or `Module` maps to a single shared `Logger` instance:
+
+~~~ruby
+SemanticLogger.cache_loggers = true
+~~~
+
+With caching enabled:
+
+- `SemanticLogger[MyClass]` and the `Loggable` mixin's `MyClass.logger` return the **same** instance.
+- Changing that logger's level or filter is visible to every holder.
+- Strings (`SemanticLogger["Name"]`) always get a fresh instance, and anonymous classes (no `name`)
+  are never cached.
+
+Enabling caching is recommended for most applications. It is opt-in (default `false`) to preserve
+existing behaviour for code that relied on getting an independent logger per call. If you need to
+discard cached loggers (for example in tests, or after redefining a class), call
+`SemanticLogger.clear_logger_cache`.
+
+#### Control characters are escaped in Syslog output
+
+To prevent log-injection via embedded control characters (newlines, escape sequences, etc.), the
+Syslog formatter now escapes control characters in records by default. If you relied on raw control
+characters reaching syslog, this output now differs. The text formatters (default, color) also gain
+an opt-in `escape_control_chars` option (default `false`) for the same protection. See the
+[Security](security.html) page for details.
+
+#### `Formatters::Base#cleanse` renamed
+
+The internal `SemanticLogger::Formatters::Base#cleanse` method was renamed to
+`#escape_control_characters`. This only affects custom formatters or subclasses that called the old
+method name directly; update them to call `#escape_control_characters`.
+
+#### Rails: appenders are now configured in one block
+
+If you use [rails_semantic_logger](rails.html), the companion gem's v5 release moves appender
+configuration into a single `config.rails_semantic_logger.appenders do |appenders| ... end` block.
+The v4 options (`format`, `add_file_appender`, `ap_options`, `filter`, `console_logger`) still work
+but are deprecated and will be removed in v6. See
+[Migrating from v4 to v5](rails.html#migrating-from-v4-to-v5) for the full before/after mapping.
+
+### Upgrading to Semantic Logger v4.18
+
+#### Async queue is now bounded by default
+
+The default `max_queue_size` for async appenders changed from `-1` (unbounded) to `10_000`.
+Under sustained high log volume, messages that exceed the queue limit will be dropped rather than
+causing memory to grow without bound.
+
+If your application logs at very high throughput and you were relying on the unbounded default,
+set the limit explicitly when adding each appender:
+
+~~~ruby
+SemanticLogger.add_appender(io: $stdout, async: true, max_queue_size: -1)
+~~~
+
+Alternatively, tune the value to match your application's tolerance for log loss vs. memory usage.
+
+#### Invalid exception values are now wrapped
+
+Previously, passing a non-`Exception` object to the `exception:` keyword argument would silently
+misbehave. It is now wrapped in an `ArgumentError` with a description of the invalid value and a
+backtrace pointing to the call site:
+
+~~~ruby
+# This previously caused undefined behaviour — now raises a descriptive ArgumentError
+logger.error("Something went wrong", exception: "a plain string")
+~~~
+
+#### New Relic formatter — trace and span metadata
+
+Previously, New Relic trace metadata (`trace.id`, `span.id`, `entity.name`, etc.) was captured at
+format time, so it worked with any appender type. In v4.18, this metadata is captured via an
+`on_log` callback that is only registered when `SemanticLogger::Appender::NewRelicLogs` is
+instantiated.
+
+If you use a plain IO or File appender with `SemanticLogger::Formatters::NewRelicLogs` as the
+formatter (rather than the dedicated `NewRelicLogs` appender), you must register the callback
+manually in your initializer, otherwise `trace.id`, `span.id`, and `entity.name` will be absent
+from all logs:
+
+~~~ruby
+require "semantic_logger/appender/new_relic_logs"
+SemanticLogger.on_log(SemanticLogger::Appender::NewRelicLogs::CAPTURE_CONTEXT)
+~~~
+
+This line should be added before any log messages are emitted.
+
+---
+
+### Upgrading to Semantic Logger v4.17
+
+#### New Relic formatter (breaking changes for New Relic users)
+
+The New Relic log formatter (`SemanticLogger::Formatters::NewRelicLogs`) has been significantly
+updated to produce a structure that New Relic can natively parse and index as individual
+attributes. Most users will benefit from richer log correlation in New Relic, but there are
+several breaking changes to be aware of.
+
+##### Log output structure
+
+Previously, all semantic fields (`payload`, `tags`, `named_tags`, `metric`, etc.) were serialized
+as a JSON string inside the `message` key. They are now emitted as top-level keys:
+
+Before (v4.16):
+~~~json
+{
+  "message": "{\"message\":\"Order processed\",\"payload\":{\"order_id\":123},\"tags\":[],\"named_tags\":{}}",
+  "timestamp": 1234567890000,
+  "log.level": "INFO"
+}
+~~~
+
+After (v4.17):
+~~~json
+{
+  "message": "Order processed",
+  "payload": { "order_id": 123 },
+  "timestamp": 1234567890000,
+  "logger": { "name": "OrderService" },
+  "thread": { "name": "main" }
+}
+~~~
+
+**Impact**: Any New Relic alert queries, dashboards, or NRQL that references fields inside the
+old `message` JSON string (e.g. `message.payload.order_id`) must be updated to reference the
+new top-level keys (e.g. `payload.order_id`).
+
+Note that New Relic's log ingestion flattens nested objects. A `payload` value that is itself a
+nested Hash may not be fully indexed as individually searchable attributes. If you need payload
+fields to be queryable in New Relic, promote them to `named_tags` instead:
+
+~~~ruby
+# Payload fields may not be individually queryable in New Relic:
+logger.info "Order processed", order_id: 123, amount: 49.99
+
+# Use named_tags for fields you want to query directly in New Relic:
+SemanticLogger.tagged(order_id: 123, amount: 49.99) do
+  logger.info "Order processed"
+end
+~~~
+
+##### named_tags are now merged to the top level
+
+Previously `named_tags` were nested inside the `message` JSON string. They are now merged
+directly into the top-level log entry, making them first-class attributes in New Relic.
+
+If a named tag key conflicts with an existing top-level key (e.g. `message`, `timestamp`), it is
+dropped and the conflicting key names are recorded under `named_tag_conflicts`.
+
+##### Error, logger, and thread fields restructured
+
+Before (v4.16):
+~~~json
+{
+  "error.message": "Something went wrong",
+  "error.class":   "RuntimeError",
+  "error.stack":   "...",
+  "log.level":     "ERROR",
+  "logger.name":   "OrderService",
+  "thread.name":   "main"
+}
+~~~
+
+After (v4.17):
+~~~json
+{
+  "error":  { "message": "Something went wrong", "class": "RuntimeError", "stack": "..." },
+  "logger": { "name": "OrderService" },
+  "thread": { "name": "main" }
+}
+~~~
+
+Update any New Relic alert conditions or NRQL queries that reference the old dot-notation keys.
+
+#### Suppressing timestamps in formatters
+
+v4.17 added `:notime` as an explicit `time_format` value to suppress timestamp output. If you
+were previously passing `time_format: nil` expecting it to suppress output, use `:notime` instead:
+
+~~~ruby
+SemanticLogger.add_appender(io: $stdout, formatter: MyFormatter.new(time_format: :notime))
+~~~
+
+---
+
+### Upgrading to Semantic Logger v4.9
+
+These changes should not be noticeable by the majority of users of Semantic Logger, since
+they are to the internal API. It is possible that advanced users may be using these internal
+API's directly.
+
+This does not affect any calls to the public api `SemanticLogger.add_appender`.
+
+File and IO are now separate appenders. When creating the File appender explicitly, its arguments
+have changed. For example, when requesting an IO stream, it needs to be changed from:
+
+~~~ruby
+SemanticLogger::Appender::File.new(io: $stderr)
+~~~
+to:
+~~~ruby
+SemanticLogger::Appender::IO.new($stderr)
+~~~
+
+Additionally, this needs to be changed from:
+~~~ruby
+SemanticLogger::Appender::File.new(file_name: "file.log")
+~~~
+to:
+~~~ruby
+SemanticLogger::Appender::File.new("file.log")
+~~~
+
+Rails Semantic Logger, if used, needs to be upgraded to v4.9 when upgrading to Semantic Logger v4.9.
+
+---
+
+### Upgrading to Semantic Logger v4.4
+
+With some forking frameworks it is necessary to call `reopen` after the fork. With v4.4 the
+workaround for Ruby 2.5 crashes is no longer needed.
+I.e. Please remove the following line if being called anywhere:
+
+~~~ruby
+SemanticLogger::Processor.instance.instance_variable_set(:@queue, Queue.new)
+~~~
+
+---
+
+### Upgrading to Semantic Logger v4.0
+
+The following changes need to be made when upgrading to V4:
+- Ruby V2.3 / JRuby V9.1 is now the minimum runtime version.
+- Replace calls to `Logger#with_payload` with `SemanticLogger.named_tagged`.
+- Replace calls to `Logger#payload` with `SemanticLogger.named_tags`.
+- MongoDB Appender requires Mongo Ruby Client V2 or greater.
+- Appenders now write payload data in a separate `:payload` tag instead of mixing them
+  directly into the root elements to avoid name clashes.
+
+As a result any calls like the following:
+
+~~~ruby
+logger.debug foo: 'foo', bar: 'bar'
+~~~
+
+Must be replaced with the following in v4:
+
+~~~ruby
+logger.debug payload: {foo: 'foo', bar: 'bar'}
+~~~
+
+Similarly, for measure blocks:
+
+~~~ruby
+logger.measure_info('How long is the sleep', foo: 'foo', bar: 'bar') { sleep 1 }
+~~~
+
+Must be replaced with the following in v4:
+
+~~~ruby
+logger.measure_info('How long is the sleep', payload: {foo: 'foo', bar: 'bar'}) { sleep 1 }
+~~~
+
+The common log call has not changed, and the payload is still logged directly:
+
+~~~ruby
+logger.debug('log this', foo: 'foo', bar: 'bar')
+~~~