semantic_logger 5.0.0 → 5.2.0

data/docs/index.md

@@ -0,0 +1,223 @@
+---
+layout: default
+heading: What is Semantic Logger?
+mermaid: true
+---
+
+
+**Contents**
+
+* TOC
+{:toc}
+
+Semantic Logger is a high-performance, asynchronous structured logging framework
+for Ruby and Rails.
+
+It does two things that ordinary loggers do not:
+
+1. **It logs structured data, not just strings.** Along with the usual text message you can
+   attach a payload (any Hash), an exception, a duration, metrics, and tags. That data is
+   preserved all the way to the destination, so it stays searchable instead of being flattened
+   into a line of text.
+2. **It logs asynchronously.** Log events are placed on an in-memory queue and written to their
+   destinations by a separate background thread. Your application is not blocked while logs are
+   written to disk, a database, or a remote service.
+
+If you are using Rails, use the companion gem
+[rails_semantic_logger](https://github.com/reidmorrison/rails_semantic_logger), which replaces
+the Rails logger automatically. See the [Rails guide](rails.html).
+
+## Why use it?
+
+### The problem with traditional logging
+
+With the standard Ruby logger you eventually end up building messages by hand:
+
+~~~ruby
+logger.info("Queried users table in #{duration} ms, with a result code of #{result}")
+~~~
+
+That reads fine for a human, but to a machine it is just a string. To answer a question like
+"show me every query that took longer than 100 ms" you have to write fragile regular expressions
+against your log files, and every developer formats their messages slightly differently.
+
+### The Semantic Logger way
+
+Log the message and the data separately:
+
+~~~ruby
+logger.info("Queried users table",
+  duration: duration,
+  result:   result,
+  table:    "users",
+  action:   "query")
+~~~
+
+Now the same event is still readable by a human, but a machine can index `duration`, `result`,
+`table`, and `action` as real fields. Send it to a JSON file, Elasticsearch, or Splunk and you can
+search, filter, and build dashboards on those fields directly, no log parsing required.
+
+### Reasons developers choose Semantic Logger
+
+* **Fast.** Logging happens on a background thread, so even logging thousands of lines per second
+  does not slow your application down. See [how it works](#how-it-works) below.
+* **Structured.** Every log entry can carry a payload, exception, duration, metrics, and tags
+  without losing that structure.
+* **Human and machine readable at the same time.** Colorized text for developers, JSON for your
+  log aggregator, written from the same log call.
+* **Many destinations at once.** Write to a file, the screen, and a remote service simultaneously,
+  each with its own log level and format. See [Appenders](appenders.html).
+* **A familiar API.** It supports the standard `debug`/`info`/`warn`/`error`/`fatal` API, so
+  existing code and other gems keep working. You mostly just change how the logger is created.
+* **Built for production.** Per-class log levels, change the log level of a running process with a
+  [signal](operations.html#linux-signals), tagged logging for tracing requests across threads, automatic exception
+  capture, and [metrics](metrics.html) for dashboards.
+
+## Quick start
+
+Install the gem:
+
+~~~bash
+gem install semantic_logger
+~~~
+
+Or add it to your `Gemfile`:
+
+~~~ruby
+gem "semantic_logger"
+~~~
+
+Configure it once when your application starts, then log:
+
+~~~ruby
+require "semantic_logger"
+
+# Log :info and above. Use :trace, :debug, :info, :warn, :error, or :fatal.
+SemanticLogger.default_level = :info
+
+# Send log output to the screen using the colorized formatter.
+SemanticLogger.add_appender(io: $stdout, formatter: :color)
+
+# Create a logger for the current class. The class name is added to every message.
+logger = SemanticLogger["MyApp"]
+
+logger.info("Application started")
+~~~
+
+That is the whole setup. The [Programmer's Guide](api.html) covers the full logging API.
+
+## A tour of the features
+
+The examples below assume you already have a `logger` from `SemanticLogger["MyApp"]`.
+
+#### Log a message with structured data
+
+~~~ruby
+logger.error("Outbound call failed", result: :failed, reason_code: -10)
+~~~
+
+#### Log an exception
+
+Pass the exception directly. The class, message, and backtrace are all captured, including any
+nested "caused by" exceptions.
+
+~~~ruby
+begin
+  # ... code that may raise
+rescue => exception
+  logger.error("Import failed", exception)
+end
+~~~
+
+#### Only evaluate expensive messages when needed
+
+Pass a block. It runs only when the log level is active, so it is skipped entirely in production
+when the level is higher.
+
+~~~ruby
+logger.debug { "Processed a total of #{records.sum(&:size)} bytes" }
+~~~
+
+#### Measure how long something takes
+
+The message is written when the block completes, together with its duration. If the block raises,
+the exception is logged with the duration and then re-raised.
+
+~~~ruby
+logger.measure_info("Called external interface") do
+  # Code to call the external service ...
+end
+~~~
+
+#### Tag a block of related log entries
+
+Every log entry inside the block carries the tags, which is invaluable for tracing one request
+across many classes and threads.
+
+~~~ruby
+SemanticLogger.tagged(user: "jbloggs", request_id: "abc123") do
+  logger.info("Hello World")
+  logger.debug("More detail")
+end
+~~~
+
+#### Name your threads
+
+So concurrent log entries are easy to tell apart.
+
+~~~ruby
+Thread.current.name = "worker-1"
+~~~
+
+## How it works
+
+Every logger forwards its log events to a single, shared background thread through an in-memory
+queue. That thread writes each event to every registered appender (destination) in turn. Because
+the writing happens off to the side, the thread that called `logger.info` returns immediately.
+
+~~~mermaid
+flowchart LR
+  subgraph callers["Application threads"]
+    direction TB
+    A["OrderProcessor<br/>logger.info"]
+    B["PaymentGateway<br/>logger.measure_info"]
+    C["Rails<br/>logger.error"]
+  end
+
+  Q["In-memory queue<br/>max_queue_size: 10,000"]
+  P(["Processor<br/>one background thread"])
+
+  subgraph appenders["Appenders"]
+    direction TB
+    D["Screen"]
+    E["File"]
+    F["Elasticsearch"]
+    G["Sentry"]
+  end
+
+  A -->|"enqueue, return"| Q
+  B --> Q
+  C --> Q
+  Q --> P
+  P --> D
+  P --> E
+  P --> F
+  P --> G
+~~~
+
+This design is why Semantic Logger is both fast and thread safe: log calls from hundreds of
+concurrent threads simply enqueue events, and each appender writes them out sequentially in the
+correct order. For the rare cases where you want logging to happen inline on the calling thread,
+see [Synchronous Operation](operations.html#synchronous-operation).
+
+This design is also tunable for production workloads. The in-memory queue is capped by default
+(blocking when full so that no log message is lost), but it can instead drop messages to protect
+application availability, give a slow appender its own thread, retry transient appender failures,
+and be monitored at runtime. See
+[Performance and reliability tuning](operations.html#performance-and-reliability-tuning) in the
+Configuration guide.
+
+### Ruby Support
+
+For the complete list of supported Ruby versions, see
+the [Testing file](https://github.com/reidmorrison/semantic_logger/blob/main/.github/workflows/ci.yml).

data/docs/log.md

@@ -0,0 +1,116 @@
+---
+layout: default
+title: Log Event
+description: >-
+  The Log event object that carries every entry through the pipeline: which
+  fields are always present, its attributes and helper methods, and how to
+  inspect one.
+---
+
+
+**Contents**
+
+* TOC
+{:toc}
+
+Every call to a logger (`logger.info`, `logger.measure_error`, and so on) builds a single **Log
+event** object. That object is the unit of work that flows through the whole pipeline: it is
+handed to each [filter](config.html#filtering), then to each appender's [formatter](config.html#custom-formatters), and
+finally written out by the [appender](appenders.html) itself, usually on a background thread so the
+calling application is not blocked.
+
+You meet the Log event whenever you customize Semantic Logger:
+
+* A **filter** Proc receives it and returns `true` to keep the entry. See [Filtering](config.html#filtering).
+* A **formatter** receives it and turns it into the final output (text, JSON, ...). See [Custom formatters](config.html#custom-formatters).
+* A custom **appender**'s `#log(log)` method receives it and writes it to a destination.
+
+In every one of those places you are reading the same object documented here.
+
+### The event is mutable
+
+The Log event is a plain mutable Ruby object: every attribute below has both a reader and a
+writer. All appenders share one event instance per log call, so change it only before it is handed
+to them: in an [`on_log`](config.html#capturing-context-with-on_log) subscriber, which runs for
+every entry, or in the `filter` of a logger instance, which runs for that logger's entries. For
+example, to add context to every entry:
+
+```ruby
+SemanticLogger.on_log do |log|
+  log.set_context(:request_id, Thread.current[:request_id])
+end
+```
+
+To redact sensitive data, see [Redacting sensitive data](security.html#redacting-sensitive-data).
+
+Do not change the event in an appender's `filter` or in a formatter: the appenders that ran before
+it have already written the event, and one that runs asynchronously may be reading it on its own
+thread at the same time.
+
+### Which fields are always present
+
+A handful of attributes are populated for **every** event:
+
+* `level`, `level_index`, `name`, `time`, `thread_name`, `tags`, `named_tags`.
+
+The rest are **situational**: they are set only when relevant to that particular call. For example
+`duration` is present only for `measure_*` calls, `exception` only when an exception was logged,
+`metric`/`metric_amount`/`dimensions` only for metrics, and `backtrace` only when backtrace
+capture is enabled (`SemanticLogger.backtrace_level`). Always guard against `nil` when reading a
+situational field.
+
+### Attributes
+
+|Attribute|Type|Description|
+|---------|----|-----------|
+|`level`|`Symbol`|Log level of the call: `:trace`, `:debug`, `:info`, `:warn`, `:error`, `:fatal`.|
+|`level_index`|`Integer`|Numeric form of the level for fast comparisons: `:trace=>0`, `:debug=>1`, `:info=>2`, `:warn=>3`, `:error=>4`, `:fatal=>5`.|
+|`name`|`String`|Class or component name supplied to the logger, for example the `"MyClass"` in `SemanticLogger["MyClass"]`.|
+|`message`|`String`|The text message. May be `nil` (for example, a metric-only event).|
+|`payload`|`Hash`|Structured data logged alongside the message. `nil` when none was supplied.|
+|`time`|`Time`|The moment the event was created.|
+|`thread_name`|`String`|Name (or id) of the thread that made the log call.|
+|`tags`|`Array<String>`|Tags active on the thread when the call was made.|
+|`named_tags`|`Hash<String, Object>`|Named tags active on the thread when the call was made.|
+|`context`|`Hash`|Named contexts captured in-line at the point the event was created. `nil` when none.|
+|`duration`|`Float`|Time in milliseconds taken by a `measure_*` call. `nil` for ordinary log calls.|
+|`exception`|`Exception`|The Ruby exception that was logged. `nil` when none. Use `each_exception` to walk a `cause` chain.|
+|`backtrace`|`Array<String>`|Backtrace captured at the call site, present only when the level is at or above `SemanticLogger.backtrace_level`.|
+|`metric`|`String`|The metric name, present for metric and `measure_*` calls. See [Metrics](metrics.html).|
+|`metric_amount`|`Float`|Numeric amount for a counter or gauge metric, for example the quantity purchased.|
+|`dimensions`|`Hash`|Dimensions (key/value labels) supplied for a metric.|
+
+### Helper methods
+
+These read-only helpers derive convenient values from the attributes above:
+
+|Method|Returns|Description|
+|------|-------|-----------|
+|`payload?`|`true`/`false`|Whether the event carries a non-empty payload.|
+|`payload_to_s`|`String` or `nil`|The payload rendered with `inspect`, or `nil` when absent.|
+|`metric_only?`|`true`/`false`|`true` when the event has a metric but no message and no exception. Text appenders typically skip these; machine-readable (JSON) appenders usually keep them.|
+|`duration_to_s`|`String` or `nil`|The duration in milliseconds as a short string, for example `"12ms"`. `nil` when there is no duration.|
+|`duration_human`|`String` or `nil`|The duration in human readable form, scaling up to seconds, minutes, hours, or days as needed.|
+|`level_to_s`|`String`|The level as a single upper-case character, for example `"I"` for `:info`.|
+|`cleansed_message`|`String`|The message with Rails/ANSI color escape codes stripped. Used to keep terminal codes out of structured output.|
+|`each_exception`|Enumerator|Iterates the exception and its nested `cause` chain, yielding `(exception, depth)`.|
+|`backtrace_to_s`|`String`|The exception backtrace as a string, including every exception in the `cause` chain. With a block, each cause's message and each backtrace line is replaced by the value the block returns, for example to escape it.|
+|`exception_details(exception, source: false)`|`String` or `nil`|The detail Ruby's `Exception#detailed_message` adds to the message of `exception` (this event's exception or one of its causes), such as did_you_mean suggestions, without the class name. `source: true` also includes error_highlight's source snippet. `nil` when there is none. Cached on the event.|
+|`file_name_and_line(short_name = false)`|`[String, Integer]` or `nil`|The file name and line number from the event's backtrace or exception. Pass `true` for just the base file name.|
+|`to_h`|`Hash`|The event as a plain Hash (via the Raw formatter), including `host`, `application`, and `environment`. Handy for inspecting an event or building a custom formatter.|
+|`set_context(key, value)`|`Hash`|Lazily initializes `context` and stores a key/value pair on it.|
+
+### Inspecting an event
+
+When writing a filter or formatter, the quickest way to see what a real event contains is to dump
+it with `to_h`:
+
+```ruby
+SemanticLogger.add_appender(
+  io:     $stdout,
+  filter: ->(log) do
+    pp log.to_h   # one-off: inspect the event during development
+    true
+  end
+)
+```

data/docs/metrics.md

@@ -0,0 +1,171 @@
+---
+layout: default
+title: Metrics
+description: >-
+  Emit named numeric metrics alongside log entries: counters, durations,
+  breaking a metric down with dimensions, and sending them on to a metrics
+  backend.
+---
+
+
+**Contents**
+
+* TOC
+{:toc}
+
+A **metric** is a named number that Semantic Logger emits alongside a log entry, so the same call
+that records *what happened* can also feed your dashboards and alerts.
+
+Use metrics to track things like:
+
+* How often an event happens (an error, a sign-up, a cache miss).
+* How long something takes (an external API call, a database query).
+* Running totals, such as the amount purchased per department over time.
+
+There are two parts to using metrics:
+
+1. **Emit** a metric by adding the `:metric` option to any log or `measure_` call. This is covered in
+   Steps 1 to 3 below.
+2. **Subscribe** a destination (Statsd, New Relic, SignalFx, and so on) that receives those metrics.
+   Until a subscriber is registered, the `:metric` option is harmless and the number goes nowhere.
+   See [Send metrics to a backend](#send-metrics-to-a-backend).
+
+Metric subscribers are notified asynchronously on the background log thread, so emitting a metric
+never slows the calling thread. A metric rides along with its log entry, so it obeys the same level
+and [filtering](config.html#filtering) rules: a metric on a `:trace` call is not emitted while the
+level is `:info`.
+
+> **Not to be confused with operational stats.** The metrics on this page are *application* metrics
+> that your code emits about what it is doing. To monitor Semantic Logger's *own* health instead,
+> such as queue sizes and the number of log entries processed or dropped, use
+> [`SemanticLogger.stats`](operations.html#monitoring-the-background-thread).
+
+## Step 1: Count something
+
+The simplest metric counts how often something happens. Add `:metric` with a name to any log call,
+and the metric is incremented by 1:
+
+~~~ruby
+logger.info(message: "Signed up", metric: "users/signup")
+~~~
+
+Supply `:metric_amount` to change the amount. To decrement:
+
+~~~ruby
+logger.info(message: "Item removed", metric: "cart/items", metric_amount: -1)
+~~~
+
+To add to a running total, such as the dollar amount of purchases per department:
+
+~~~ruby
+logger.info(message: "Purchase complete", metric: "departments/clothing", metric_amount: 189.42)
+~~~
+
+Note: Statsd counters are integers, so float amounts are rounded to the nearest integer.
+
+## Step 2: Measure how long something takes
+
+A duration metric records elapsed time. The easiest way is to add `:metric` to a `measure_` block
+(see [Measure how long something takes](api.html#step-6-measure-how-long-something-takes) in the
+Guide). The measured duration becomes the metric:
+
+~~~ruby
+logger.measure_info("Called supplier", metric: "supplier/add_user") do
+  # Code to call the external service ...
+end
+~~~
+
+When you already have the duration, emit it on a plain log call by supplying both `:metric` and
+`:duration` (in milliseconds):
+
+~~~ruby
+logger.info(message: "Called supplier", metric: "supplier/add_user", duration: 100.23)
+~~~
+
+## Step 3: Break a metric down with dimensions
+
+Dimensions are key/value labels attached to a metric, so a backend that supports them (such as
+SignalFx) can slice it, for example by user, action, or state. Add them with the `:dimensions`
+option:
+
+~~~ruby
+# A counter, broken down by user:
+logger.info(metric: "filters.count", dimensions: {user: "jbloggs"})
+
+# A gauge with an amount and dimensions:
+logger.info(metric: "filters.average", metric_amount: 1.2, dimensions: {user: "jbloggs"})
+~~~
+
+Not every backend supports dimensions. **Statsd** and **New Relic** ignore any metric that carries
+dimensions; **SignalFx** is built around them. See each subscriber below.
+
+## Send metrics to a backend
+
+A metric goes nowhere until a **metric subscriber** is registered. Add one with
+`SemanticLogger.add_appender(metric: ...)`, usually when your application starts. A subscriber
+receives every logged entry that has a `:metric`, asynchronously on the background thread.
+
+### Statsd
+
+Send metrics to [Statsd](https://github.com/statsd/statsd) over UDP, which can roll them up and
+forward them to [Graphite](https://graphiteapp.org), MongoDB, and others (requires the
+`statsd-ruby` gem):
+
+~~~ruby
+SemanticLogger.add_appender(metric: :statsd, url: "udp://localhost:8125")
+~~~
+
+Counters are integers (float amounts are rounded). Does not support dimensions.
+
+### New Relic
+
+Forward metrics to New Relic so they can be displayed on custom dashboards (requires the
+`newrelic_rpm` gem):
+
+~~~ruby
+SemanticLogger.add_appender(metric: :new_relic)
+~~~
+
+Does not support dimensions.
+
+### SignalFx
+
+Forward metrics to [SignalFx](https://www.splunk.com/en_us/products/infrastructure-monitoring.html),
+which is built around dimensions:
+
+~~~ruby
+SemanticLogger.add_appender(metric: :signalfx, token: "SIGNALFX_ORG_ACCESS_TOKEN")
+~~~
+
+`application` and `host` are always sent as dimensions. To also forward specific named tags as
+dimensions whenever they are present on a log entry, list them:
+
+~~~ruby
+SemanticLogger.add_appender(
+  metric:     :signalfx,
+  token:      "SIGNALFX_ORG_ACCESS_TOKEN",
+  dimensions: [:user_id, :state]
+)
+~~~
+
+When a duration metric has no dimensions, SignalFx receives both a gauge and a counter, so you can
+chart both the timing and the number of occurrences. When dimensions are present, the metric is sent
+as-is.
+
+### Elasticsearch and Splunk
+
+These are ordinary log [appenders](appenders.html), not metric subscribers, but a metric is part of
+the log entry, so `metric`, `metric_amount`, and any dimensions are written into the document
+automatically. You can build dashboards on those fields directly, without registering a separate
+metric subscriber.
+
+## How metrics behave
+
+**Asynchronous.** Subscribers run on the background logging thread, so emitting a metric never slows
+down the thread that logged it.
+
+**Follows the log level.** A metric is emitted only when its log entry is actually logged, so it
+obeys the same log level and filtering as any other entry. Use this to your advantage: leave detailed
+`:trace` level metrics in the code where they stay dormant under an `:info` level, then turn them on
+when needed, for example by sending the `SIGUSR2` [signal](operations.html#linux-signals) to lower
+the log level on a running process.