semantic_logger 5.0.0 → 5.1.0

data/docs/security.md

@@ -0,0 +1,119 @@
+---
+layout: default
+---
+
+## Security
+{:.no_toc}
+
+**Contents**
+
+* TOC
+{:toc}
+
+Logging frameworks sit on a sensitive boundary: they take data from all over an
+application, including data that originated from untrusted users, and write it to
+files, terminals, and centralized logging systems. This page describes the security
+properties of Semantic Logger and how to configure it when logging untrusted data.
+
+### Log injection and forging
+
+When a log message, tag, or exception message contains untrusted, attacker-controlled
+data (for example a user name, request parameter, or `User-Agent` header), control
+characters in that data can be abused:
+
+* A newline can forge an additional, fake log entry ("log forging"), or split one
+  record into two at a collector that frames records with a separator.
+* An ANSI escape sequence can spoof or hide terminal output when the log is viewed in
+  a terminal.
+
+The structured formatters, such as `:json`, are **not** affected, because JSON encoding
+always escapes control characters. They are the recommended choice when forwarding logs
+that may contain untrusted data to a centralized logging system.
+
+By design the human readable text formatters (`:default` and `:color`) preserve
+newlines and ANSI color codes, since multi-line and colorized output is useful when
+reading logs locally. When the text output may contain untrusted data, enable the
+`escape_control_chars` option to replace control characters with a printable, escaped
+form (for example a newline becomes `\n`):
+
+~~~ruby
+SemanticLogger.add_appender(file_name: "production.log", formatter: {default: {escape_control_chars: true}})
+~~~
+
+See [Escaping Control Characters](config.html#escaping-control-characters) for
+details.
+
+The record-framed network appenders are already safe by default:
+
+* The **TCP** and **UDP** appenders default to the JSON formatter.
+* The **Syslog** formatters escape control characters by default, since syslog frames
+  each record.
+
+If you replace the formatter on any of these appenders with a text formatter, enable
+`escape_control_chars` as shown above.
+
+### Redacting sensitive data
+
+Semantic Logger does not automatically redact secrets or personal information; it logs
+the message and payload it is given. Avoid logging passwords, tokens, full credit card
+numbers, and similar values in the first place.
+
+When sensitive values can reach the logs indirectly, redact them. A global `on_log`
+subscriber runs inline, once per log event, before the event is handed to any appender,
+so it can scrub the message or payload for every destination:
+
+~~~ruby
+SECRET_KEYS = %i[password token authorization secret].freeze
+
+SemanticLogger.on_log do |log|
+  log.payload&.each_key do |key|
+    log.payload[key] = "[REDACTED]" if SECRET_KEYS.include?(key)
+  end
+end
+~~~
+
+Redaction can also be applied to a single logger or appender with a `filter` that
+mutates the log and returns `true`. See [Filtering](config.html#filtering) for examples.
+
+Rails applications should use the sister gem
+[rails_semantic_logger](https://github.com/reidmorrison/rails_semantic_logger), which
+honours `config.filter_parameters` so that parameters Rails already considers sensitive
+are filtered out of the logs.
+
+### Transport encryption
+
+Appenders that send logs over the network support TLS. The appenders built on the
+`:http` appender (`:http`, `:splunk_http`, and `:elasticsearch_http`) default to
+verifying the server certificate (`OpenSSL::SSL::VERIFY_PEER`) when an `https` URL is
+used.
+
+Do not disable certificate verification in production. Settings such as
+`ssl: {verify_mode: OpenSSL::SSL::VERIFY_NONE}` expose the connection to
+man-in-the-middle attacks and should be limited to local development against
+self-signed certificates.
+
+### Log file permissions
+
+Log files frequently contain sensitive information. By default the file appender
+creates files using the process umask, the standard Ruby behavior. To restrict access,
+supply the `permissions` option, which is applied both when the file is created and to
+an existing log file:
+
+~~~ruby
+# Owner read/write, group read, no access for others:
+SemanticLogger.add_appender(file_name: "production.log", permissions: 0o640)
+~~~
+
+### Dependencies and supply chain
+
+Semantic Logger has a single runtime dependency, `concurrent-ruby`. Appenders for
+third-party services (Kafka, MongoDB, Sentry, etc.) keep their backing gem optional: it
+is required lazily inside the appender only when that appender is used, and is never
+added to the gemspec. This keeps the dependency surface of an application that only uses
+the built-in file and IO appenders very small.
+
+Auditing the resolved dependency versions for known advisories is the responsibility of
+the consuming application, which should run a tool such as
+[bundler-audit](https://github.com/rubysec/bundler-audit) against its own
+`Gemfile.lock`. Pinning dependencies with a committed lockfile, and verifying it in CI
+with `bundle install --frozen`, prevents an unexpected dependency from being pulled in.

data/docs/testing.md

@@ -0,0 +1,285 @@
+---
+layout: default
+---
+
+## Testing
+{:.no_toc}
+
+**Contents**
+
+* TOC
+{:toc}
+
+When testing application code, you often want to assert that it logged the right thing: the expected
+message, level, payload, or metric. Semantic Logger logs through a global, asynchronous pipeline, so
+you cannot simply read it back from a normal appender. Instead it ships a test helper that **captures
+log events in memory** during a block, so you can make assertions on them.
+
+The events are captured as raw `SemanticLogger::Log` objects, before any appender or formatter runs,
+so your assertions are not affected by how logging happens to be configured.
+
+Helpers are provided for both Minitest and RSpec. Other frameworks are covered further down.
+
+## Minitest
+
+### Step 1: install the helpers
+
+Add the helper methods to your tests, once, in `test_helper.rb`:
+
+~~~ruby
+Minitest::Test.include SemanticLogger::Test::Minitest
+~~~
+
+### Step 2: capture events
+
+Wrap the code under test in `semantic_logger_events`. It returns every log event created during the
+block:
+
+~~~ruby
+messages = semantic_logger_events do
+  User.new.enable!
+end
+~~~
+
+By default it captures events from every class. To capture only the events from one class, pass that
+class:
+
+~~~ruby
+messages = semantic_logger_events(ApiClient) do
+  # Only ApiClient log events created during this block are captured.
+end
+~~~
+
+### Step 3: assert on the events
+
+Check how many events were produced, then assert on each one with `assert_semantic_logger_event`:
+
+~~~ruby
+require_relative "test_helper"
+
+class UserTest < ActiveSupport::TestCase
+  describe User do
+    it "logs message" do
+      messages = semantic_logger_events do
+        User.new.enable!
+      end
+
+      # How many events were logged
+      assert_equal 2, messages.count, messages
+
+      # The first event
+      assert_semantic_logger_event(
+        messages[0],
+        level:   :info,
+        message: "User enabled"
+      )
+
+      # The second event
+      assert_semantic_logger_event(
+        messages[1],
+        level:   :debug,
+        message: "Completed"
+      )
+    end
+  end
+end
+~~~
+
+Every argument other than the event itself is optional, so you assert only on what matters to the
+test. The available checks are:
+
+| Argument | Checks |
+|----------|--------|
+| `message` | Exact match of the text message. |
+| `message_includes` | Partial (substring) match of the message. |
+| `level` | Log level: `:trace`, `:debug`, `:info`, `:warn`, `:error`, `:fatal`. |
+| `payload` | Exact match of the payload Hash. |
+| `payload_includes` | Partial match: the given keys/values, ignoring any others. |
+| `tags` | Tags active on the thread when logged. |
+| `named_tags` | Named tags active on the thread when logged. |
+| `name` | Class name of the logger. |
+| `exception` | The Ruby exception that was logged. |
+| `metric` | The metric name. |
+| `metric_amount` | The metric amount. |
+| `dimensions` | The metric dimensions. |
+| `context` | Named contexts captured with the entry. |
+| `time` | When the entry was created. |
+| `duration` | Duration of a measure call, in milliseconds. |
+| `backtrace` | The captured backtrace, if any. |
+| `thread_name` | Name of the thread that logged the entry. |
+
+### Match part of a message or payload
+
+For assertions that should not break on incidental detail, match partially.
+
+Use `message_includes` for a substring of the message:
+
+~~~ruby
+assert_semantic_logger_event(
+  messages[0],
+  level:            :info,
+  message_includes: "enabled"
+)
+~~~
+
+Use `payload_includes` to assert specific payload keys while ignoring any extras. Compare with
+`payload`, which must match the whole Hash exactly:
+
+~~~ruby
+# Exact: the payload must be exactly these keys and values
+assert_semantic_logger_event(
+  messages[0],
+  level:   :info,
+  message: "User enabled",
+  payload: {first_name: "Jack", last_name: "Jones"}
+)
+
+# Partial: first_name must be present, other keys are ignored
+assert_semantic_logger_event(
+  messages[0],
+  level:            :info,
+  message:          "User enabled",
+  payload_includes: {first_name: "Jack"}
+)
+~~~
+
+For more examples, see the
+[Rails Semantic Logger tests](https://github.com/reidmorrison/rails_semantic_logger/blob/main/test/active_record_test.rb).
+
+## RSpec
+
+The RSpec helpers mirror the Minitest ones: a capture helper plus matchers for asserting on the
+captured events.
+
+### Step 1: install the helpers
+
+Require the helpers and include them, once, in `spec_helper.rb`:
+
+~~~ruby
+require "semantic_logger/test/rspec"
+
+RSpec.configure do |config|
+  config.include SemanticLogger::Test::RSpec
+end
+~~~
+
+### Step 2: capture events
+
+Wrap the code under test in `capture_semantic_logger_events`. It returns every log event created
+during the block, regardless of the global default log level:
+
+~~~ruby
+events = capture_semantic_logger_events do
+  User.new.enable!
+end
+~~~
+
+By default it captures events from every class. To capture only the events from one class, pass that
+class:
+
+~~~ruby
+events = capture_semantic_logger_events(ApiClient) do
+  # Only ApiClient log events created during this block are captured.
+end
+~~~
+
+### Step 3: assert on the events
+
+Use `be_a_semantic_logger_event` to assert on a single captured event:
+
+~~~ruby
+RSpec.describe User do
+  it "logs message" do
+    events = capture_semantic_logger_events do
+      User.new.enable!
+    end
+
+    expect(events.count).to eq(2)
+
+    expect(events[0]).to be_a_semantic_logger_event(
+      level:   :info,
+      message: "User enabled"
+    )
+
+    expect(events[1]).to be_a_semantic_logger_event(
+      level:   :debug,
+      message: "Completed"
+    )
+  end
+end
+~~~
+
+Every argument is optional, so you assert only on what matters to the test. The available checks are
+the same as the [Minitest table above](#step-3-assert-on-the-events), plus `message_includes`,
+`payload_includes`, and `exception_includes` for partial matches:
+
+~~~ruby
+expect(events[0]).to be_a_semantic_logger_event(
+  level:            :info,
+  message_includes: "enabled",
+  payload_includes: {first_name: "Jack"}
+)
+~~~
+
+An expected value that is a Class matches by type, and `:nil` asserts the attribute is `nil`:
+
+~~~ruby
+expect(events[0]).to be_a_semantic_logger_event(time: Time, exception: :nil)
+~~~
+
+### Match against a list of events
+
+`a_semantic_logger_event` is a composable alias, so it works inside `include` and other matchers:
+
+~~~ruby
+expect(events).to include(
+  a_semantic_logger_event(message: "User enabled")
+)
+~~~
+
+### Assert directly on a block
+
+`log_semantic_logger_event` captures the events for you and passes if any of them match. Pass `on:`
+to capture only one class's events:
+
+~~~ruby
+expect { User.new.enable! }.to(
+  log_semantic_logger_event(level: :info, message: "User enabled")
+)
+
+expect { ApiClient.new.call }.to(
+  log_semantic_logger_event(on: ApiClient, metric: "ApiClient/call")
+)
+~~~
+
+## Other test frameworks
+
+The same approach works anywhere: replace the logger with a
+`SemanticLogger::Test::CaptureLogEvents` instance. It looks and behaves like a normal logger, but
+keeps the raw log events in memory instead of writing them, so your assertions are unaffected by the
+configured appenders or their formats.
+
+~~~ruby
+logger = SemanticLogger::Test::CaptureLogEvents.new
+~~~
+
+Stub it onto a single class to capture just that class's logging:
+
+~~~ruby
+User.stub(:logger, logger) do
+  # Capture all logging calls to the User logger.
+end
+~~~
+
+Or stub the global processor to capture logging from every class:
+
+~~~ruby
+SemanticLogger::Logger.stub(:processor, logger) do
+  # Capture all log events during the block.
+end
+~~~
+
+Either way, the captured events are available as `logger.events`.
+
+If you add helper methods for another framework like the Minitest ones, a pull request would be
+welcome.