rails_semantic_logging 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 486b6add6a541fe35dfc1f09e09c6c028f4e9dfa1d62ac49e219d668f26b6a40
-  data.tar.gz: 85c08bdb85fb2372486966cc278b6d1d82e9c89879dc121da9d4a17ddd1f3086
+  metadata.gz: 45760af9983f2c22b841521dede71966f21d51b02ac68909adc334a9c59d1347
+  data.tar.gz: d143a49b3a2044e30ac92e20d0ea85d81c4634322526e1e03480e4b9f731e0e0
 SHA512:
-  metadata.gz: a5d4cae0b1752ba768a769c435a4f46c2bb43c3ac51b3e9bc4178b2e950dc39cb758091d7a6ca2e813d53a80e5932990d0225307d7bdf1f2bc41d60763ba2c95
-  data.tar.gz: f6640bcfffb3adc4c2773112b2e85b1da7238a8992bafc54b3ec5123b06ccb4373926f378be43851836f1578e228aeead1377d129dd8b1fb4d486d4771963024
+  metadata.gz: cd642159648180526467d511be9f232571e9f7fffcac3840c02706eb1e4f0a18143921d612d8a3cb356d04ed8b51127fa2a05d746791f3a2655eb1caa87a58a2
+  data.tar.gz: cf2ce998170d859f106e83b9b11648d1135a5015abd4a8790c67e4ce076bfaf685f1e3622eaaddf9a7127720afbc2378d01fcb8b320672ac4a4a39e9b3536c9e

data/README.md

@@ -7,7 +7,7 @@ Opinionated Rails semantic logger configuration with Datadog support. Provides a
 - **Datadog formatter** with [Standard Attributes](https://docs.datadoghq.com/standard-attributes/) mapping
 - **Default payload enrichment** for controllers (host, user_agent, referer) mapped to `http.*`
 - **JSON formatter** for structured logging without Datadog-specific fields
-- **ActiveJob integration** with named tags (`job_class`, `job_id`, `queue`) instead of array tags
+- **ActiveJob integration** with named tags (`job_class`, `job_id`, `queue`, `executions`, and `provider_job_id` when the adapter assigns one) instead of array tags
 - **Sidekiq integration** with job context in all log lines
 - **Configurable** via [anyway_config](https://github.com/palkan/anyway_config) (YAML, env vars, or code)
 - **Environment-aware** defaults (Datadog JSON in production, color in development, fatal in test)
@@ -130,6 +130,17 @@ When `default_payload` is enabled (default), controller request logs include:
 
 When Datadog tracing is active, the formatter injects: `dd.trace_id`, `dd.span_id`, `dd.env`, `dd.service`, `dd.version`.
 
+#### Routing error enrichment
+
+`rails_semantic_logger` logs unmatched routes as a real `ActionController::RoutingError`.
+When the formatter sees one, it parses the standard `No route matches [GET] "/foo"`
+message and populates `http.method`, `http.url_details.path` and `http.status_code`
+(the canonical `404`) on the log event. The exception message is also used as the
+top-level `message` so the log line is never empty. This keeps unmatched-route logs
+correlated with the original request URL in Datadog without requiring a custom
+server-side log pipeline. Actual request data (when present in the payload) always
+wins over the message-derived fields.
+
 ### Example: Complete request log (production)
 
 ```json
@@ -163,11 +174,21 @@ When Datadog tracing is active, the formatter injects: `dd.trace_id`, `dd.span_i
   "named_tags": {
     "job_class": "ImportBikesJob",
     "job_id": "abc-123",
-    "queue": "default"
+    "queue": "default",
+    "executions": 0,
+    "provider_job_id": "9632"
   }
 }
 ```
 
+`executions` is the ActiveJob attempt counter (0 on the first run), so retries
+are visible directly in the logs. `provider_job_id` is the queue backend's
+native identifier — the `solid_queue_jobs` row id under Solid Queue, the `jid`
+under Sidekiq — which lets a log line be correlated with the job in Mission
+Control and the backend's own tables. It is only present once the job has been
+enqueued, so synchronous (`perform_now` / `:inline` / `:test`) executions omit
+it.
+
 ### Example: Error log
 
 ```json
@@ -204,9 +225,9 @@ Plain structured JSON without Datadog-specific field mapping. Useful for non-Dat
 
 Uses the built-in `SemanticLogger::Formatters::Color` for human-readable development output.
 
-## RSpec Matcher
+## RSpec Matchers
 
-Include the matcher module in your spec config:
+Include the matchers module in your spec config:
 
 ```ruby
 require 'rails_semantic_logging/rspec/matchers'
@@ -216,7 +237,11 @@ RSpec.configure do |config|
 end
 ```
 
-Usage:
+Two matchers are provided. They cover complementary use cases.
+
+### `log_semantic` — declarative "was X logged?" assertion
+
+Returns a boolean: true if **any** captured log line matches all the supplied criteria. Use it when the question is "did this happen?".
 
 ```ruby
 expect { logger.info("hello") }.to log_semantic(level: :info, message: /hello/)
@@ -224,6 +249,34 @@ expect { logger.warn("oops", key: "val") }.to log_semantic(payload: { key: "val"
 expect { do_work }.to log_semantic(named_tags: { job_class: 'MyJob' })
 ```
 
+### `have_logged_message` — find a specific log and assert on it
+
+Finds the first log event whose message matches the expected pattern, then hands the raw `SemanticLogger::Log` (and, optionally, a formatted version) over to a user block so you can make arbitrary assertions on it.
+
+The expected message can be a String, a Regexp, or any object with `===` (so RSpec built-in matchers like `a_string_starting_with("...")` work too). An optional second argument constrains the level.
+
+```ruby
+expect { service.call }.to have_logged_message("Imported")
+expect { service.call }.to have_logged_message(/import/, :warn)
+
+expect { service.call }
+  .to have_logged_message("Imported").with_formatted_event { |event, formatted|
+    expect(event.payload).to include(items: 3)
+    expect(event.named_tags).to include(tenant: 'eu')
+    expect(formatted.dig('http', 'status_code')).to eq(200)
+  }
+```
+
+`with_formatted_event` defaults to the gem's Datadog formatter; pass any `SemanticLogger::Formatters::Base` subclass to use a different one:
+
+```ruby
+.with_formatted_event(SemanticLogger::Formatters::Json) { |event, formatted| ... }
+```
+
+`formatted` is the formatter's JSON output, re-parsed with indifferent access — so both `formatted[:status]` and `formatted['status']` work.
+
+The matcher passes a **duplicated** copy of the log event to the formatter, so a formatter that mutates `log.payload` / `log.named_tags` cannot leak those mutations into the event you receive in the block.
+
 ## Puma Integration
 
 When using Puma in clustered mode, reopen log appenders after forking:

data/VERSION

@@ -1 +1 @@
-0.1.2
+0.2.0

data/lib/rails_semantic_logging/datadog/log_injection.rb

@@ -1,7 +1,22 @@
-# Monkey patch for Datadog's ActiveJob LogInjection to use hash-style
-# correlation tags instead of string-style. This is necessary because
-# we patch ActiveJob::Logging to use named tags (job_class, job_id, queue),
-# so Datadog must also use hash-style tags for compatibility with SemanticLogger.
+# Monkey patches Datadog's ActiveJob LogInjection to use hash-style
+# correlation tags instead of string-style.
+#
+# Background: Datadog ships LogInjection sub-modules (PerformNowPatch for
+# Rails 6+, EnqueuePatch for the singleton enqueue path) and patches
+# ActiveJob::Base by `prepend`ing them. By default they call
+# `Datadog::Tracing.log_correlation`, which returns a flat string
+# ("dd.trace_id=... dd.span_id=...") and ends up under `event.tags`.
+#
+# Once we patch ActiveJob::Logging#tag_logger to emit named tags, we want
+# Datadog correlation to land in `named_tags` too, so traces and logs
+# correlate properly in the Datadog UI. We achieve that by re-defining
+# `perform_now` and `enqueue` on Datadog's own patch modules to call
+# `Datadog::Tracing.correlation.to_h` and pass the resulting hash to
+# `logger.tagged`.
+#
+# Because Datadog prepends those modules into ActiveJob::Base, our
+# re-definitions take effect on every method dispatch — independent of
+# load order, as long as `apply!` runs before any job is executed.
 
 module RailsSemanticLogging
   module Datadog
@@ -9,24 +24,37 @@ module RailsSemanticLogging
       def self.apply!
         return unless defined?(::Datadog::Tracing::Contrib::ActiveJob::LogInjection)
 
-        # Ensure the SemanticLogger ActiveJob extension is loaded first
+        # Ensure the SemanticLogger ActiveJob extension is loaded before we
+        # override Datadog's patches, so the named-tags `tag_logger` is in place.
         require 'rails_semantic_logger/extensions/active_job/logging'
 
-        # Replace the original module with our patched version
-        ::Datadog::Tracing::Contrib::ActiveJob.send(:remove_const, :LogInjection)
-        ::Datadog::Tracing::Contrib::ActiveJob.const_set(:LogInjection, PatchedLogInjection)
+        patch_perform_now
+        patch_enqueue
       end
 
-      # Replacement module that uses correlation.to_h instead of log_correlation
-      module PatchedLogInjection
-        def self.included(base)
-          base.class_eval do
-            around_perform do |_, block|
-              if ::Datadog.configuration.tracing.log_injection && logger.respond_to?(:tagged)
-                logger.tagged(::Datadog::Tracing.correlation.to_h, &block)
-              else
-                block.call
-              end
+      def self.patch_perform_now
+        return unless defined?(::Datadog::Tracing::Contrib::ActiveJob::LogInjection::PerformNowPatch)
+
+        ::Datadog::Tracing::Contrib::ActiveJob::LogInjection::PerformNowPatch.module_eval do
+          define_method(:perform_now) do
+            if ::Datadog.configuration.tracing.log_injection && logger.respond_to?(:tagged)
+              logger.tagged(::Datadog::Tracing.correlation.to_h) { super() }
+            else
+              super()
+            end
+          end
+        end
+      end
+
+      def self.patch_enqueue
+        return unless defined?(::Datadog::Tracing::Contrib::ActiveJob::LogInjection::EnqueuePatch)
+
+        ::Datadog::Tracing::Contrib::ActiveJob::LogInjection::EnqueuePatch.module_eval do
+          define_method(:enqueue) do |*args, **kwargs, &block|
+            if ::Datadog.configuration.tracing.log_injection && logger.respond_to?(:tagged)
+              logger.tagged(::Datadog::Tracing.correlation.to_h) { super(*args, **kwargs, &block) }
+            else
+              super(*args, **kwargs, &block)
             end
           end
         end

data/lib/rails_semantic_logging/formatters/datadog.rb

@@ -33,12 +33,27 @@ module RailsSemanticLogging
         user_role: :role
       }.freeze
 
+      # Matches ActionController::RoutingError messages like:
+      #   No route matches [GET] "/some/path"
+      # Used to extract http.method and http.url_details.path from the exception.
+      ROUTING_ERROR_MESSAGE = /\ANo route matches \[(?<method>\w+)\]\s+"(?<path>[^"]+)"/
+
       def initialize(time_format: :iso_8601, time_key: :timestamp, **args) # rubocop:disable Naming/VariableNumber
         super(time_format:, time_key:, log_application: false, log_host: true, log_environment: false, **args)
       end
 
       def call(log, logger)
         super
+        # SemanticLogger::Formatters::Raw assigns log.payload and log.named_tags
+        # to hash[:payload] / hash[:named_tags] BY REFERENCE. We mutate those
+        # below (remap_http_payload deletes :host/:method/etc., remap_named_tags
+        # deletes :request_id/:client_ip/:dd/etc., deep_compact_blank! drops
+        # every blank value). Without these dups the formatter strips keys off
+        # the underlying log event, which breaks any consumer that reads
+        # log.payload / log.named_tags after formatting (other appenders,
+        # RSpec matchers that re-inspect the captured event).
+        hash[:payload] = hash[:payload].dup if hash[:payload].is_a?(Hash)
+        hash[:named_tags] = hash[:named_tags].dup if hash[:named_tags].is_a?(Hash)
         remap_named_tags
         remap_http_payload
         parse_url_details
@@ -59,6 +74,16 @@ module RailsSemanticLogging
         hash[:status] = log.level
       end
 
+      # Fall back to the exception message when the log carries no message of
+      # its own. rails_semantic_logger logs unmatched routes (and other
+      # rescued exceptions) via ActionDispatch::DebugExceptions by passing only
+      # the exception object, so without this the Datadog `message` field would
+      # be empty for those events.
+      def message
+        super
+        hash[:message] ||= log.exception&.message
+      end
+
       def duration
         # Propagate duration from payload if not set on log
         log.duration = log.payload[:duration] if log.duration.nil? && log.payload&.dig(:duration)
@@ -78,10 +103,45 @@ module RailsSemanticLogging
           message: log.exception.message,
           stack: log.exception.backtrace&.join("\n")
         }
+
+        parse_routing_error
+      end
+
+      # SemanticLogger's :dimensions log attribute carries metric tags
+      # (e.g. `logger.info('Processed feed', metric: 'feed_size', metric_amount: 12,
+      # dimensions: { feed: 'amazon' })`). The Raw formatter sets :metric and
+      # :metric_amount but not :dimensions — surface it at the top level so the
+      # Datadog UI can filter on those tags directly without diving into named_tags.
+      def metric
+        super
+        hash[:dimensions] = log.dimensions if log.dimensions.respond_to?(:any?) && log.dimensions.any?
       end
 
       private
 
+      # Extracts http.method and http.url_details.path from
+      # ActionController::RoutingError messages so logs for unmatched routes
+      # are still correlated with the request URL in Datadog.
+      #
+      # rails_semantic_logger patches ActionDispatch::DebugExceptions to log the
+      # real exception object, so unmatched-route requests reach this formatter
+      # as a RoutingError carrying a message like: No route matches [GET] "/foo".
+      def parse_routing_error
+        # Fully qualified so the constant resolves to Rails' ActionController and
+        # not the gem's own RailsSemanticLogging::ActionController namespace.
+        return unless log.exception.is_a?(::ActionController::RoutingError)
+        return unless (match = ROUTING_ERROR_MESSAGE.match(log.exception.message))
+
+        hash[:http] ||= {}
+        hash[:http][:method] ||= match[:method]
+        # The DebugExceptions log carries no HTTP status (there is no completed
+        # request), so map the exception to its canonical status the same way
+        # rails_semantic_logger does (RoutingError => 404).
+        hash[:http][:status_code] ||= ::ActionDispatch::ExceptionWrapper.status_code_for_exception(log.exception.class.name)
+        url_details = hash[:http][:url_details] ||= {}
+        url_details[:path] ||= match[:path]
+      end
+
       # Parses http.url into url_details with host, path and queryString
       def parse_url_details
         return unless hash.dig(:http, :url)
@@ -176,14 +236,25 @@ module RailsSemanticLogging
         hash[:http].is_a?(Hash) ? hash[:http].merge!(http) : hash[:http] = http
       end
 
-      # Recursively removes blank values from a hash
+      # Recursively removes empty values from a hash. nil, "", [], {} are dropped
+      # to keep the log payload tidy, but `false` and `0` are kept because they
+      # are meaningful values in things like `bot: false` or `count: 0`.
+      # (ActiveSupport's `blank?` treats `false` as blank, which would silently
+      # strip boolean flags from payloads.)
       def deep_compact_blank!(h)
         h.each do |key, value|
           deep_compact_blank!(value) if value.is_a?(Hash)
-          h.delete(key) if value.blank?
+          h.delete(key) if blank_value?(value)
         end
         h
       end
+
+      def blank_value?(value)
+        return true if value.nil?
+        return value.empty? if value.respond_to?(:empty?)
+
+        false
+      end
     end
   end
 end

data/lib/rails_semantic_logging/job_logging/active_job_patch.rb

@@ -4,13 +4,23 @@ module RailsSemanticLogging
   module JobLogging
     # ActiveJob patch to provide named tags instead of array tags.
     # Converts the default tag_logger(class, id) call into named tags
-    # (job_class, job_id, queue) for structured logging.
+    # (job_class, job_id, queue, provider_job_id, executions) for
+    # structured logging.
+    #
+    # `provider_job_id` is the queue backend's native id (e.g. the
+    # solid_queue_jobs row id, or the Sidekiq jid) and lets a log line be
+    # correlated with the job in Mission Control / the solid_queue_* tables.
+    # It is only present once the job has been enqueued, so it is added
+    # conditionally. `executions` is the ActiveJob attempt counter, which
+    # makes retries visible in the logs.
     module ActiveJobPatch
       extend ActiveSupport::Concern
 
       def tag_logger(job_class = nil, job_id = nil, &)
         if job_class && job_id
-          super(job_class: job_class, job_id: job_id, queue: queue_name, &)
+          tags = { job_class: job_class, job_id: job_id, queue: queue_name, executions: executions }
+          tags[:provider_job_id] = provider_job_id if provider_job_id
+          super(**tags, &)
         else
           super(&)
         end

data/lib/rails_semantic_logging/rspec/matchers.rb

@@ -1,4 +1,6 @@
+require 'json'
 require 'semantic_logger'
+require_relative '../formatters/datadog'
 
 module RailsSemanticLogging
   module RSpec
@@ -111,10 +113,144 @@ module RailsSemanticLogging
       end
     end
 
+    # Block matcher that finds a specific log event by message and exposes both
+    # the raw SemanticLogger::Log and an optionally formatted version to a user
+    # block, so the test can make arbitrary assertions on the event.
+    #
+    # Complementary to `log_semantic`:
+    # * `log_semantic` is declarative ("was anything matching these criteria
+    #   logged?") and returns a boolean.
+    # * `have_logged_message` is imperative — it finds the event, hands it over
+    #   to the test, and lets the test assert whatever it needs.
+    #
+    # Usage:
+    #   expect { logger.info("hello", key: "val") }.to have_logged_message("hello")
+    #   expect { do_work }.to have_logged_message(/import/, :warn)
+    #
+    #   expect { service.call }
+    #     .to have_logged_message("Completed").with_formatted_event { |event, formatted|
+    #       expect(event.payload).to include(order_id: order.id)
+    #       expect(formatted.dig("http", "status_code")).to eq(200)
+    #     }
+    #
+    # The optional `with_formatted_event` block receives `(event, formatted)`
+    # where `formatted` is the JSON output of the chosen formatter parsed back
+    # with indifferent access. By default the gem's Datadog formatter is used;
+    # pass any `SemanticLogger::Formatters::Base` subclass to override:
+    #
+    #   .with_formatted_event(SemanticLogger::Formatters::Json) { |event, formatted| ... }
+    class HaveLoggedMessageMatcher
+      attr_reader :event_block
+
+      DEFAULT_FORMATTER = ::RailsSemanticLogging::Formatters::Datadog
+
+      def initialize(expected, expected_level = nil)
+        @expected_message = expected
+        @expected_level = expected_level
+        @formatter_class = DEFAULT_FORMATTER
+      end
+
+      def matches?(block)
+        # Capture the live appender BEFORE we attach our test appender. The
+        # Datadog formatter reads `logger.host` on the appender it receives, so
+        # the formatter call later on needs a real one (any subscriber will do).
+        @live_appender = ::SemanticLogger.appenders.first
+        @log_event = capture_first_matching_event(block)
+        return false unless @log_event
+        return false if @expected_level && @log_event.level != @expected_level
+
+        @event_block&.call(@log_event, formatted)
+        true
+      rescue ::RSpec::Expectations::ExpectationNotMetError => e
+        @error = e
+        false
+      end
+
+      def supports_block_expectations?
+        true
+      end
+
+      # Attach a block that receives the matched `(log_event, formatted)` pair.
+      # `formatted` is the JSON output of the configured formatter, parsed back
+      # with `HashWithIndifferentAccess` so callers can navigate the structure
+      # with either string or symbol keys.
+      def with_formatted_event(formatter_class = nil, &block)
+        raise ArgumentError, 'block is required' unless block
+        if formatter_class && !(formatter_class < ::SemanticLogger::Formatters::Base)
+          raise ArgumentError,
+                "formatter must inherit from SemanticLogger::Formatters::Base, got #{formatter_class.inspect}"
+        end
+
+        @formatter_class = formatter_class if formatter_class
+        @event_block = block
+        self
+      end
+
+      def failure_message
+        return @error.message if @error
+        if @log_event && @expected_level && @log_event.level != @expected_level
+          return "expected log level #{@expected_level.inspect}, got #{@log_event.level.inspect}"
+        end
+
+        "expected block to log a message matching #{@expected_message.inspect}"
+      end
+
+      def failure_message_when_negated
+        "expected block NOT to log a message matching #{@expected_message.inspect}, but a matching event was emitted"
+      end
+
+      private
+
+      def capture_first_matching_event(block)
+        capture = InMemoryAppender.new
+        with_capture_appender(capture) { block.call }
+        capture.logs.find { |evt| message_matches?(evt.message) }
+      end
+
+      def with_capture_appender(capture)
+        ::SemanticLogger.add_appender(appender: capture)
+        previous_level = ::SemanticLogger.default_level
+        ::SemanticLogger.default_level = :trace
+        yield
+        ::SemanticLogger.flush
+      ensure
+        ::SemanticLogger.default_level = previous_level
+        ::SemanticLogger.remove_appender(capture)
+      end
+
+      # Matches String, Regex and any object that implements `===` (incl. RSpec
+      # built-in matchers like `a_string_starting_with("...")`).
+      def message_matches?(actual)
+        @expected_message === actual.to_s # rubocop:disable Style/CaseEquality
+      end
+
+      # Format a *copy* of the log event so a buggy formatter that mutates
+      # log.payload / log.named_tags cannot leak back into the event we hand
+      # off to the user block.
+      def formatted
+        @formatted ||= begin
+          safe_log = duped_log_event
+          ::JSON.parse(@formatter_class.new.call(safe_log, @live_appender)).with_indifferent_access
+        end
+      end
+
+      def duped_log_event
+        copy = @log_event.dup
+        copy.payload = @log_event.payload.dup if @log_event.payload.is_a?(::Hash)
+        copy.named_tags = @log_event.named_tags.dup if @log_event.named_tags.is_a?(::Hash)
+        copy.tags = @log_event.tags.dup if @log_event.tags.is_a?(::Array)
+        copy
+      end
+    end
+
     module Matchers
       def log_semantic(expected = {})
         LogSemanticMatcher.new(expected)
       end
+
+      def have_logged_message(expected, level = nil) # rubocop:disable Naming/PredicatePrefix
+        HaveLoggedMessageMatcher.new(expected, level)
+      end
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_semantic_logging
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.2.0
 platform: ruby
 authors:
 - Fabio Napoleoni