rails_semantic_logger 4.15.0 → 4.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3bb41dd02df0dc0dc472a66f83a82146878942a309670d4fb10da64bfdae4bc4
-  data.tar.gz: c1463e8da359c87fffc4e7d4e56919609320473fa0702b6a7f9b9fec1bdc5050
+  metadata.gz: 7da18e2122c13f2b4325a6f3000dd74d22517608b3a32355ca3d2bf4e5f08789
+  data.tar.gz: b0503c3227cb0bfa5faa1c6f00b408968783c8d09092a960497ab9f004798875
 SHA512:
-  metadata.gz: 1604cdbeed7b9bda178517f7cd7f72e4721523cf222aa6b0a2dcce36fc77c62807641ca32aef349f91c9e5aa86d04e9202e15adff71e4b5e4e29798a3c9dde16
-  data.tar.gz: b0e79372c2111bb823c191595a600c8c4c218e6989249f66c0ec46d66826116abfe2f6c5c761d33ae23f1494e7a74b8299d70b037410570757195115b212dced
+  metadata.gz: 30446e6a7553868e37cb56550d0a3e5c15b137991609e2cbb7aee84c7c7f5a75abe0bdd7f5e6263418546645f7a0892472e042ee09e0764be9f0ff9d90311917
+  data.tar.gz: 051ff94e24b1d9a4231f01d4148a966333b8d7b57539f7fa15ab953ad0a2fc9c998b61413ae2350716763f323eafa6d7302893b0c7b691e165c2200f7a8f3ef1

data/README.md

@@ -3,12 +3,112 @@
 
 Rails Semantic Logger replaces the Rails default logger with [Semantic Logger](https://logger.rocketjob.io/)
 
+When any large Rails application is deployed to production one of the first steps is to move to centralized logging, so that logs can be viewed and searched from a central location.
+
+Centralized logging quickly falls apart when trying to consume the current human readable log files:
+- Log entries often span multiple lines, resulting in unrelated log lines in the centralized logging system. For example, stack traces.
+- Complex Regular Expressions are needed to parse the text lines and make them machine readable. For example to build queries, or alerts that are looking for specific elements in the message.
+- Writing searches, alerts, or dashboards based on text logs is incredibly brittle, since a small change to the text logged can often break the parsing of those logs.
+- Every log entry often has a completely different format, making it difficult to make consistent searches against the data.
+
+For these and many other reasons switching to structured logging, or logs in JSON format, in testing and production makes centralized logging incredibly powerful.
+
+For example, adding these lines to `config/application.rb` and removing any other log overrides from other environments, will switch automatically to structured logging when running inside Kubernetes:
+~~~ruby
+    # Setup structured logging
+    config.semantic_logger.application = "my_application"
+    config.semantic_logger.environment = ENV["STACK_NAME"] || Rails.env
+    config.log_level = ENV["LOG_LEVEL"] || :info
+
+    # Switch to JSON Logging output to stdout when running on Kubernetes
+    if ENV["LOG_TO_CONSOLE"] || ENV["KUBERNETES_SERVICE_HOST"]
+      config.rails_semantic_logger.add_file_appender = false
+      config.semantic_logger.add_appender(io: $stdout, formatter: :json)
+    end
+~~~
+
+Then configure the centralized logging system to tell it that the data is in JSON format, so that it will parse it for you into a hierarchy.
+
+For example, the following will instruct [Observe](https://www.observeinc.com/) to parse the JSON data and create machine readable data from it:
+~~~ruby
+interface "log", "log":log
+
+make_col event:parse_json(log)
+
+make_col
+   time:parse_isotime(event.timestamp),
+   application:string(event.application),
+   environment:string(event.environment),
+   duration:duration_ms(event.duration_ms),
+   level:string(event.level),
+   name:string(event.name),
+   message:string(event.message),
+   named_tags:event.named_tags,
+   payload:event.payload,
+   metric:string(event.metric),
+   metric_amount:float64(event.metric_amount),
+   tags:array(event.tags),
+   exception:event.exception,
+   host:string(event.host),
+   pid:int64(event.pid),
+   thread:string(event.thread),
+   file:string(event.file),
+   line:int64(event.line),
+   dimensions:event.dimensions,
+   backtrace:array(event.backtrace),
+   level_index:int64(event.level_index)
+
+set_valid_from(time)
+drop_col timestamp, log, event, stream
+rename_col timestamp:time
+~~~
+
+Now queries can be built to drill down into each of these fields, including `payload` which is a nested object.
+
+For example to find all failed Sidekiq job calls where the causing exception class name is `NoMethodError`:
+~~~ruby
+filter environment = "uat2"
+filter level = "error"
+filter metric = "sidekiq.job.perform"
+filter (string(exception.cause.name) = "NoMethodError")
+~~~
+
+Example: create a dashboard showing the duration of all successful Sidekiq jobs:
+~~~ruby
+filter environment = "production"
+filter level = "info"
+filter metric = "sidekiq.job.perform"
+timechart duration:avg(duration), group_by(name)
+~~~
+
+Example: create a dashboard showing the queue latency of all Sidekiq jobs. 
+The queue latency is the time between when the job was enqueued and when it was started:
+~~~ruby
+filter environment = "production"
+filter level = "info"
+filter metric = "sidekiq.queue.latency"
+timechart duration:avg(duration), group_by(name)
+~~~
+
 * http://github.com/reidmorrison/rails_semantic_logger
 
 ## Documentation
 
 For complete documentation see: https://logger.rocketjob.io/rails
 
+## Upgrading to Semantic Logger V4.16 - Sidekiq Metrics Support
+
+Rails Semantic Logger now supports Sidekiq metrics. 
+Below are the metrics that are now available when the JSON logging format is used:
+- `sidekiq.job.perform` - The duration of each Sidekiq job.
+- `sidekiq.queue.latency` - The time between when a Sidekiq job was enqueued and when it was started.
+
+## Upgrading to Semantic Logger v4.15 & V4.16 - Sidekiq Support
+
+Rails Semantic Logger introduces direct support for Sidekiq v4, v5, v6, and v7. 
+Please remove any previous custom patches or configurations to make Sidekiq work with Semantic Logger.
+To see the complete list of patches being made, and to contribute your own changes, see: [Sidekiq Patches](https://github.com/reidmorrison/rails_semantic_logger/blob/master/lib/rails_semantic_logger/extensions/sidekiq/sidekiq.rb)
+
 ## Upgrading to Semantic Logger v4.4
 
 With some forking frameworks it is necessary to call `reopen` after the fork. With v4.4 the
@@ -19,7 +119,18 @@ I.e. Please remove the following line if being called anywhere:
 SemanticLogger::Processor.instance.instance_variable_set(:@queue, Queue.new)
 ~~~
 
-## Supports
+## New Versions of Rails, etc.
+
+The primary purpose of the Rails Semantic Logger gem is to patch other gems, primarily Rails, to make them support structured logging though Semantic Logger.
+
+When new versions of Rails and other gems are published they often make changes to the internals, so the existing patches stop working.
+
+Rails Semantic Logger survives only when someone in the community upgrades to a newer Rails or other supported libraries, runs into problems, 
+and then contributes the fix back to the community by means of a pull request.
+
+Additionally, when new popular gems come out, we rely only the community to supply the necessary patches in Rails Semantic Logger to make those gems support structured logging.
+
+## Supported Platforms
 
 For the complete list of supported Ruby and Rails versions, see the [Testing file](https://github.com/reidmorrison/rails_semantic_logger/blob/master/.github/workflows/ci.yml).
 

data/lib/rails_semantic_logger/extensions/sidekiq/sidekiq.rb

@@ -33,18 +33,26 @@ module Sidekiq
   if defined?(::Sidekiq::JobLogger)
     # Let Semantic Logger handle duration logging
     class JobLogger
-      def call(item, queue)
+      def call(item, queue, &block)
         klass  = item["wrapped"] || item["class"]
-        metric = "Sidekiq/#{klass}/perform" if klass
         logger = klass ? SemanticLogger[klass] : Sidekiq.logger
-        logger.info("Start #perform")
-        logger.measure_info(
-          "Completed #perform",
-          on_exception_level: :error,
-          log_exception:      :full,
-          metric:             metric
-        ) do
-          yield
+
+        SemanticLogger.tagged(queue: queue) do
+          # Latency is the time between when the job was enqueued and when it started executing.
+          logger.info(
+            "Start #perform",
+            metric:        "sidekiq.queue.latency",
+            metric_amount: job_latency_ms(item)
+          )
+
+          # Measure the duration of running the job
+          logger.measure_info(
+            "Completed #perform",
+            on_exception_level: :error,
+            log_exception:      :full,
+            metric:             "sidekiq.job.perform",
+            &block
+          )
         end
       end
 
@@ -60,14 +68,18 @@ module Sidekiq
       end
 
       def job_hash_context(job_hash)
-        h        = {
-          class: job_hash["display_class"] || job_hash["wrapped"] || job_hash["class"],
-          jid:   job_hash["jid"]
-        }
-        h[:bid]  = job_hash["bid"] if job_hash["bid"]
-        h[:tags] = job_hash["tags"] if job_hash["tags"]
+        h         = {jid: job_hash["jid"]}
+        h[:bid]   = job_hash["bid"] if job_hash["bid"]
+        h[:tags]  = job_hash["tags"] if job_hash["tags"]
+        h[:queue] = job_hash["queue"] if job_hash["queue"]
         h
       end
+
+      def job_latency_ms(job)
+        return unless job && job["enqueued_at"]
+
+        (Time.now.to_f - job["enqueued_at"].to_f) * 1000
+      end
     end
   end
 
@@ -80,48 +92,47 @@ module Sidekiq
       end
 
       def self.job_hash_context(job_hash)
-        klass       = job_hash["wrapped"] || job_hash["class"]
-        event       = { class: klass, jid: job_hash["jid"] }
-        event[:bid] = job_hash["bid"] if job_hash["bid"]
-        event
+        h         = {jid: job_hash["jid"]}
+        h[:bid]   = job_hash["bid"] if job_hash["bid"]
+        h[:queue] = job_hash["queue"] if job_hash["queue"]
+        h
       end
     end
   end
 
   # Exception is already logged by Semantic Logger during the perform call
-  # Sidekiq <= v6.5
   if defined?(::Sidekiq::ExceptionHandler)
+    # Sidekiq <= v6.5
     module ExceptionHandler
       class Logger
-        def call(ex, ctx)
-          unless ctx.empty?
-            job_hash = ctx[:job] || {}
-            klass = job_hash["display_class"] || job_hash["wrapped"] || job_hash["class"]
-            logger = klass ? SemanticLogger[klass] : Sidekiq.logger
-            ctx[:context] ? logger.warn(ctx[:context], ctx) : logger.warn(ctx)
-          end
+        def call(_exception, ctx)
+          return if ctx.empty?
+
+          job_hash = ctx[:job] || {}
+          klass    = job_hash["display_class"] || job_hash["wrapped"] || job_hash["class"]
+          logger   = klass ? SemanticLogger[klass] : Sidekiq.logger
+          ctx[:context] ? logger.warn(ctx[:context], ctx) : logger.warn(ctx)
         end
       end
     end
-  # Sidekiq >= v7
   elsif defined?(::Sidekiq::Config)
+    # Sidekiq >= v7
     class Config
       remove_const :ERROR_HANDLER
 
-      ERROR_HANDLER = ->(ex, ctx, cfg = Sidekiq.default_configuration) {
+      ERROR_HANDLER = ->(ex, ctx, cfg = Sidekiq.default_configuration) do
         unless ctx.empty?
           job_hash = ctx[:job] || {}
-          klass = job_hash["display_class"] || job_hash["wrapped"] || job_hash["class"]
-          logger = klass ? SemanticLogger[klass] : Sidekiq.logger
+          klass    = job_hash["display_class"] || job_hash["wrapped"] || job_hash["class"]
+          logger   = klass ? SemanticLogger[klass] : Sidekiq.logger
           ctx[:context] ? logger.warn(ctx[:context], ctx) : logger.warn(ctx)
         end
-      }
+      end
     end
   else
     # Sidekiq >= 6.5
-    # TODO: Not taking effect. See test/sidekiq_test.rb
-    def self.default_error_handler(ex, ctx)
-      binding.irb
+    Sidekiq.error_handlers.delete(Sidekiq::DEFAULT_ERROR_HANDLER)
+    Sidekiq.error_handlers << ->(ex, ctx) do
       unless ctx.empty?
         job_hash = ctx[:job] || {}
         klass = job_hash["display_class"] || job_hash["wrapped"] || job_hash["class"]
@@ -132,10 +143,13 @@ module Sidekiq
   end
 
   # Logging within each worker should use its own logger
-  if Sidekiq::VERSION.to_i == 4
+  case Sidekiq::VERSION.to_i
+  when 4
     module Worker
       def self.included(base)
-        raise ArgumentError, "You cannot include Sidekiq::Worker in an ActiveJob: #{base.name}" if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+        if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+          raise ArgumentError, "You cannot include Sidekiq::Worker in an ActiveJob: #{base.name}"
+        end
 
         base.extend(ClassMethods)
         base.include(SemanticLogger::Loggable)
@@ -144,10 +158,12 @@ module Sidekiq
         base.class_attribute :sidekiq_retries_exhausted_block
       end
     end
-  elsif Sidekiq::VERSION.to_i == 5
+  when 5
     module Worker
       def self.included(base)
-        raise ArgumentError, "You cannot include Sidekiq::Worker in an ActiveJob: #{base.name}" if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+        if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+          raise ArgumentError, "You cannot include Sidekiq::Worker in an ActiveJob: #{base.name}"
+        end
 
         base.extend(ClassMethods)
         base.include(SemanticLogger::Loggable)
@@ -156,10 +172,12 @@ module Sidekiq
         base.sidekiq_class_attribute :sidekiq_retries_exhausted_block
       end
     end
-  elsif Sidekiq::VERSION.to_i == 6
+  when 6
     module Worker
       def self.included(base)
-        raise ArgumentError, "Sidekiq::Worker cannot be included in an ActiveJob: #{base.name}" if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+        if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+          raise ArgumentError, "Sidekiq::Worker cannot be included in an ActiveJob: #{base.name}"
+        end
 
         base.include(Options)
         base.extend(ClassMethods)
@@ -169,7 +187,9 @@ module Sidekiq
   else
     module Job
       def self.included(base)
-        raise ArgumentError, "Sidekiq::Job cannot be included in an ActiveJob: #{base.name}" if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+        if base.ancestors.any? { |c| c.name == "ActiveJob::Base" }
+          raise ArgumentError, "Sidekiq::Job cannot be included in an ActiveJob: #{base.name}"
+        end
 
         base.include(Options)
         base.extend(ClassMethods)
@@ -178,14 +198,15 @@ module Sidekiq
     end
   end
 
-  if Sidekiq::VERSION.to_i == 4
+  if defined?(::Sidekiq::Middleware::Server::Logging)
+    # Sidekiq v4
     # Convert string to machine readable format
     class Processor
       def log_context(job_hash)
-        klass       = job_hash["wrapped"] || job_hash["class"]
-        event       = { class: klass, jid: job_hash["jid"] }
-        event[:bid] = job_hash["bid"] if job_hash["bid"]
-        event
+        h         = {jid: job_hash["jid"]}
+        h[:bid]   = job_hash["bid"] if job_hash["bid"]
+        h[:queue] = job_hash["queue"] if job_hash["queue"]
+        h
       end
     end
 
@@ -194,16 +215,26 @@ module Sidekiq
       module Server
         class Logging
           def call(worker, item, queue)
-            worker.logger.info("Start #perform")
-            worker.logger.measure_info(
-              "Completed #perform",
-              on_exception_level: :error,
-              log_exception:      :full,
-              metric:             "Sidekiq/#{worker.class.name}/perform"
-            ) do
-              yield
+            SemanticLogger.tagged(queue: queue) do
+              worker.logger.info(
+                "Start #perform",
+                metric:        "sidekiq.queue.latency",
+                metric_amount: job_latency_ms(item)
+              )
+              worker.logger.measure_info(
+                "Completed #perform",
+                on_exception_level: :error,
+                log_exception:      :full,
+                metric:             "sidekiq.job.perform"
+              ) { yield }
             end
           end
+
+          def job_latency_ms(job)
+            return unless job && job["enqueued_at"]
+
+            (Time.now.to_f - job["enqueued_at"].to_f) * 1000
+          end
         end
       end
     end

data/lib/rails_semantic_logger/version.rb

@@ -1,3 +1,3 @@
 module RailsSemanticLogger
-  VERSION = "4.15.0".freeze
+  VERSION = "4.16.0".freeze
 end

data/lib/rails_semantic_logger.rb

@@ -69,7 +69,7 @@ require("rails_semantic_logger/extensions/active_support/logger") if defined?(Ac
 require("rails_semantic_logger/extensions/active_support/log_subscriber") if defined?(ActiveSupport::LogSubscriber)
 
 begin
-  require 'rackup'
+  require "rackup"
 rescue LoadError
   # No need to do anything, will fall back to Rack
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rails_semantic_logger
 version: !ruby/object:Gem::Version
-  version: 4.15.0
+  version: 4.16.0
 platform: ruby
 authors:
 - Reid Morrison
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-06-27 00:00:00.000000000 Z
+date: 2024-07-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rack
@@ -92,7 +92,7 @@ licenses:
 metadata:
   bug_tracker_uri: https://github.com/reidmorrison/rails_semantic_logger/issues
   documentation_uri: https://logger.rocketjob.io
-  source_code_uri: https://github.com/reidmorrison/rails_semantic_logger/tree/v4.15.0
+  source_code_uri: https://github.com/reidmorrison/rails_semantic_logger/tree/v4.16.0
   rubygems_mfa_required: 'true'
 post_install_message:
 rdoc_options: []