semantic_logger 4.18.0 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 98c3a3e01f125e81eb58559ca0f7931c55bb787102d89d0c10dd9c9a7ea2e221
-  data.tar.gz: 8d82c20b77fe2c98773b72ab2483bdfb2d1267c65f9b8102cb2ffba00c141ee9
+  metadata.gz: ff43a3438164b6bf70367a7b2806350e65da8762fc6fb4e333a769cdf200e95b
+  data.tar.gz: 3d12647aaed6e22b7abc7a4c377c0d0470f00ffee3401d2db6043aece3ce09c1
 SHA512:
-  metadata.gz: c014af737549553d82d1d65ac1f46fbb09cefae9cb5aca3d5c39a4bbc180587fea4a65fa6756cad87eb031af52806693ddf11b9c2a4564bb140b1b1261b216c7
-  data.tar.gz: f0c050cb59d1a3022fbf2b5ca180b644493c8630103e09498e7a9097bbda97df3cb3a28b84f125331196819123cad128f85a142c7952918d9df376acfd1dff69
+  metadata.gz: 2893b8e69a4d246c1346e5fc6b20ca6a8fcdc3a29e78a44e4e1c8dae5466971298d9cfcabd736080beef17c2eb68914cb90ddef844bf6afd3ede388850232a90
+  data.tar.gz: 944f2cbf7232a4cae3ecee1f67d770d4b6a9306fae810b6045710c6920cc5876a5d56246d0a2605c7e066722df4abba1d8dccc29b43964c677a1bd33296dac73

data/README.md

@@ -1,13 +1,41 @@
 # Semantic Logger
 [![Gem Version](https://img.shields.io/gem/v/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![Build Status](https://github.com/reidmorrison/semantic_logger/workflows/build/badge.svg)](https://github.com/reidmorrison/semantic_logger/actions?query=workflow%3Abuild) [![Downloads](https://img.shields.io/gem/dt/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
 
-Semantic Logger is a feature rich logging framework, and replacement for existing Ruby & Rails loggers.
+Semantic Logger is a high-performance, asynchronous structured logging framework for Ruby
+and Rails.
 
-* https://logger.rocketjob.io/
+It differs from ordinary loggers in two important ways:
+
+1. **It logs structured data, not just strings.** Along with the text message, each log entry can
+   carry a payload (any Hash), an exception, a duration, metrics, and tags. That data is preserved
+   all the way to the destination, so it stays searchable instead of being flattened into text.
+2. **It logs asynchronously.** Log events are pushed onto an in-memory queue and written to their
+   destinations by a separate background thread, so your application is not blocked while logs are
+   written. Semantic Logger can log thousands of lines per second without slowing the application
+   down.
+
+```ruby
+require "semantic_logger"
+
+SemanticLogger.default_level = :info
+SemanticLogger.add_appender(io: $stdout, formatter: :color)
+
+logger = SemanticLogger["MyApp"]
+
+# A plain message, plus structured data that stays searchable
+logger.info("Queried users table", duration: 54, result: :ok, table: "users")
+```
+
+When running Rails, use
+[rails_semantic_logger](https://github.com/reidmorrison/rails_semantic_logger) instead, since it
+replaces the Rails default logger with Semantic Logger automatically.
 
 ## Documentation
 
-[Semantic Logger Guide](https://logger.rocketjob.io/)
+Start with the [Introduction](https://logger.rocketjob.io/), then the
+[Programmer's Guide](https://logger.rocketjob.io/api.html).
+
+* Full guide: [https://logger.rocketjob.io/](https://logger.rocketjob.io/)
 
 ## Logging Destinations
 
@@ -28,6 +56,7 @@ Logging to the following destinations are all supported "out-of-the-box":
 * UDP
 * Syslog
 * CloudWatch Logs
+* OpenTelemetry
 * Add any existing Ruby logger as another destination.
 * Roll-your-own
 
@@ -63,89 +92,16 @@ and are therefore not automatically included by this gem:
 - Syslog Appender to a remote syslogng server over TCP or UDP: gem 'net_tcp_client'
 - Splunk Appender: gem 'splunk-sdk-ruby'
 - Elasticsearch Appender: gem 'elasticsearch'
+- OpenSearch Appender: gem 'opensearch-ruby'
 - Kafka Appender: gem 'ruby-kafka'
 - Legacy Sentry Appender: gem 'sentry-raven' (deprecated)
 - Sentry Appender: gem 'sentry-ruby'
+- OpenTelemetry Appender: gem 'opentelemetry-logs-sdk' (plus an exporter, e.g. 'opentelemetry-exporter-otlp-logs')
 
-## Upgrading to Semantic Logger v4.9
+## Upgrading
 
-These changes should not be noticeable by the majority of users of Semantic Logger, since
-they are to the internal API. It is possible that advanced users may be using these internal
-API's directly.
-
-This does not affect any calls to the public api `SemanticLogger.add_appender`.
-
-File and IO are now separate appenders. When creating the File appender explicitly, its arguments
-have changed. For example, when requesting an IO stream, it needs to be changed from:
-
-~~~ruby
-SemanticLogger::Appender::File.new(io: $stderr)
-~~~
-to:
-~~~ruby
-SemanticLogger::Appender::IO.new($stderr)
-~~~
-
-Additionally, this needs to be changed from:
-~~~ruby
-SemanticLogger::Appender::File.new(file_name: "file.log")
-~~~
-to:
-~~~ruby
-SemanticLogger::Appender::File.new("file.log")
-~~~
-
-Rails Semantic Logger, if used, needs to be upgraded to v4.9 when upgrading to Semantic Logger v4.9.
-
-## Upgrading to Semantic Logger v4.4
-
-With some forking frameworks it is necessary to call `reopen` after the fork. With v4.4 the
-workaround for Ruby 2.5 crashes is no longer needed.
-I.e. Please remove the following line if being called anywhere:
-
-~~~ruby
-SemanticLogger::Processor.instance.instance_variable_set(:@queue, Queue.new)
-~~~
-
-## Upgrading to Semantic Logger v4.0
-
-The following changes need to be made when upgrading to V4:
-- Ruby V2.3 / JRuby V9.1 is now the minimum runtime version.
-- Replace calls to Logger#with_payload with SemanticLogger.named_tagged.
-- Replace calls to Logger#payload with SemanticLogger.named_tags.
-- MongoDB Appender requires Mongo Ruby Client V2 or greater.
-- Appenders now write payload data in a seperate :payload tag instead of mixing them
-  directly into the root elements to avoid name clashes.
-
-As a result any calls like the following:
-
-~~~ruby
-logger.debug foo: 'foo', bar: 'bar'
-~~~
-
-Must be replaced with the following in v4:
-
-~~~ruby
-logger.debug payload: {foo: 'foo', bar: 'bar'}
-~~~
-
-Similarly, for measure blocks:
-
-~~~ruby
-logger.measure_info('How long is the sleep', foo: 'foo', bar: 'bar') { sleep 1 }
-~~~
-
-Must be replaced with the following in v4:
-
-~~~ruby
-logger.measure_info('How long is the sleep', payload: {foo: 'foo', bar: 'bar'}) { sleep 1 }
-~~~
-
-The common log call has not changed, and the payload is still logged directly:
-
-~~~ruby
-logger.debug('log this', foo: 'foo', bar: 'bar')
-~~~
+See the [Upgrading Guide](https://logger.rocketjob.io/upgrading.html) for instructions on
+upgrading between major versions.
 
 ## Install
 
@@ -165,6 +121,11 @@ SemanticLogger.add_appender(file_name: 'development.log', formatter: :color)
 
 If running rails, see: [Semantic Logger Rails](https://logger.rocketjob.io/rails.html)
 
+## Contributing
+
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to set up the project, run the
+tests, and an overview of the architecture, including a class diagram.
+
 ## Author
 
 [Reid Morrison](https://github.com/reidmorrison)

data/Rakefile

@@ -9,8 +9,8 @@ task :gem do
 end
 
 task publish: :gem do
-  # system "git tag -a v#{SemanticLogger::VERSION} -m 'Tagging #{SemanticLogger::VERSION}'"
-  # system "git push --tags"
+  system "git tag -a v#{SemanticLogger::VERSION} -m 'Tagging #{SemanticLogger::VERSION}'"
+  system "git push --tags"
   system "gem push semantic_logger-#{SemanticLogger::VERSION}.gem"
   system "rm semantic_logger-#{SemanticLogger::VERSION}.gem"
 end
@@ -21,4 +21,11 @@ Rake::TestTask.new(:test) do |t|
   t.warning = false
 end
 
-task default: :test
+begin
+  require "rspec/core/rake_task"
+  RSpec::Core::RakeTask.new(:spec)
+rescue LoadError
+  # RSpec is only available in the test/development environment.
+end
+
+task default: %i[test spec]

data/lib/semantic_logger/appender/async.rb

@@ -2,28 +2,39 @@ require "forwardable"
 
 module SemanticLogger
   module Appender
-    # Allow any appender to run asynchronously in a separate thread.
+    # Proxy that allows any appender to run asynchronously in a separate thread.
+    #
+    # The worker thread and the in-memory queue are owned by an internal
+    # SemanticLogger::QueueProcessor. This proxy forwards log/flush/close/reopen on to the
+    # processor, and forwards the remaining appender methods (name, level, ...) directly to
+    # the wrapped appender so they keep executing on the caller's thread.
     class Async
       extend Forwardable
 
-      attr_accessor :lag_check_interval, :lag_threshold_s
-      attr_reader :queue, :appender, :max_queue_size
+      attr_reader :appender, :processor
 
-      # Forward methods that can be called directly
-      def_delegator :@appender, :name
-      def_delegator :@appender, :should_log?
-      def_delegator :@appender, :filter
-      def_delegator :@appender, :host
-      def_delegator :@appender, :application
-      def_delegator :@appender, :environment
-      def_delegator :@appender, :level
-      def_delegator :@appender, :level=
-      def_delegator :@appender, :logger
-      def_delegator :@appender, :logger=
+      # Methods forwarded directly to the wrapped appender (run on the caller's thread).
+      def_delegators :@appender,
+                     :name, :should_log?, :filter, :host, :application, :environment,
+                     :level, :level=, :logger, :logger=, :console_stream, :console_output?
+
+      # Methods forwarded to the queue processor that owns the thread and queue.
+      # Tuning options (max_queue_size, batch_size, non_blocking, ...) are set once at
+      # construction via SemanticLogger.add_appender, so only the values that callers actually
+      # read back are re-exposed here. lag_* backs the public SemanticLogger.lag_* API.
+      def_delegators :@processor,
+                     :log, :flush, :close, :thread, :active?, :queue, :max_queue_size,
+                     :capped?, :non_blocking?, :batch?,
+                     :lag_check_interval, :lag_check_interval=,
+                     :lag_threshold_s, :lag_threshold_s=,
+                     :processed_count, :dropped_count
 
       # Appender proxy to allow an existing appender to run asynchronously in a separate thread.
       #
       # Parameters:
+      #   appender: [SemanticLogger::Subscriber]
+      #     The appender to log to in a separate thread.
+      #
       #   max_queue_size: [Integer]
       #     The maximum number of log messages to hold on the queue before blocking attempts to add to the queue.
       #     -1: The queue size is uncapped and will never block no matter how long the queue is.
@@ -36,172 +47,74 @@ module SemanticLogger
       #   lag_check_interval: [Integer]
       #     Number of messages to process before checking for slow logging.
       #     Default: 1,000
-      def initialize(appender:,
-                     max_queue_size: 10_000,
-                     lag_check_interval: 1_000,
-                     lag_threshold_s: 30)
-        @appender           = appender
-        @lag_check_interval = lag_check_interval
-        @lag_threshold_s    = lag_threshold_s
-        @thread             = nil
-        @max_queue_size     = max_queue_size
-        create_queue
-        thread
+      #     Note: Not applicable when batch: true.
+      #
+      #   batch: [true|false]
+      #     Process log messages in batches via the appender's #batch method, instead of one at a time.
+      #     The appender must implement #batch.
+      #     Default: false
+      #
+      #   batch_size: [Integer]
+      #     Maximum number of messages to batch up before sending.
+      #     Default: 300
+      #     Note: Only applicable when batch: true.
+      #
+      #   batch_seconds: [Integer]
+      #     Maximum number of seconds between sending batches.
+      #     Default: 5
+      #     Note: Only applicable when batch: true.
+      #
+      #   non_blocking: [true|false]
+      #     Whether to drop log messages instead of blocking the calling thread when the queue is full.
+      #     When true and the queue is capped, attempts to add to a full queue return immediately and
+      #     the message is dropped. The number of dropped messages is logged to the internal logger
+      #     periodically (see dropped_message_report_seconds). Only applies to a capped queue.
+      #     Default: false
+      #
+      #   dropped_message_report_seconds: [Integer]
+      #     When non_blocking is enabled, log the count of dropped messages to the internal logger
+      #     at most once every this number of seconds.
+      #     Default: 30
+      #
+      #   async_max_retries: [Integer]
+      #     Maximum number of consecutive times to restart the worker thread (with a back-off)
+      #     after it raises an exception while processing messages, before giving up and stopping
+      #     the thread. The counter resets after any message is processed successfully.
+      #     -1: Retry indefinitely and never stop the thread (the pre-v5 behaviour).
+      #     Default: 100
+      def initialize(appender:, **args)
+        @appender  = appender
+        @processor = QueueProcessor.start(appender: appender, **args)
       end
 
-      # Re-open appender after a fork
+      # Re-open appender after a fork.
       def reopen
-        # Workaround CRuby crash on fork by recreating queue on reopen
-        #   https://github.com/reidmorrison/semantic_logger/issues/103
-        @queue&.close
-        create_queue
-
         appender.reopen if appender.respond_to?(:reopen)
-
-        @thread&.kill if @thread&.alive?
-        @thread = Thread.new { process }
-      end
-
-      # Returns [true|false] if the queue has a capped size.
-      def capped?
-        @capped
+        processor.reopen
       end
 
-      # Returns [Thread] the worker thread.
+      # Returns [Hash] operational statistics for this appender.
       #
-      # Starts the worker thread if not running.
-      def thread
-        return @thread if @thread&.alive?
-
-        @thread = Thread.new { process }
-      end
-
-      # Returns true if the worker thread is active
-      def active?
-        @thread&.alive?
-      end
-
-      # Add log message for processing.
-      def log(log)
-        queue << log
-      end
-
-      # Flush all queued log entries disk, database, etc.
-      #  All queued log messages are written and then each appender is flushed in turn.
-      def flush
-        submit_request(:flush)
-      end
-
-      # Close all appenders and flush any outstanding messages.
-      def close
-        # TODO: Prevent new close requests once this appender has been closed.
-        submit_request(:close)
-      end
-
-      private
-
-      def create_queue
-        if max_queue_size == -1
-          @queue  = Queue.new
-          @capped = false
-        else
-          @queue  = SizedQueue.new(max_queue_size)
-          @capped = true
-        end
-      end
-
-      # Separate thread for batching up log messages before writing.
-      def process
-        # This thread is designed to never go down unless the main thread terminates
-        # or the appender is closed.
-        Thread.current.name = logger.name
-        logger.trace "Async: Appender thread active"
-        begin
-          process_messages
-        rescue StandardError => e
-          # This block may be called after the file handles have been released by Ruby
-          begin
-            logger.error("Async: Restarting due to exception", e)
-          rescue StandardError
-            nil
-          end
-          retry
-        rescue Exception => e
-          # This block may be called after the file handles have been released by Ruby
-          begin
-            logger.error("Async: Stopping due to fatal exception", e)
-          rescue StandardError
-            nil
-          end
-        ensure
-          @thread = nil
-          # This block may be called after the file handles have been released by Ruby
-          begin
-            logger.trace("Async: Thread has stopped")
-          rescue StandardError
-            nil
-          end
-        end
-      end
-
-      def process_messages
-        count = 0
-        while (message = queue.pop)
-          if message.is_a?(Log)
-            appender.log(message)
-            count += 1
-            # Check every few log messages whether this appender thread is falling behind
-            if count > lag_check_interval
-              check_lag(message)
-              count = 0
-            end
-          else
-            break unless process_message(message)
-          end
-        end
-        logger.trace "Async: Queue Closed"
-      end
-
-      # Returns false when message processing should be stopped
-      def process_message(message)
-        case message[:command]
-        when :flush
-          appender.flush
-          message[:reply_queue] << true if message[:reply_queue]
-        when :close
-          appender.close
-          message[:reply_queue] << true if message[:reply_queue]
-          return false
-        else
-          logger.warn "Async: Appender thread: Ignoring unknown command: #{message[:command]}"
-        end
-        true
-      end
-
-      def check_lag(log)
-        diff = Time.now - log.time
-        return unless diff > lag_threshold_s
-
-        logger.warn "Async: Appender thread has fallen behind by #{diff} seconds with #{queue.size} messages queued up. Consider reducing the log level or changing the appenders"
-      end
-
-      # Submit command and wait for reply
-      def submit_request(command)
-        return false unless active?
-
-        queue_size = queue.size
-        msg        = "Async: Queued log messages: #{queue_size}, running command: #{command}"
-        if queue_size > 1_000
-          logger.warn msg
-        elsif queue_size > 100
-          logger.info msg
-        elsif queue_size.positive?
-          logger.trace msg
-        end
-
-        reply_queue = Queue.new
-        queue << {command: command, reply_queue: reply_queue}
-        reply_queue.pop
+      #   name:           [String]  Name of the wrapped appender.
+      #   async:          [true]    This appender logs asynchronously via a separate thread.
+      #   thread_active:  [Boolean] Whether the worker thread is currently running.
+      #   queue_size:     [Integer] Number of log messages currently waiting to be written.
+      #   capped:         [Boolean] Whether the queue has a maximum size.
+      #   max_queue_size: [Integer] Maximum queue size, or nil when uncapped.
+      #   processed:      [Integer] Cumulative number of log messages written since startup.
+      #   dropped:        [Integer] Cumulative number of log messages dropped because the queue
+      #                             was full (only possible when non_blocking is enabled).
+      def stats
+        {
+          name:           name,
+          async:          true,
+          thread_active:  active? || false,
+          queue_size:     queue.size,
+          capped:         capped?,
+          max_queue_size: capped? ? max_queue_size : nil,
+          processed:      processed_count,
+          dropped:        dropped_count
+        }
       end
     end
   end

data/lib/semantic_logger/appender/cloudwatch_logs.rb

@@ -54,7 +54,7 @@ module SemanticLogger
       #     Note that currently CloudWatch Logs has 10000 hard limit.
       #     Default: 4000
       def initialize(
-        *args,
+        *,
         group:,
         client_kwargs: {},
         stream: nil,
@@ -62,8 +62,8 @@ module SemanticLogger
         create_stream: true,
         force_flush_interval_seconds: 5,
         max_buffered_events: 4_000,
-        **kwargs,
-        &block
+        **,
+        &
       )
         @group = group
         @client_kwargs = client_kwargs
@@ -73,7 +73,7 @@ module SemanticLogger
         @force_flush_interval_seconds = force_flush_interval_seconds
         @max_buffered_events = max_buffered_events
 
-        super(*args, **kwargs, &block)
+        super(*, **, &)
         reopen
       end