semantic_logger 4.0.0 → 4.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b013a6597ef91c2f2b5da17fad3081736d000ae4
-  data.tar.gz: 8893e45e15e2ff4c6c397b5892c63d72acdc7faf
+  metadata.gz: b0ad46c624c9904d87ec5b38c7fdc28e12dcc62d
+  data.tar.gz: 92ce4e537d95a70422e9662001683c82a75ce7dd
 SHA512:
-  metadata.gz: 365fe77c43d97fe8923a299bde0113095853b0083610dea1df76977e68d095c673eb393ac79c3baf659d8ddb150ba6f091a92aa4b2325cde936026a7102a6eff
-  data.tar.gz: bddea8b37c3d837690b56c5734ba22d05e47877f4aaf0a830545a8e3156f0a3170c2fe3953df483d5a31044b394978d9f50f652e1aea8d955f2e69152379082c
+  metadata.gz: fb9845a97ecaed5b6e91f0a97e6fc61464411969af7798ebd04e336eea66e115df1447514a3fd80c3d1e36236f79071ebd8dfc2b9bdcf2af1100deeb2297d389
+  data.tar.gz: 23cd01a8709c925132f82ddf697aa8b29a57a35a595ad308ba772c948cef722b25d9aacbd6c89d56531f08dcfdd25727dc486223d867db7e868564e2d29e0f86

data/README.md

@@ -1,7 +1,7 @@
 # semantic_logger
 [![Gem Version](https://img.shields.io/gem/v/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![Build Status](https://travis-ci.org/rocketjob/semantic_logger.svg?branch=master)](https://travis-ci.org/rocketjob/semantic_logger) [![Downloads](https://img.shields.io/gem/dt/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) [![Gitter chat](https://img.shields.io/badge/IRC%20(gitter)-Support-brightgreen.svg)](https://gitter.im/rocketjob/support)
 
-Low latency, high throughput, enterprise-scale logging system for Ruby
+Low latency, high throughput, enterprise-scale logging system for Ruby.
 
 * http://github.com/rocketjob/semantic_logger
 
@@ -17,29 +17,44 @@ Logging to the following destinations are all supported "out-of-the-box":
 
 * File
 * Screen
-* MongoDB
+* ElasticSearch. (Use with Kibana for Dashboards and Visualizations)
+* Graylog
 * BugSnag
 * NewRelic
 * Splunk
+* MongoDB
+* Honeybadger
+* Sentry
+* HTTP
+* TCP
+* UDP
 * Syslog
+* Add any existing Ruby logger as another destination.
 * Roll-your-own
 
 Semantic Logger is capable of logging thousands of lines per second without slowing
 down the application. Traditional logging systems make the application wait while
 the log information is being saved. Semantic Logger avoids this slowdown by pushing
 log events to an in-memory queue that is serviced by a separate thread that only
-handles saving log information to multiple destinations at the same time.
+handles saving log information to multiple destinations / appenders.
 
 ## Rails
 
 When running Rails, use [rails_semantic_logger](http://github.com/rocketjob/rails_semantic_logger)
 instead of Semantic Logger directly since it will automatically replace the Rails default logger with Semantic Logger.
 
+## Rocket Job
+
+Checkout the sister project [Rocket Job](http://rocketjob.io): Ruby's missing batch system.
+
+Fully supports Semantic Logger when running jobs in the background. Complete support for job metrics
+sent via Semantic Logger to your favorite dashboards.
+
 ## Supports
 
 Semantic Logger is tested and supported on the following Ruby platforms:
-- Ruby 2.1, 2.2, 2.3, and above
-- JRuby 1.7.23, 9.0.5 and above
+- Ruby 2.1 and higher.
+- JRuby 9.1 and higher.
 
 The following gems are only required when their corresponding appenders are being used,
 and are therefore not automatically included by this gem:
@@ -49,6 +64,8 @@ and are therefore not automatically included by this gem:
 - Syslog Appender: gem 'syslog_protocol' 0.9.2 or above
 - Syslog Appender to a remote syslogng server over TCP or UDP: gem 'net_tcp_client'
 - Splunk Appender: gem 'splunk-sdk-ruby'
+- Elasticsearch Appender: gem 'elasticsearch'
+- Kafka Appender: gem 'ruby-kafka'
 
 ## V4 Upgrade notes
 
@@ -56,9 +73,39 @@ The following changes need to be made when upgrading to V4:
 - Ruby V2.1 / JRuby V9.1 is now the minimum runtime version.
 - Replace calls to Logger#with_payload with SemanticLogger.named_tagged.
 - Replace calls to Logger#payload with SemanticLogger.named_tags.
+- MongoDB Appender requires Mongo Ruby Client V2 or greater.
 - Appenders now write payload data in a seperate :payload tag instead of mixing them.
   directly into the root elements to avoid name clashes.
-- MongoDB Appender requires Mongo Ruby Client V2 or greater.
+
+As a result any calls like the following:
+
+~~~ruby
+logger.debug foo: 'foo', bar: 'bar'
+~~~
+
+Must be replaced with the following in v4:
+
+~~~ruby
+logger.debug payload: {foo: 'foo', bar: 'bar'}
+~~~
+
+Similarly, for measure blocks:
+
+~~~ruby
+logger.measure_info('How long is the sleep', foo: 'foo', bar: 'bar') { sleep 1 } 
+~~~
+
+Must be replaced with the following in v4:
+
+~~~ruby
+logger.measure_info('How long is the sleep', payload: {foo: 'foo', bar: 'bar'}) { sleep 1 } 
+~~~
+
+The common log call has not changed, and the payload is still logged directly: 
+
+~~~ruby
+logger.debug('log this', foo: 'foo', bar: 'bar')
+~~~
 
 ## Install
 
@@ -66,7 +113,7 @@ The following changes need to be made when upgrading to V4:
 
 To configure a stand-alone application for Semantic Logger:
 
-```ruby
+~~~ruby
 require 'semantic_logger'
 
 # Set the global default log level
@@ -74,7 +121,7 @@ SemanticLogger.default_level = :trace
 
 # Log to a file, and use the colorized formatter
 SemanticLogger.add_appender(file_name: 'development.log', formatter: :color)
-```
+~~~
 
 If running rails, see: [Semantic Logger Rails](http://rocketjob.github.io/semantic_logger/rails.html)
 

data/lib/semantic_logger.rb

@@ -44,6 +44,5 @@ end
 # Close and flush all appenders at exit, waiting for outstanding messages on the queue
 # to be written first
 at_exit do
-  # Cannot call #close since test frameworks use at_exit to run loaded tests
-  SemanticLogger.flush
+  SemanticLogger.close
 end

data/lib/semantic_logger/ansi_colors.rb

@@ -12,8 +12,7 @@ module SemanticLogger
     CYAN      = "\e[36m"
     WHITE     = "\e[37m"
 
-    # Maps the log level to a color for colorized formatters
-    # Since this map is not frozen, it can be modified as needed
+    # DEPRECATED - NOT USED
     LEVEL_MAP = {
       trace: MAGENTA,
       debug: GREEN,

data/lib/semantic_logger/appender.rb

@@ -1,21 +1,23 @@
 module SemanticLogger
   module Appender
     # @formatter:off
-    autoload :Bugsnag,          'semantic_logger/appender/bugsnag'
-    autoload :Elasticsearch,    'semantic_logger/appender/elasticsearch'
-    autoload :File,             'semantic_logger/appender/file'
-    autoload :Graylog,          'semantic_logger/appender/graylog'
-    autoload :Honeybadger,      'semantic_logger/appender/honeybadger'
-    autoload :Sentry,           'semantic_logger/appender/sentry'
-    autoload :Http,             'semantic_logger/appender/http'
-    autoload :MongoDB,          'semantic_logger/appender/mongodb'
-    autoload :NewRelic,         'semantic_logger/appender/new_relic'
-    autoload :Splunk,           'semantic_logger/appender/splunk'
-    autoload :SplunkHttp,       'semantic_logger/appender/splunk_http'
-    autoload :Syslog,           'semantic_logger/appender/syslog'
-    autoload :Tcp,              'semantic_logger/appender/tcp'
-    autoload :Udp,              'semantic_logger/appender/udp'
-    autoload :Wrapper,          'semantic_logger/appender/wrapper'
+    autoload :Bugsnag,           'semantic_logger/appender/bugsnag'
+    autoload :Elasticsearch,     'semantic_logger/appender/elasticsearch'
+    autoload :ElasticsearchHttp, 'semantic_logger/appender/elasticsearch_http'
+    autoload :File,              'semantic_logger/appender/file'
+    autoload :Graylog,           'semantic_logger/appender/graylog'
+    autoload :Honeybadger,       'semantic_logger/appender/honeybadger'
+    autoload :Kafka,             'semantic_logger/appender/kafka'
+    autoload :Sentry,            'semantic_logger/appender/sentry'
+    autoload :Http,              'semantic_logger/appender/http'
+    autoload :MongoDB,           'semantic_logger/appender/mongodb'
+    autoload :NewRelic,          'semantic_logger/appender/new_relic'
+    autoload :Splunk,            'semantic_logger/appender/splunk'
+    autoload :SplunkHttp,        'semantic_logger/appender/splunk_http'
+    autoload :Syslog,            'semantic_logger/appender/syslog'
+    autoload :Tcp,               'semantic_logger/appender/tcp'
+    autoload :Udp,               'semantic_logger/appender/udp'
+    autoload :Wrapper,           'semantic_logger/appender/wrapper'
     # @formatter:on
 
     # DEPRECATED, use SemanticLogger::AnsiColors

data/lib/semantic_logger/appender/bugsnag.rb

@@ -27,19 +27,20 @@ class SemanticLogger::Appender::Bugsnag < SemanticLogger::Subscriber
   #     regular expression. All other messages will be ignored.
   #     Proc: Only include log messages where the supplied Proc returns true
   #           The Proc must return true or false.
-  def initialize(level: :error, formatter: nil, filter: nil, host: SemanticLogger.host, application: SemanticLogger.application, &block)
-    raise 'Bugsnag only supports :info, :warn, or :error log levels' unless [:info, :warn, :error].include?(level)
+  def initialize(level: :error, formatter: nil, filter: nil, application: nil, host: nil, &block)
+    raise 'Bugsnag only supports :info, :warn, or :error log levels' unless [:info, :warn, :error, :fatal].include?(level)
 
     # Replace the Bugsnag logger so that we can identify its log messages and not forward them to Bugsnag
     Bugsnag.configure { |config| config.logger = SemanticLogger[Bugsnag] }
 
-    super(level: level, formatter: formatter, filter: filter, host: host, application: application, &block)
+    super(level: level, formatter: formatter, filter: filter, application: application, host: host, &block)
   end
 
   # Returns [Hash] of parameters to send to Bugsnag.
   def call(log, logger)
-    h            = log.to_h(host, application)
+    h = SemanticLogger::Formatters::Raw.new.call(log, logger)
     h[:severity] = log_level(log)
+    h.delete(:message) if h[:exception] && (h[:message] == h[:exception][:message])
     h.delete(:time)
     h.delete(:exception)
     h

data/lib/semantic_logger/appender/elasticsearch.rb

@@ -1,17 +1,29 @@
+begin
+  require 'elasticsearch'
+rescue LoadError
+  raise 'Gem elasticsearch is required for logging to Elasticsearch. Please add the gem "elasticsearch" to your Gemfile.'
+end
+
 require 'date'
+
 # Forward all log messages to Elasticsearch.
 #
 # Example:
+#
 #   SemanticLogger.add_appender(
 #     appender: :elasticsearch,
 #     url:      'http://localhost:9200'
 #   )
-class SemanticLogger::Appender::Elasticsearch < SemanticLogger::Appender::Http
-  attr_accessor :index, :type
+class SemanticLogger::Appender::Elasticsearch < SemanticLogger::Subscriber
+  attr_accessor :url, :index, :type, :client, :flush_interval, :timeout_interval, :batch_size
 
   # Create Elasticsearch appender over persistent HTTP(S)
   #
   # Parameters:
+  #   url: [String]
+  #     Fully qualified address to the Elasticsearch service.
+  #     Default: 'http://localhost:9200'
+  #
   #   index: [String]
   #     Prefix of the index to store the logs in Elasticsearch.
   #     The final index appends the date so that indexes are used per day.
@@ -22,6 +34,18 @@ class SemanticLogger::Appender::Elasticsearch < SemanticLogger::Appender::Http
   #     Document type to associate with logs when they are written.
   #     Default: 'log'
   #
+  #   batch_size: [Fixnum]
+  #     Size of list when sending to Elasticsearch. May be smaller if flush is triggered early.
+  #     Default: 500
+  #
+  #   flush_interval: [Fixnum]
+  #     Seconds to wait before attempting a flush to Elasticsearch. If no messages queued it's a NOOP.
+  #     Default: 1
+  #
+  #   timeout_interval: [Fixnum]
+  #     Seconds to allow the Elasticsearch client to flush the bulk message.
+  #     Default: 10
+  #
   #   level: [:trace | :debug | :info | :warn | :error | :fatal]
   #     Override the log level for this appender.
   #     Default: SemanticLogger.default_level
@@ -29,7 +53,7 @@ class SemanticLogger::Appender::Elasticsearch < SemanticLogger::Appender::Http
   #   formatter: [Object|Proc|Symbol|Hash]
   #     An instance of a class that implements #call, or a Proc to be used to format
   #     the output from this appender
-  #     Default: Use the built-in formatter (See: #call)
+  #     Default: :raw_json (See: #call)
   #
   #   filter: [Regexp|Proc]
   #     RegExp: Only include log messages where the class name matches the supplied.
@@ -44,26 +68,88 @@ class SemanticLogger::Appender::Elasticsearch < SemanticLogger::Appender::Http
   #   application: [String]
   #     Name of this application to appear in log messages.
   #     Default: SemanticLogger.application
-  def initialize(options, &block)
-    options       = options.dup
-    @index        = options.delete(:index) || 'semantic_logger'
-    @type         = options.delete(:type) || 'log'
-    options[:url] ||= 'http://localhost:9200'
-    super(options, &block)
-    @request_path = "#{@path.end_with?('/') ? @path : "#{@path}/"}#{@index}-%Y.%m.%d"
-    @logging_path = "#{@request_path}/#{type}"
+  def initialize(url: 'http://localhost:9200', index: 'semantic_logger', type: 'log', flush_interval: 1, timeout_interval: 10, batch_size: 500,
+    level: nil, formatter: nil, filter: nil, application: nil, host: nil, &block)
+
+    @url              = url
+    @index            = index
+    @type             = type
+    @flush_interval   = flush_interval
+    @timeout_interval = timeout_interval
+    @batch_size       = batch_size
+
+    @messages_mutex = Mutex.new
+    @messages       = Array.new
+
+    super(level: level, formatter: formatter, filter: filter, application: application, host: host, &block)
+    reopen
+  end
+
+  def reopen
+    @client = Elasticsearch::Client.new(url: url, logger: SemanticLogger::Processor.logger.clone)
+
+    @messages_mutex.synchronize { @messages = [] }
+
+    @flush_task = Concurrent::TimerTask.new(execution_interval: flush_interval, timeout_interval: timeout_interval) do
+      flush
+    end.execute
+  end
+
+  def close
+    @flush_task.shutdown if @flush_task
+    @flush_task = nil
+    # No api to close connections in the elasticsearch client!
+    #@client.close if @client
+    #@client = nil
+  end
+
+  def call(log, logger)
+    h = SemanticLogger::Formatters::Raw.new.call(log, logger)
+    h.delete(:time)
+    h[:timestamp] = log.time.utc.iso8601(SemanticLogger::Formatters::Base::PRECISION)
+    h
+  end
+
+  def flush
+    collected_messages = nil
+    @messages_mutex.synchronize do
+      if @messages.length > 0
+        collected_messages = @messages
+        @messages          = []
+      end
+    end
+
+    if collected_messages
+      bulk_result = @client.bulk(body: collected_messages)
+      if bulk_result["errors"]
+        failed = bulk_result["items"].select { |x| x["status"] != 201 }
+        SemanticLogger::Processor.logger.error("ElasticSearch: Write failed. Messages discarded. : #{failed}")
+      end
+    end
+  rescue Exception => exc
+    SemanticLogger::Processor.logger.error('ElasticSearch: Failed to bulk insert log messages', exc)
   end
 
   # Log to the index for today
   def log(log)
     return false unless should_log?(log)
 
-    post(formatter.call(log, self), log.time.strftime(@logging_path))
-  end
+    daily_index = log.time.strftime("#{@index}-%Y.%m.%d")
 
-  # Deletes all log data captured for a day
-  def delete_all(date = Date.today)
-    delete(date.strftime(@request_path))
+    bulk_index   = {'index' => {'_index' => daily_index, '_type' => @type}}
+    bulk_payload = formatter.call(log, self)
+
+    enqueue(bulk_index, bulk_payload)
   end
 
+  def enqueue(bulk_index, bulk_payload)
+    messages_len =
+      @messages_mutex.synchronize do
+        @messages.push(bulk_index)
+        @messages.push(bulk_payload)
+        @messages.length
+      end
+
+    flush if messages_len >= batch_size
+  end
 end

data/lib/semantic_logger/appender/elasticsearch_http.rb

@@ -0,0 +1,76 @@
+require 'date'
+# Forward all log messages to Elasticsearch one at a time via a HTTP post.
+#
+# Note:
+# * Other than in very low volume environments it is recommended to rather use the Elasticsearch appender,
+#   since it supports bulk logging.
+#
+# Example:
+#   SemanticLogger.add_appender(
+#     appender: :elasticsearch_http,
+#     url:      'http://localhost:9200'
+#   )
+class SemanticLogger::Appender::ElasticsearchHttp < SemanticLogger::Appender::Http
+  attr_accessor :index, :type
+
+  # Create Elasticsearch appender over persistent HTTP(S)
+  #
+  # Parameters:
+  #   index: [String]
+  #     Prefix of the index to store the logs in Elasticsearch.
+  #     The final index appends the date so that indexes are used per day.
+  #       I.e. The final index will look like 'semantic_logger-YYYY.MM.DD'
+  #     Default: 'semantic_logger'
+  #
+  #   type: [String]
+  #     Document type to associate with logs when they are written.
+  #     Default: 'log'
+  #
+  #   level: [:trace | :debug | :info | :warn | :error | :fatal]
+  #     Override the log level for this appender.
+  #     Default: SemanticLogger.default_level
+  #
+  #   formatter: [Object|Proc|Symbol|Hash]
+  #     An instance of a class that implements #call, or a Proc to be used to format
+  #     the output from this appender
+  #     Default: Use the built-in formatter (See: #call)
+  #
+  #   filter: [Regexp|Proc]
+  #     RegExp: Only include log messages where the class name matches the supplied.
+  #     regular expression. All other messages will be ignored.
+  #     Proc: Only include log messages where the supplied Proc returns true
+  #           The Proc must return true or false.
+  #
+  #   host: [String]
+  #     Name of this host to appear in log messages.
+  #     Default: SemanticLogger.host
+  #
+  #   application: [String]
+  #     Name of this application to appear in log messages.
+  #     Default: SemanticLogger.application
+  def initialize(index: 'semantic_logger', type: 'log',
+    url: 'http://localhost:9200', compress: false, ssl: {}, open_timeout: 2.0, read_timeout: 1.0, continue_timeout: 1.0,
+    level: nil, formatter: nil, filter: nil, application: nil, host: nil, &block)
+
+    @index = index
+    @type  = type
+    super(url: url, compress: compress, ssl: ssl, open_timeout: 2.0, read_timeout: open_timeout, continue_timeout: continue_timeout,
+      level:   level, formatter: formatter, filter: filter, application: application, host: host, &block)
+
+    @request_path = "#{@path.end_with?('/') ? @path : "#{@path}/"}#{@index}-%Y.%m.%d"
+    @logging_path = "#{@request_path}/#{type}"
+  end
+
+  # Log to the index for today.
+  def log(log)
+    return false unless should_log?(log)
+
+    post(formatter.call(log, self), log.time.strftime(@logging_path))
+  end
+
+  # Deletes all log data captured for a day.
+  def delete_all(date = Date.today)
+    delete(date.strftime(@request_path))
+  end
+
+end