semantic_logger 0.3.3 → 0.4.0

data/README.md

@@ -3,11 +3,9 @@ semantic-logger-ruby
 
 * http://github.com/ClarityServices/semantic-logger-ruby
 
-Feedback is welcome and appreciated :)
-
 ### Introduction
 
-SemanticLogger is a Logger that supports logging of meta-data, not only text messages
+SemanticLogger is a Logger that supports logging of meta-data, along with text messages
 
 Machines can understand the logged data without having to use
 complex Regular Expressions or other text parsing techniques
@@ -36,30 +34,348 @@ This can be changed over time to:
            :table    => "users",
            :action   => "query")
 
-Using this semantic example, a machine can easily find all queries for table
-'users' that took longer than 100 ms.
+Using the MongoDB appender, we can easily find all queries for table 'users'
+that took longer than 100 ms:
 
-    db.logs.find({"metadata.table":"users", "metadata.action":"query"})
+    db.logs.find({"payload.table":"users", "payload.action":"query", "payload.duration":{$gt:100} })
 
 Since SemanticLogger can call existing Loggers, it does not force end-users
 to have to adopt a Semantic aware adapter. Although, such adapters create
 tremendous value in the problem monitoring and determination processes.
 
-For ease of use each Appender can be used directly if required. It is preferred
-however to use the SemanticLogger interface and supply the configuration to it.
-In this way the adapter can be changed without modifying the application
-source code.
+### Logging API
+
+#### Standard Logging methods
+
+The Semantic Logger logging API supports the existing logging interface for
+the Rails and Ruby Loggers. For example:
+
+    logger.info("Hello World")
+
+Or to query whether a specific log level is set
+
+    logger.info?
+
+The following logging methods are all available
+
+    logger.trace(message, payload) { # optional block }
+    logger.debug(message, payload) { # optional block }
+    logger.info(message, payload) { # optional block }
+    logger.warn(message, payload) { # optional block }
+    logger.error(message, payload) { # optional block }
+    logger.fatal(message, payload) { # optional block }
+
+Parameters
+
+- message: The text message to log.
+  Mandatory only if no block is supplied
+- payload: Optional, either a Ruby Exception object or a Hash
+- block:   The optional block is executed only if the corresponding log level
+  is active. Can be used to prevent unnecessary calculations of debug data in
+  production.
+
+Examples:
+
+    logger.debug("Calling Supplier")
+
+    logger.debug("Calling Supplier", :request => 'update', :user => 'Jack')
+
+    logger.debug { "A total of #{result.inject(0) {|sum, i| i+sum }} were processed" }
+
+### Exceptions
+
+The Semantic Logger adds an extra parameter to the existing log methods so that
+a corresponding Exception can be logged in a standard way
+
+    begin
+      # ... Code that can raise an exception
+    rescue Exception => exception
+      logger.error("Oops external call failed", exception)
+      # Re-raise or handle the exception
+      raise exception
+    end
+
+
+#### Payload
+
+The Semantic Logger adds an extra parameter to the existing log methods so that
+additional payload can be logged, such as a Hash or a Ruby Exception object.
+
+    logger.info("Oops external call failed", :result => :failed, :reason_code => -10)
+
+The additional payload is machine readable so that we don't have to write complex
+regular expressions so that a program can analyze log output. With the MongoDB
+appender the payload is written directly to MongoDB as part of the document and
+is therefore fully searchable
+
+#### Benchmarking
+
+Another common logging requirement is to measure the time it takes to execute a block
+of code based on the log level. For example:
+
+    Rails.logger.benchmark_info "Calling external interface" do
+      # Code to call external service ...
+    end
+
+The following output will be written to file:
+
+    2012-08-30 15:37:29.474 I [48308:ScriptThreadProcess: script/rails] Rails -- Calling external interface (5.2ms)
+
+If an exception is raised during the block the exception is logged
+at the same log level as the benchmark along with the duration and message
 
-# Future
-- Web Interface to view and search log information
-- Override the log level at the appender level
+#### Logging levels
 
-# Configuration
+The following logging levels are available through Semantic Logger
+
+    :trace, :debug, :info, :warn, :error, :fatal
+
+The log levels are listed above in the order of precedence with the most detail to the least.
+For example :debug would include :info, :warn, :error, :fatal levels but not :trace
+And :fatal would only log :fatal error messages and nothing else
+
+:unknown has been mapped to :fatal for Rails and Ruby Logger
+
+:trace is a new level that is often used for tracing low level calls such
+as the data sent or received to external web services. It is also commonly used
+in the development environment for low level trace logging of methods calls etc.
+
+If only the rails logger is being used, then :trace level calls will be logged
+as debug calls only if the log level is set to trace
+
+#### Changing the Class name for Log Entries
+
+When Semantic Logger is included on a Rails project it automatically replaces the
+loggers for Rails, ActiveRecord::Base, ActionController::Base, and ActiveResource::Base
+with wrappers that set their Class name. For example in railtie.rb:
+
+      ActiveRecord::Base.logger = SemanticLogger::Logger.new(ActiveRecord)
+
+By replacing their loggers we now get the class name in the text logging output:
+
+    2012-08-30 15:24:13.439 D [47900:main] ActiveRecord --   SQL (12.0ms)  SELECT `schema_migrations`.`version` FROM `schema_migrations`
+
+It is recommended to include a class specific logger for all major classes that will
+be logging. For Example:
+
+    class ExternalSupplier
+      # Gem sync_attr is a dependency of semantic_logger so is already installed
+      include SyncAttr
+
+      # Lazy initializes the class logger on it's first call in a thread-safe way
+      sync_cattr_reader :logger do
+        SemanticLogger::Logger.new(self)
+      end
+
+      def call(params)
+        self.class.logger.benchmark_info "Calling external interface" do
+            # Code to call external service ...
+        end
+      end
+    end
+
+This will result in the log output identitying the log entry as from the ExternalSupplier class
+
+    2012-08-30 15:37:29.474 I [48308:ScriptThreadProcess: script/rails] ExternalSupplier -- Calling external interface (5.2ms)
+
+#### Tagged Logging
+
+    logger.with_tags(tracking_number) do
+      logger.debug("Hello World")
+      # ...
+    end
+
+#### Payload injected logging
+
+    logger.with_payload(:user => 'Jack') do
+      logger.debug("Hello World")
+      # ...
+    end
+
+### Configuration
 
 The Semantic Logger follows the principle where multiple appenders can be active
 at the same time. This allows one to log to MongoDB and the Rails
 ActiveResource::BufferedLogger at the same time.
 
+#### Rails Configuration
+
+Add the following line to Gemfile
+
+    gem 'semantic_logger'
+
+Also add the following line to Gemfile if you want to log to MongoDB
+
+    gem 'mongo'
+
+Install required gems with bundler
+
+    bundle install
+
+This will automatically replace the standard Rails logger with Semantic Logger
+which will write all log data to the configured Rails logger.
+
+By default Semantic Logger will detect the log level from Rails. To set the
+log level explicitly, add the following line to
+config/environments/production.rb inside the Application.configure block
+
+    config.log_level = :trace
+
+To log to both the Rails logger and MongoDB add the following lines to
+config/environments/production.rb inside the Application.configure block
+
+    config.after_initialize do
+      # Re-use the existing MongoDB connection, or create a new one here
+      db = Mongo::Connection.new['production_logging']
+
+      # Besides logging to the standard Rails logger, also log to MongoDB
+      config.semantic_logger.appenders << SemanticLogger::Appender::MongoDB.new(
+        :db              => db,
+        :collection_size => 25.gigabytes
+      )
+    end
+
+#### Custom Formatters
+
+The formatting for each appender can be replaced with custom code. To replace the
+existing formatter supply a block of code when creating the appender.
+
+For example to replace the Rails or Ruby text log formatter, in the environment configuration file:
+
+    config.after_initialize do
+      # Since the Rails logger is already initialized, replace its default formatter
+      config.semantic_logger.appenders.first.formatter = Proc.new do |log|
+          # log is a struct with the following fields:
+          #
+          # level
+          #   Log level of the supplied log call
+          #   :trace, :debug, :info, :warn, :error, :fatal
+          #
+          # thread_name
+          #   Name of the thread in which the logging call was called
+          #
+          # name
+          #   Class name supplied to the logging instance
+          #
+          # message
+          #   Text message to be logged
+          #
+          # payload
+          #   Optional Hash or Ruby Exception object to be logged
+          #
+          # time
+          #   The time at which the log entry was created
+          #
+          # duration
+          #   The time taken to complete a benchmark call
+          #
+          # tags
+          #   Any tags active on the thread when the log call was made
+          #
+
+          message = log.message.to_s
+          tags = log.tags.collect { |tag| "[#{tag}]" }.join(" ") + " " if log.tags && (log.tags.size > 0)
+
+          if log.payload
+            if log.payload.is_a?(Exception)
+              exception = log.payload
+              message << " -- " << "#{exception.class}: #{exception.message}\n#{(exception.backtrace || []).join("\n")}"
+            else
+              message << " -- " << log.payload.inspect
+            end
+          end
+
+          str = "#{log.time.strftime("%Y-%m-%d %H:%M:%S")}.#{"%03d" % (log.time.usec/1000)} #{log.level.to_s[0..0].upcase} [#{$$}:#{log.thread_name}] #{tags}#{log.name} -- #{message}"
+          str << " (#{'%.1f' % log.duration}ms)" if log.duration
+          str
+      end
+    end
+
+For example to replace the MongoDB formatter, in the environment configuration file:
+
+    config.after_initialize do
+      # Log to MongoDB and supply a custom document formatter
+      mongodb_appender = SemanticLogger::Appender::MongoDB.new(
+        :db              => Cache::Work.db,
+        :collection_size => 25.gigabytes
+      ) do |log|
+          # log is a struct with the following fields:
+          # level
+          #   Log level of the supplied log call
+          #   :trace, :debug, :info, :warn, :error, :fatal
+          #
+          # thread_name
+          #   Name of the thread in which the logging call was called
+          #
+          # name
+          #   Class name supplied to the logging instance
+          #
+          # message
+          #   Text message to be logged
+          #
+          # payload
+          #   Optional Hash or Ruby Exception object to be logged
+          #
+          # time
+          #   The time at which the log entry was created
+          #
+          # duration
+          #   The time taken to complete a benchmark call
+          #
+          # tags
+          #   Any tags active on the thread when the log call was made
+          #
+
+          # Return a document (Hash) of the data to be saved to MongoDB
+          document = {
+            :time        => log.time,
+            :host_name   => SemanticLogger::Appender::MongoDB.host_name,
+            :pid         => $PID,
+            :thread_name => log.thread_name,
+            :name        => log.name,
+            :level       => log.level,
+          }
+          document[:application] = 'MyApplication'
+          document[:message]     = SemanticLogger::Appender::MongoDB.strip_colorizing(log.message) if log.message
+          document[:duration]    = log.duration if log.duration
+          document[:tags]        = log.tags if log.tags && (log.tags.size > 0)
+
+          if log.payload
+            if log.payload.is_a?(Exception)
+              exception = log.payload
+              document[:payload] = {
+                :exception => exception.class.name,
+                :message   => exception.message,
+                :backtrace => exception.backtrace
+              }
+            else
+              document[:payload] = log.payload
+            end
+          end
+          document
+      end
+      config.semantic_logger.appenders << mongodb_appender
+    end
+
+### Architecture & Performance
+
+In order to ensure that logging does not hinder the performance of the application
+all log entries are written to thread-safe Queue. A separate thread is responsible
+for writing the log entries to each of the appenders.
+
+In this way formatting and disk or network write delays will not affect the
+performance of the application. Also adding more than one appender does not affect
+the runtime performance of the application
+
+The additional thread is automatically started on initialization. When the program
+terminates it will complete writing out all log data and flush the appenders before
+the program exits.
+
+Calling SemanticLogger::Logger#flush will wait until all outstanding log messages
+have been written and flushed to their respective appenders before returning.
+Since all logging is now from this thread calling flush is no longer thread
+specific.
+
 ### Dependencies
 
 - Ruby MRI 1.8.7 (or above) Or, JRuby 1.6.3 (or above)
@@ -68,7 +384,15 @@ ActiveResource::BufferedLogger at the same time.
 
 ### Install
 
-  gem install semantic-logger
+    gem install semantic-logger
+
+To log to MongoDB
+
+    gem install mongo
+
+### Future
+
+- Web Interface to view and search log information stored in MongoDB
 
 Development
 -----------
@@ -94,9 +418,6 @@ Once you've made your great commits:
 4. Create an [Issue](http://github.com/ClarityServices/semantic-logger/issues) with a link to your branch
 5. That's it!
 
-We are looking for volunteers to create implementations in other languages such as:
-* Java, C# and Go
-
 Meta
 ----
 

data/Rakefile

@@ -21,6 +21,7 @@ task :gem  do |t|
     spec.description = "Machine readable document oriented logging with support for MongoDB and text files"
     spec.files       = FileList["./**/*"].exclude('*.gem', 'nbproject').map{|f| f.sub(/^\.\//, '')}
     spec.has_rdoc    = true
+    spec.add_dependency 'sync_attr'
     spec.add_development_dependency 'shoulda'
   end
   Gem::Builder.new(gemspec).build

data/lib/semantic_logger.rb

@@ -1,4 +1,4 @@
-# Temp until in Gemfile
+# Include sync_attr dependency
 require 'sync_attr'
 
 module SemanticLogger

data/lib/semantic_logger/appender/logger.rb

@@ -8,6 +8,7 @@ module SemanticLogger
   module Appender
     class Logger
       attr_reader :logger
+      attr_accessor :formatter
 
       # Create a Logger or Rails Logger appender instance
       #

data/lib/semantic_logger/appender/mongodb.rb

@@ -109,8 +109,11 @@ module SemanticLogger
       # * Document updates cannot make them any larger
       # * Documents are always stored in insertion order
       #   * A find will always return the documents in their insertion order
+      #
+      # Creates an index based on tags to support faster lookups
       def create_indexes
         db.create_collection(collection_name, {:capped => true, :size => @collection_size, :max => @collection_max})
+        collection.ensure_index('tags')
       end
 
       # Purge all data from the capped collection by dropping the collection
@@ -171,6 +174,16 @@ module SemanticLogger
         message.to_s.gsub(/(\e(\[([\d;]*[mz]?))?)?/, '').strip
       end
 
+      # Default host_name to use if none is supplied to the appenders initializer
+      def self.host_name
+        @@host_name ||= Socket.gethostname.split('.').first
+      end
+
+      # Override the default host_name
+      def self.host_name=(host_name)
+        @@host_name = host_name
+      end
+
       # Log the message to MongoDB
       def log(log)
         # Insert log entry into Mongo

data/lib/semantic_logger/logger.rb

@@ -12,16 +12,23 @@
 #   log = Logger.new(STDOUT)
 #   log.level = Logger::DEBUG
 #
-#   SemanticLogger::Manager.register_appender(SemanticLogger::Appender::Logger.new(log))
+#   SemanticLogger::Logger.appenders << SemanticLogger::Appender::Logger.new(log)
 #
 #   logger = SemanticLogger::Logger.new("my.app.class")
 #   logger.debug("Login time", :user => 'Joe', :duration => 100, :ip_address=>'127.0.0.1')
 #
-# # Now log to the Logger above as well as Mongo at the same time
+#   # Now log to the Logger above as well as MongoDB at the same time
 #
-#   SemanticLogger::Manager.register_appender(SemanticLogger::Appender::Mongo.new(cfg))
+#   db = Mongo::Connection.new['production_logging']
+#
+#   SemanticLogger::Logger.appenders << SemanticLogger::Appender::MongoDB.new(
+#     :db              => db,
+#     :collection_size => 25.gigabytes
+#   )
 # ...
+#   # This will be logged to both the Ruby Logger and MongoDB
 #   logger.debug("Login time", :user => 'Mary', :duration => 230, :ip_address=>'192.168.0.1')
+#
 module SemanticLogger
   class Logger
     include SyncAttr
@@ -213,12 +220,6 @@ module SemanticLogger
       queue.size
     end
 
-    # Flush all pending log entry disk, database, etc.
-    #  All pending log writes are completed and each appender is flushed in turn
-    def flush
-      self.class.flush
-    end
-
     # Flush all pending log entry disk, database, etc.
     #  All pending log writes are completed and each appender is flushed in turn
     def self.flush
@@ -251,6 +252,33 @@ module SemanticLogger
       Queue.new
     end
 
+    # Struct Log
+    #
+    # level
+    #   Log level of the supplied log call
+    #   :trace, :debug, :info, :warn, :error, :fatal
+    #
+    # thread_name
+    #   Name of the thread in which the logging call was called
+    #
+    # name
+    #   Class name supplied to the logging instance
+    #
+    # message
+    #   Text message to be logged
+    #
+    # payload
+    #   Optional Hash or Ruby Exception object to be logged
+    #
+    # time
+    #   The time at which the log entry was created
+    #
+    # duration
+    #   The time taken to complete a benchmark call
+    #
+    # tags
+    #   Any tags active on the thread when the log call was made
+    #
     Log = Struct.new(:level, :thread_name, :name, :message, :payload, :time, :duration, :tags)
 
     # For JRuby include the Thread name rather than its id

data/lib/semantic_logger/version.rb

@@ -1,3 +1,3 @@
 module SemanticLogger #:nodoc
-  VERSION = "0.3.3"
+  VERSION = "0.4.0"
 end

data/test/logger_test.rb

@@ -33,7 +33,7 @@ class LoggerTest < Test::Unit::TestCase
       SemanticLogger::Logger::LEVELS.each do |level|
         should "log #{level} info" do
           @logger.send(level, 'hello world', @hash) { "Calculations" }
-          @logger.flush
+          SemanticLogger::Logger.flush
           assert_match /\d+-\d+-\d+ \d+:\d+:\d+.\d+ \w \[\d+:.+\] LoggerTest -- hello world -- Calculations -- \{:session_id=>\"HSSKLEU@JDK767\", :tracking_number=>12345\}/, @mock_logger.message
         end
       end
@@ -42,7 +42,7 @@ class LoggerTest < Test::Unit::TestCase
         should "add tags to log entries" do
           @logger.with_tags('12345', 'DJHSFK') do
             @logger.info('Hello world')
-            @logger.flush
+            SemanticLogger::Logger.flush
             assert_match /\d+-\d+-\d+ \d+:\d+:\d+.\d+ \w \[\d+:.+\] \[12345\] \[DJHSFK\] LoggerTest -- Hello world/, @mock_logger.message
           end
         end
@@ -51,7 +51,7 @@ class LoggerTest < Test::Unit::TestCase
           @logger.with_tags('First Level', 'tags') do
             @logger.with_tags('Second Level') do
               @logger.info('Hello world')
-              @logger.flush
+              SemanticLogger::Logger.flush
               assert_match /\d+-\d+-\d+ \d+:\d+:\d+.\d+ \w \[\d+:.+\] \[First Level\] \[tags\] \[Second Level\] LoggerTest -- Hello world/, @mock_logger.message
             end
           end
@@ -61,7 +61,7 @@ class LoggerTest < Test::Unit::TestCase
           @logger.with_payload(:tracking_number => '123456') do
             @logger.with_payload(:more => 'data', :even => 2) do
               @logger.info('Hello world')
-              @logger.flush
+              SemanticLogger::Logger.flush
               assert_match /\d+-\d+-\d+ \d+:\d+:\d+.\d+ \w \[\d+:.+\] LoggerTest -- Hello world -- \{:more=>\"data\", :even=>2, :tracking_number=>\"123456\"\}/, @mock_logger.message
             end
           end

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 0
-  - 3
-  - 3
-  version: 0.3.3
+  - 4
+  - 0
+  version: 0.4.0
 platform: ruby
 authors: 
 - Reid Morrison
@@ -14,11 +14,11 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2012-08-30 00:00:00 -04:00
+date: 2012-09-05 00:00:00 -04:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
-  name: shoulda
+  name: sync_attr
   prerelease: false
   requirement: &id001 !ruby/object:Gem::Requirement 
     requirements: 
@@ -27,8 +27,20 @@ dependencies:
         segments: 
         - 0
         version: "0"
-  type: :development
+  type: :runtime
   version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: shoulda
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id002
 description: Machine readable document oriented logging with support for MongoDB and text files
 email: 
 - reidmo@gmail.com