soar_auditing_provider 0.7.0 → 0.9.3

data/.ruby-version

@@ -1 +1 @@
-ruby-2.2
+ruby-2.3.0

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2016 Ernst Van Graan
+Copyright (c) 2016 Barney de Villiers
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,14 +1,8 @@
 # SoarAuditingProvider
 
-This gem provides the auditing API that audit providers for the SOAR architecture should adhere to. 
+[![Gem Version](https://badge.fury.io/rb/soar_auditing_provider.png)](https://badge.fury.io/rb/soar_auditing_provider) [<img src="http://soar-ci.dev.auto-h.net:8080/job/soar-auditing-provider/badge/icon">](http://soar-ci.dev.auto-h.net:8080/job/soar-auditing-provider)
 
-Providers are initialized with a list of auditors (already configured) and their configurations, which include the NFRs they support (key: 'nfrs'), and is responsible for selecting an auditor given a set of NFRs. Such selection is made by calling the select(nfrs) method. The auditing API as set out below then delegates auditing actions to the selected auditor.
-
-The API provides default delegation behaviour that calls the API methods verbatim on the auditor. NFRs can be anything you want. You can ask the auditing provider to select an auditor based on a set of NFRs. The default behaviour is that the first auditor which matches all NFRs exactly is returned. You can override this behaviour by overriding the select(nfrs) method.
-
-If you'd simply like the auditing provider to use the first (perhaps only?) auditor that it knows about, you can tell it to select DEFAULT so:
-
-auditor = auditing_provider.select(SoarAuditingProvider::AuditingProviderAPI::DEFAULT)
+This gem provides an auditing provider for the SOAR architecture.
 
 ## Installation
 
@@ -18,6 +12,16 @@ Add this line to your application's Gemfile:
 gem 'soar_auditing_provider'
 ```
 
+Note that the auditing provider will only be useful when configured with an auditor that extends from soar_auditor_api. Recommend using log4r_auditor for auditing to a local file and standard stream. Alternatively use logstash_auditor for auditing to a centralized system.
+
+```ruby
+gem 'log4r_auditor'
+```
+and/or
+```ruby
+gem 'logstash_auditor'
+```
+
 And then execute:
 
     $ bundle
@@ -26,81 +30,143 @@ Or install it yourself as:
 
     $ gem install soar_auditing_provider
 
-## Usage
 
-Extend the SoarAuditingProvider::AuditingProviderAPI to create a new auditing provider, or instantiate the API directly for an out-of-the-box auditing provider with default behaviour.
+## Testing
 
-```
-class MyAuditingProvider < SoarAuditingProvider::AuditingProviderAPI
-end
-```
+Run the rspec test tests:
 
-or
+    $ bundle exec rspec -cfd spec
 
-```
-auditor_provider = SoarAuditingProvider::AuditingProviderAPI.new(auditors)
-```
 
-Initialize the provider so:
+## Usage
 
-```
-auditor = MyAuditor.new
-auditor_nfrs = { 'nfrs' => { 'accessibility' => 'local' }, 'privacy' => 'not encrypted' }
-@iut = MyAuditingProvider.new( { auditor => auditor_nfrs } )
+Require the gems responsible for various aspects of the auditing
+```ruby
+require 'soar_auditing_provider'
+require 'log4r_auditor'
+require 'soar_flow'
 ```
 
-Select an auditor and audit using the API methods. The auditing provider remembers your selection, e.g.:
-
-```
-@iut.select( {'accessibility' => 'local' })
-@iut.info("This is info")
-@iut.debug(some_debug_object)
-@iut.warn("Statistics show that dropped packets have increased to #{dropped}%")
-@iut.error("Could not resend some dropped packets. They have been lost. All is still OK, I could compensate")
-@iut.fatal("Unable to perform action, too many dropped packets. Functional degradation.")
-@iut << 'Rack::CommonLogger requires this'
+Initialize and configure the provider.
+```ruby
+AUDITING_CONFIGURATION = {
+  'auditing' => {
+    'level' => 'debug',
+    'queue_worker' => {
+      'queue_size' => 1000,
+      'initial_back_off_in_seconds' => 1,
+      'back_off_multiplier' => 2,
+      'back_off_attempts' => 5
+    },
+    'default_nfrs' => {
+      'accessibility' => 'local',
+      'privacy' => 'not encrypted',
+      'reliability' => 'instance',
+      'performance' => 'high'
+    },
+    'auditors' => {
+      'log4r' => {
+        'adaptor' => 'Log4rAuditor::Log4rAuditor',
+        'file_name' => 'soar_sc.log',
+        'standard_stream' => 'stdout',
+        'nfrs' => {
+          'accessibility' => 'local',
+          'privacy' => 'not encrypted',
+          'reliability' => 'instance',
+          'performance' => 'high'
+        }
+      }
+    }
+  }
+}
+myauditing = SoarAuditingProvider::AuditingProvider.new( AUDITING_CONFIGURATION['auditing'] )
 ```
 
-Alternatively, to use different auditors for different sets of NFRs in the same code, do as below. The auditing provider remembers the last auditor selected.
-
+In order to associate all startup and shutdown related audit events with each other it is useful to set an instance flow identifier. Generate an unique flow identifier and pass to the auditing provider.  Optional but very useful.
+```ruby
+myauditing.instance_flow_identifier = SoarFlow::ID::generate_flow_id
 ```
-  auditor_A = @iut.select( {'my nfr 1' => 'criteria 1' })
-  auditor_B = @iut.select( {'my nfr 2' => 'criteria 2' })
-
-  auditor_A.debug('debug')
-  auditor_B.warn('warn')
 
-  @iut.error('error') # Uses auditor_B since it was selected last
+When auditing to a local file there is no need to identify each audit event with a specific service since each service probably has its own audit file.  However, when merging audit events to a centralized system it is vital to associate each audit event with a specific service and instance thereof.  Set an unique service identifer that will form part of each audit event as follow:
+```ruby
+myauditing.service_identifier = 'my-test-service.com'
 ```
 
-The API also supports appending as below, enabling support, e.g. for Rack::CommonLogger, etc.:
-
+Configure the audit level of the auditing provider either inside the configuration or by calling the set_audit_level method. This will result in audit events of a lower level being ignored.
+```ruby
+myauditing.set_audit_level(:info)
 ```
-<<
+
+Generate audit events by passing in anything that implements the to_s method. The set and order of audit levels are [debug,info,warn,error,fatal]. Note that passing in the flow identifier is optional but recommended for each audit event.
+```ruby
+some_debug_object = 123
+myauditing.info("This is info",flow_id)
+myauditing.debug(some_debug_object,flow_id)
+dropped = 95
+myauditing.warn("Statistics show that dropped packets have increased to #{dropped}%",flow_id)
+myauditing.error("Could not resend some dropped packets. They have been lost. All is still OK, I could compensate",flow_id)
+myauditing.fatal("Unable to perform action, too many dropped packets. Functional degradation.",flow_id)
+myauditing << 'Rack::CommonLogger requires this'
 ```
 
 ## Detailed example
 
-```
-require 'log4r'
+```ruby
 require 'soar_auditing_provider'
+require 'log4r_auditor'
+require 'soar_flow'
 
 class Main
-  include Log4r
-  
+
+  AUDITING_CONFIGURATION = {
+    'auditing' => {
+      'level' => 'debug',
+      'queue_worker' => {
+        'queue_size' => 1000,
+        'initial_back_off_in_seconds' => 1,
+        'back_off_multiplier' => 2,
+        'back_off_attempts' => 5
+      },
+      'default_nfrs' => {
+        'accessibility' => 'local',
+        'privacy' => 'not encrypted',
+        'reliability' => 'instance',
+        'performance' => 'high'
+      },
+      'auditors' => {
+        'log4r' => {
+          'adaptor' => 'Log4rAuditor::Log4rAuditor',
+          'file_name' => 'soar_sc.log',
+          'standard_stream' => 'stdout',
+          'nfrs' => {
+            'accessibility' => 'local',
+            'privacy' => 'not encrypted',
+            'reliability' => 'instance',
+            'performance' => 'high'
+          }
+        }
+      }
+    }
+  }
+
   def test_sanity
-    auditor = Logger.new 'sanity'
-    auditor.outputters = Outputter.stdout
-    @iut = SoarAuditingProvider::AuditingProviderAPI.new( { auditor => { 'nfrs' => {'accessibility' => 'local'} } } )
-    @iut.select(SoarAuditingProvider::AuditingProviderAPI::DEFAULT)
+    #create and configure auditing instance
+    myauditing = SoarAuditingProvider::AuditingProvider.new( AUDITING_CONFIGURATION['auditing'] )
+    myauditing.instance_flow_identifier = SoarFlow::ID::generate_flow_id
+    myauditing.service_identifier = 'my-test-service.com'
+
+    #associate a set of auditing entries with a flow by generating a flow identifiers
+    flow_id = SoarFlow::ID::generate_flow_id
+
+    #generate audit events
     some_debug_object = 123
-    @iut.info("This is info")
-    @iut.debug(some_debug_object)
+    myauditing.info("This is info",flow_id)
+    myauditing.debug(some_debug_object,flow_id)
     dropped = 95
-    @iut.warn("Statistics show that dropped packets have increased to #{dropped}%")
-    @iut.error("Could not resend some dropped packets. They have been lost. All is still OK, I could compensate")
-    @iut.fatal("Unable to perform action, too many dropped packets. Functional degradation.")
-    @iut << 'Rack::CommonLogger requires this'
+    myauditing.warn("Statistics show that dropped packets have increased to #{dropped}%",flow_id)
+    myauditing.error("Could not resend some dropped packets. They have been lost. All is still OK, I could compensate",flow_id)
+    myauditing.fatal("Unable to perform action, too many dropped packets. Functional degradation.",flow_id)
+    myauditing << 'Rack::CommonLogger requires this'
   end
 end
 
@@ -108,6 +174,28 @@ main = Main.new
 main.test_sanity
 ```
 
+## Out of Band Auditing
+
+In order prevent auditing from affecting normal execution the audit events are buffered and published to the auditor using a separate thread.  
+
+The buffer size can be configured.  When full, new entries are discarded in favor of old entries and the overflow counter incremented. The separate thread will use the auditor to publish the buffered audit events in order irrespective of level.  Failures are retried using a configurable exponential back off scheme.
+
+## At_exit Hook
+
+The auditing provider automatically chains a hook into the Kernel at_exit method in order to flush the audit entries at shutdown. It will generate a final audit entry (info level) stating "Application exit" with the flow identifier set using the instance_flow_id= method. Thereafter it attempts to flush the remaining entries to the selected auditor.  Failing that, it will flush the entries to the standard error stream and exit.
+
+## Status
+
+Provision has been made for out-of-band status/statistics gathering inside the auditing provider.  The hash containing the status/statistics is accessible using the status method call:
+```ruby
+myauditing.get_status
+```
+
+At present only the buffer overflow count is avialable:
+```ruby
+  { 'audit_buffer_overflows' => 123 }
+```
+
 ## Contributing
 
 Bug reports and feature requests are welcome by email to ernst dot van dot graan at hetzner dot co dot za. This gem is sponsored by Hetzner (Pty) Ltd (http://hetzner.co.za)
@@ -119,4 +207,3 @@ Though out of scope for the provider, auditors should take into account encoding
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
-

data/lib/soar_auditing_provider/auditing_overflow_error.rb

@@ -0,0 +1,4 @@
+module SoarAuditingProvider
+  class AuditingOverflowError < RuntimeError
+  end
+end

data/lib/soar_auditing_provider/auditing_provider.rb

@@ -0,0 +1,150 @@
+require 'soar_auditing_provider_api'
+require 'soar_auditing_format'
+require 'soar_configured_factory'
+require 'time'
+require 'securerandom'
+
+module SoarAuditingProvider
+  class AuditingProvider < SoarAuditingProviderAPI::AuditingProviderAPI
+    private
+
+    #Aliases for bypassing overridden methods when accessing underlying super class api
+    alias :super_debug :debug
+    alias :super_info  :info
+    alias :super_warn  :warn
+    alias :super_error :error
+    alias :super_fatal :fatal
+
+    public
+
+    attr_accessor :instance_flow_identifier
+    attr_accessor :service_identifier
+    attr_reader   :configuration
+
+    def initialize(configuration)
+      validate_provider_configuration(configuration)
+      @configuration = configuration
+      super(create_auditors(configuration))
+      select_auditor(configuration['default_nfrs'])
+      create_auditing_worker
+      @buffer_overflow_count = 0
+      install_at_exit_handler
+    end
+
+    def select_auditor(nfrs)
+      select(nfrs)
+      set_audit_level(@configuration['level'].to_sym)
+    end
+
+    def set_audit_level(level)
+      @auditor.set_audit_level(level)
+    rescue ArgumentError
+      $stderr.puts 'Invalid auditing level'
+      raise
+    end
+
+    def debug(data, flow_identifier = nil)
+      enqueue(:debug, format(:debug, data, flow_identifier))
+    end
+
+    def info(data, flow_identifier = nil)
+      enqueue(:info, format(:info, data, flow_identifier))
+    end
+
+    def warn(data, flow_identifier = nil)
+      enqueue(:warn, format(:warn, data, flow_identifier))
+    end
+
+    def error(data, flow_identifier = nil)
+      enqueue(:error, format(:error, data, flow_identifier))
+    end
+
+    def fatal(data, flow_identifier = nil)
+      enqueue(:fatal, format(:fatal, data, flow_identifier))
+    end
+
+    def <<(data, flow_identifier = nil)
+      enqueue(:info, format(:info, data, flow_identifier))
+    end
+
+    def get_status
+      { 'audit_buffer_overflows' => @buffer_overflow_count }
+    end
+
+    def flush
+      @worker.flush
+    end
+
+    private
+
+    def install_at_exit_handler
+      if @configuration['install_exit_handler'] == 'true'
+        Kernel.at_exit do
+          exit_cleanup
+        end
+      end
+    end
+
+    def exit_cleanup(exception = nil)
+      audit_exception_message(exception) if exception
+      info("Application exit",@instance_flow_identifier)
+      flush
+    end
+
+    def audit_exception_message(exception)
+      exception_message = "#{exception.class}: #{exception.message}"
+      exception_message = exception_message + ":\n\t" + exception.backtrace.join("\n\t") if ENV['RACK_ENV'] == 'development'
+      fatal(exception_message,@instance_flow_identifier)
+    end
+
+    def enqueue(level, data)
+      @worker.enqueue(level, data)
+    rescue AuditingOverflowError
+      increase_buffer_overflow_count
+      $stderr.puts "Audit buffer full, unable to audit event : #{level} : #{data}"
+    end
+
+    def increase_buffer_overflow_count
+      @buffer_overflow_count += 1
+    end
+
+    def validate_provider_configuration(configuration)
+      raise 'Missing auditing level' if configuration['level'].nil?
+    end
+
+    def format(level, data, flow_identifier)
+      SoarAuditingFormatter::Formatter.format(level,@service_identifier,flow_identifier,Time.now.utc.iso8601(3),data)
+    end
+
+    def create_auditing_worker
+      @worker = AuditingWorker.new
+      @worker.configure(queue_worker_configuration: @configuration['queue_worker'], auditor_audit_method: method(:super_class_caller))
+      @worker.start
+    end
+
+    def super_class_caller(level, data)
+      send("super_#{level}",data)
+    end
+
+    def create_auditors(configuration)
+      auditor_factory = SoarConfiguredFactory::ConfiguredFactory.new(configuration['auditors'])
+      auditors = {}
+      configuration['auditors'].each do |auditor_name, auditor_configuration|
+        raise 'Missing auditor configuration' if auditor_configuration.nil?
+        auditor = create_auditor(auditor_factory,auditor_name)
+        auditors[auditor] = { 'name' => auditor_name, 'nfrs' => auditor_configuration['nfrs'] }
+      end
+      auditors
+    rescue
+      $stderr.puts 'Failure initializing auditor'
+      raise
+    end
+
+    def create_auditor(auditor_factory,auditor_name)
+      auditor_factory.create(auditor_name)
+    rescue
+      $stderr.puts 'Invalid auditor configuration'
+      raise
+    end
+  end
+end

data/lib/soar_auditing_provider/auditing_worker.rb

@@ -0,0 +1,153 @@
+require 'soar_thread_worker/thread_worker'
+
+module SoarAuditingProvider
+  class AuditingWorker < SoarThreadWorker::ThreadWorker
+    def initialize
+      @queue = Queue.new
+      @start_mutex = Mutex.new
+    end
+
+    def configure(queue_worker_configuration: ,auditor_audit_method: )
+      validate_configuration(queue_worker_configuration)
+      @maximum_queue_size = queue_worker_configuration['queue_size'].to_i
+      @initial_back_off_in_seconds = queue_worker_configuration['initial_back_off_in_seconds'].to_i
+      @back_off_multiplier = queue_worker_configuration['back_off_multiplier'].to_i
+      @maximum_back_off_attempts = queue_worker_configuration['back_off_attempts'].to_i
+      @auditor_audit_method = auditor_audit_method
+    end
+
+    def enqueue(level, data)
+      if @queue.size < @maximum_queue_size then
+        @queue.push({:level => level, :data => data})
+      else
+        raise AuditingOverflowError
+      end
+      ensure_worker_is_running
+    end
+
+    def start(verbose: false)
+      @start_mutex.synchronize {
+        if not running? then
+          super()
+          $stderr.puts("Auditing worker was not running and respawned") if verbose
+        end
+      }
+    end
+
+    def execute
+      audit_event = @queue.pop
+      failed_before = false
+      begin
+        if @stopping
+          @queue.push(audit_event) if audit_event #push the event back into the queue so that fallback flush mechanism can deal with this audit event
+          return true #indicates to thread worder that we are done executing since we are in the process of stopping
+        end
+        exponential_back_off(start_at_last_attempt: failed_before) {
+          @auditor_audit_method.call(audit_event[:level],audit_event[:data])
+        }
+      rescue Exception => e
+        print_exception_with_message_to_stderr(nil,e)
+        failed_before = true
+        retry
+      end
+      return false #indicates to thread worder that we are not done executing
+    end
+
+    def flush(timeout = 1)
+      ensure_worker_is_running
+      wait_for_worker_to_clear_queue(timeout)
+      fallback_flush_to_stderr if @queue.size > 0
+    end
+
+    private
+
+    def wait_for_worker_to_clear_queue(timeout = 1)
+      start_time = Time.now
+      until ((@queue.size == 0) or ((Time.now - start_time) >= timeout)) do
+        sleep(0.1)
+      end
+    end
+
+    def fallback_flush_to_stderr(timeout = 1)
+      $stderr.puts 'Unable to flush audit entries to auditor, stopping worker and flushing to stderr'
+      ensure_worker_is_stopped
+      start_time = Time.now
+      until ((@queue.size == 0) or ((Time.now - start_time) >= timeout)) do
+        audit_event = @queue.pop
+        $stderr.puts audit_event[:data].to_s
+      end
+    rescue Exception => e
+      print_exception_with_message_to_stderr('Failure during fallback attempt to flush audit entries to stderr',e)
+      raise
+    end
+
+    def print_exception_with_message_to_stderr(notification,exception)
+      message = "#{exception.class}: #{exception.message}"
+      message = message + ":\n\t" + exception.backtrace.join("\n\t") if ENV['RACK_ENV'] == 'development'
+      $stderr.puts "#{notification}: #{message}"
+    end
+
+    def ensure_worker_is_running
+      start(verbose: true)
+    end
+
+    def ensure_worker_is_stopped
+      attempt_graceful_stop
+      sleep_while_still_running(5)
+      force_stop
+    end
+
+    def attempt_graceful_stop
+      @stopping = true
+    end
+
+    def force_stop
+      stop
+    end
+
+    def validate_configuration(queue_worker_configuration)
+      raise ArgumentError.new("Invalid queue size (#{queue_worker_configuration['queue_size'].to_i})") if queue_worker_configuration['queue_size'].to_i < 1
+      raise ArgumentError.new("Invalid number of back off attempts (#{queue_worker_configuration['back_off_attempts'].to_i})") if queue_worker_configuration['back_off_attempts'].to_i < 1
+    end
+
+    def exponential_back_off(start_at_last_attempt: false)
+      attempt = 1
+      if start_at_last_attempt
+        attempt = @maximum_back_off_attempts
+        sleep_unless_stopping(calculate_back_off_delay(@maximum_back_off_attempts))
+      end
+      begin
+        yield
+      rescue StandardError
+        # Any exception derived from StandardError is assumed to be a failure and
+        # attempted again until it completes without an exception or an exception
+        # not derived from StandardError
+        if ((attempt <= @maximum_back_off_attempts) and (not @stopping)) then
+          sleep_unless_stopping(calculate_back_off_delay(attempt))
+          attempt = attempt + 1
+          retry
+        else
+          raise
+        end
+      end
+    end
+
+    def calculate_back_off_delay(attempt)
+      @initial_back_off_in_seconds * (@back_off_multiplier ** (attempt-1))
+    end
+
+    def sleep_unless_stopping(desired_delay)
+      start_time = Time.now
+      until (@stopping or ((Time.now - start_time) >= desired_delay)) do
+        sleep(0.1)
+      end
+    end
+
+    def sleep_while_still_running(desired_delay)
+      start_time = Time.now
+      until ((false == @running) or ((Time.now - start_time) >= desired_delay)) do
+        sleep(0.1)
+      end
+    end
+  end
+end