outliers 0.3.3 → 0.5.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 7a3abdedf8279901f986e89f931eea7027945348
-  data.tar.gz: 343deb6efbe0455784e35acfdd0b15dce25bec44
+  metadata.gz: e21b9c0b8c0b80d467c4deedbd9e2f0911b44fcc
+  data.tar.gz: e14dbca5caad874380dc33a7e2e8edebd4884556
 SHA512:
-  metadata.gz: 82e1db72c5424b718a5ed6e9a4550315f5c9b457b3959a58d9581495f3c8971794ac1096b4cb24ae712c2f336c83213bb8a0e439a303f5184a48f12635244699
-  data.tar.gz: 89e55d536770ffbca617679940e3f947c07e30754837a8f86964810959cecfed97ddb675c6d40e7814c912e64b22375f2c4f0af93449c9a10b26923f50b65660
+  metadata.gz: 6aa6ca8c8f0bcd4a4923f9c89b61cf93c5224754fdce113b69c1f4f039d7cf97a1d9cacb68781081502d8ceaff50089368e7f174d8cc262eda6c6a3d8c5a0511
+  data.tar.gz: a957f1bd5eeb096f930ff27ecdaf864dbba496d2aa8732081a4921f7ddb87bfddd65f62c53045d718e641963e4dbd6436ba4558a5e6f71fb5866c8f2a25daa1e

data/CHANGELOG.md

@@ -1,4 +1,14 @@
-## HEAD
+## 0.5.0
+
+* Added exclude filters
+* Refactored and simplified DSL
+* Added basic handler
+
+## 0.4.0
+
+* Renamed credentials to account
+* Adding JSON support to result
+* Refactor results
 
 ## 0.3.3
 

data/README.md

@@ -1,4 +1,5 @@
 [![Build Status](https://secure.travis-ci.org/brettweavnet/outliers.png)](http://travis-ci.org/brettweavnet/outliers)
+[![Code Climate](https://codeclimate.com/github/brettweavnet/outliers.png)](https://codeclimate.com/github/brettweavnet/outliers)
 
 # Outliers
 
@@ -6,32 +7,26 @@ A framework to detect misconfigurations (Outliers).
 
 ## Overview
 
-To detect misconfigurations at scale, Outliers provides a framework for performing complex evaluations:
+To detect misconfigurations at scale, Outliers provides a framework for performing complex evaluations of cloud resources based on the following:
 
 * Applications rely on **resources** delivered from multiple **providers** (EC2, S3, etc).
 * Resource configuration can be evaluated against specific **verifications**  (Instance launched from given AMI, S3 bucket contains no public objects, etc).
 * Verifications can be performed against a subset of resources based on a **filter**.
 * Those not passing verification, are flagged as Outliers.
 
-Evalutions are read from from files ending with **.rb** within a target directory.
-
-Multiple evaluations can be specified in a file, with multiple files in directory.
-
 ## Requirements
 
-* Ruby 1.9.3 or greater.
+* Ruby 1.9.3 or greater
 
 ## Installation
 
-Install the gem:
-
     gem install outliers
 
 ## Getting Started
 
-Create **~/outliers.yml** with a list of credentials in the following format:
+Create **~/outliers.yml** with a list of accounts in the following format:
 
-    credential_name:
+    account_name:
       region: AWS_REGION
       access_key_id: AWS_ACCESS_ID
       secret_access_key: AWS_SECRET_KEY
@@ -43,11 +38,11 @@ For example:
       access_key_id: abcd1234abcd1234abcd
       secret_access_key: abcd1234abcd1234abcdabcd1234abcd1234abcd
 
-Outliers provides a DSL which can be used to build up a comprehensive list of evaluations. Create a directory to store your evaluations. 
+Create a directory to store your evaluations.
 
     mkdir ~/outliers
 
-To verify all instances are in a VPC, create a file **ec2.rb** in **~/outliers** containing:
+To verify all instances in aws_prod are in a VPC, create ec2.rb in ~/outliers containing:
 
     evaluate do
       connect 'aws_prod', provider: 'aws_ec2'
@@ -76,13 +71,22 @@ Sample Output:
     I, [2013-09-24T09:42:44.804147 #4940]  INFO -- : Evaluations completed.
     I, [2013-09-24T09:42:44.804211 #4940]  INFO -- : (0 evaluations failed, 1 evaluations passed.)
 
-* Resources can be targeted or excluded by their ID (EC2 Instance ID, S3 Object Key, etc).
-* Resources can be targeted or excluded by matching a filter (Instance has tag 'x' with value 'y').
+## Results
+
+To return outliers results, you must set the **OUTLIERS_KEY** environment variable.
+
+    export OUTLIERS_KEY=abcd1234
+
+To modify the URL where reuslts are sent, set **OUTLIERS_URL** environment variable.
+
+By default, results are sent to **https://api.getoutliers.com**
+
+    export OUTLIERS_URL=http://localhost:3000
 
 ## Examples
 
-See [examples](http://brettweavnet.github.io/outliers/examples) for a list of more advanced evaluations.
+See [examples](http://www.getoutliers.com/documentation/examples) for a list of more advanced evaluations.
 
-## References
+## Documentation
 
-See the [providers](http://brettweavnet.github.io/outliers/providers), [resources](http://brettweavnet.github.io/outliers/resources) and [filters](http://brettweavnet.github.io/outliers/filters) pages for additional documentation.
+See [providers](http://www.getoutliers.com/documentation/providers), [resources](http://www.getoutliers.com/documentation/resources) and [filters](http://www.getoutliers.com/documentation/filters) for additional documentation.

data/lib/outliers/{credentials.rb → account.rb}

@@ -1,14 +1,14 @@
 module Outliers
-  module Credentials
+  module Account
     module_function
 
     def load_from_file(file)
-      credentials = {}
+      account = {}
       contents = File.read file
       YAML.load(contents).each_pair do |k,v|
-        credentials[k] = v
+        account[k] = v
       end
-      credentials
+      account
     end
 
   end

data/lib/outliers/cli/process.rb

@@ -2,7 +2,7 @@ module Outliers
   module CLI
     class Process
       def process
-        @options = { threads: 1 }
+        @options = { threads: 1, log_level: 'info' }
 
         option_parser.parse!
 
@@ -16,31 +16,53 @@ module Outliers
           @run.thread_count = @options[:threads]
         end
 
+        log_level = @options.fetch(:log_level).upcase
+
+        unless ["DEBUG", "INFO", "WARN", "ERROR"].include? log_level
+          @logger.error "Invalid log level. Valid levels are debug, info, warn, error."
+          exit 1
+        end
+
+        @logger.level = Logger.const_get log_level
+
         begin
-          @run.credentials = Credentials.load_from_file "#{ENV['HOME']}/.outliers.yml"
+          @run.account = Account.load_from_file "#{ENV['HOME']}/.outliers.yml"
           @run.process_evaluations_in_dir
         rescue Outliers::Exceptions::Base => e
           @logger.error e.message
           exit 1
         end
 
-        passed = @run.passed.count
-        failed = @run.failed.count
+        passing_count = @run.passing_results.count
+        failing_count = @run.failing_results.count
 
         @logger.info "Evaluations completed."
 
-        @run.failed.each do |f|
-          if f.evaluation
-            @logger.info "Evaluation '#{f.evaluation}' verification '#{f.verification}' of '#{f.resource}' failed."
+        if key
+          @logger.info "Running report handlers."
+          @run.results.each do |result|
+            unless Outliers::Handlers::OutliersApi.new.post result, key, results_url
+              @logger.error "Report handler failed."
+              exit 1
+            end
+          end
+          @logger.info "Report handlers completed."
+        else
+          @logger.info "OUTLIERS_KEY not set, not sending results."
+        end
+
+        @run.failing_results.each do |r|
+          if r.name
+            @logger.info "Results of '#{r.name}', verifying '#{r.verification_name}' of '#{r.provider_name}:#{r.resource_name}' via '#{r.account_name}' failed."
           else
-            @logger.info "Verification '#{f.verification}' of '#{f.resource}' failed."
+            @logger.info "Verification '#{r.verification_name}' of '#{r.provider_name}:#{r.resource_name}' via '#{r.account_name}' failed."
           end
-          @logger.info "Failing resource IDs '#{f.failing_resources.map{|r| r.id}.join(', ')}'"
+          @logger.info "Failing resource IDs '#{r.failing_resources.map{|r| r.id}.join(', ')}'"
         end
 
-        @logger.info "(#{failed} evaluations failed, #{passed} evaluations passed.)"
+        @logger.info "(#{failing_count} evaluations failed, #{passing_count} evaluations passed.)"
 
-        exit 1 unless failed.zero?
+        exit 1 unless failing_count.zero?
       end
 
       def command_name
@@ -53,17 +75,33 @@ module Outliers
 
       private
 
+      def key
+        ENV['OUTLIERS_KEY']
+      end
+
+      def url
+        ENV['OUTLIERS_URL'] ||= 'https://api.getoutliers.com'
+      end
+
+      def results_url
+        "#{url}/results"
+      end
+
       def option_parser
         OptionParser.new do |opts|
           opts.banner = "Usage: outliers process [options]"
 
-          opts.on("-t", "--threads [THREADS]", "Maximum number of evaluations threads to run concurrently (Default: 1).") do |o|
-            @options[:threads] = o.to_i
-          end
-
           opts.on("-d", "--directory [DIRECTORY]", "Directory containing evaluations to load.") do |o|
             @options[:directory] = o
           end
+
+          opts.on("-l", "--log_level [LOG_LEVEL]", "Log level (Default: info).") do |o|
+            @options[:log_level] = o
+          end
+
+          opts.on("-t", "--threads [THREADS]", "Maximum number of evaluations threads to run concurrently (Default: 1).") do |o|
+            @options[:threads] = o.to_i
+          end
         end
       end
 

data/lib/outliers/cli.rb

@@ -31,7 +31,7 @@ module Outliers
       puts ''
       puts 'Append -h for help on specific subcommand.'
       puts ''
-      puts 'See http://brettweavnet.github.io/outliers for documentation.'
+      puts 'See http://www.getoutliers.com/documentation for documentation.'
       puts ''
 
       puts 'Commands:'

data/lib/outliers/collection.rb

@@ -43,36 +43,44 @@ module Outliers
     end
 
     def exclude_by_key(exclusions)
-      @logger.info "Excluding the following resources: '#{exclusions.join(',')}'."
       save = list.reject {|u| exclusions.include? u.public_send key}
       @list = save
     end
 
-    def filter(args)
+    def filter(action, args)
       name  = args.keys.first
       value = args.fetch name
 
-      logger.info "Applying filter '#{name}' with value '#{value}'."
-
       unless self.public_methods.include? "filter_#{name}".to_sym
         raise Exceptions::UnknownFilter.new "Unknown filter '#{name}'."
       end
 
       filtered_list = self.public_send "filter_#{name}", value
 
-      logger.warn "No resources match filter." unless filtered_list.any?
-
-      @list = filtered_list
+      case action
+      when 'include'
+        logger.info "Including resources filtered by '#{name}' with value '#{value}'."
+        logger.warn "No resources match filter." unless filtered_list.any?
+        @list = filtered_list & @list
+      when 'exclude'
+        logger.info "Excluding resources filtered by '#{name}' with value '#{value}'."
+        @list -= filtered_list
+      else
+        raise Exceptions::UnknownFilterAction.new "Filters must be either 'include' or 'exclude'."
+      end
     end
 
-    def verify(name, arguments={})
-      name << "?" unless name =~ /^.*\?$/
+    def verify(name, arguments=nil)
+      logger.debug "Verifying '#{name}'."
+
+      name += "?" unless name =~ /^.*\?$/
 
       unless list.any?
         return { failing_resources: [], passing_resources: [] }
       end
 
-      logger.info "Verifying '#{name}'."
+      set_target_resources name if targets.any?
+
       logger.debug "Target resources '#{list_by_key.join(', ')}'."
 
       unless verification_exists? name
@@ -124,7 +132,7 @@ module Outliers
     end
 
     def set_target_resources(verification)
-      logger.info "Verifying target '#{targets.join(', ')}'."
+      logger.debug "Setting target resource(s) to '#{targets.join(', ')}'."
 
       @list = list.select {|r| targets.include? r.id }
 
@@ -136,12 +144,10 @@ module Outliers
     end
 
     def send_resources_verification(verification, arguments)
-      set_target_resources verification if targets.any?
-
       failing_resources = reject do |resource|
-        result = send_verification resource, verification, arguments
-        logger.debug "Verification of resource '#{resource.id}' #{result ? 'passed' : 'failed'}."
-        result
+        r = send_verification resource, verification, arguments
+        logger.debug "Verification of resource '#{resource.id}' #{r ? 'passed' : 'failed'}."
+        r
       end
       { failing_resources: failing_resources, passing_resources: list - failing_resources }
     end
@@ -153,12 +159,12 @@ module Outliers
 
     def send_verification(object, verification, arguments) 
       if object.method(verification).arity.zero?
-        if arguments.any?
+        unless arguments.nil?
           raise Outliers::Exceptions::NoArgumentRequired.new "Verification '#{verification}' does not require an arguments."
         end
         object.public_send verification
       else
-        if arguments.none?
+        if arguments.nil?
           raise Outliers::Exceptions::ArgumentRequired.new "Verification '#{verification}' requires arguments."
         end
         object.public_send verification, arguments

data/lib/outliers/evaluation.rb

@@ -1,63 +1,92 @@
 module Outliers
   class Evaluation
 
-    attr_reader :collection, :provider_name, :provider_name_array
+    attr_reader :resource_collection, :provider_name, :provider_name_array
 
     def initialize(args)
       @run  = args[:run]
       @name = args[:name]
     end
 
-    def connect(name, options={})
-      @provider_name = merged_credentials(name, options).fetch 'provider'
+    def connect(account_name, options={})
+      @account_name  = account_name
+      @provider_name = merged_account(account_name, options).fetch 'provider'
 
-      logger.info "Connecting via '#{name}' to '#{@provider_name}'."
+      logger.info "Connecting via '#{account_name}' to '#{@provider_name}'."
       logger.info "Including connection options '#{options.map {|k,v| "#{k}=#{v}"}.join(',')}'." if options.any?
 
       set_provider_name_array
 
-      @provider = Outliers::Provider.connect_to merged_credentials(name, options)
+      @provider = Outliers::Provider.connect_to merged_account(account_name, options)
     end
 
-    def resources(name, targets=[])
-      logger.info "Loading '#{name}' resource collection."
-      @collection = collection_object name
+    def resources(name, targets=nil)
+      logger.debug "Loading '#{name}' resource collection."
 
-      targets_array = Array(targets)
+      @resource_name       = name
+      @resource_collection = collection_object name
 
-      if targets_array.any?
-        logger.info "Verifying '#{targets_array.join(', ')}' from '#{name}' collection."
-        collection.targets = targets_array
-      end
-      collection
+      load_targets targets
+      resource_collection
     end
 
-    def exclude(exclusions)
-      collection.exclude_by_key Array(exclusions)
+    def filter(action, args)
+      resource_collection.filter action, args.keys_to_s
     end
 
-    def filter(args)
-      collection.filter args.keys_to_s
-    end
+    def verify(verification_name, arguments=nil)
+      @resources_loaded ||= resource_collection.load_all
 
-    def verify(verification, arguments={})
-      @resources_loaded ||= collection.load_all
+      args_to_send = convert_verification_arguments arguments
 
-      verification_result = collection.verify verification, arguments.keys_to_sym
+      verification_result = resource_collection.verify verification_name, args_to_send
 
-      result = Outliers::Result.new evaluation:        @name,
+      result = Outliers::Result.new account_name:      @account_name,
                                     failing_resources: verification_result.fetch(:failing_resources),
+                                    name:              @name,
                                     passing_resources: verification_result.fetch(:passing_resources),
-                                    resource:          @collection,
-                                    verification:      verification
+                                    arguments:         Array(args_to_send),
+                                    provider_name:     @provider_name,
+                                    resource_name:     @resource_name,
+                                    verification_name: verification_name
 
-      logger.info "Verification '#{verification}' #{result}."
+      logger.info "Verification '#{verification_name}' #{result.passed? ? 'passed' : 'failed'}."
 
       @run.results << result
     end
 
     private
 
+    def load_targets(targets)
+      case targets.class.to_s
+      when "Hash"
+        t = targets.keys_to_sym
+        if t.has_key? :include
+          list = Array(t.fetch :include)
+          logger.info "Targeting '#{list.join(', ')}' from '#{@resource_name}' collection."
+          resource_collection.targets = list
+        elsif t.has_key? :exclude
+          list = Array(t.fetch :exclude)
+          logger.info "Excluding '#{list.join(', ')}' from '#{@resource_name}' collection."
+          resource_collection.exclude_by_key list
+        else
+          logger.info "Targeting all resources in '#{@resource_name}' collection."
+        end
+      when "String", "Array"
+        list = Array(targets)
+        logger.info "Targeting '#{list.join(', ')}' from '#{@resource_name}' collection."
+        resource_collection.targets = list
+      when "Nil"
+        logger.info "Targeting all resources in '#{@resource_name}' collection."
+      end
+    end
+
+    def convert_verification_arguments(arguments)
+      return Array(arguments) if arguments.is_a?(Array) || arguments.is_a?(String)
+      return nil if arguments.is_a?(NilClass)
+      raise Outliers::Exceptions::InvalidArguments.new "Verification arguments '#{arguments}' invalid. Must be a string or array."
+    end
+
     def set_provider_name_array
       begin
         array = Outliers::Providers.name_map.fetch(provider_name).to_s.split('::')
@@ -75,13 +104,13 @@ module Outliers
       raise Outliers::Exceptions::UnknownCollection.new "Unknown collection '#{name}'."
     end
 
-    def credentials(name)
-      @run.credentials.fetch name
+    def account(name)
+      @run.account.fetch name
     end
 
-    def merged_credentials(name, options)
-      credentials(name).merge! options.keys_to_s
-      credentials(name).merge :name => name
+    def merged_account(name, options)
+      account(name).merge! options.keys_to_s
+      account(name).merge :name => name
     end
 
     def logger
@@ -90,4 +119,3 @@ module Outliers
 
   end
 end
-

data/lib/outliers/exceptions.rb

@@ -12,16 +12,22 @@ module Outliers
     class ArgumentRequired < Base
     end
 
+    class HandlerError < Base
+    end
+
     class InvalidBucket < Base
     end
 
+    class InvalidArguments < Base
+    end
+
     class NoArgumentRequired < Base
     end
 
     class UnknownCollection < Base
     end
 
-    class UnknownCredentials < Base
+    class UnknownAccount < Base
     end
 
     class UnknownVerification < Base
@@ -30,6 +36,9 @@ module Outliers
     class UnknownFilter < Base
     end
 
+    class UnknownFilterAction < Base
+    end
+
     class UnknownProvider < Base
     end
 

data/lib/outliers/filters/aws/ec2/tags.rb

@@ -7,12 +7,12 @@ module Outliers
           def filter_tag(value)
             tag_name  = value.split(':').first
             tag_value = value.split(':').last
-            logger.info "Filtering by tag '#{tag_name}' equals '#{tag_value}'."
+            logger.info "Loading filter by tag '#{tag_name}' equals '#{tag_value}'."
             list.select do |r|
               if r.tags.has_key? tag_name
                 value = r.tags[tag_name]
                 result = value == tag_value
-                logger.debug "'#{r.id}' has tag with value '#{value}'. #{result ? 'Matches' : 'Does not match'} filter."
+                logger.debug "'#{r.id}' has tag '#{tag_name}' with value '#{value}'. #{result ? 'Matches' : 'Does not match'} filter."
                 result
               else
                 logger.debug "'#{r.id}' does not have tag '#{tag_name}'"

data/lib/outliers/handlers/json.rb

@@ -0,0 +1,36 @@
+require 'net/http'
+
+module Outliers
+  module Handlers
+    class JSON
+      def post(result)
+        host = 'localhost'
+        port = '3000'
+        path = '/results'
+
+        req = Net::HTTP::Post.new(path, initheader = { 'Content-Type' => 'application/json',
+                                                       'Accept'       => 'application/vnd.outliers-v1+json' })
+        req.body = result.to_json
+        
+        response = Net::HTTP.new(host, port).start {|http| http.request(req) }
+
+        logger.debug response.body
+
+        if response
+          logger.debug "Handler completed succesfully."
+          true
+        else
+          logger.debug "Handler failed."
+          false
+        end
+      end
+
+      private
+
+      def logger
+        @logger ||= Outliers.logger
+      end
+
+    end
+  end
+end

data/lib/outliers/handlers/outliers_api.rb

@@ -0,0 +1,62 @@
+require 'json'
+require 'net/http'
+require 'uri'
+
+module Outliers
+  module Handlers
+    class OutliersApi
+      def post(result, key, url)
+        uri = URI.parse url
+
+        host = uri.host
+        port = uri.port
+        path = uri.path
+        use_ssl = uri.scheme == 'https'
+
+        req = Net::HTTP::Post.new path, header
+        
+        req.body = body(key, result)
+
+        logger.debug "Hanlder URL: #{url}"
+        logger.debug "Posting: #{body("XXX", result)}"
+
+        session = Net::HTTP.new(host, port)
+        session.use_ssl = use_ssl
+
+        begin
+          response = session.start {|http| http.request(req) }
+        rescue Errno::ECONNREFUSED
+          logger.error "Connection to '#{url}' refused."
+          return false
+        end
+
+        if response.code == "200"
+          logger.debug "Received: #{response.body}"
+          logger.debug "Handler completed succesfully."
+          true
+        else
+          logger.error "Received: #{response.body}"
+          logger.error "Handler failed with code #{response.code}."
+          false
+        end
+      end
+
+      private
+
+      def logger
+        @logger ||= Outliers.logger
+      end
+
+      def header
+        { 'Content-Type' => 'application/json',
+          'Accept'       => 'application/vnd.outliers-v1+json' }
+      end
+
+      def body(key, result)
+        { 'key'    => key,
+          'result' => result.to_hash }.to_json
+      end
+
+    end
+  end
+end

data/lib/outliers/handlers.rb

@@ -0,0 +1 @@
+require 'outliers/handlers/outliers_api'