cfn-guardian 0.4.0 → 0.6.6

data/docs/notifiers.md

@@ -0,0 +1,33 @@
+# Guardian Notifiers
+
+## SNS Notification
+
+There are 4 default notification levels used by Guardian Critical, Warning, Task, Informational. If you wish to recieve notifications for each of these you need to supply an sns topic arn in the alarms.yaml
+
+```yaml
+Topics:
+  Critical: arn:aws:sns:ap-southeast-2:123456789012:Critical
+  Warning: arn:aws:sns:ap-southeast-2:123456789012:Warning
+  Task: arn:aws:sns:ap-southeast-2:123456789012:Task
+  Informational: arn:aws:sns:ap-southeast-2:123456789012:Informational
+```
+
+Each alarm has a default notification level but can be overriden in the config using the `AlarmAction` property at either the alarm group or alarm level. See the [Overriding Defaults](#overriding-defaults) section on how to do that.
+
+You can add your own notification topics to the topics section and combine them with the existing topics. `AlarmAction` property will accept both a string and array of notication topics.
+
+```yaml
+Topics:
+  Critical: arn:aws:sns:ap-southeast-2:123456789012:Critical
+  Warning: arn:aws:sns:ap-southeast-2:123456789012:Warning
+  Task: arn:aws:sns:ap-southeast-2:123456789012:Task
+  Informational: arn:aws:sns:ap-southeast-2:123456789012:Informational
+  Custom: arn:aws:sns:ap-southeast-2:123456789012:Custom
+
+Template:
+  Ec2Instance:
+    GroupOverrides:
+      AlarmActions:
+      - Critical
+      - Custom
+```

data/docs/overview.md

@@ -0,0 +1,22 @@
+# Guardian Documentation
+
+## Table of Contents
+1. [CLI Commands](cli.md)
+2. [Resources](resources.md)
+3. [Alarm Templates](alarm_templates.md)
+4. Custom Checks
+    1. [HTTP](custom_checks/http.md)
+    2. [Domain Expirey](custom_checks/domain_expirey.md)
+    3. [LogGroup Metric Filters](custom_checks/log_group_metric_filters.md)
+    4. [NRPE](custom_checks/nrpe.md)
+    5. [Port](custom_checks/port.md)
+    6. [SFTP](custom_checks/sftp.md)
+    7. [SQL](custom_checks/sql.md)
+    8. [TLS](custom_checks/tls.md)
+    9. [Azure File Check](custom_checks/azure_file_check.md)
+5. [Event Subscriptions](event_subscriptions.md)
+6. [Notifiers](notifiers.md)
+7. [Maintenance Mode](maintenance_mode.md)
+8. [Composite Alarms](composite_alarms.md)
+9. [Alarms for Custom Metrics](custom_metrics.md)
+10. [Dimension Variables](variables.md)

data/docs/resources.md

@@ -0,0 +1,93 @@
+# Resources
+
+Resources are AWS resources grouped by the resource type such as `RDSInstance`, `Ec2Instance`, `ApplicationTargetGroup` etc. These are defined under the top level key `Resources` in the yaml config file. The resource group is then used to generate standard set of alarms.
+
+Custom resource groups can be created however matching alarm templates must be created to create alarms.
+
+## Resource lookup
+
+Resources can be looked up within an account using the tool [monitorable](https://github.com/base2Services/monitorable). This tool will scan every region within an account for AWS resources that can be monitored and return a valid Guardian yaml config using the `--format cfn-guardian` flag.
+
+```sh
+./monitorable.py --format cfn-guardian
+```
+
+## YAML Configuration
+
+The resources key is where the resources are defined.
+
+```yaml
+Resources:
+  # resource group
+  Ec2Instance:
+  # Array of resources defining the resource id with the Id: key
+  - Id: i-1a2b3c4d5e
+```
+
+There are some resources that require more that the resource id to generate the alarm, for these cases addition key:values are required.
+
+```yaml
+Resources:
+  ApplicationTargetGroup:
+  - Id: target-group-id
+    # Target group requires the loadbalancer id for the alarm
+    Loadbalancer: app/application-loadbalancer-id
+```
+
+| Resource Group              | Require Keys     |
+| --------------------------- | ---------------- |
+| ApiGateway                  | Id               |
+| AmazonMQBroker              | Id               |
+| AutoScalingGroup            | Id               |
+| DynamoDBTable               | Id               |
+| ElastiCacheReplicationGroup | Id               |
+| ElasticFileSystem           | Id               |
+| Ec2Instance                 | Id               |
+| EcsCluster                  | Id               |
+| EcsService                  | Id, Cluster      |
+| NetworkTargetGroup          | Id, LoadBalancer |
+| ApplicationTargetGroup      | Id, LoadBalancer |
+| ElasticLoadBalancer         | Id               |
+| RDSInstance                 | Id               |
+| RDSClusterInstance          | Id               |
+| RedshiftCluster             | Id               |
+| Lambda                      | Id               |
+| CloudFrontDistribution      | Id               |
+| SQSQueue                    | Id               |
+
+
+## Custom Resource Groups
+
+You may want to create a custom resource group if some of the resources require differewnt alarm configurations. To create a custom resource group create new name for the group and add the resources, then create the alarm template and inherit the desired alarms.
+
+```yaml
+Resources:
+  # default resource group
+  Ec2Instance:
+  - Id: i-1a2b3c4d5e
+  - Id: i-9z8y7x6w5v
+  # custom resource group
+  CustomEc2Instance:
+  - Id: i-6fefg5qe4e
+
+Templates:
+  # create a new alarm template with the same group name 
+  CustomEc2Instance:
+    # inherit the ec2 alarms
+    Inherit: Ec2Instance
+    # alter the alarms
+    CPUUtilizationHigh: false
+```
+
+## Friendly Resource Names
+
+You can set a friendly name which will replace the resource id in the alarm name.
+The resource id will still be available in the alarm description.
+
+```yaml
+Resources:
+  ApplicationTargetGroup:
+  - Id: target-group-id
+    Loadbalancer: app/application-loadbalancer-id
+    Name: webapp
+```

data/docs/variables.md

@@ -0,0 +1,58 @@
+## Dimension Variables
+
+variables can be used to reference resource group values such as the resource Id within the dimensions section of an alarm template.
+
+For example here we are creating an alarm for a disk usage metric for a group of EC2 instances.
+
+```yaml
+Templates:
+  Ec2Instance:
+    LowDiskSpaceRootVolume:
+      Namespace: CWAgent
+      MetricName: DiskSpaceUsedPercent
+      Dimensions:
+        path: '/'
+        # Reference the resource Id from the resource group
+        host: ${Resource::Id}
+        device: 'xvda1'
+        fstype: 'ext4'
+      Statistic: Maximum
+      Threshold: 85  
+      Period: 60
+      EvaluationPeriods: 1
+      TreatMissingData: breaching
+
+Resources:
+  Ec2Instance:
+    - Id: i-12345678
+    - Id: i-abcdefgh
+```
+
+custom variables can be referenced if you have different dimensions for each resource. using the example above, you may have different file system types on each instance. 
+
+```yaml
+Templates:
+  Ec2Instance:
+    LowDiskSpaceRootVolume:
+      Namespace: CWAgent
+      MetricName: DiskSpaceUsedPercent
+      Dimensions:
+        path: '/'
+        # Reference the resource Id from the resource group
+        host: ${Resource::Id}
+        device: 'xvda1'
+        # Reference the resource FileSystemType from the resource group
+        fstype: ${Resource::FileSystemType}
+      Statistic: Maximum
+      Threshold: 85  
+      Period: 60
+      EvaluationPeriods: 1
+      TreatMissingData: breaching
+
+Resources:
+  Ec2Instance:
+    - Id: i-12345678
+      FileSystemType: ext4
+    - Id: i-abcdefgh
+      FileSystemType: ext4
+```

data/lib/cfnguardian.rb

@@ -33,6 +33,13 @@ module CfnGuardian
     method_option :bucket, type: :string, desc: "provide custom bucket name, will create a default bucket if not provided"
     method_option :path, type: :string, default: "guardian", desc: "S3 path location for nested stacks"
     method_option :region, aliases: :r, type: :string, desc: "set the AWS region"
+    method_option :template_config, type: :boolean, default: false, desc: "Genrates an AWS CodePipeline cloudformation template configuration file to override parameters"
+    method_option :sns_critical, type: :string, desc: "sns topic arn for the critical alarms"
+    method_option :sns_warning, type: :string, desc: "sns topic arn for the warning alarms"
+    method_option :sns_task, type: :string, desc: "sns topic arn for the task alarms"
+    method_option :sns_informational, type: :string, desc: "sns topic arn for the informational alarms"
+    method_option :sns_events, type: :string, desc: "sns topic arn for the informational alarms"
+
 
     def compile
       set_log_level(options[:debug])
@@ -43,7 +50,7 @@ module CfnGuardian
       compiler = CfnGuardian::Compile.new(options[:config])
       compiler.get_resources
       compiler.compile_templates(s3.bucket,s3.path)
-      logger.info "Clouformation templates compiled successfully in out/ directory"
+      logger.info "Cloudformation templates compiled successfully in out/ directory"
       if options[:validate]
         s3.create_bucket_if_not_exists()
         validator = CfnGuardian::Validate.new(s3.bucket)
@@ -51,10 +58,16 @@ module CfnGuardian
           logger.error("One or more templates failed to validate")
           exit(1)
         else
-          logger.info "Clouformation templates were validated successfully"
+          logger.info "Cloudformation templates were validated successfully"
         end
       end
       logger.warn "AWS cloudwatch alarms defined in the templates will cost roughly $#{'%.2f' % compiler.cost} per month"
+
+      if options[:template_config]
+        logger.info "Generating a AWS CodePipeline template configuration file template-config.guardian.json"
+        parameters = compiler.load_parameters(options)
+        compiler.genrate_template_config(parameters)
+      end
     end
 
     desc "deploy", "Generates and deploys monitoring CloudFormation templates"
@@ -67,10 +80,11 @@ module CfnGuardian
     method_option :path, type: :string, default: "guardian", desc: "S3 path location for nested stacks"
     method_option :region, aliases: :r, type: :string, desc: "set the AWS region"
     method_option :stack_name, aliases: :s, type: :string, desc: "set the Cloudformation stack name. Defaults to `guardian`"
-    method_option :sns_critical, type: :string, desc: "sns topic arn for the critical alamrs"
-    method_option :sns_warning, type: :string, desc: "sns topic arn for the warning alamrs"
-    method_option :sns_task, type: :string, desc: "sns topic arn for the task alamrs"
-    method_option :sns_informational, type: :string, desc: "sns topic arn for the informational alamrs"
+    method_option :sns_critical, type: :string, desc: "sns topic arn for the critical alarms"
+    method_option :sns_warning, type: :string, desc: "sns topic arn for the warning alarms"
+    method_option :sns_task, type: :string, desc: "sns topic arn for the task alarms"
+    method_option :sns_informational, type: :string, desc: "sns topic arn for the informational alarms"
+    method_option :sns_events, type: :string, desc: "sns topic arn for the informational alarms"
 
     def deploy
       set_log_level(options[:debug])
@@ -81,7 +95,8 @@ module CfnGuardian
       compiler = CfnGuardian::Compile.new(options[:config])
       compiler.get_resources
       compiler.compile_templates(s3.bucket,s3.path)
-      logger.info "Clouformation templates compiled successfully in out/ directory"
+      parameters = compiler.load_parameters(options)
+      logger.info "Cloudformation templates compiled successfully in out/ directory"
 
       s3.create_bucket_if_not_exists
       validator = CfnGuardian::Validate.new(s3.bucket)
@@ -89,10 +104,10 @@ module CfnGuardian
         logger.error("One or more templates failed to validate")
         exit(1)
       else
-        logger.info "Clouformation templates were validated successfully"
+        logger.info "Cloudformation templates were validated successfully"
       end
       
-      deployer = CfnGuardian::Deploy.new(options,s3.bucket)
+      deployer = CfnGuardian::Deploy.new(options,s3.bucket,parameters)
       deployer.upload_templates
       change_set, change_set_type = deployer.create_change_set()
       deployer.wait_for_changeset(change_set.id)
@@ -135,14 +150,11 @@ module CfnGuardian
     method_option :config, aliases: :c, type: :string, desc: "yaml config file"
     method_option :defaults, type: :boolean, desc: "display default alarms and properties"
     method_option :region, aliases: :r, type: :string, desc: "set the AWS region"
-    method_option :group, aliases: :g, type: :string, desc: "resource group"
-    method_option :alarm, aliases: :a, type: :string, desc: "alarm name"
-    method_option :id, type: :string, desc: "resource id"
+    method_option :filter, type: :hash, default: {}, desc: "filter the displayed alarms by [group, resource-id, alarm, stack-id, topic, maintenance-group]"
     method_option :compare, type: :boolean, default: false, desc: "compare config to deployed alarms"
     
     def show_alarms
       set_log_level(options[:debug])
-      
       set_region(options[:region],options[:compare])
       
       if options[:config]
@@ -156,7 +168,7 @@ module CfnGuardian
       
       compiler = CfnGuardian::Compile.new(config_file)
       compiler.get_resources
-      alarms = filter_alarms(compiler.alarms,options)
+      alarms = filter_compiled_alarms(compiler.alarms,options[:filter])
 
       if alarms.empty?
         logger.error "No matches found" 
@@ -167,13 +179,15 @@ module CfnGuardian
       formatter = CfnGuardian::DisplayFormatter.new(alarms)
       
       if options[:compare] && !options[:defaults]
-        metric_alarms = CfnGuardian::CloudWatch.get_alarms(alarms)
+        metric_alarms = CfnGuardian::CloudWatch.get_alarms_by_prefix(prefix: 'guardian')
+        metric_alarms = CfnGuardian::CloudWatch.filter_alarms(filters: options[:filter], alarms: metric_alarms)
+
         formatted_alarms = formatter.compare_alarms(metric_alarms)
         headings.push('Deployed')
       else
         formatted_alarms = formatter.alarms()
       end
-      
+
       if formatted_alarms.any?
         formatted_alarms.each do |fa|
           puts Terminal::Table.new( 
@@ -192,37 +206,34 @@ module CfnGuardian
     
     desc "show-state", "Shows alarm state in cloudwatch"
     long_desc <<-LONG
-    Displays the current cloudwatch alarm state
+    Displays the current cloudwatch alarm state. By default it will return all the guardian alarms.
     LONG
-    method_option :config, aliases: :c, type: :string, desc: "yaml config file"
     method_option :region, aliases: :r, type: :string, desc: "set the AWS region"
-    method_option :group, aliases: :g, type: :string, desc: "resource group"
-    method_option :alarm, aliases: :a, type: :string, desc: "alarm name"
-    method_option :id, type: :string, desc: "resource id"
     method_option :state, aliases: :s, type: :string, enum: %w(OK ALARM INSUFFICIENT_DATA), desc: "filter by alarm state"
-    method_option :alarm_names, type: :array, desc: "CloudWatch alarm name if not providing config"
-    method_option :alarm_prefix, type: :string, desc: "CloudWatch alarm name prefix if not providing config"
-    
+    method_option :alarm_names, type: :array, desc: "list of cloudwatch alarm names"
+    method_option :alarm_prefix, type: :string, default: "guardian", desc: "cloudwatch alarm name prefix"
+    method_option :filter, type: :hash, default: {}, desc: "filter the displayed alarms by [group, resource-id, alarm, stack-id, topic, maintenance-group]"
+
     def show_state
       set_log_level(options[:debug])
       set_region(options[:region],true)
-      
-      formatter = CfnGuardian::DisplayFormatter.new()
-      
-      if !options[:config].nil?
-        compiler = CfnGuardian::Compile.new(options[:config])
-        compiler.get_resources
-        alarms = filter_alarms(compiler.alarms,options)
-        metric_alarms = CfnGuardian::CloudWatch.get_alarm_state(config_alarms: alarms, state: options[:state])
-      elsif !options[:alarm_names].nil?
-        metric_alarms = CfnGuardian::CloudWatch.get_alarm_state(alarm_names: options[:alarm_names], state: options[:state])
-      elsif !options[:alarm_prefix].nil?
-        metric_alarms = CfnGuardian::CloudWatch.get_alarm_state(alarm_prefix: options[:alarm_prefix], state: options[:state])
+      action_prefix = nil
+
+      if options[:filter].has_key?('topic')
+        action_prefix = get_topic_arn_from_stack(options[:filter]['topic'])
+      elsif options[:filter].has_key?('maintenance-group')
+        action_prefix = "arn:aws:sns:#{Aws.config[:region]}:#{CfnGuardian::CloudWatch.aws_account_id()}:#{options[:filter]['maintenance-group']}MaintenanceGroup"
+      end
+
+      if options[:alarm_names]
+        metric_alarms = CfnGuardian::CloudWatch.get_alarms_by_name(alarm_names: options[:alarm_names], state: options[:state], action_prefix: action_prefix)
       else
-        logger.error "one of `--config` `--alarm-prefix` `--alarm-names` must be supplied"
-        exit 1
+        metric_alarms = CfnGuardian::CloudWatch.get_alarms_by_prefix(prefix: options[:alarm_prefix], state: options[:state], action_prefix: action_prefix)
       end
-      
+
+      metric_alarms = CfnGuardian::CloudWatch.filter_alarms(filters: options[:filter], alarms: metric_alarms)
+
+      formatter = CfnGuardian::DisplayFormatter.new()
       rows = formatter.alarm_state(metric_alarms)
       
       if rows.any?
@@ -239,31 +250,24 @@ module CfnGuardian
     long_desc <<-LONG
     Displays the alarm state or config history for the last 7 days
     LONG
-    method_option :config, aliases: :c, type: :string, desc: "yaml config file"
     method_option :region, aliases: :r, type: :string, desc: "set the AWS region"
-    method_option :group, aliases: :g, type: :string, desc: "resource group"
-    method_option :alarm, aliases: :a, type: :string, desc: "alarm name"
-    method_option :alarm_names, type: :array, desc: "CloudWatch alarm name if not providing config"
-    method_option :id, type: :string, desc: "resource id"
+    method_option :alarm_names, type: :array, desc: "list of cloudwatch alarm names"
     method_option :type, aliases: :t, type: :string, 
         enum: %w(state config), default: 'state', desc: "filter by alarm state"
+    method_option :alarm_prefix, type: :string, default: "guardian", desc: "cloudwatch alarm name prefix"
+    method_option :filter, type: :hash, desc: "filter the displayed alarms by [group, resource-id, alarm, stack-id]"
     
     def show_history
       set_log_level(options[:debug])
       set_region(options[:region],true)
       
-      if !options[:config].nil?
-        compiler = CfnGuardian::Compile.new(options[:config])
-        compiler.get_resources
-        config_alarms = filter_alarms(compiler.alarms,options)
-        alarms = config_alarms.map {|alarm| CfnGuardian::CloudWatch.get_alarm_name(alarm)}
-      elsif !options[:alarm_names].nil?
-        alarms = options[:alarm_names]
+      if options[:alarm_names]
+        metric_alarms = CfnGuardian::CloudWatch.get_alarms_by_name(alarm_names: options[:alarm_names], state: options[:state])
       else
-        logger.error "one of `--config` `--alarm-names` must be supplied"
-        exit 1
+        metric_alarms = CfnGuardian::CloudWatch.get_alarms_by_prefix(prefix: options[:alarm_prefix], state: options[:state])
       end
-        
+
+      metric_alarms = CfnGuardian::CloudWatch.filter_alarms(filters: options[:filter], alarms: metric_alarms)       
       
       case options[:type]
       when 'state'
@@ -276,12 +280,12 @@ module CfnGuardian
       
       formatter = CfnGuardian::DisplayFormatter.new()
       
-      alarms.each do |alarm|
-        history = CfnGuardian::CloudWatch.get_alarm_history(alarm,type)
+      metric_alarms.each do |alarm|
+        history = CfnGuardian::CloudWatch.get_alarm_history(alarm.alarm_name,type)
         rows = formatter.alarm_history(history,type)
         if rows.any?     
           puts Terminal::Table.new( 
-                  :title => alarm.green, 
+                  :title => alarm.alarm_name.green, 
                   :headings => headings, 
                   :rows => rows)
           puts "\n"
@@ -293,17 +297,20 @@ module CfnGuardian
     long_desc <<-LONG
     Shows the last 10 commits made to the codecommit repo
     LONG
-    method_option :config, aliases: :c, type: :string, desc: "yaml config file"
     method_option :region, aliases: :r, type: :string, desc: "set the AWS region"
     method_option :repository, type: :string, default: 'guardian', desc: "codecommit repository name"
+    method_option :branch, type: :string, default: 'master', desc: "codecommit branch"
+    method_option :count, type: :numeric, default: 10, desc: "number of last commits to retrieve"
     
     def show_config_history
       set_region(options[:region],true)
     
-      history = CfnGuardian::CodeCommit.new(options[:repository]).get_commit_history()
-      puts Terminal::Table.new(
-        :headings => history.first.keys.map{|h| h.to_s.to_heading}, 
-        :rows => history.map(&:values))
+      history = CfnGuardian::CodeCommit.new(options[:repository]).get_commit_history(options[:branch], options[:count])
+      if history.any?
+        puts Terminal::Table.new(
+          :headings => history.first.keys.map{|h| h.to_s.to_heading}, 
+          :rows => history.map(&:values))
+      end
     end
     
     desc "show-pipeline", "Shows the current state of the AWS code pipeline"
@@ -405,16 +412,27 @@ module CfnGuardian
       logger.level = debug ? Logger::DEBUG : Logger::INFO
     end
     
-    def filter_alarms(alarms,options)
-      alarms.select! {|alarm| alarm.group.downcase == options[:group].downcase} if options[:group]
-      alarms.select! {|alarm| alarm.resource_id.downcase == options[:id].downcase} if options[:id]
-      alarms.select! {|alarm| alarm.name.downcase.include? options[:alarm].downcase} if options[:alarm]
+    def filter_compiled_alarms(alarms,filters)
+      filters = filters.slice('group', 'resource', 'alarm', 'topic', 'maintenance-group')
+      alarms.select! {|alarm| alarm.group.downcase == filters['group'].downcase} if filters.has_key?('group')
+      alarms.select! {|alarm| alarm.resource_id.downcase == filters['resource'].downcase} if filters.has_key?('resource')
+      alarms.select! {|alarm| alarm.name.downcase.include? filters['alarm'].downcase} if filters.has_key?('alarm')
+      alarms.select! {|alarm| alarm.alarm_action.include? filters['topic']} if filters.has_key?('topic')
+      alarms.select! {|alarm| alarm.maintenance_groups.include? "#{filters['maintenance-group']}MaintenanceGroup"} if filters.has_key?('maintenance-group')
       return alarms
     end
     
     def default_config()
       return "#{File.expand_path(File.dirname(__FILE__))}/cfnguardian/config/defaults.yaml"
     end
+
+    def get_topic_arn_from_stack(topic)
+      client = Aws::CloudFormation::Client.new()
+      resp = client.describe_stacks({ stack_name: @stack_name })
+      stack = resp.stacks.first
+      parameter = stack.parameters.find {|p| p.parameter_key == topic}
+      return !parameter.nil? ? parameter.parameter_value : nil
+    end
     
   end
 end