cloud-mu 1.9.0.pre.beta

data/modules/mu/clouds/README.md

@@ -0,0 +1,201 @@
+This is where we implement provisioning layers. This can be a full-featured
+cloud provider, like Amazon Web Services, or a simple node virtualization
+or containerization layer, like VMWare or Docker.
+
+Implementing a New Cloud Resource Type
+--------------------------------------
+
+The following is a walkthrough for developers adding an entirely new cloud
+resource type (e.g. `Server`, `Alarm`, etc), one which has not yet been defined
+or implemented for **any** cloud layer. For this example we'll discuss the
+`Function` resource type, which in cloud provider terms will be an implemention
+for serverless code services such as AWS Lambda, Google Cloud Functions, etc.
+
+
+1. Add stubs to `modules/mu/cloud.rb`. Edit files file, and you'll see a
+`@@resource_types` hash describing each cloud resource type Mu supports. The
+configuration here governs a lot of Mu deployment engine behaviors, as well as
+interaction with the configuration parser. For our `Function` type, we'll keep
+it simple:
+
+
+```
+      :Function => {
+        :has_multiples => false,
+        :can_live_in_vpc => true,
+        :cfg_name => "function",
+        :cfg_plural => "functions",
+        :interface => self.const_get("Function"),
+        :deps_wait_on_my_creation => true,
+        :waits_on_parent_completion => false,
+        :class => generic_class_methods,
+        :instance => generic_instance_methods
+      }
+
+```
+
+2. Add our new type to the configuration schema in `modules/mu/config`. This is where our parser learns to look for our type when loading Basket of Kittens YAML or JSON files. Let's start by adding to the top-level of the `@@schema` hash, which is actually a [http://json-schema.org/](JSON Schema) definition.
+
+Each of these stubs must be subclass of `MU::Config` that implements a class with the same class name as you declared in step 1. This class must, at minimum, implement `self.schema` and `self.validate` methods. Minimal example:
+
+```
+module MU
+  class Config
+    class Function
+
+      def self.schema
+        {
+          "type" => "object",
+          "title" => "Functions",
+          "additionalProperties" => false,
+          "description" => "Create a serverless cloud function.",
+          "properties" => {
+            "cloud" => MU::Config.cloud_primitive,
+            "name" => {"type" => "string"},
+            "region" => MU::Config.region_primitive,
+            "vpc" => MU::Config::VPC.reference(MU::Config::VPC::ONE_SUBNET+MU::Config::VPC::MANY_SUBNETS, MU::Config::VPC::NO_NAT_OPTS, "all_private"),
+            "dependencies" => MU::Config.dependencies_primitive,
+            "tags" => MU::Config.tags_primitive,
+            "optional_tags" => {
+              "type" => "boolean",
+              "description" => "Tag the resource with our optional tags (MU-HANDLE, MU-MASTER-NAME, MU-OWNER). Defaults to true",
+            }
+          }
+        }
+      end
+
+      # Generic pre-processing of {MU::Config::BasketofKittens::functions}, bare and unvalidated.
+      # @param function [Hash]: The resource to process and validate
+      # @param configurator [MU::Config]: The overall deployment configurator of which this resource is a member
+      # @return [Boolean]: True if validation succeeded, False otherwise
+      def self.validate(function, configurator)
+        ok = true
+        ok
+      end
+
+    end
+  end
+end
+```
+
+
+3. Define an empty implemention. Remember in step 1 where we had a couple configuration variables in our little hash (`:class => generic_class_methods`, `:instance => generic_instance_methods`)? Well those are lists of class methods and instance methods that any implemention will be required to have, no matter which
+cloud provider it's for.
+
+Looking elsewhere in `cloud.rb` let's see what all we have to do:
+
+```
+    generic_class_methods = [:find, :cleanup, :validateConfig, :schema]
+    generic_instance_methods = [:create, :notify, :mu_name, :cloud_id, :config]
+```
+
+Just the basics, for now. Here's what that will look like in the AWS layer, in the file `modules/mu/clouds/aws/function.rb`:
+
+```
+module MU
+  class Cloud
+    class AWS
+      # A function as configured in {MU::Config::BasketofKittens::functions}
+      class Function < MU::Cloud::Function
+        @deploy = nil
+        @config = nil
+        attr_reader :mu_name
+        attr_reader :config
+        attr_reader :cloud_id
+
+        @cloudformation_data = {}
+        attr_reader :cloudformation_data
+
+        # @param mommacat [MU::MommaCat]: A {MU::Mommacat} object containing the deploy of which this resource is/will be a member.
+        # @param kitten_cfg [Hash]: The fully parsed and resolved {MU::Config} resource descriptor as defined in {MU::Config::BasketofKittens::functions}
+        def initialize(mommacat: nil, kitten_cfg: nil, mu_name: nil, cloud_id: nil)
+          @deploy = mommacat
+          @config = MU::Config.manxify(kitten_cfg)
+          @cloud_id ||= cloud_id
+          @mu_name ||= @deploy.getResourceName(@config["name"])
+        end
+
+        # Called automatically by {MU::Deploy#createResources}
+        def create
+        end
+
+        # Return the metadata for this Function rule
+        # @return [Hash]
+        def notify
+          deploy_struct = {
+          }
+          return deploy_struct
+        end
+
+        # Remove all functions associated with the currently loaded deployment.
+        # @param noop [Boolean]: If true, will only print what would be done
+        # @param ignoremaster [Boolean]: If true, will remove resources not flagged as originating from this Mu server
+        # @param region [String]: The cloud provider region
+        # @return [void]
+        def self.cleanup(noop: false, ignoremaster: false, region: MU.curRegion, flags: {})
+        end
+
+        # Locate an existing function.
+        # @param cloud_id [String]: The cloud provider's identifier for this resource.
+        # @param region [String]: The cloud provider region.
+        # @param flags [Hash]: Optional flags
+        # @return [OpenStruct]: The cloud provider's complete descriptions of matching function.
+        def self.find(cloud_id: nil, region: MU.curRegion, flags: {})
+        end
+
+        # Cloud-specific configuration properties.
+        # @param config [MU::Config]: The calling MU::Config object
+        # @return [Array<Array,Hash>]: List of required fields, and json-schema Hash of cloud-specific configuration parameters for this resource
+        def self.schema(config)
+          toplevel_required = []
+          schema = {}
+          [toplevel_required, schema]
+        end
+
+        # Cloud-specific pre-processing of {MU::Config::BasketofKittens::functions}, bare and unvalidated.
+        # @param function [Hash]: The resource to process and validate
+        # @param configurator [MU::Config]: The overall deployment configurator of which this resource is a member
+        # @return [Boolean]: True if validation succeeded, False otherwise
+        def self.validateConfig(function, configurator)
+          ok = true
+#          if something_bad
+#            ok = false
+#          end
+
+          ok
+        end
+
+      end
+    end
+  end
+end
+```
+
+As you press forward in your implementation, you may find that it makes sense
+to add other requirements to this resource type.  For example, many resource
+types, such as `Servers`, have a `groom` phase that is invoked after initial
+creation, but which can be run asynchronously with dependent child resources.
+Others have methods that are peculiar to the resource type, such as the
+`addRule` method for the `FirewallRule` type.
+
+Most of these methods are self-explanatory, and you can look at other resource types for examples of actual implementation.
+
+A bit about the purpose of the `self.schema` class method. This is for
+injecting cloud-specific properties into the schema we defined in
+step 2. Cloud providers often have nuanced functionality that doesn't map to
+similar behavior in other providers, so we insert that here in the cloud layer
+rather than the global config schema. Here's an example in the AWS `Server` implementation:
+
+```
+        def self.schema(config)
+          toplevel_required = []
+          schema = {
+            "ami_id" => {
+              "type" => "string",
+              "description" => "The Amazon EC2 AMI on which to base this instance. Will use the default appropriate for the platform, if not specified."
+            }
+          }
+          [toplevel_required, schema]
+        end
+
+```

data/modules/mu/clouds/aws/alarm.rb

@@ -0,0 +1,319 @@
+# Copyright:: Copyright (c) 2014 eGlobalTech, Inc., all rights reserved
+#
+# Licensed under the BSD-3 license (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License in the root of the project or at
+#
+#     http://egt-labs.com/mu/LICENSE.html
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module MU
+  class Cloud
+    class AWS
+      # A alarm as configured in {MU::Config::BasketofKittens::alarms}
+      class Alarm < MU::Cloud::Alarm
+        @deploy = nil
+        @config = nil
+        attr_reader :mu_name
+        attr_reader :config
+        attr_reader :cloud_id
+
+        @cloudformation_data = {}
+        attr_reader :cloudformation_data
+
+        # @param mommacat [MU::MommaCat]: A {MU::Mommacat} object containing the deploy of which this resource is/will be a member.
+        # @param kitten_cfg [Hash]: The fully parsed and resolved {MU::Config} resource descriptor as defined in {MU::Config::BasketofKittens::alarms}
+        def initialize(mommacat: nil, kitten_cfg: nil, mu_name: nil, cloud_id: nil)
+          @deploy = mommacat
+          @config = MU::Config.manxify(kitten_cfg)
+          @cloud_id ||= cloud_id
+          @mu_name ||= @deploy.getResourceName(@config["name"])
+        end
+
+        # Called automatically by {MU::Deploy#createResources}
+        def create
+          if @config["dimensions"]
+            dimensions = []
+            @config["dimensions"].each { |dimension|
+              cloudid = 
+                if dimension["name"] and dimension["depclass"]
+                  if @dependencies.has_key?(dimension["depclass"])
+                    @dependencies[dimension["depclass"]][dimension["name"]].cloudobj.cloud_id
+                  end
+                elsif dimension["mu_name"] and dimension["deploy_id"]
+                  found = MU::MommaCat.findStray("AWS", deps_class, deploy_id: dimension["deploy_id"], mu_name: dimension["mu_name"], region: @config["region"])
+                  raise MuError, "Couldn't find #{deps_class} #{dimension["mu_name"]}" if found.nil? || found.empty?
+                  resp = found.first.deploydata["cloud_id"]
+                  resp.downcase if %w{database cache_cluster}.include?(deps_class)
+                else
+                  dimension["cloud_id"]
+                end
+              dimensions << {:name => dimension["cloud_class"], :value => cloudid}
+            }
+            @config["dimensions"] = dimensions
+          end
+
+          if @config["enable_notifications"]
+            @config["alarm_actions"] = [] if @config["alarm_actions"].nil?
+            @config["ok_actions"] = [] if @config["ok_actions"].nil?
+
+            topic_arn = MU::Cloud::AWS::Notification.createTopic(@config["notification_group"], region: @config["region"])
+            MU::Cloud::AWS::Notification.subscribe(arn: topic_arn, protocol: @config["notification_type"], endpoint: @config["notification_endpoint"], region: @config["region"])
+
+            @config["alarm_actions"] << topic_arn
+            @config["ok_actions"] << topic_arn
+          end
+          @config["ok_actions"].uniq!
+          @config["alarm_actions"].uniq!
+
+          MU::Cloud::AWS::Alarm.setAlarm(
+            name: @mu_name,
+            ok_actions: @config["ok_actions"],
+            alarm_actions: @config["alarm_actions"],
+            insufficient_data_actions: @config["no_data_actions"],
+            metric_name: @config["metric_name"],
+            namespace: @config["namespace"],
+            statistic: @config["statistic"],
+            dimensions: @config["dimensions"],
+            period: @config["period"],
+            unit: @config["unit"],
+            evaluation_periods: @config["evaluation_periods"],
+            threshold: @config["threshold"],
+            comparison_operator: @config["comparison_operator"],
+            region: @config["region"]
+          )
+
+          @cloud_id = @mu_name
+        end
+
+        # Canonical Amazon Resource Number for this resource
+        # @return [String]
+        def arn
+          cloud_desc.alarm_arn
+        end
+
+        # Return the metadata for this Alarm rule
+        # @return [Hash]
+        def notify
+          deploy_struct = {
+            "ok_actions" => @config["ok_actions"],
+            "alarm_actions" => @config["alarm_actions"],
+            "insufficient_data_actions" => @config["no_data_actions"],
+            "metric_name" => @config["metric_name"],
+            "namespace" => @config["namespace"],
+            "statistic" => @config["statistic"],
+            "dimensions" => @config["dimensions"],
+            "period" => @config["period"],
+            "unit" => @config["unit"],
+            "evaluation_periods" => @config["evaluation_periods"],
+            "threshold" => @config["threshold"],
+            "comparison_operator" => @config["comparison_operator"]
+          }
+          return deploy_struct
+        end
+
+        # Remove all alarms associated with the currently loaded deployment.
+        # @param noop [Boolean]: If true, will only print what would be done
+        # @param ignoremaster [Boolean]: If true, will remove resources not flagged as originating from this Mu server
+        # @param region [String]: The cloud provider region
+        # @return [void]
+        def self.cleanup(noop: false, ignoremaster: false, region: MU.curRegion, flags: {})
+          alarms = []
+          # We don't have a way to tag alarms, so we try to delete them by the deploy ID. 
+          # This can miss alarms in some cases (eg. cache_cluster) so we might want to delete alarms from each API as well.
+          MU::Cloud::AWS.cloudwatch(region).describe_alarms.each { |page|
+            page.metric_alarms.map(&:alarm_name).each { |alarm_name|
+              alarms << alarm_name if alarm_name.match(MU.deploy_id)
+            }
+          }
+
+          if !alarms.empty?
+            MU::Cloud::AWS.cloudwatch(region).delete_alarms(alarm_names: alarms) unless noop
+            MU.log "Deleted alarms #{alarms.join(', ')}"
+          end
+        end
+
+        # Locate an existing alarm.
+        # @param cloud_id [String]: The cloud provider's identifier for this resource.
+        # @param region [String]: The cloud provider region.
+        # @param flags [Hash]: Optional flags
+        # @return [OpenStruct]: The cloud provider's complete descriptions of matching alarm.
+        def self.find(cloud_id: nil, region: MU.curRegion, flags: {})
+          MU::Cloud::AWS::Alarm.getAlarmByName(cloud_id, region: region)
+        end
+
+        # Create an alarm.
+        def self.setAlarm(
+                name: nil, ok_actions: [], alarm_actions: [], insufficient_data_actions: [], metric_name: nil, namespace: nil, statistic: nil,
+                dimensions: [], period: nil, unit: nil, evaluation_periods: nil, threshold: nil, comparison_operator: nil, region: MU.curRegion
+               )
+
+          # If the alarm already exists, then assume we're updating it and
+          # munge in potentially new arguments.
+          ext_alarm = getAlarmByName(name, region: region)
+          if ext_alarm
+            if !ext_alarm.dimensions.empty?
+              ext_alarm.dimensions.each { |dim|
+                dimensions << dim.to_h
+              }
+              dimensions.uniq!
+            end
+            if alarm_actions
+              alarm_actions.concat(ext_alarm.alarm_actions)
+              alarm_actions.uniq!
+            end
+            if ok_actions
+              ok_actions.concat(ext_alarm.ok_actions)
+              ok_actions.uniq!
+            end
+            if insufficient_data_actions
+              insufficient_data_actions.concat(ext_alarm.insufficient_data_actions)
+              insufficient_data_actions.uniq!
+            end
+            MU.log "Modifying alarm #{name}"
+          else
+            MU.log "Creating alarm #{name}"
+          end
+
+          begin
+            MU::Cloud::AWS.cloudwatch(region).put_metric_alarm(
+              alarm_name: name,
+              alarm_description: name,
+              actions_enabled: true,
+              ok_actions: ok_actions,
+              alarm_actions: alarm_actions,
+              insufficient_data_actions: insufficient_data_actions,
+              metric_name: metric_name,
+              namespace: namespace,
+              statistic: statistic,
+              dimensions: dimensions,
+              period: period,
+              unit: unit,
+              evaluation_periods: evaluation_periods,
+              threshold: threshold,
+              comparison_operator: comparison_operator
+            )
+          rescue Aws::CloudWatch::Errors::ValidationError => e
+            # Dopey but ultimately harmless race condition
+            if e.message.match(/A separate request to update this alarm is in progress/)
+              MU.log "Duplicate request to create alarm #{name}. This one came from #{caller[0]}", MU::WARN
+              sleep 15
+              retry
+            else
+              raise e
+            end
+          end
+
+        end
+
+        # Retrieve the complete cloud provider description of a alarm.
+        # @param name [String]: The cloud provider's identifier for this alarm.
+        # @param region [String]: The cloud provider region
+        # @return [OpenStruct]
+        def self.getAlarmByName(name, region: MU.curRegion)
+          MU::Cloud::AWS.cloudwatch(region).describe_alarms(alarm_names: [name]).metric_alarms.first
+        end
+
+        # Publish logging data, or create a new custom container/group for your logging data
+        # @param namespace [String]: The name of the container, or group the data will be added to to.
+        # @param metric_data [Array]: The data points describing your new metric.
+        # @param region [String]: The cloud provider region.
+        def self.createMetric(namespace: nil, metric_data: [], region: MU.curRegion)
+          MU::Cloud::AWS.cloudwatch(region).put_metric_data(namespace: namespace, metric_data: metric_data, region: region)
+        end
+
+        # Enable the state of the alarm
+        # @param name [String]: The cloud provider's identifier for this alarm.
+        # @param region [String]: The cloud provider region.
+        def self.enableAlarmAction(name, region: MU.curRegion)
+          MU::Cloud::AWS.cloudwatch(region).enable_alarm_actions(alarm_names: [name])
+        end
+
+        # Cloud-specific configuration properties.
+        # @param config [MU::Config]: The calling MU::Config object
+        # @return [Array<Array,Hash>]: List of required fields, and json-schema Hash of cloud-specific configuration parameters for this resource
+        def self.schema(config)
+          toplevel_required = []
+          schema = {}
+          [toplevel_required, schema]
+        end
+
+        # Cloud-specific pre-processing of {MU::Config::BasketofKittens::alarms}, bare and unvalidated.
+        # @param alarm [Hash]: The resource to process and validate
+        # @param configurator [MU::Config]: The overall deployment configurator of which this resource is a member
+        # @return [Boolean]: True if validation succeeded, False otherwise
+        def self.validateConfig(alarm, configurator)
+          ok = true
+          alarm["dimensions"] ||= []
+
+          if alarm["#TARGETCLASS"] == "cache_cluster"
+            alarm['dimensions'] << { "name" => alarm["#TARGETCLASS"], "cloud_class" => "CacheClusterId" }
+            alarm["namespace"] = "AWS/ElastiCache" if alarm["namespace"].nil?
+          elsif alarm["#TARGETCLASS"] == "server"
+            alarm['dimensions'] << { "name" => alarm["#TARGETCLASS"], "cloud_class" => "InstanceId" }
+            alarm["namespace"] = "AWS/EC2" if alarm["namespace"].nil?
+          elsif alarm["#TARGETCLASS"] == "database"
+            alarm['dimensions'] << { "name" => alarm["#TARGETCLASS"], "cloud_class" => "DBInstanceIdentifier" }
+            alarm["namespace"] = "AWS/RDS" if alarm["namespace"].nil?
+          end
+
+          alarm.delete("#TARGETCLASS")
+          alarm.delete("#TARGETNAME")
+
+          if alarm["dimensions"]
+            alarm["dimensions"].each{ |dimension|
+              if dimension["cloud_class"].nil?
+                MU.log "You must specify 'cloud_class'", MU::ERR
+                ok = false
+              end
+  
+              alarm["namespace"], depclass = 
+                if ["InstanceId", "server", "Server"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "InstanceId"
+                  ["AWS/EC2", "server"]
+                elsif ["AutoScalingGroupName", "server_pool", "ServerPool"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "AutoScalingGroupName"
+                  ["AWS/EC2", "server_pool"]
+                elsif ["DBInstanceIdentifier", "database", "Database"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "DBInstanceIdentifier"
+                  ["AWS/RDS", "database"]
+                elsif ["LoadBalancerName", "loadbalancer", "LoadBalancer"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "LoadBalancerName"
+                  ["AWS/ELB", "loadbalancer"]
+                elsif ["CacheClusterId", "cache_cluster", "CacheCluster"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "CacheClusterId"
+                  ["AWS/ElastiCache", "cache_cluster"]
+                elsif ["VolumeId", "volume", "Volume"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "VolumeId"
+                  ["AWS/EBS", nil]
+                elsif ["BucketName", "bucket", "Bucket"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "BucketName"
+                  ["AWS/S3", nil]
+                elsif ["TopicName", "notification", "Notification"].include?(dimension["cloud_class"])
+                  dimension["cloud_class"] = "TopicName"
+                  ["AWS/SNS", nil]
+                end
+  
+              if !depclass.nil?
+                dimension["depclass"] = depclass
+                if !dimension["name"].nil? and !dimension["name"].empty?
+                  alarm["dependencies"] << { "name" => dimension["name"], "type" => depclass }
+                end
+              end
+            }
+          end
+
+          ok = false unless MU::Config::Alarm.validate(alarm, configurator) # XXX the stuff in this method is probably also AWS-specific
+
+          ok
+        end
+
+      end
+    end
+  end
+end