geoengineer 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ca09f8d5b8d93ce13661980ff56dd0e28763ecbf
-  data.tar.gz: 4d9e40dfd4402a197a4bab1fb3a184966045fd24
+  metadata.gz: dda3981b643cb0887fcdb1e794abfa4b8c6ac9c0
+  data.tar.gz: 68d052b3c9cd6a7a3f77571837cfda41b1413161
 SHA512:
-  metadata.gz: d99a2fe4223c269d192a8c662626fba2aacddc813ee12fc03fe1a58179a4e6ab5f2a38c21d58cb8aa12ca1e53097906cee497b4853e8ce407003c9890eaf1f7e
-  data.tar.gz: cb63c4c1a257d390ee22f992a725adb180e8a64d9374875b8cc185efb35b9114522248412cf60db060d7128fde5f7e3c6a563bdedee5b0b7be88c0381ccd695d
+  metadata.gz: 8b2bae296535675da928d1decd4e9d96f448876642e953f40d50e6f4ed919ddb3cd6bebc89380a19dddf27305735d8ed2d212af29485baf42ccd582a54751432
+  data.tar.gz: 7e9eb34de6c0d0cb88d2f182ed1e618570404d30bc6c84f97c3a04bd13544e2191ba4bb82cc29e1e333f554881701f1630702072a60dea4ae005ca04d303a434

data/README.md

@@ -4,7 +4,7 @@
 
 <a href="https://commons.wikimedia.org/wiki/File:Mantle_of_Responsibility.png"><img src="./assets/mantle.png" align="right" alt="Mantle_of_Responsibility" /></a>
 
-GeoEngineer provides a Ruby DSL and command line tool (`geo`) to *codeify*  then plan and execute changes to cloud resources.
+GeoEngineer provides a Ruby DSL and command line tool (`geo`) to *codify*  then plan and execute changes to cloud resources.
 
 GeoEngineer's goals/requirements/features are:
 
@@ -29,9 +29,19 @@ Instructions to install Ruby can be found [here](https://www.ruby-lang.org/en/do
 
 ### Install GeoEngineer
 
+Secure install (this will validate the geoengineer gem):
+
+```
+gem cert --add <(curl -Ls https://raw.githubusercontent.com/coinbase/geoengineer/master/certs/geoengineer-gem.pem)
+gem install geoengineer --trust-policy MediumSecurity
+```
+
+*Note: HighSecurity will not work because aws-sdk and other gems are not signed*
+
+Or just install normally:
+
 ```
-gem cert --add <(https://raw.githubusercontent.com/coinbase/geoengineer/master/certs/geoengineer-gem.pem)
-gem install geoengineer --trust-policy HighSecurity
+gem install geoengineer
 ```
 
 Test it is installed correctly with:
@@ -353,7 +363,7 @@ The core models in GeoEngineer are:
 
 1. `Environment` contains many resources that may exist outside of a project, like VPCs or routing tables. Also every project defined to be in the environment, for example the `test_www` project is in `staging` but `monorail` is in `staging` and `production` environments.
 2. `Project` contains many resources and services grouped together into a name.
-3. `Template` has a `type` and `name`, and a group of resources that are defined in a pattern, e.g. every Load Balancer requires a unique security group that allows traffic in. It is an simple abstraction that can dramatically simplify and standardize cloud resources.
+3. `Template` has a `type` and `name`, and a group of resources that are defined in a pattern, e.g. every Load Balancer requires a unique security group that allows traffic in. It is a simple abstraction that can dramatically simplify and standardize cloud resources.
 4. `Resource` and `SubResource` are based off of how terraform models cloud resources. A `Resource` instance can have many `SubResource` instances, but a `SubResource` instance belongs to only one `Resource` instance, e.g. a load balancer resource may have a `health_check` sub-resource to only allow specific incoming ports.
 
 All these models can have arbitrary attributes assigned to them either by directly assigning on the instance, or through passing a block to the constructor. For example:
@@ -390,7 +400,7 @@ puts resource.lazy_attr
 
 ### Environment
 
-The top level class in GeoEngineer is the `environment`, it contains all projects, resources and services, there should only ever be one initialized at a time.
+The top level class in GeoEngineer is the `environment`: it contains all projects, resources and services, and there should only ever be one initialized at a time.
 
 An environment can mean many things to different people, e.g. an AWS account, an AWS region, or a specific AWS VPC. The only real constraint is that a resource has one instance per environment, e.g. a load balancer that is defined to be in `staging` and `production` environments, will have an instance in each.
 
@@ -418,14 +428,14 @@ project = project('org', 'project_name') {
 
 This projects organization is `org`, its name `project_name` and will be provisioned in the `staging` and `production` environments. The `org` and `name` must be unique across all other projects.
 
-The method `project` will automatically add the project to the instantiated environment object **only if** that environments name is in the list of environments, otherwise it is ignored.
+The method `project` will automatically add the project to the instantiated environment object **only if** that environment's name is in the list of environments, otherwise it is ignored.
 
 ### Templates
 
 A template is used to create a group of resources in a recommended pattern. For example, an HTTP service could create a load balancer, a load balancer security group, and a ec2 security group.
 
 ```ruby
-template_instance = project.from_template('template_type', 'name', { parameter: 'helloworld') { |resource, resource_sg|
+template_instance = project.from_template('template_type', 'name', parameter: 'helloworld') { |resource, resource_sg|
 
   # set attribute
   attribute "custom attribute"
@@ -471,6 +481,6 @@ template.resource('type', 'identifier') {
 
 The `type` of a resource must be a valid terraform type, where AWS types are listed [here](https://www.terraform.io/docs/providers/aws/index.html). Some resources are not supported yet by GeoEngineer.
 
-`identifier` is used by GeoEngineer and terraform to reference this resource so must be unique, however it is not stored in the cloud so can be changed without affecting a plan.
+`identifier` is used by GeoEngineer and terraform to reference this resource must be unique, however it is not stored in the cloud so can be changed without affecting a plan.
 
 A resource also has a ruby block sent to it that contains parameters and sub-resources. These values are defined by terraform so for reference to what values are required please refer to the [terraform docs](https://www.terraform.io/docs/providers/aws/index.html).

data/lib/geoengineer/cli/geo_cli.rb

@@ -140,7 +140,10 @@ class GeoCLI
 
       @environment.execute_lifecycle(:before, action_name.to_sym)
       errs = @environment.errors.flatten.sort
-      return print_validation_errors(errs) unless errs.empty?
+      unless errs.empty?
+        print_validation_errors(errs)
+        exit 1
+      end
 
       yield args, options
       @environment.execute_lifecycle(:after, action_name.to_sym)

data/lib/geoengineer/environment.rb

@@ -64,7 +64,7 @@ class GeoEngineer::Environment
     # do not add the project if the project is not supported by this environment
     return NullObject.new unless supported_environments.include?(@name)
 
-    projects[name] = project
+    project
   end
 
   def resource(type, id, &block)

data/lib/geoengineer/resources/aws_cloudtrail.rb

@@ -0,0 +1,20 @@
+########################################################################
+# AwsCloudTrail is the +aws_cloudtrail+ terrform resource,
+#
+# {https://www.terraform.io/docs/providers/aws/r/cloudtrail.html}
+########################################################################
+class GeoEngineer::Resources::AwsCloudtrail < GeoEngineer::Resource
+  validate -> { validate_required_attributes([:name, :s3_bucket_name]) }
+
+  after :initialize, -> { _terraform_id -> { name } }
+
+  def self._fetch_remote_resources
+    cloudtrails = AwsClients.cloudtrail.describe_trails["trail_list"].map(&:to_h)
+
+    cloudtrails.map do |ct|
+      ct[:_terraform_id] = ct[:name]
+      ct[:_geo_id] = ct[:name]
+      ct
+    end
+  end
+end

data/lib/geoengineer/resources/aws_cloudwatch_event_rule.rb

@@ -0,0 +1,26 @@
+########################################################################
+# AwsCloudwatchEventRule is the +aws_cloudwatch_event_rule+ terrform resource,
+#
+# {https://www.terraform.io/docs/providers/aws/r/cloudwatch_event_rule.html Terraform Docs}
+########################################################################
+class GeoEngineer::Resources::AwsCloudwatchEventRule < GeoEngineer::Resource
+  validate -> { validate_required_attributes([:name, :schedule_expression]) }
+
+  after :initialize, -> { _terraform_id -> { NullObject.maybe(remote_resource)._terraform_id } }
+  after :initialize, -> { _geo_id       -> { self[:name] } }
+
+  def support_tags?
+    false
+  end
+
+  def self._fetch_remote_resources
+    AwsClients.cloudwatchevents.list_rules.rules.map(&:to_h).map do |cloudwatch_event_rule|
+      cloudwatch_event_rule.merge(
+        {
+          _terraform_id: cloudwatch_event_rule[:name],
+          _geo_id: cloudwatch_event_rule[:name]
+        }
+      )
+    end
+  end
+end

data/lib/geoengineer/resources/aws_cloudwatch_event_target.rb

@@ -0,0 +1,50 @@
+########################################################################
+# AwsCloudwatchEventTarget is the +aws_cloudwatch_event_target+ terrform resource,
+#
+# {https://www.terraform.io/docs/providers/aws/r/cloudwatch_event_target.html Terraform Docs}
+########################################################################
+class GeoEngineer::Resources::AwsCloudwatchEventTarget < GeoEngineer::Resource
+  validate -> { validate_required_attributes([:rule, :arn, :target_id]) }
+
+  after :initialize, -> { _terraform_id -> { NullObject.maybe(remote_resource)._terraform_id } }
+  after :initialize, -> { _geo_id       -> { "#{self.rule}-#{self.target_id}" } }
+
+  def to_terraform_state
+    tfstate = super
+    tfstate[:primary][:attributes] = {
+      'target_id' => self.target_id,
+      'rule' => self.rule
+    }
+    tfstate
+  end
+
+  def support_tags?
+    false
+  end
+
+  def self._fetch_remote_resources
+    AwsClients
+      .cloudwatchevents
+      .list_rules
+      .rules
+      .map(&:to_h)
+      .map { |rule| _get_rule_targets(rule) }
+      .flatten
+      .map do |rule_target|
+      rule_target.merge(
+        {
+          _terraform_id: "#{rule_target[:rule_name]}-#{rule_target[:id]}",
+          _geo_id: "#{rule_target[:rule_name]}-#{rule_target[:id]}"
+        }
+      )
+    end
+  end
+
+  def self._get_rule_targets(rule)
+    AwsClients
+      .cloudwatchevents
+      .list_targets_by_rule({ rule: rule[:name] })[:targets]
+      .map(&:to_h)
+      .map { |target| target.merge({ rule_name: rule[:name] }) }
+  end
+end

data/lib/geoengineer/resources/aws_db_instance.rb

@@ -14,7 +14,11 @@ class GeoEngineer::Resources::AwsDbInstance < GeoEngineer::Resource
       )
     end
   }
-  validate -> { validate_required_attributes([:password, :username, :name]) if new? }
+  validate -> {
+    if new? && !(snapshot_identifier || replicate_source_db)
+      validate_required_attributes([:password, :username, :name])
+    end
+  }
   validate -> { validate_required_attributes([:instance_class, :engine]) }
   validate -> { validate_subresource_required_attributes(:access_logs, [:bucket]) }
 

data/lib/geoengineer/resources/aws_dynamodb_table.rb

@@ -0,0 +1,24 @@
+########################################################################
+# AwsDynamodbTable +aws_dynamodb_table+ terrform resource,
+#
+# {https://www.terraform.io/docs/providers/aws/r/aws_dynamodb_table.html Terraform Docs}
+########################################################################
+class GeoEngineer::Resources::AwsDynamodbTable < GeoEngineer::Resource
+  validate -> { validate_required_attributes([:name, :read_capacity, :write_capacity, :hash_key]) }
+
+  after :initialize, -> { _terraform_id -> { name } }
+
+  def support_tags?
+    false
+  end
+
+  def self._fetch_remote_resources
+    AwsClients.dynamo.list_tables['table_names'].map { |name|
+      {
+        name: name,
+        _geo_id: name,
+        _terraform_id: name
+      }
+    }
+  end
+end

data/lib/geoengineer/resources/aws_elasticache_cluster.rb

@@ -4,6 +4,8 @@
 # {https://www.terraform.io/docs/providers/aws/r/elasticache_cluster.html Terraform Docs}
 ########################################################################
 class GeoEngineer::Resources::AwsElasticacheCluster < GeoEngineer::Resource
+  class SecurityGroupError < StandardError; end
+
   validate -> {
     validate_required_attributes(
       [
@@ -21,14 +23,17 @@ class GeoEngineer::Resources::AwsElasticacheCluster < GeoEngineer::Resource
 
   def to_terraform_state
     tfstate = super
-    attributes = {
-      'port' => port.to_s,
-      'parameter_group_name' => parameter_group_name
-    }
+    attributes = { 'port' => port.to_s, 'parameter_group_name' => parameter_group_name }
 
     # Security groups workaround
     security_group_ids.each_with_index do |sg, i|
-      attributes["security_group_ids.#{i}"] = sg._terraform_id
+      if sg.is_a?(GeoEngineer::Resource)
+        attributes["security_group_ids.#{i}"] = sg._terraform_id
+      elsif sg.is_a?(String)
+        attributes["security_group_ids.#{i}"] = sg
+      else
+        raise SecurityGroupError, "Please pass a Geo Resource or string ID"
+      end
     end
     attributes['security_group_ids.#'] = security_group_ids.count.to_s
 
@@ -41,10 +46,18 @@ class GeoEngineer::Resources::AwsElasticacheCluster < GeoEngineer::Resource
   end
 
   def self._fetch_remote_resources
-    AwsClients.elasticache.describe_cache_clusters['cache_clusters'].map(&:to_h).map do |ec|
-      ec[:_geo_id] = ec[:cache_cluster_id]
-      ec[:_terraform_id] = ec[:cache_cluster_id]
-      ec
+    AwsClients
+      .elasticache
+      .describe_cache_clusters['cache_clusters']
+      .map(&:to_h)
+      .select { |ec| ec[:replication_group_id].nil? }
+      .map do |ec|
+      ec.merge(
+        {
+          _terraform_id: ec[:cache_cluster_id],
+          _geo_id: ec[:cache_cluster_id]
+        }
+      )
     end
   end
 end

data/lib/geoengineer/resources/aws_iam_policy.rb

@@ -5,6 +5,7 @@
 ########################################################################
 class GeoEngineer::Resources::AwsIamPolicy < GeoEngineer::Resource
   validate -> { validate_required_attributes([:name, :policy]) }
+  validate -> { validate_policy_length(self.policy) }
 
   after :initialize, -> {
     _terraform_id -> { NullObject.maybe(remote_resource)._terraform_id }

data/lib/geoengineer/resources/aws_iam_role_policy.rb

@@ -0,0 +1,63 @@
+########################################################################
+# AwsIamRolePolicy +aws_iam_role_policy+ terrform resource,
+#
+# {https://www.terraform.io/docs/providers/aws/r/iam_role_policy.html Terraform Docs}
+########################################################################
+class GeoEngineer::Resources::AwsIamRolePolicy < GeoEngineer::Resource
+  validate -> { validate_required_attributes([:name, :policy, :role]) }
+  validate -> { validate_policy_length(self.policy) }
+
+  after :initialize, -> {
+    _terraform_id -> { "#{role}:#{name}" }
+  }
+
+  def to_terraform_state
+    tfstate = super
+    attributes = {
+      'policy' => policy,
+      'role' => role,
+      'name' => name
+    }
+
+    tfstate[:primary][:attributes] = attributes
+
+    tfstate
+  end
+
+  def support_tags?
+    false
+  end
+
+  def _policy_file(path, binding_obj = nil)
+    _json_file(:policy, path, binding_obj)
+  end
+
+  def self._fetch_remote_resources
+    AwsClients
+      .iam
+      .list_roles
+      .roles
+      .map(&:to_h)
+      .map { |role| _get_role_policies(role) }
+      .flatten
+      .compact
+      .map { |role_policy| _get_policy(role_policy) }
+  end
+
+  def self._get_role_policies(role)
+    AwsClients
+      .iam
+      .list_role_policies({ role_name: role[:role_name] })
+      .map(&:policy_names)
+      .flatten
+      .map { |policy| { role_name: role[:role_name], policy_name: policy } }
+  end
+
+  def self._get_policy(role_policy)
+    AwsClients
+      .iam
+      .get_role_policy(role_policy)
+      .to_h
+      .merge({ _terraform_id: "#{role_policy[:role_name]}:#{role_policy[:policy_name]}" })
+  end
+end

data/lib/geoengineer/resources/aws_kinesis_stream.rb

@@ -0,0 +1,40 @@
+########################################################################
+# AwsKinesisStream is the +aws_kinesis_stream+ terrform resource,
+#
+# {https://www.terraform.io/docs/providers/aws/r/aws_kinesis_stream.html Terraform Docs}
+########################################################################
+class GeoEngineer::Resources::AwsKinesisStream < GeoEngineer::Resource
+  validate -> { validate_required_attributes([:name, :shard_count]) }
+  validate -> { validate_has_tag(:Name) }
+
+  after :initialize, -> { _terraform_id -> { NullObject.maybe(remote_resource)._terraform_id } }
+  after :initialize, -> { _geo_id       -> { self[:name] } }
+
+  def to_terraform_state
+    tfstate = super
+    tfstate[:primary][:attributes] = {
+      'name' => self.name,
+      'shard_count' => self.shard_count.to_s
+    }
+    tfstate
+  end
+
+  def self._all_streams
+    streams = []
+    AwsClients.kinesis.list_streams[:stream_names].each do |stream_name|
+      AwsClients.kinesis.describe_stream({ stream_name: stream_name }).map(&:to_h).map do |stream|
+        streams << stream[:stream_description]
+      end
+    end
+    streams
+  end
+
+  def self._fetch_remote_resources
+    self._all_streams.map do |stream|
+      stream.merge({
+                     _terraform_id: stream[:stream_arn],
+                     _geo_id: stream[:stream_name]
+                   })
+    end
+  end
+end

data/lib/geoengineer/resources/aws_lambda_alias.rb

@@ -4,12 +4,18 @@
 # {https://www.terraform.io/docs/providers/aws/r/lambda_alias.html Terraform Docs}
 ########################################################################
 class GeoEngineer::Resources::AwsLambdaAlias < GeoEngineer::Resource
+  # Note: function_name here actually means function_arn, even though
+  # function_name is also a key on a lambda function.
   validate -> { validate_required_attributes([:name, :function_name, :function_version]) }
   validate -> {
-    !(name =~ /(?!^[0-9]+$)([a-zA-Z0-9\-_]+)/).nil? if name
+    if name && (name =~ /(?!^[0-9]+$)([a-zA-Z0-9\-_]+)/).nil?
+      "#{name} must match: /(?!^[0-9]+$)([a-zA-Z0-9\-_]+)/"
+    end
   }
   validate -> {
-    !(function_version =~ /(\$LATEST|[0-9]+)/).nil? if function_version
+    if function_version && (function_version =~ /(\$LATEST|[0-9]+)/).nil?
+      "#{function_version} must match: /(\$LATEST|[0-9]+)/"
+    end
   }
 
   after :initialize, -> { _terraform_id -> { NullObject.maybe(remote_resource)._terraform_id } }
@@ -19,6 +25,15 @@ class GeoEngineer::Resources::AwsLambdaAlias < GeoEngineer::Resource
     false
   end
 
+  def to_terraform_state
+    tfstate = super
+    tfstate[:primary][:attributes] = {
+      'function_name' => function_name,
+      'name' => name
+    }
+    tfstate
+  end
+
   # TODO(Brad) - May need to implement solution for pagination...
   def self._fetch_functions
     AwsClients
@@ -31,7 +46,7 @@ class GeoEngineer::Resources::AwsLambdaAlias < GeoEngineer::Resource
   def self._fetch_aliases(function)
     options = { function_name: function[:function_name] }
     AwsClients.lambda.list_aliases(options)[:aliases].map(&:to_h).map do |f_alias|
-      geo_id_components = [f_alias[:name], f_alias[:function_name], f_alias[:function_version]]
+      geo_id_components = [f_alias[:name], function[:function_arn], f_alias[:function_version]]
       f_alias.merge(
         {
           _terraform_id: f_alias[:alias_arn],