chef-provisioning-aws 1.4.1 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e520aad6f6cafb84d380454daf8dd65285e194cd
-  data.tar.gz: b2ec1a747718d0cad0c4bdc9ca6561d1e0daeb6f
+  metadata.gz: 1ac3b4009283033918629bde10eae733c25b8a73
+  data.tar.gz: 486f831a4168454ad62c9b5ebccd48e7f57249d5
 SHA512:
-  metadata.gz: 0c3d1891e4c22d14f3b03506f39105a884dae7bc00ad7dc2ace729f5325f7fbc935b29be11631387563529b19efcad224bb0e2447b77a0365dfd5f6969cfcc1c
-  data.tar.gz: a52717acff8c3ccf2945863f616fd8fce022b11fa813b0c8ace68b737be6a797dbb8364cdd94d74884e206357ea786741c657cc1d581ec90502459e8d7e1f6b4
+  metadata.gz: 83fc3ef810c239b52c0c00246f08ba97fa5f4b4af3ab116324a7cdf52929815e9006ecc89e92837f1311510b0b9960ca43c41a50385e856bcc12a022a602efa6
+  data.tar.gz: 2dd1b0c595deccae3767740d0abe2d6058c2322ef214a16640dc4386c8586b09efcd39ad667ec29f21aadedbb835aba29b37556b67fb84e0d5bbf661946d5afd

data/Gemfile

@@ -0,0 +1,8 @@
+source "https://rubygems.org"
+gem "simplecov"
+gemspec
+
+#gem 'chef-provisioning', path: '../chef-provisioning'
+#gem 'chef-provisioning', github: 'chef/chef-provisioning', branch: 'master'
+#gem "pry-byebug"
+#gem "pry-stack_explorer"

data/README.md

@@ -6,12 +6,20 @@ This README is a work in progress.  Please add to it!
 
 ## Credentials
 
-AWS credentials should be specified in your `~/.aws/credentials` file as documented [here](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files).  We support the use of profiles as well.  If you do not specify a profile then we use the `default` profile.
+There are 3 ways you can provide your AWS Credentials.  We will look for credentials in the order from below and use the first one found.  This precedence order is taken from http://docs.aws.amazon.com/sdkforruby/api/index.html#Configuration:
 
-You can specify a profile as the middle section of the semi-colon seperated driver url.  For example, a driver url of `aws:staging:us-east-1` would use the profile `staging`.
+1. Through the environment variables `ENV["AWS_ACCESS_KEY_ID"]`, `ENV["AWS_SECRET_ACCESS_KEY"]` and optionally `ENV["AWS_SESSION_TOKEN"]`
+2. The shared credentials ini file.  The default location is `~/.aws/credentials` but you can overwrite this by specifying `ENV["AWS_CONFIG_FILE"]`.  You can specify
+multiple profiles in this file and select one with the `ENV["AWS_DEFAULT_PROFILE"]`
+environment variable or via the driver url.  For example, a driver url of `aws:staging:us-east-1` would use the profile `staging`.  If you do not specify a profile then the `default` one is used.  Read
+[this](http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs) for more information about profiles.
+3. From an instance profile when running on EC2.  This accesses the local
+metadata service to discover the local instance's IAM instance profile.
 
 ## Configurable Options
 
+### aws_retry_limit
+
 When using `machine_batch` with a large number of machines it is possible to overwhelm the AWS SDK until it starts returning `AWS::EC2::Errors::RequestLimitExceeded`.  You can configure the AWS SDK to retry these errors automatically by specifying
 
 ```ruby
@@ -20,14 +28,22 @@ chef_provisioning({:aws_retry_limit => 10})
 
 in your client.rb for the provisioning workstation.  The default `:aws_retry_limit` is 5.
 
+### image_max_wait_time and machine_max_wait_time
+
+By default, the time we will wait for a `machine` to become ready or for the transport to become ready is 120 seconds (each).
+For a `machine_image` we wait 300 seconds for the AMI to be created.  These timeouts can be configured with
+
+```ruby
+chef_provisioning({:image_max_wait_time => 600, :machine_max_wait_time => 240})
+```
+
+in your client.rb for the provisioning workstation.
+
 # Resources
 
 TODO: List out weird/unique things about resources here.  We don't need to document every resource
 because users can look at the resource model.
 
-TODO: document `aws_object` and `get_aws_object` and how you can get the aws object for a base
-chef-provisioning resource like machine or load_balancer
-
 ## aws_key_pair
 
 You can specify an existing key pair to upload by specifying the following:
@@ -171,7 +187,6 @@ The available parameters for `load_balancer_options` can be viewed in the [aws d
 NOTES:
 
 1. You can specify either `ssl_certificate_id` or `server_certificate` in a listener but the value to both parameters should be the ARN of an existing IAM::ServerCertificate object.
-2. Instead of specifying `tags` in the `load_balancer_options`, you should specify `aws_tags`.  See the note on [tagging base resources](https://github.com/chef/chef-provisioning-aws#base-resources).
 
 # RDS Instance Options
 
@@ -260,8 +275,11 @@ Finally, you should add 3 standard tests for taggable objects - 1) Tags can be c
 
 ## \#aws\_object
 
-All chef-provisioning-aws resources have a `aws_object` method that will return the AWS object.  The AWS
-object won't exist until the resource converges, however.  An example of how to do this looks like:
+All chef-provisioning-aws resources have a `aws_object` method that will return the AWS object.  The base
+resources `machine`, `machine_image` and `load_balancer` are monkeypatched to also include the `aws_object`
+method and should respond to it like all other resources.
+
+The AWS object won't exist until the resource converges, however.  An example of how to do this looks like:
 
 ```ruby
 my_vpc = aws_vpc 'my_vpc' do
@@ -336,37 +354,6 @@ perform [lookup_options](https://github.com/chef/chef-provisioning-aws/blob/mast
 This tries to turn elements with names like `vpc`, `security_group_ids`, `machines`, `launch_configurations`,
 `load_balancers`, etc. to the correct AWS object.
 
-## Looking up chef-provisioning resources
-
-The base chef-provisioning resources (machine, machine_batch, load_balancer, machine_image) don't
-have the `aws_object` method defined on them because they are not `AWSResource` classes.  To
-look them up use the class method `get_aws_object` defined on the chef-provisioning-aws specific
-resource:
-
-```ruby
-machine_image 'my_image' do
-  ...
-end
-
-ruby_block "look up machine_image object" do
-  block do
-    aws_object = Chef::Resource::AwsImage.get_aws_object(
-      'my_image',
-      run_context: run_context,
-      driver: run_context.chef_provisioning.current_driver,
-      managed_entry_store: Chef::Provisioning.chef_managed_entry_store(run_context.cheffish.current_chef_server)
-    )
-  end
-end
-```
-
-To look up a machine, use the `AwsInstance` class, to look up a load balancer use the `AwsLoadBalancer`
-class, etc.  The first parameter you pass should be the same resource name as used in the base
-chef-provisioning resource.
-
-Again, the AWS object will not exist until the converge phase, so the aws_object will only be
-available using a `lazy` attribute modifier or in a `ruby_block`.
-
 # Running Integration Tests
 
 To run the integration tests execute `bundle exec rspec`.  If you have not set it up,

data/Rakefile

@@ -14,14 +14,15 @@ RSpec::Core::RakeTask.new(:spec) do |spec|
 end
 
 desc "run integration specs"
-RSpec::Core::RakeTask.new(:integration) do |spec|
-  spec.pattern = 'spec/integration/**/*_spec.rb'
+RSpec::Core::RakeTask.new(:integration, [:pattern]) do |spec, args|
+  spec.pattern = args[:pattern] || 'spec/integration/**/*_spec.rb'
+  spec.rspec_opts = "-b"
 end
 
 desc "run :super_slow specs (machine/machine_image)"
-RSpec::Core::RakeTask.new(:slow) do |spec|
-  spec.pattern = 'spec/**/*_spec.rb'
-  spec.rspec_opts = "-t super_slow"
+RSpec::Core::RakeTask.new(:super_slow, [:pattern]) do |spec, args|
+  spec.pattern = args[:pattern] || 'spec/integration/**/*_spec.rb'
+  spec.rspec_opts = "-b -t super_slow"
 end
 
 desc "run all specs, except :super_slow"
@@ -35,3 +36,10 @@ task :all_slow do
     Rake::Task[t].invoke
   end
 end
+
+desc "travis specific task - runs CI integration tests (regular and super_slow in parallel) and sets up travis specific ENV variables"
+task :travis, [:sub_task] do |t, args|
+  pattern = "load_balancer_spec.rb,machine_image_spec.rb" # This is a comma seperated list
+  pattern = pattern.split(",").map {|p| "spec/integration/**/*#{p}"}.join(",")
+  Rake::Task[args[:sub_task]].invoke(pattern)
+end

data/lib/chef/provider/aws_iam_instance_profile.rb

@@ -0,0 +1,60 @@
+require 'chef/provisioning/aws_driver/aws_provider'
+
+class Chef::Provider::AwsInstanceProfile < Chef::Provisioning::AWSDriver::AWSProvider
+  provides :aws_iam_instance_profile
+
+  def action_create
+    iam_instance_profile = super
+
+    update_attached_role(iam_instance_profile)
+  end
+
+
+  protected
+
+  def detach_role(iam_instance_profile)
+    iam_instance_profile.roles.each do |r|
+      converge_by "detaching role #{r.name} from instance profile #{new_resource.name}" do
+        iam_instance_profile.remove_role(role_name: r.name)
+      end
+    end
+  end
+
+  def update_attached_role(iam_instance_profile)
+    options = Chef::Provisioning::AWSDriver::AWSResource.lookup_options({ iam_role: new_resource.role }, resource: new_resource)
+    role = options[:iam_role]
+
+    if new_resource.role && !iam_instance_profile.roles.map(&:name).include?(role)
+      detach_role(iam_instance_profile)
+      converge_by "associating role #{role} with instance profile #{new_resource.name}" do
+        # Despite having collection methods for roles, instance profile can only have single role associated
+        iam_instance_profile.add_role({
+          role_name: role
+        })
+      end
+    end
+  end
+
+  def create_aws_object
+    converge_by "create IAM instance profile #{new_resource.name}" do
+      new_resource.driver.iam_resource.create_instance_profile({
+        path: new_resource.path || "/",
+        instance_profile_name: new_resource.name
+      })
+    end
+  end
+
+  def update_aws_object(iam_instance_profile)
+    # Nothing to update on our object because the role relationship is managed
+    # through the action
+    iam_instance_profile
+  end
+
+  def destroy_aws_object(iam_instance_profile)
+    detach_role(iam_instance_profile)
+    converge_by "delete #{iam_instance_profile.name}" do
+      iam_instance_profile.delete
+    end
+  end
+
+end

data/lib/chef/provider/aws_iam_role.rb

@@ -0,0 +1,98 @@
+require 'chef/provisioning/aws_driver/aws_provider'
+require 'chef/json_compat'
+
+class Chef::Provider::AwsIamRole < Chef::Provisioning::AWSDriver::AWSProvider
+  provides :aws_iam_role
+
+  def iam_client
+    new_resource.driver.iam_client
+  end
+
+  def iam_resource
+    new_resource.driver.iam_resource
+  end
+
+  def action_create
+    role = super
+
+    if !new_resource.inline_policies.nil?
+      update_inline_policy(role)
+    end
+  end
+
+  protected
+
+  def create_aws_object
+    converge_by "create IAM Role #{new_resource.name}" do
+      iam_resource.create_role({
+        path: new_resource.path,
+        role_name: new_resource.name,
+        assume_role_policy_document: new_resource.assume_role_policy_document
+      })
+    end
+    iam_resource.role(new_resource.name)
+  end
+
+  def update_aws_object(role)
+    if new_resource.path && new_resource.path != role.path
+      raise "Path of IAM Role #{new_resource.name} is #{role.path}, but desired path is #{new_resource.path}.  IAM Role paths cannot be updated!"
+    end
+    if new_resource.assume_role_policy_document && policy_update_required?(role.assume_role_policy_document, new_resource.assume_role_policy_document)
+      converge_by "update IAM Role #{role.name} assume_role_policy_document" do
+        iam_client.update_assume_role_policy({
+          role_name: new_resource.name,
+          policy_document: new_resource.assume_role_policy_document
+        })
+      end
+    end
+  end
+
+  def destroy_aws_object(role)
+    converge_by "delete IAM Role #{role.name}" do
+      role.instance_profiles.each do |profile|
+        profile.remove_role(role_name: role.name)
+      end
+      role.policies.each do |policy|
+        converge_by "delete IAM Role inline policy #{policy.name}" do
+          policy.delete
+        end
+      end
+      role.delete
+    end
+  end
+
+  private
+
+  def update_inline_policy(role)
+    desired_inline_policies = Hash[new_resource.inline_policies.map {|k, v| [k.to_s, v]}]
+    current_inline_policies = Hash[role.policies.map {|p| [p.name, p.policy_document]}]
+
+    policies_to_put = desired_inline_policies.reject {|k,v| current_inline_policies[k] && !policy_update_required?(current_inline_policies[k], v)}
+    policies_to_delete = current_inline_policies.keys - desired_inline_policies.keys
+
+    policies_to_put.each do |policy_name, policy|
+      converge_by "Adding or updating inline Role policy #{policy_name}" do
+        iam_client.put_role_policy({
+          role_name: role.name,
+          policy_name: policy_name,
+          policy_document: policy
+        })
+      end
+    end
+
+    policies_to_delete.each do |policy_name|
+      converge_by "Deleting inline Role policy #{policy_name}" do
+        iam_client.delete_role_policy({
+          role_name: role.name,
+          policy_name: policy_name
+        })
+      end
+    end
+  end
+
+  def policy_update_required?(current_policy, desired_policy)
+    # We parse the JSON into a hash to get rid of whitespace and ordering issues
+    Chef::JSONCompat.parse(URI.decode(current_policy)) != Chef::JSONCompat.parse(desired_policy)
+  end
+
+end

data/lib/chef/provider/aws_image.rb

@@ -29,7 +29,7 @@ class Chef::Provider::AwsImage < Chef::Provisioning::AWSDriver::AWSProvider
       # destroyed - we just need to make sure that has completed successfully
       instance = new_resource.driver.ec2.instances[instance_id]
       converge_by "waiting until instance #{instance.id} is :terminated" do
-        wait_for_status(instance, :terminated, [AWS::EC2::Errors::InvalidInstanceID::NotFound])
+        wait_for_status(instance, :terminated, [AWS::EC2::Errors::InvalidInstanceID::NotFound, AWS::Core::Resource::NotFound])
       end
     end
   end

data/lib/chef/provider/aws_internet_gateway.rb

@@ -0,0 +1,75 @@
+require 'chef/provisioning/aws_driver/aws_provider'
+require 'retryable'
+
+class Chef::Provider::AwsInternetGateway < Chef::Provisioning::AWSDriver::AWSProvider
+  include Chef::Provisioning::AWSDriver::TaggingStrategy::EC2ConvergeTags
+
+  provides :aws_internet_gateway
+
+  def action_detach
+    internet_gateway = Chef::Resource::AwsInternetGateway.get_aws_object(new_resource.name, resource: new_resource)
+    detach_vpc(internet_gateway)
+  end
+
+  protected
+
+  def create_aws_object
+    desired_vpc = Chef::Resource::AwsVpc.get_aws_object(new_resource.vpc, resource: new_resource) if new_resource.vpc
+
+    converge_by "create internet gateway #{new_resource.name} in region #{region}" do
+      internet_gateway = new_resource.driver.ec2.internet_gateways.create
+      retry_with_backoff(AWS::EC2::Errors::InvalidInternetGatewayID::NotFound) do
+        internet_gateway.tags['Name'] = new_resource.name
+      end
+
+      if desired_vpc
+        attach_vpc(desired_vpc, internet_gateway)
+      end
+
+      internet_gateway
+    end
+  end
+
+  def update_aws_object(internet_gateway)
+    current_vpc = internet_gateway.vpc
+
+    if new_resource.vpc
+      desired_vpc = Chef::Resource::AwsVpc.get_aws_object(new_resource.vpc, resource: new_resource)
+      if current_vpc != desired_vpc
+        attach_vpc(desired_vpc, internet_gateway)
+      end
+    end
+  end
+
+  def destroy_aws_object(internet_gateway)
+    converge_by "delete internet gateway #{new_resource.name} in region #{region}" do
+      detach_vpc(internet_gateway)
+      internet_gateway.delete
+    end
+  end
+
+  private
+
+  def attach_vpc(vpc, desired_gateway)
+    if vpc.internet_gateway && vpc.internet_gateway != desired_gateway
+      Cheffish.inline_resource(self, action) do
+        aws_vpc vpc.id do
+          cidr_block vpc.cidr_block
+          internet_gateway false
+        end
+      end
+    end
+    converge_by "attach vpc #{vpc.id} to #{desired_gateway.id}" do
+      desired_gateway.vpc = vpc
+    end
+  end
+
+  def detach_vpc(internet_gateway)
+    if internet_gateway.vpc
+      converge_by "detach vpc #{internet_gateway.vpc.id} from internet gateway #{internet_gateway.id}" do
+        internet_gateway.detach(internet_gateway.vpc)
+      end
+    end
+  end
+
+end

data/lib/chef/provider/aws_route_table.rb

@@ -100,7 +100,8 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
 
     # Delete anything that's left (that wasn't replaced)
     current_routes.values.each do |current_route|
-      action_handler.perform_action "remove route sending #{current_route.destination_cidr_block} to #{current_route.target.id}" do
+      current_target = current_route.gateway_id || current_route.instance_id || current_route.network_interface_id || current_route.vpc_peering_connection_id
+      action_handler.perform_action "remove route sending #{current_route.destination_cidr_block} to #{current_target}" do
         current_route.delete
       end
     end
@@ -141,7 +142,7 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
       route_target = { internet_gateway: route_target }
     when /^eni-[A-Fa-f0-9]{8}$/, Chef::Resource::AwsNetworkInterface, AWS::EC2::NetworkInterface
       route_target = { network_interface: route_target }
-    when /^pcx-[A-Fa-f0-9]{8}$/, Chef::Resource::AwsVpcPeeringConnection, ::Aws::EC2::AwsVpcPeeringConnection
+    when /^pcx-[A-Fa-f0-9]{8}$/, Chef::Resource::AwsVpcPeeringConnection, ::Aws::EC2::VpcPeeringConnection
       route_target = { vpc_peering_connection: route_target }
     when String, Chef::Resource::AwsInstance
       route_target = { instance: route_target }

data/lib/chef/provider/aws_s3_bucket.rb

@@ -50,7 +50,7 @@ class Chef::Provider::AwsS3Bucket < Chef::Provisioning::AWSDriver::AWSProvider
 
   def create_aws_object
     converge_by "create S3 bucket #{new_resource.name}" do
-      new_resource.driver.s3.buckets.create(new_resource.name)
+      new_resource.driver.s3.buckets.create(new_resource.name, new_resource.options)
       # S3 buckets already have a top level name property so they don't need
       # a 'Name' tag
     end
@@ -60,6 +60,9 @@ class Chef::Provider::AwsS3Bucket < Chef::Provisioning::AWSDriver::AWSProvider
   end
 
   def destroy_aws_object(bucket)
+    if purging
+      new_resource.recursive_delete(true)
+    end
     converge_by "delete S3 bucket #{new_resource.name}" do
       if new_resource.recursive_delete
         bucket.delete!

data/lib/chef/provider/aws_security_group.rb

@@ -24,7 +24,7 @@ class Chef::Provider::AwsSecurityGroup < Chef::Provisioning::AWSDriver::AWSProvi
       Chef::Log.debug("VPC: #{options[:vpc]}")
 
       sg = new_resource.driver.ec2.security_groups.create(new_resource.name, options)
-      retry_with_backoff(AWS::EC2::Errors::InvalidSecurityGroupsID::NotFound) do
+      retry_with_backoff(AWS::EC2::Errors::InvalidSecurityGroupsID::NotFound, AWS::EC2::Errors::InvalidGroup::NotFound) do
         sg.tags['Name'] = new_resource.name
       end
       sg