chef-provisioning-aws 1.1.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9a8ee86c530dd398dad4b925b119b8d88db9f55a
-  data.tar.gz: 367430b7e0cdee74064297461617a9545c9c1aa1
+  metadata.gz: c249f182469f97f148ae38d524c4d968cdd8f1e3
+  data.tar.gz: da61c4cfe294ca2f50605e63ff77042feffb1b74
 SHA512:
-  metadata.gz: 76da770b34ea861319e125e2125aee36d61b2ce47ed333a26aceae6939576c5f498d054b9f84eff7f1ad46b7d7ca61e6f5cbb5c236742dd761697b8b25eb50f1
-  data.tar.gz: b0c8b49ae247b81214853e172b417c77ed40928da2903f6a4b79085bac2fb4cd91f108a827a9059d60259daa712b2084be4414dcd5202f1d37f1ce4649e98339
+  metadata.gz: 0cce99948b3f6f240fddae731ce3de418c9a239761a8a18776629e27e0b8c82843b202d4b91061d4cd002344c03e14ad36e02f6eecd4437fbbadd0347558affb
+  data.tar.gz: 28e75762b4b912ff030150ea2c9ca09290286b9eda36d6efbe0ed5e15b93f31ba16ffec6657220eabf0fbd85452a2ea9e8439cfd3f78082c10f7c1b6a2c230e6

data/README.md

@@ -1,17 +1,190 @@
-# chef-provisioning-aws
+# Chef Provisioning AWS
 
-An implementation of the AWS driver using the AWS Ruby SDK (v1).  It also implements a large number of AWS-specific resources such as:
+This README is a work in progress.  Please add to it!
 
-* SQS Queues
-* SNS Topics
-* Elastic Load Balancers
-* VPCs
-* Security Groups
-* Instances
-* Images
-* Autoscaling Groups
-* SSH Key pairs
-* Launch configs
+# Resources
+
+TODO: List out weird/unique things about resources here.  We don't need to document every resource
+because users can look at the resource model.
+
+## aws_vpc
+
+If you specify `internet_gateway true` the VPC will create and manage its own internet gateway.
+Specifying `internet_gateway false` will delete that managed internet gateway.
+
+Specifying `main_routes` without `main_route_table` will update the 'default' route table
+that is created when AWS creates the VPC.
+
+Specifying `main_route_table` without specifying `main_routes` will update the main route
+association to point to the provided route table.
+
+If you specify both `main_routes` and `main_route_table` we will update the `main_route_table`
+to have the specified `main_routes`.  IE, running
+
+```ruby
+aws_route_table 'ref-main-route-table' do
+  vpc 'ref-vpc'
+  routes '0.0.0.0/0' => :internet_gateway
+end
+
+aws_vpc 'ref-vpc' do
+  main_route_table 'ref-main-route-table'
+  main_routes '0.0.0.0/1' => :internet_gateway
+end
+
+aws_vpc 'ref-vpc' do
+  main_routes '0.0.0.0/2' => :internet_gateway
+end
+```
+
+will cause resource flapping.  The `ref-main-route-table` resource will set the routes to `/0`
+and then the vpc will set the routes to `/1`.  Then because `ref-main-route-table` is set
+to the main route for `ref-vpc` the third resource will set the routes to `/2`.
+
+The takeaway from this is that you should either specify `main_routes` on your VPC and only
+manage the routes through that, OR only specify `main_route_table` and manage the routes
+through the `aws_route_table` resource.
+
+### Purging
+
+If you specify `action :purge` on the VPC it will attempt to delete ALL resources contained in this
+VPC before deleting the actual VPC.
+
+A potential danger of this is that it does not delete the data bag entries for tracked AWS objects.
+If you `:purge` a VPC and it has `aws_route_table[ref-route]` in it, the data bag entry for
+`ref-route` is not automatically destroyed.  Purge is most useful for testing to ensure no objects
+are left that AWS can charge for.
+
+## aws_key_pair
+
+TODO - document how to specify an existing local key 
+
+## Machine Options
+
+You can pass machine options that will be used by `machine`, `machine_batch` and `machine_image` to
+configure the machine.  These are all the available options:
+
+```ruby
+with_machine_options({
+  bootstrap_options: {
+    key_name: 'ref-key-pair',
+    ...
+  },
+  ...
+})
+```
+
+This options hash can be supplied to either `with_machine_options` or directly into the `machine_options`
+attribute.
+
+## Looking up AWS objects
+
+### \#aws\_object
+
+All chef-provisioning-aws resources have a `aws_object` method that will return the AWS object.  The AWS
+object won't exist until the resource converges, however.  An example of how to do this looks like:
+
+```ruby
+my_vpc = aws_vpc 'my_vpc' do
+  cidr_block '10.0.0.0/24'
+  main_routes '0.0.0.0/0' => :internet_gateway
+  internet_gateway true
+end
+
+my_sg = aws_security_group 'my_sg' do
+  vpc lazy { my_vpc.aws_object.id }
+  inbound_rules '0.0.0.0/0' => [ 22, 80 ]
+end
+
+my_subnet = aws_subnet 'my_subnet' do
+  vpc lazy { my_vpc.aws_object.id }
+  cidr_block '10.0.0.0/24'
+  availability_zone 'eu-west-1a'
+  map_public_ip_on_launch true
+end
+
+machine 'my_machine' do
+  machine_options(
+    lazy do
+      {
+        bootstrap_options: {
+          subnet_id: my_subnet.aws_object.id,
+          security_group_ids: [my_sg.aws_object.id]
+        }
+      }
+    end
+  )
+end
+```
+
+Note the use of the `lazy` attribute modifier.  This is necessary because when the resources are compiled
+the aws_objects do not exist yet, so we must wait to reference them until the converge phase.
+
+### \#lookup\_options
+
+You have access to the aws object when necessary, but often it isn't needed.  The above example is better
+written as:
+
+```ruby
+aws_vpc 'my_vpc' do
+  cidr_block '10.0.0.0/24'
+  main_routes '0.0.0.0/0' => :internet_gateway
+  internet_gateway true
+end
+
+aws_security_group 'my_sg' do
+  vpc 'my_vpc'
+  inbound_rules '0.0.0.0/0' => [ 22, 80 ]
+end
+
+aws_subnet 'my_subnet' do
+  vpc 'my_vpc'
+  cidr_block '10.0.0.0/24'
+  availability_zone 'eu-west-1a'
+  map_public_ip_on_launch true
+end
+
+machine 'my_machine' do
+  machine_options bootstrap_options: {
+    subnet_id: 'my_subnet',
+    security_group_ids: ['my_sg']
+  }
+end
+```
+
+When specifying `bootstrap_options` and any attributes which reference another aws resource, we
+perform [lookup_options](https://github.com/chef/chef-provisioning-aws/blob/master/lib/chef/provisioning/aws_driver/aws_resource.rb#L63-L91).
+This tries to turn elements with names like `vpc`, `security_group_ids`, `machines`, `launch_configurations`,
+`load_balancers`, etc. to the correct AWS object.
+
+### Looking up chef-provisioning resources
+
+The base chef-provisioning resources (machine, machine_batch, load_balancer, machine_image) don't
+have the `aws_object` method defined on them because they are not `AWSResource` classes.  To
+look them up use the class method `get_aws_object` defined on the chef-provisioning-aws specific
+resource:
+
+```ruby
+machine_image 'my_image' do
+  ...
+end
+
+ruby_block "look up machine_image object" do
+  aws_object = Chef::Resource::AwsImage.get_aws_object(
+    'my_image',
+    run_context: run_context,
+    driver: run_context.chef_provisioning.current_driver,
+    managed_entry_store: Chef::Provisioning.chef_managed_entry_store(self.chef_server)
+  )
+end
+```
+
+To look up a machine, use the `AwsInstance` class, to look up a load balancer use the `AwsLoadBalancer`
+class, etc.  The first parameter you pass should be the same resource name as used in the base
+chef-provisioning resource.
+
+Again, the AWS object will not exist until the converge phase, so the aws_object will only be
+available using a `lazy` attribute modifier or in a `ruby_block`.
 
 # Running Integration Tests
 
@@ -30,3 +203,55 @@ you!
 If you find the tests leaving behind resources during normal conditions (IE, not when there is an
 unexpected exception) please file a bug.  Most objects can be cleaned up by deleting the `test_vpc`
 from within the AWS browser console.
+
+# Tagging Resources
+
+## Aws Resources
+
+All resources which extend Chef::Provisioning::AWSDriver::AWSResourceWithEntry support the ability
+to add tags, except AwsEipAddress.  AWS does not support tagging on AwsEipAddress.  To add a tag
+to any aws resource, us the `aws_tags` attribute and provide it a hash:
+
+```ruby
+aws_ebs_volume 'ref-volume' do
+  aws_tags company: 'my_company', 'key_as_string' => :value_as_symbol
+end
+
+aws_vpc 'ref-vpc' do
+  aws_tags 'Name' => 'custom-vpc-name'
+end
+```
+
+The hash of tags can use symbols or strings for both keys and values.  The tags will be converged
+idempotently, meaning no write will occur if no tags are changing.
+
+We will not touch the `'Name'` tag UNLESS you specifically pass it.  If you do not pass it, we
+leave it alone.
+
+## Base Resources
+
+Because base resources from chef-provisioning do not have the `aws_tag` attribute, they must be
+tagged in their options:
+
+```ruby
+machine 'ref-machine-1' do
+  machine_options :aws_tags => {:marco => 'polo', :happyhappy => 'joyjoy'}
+end
+
+machine_batch "ref-batch" do
+  machine 'ref-machine-2' do
+    machine_options :aws_tags => {:marco => 'polo', :happyhappy => 'joyjoy'}
+    converge false
+  end
+  machine 'ref-machine-3' do
+    machine_options :aws_tags => {:othercustomtags => 'byebye'}
+    converge false
+  end
+end
+
+load_balancer 'ref-elb' do
+  load_balancer_options :aws_tags => {:marco => 'polo', :happyhappy => 'joyjoy'}
+end
+```
+
+See `docs/examples/aws_tags.rb` for further examples.

data/lib/chef/provider/aws_dhcp_options.rb

@@ -1,4 +1,5 @@
 require 'chef/provisioning/aws_driver/aws_provider'
+require 'retryable'
 
 class Chef::Provider::AwsDhcpOptions < Chef::Provisioning::AWSDriver::AWSProvider
   protected
@@ -11,7 +12,9 @@ class Chef::Provider::AwsDhcpOptions < Chef::Provisioning::AWSDriver::AWSProvide
 
     converge_by "create new dhcp_options #{new_resource.name} in #{region}" do
       dhcp_options = new_resource.driver.ec2.dhcp_options.create(options)
-      dhcp_options.tags['Name'] = new_resource.name
+      Retryable.retryable(:tries => 15, :sleep => 1, :on => AWS::EC2::Errors::InvalidDhcpOptionsID::NotFound) do
+        dhcp_options.tags['Name'] = new_resource.name
+      end
       dhcp_options
     end
   end

data/lib/chef/provider/aws_load_balancer.rb

@@ -2,7 +2,7 @@ require 'chef/provisioning/aws_driver/aws_provider'
 
 class Chef::Provider::AwsLoadBalancer < Chef::Provisioning::AWSDriver::AWSProvider
   def destroy_aws_object(load_balancer)
-    converge_by "delete load balancer #{new_resource.name} (#{load_balancer.id}) in VPC #{load_balancer.vpc.id} in #{region}" do
+    converge_by "delete load balancer #{new_resource.name} (#{load_balancer.name}) in #{region}" do
       load_balancer.delete
     end
   end

data/lib/chef/provider/aws_route_table.rb

@@ -1,4 +1,5 @@
 require 'chef/provisioning/aws_driver/aws_provider'
+require 'retryable'
 
 class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
 
@@ -6,8 +7,10 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
     route_table = super
 
     if !new_resource.routes.nil?
-      update_routes(vpc, route_table)
+      update_routes(vpc, route_table, new_resource.ignore_route_targets)
     end
+
+    update_virtual_private_gateways(route_table, new_resource.virtual_private_gateways)
   end
 
   protected
@@ -20,7 +23,9 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
 
     converge_by "create new route table #{new_resource.name} in VPC #{new_resource.vpc} (#{vpc.id}) and region #{region}" do
       route_table = new_resource.driver.ec2.route_tables.create(options)
-      route_table.tags['Name'] = new_resource.name
+      Retryable.retryable(:tries => 15, :sleep => 1, :on => AWS::EC2::Errors::InvalidRouteTableID::NotFound) do
+        route_table.tags['Name'] = new_resource.name
+      end
       route_table
     end
   end
@@ -31,14 +36,18 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
     if new_resource.vpc
       desired_vpc = Chef::Resource::AwsVpc.get_aws_object(new_resource.vpc, resource: new_resource)
       if vpc != desired_vpc
-        raise "VPC of route table #{new_resource.name} (#{route_table.id}) is #{route_table.vpc.id}, but desired vpc is #{new_resource.vpc}!  Moving (or rather, recreating) a route table is not yet supported."
+        raise "VPC of route table #{new_resource.to_s} is #{route_table.vpc.id}, but desired vpc is #{new_resource.vpc}!  The AWS SDK does not support updating the main route table except by creating a new route table."
       end
     end
   end
 
   def destroy_aws_object(route_table)
-    converge_by "delete route table #{new_resource.name} (#{route_table.id}) in #{region}" do
-      route_table.delete
+    converge_by "delete #{new_resource.to_s} in #{region}" do
+      begin
+        route_table.delete
+      rescue AWS::EC2::Errors::DependencyViolation
+        raise "#{new_resource.to_s} could not be deleted because it is the main route table for #{route_table.vpc.id} or it is being used by a subnet"
+      end
     end
   end
 
@@ -46,12 +55,13 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
 
   attr_accessor :vpc
 
-  def update_routes(vpc, route_table)
+  def update_routes(vpc, route_table, ignore_route_targets = [])
     # Collect current routes
     current_routes = {}
     route_table.routes.each do |route|
       # Ignore the automatic local route
       next if route.target.id == 'local'
+      next if ignore_route_targets.find { |target| route.target.id.match(/#{target}/) }
       current_routes[route.destination_cidr_block] = route
     end
 
@@ -82,6 +92,30 @@ class Chef::Provider::AwsRouteTable < Chef::Provisioning::AWSDriver::AWSProvider
     end
   end
 
+  def update_virtual_private_gateways(route_table, gateway_ids)
+    current_propagating_vgw_set = route_table.client.describe_route_tables(route_table_ids: [route_table.id]).route_table_set.first.propagating_vgw_set
+
+    # Add propagated routes
+    if gateway_ids
+      gateway_ids.each do |gateway_id|
+        if !current_propagating_vgw_set.reject! { |vgw_set| vgw_set[:gateway_id] == gateway_id }
+          action_handler.perform_action "enable route propagation for route table #{route_table.id} to virtual private gateway #{gateway_id}" do
+            route_table.client.enable_vgw_route_propagation(route_table_id: route_table.id, gateway_id: gateway_id)
+          end
+        end
+      end
+    end
+
+    # Delete anything that's left
+    if current_propagating_vgw_set
+      current_propagating_vgw_set.each do |vgw_set|
+        action_handler.perform_action "disabling route propagation for route table #{route_table.id} from virtual private gateway #{vgw_set[:gateway_id]}" do
+          route_table.client.disable_vgw_route_propagation(route_table_id: route_table.id, gateway_id: vgw_set[:gateway_id])
+        end
+      end
+    end
+  end
+
   def get_route_target(vpc, route_target)
     case route_target
     when :internet_gateway

data/lib/chef/provider/aws_security_group.rb

@@ -38,7 +38,7 @@ class Chef::Provider::AwsSecurityGroup < Chef::Provisioning::AWSDriver::AWSProvi
   end
 
   def destroy_aws_object(sg)
-    converge_by "Deleting SG #{new_resource.name} in #{region}" do
+    converge_by "delete #{new_resource.to_s} in #{region}" do
       sg.delete
     end
   end

data/lib/chef/provider/aws_subnet.rb

@@ -61,8 +61,15 @@ class Chef::Provider::AwsSubnet < Chef::Provisioning::AWSDriver::AWSProvider
           end
         end
       end
+      p.parallel_do(subnet.network_interfaces.to_a) do |network|
+        Cheffish.inline_resource(self, action) do
+          aws_network_interface network do
+            action :purge
+          end
+        end
+      end
     end
-    converge_by "delete subnet #{new_resource.name} in VPC #{new_resource.vpc} in #{region}" do
+    converge_by "delete #{new_resource.to_s} in VPC #{new_resource.vpc} in #{region}" do
       # If the subnet doesn't exist we can't check state on it - state can only be :pending or :available
       begin
         subnet.delete

data/lib/chef/provider/aws_vpc.rb

@@ -1,5 +1,7 @@
 require 'chef/provisioning/aws_driver/aws_provider'
 require 'date'
+require 'chef/provisioning'
+require 'retryable'
 
 class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
 
@@ -18,12 +20,12 @@ class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
 
     # Replace the main route table for the VPC
     if !new_resource.main_route_table.nil?
-      main_route_table = update_main_route_table(vpc)
+      update_main_route_table(vpc)
     end
 
     # Update the main route table
     if !new_resource.main_routes.nil?
-      update_main_routes(vpc, main_route_table)
+      update_main_routes(vpc, new_resource.main_route_table)
     end
 
     # Update DHCP options
@@ -57,7 +59,7 @@ class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
   def destroy_aws_object(vpc)
     if purging
       vpc.subnets.each do |s|
-        Cheffish.inline_resource(self, action) do # if action isn't defined, we want :purge
+        Cheffish.inline_resource(self, action) do
           aws_subnet s do
             action :purge
           end
@@ -66,27 +68,49 @@ class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
       # If any of the below resources start needing complicated delete logic (dependent resources needing to
       # be deleted) move that logic into `delete_aws_resource` and add the purging logic to the resource
       vpc.network_acls.each       { |o| o.delete unless o.default? }
-      vpc.network_interfaces.each { |o| o.delete }
-      vpc.route_tables.each       { |o| o.delete unless o.main? }
-      vpc.security_groups.each    { |o| o.delete unless o.name == 'default' }
+      vpc.network_interfaces.each do |ni|
+        Cheffish.inline_resource(self, action) do
+          aws_network_interface ni do
+            action :purge
+          end
+        end
+      end
+      vpc.route_tables.each do |rt|
+        unless rt.main?
+          Cheffish.inline_resource(self, action) do
+            aws_route_table rt do
+              action :purge
+            end
+          end
+        end
+      end
+      vpc.security_groups.each do |sg|
+        unless sg.name == 'default'
+          Cheffish.inline_resource(self, action) do
+            aws_security_group sg do
+              action :purge
+            end
+          end
+        end
+      end
     end
 
     # Detach or destroy the internet gateway
     ig = vpc.internet_gateway
     if ig
-      converge_by "detach Internet Gateway #{ig.id} in #{region} from VPC #{new_resource.name} (#{vpc.id}" do
+      converge_by "detach Internet Gateway #{ig.id} in #{region} from #{new_resource.to_s}" do
         ig.detach(vpc.id)
       end
       if ig.tags['OwnedByVPC'] == vpc.id
-        converge_by "destroy Internet Gateway #{ig.id} in #{region} (owned by VPC #{new_resource.name} (#{vpc.id}))" do
+        converge_by "destroy Internet Gateway #{ig.id} in #{region} (owned by #{new_resource.to_s})" do
           ig.delete
         end
       end
     end
 
-    # TODO delete main route table & routes if they exist and we created them
+    # We cannot delete the main route table, and it will be deleted when the VPC is deleted anyways
 
-    converge_by "delete VPC #{new_resource.name} (#{vpc.id}) in #{region}" do
+    converge_by "delete #{new_resource.to_s} in #{region}" do
       vpc.delete
     end
   end
@@ -140,6 +164,9 @@ class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
       if !current_ig
         converge_by "attach new Internet Gateway to VPC #{vpc.id}" do
           current_ig = AWS.ec2(config: vpc.config).internet_gateways.create
+          Retryable.retryable(:tries => 15, :sleep => 1, :matching => /never obtained existence/) do
+            raise "internet gateway for VPC #{vpc.id} never obtained existence" unless current_ig.exists?
+          end
           action_handler.report_progress "create Internet Gateway #{current_ig.id}"
           current_ig.tags['OwnedByVPC'] = vpc.id
           action_handler.report_progress "tag Internet Gateway #{current_ig.id} as OwnedByVpc: #{vpc.id}"
@@ -166,22 +193,27 @@ class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
     if current_route_table != desired_route_table
       main_association = current_route_table.associations.select { |a| a.main? }.first
       if !main_association
-        raise "No main route table association found for VPC #{new_resource.name} (#{vpc.id})'s current main route table #{current_route_table.id}: error!  Probably a race condition."
+        raise "No main route table association found for #{new_resource.to_s} current main route table #{current_route_table.id}: error!  Probably a race condition."
       end
-      converge_by "change main route table for VPC #{new_resource.name} (#{vpc.id}) to #{desired_route_table.id} (was #{current_route_table.id})" do
+      converge_by "change main route table for #{new_resource.to_s} to #{desired_route_table.id} (was #{current_route_table.id})" do
         vpc.client.replace_route_table_association(
           association_id: main_association.id,
-          route_table_id: desired_route_table.id)
+          route_table_id: desired_route_table.id
+        )
       end
     end
     desired_route_table
   end
 
   def update_main_routes(vpc, main_route_table)
+    # If no route table is provided and we fetch the current main one from AWS,
+    # there is no guarantee that is the 'default' route table created when
+    # creating the VPC
     main_route_table ||= vpc.route_tables.main_route_table
+    main_routes = new_resource.main_routes
     aws_route_table main_route_table do
       vpc vpc
-      routes new_resource.main_routes
+      routes main_routes
     end
     main_route_table
   end
@@ -190,7 +222,7 @@ class Chef::Provider::AwsVpc < Chef::Provisioning::AWSDriver::AWSProvider
     dhcp_options = vpc.dhcp_options
     desired_dhcp_options = Chef::Resource::AwsDhcpOptions.get_aws_object(new_resource.dhcp_options, resource: new_resource)
     if dhcp_options != desired_dhcp_options
-      converge_by "change DHCP options for VPC #{new_resource.name} (#{vpc.id}) to #{new_resource.dhcp_options} (#{desired_dhcp_options.id}) - was #{dhcp_options.id}" do
+      converge_by "change DHCP options for #{new_resource.to_s} to #{new_resource.dhcp_options} (#{desired_dhcp_options.id}) - was #{dhcp_options.id}" do
         vpc.dhcp_options = desired_dhcp_options
       end
     end