cfer 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 81546f97136e9bea0bb5197e14b7cded7ab7c440
-  data.tar.gz: da24c76c102c3988c08790805e9d587c015c66bc
+  metadata.gz: 9e97e0ff7a93e2064e001ebe5d02275d10d652b4
+  data.tar.gz: 6814fbea31c860591f6034facf0db09aff48bd6e
 SHA512:
-  metadata.gz: 129f8123bdfff9e159c8d98282e5a084d69bb14e5701562c009832d93dc668f2cd432e9dc440fdc5e06cc4459c02c18429b21f25b356df49621b5667bd96286d
-  data.tar.gz: ae98b4262c9008926894e1cf6a81331809706efa0f5e2e8f9147e49032281cf6c0477eebc1d584ed0fc457d1aed75a556da1f59d0351d57c9411820bd7316b3b
+  metadata.gz: c39189c5fd1dea4a81cd95837d512e71342d60361aa310e7619af2c0abbb8c5f68525263048734864775199a1a25316ede965c01aad1ece07722e02151c00ce7
+  data.tar.gz: dcac2d43384fc8cde476c1f3ad92a4d2153fd66fb48da3da73aa00872ee7d22367a235a903ac2b7fbfd7640ae6713e47aa92d5f54492b7eb83cd3e70bd2d84a8

data/.gitignore

@@ -8,6 +8,6 @@
 /spec/reports/
 /tmp/
 *.swp
-*/.tags
+**/.tags
 
 !/doc/cfer-demo.gif

data/README.md

@@ -15,7 +15,9 @@ Read about Cfer [here](http://tilmonedwards.com/2015/07/28/cfer.html).
 
 Cfer is pre-1.0 software, and may contain bugs or incomplete features. Please see the [license](https://github.com/seanedwards/cfer/blob/master/LICENSE.txt) for disclaimers.
 
-If you would like support or guidance on Cfer, or CloudFormation in general, I offer DevOps consulting services. Please [Contact Bitlancer](http://www.bitlancer.com/contact-us/) and we'll be happy to discuss your needs.
+If you would like support or guidance on Cfer, or CloudFormation in general, I offer DevOps consulting services. Please [Contact me](mailto:stedwards87+cfer@gmail.com) and I'll be happy to discuss your needs.
+
+You can also find me at [@tilmonedwards](https://twitter.com/tilmonedwards). If you use Cfer, or are considering it, I'd love to hear from you.
 
 ## Installation
 
@@ -77,9 +79,14 @@ The following options may be used with the `converge` command:
 * `--follow` (`-f`): Follows stack events on standard output as the create/update process takes place.
 * `--stack-file <template.rb>`: Reads this file from the filesystem, rather than the default `<stack-name>.rb`
 * `--parameters <Key1>:<Value1> <Key2>:<Value2> ...`: Specifies input parameters, which will be available to Ruby in the `parameters` hash, or to CloudFormation by using the `Fn::ref` function
+* `--parameter-file <params_file.[yaml|json]`: Specifies input parameters from a YAML or JSON file
+* `--parameter-environment <env_name>`: Requires `--parameter-file`. Merges the specified key in the YAML or JSON file into the root of the parameter file before passing it into the Cfer stack, i.e. to provide different constants for different AWS environments. The priority for parameters is, in ascending order, **stack default**, **file**, **environment**, and **command line**.
 * `--on-failure <DELETE|ROLLBACK|DO_NOTHING>`: Specifies the action to take when a stack creation fails. Has no effect if the stack already exists and is being updated.
 * `--stack-policy <filename|URL|JSON string>` (`-s`): Stack policy to apply to the stack in order to control updates; takes a local filename containing the policy, a URL to an S3 object, or a raw JSON string.
 * `--stack-policy-during-update <filename|URL|JSON string>` (`-u`): Stack policy as in `--stack-policy` option above, but applied as a temporary override to the permanent policy during stack update.
+* `--s3-path <S3_PATH>`: Path to an S3 bucket location where the template will be stored. This is required if the template output exceeds [51,200 bytes](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html).
+* `--force-s3`: Forces Cfer to upload the template to S3, even if it's small enough to be uploaded directly to the cloudformation API.
+* `--change <CHANGE_NAME>`: Creates a [CloudFormation Change Set](https://aws.amazon.com/blogs/aws/new-change-sets-for-aws-cloudformation/), rather than immediately updating the stack.
 
 #### `tail <stack-name>`
 
@@ -90,6 +97,19 @@ The following options may be used with the `tail` command:
 * `--follow` (`-f`): Follows stack events on standard output as the create/update process takes place.
 * `--number` (`-n`): Print the last `n` stack events.
 
+#### `remove <stack-name>`
+
+Removes or deletes an existing CloudFormation stack. 
+
+```bash
+cfer remove vpc --profile [YOUR-PROFILE] --region [YOUR-REGION]
+```
+
+This can also be done in the following way with the awscli tools, but now eliminates the need to install that package.
+```bash
+aws cloudformation delete-stack --stack-name vpc
+```
+
 ### Template Anatomy
 
 See the `examples` directory for some examples of complete templates.
@@ -278,6 +298,24 @@ This project also contains a [Code of Conduct](https://github.com/seanedwards/cf
 
 # Release Notes
 
+## 0.4.0
+
+### **BREAKING CHANGES**
+* Provisioning is removed from Cfer core and moved to [cfer-provisioning](https://github.com/seanedwards/cfer-provisioning)
+
+### Enhancements
+* Adds support for assume-role authentication with MFA (see: https://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html)
+* Adds support for yml-format parameter files with environment-specific sections.
+* Adds a DSL for IAM policies.
+* Adds `cfer estimate` command to estimate the cost of a template using the AWS CloudFormation cost estimation API.
+* Enhancements to chef provisioner to allow for references in chef attributes. (Thanks to @eropple)
+* Adds continue/rollback/quit selection when `^C` is caught during a converge.
+* Stores Cfer version and Git repo information in the Repo metadata.
+* Added support for uploading templates to S3 with the `--s3-path` and `--force-s3` options.
+* Added new way of extending resources, making plugins easier.
+
+### Bugfixes
+
 ## 0.3.0
 
 ### Enhancements:

data/Rakefile

@@ -1,56 +1 @@
-#require "bundler/gem_tasks"
-gem 'cfer'
-require 'cfer'
-require 'highline'
-
-task :default => [:spec]
-
-task :config_aws, [:profile] do |t, args|
-  Aws.config.update region: ENV['AWS_REGION'] || ask('AWS Region?') { |q| q.default = 'us-east-1' },
-    credentials: Aws::SharedCredentials.new(profile_name: ENV['AWS_PROFILE'] || ask('AWS Profile?') { |q| q.default = 'default' })
-end
-
-task :vpc => :config_aws do |t, args|
-  Cfer.converge! 'vpc',
-    template: 'examples/vpc.rb',
-    follow: true
-end
-
-task :describe_vpc => :config_aws do
-  Cfer.describe! 'vpc'
-end
-
-task :instance => :vpc do |t, args|
-  key_pair = ask("Enter your EC2 KeyPair name: ")
-
-  Cfer.converge! 'instance',
-    template: 'examples/instance.rb',
-    parameters: {
-      :KeyName => key_pair
-    },
-    follow: true
-end
-
-task :describe_instance => :config_aws do
-  Cfer.describe! 'instance'
-end
-
-task :converge => [:vpc, :instance]
-
-
-########################
-##### END OF DEMO ######
-########################
-
-
-# This task isn't really part of Cfer.
-# It just makes it easier for me to release new versions.
-task :release do
-  `git checkout master`
-  `git merge develop --no-ff -m 'Merge from develop for release'`
-
-  require_relative 'lib/cfer/version.rb'
-
-  `git tag -m "Release v#{Cfer::VERSION}" #{Cfer::VERSION}`
-end
-
+require "bundler/gem_tasks"

data/cfer.gemspec

@@ -19,19 +19,18 @@ Gem::Specification.new do |spec|
   spec.executables   = 'cfer'
   spec.require_paths = ["lib"]
 
-  spec.add_runtime_dependency 'docile'
-  spec.add_runtime_dependency 'thor'
-  spec.add_runtime_dependency 'activesupport'
-  spec.add_runtime_dependency 'aws-sdk'
-  spec.add_runtime_dependency 'aws-sdk-resources'
-  spec.add_runtime_dependency 'preconditions'
-  spec.add_runtime_dependency 'semantic'
-  spec.add_runtime_dependency 'rainbow'
-  spec.add_runtime_dependency 'highline'
-  spec.add_runtime_dependency 'table_print'
-  spec.add_runtime_dependency "rake"
-  spec.add_runtime_dependency "erubis"
+  spec.add_runtime_dependency 'docile', '~> 1.1', '>= 1.1.5'
+  spec.add_runtime_dependency 'thor', '~> 0.19.1'
+  spec.add_runtime_dependency 'activesupport', '~> 4.2', '>= 4.2.6'
+  spec.add_runtime_dependency 'aws-sdk', '~> 2.2', '>= 2.2.33'
+  spec.add_runtime_dependency 'aws-sdk-resources', '~> 2.2', '>= 2.2.33'
+  spec.add_runtime_dependency 'preconditions', '~> 0.3.0'
+  spec.add_runtime_dependency 'semantic', '~> 1.4', '>= 1.4.1'
+  spec.add_runtime_dependency 'rainbow', '~> 2.1'
+  spec.add_runtime_dependency 'highline', '~> 1.7', '>= 1.7.8'
+  spec.add_runtime_dependency 'table_print', '~> 1.5', '>= 1.5.6'
+  spec.add_runtime_dependency "git", '~> 1.3'
+  spec.add_runtime_dependency "bundler"
 
-  spec.add_development_dependency "bundler"
-  spec.add_development_dependency "yard"
+  spec.add_development_dependency "yard", '~> 0.8.7.6'
 end

data/examples/common/instance_deps.rb

@@ -8,12 +8,12 @@ parameter :KeyName
 # If you created the VPC stack with a different name, you can overwrite these default values
 # by adding `Vpc:<vpc_stack_name> to your `--parameters` option
 parameter :Vpc, default: 'vpc'
-parameter :VpcId, default: lookup_output(parameters[:Vpc], 'vpcid')
-parameter :SubnetId, default: lookup_output(parameters[:Vpc], 'subnetid1')
+parameter :VpcId, default: (lookup_output(parameters[:Vpc], 'vpcid') rescue nil)
+parameter :SubnetId, default: (lookup_output(parameters[:Vpc], 'subnetid1') rescue nil)
 
 # This is the Ubuntu 14.04 LTS HVM AMI provided by Amazon.
 parameter :ImageId, default: 'ami-fce3c696'
-parameter :InstanceType, default: 't2.nano'
+parameter :InstanceType, default: 't2.micro'
 
 # Define a security group to be applied to an instance.
 # This one will allow SSH access from anywhere, and no other inbound traffic.

data/examples/instance.rb

@@ -2,30 +2,24 @@ description 'Example stack template for a small EC2 instance'
 
 # NOTE: This template depends on vpc.rb
 
-# Include common template code that will be used for examples that create EC2 instances.
+# You can use the `include_template` function to include other ruby files into this Cloudformation template.
 include_template 'common/instance_deps.rb'
 
-# We can define extension objects, which extend the basic JSON-building
+# We can define extensions to resources, which extend the basic JSON-building
 # functionality of Cfer. Cfer provides a few of these, but you're free
-# to define your own by creating a class that matches the name of an
-# CloudFormation resource type, inheriting from `Cfer::AWS::Resource`
-# inside the `CferExt` module:
-module CferExt::AWS::EC2
-  # This class adds methods to resources with the type `AWS::EC2::Instance`
-  # Remember, this class could go in your own gem to be shared between your templates
-  # in a way that works with the rest of your infrastructure.
-  class Instance < Cfer::Cfn::Resource
-    def boot_script(data)
-      # This function simply wraps a bash script in the little bit of extra
-      # sugar (hashbang + base64 encoding) that EC2 requires for userdata boot scripts.
-      # See the AWS docs here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
-      script = <<-EOS.strip_heredoc
-      #!/bin/bash
-      #{data}
-      EOS
-
-      user_data Base64.encode64(script)
-    end
+# to define your own by using `Cfer::Core::Resource.extend_resource` and specifying a
+# CloudFormation resource type. Inside the block, define any methods you'd like to use:
+Cfer::Core::Resource.extend_resource "AWS::EC2::Instance" do
+  def boot_script(data)
+    # This function simply wraps a bash script in the little bit of extra
+    # sugar (hashbang + base64 encoding) that EC2 requires for userdata boot scripts.
+    # See the AWS docs here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
+    script = <<-EOS.strip_heredoc
+    #!/bin/bash
+    #{data}
+    EOS
+
+    user_data Base64.encode64(script)
   end
 end
 

data/lib/cfer.rb

@@ -56,12 +56,34 @@ module Cfer
       cfn = options[:aws_options] || {}
 
       cfn_stack = options[:cfer_client] || Cfer::Cfn::Client.new(cfn.merge(stack_name: stack_name))
-      stack = options[:cfer_stack] || Cfer::stack_from_file(tmpl, options.merge(client: cfn_stack))
+      stack = options[:cfer_stack] ||
+              Cfer::stack_from_file(tmpl,
+                options.merge(
+                  client: cfn_stack,
+                  parameters: generate_final_parameters(options)
+                )
+              )
 
       begin
-        cfn_stack.converge(stack, options)
-        if options[:follow]
-          tail! stack_name, options
+        operation = stack.converge!(options)
+        if options[:follow] && !options[:change]
+          begin
+            tail! stack_name, options.merge(cfer_client: cfn_stack)
+          rescue Interrupt
+            puts "Caught interrupt. What would you like to do?"
+            case HighLine.new($stdin, $stderr).choose('Continue', 'Quit', 'Rollback')
+            when 'Continue'
+              retry
+            when 'Rollback'
+              case operation
+              when :created
+                cfn_stack.delete_stack stack_name: stack_name
+              when :updated
+                cfn_stack.cancel_update_stack stack_name: stack_name
+              end
+              retry
+            end
+          end
         end
       rescue Aws::CloudFormation::Errors::ValidationError => e
         Cfer::LOGGER.info "CFN validation error: #{e.message}"
@@ -118,11 +140,31 @@ module Cfer
 
     def generate!(tmpl, options = {})
       config(options)
-      cfn_stack = Cfer::Cfn::Client.new(options[:aws_options] || {})
-      stack = Cfer::stack_from_file(tmpl, options.merge(client: cfn_stack)).to_h
+      cfn = options[:aws_options] || {}
+
+      cfn_stack = options[:cfer_client] || Cfer::Cfn::Client.new(cfn)
+      stack = options[:cfer_stack] || Cfer::stack_from_file(tmpl,
+        options.merge(client: cfn_stack, parameters: generate_final_parameters(options))).to_h
       puts render_json(stack, options)
     end
 
+    def estimate!(tmpl, options = {})
+      config(options)
+      cfn = options[:aws_options] || {}
+
+      cfn_stack = options[:cfer_client] || Cfer::Cfn::Client.new(cfn)
+      stack = options[:cfer_stack] || Cfer::stack_from_file(tmpl,
+        options.merge(client: cfn_stack, parameters: generate_final_parameters(options)))
+      puts cfn_stack.estimate(stack)
+    end
+
+    def delete!(stack_name, options = {})
+      config(options)
+      cfn = options[:aws_options] || {}
+      cfn_stack = options[:cfer_client] || cfn_stack = Cfer::Cfn::Client.new(cfn.merge(stack_name: stack_name))
+      cfn_stack.delete_stack(stack_name)
+    end
+
     # Builds a Cfer::Core::Stack from a Ruby block
     #
     # @param options [Hash] The stack options
@@ -156,8 +198,40 @@ module Cfer
       Cfer::LOGGER.debug "Options: #{options}"
       Cfer::LOGGER.level = Logger::DEBUG if options[:verbose]
 
+      require 'rubygems'
+      require 'bundler/setup'
+
       Aws.config.update region: options[:region] if options[:region]
-      Aws.config.update credentials: Aws::SharedCredentials.new(profile_name: options[:profile]) if options[:profile]
+      Aws.config.update credentials: Cfer::Cfn::CferCredentialsProvider.new(profile_name: options[:profile]) if options[:profile]
+    end
+
+    def generate_final_parameters(options)
+      raise "parameter-environment set but parameter_file not set" \
+        if options[:parameter_environment] && options[:parameter_file].nil?
+
+      file_params =
+        if options[:parameter_file]
+          case File.extname(options[:parameter_file])
+          when '.yaml'
+            require 'yaml'
+            YAML.load_file(options[:parameter_file])
+          when '.json'
+            JSON.parse(IO.read(options[:parameter_file]))
+          else
+            raise "Unrecognized parameter file format: #{File.extname(options[:parameter_file])}"
+          end
+        else
+          {}
+        end
+
+      if options[:parameter_environment]
+        raise "no key '#{options[:parameter_environment]}' found in parameters file." \
+          unless file_params.key?(options[:parameter_environment])
+
+        file_params = file_params.deep_merge(file_params[options[:parameter_environment]])
+      end
+
+      file_params.deep_merge(options[:parameters] || {})
     end
 
     def render_json(obj, options = {})
@@ -170,6 +244,8 @@ module Cfer
 
     def templatize_errors(base_loc)
       yield
+    rescue Cfer::Util::CferError => e
+      raise e
     rescue SyntaxError => e
       raise Cfer::Util::TemplateError.new([]), e.message
     rescue StandardError => e

data/lib/cfer/block.rb

@@ -30,4 +30,46 @@ module Cfer
     def post_block
     end
   end
+
+  class BlockHash < Block
+    NON_PROXIED_METHODS = [:parameters, :options, :lookup_output]
+
+    def properties(keyvals = {})
+      self.merge!(keyvals)
+    end
+
+    def get_property(key)
+      self.fetch key
+    end
+
+    def respond_to?(method_sym)
+      !non_proxied_methods.include?(method_sym)
+    end
+
+    def method_missing(method_sym, *arguments, &block)
+      key = camelize_property(method_sym)
+      properties key =>
+        case arguments.size
+        when 0
+          if block
+            BlockHash.new.build_from_block(&block)
+          else
+            raise "Expected a value or block when setting property #{key}"
+          end
+        when 1
+          arguments.first
+        else
+          arguments
+        end
+    end
+
+    private
+    def non_proxied_methods
+      NON_PROXIED_METHODS
+    end
+
+    def camelize_property(sym)
+      sym.to_s.camelize.to_sym
+    end
+  end
 end

data/lib/cfer/cfn/cfer_credentials_provider.rb

@@ -0,0 +1,72 @@
+require 'yaml'
+
+module Cfer
+  module Cfn
+    class CferCredentialsProvider < Aws::SharedCredentials
+      private
+
+      def load_from_path
+        profile = load_profile
+        credentials = Aws::Credentials.new(
+          profile['aws_access_key_id'],
+          profile['aws_secret_access_key'],
+          profile['aws_session_token']
+        )
+        @credentials =
+          if role_arn = profile['role_arn']
+            role_creds =
+              begin
+                YAML::load_file('.cfer-role')
+              rescue
+                {}
+              end
+
+            if stored_creds = role_creds[profile_name]
+              if (Time.now.to_i + 5 * 60) > stored_creds[:expiration].to_i
+                stored_creds = nil
+              end
+            end
+
+            if stored_creds == nil
+              role_credentials_options = {
+                role_session_name: [*('A'..'Z')].sample(16).join,
+                role_arn: role_arn,
+                credentials: credentials
+              }
+
+              if profile['mfa_serial']
+                role_credentials_options[:serial_number] ||= profile['mfa_serial']
+                role_credentials_options[:token_code] ||= HighLine.new($stdin, $stderr).ask('Enter MFA Code:')
+              end
+
+              creds = Aws::AssumeRoleCredentials.new(role_credentials_options)
+              stored_creds = {
+                expiration: creds.expiration,
+                credentials: creds.credentials
+              }
+              role_creds[profile_name] = stored_creds
+            end
+
+            IO.write('.cfer-role', YAML.dump(role_creds))
+            stored_creds[:credentials]
+          else
+            credentials
+          end
+      end
+
+      def load_profile
+        if profile = profiles[profile_name]
+          # Add all options from source profile
+          if source = profile.delete('source_profile')
+            profiles[source].merge(profile)
+          else
+            profile
+          end
+        else
+          msg = "Profile `#{profile_name}' not found in #{path}"
+          raise Aws::Errors::NoSuchProfileError, msg
+        end
+      end
+    end
+  end
+end