clouds 0.1.0

data/README.md

@@ -0,0 +1,97 @@
+# clouds
+
+This is a simple tool that aims to ease the handling of [AWS CloudFormation](https://aws.amazon.com/cloudformation/) stacks, following the [infrastructure as code](http://sdarchitect.wordpress.com/2012/12/13/infrastructure-as-code/) philosophy in ways that are not possible with the AWS console. Even though you can achieve pretty much the same using the AWS command line tools, `clouds` aims to be much easier to use, cleaner and more specialized in handling CloudFormation stacks as code.
+
+## Features
+- Upload templates and parameters from the current directory into AWS with a single intuitive command. They need to be located in stacks/stack_name/{template.json,parameters.yaml}
+- List existing stacks defined in AWS and also those only defined locally
+- Dump one or all of the stacks from your currently selected AWS account into the current directory (preferably stored in a version control system) under the 'stacks' directory, including both the template JSON code and the parameters that were used for launching it, which will then be saved in a human-friendly YAML file.
+- Perform updates on the AWS stacks once you modified the dumped data (template or parameters)
+- Validate templates for JSON correctness when uploading or updating a stack. The error messages should help you debug syntax errors.
+- Clone existing stacks to create new ones. It defaults to local clone, which then needs another update call to get in effect. This is an easy way to migrate infrastructure code and later to promote changes between environments, and can also be used easily with projects running in different AWS accounts, especially if using a simple AWS profile switcher shell alias like implemented on oh-my-zsh.
+- Perform cost calculations when creating a new stack.
+- Delete existing stacks with a simple command
+- For authentication it can use AWS account credentials defined using either the AWS_ACCESS_KEY_ID&AWS_SECRET_ACCESS_KEY combination, or the AWS_DEFAULT_PROFILE environment variables
+- Able to use temporary credentials, if the AWS_SECURITY_TOKEN variable is defined in the environment.
+
+## Installation
+
+    gem install clouds
+
+or using a [bundler](http://bundler.io/) Gemfile:
+
+    gem 'clouds'
+
+Note: On systems running Ruby 1.8.x such as RHEL5 and clones, you need to be aware of the incompatibility between the more recent versions of `nokogiri`(a dependency of the `aws-sdk`, that `clouds` indirectly depends on) and Ruby 1.8. This is pretty well-documented [here](http://ruby.awsblog.com/post/Tx2T9MFQJK7U74N/AWS-SDK-for-Ruby-and-Nokogiri). **You will need to manually install the latest compatible version of nokogiri and the AWS SDK for Ruby as documented on that page.**
+
+## Running
+Execute this in your shell:
+
+    clouds -h
+
+In order to do real stuff you need a profile to be defined in .aws/config for the aws command line tool, and to have it referenced by the AWS_DEFAULT_PROFILE environment variable. The AWS_ACCESS_KEY_ID&AWS_SECRET_ACCESS_KEY combination are also supported. You can also use temporary credentials, if the AWS_SECURITY_TOKEN variable is properly defined in the environment when using the temporary access key and secret.
+
+    export AWS_DEFAULT_PROFILE=my_aws_profile
+or
+
+    export AWS_ACCESS_KEY_ID=my_key_id
+    export AWS_SECRET_ACCESS_KEY=my_key_secret
+
+The oh-my-zsh users can also use the `asp` command implemented in the [AWS zsh plugin](https://github.com/robbyrussell/oh-my-zsh/wiki/Plugins#aws), which makes it very easy to switch between different profiles defined in .aws/config.
+
+## Use cases
+
+Dump all the stacks from your account into the current directory
+
+    clouds dump --all
+
+Once you edit a stack source code, this command would update it on AWS
+
+    clouds update stack_name
+
+Clone a stack (locally only, so you can perform some changes)
+
+    clouds clone stack new_stack
+
+Upload the cloned stack to AWS CloudFormation
+
+    clouds update new_stack -c
+
+Delete the new stack (needs --force)
+
+    clouds delete new_stack
+
+
+## Development
+
+### Build requirements
+- Ruby, rubygems and rake (some might already be there on Ruby > 1.9)
+- Development headers for libxml2 and libxslt (-devel packages)
+- Bundler
+
+    gem install bundler
+
+### Install dependencies
+Only in case you didn't do it before
+
+    bundle install
+
+### Running for development
+
+    bundle exec bin/clouds
+
+### Updating the installed gem
+
+    rake repackage
+    sudo gem install pkg/clouds-0.1.0.gem
+
+### Build instructions
+
+Simply run (once you have the stated requirements satisfied)
+
+    rake package
+
+### Installing your own gem
+
+    sudo gem install pkg/clouds-0.1.0.gem
+

data/bin/clouds

@@ -0,0 +1,162 @@
+#!/usr/bin/env ruby
+# For development, you need to use `bundle exec bin/clouds` to run this code
+# At install-time, RubyGems will make sure the lib, etc. are in the load path
+
+require 'gli'
+require 'clouds'
+
+include GLI::App
+
+program_desc 'Tool that eases handling of AWS Cloudformation stacks as code'
+
+version Clouds::VERSION
+
+desc 'Dry run, nothing would be done NOT IMPLEMENTED YET'
+switch [:d,:dry], :negatable => false
+
+
+desc 'Force overwrites of existing data'
+switch [:f,:force], :negatable => false
+
+desc 'List CloudFormation stacks from your account'
+arg_name 'Arguments to list'
+command :list do |c|
+
+  c.desc 'List all the stacks available within your account'
+
+  c.action do |global_options,options,args|
+
+    raise 'This command takes no arguments.' unless args.empty?
+    list_stacks()
+  end
+end
+
+desc 'Dump CloudFormation stacks from your account into the current directory'
+arg_name 'Arguments to dump'
+command :dump do |c|
+
+  c.desc 'Dump all the stacks available within your account into the current directory'
+  c.switch [:a,:all], :negatable => false
+  c.desc 'Force overwrites of existing data'
+  c.switch [:f,:force], :negatable => false
+
+  c.action do |global_options,options,args|
+    if args.empty? && options[:all]
+      puts 'dumping all stacks'
+      dump_all_stacks(global_options[:force] || options[:force])
+    elsif args.length >= 1
+      puts "dumping the following stacks: #{args.inspect}"
+      dump_stacks(args, global_options[:force] || options[:force])
+    else # no arguments and --all flag
+      raise 'This command needs a list of stacks, or the --all flag in order to dump all the stacks.'
+    end
+  end
+end
+
+desc 'Edit a CloudFormation stack locally'
+arg_name 'Arguments to edit'
+command :edit do |c|
+
+  c.desc 'Force overwrites of existing data'
+  c.switch [:p,:parameters], :negatable => false
+  c.switch [:t,:template], :negatable => false
+
+  c.action do |global_options,options,args|
+    if args.length == 1
+      stack = args[0]
+      if options[:parameters]
+        puts 'editing parameters'
+        system("#{ENV['EDITOR']} #{get_parameters_path(stack)}")
+      elsif options[:template]
+        puts 'editing template'
+        system("#{ENV['EDITOR']} #{get_template_path(stack)}")
+      else
+        puts options.inspect
+        puts 'editing both the template and the parameters'
+        system("#{ENV['EDITOR']} #{get_template_path(stack)} #{get_parameters_path(stack)}")
+      end
+    else
+      raise 'This command needs a single stack as an argument.'
+    end
+  end
+end
+
+
+desc 'Clones an existing stack into a new one'
+arg_name 'Arguments: stack new_stack'
+command :clone do |c|
+  c.desc 'Force overwrites of existing data'
+  c.switch [:f,:force], :negatable => false
+
+  c.desc 'Commit changes to AWS, otherwise just copy the stack locally'
+  c.switch [:c,:commit], :negatable => false
+
+  c.action do |global_options,options,args|
+    if args.length == 2
+      clone_stack(args.first, args.last, global_options[:force] || options[:force], options['commit'])
+    else
+      raise 'Needed arguments: source_stack destination_stack'
+    end
+  end
+end
+
+desc 'Updated a list of CloudFormation stacks based on information from the current directory'
+arg_name 'Arguments to update'
+command :update do |c|
+
+  c.desc 'Update all the stacks within the current directory'
+  c.switch [:a,:all], :negatable => false
+  c.desc 'Create AWS stacks if the required sources exist locally'
+  c.switch [:c,:create_missing], :negatable => false
+
+  c.action do |global_options,options,args|
+    if args.empty? && options[:all]
+      puts 'dumping all stacks'
+      update_all_stacks(options[:create_missing])
+    elsif args.length >= 1
+      puts "updating the following stacks: #{args.inspect}"
+      update_stacks(args,options[:create_missing])
+    else # no arguments and --all flag
+      raise 'This command needs a list of stacks, or the --all flag in order to update all the stacks.'
+    end
+  end
+end
+
+#TODO: support deletion of multiple stacks, and of all stacks.
+desc 'Deletes a stack'
+arg_name 'Arguments: stack'
+command :delete do |c|
+  c.desc 'Force deletion of the running stack'
+  c.switch [:f,:force], :negatable => false
+  c.action do |global_options,options,args|
+    raise "This command might impact running environments and it needs the --force flag, hopefully you know what you are doing"  unless global_options[:force] || options[:force]
+    if args.length == 1
+      delete_stack(args.first)
+    else
+      raise 'Needed arguments: stack'
+    end
+  end
+end
+
+pre do |global,command,options,args|
+  # Pre logic here
+  # Return true to proceed; false to abort and not call the
+  # chosen command
+  # Use skips_pre before a command to skip this block
+  # on that command only
+  true
+end
+
+post do |global,command,options,args|
+  # Post logic here
+  # Use skips_post before a command to skip this
+  # block on that command only
+end
+
+on_error do |exception|
+  # Error logic here
+  # return false to skip default error handling
+  true
+end
+
+exit run(ARGV)

data/clouds.rdoc

@@ -0,0 +1,5 @@
+= clouds
+
+Generate this with
+    clouds rdoc
+After you have described your command line interface

data/lib/clouds/version.rb

@@ -0,0 +1,3 @@
+module Clouds
+  VERSION = '0.1.0'
+end

data/lib/clouds.rb

@@ -0,0 +1,227 @@
+require 'clouds/version.rb'
+
+require 'aws-sdk'
+
+require 'inifile'
+require 'fileutils'
+require 'yaml'
+
+def configure(profile=nil)
+  if profile.nil? && ENV['AWS_ACCESS_KEY_ID'] && ENV['AWS_SECRET_ACCESS_KEY']
+    @aws_access_key_id = ENV['AWS_ACCESS_KEY_ID']
+    @aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+    @aws_session_token = ENV['AWS_SECURITY_TOKEN']
+    @region = ENV['AWS_DEFAULT_REGION'] || 'us-east-1'
+  else
+    profile = profile || ENV['AWS_DEFAULT_PROFILE']
+    aws_config_file = File.join(File.expand_path('~'), '.aws', 'config')
+
+    if profile.nil?
+      profile = "default"
+      puts "Using the default profile"
+    else
+      profile = "profile #{profile}"
+      puts "Using the profile '#{profile}'"
+    end
+    aws_config = IniFile.load(aws_config_file)[profile]
+
+    @aws_access_key_id = aws_config['aws_access_key_id']
+    @aws_secret_access_key = aws_config['aws_secret_access_key']
+    @region = aws_config['region']
+  end
+
+  proxy = ENV['http_proxy'] || ENV['HTTP_PROXY']
+
+  AWS.config({:access_key_id => @aws_access_key_id,
+              :secret_access_key => @aws_secret_access_key,
+              :session_token => @aws_session_token,
+              :region => @region,
+              :proxy_uri => proxy,
+  })
+
+  @cfn = AWS::CloudFormation.new
+end
+
+def write_file(file_name, content, force)
+  puts "Creating #{file_name}"
+  if File.exist?(file_name) and force == false
+    raise 'The file already exists, use --force to override'
+  end
+  File.open(file_name, 'w')  {|f| f.write(content)}
+  puts "Created #{file_name}"
+end
+
+def read_file(file_name)
+  begin
+    File.open(file_name,'rb').read
+  rescue => e
+    puts e
+  end
+end
+
+def get_stack_directory(stack_name)
+  File.join(File.expand_path('.'),'stacks', stack_name)
+end
+
+def create_stack_directory(stack_name)
+  FileUtils.mkdir_p(get_stack_directory(stack_name))
+end
+
+
+def get_template_path(stack_name)
+  File.join(get_stack_directory(stack_name), 'template.json')
+end
+
+def get_parameters_path(stack_name)
+  File.join(get_stack_directory(stack_name), 'parameters.yaml')
+end
+
+def list_stacks()
+  configure()
+  max_stack_length = 0
+  stacks = Hash.new()
+
+  local_stacks = list_local_stacks()
+  local_stacks.each do |s|
+    max_stack_length = s.length if s.length > max_stack_length
+    stacks[s] = 'LOCAL-ONLY'
+  end
+
+  begin
+    @cfn.stacks.each do |stack|
+      stacks[stack.name] = stack.status
+      max_stack_length = stack.name.length if stack.name.length > max_stack_length
+    end
+  rescue => e
+    puts e
+  end
+  puts 'Stack list and stack status:'
+  stacks.keys.sort.each do |key|
+    printf("%#{max_stack_length}s %s\n",key, stacks[key])
+  end
+end
+
+def list_local_stacks()
+  list = []
+  return [] unless File.directory?('stacks')
+  Dir.foreach('stacks') do |item|
+    next if item == '.' or item == '..'
+    list << item
+  end
+list
+end
+
+def dump_stacks(stack_list, force=false)
+  stack_list.each do |stack_name|
+    configure()
+    begin
+      stack = @cfn.stacks[stack_name]
+      puts "Dumping stack #{stack.name}"
+
+      template_content = stack.template
+      parameters = stack.parameters
+      create_stack_directory(stack_name)
+      write_file(get_template_path(stack_name), template_content, force)
+      write_file(get_parameters_path(stack_name), parameters.to_yaml, force)
+    rescue => e
+      puts "dump failed: #{e}"
+    end
+  end
+end
+
+def dump_all_stacks(force=false)
+  configure()
+  begin
+    @cfn.stacks.each do |stack|
+      dump_stacks([stack.name],force)
+    end
+  rescue => e
+    puts e
+  end
+end
+
+def update_stack(stack_name, create_if_missing=false)
+  configure()
+  stack=nil
+
+  template_content = read_file(get_template_path(stack_name))
+  parameters_content = read_file(get_parameters_path(stack_name))
+
+  begin
+    parameters_hash = YAML.load(parameters_content)
+  rescue => e
+    puts e
+    raise e
+  end
+
+  raise 'Empty stack template' if template_content.nil? || template_content.empty?
+
+  template_validation = @cfn.validate_template(template_content)
+  raise template_validation[:message] unless template_validation[:message].nil?
+
+  begin
+    if @cfn.stacks[stack_name].exists?
+      puts "Updating stack #{stack_name}"
+      stack = @cfn.stacks[stack_name]
+      stack_capabilities = stack.capabilities
+      stack.update(:template => template_content,
+                   :parameters => parameters_hash,
+                   :capabilities => stack_capabilities)
+    elsif create_if_missing
+      puts "Creating stack #{stack_name}"
+      stack = @cfn.stacks.create(stack_name,
+                                 template_content,
+                                 { :parameters => parameters_hash,
+                                   :capabilities => ['CAPABILITY_IAM']})
+    else
+      puts "Skipping stack #{stack_name} since it's not defined in this AWS account, if the stack exists locally you might use the -c flag"
+    end
+    unless stack.nil?
+      estimated_cost = stack.estimate_template_cost()
+      puts "Estimated costs for the stack #{stack_name} is #{estimated_cost}"
+    end
+  rescue => e
+    puts e
+  end
+end
+
+def update_stacks(stack_list, create_if_missing=false)
+  stack_list.each do |stack|
+    update_stack(stack, create_if_missing)
+  end
+end
+
+def update_all_stacks()
+  Dir.foreach('stacks') do |stack|
+    next if item == '.' or item == '..'
+    update_stack(stack)
+  end
+end
+
+
+def clone_stack(stack, new_stack, force=false, commit=false)
+  if File.exist?(get_stack_directory(new_stack)) and force == false
+    raise 'The stack already exists, use --force to override'
+  elsif force == true
+    FileUtils.rm_rf(get_stack_directory(new_stack))
+  end
+  FileUtils.cp_r(get_stack_directory(stack), get_stack_directory(new_stack))
+  if commit
+    puts 'Committing changes to AWS'
+    update_stack(new_stack, true)
+  else
+    puts 'Only copied the stack template and parameters locally, use the --commit flag in order to do commit the changes to AWS'
+  end
+end
+
+def delete_stack(stack_name)
+  configure()
+  FileUtils.rm_rf(get_stack_directory(stack_name))
+  if @cfn.stacks[stack_name].exists?
+    puts "Deleting stack #{stack_name}"
+    @cfn.stacks[stack_name].delete
+  else
+    puts "Stack #{stack_name} does not exist, nothing to delete here..."
+  end
+end
+

metadata

@@ -0,0 +1,188 @@
+--- !ruby/object:Gem::Specification
+name: clouds
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Cristian Magherusan-Stanciu
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-05-22 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 10.2.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 10.2.2
+- !ruby/object:Gem::Dependency
+  name: rdoc
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 4.1.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 4.1.1
+- !ruby/object:Gem::Dependency
+  name: aruba
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.5.4
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.5.4
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.6.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.6.1
+- !ruby/object:Gem::Dependency
+  name: gli
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.9.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.9.0
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.38.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.38.0
+- !ruby/object:Gem::Dependency
+  name: inifile
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.0.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.0.2
+description: A layer of syntax sugar around the AWS Cloudformation API that allows
+  you to handle Cloudformation-defined infrastructure as code with a simple tool
+email: cristian.magherusan-stanciu@here.com
+executables:
+- clouds
+extensions: []
+extra_rdoc_files:
+- README.md
+- clouds.rdoc
+files:
+- bin/clouds
+- lib/clouds/version.rb
+- lib/clouds.rb
+- README.md
+- clouds.rdoc
+homepage: http://here.com
+licenses:
+- GPL-2
+post_install_message: 
+rdoc_options:
+- --title
+- clouds
+- --main
+- README.md
+- -ri
+require_paths:
+- lib
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: AWS Cloudformation API
+test_files: []