sparkle_formation 0.2.0 → 0.2.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: cba70b56deef5fd41dfd28df337ac9f1af137119
+  data.tar.gz: 08122ae8c40990d331d8be3bbff0b58293162f18
+SHA512:
+  metadata.gz: 5fdcff8822399fa54d5274642e865198514607ef84c8eacb70bbe6ab06e86da2fb9fdd97c4092e5abfd2c3a991a4c1bf2c61c8b7f57c31b59c89276ef8604411
+  data.tar.gz: 027bae7862955e2e0d3ff4aac121daa4a4f67a1937850ce62d90a3d3528c6470b3f8f51009ae58989af726c7c06e11400a38826b9a46cbb7e6d220e732bc8a0c

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## v0.2.2
+* Lots of translation updates (AWS -> RS "hot")
+* User Documentation!
+
 ## v0.2.0
 * Add Registry helper into Utils
 * Add script for collecting cfn and hot resources

data/Gemfile.lock

@@ -1,16 +1,14 @@
 PATH
   remote: .
   specs:
-    sparkle_formation (0.2.0)
+    sparkle_formation (0.2.2)
       attribute_struct (~> 0.2.2)
       multi_json
 
 GEM
   remote: https://rubygems.org/
   specs:
-    attribute_struct (0.2.2)
-      hashie (>= 2.0.0)
-    hashie (2.1.1)
+    attribute_struct (0.2.8)
     multi_json (1.10.1)
 
 PLATFORMS

data/docs/README.md

@@ -0,0 +1,177 @@
+## Overview
+SparkleFormation is a Ruby DSL for programatically composing
+[AWS Cloudformation][cloudformation], [OpenStack Heat][heat]
+provisioning templates for the purpose of interacting with cloud
+infrastructure orchestration APIs.
+
+SparkleFormation templates describe the creation and configuration of
+collections of cloud resources (a stack) as code, allowing you to
+provision stacks in a predictable and repeatable manner. Stacks can be
+managed as single unit, allowing you to create, modify, or delete
+collections of resources via a single API call.
+
+SparkleFormation composes templates in the native cloud orchestration
+formats for AWS, Rackspace, Google Compute, and similar services.
+
+## Table of Contents
+
+- [Getting Started](#getting-started)
+- [What It Looks Like](#what-it-looks-like)
+- [Building Blocks](building-blocks.md)
+  - [Components](building-blocks.md#components)
+  - [Dynamics](building-blocks.md#dynamics)
+  - [Registries](building-blocks.md#registries)
+- [Template Anatomy](anatomy.md)
+  - [Parameters](anatomy.md#parameters)
+  - [Resources](anatomy.md#resources)
+  - [Mappings](anatomy.md#mappings)
+  - [Outputs](anatomy.md#outputs)
+- [Intrinsic Functions](functions.md)
+  - [Ref](functions.md#ref)
+  - [Attr](functions.md#attr)
+  - [Join](functions.md#join)
+- [Universal Properties](properties.md)
+ - [Tags](properties.md#tags)
+
+## Getting Started
+### Gemfile
+SparkleFormation is in active development. To access all the features
+detailed in the documentation (using the knife plugin CLI), you should
+install the plugin and supporting libraries from git:
+
+```ruby
+gem 'fog', :git => 'https://github.com/chrisroberts/fog.git', :ref => 'feature/orchestration'
+gem 'fog-core', :git => 'https://github.com/chrisroberts/fog-core.git', :ref => 'feature/orchestration'
+gem 'knife-cloudformation', :git => 'https://github.com/heavywater/knife-cloudformation.git', :ref => 'feature/fog-model'
+```
+
+The Knife Cloudformation gem is only needed for stack provisioning via
+knife. You could also upload SparkleFormation generated templates to AWS via the WebUI.
+
+### Knife Config
+To use Knife for provisioning, you will need to add the following to
+your `knife.rb` file:
+
+```ruby
+knife[:aws_access_key_id] = ENV['AWS_ACCESS_KEY_ID']
+knife[:aws_secret_access_key] = ENV['AWS_SECRET_ACCESS_KEY']
+
+[:cloudformation, :options].inject(knife){ |m,k| m[k] ||= Mash.new }
+knife[:cloudformation][:options][:disable_rollback] = ENV['AWS_CFN_DISABLE_ROLLBACK'].to_s.downcase == 'true'
+knife[:cloudformation][:options][:capabilities] = ['CAPABILITY_IAM']
+knife[:cloudformation][:processing] = true
+knife[:cloudformation][:credentials] = {
+  :aws_access_key_id => knife[:aws_access_key_id],
+  :aws_secret_access_key => knife[:aws_secret_access_key]
+}
+```
+
+| Attribute                                        | Function                                                                                                       |
+|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------|
+| `[:cloudformation][:options][:disable_rollback]` | Disables rollback if stack is unsuccessful. Useful for debugging.                                              |
+| `[:cloudformation][:credentials]`                | Credentials for a user that is allowed to create stacks.                                                       |
+| `[:cloudformation][:options][:capabilities]`     | Enables IAM creation (AWS only). Options are `nil` or `['CAPABILITY_IAM']`                                     |
+| `[:cloudformation][:processing]`                 | Enables processing SparkleFormation templates (otherwise knife cloudformation will expect a JSON CFN template. |
+
+## What it Looks Like
+Below is a basic SparkleFormation template which would provision an
+elastic load balancer forwarding port 80 to an autoscaling group of
+ec2 instances.
+
+```ruby
+SparkleFormation.new('website') do
+
+  set!('AWSTemplateFormatVersion', '2010-09-09')
+
+  description 'Supercool Website'
+
+  resources.cfn_user do
+    type 'AWS::IAM::User'
+    properties.path '/'
+    properties.policies _array(
+      -> {
+        policy_name 'cfn_access'
+        policy_document.statement _array(
+          -> {
+            effect 'Allow'
+            action 'cloudformation:DescribeStackResource'
+            resource '*'
+          }
+        )
+      }
+    )
+  end
+
+  resources.cfn_keys do
+    type 'AWS::IAM::AccessKey'
+    properties.user_name ref!(:cfn_user)
+  end
+
+  parameters.web_nodes do
+    type 'Number'
+    description 'Number of web nodes for ASG.'
+    default 2
+  end
+
+  resources.website_autoscale do
+    type 'AWS::AutoScaling::AutoScalingGroup'
+    properties do
+      availability_zones({'Fn::GetAZs' => ''})
+      launch_configuration_name ref!(:website_launch_config)
+      min_size ref!(:web_nodes)
+      max_size ref!(:web_nodes)
+    end
+  end
+
+  resources.website_launch_config do
+    type 'AWS::AutoScaling::LaunchConfiguration'
+    properties do
+      image_id 'ami-123456'
+      instance_type 'm3.medium'
+    end
+  end
+
+  resources.website_elb do
+    type 'AWS::ElasticLoadBalancing::LoadBalancer'
+    properties do
+      availability_zones._set('Fn::GetAZs', '')
+      listeners _array(
+        -> {
+          load_balancer_port '80'
+          protocol 'HTTP'
+          instance_port '80'
+          instance_protocol 'HTTP'
+        }
+      )
+      health_check do
+        target 'HTTP:80/'
+        healthy_threshold '3'
+        unhealthy_threshold '3'
+        interval '5'
+        timeout '15'
+      end
+    end
+  end
+end
+```
+
+This template is 74 lines long (with generous spacing for
+readability). The [json template this
+renders](examples/template_json/website.json) is 88 lines, without
+spacing). This can be improved, though. SparkleFormation allows you to
+create resusable files such that the above template can become :
+
+```ruby
+SparkleFormation.new(:website).load(:base).overrides do
+
+  description 'Supercool Website'
+
+  dynamic!(:autoscale, 'website', :nodes => 2)
+  dynamic!(:launch_config, 'website', :image_id => 'ami-123456', :instance_type => 'm3.medium')
+  dynamic!(:elb, 'website')
+
+end
+```
+
+[cloudformation]: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-guide.html
+[heat]: http://docs.openstack.org/developer/heat/template_guide/index.html

data/docs/anatomy.md

@@ -0,0 +1,203 @@
+## Template Anatomy
+
+### Parameters
+Parameters are prompts for stack specific values. A default may be
+specified, but is not required. Every parameter must have a value at runtime.
+
+In the Getting Started example we had one parameter, `web_nodes` which
+set the min and max for the autoscaling group:
+
+```ruby
+parameters.web_nodes do
+  type 'Number'
+  description 'Number of web nodes for ASG.'
+  default 2
+end
+```
+
+Every parameter must have a type and a description. Available types are `String`,
+`Number` (an integer), and `CommaDelimitedList` (an array of
+strings, as-in: `['alpha', 'beta', '1', 2']`). The descriptions is a
+string describing the resource. 
+
+Parameters support optional default values, declared as
+above. An array of accepted values may be set, as well:
+
+```ruby
+parameters.web_nodes do
+  type 'Number'
+  description 'Number of web nodes for ASG.'
+  default 2
+  allowed_values [1, 2, 3, 5]
+end
+```
+
+### Resources
+Resources are the infrastructure resources that are provisioned with
+the stack. Every resource must have a type that corresponds to a
+supported cloud resource. Resources typically have a properties hash
+that configures the resource. Some resources also have metadata. For
+the complete list of required and optional options, see the
+individual resource documentation.
+
+Resource availability is not consistent across
+providers. SparkleFormation's resources support is based on AWS, and
+not all resources will be available on other platforms. See the
+[resource reference](resource-reference.md) table for more information.
+
+The prior example included the following resources:
+- cfn_user: The IAM user for the stack, which will be used to
+provision stack resources.
+
+```ruby
+resources.cfn_user do
+  type 'AWS::IAM::User'
+  properties.path '/'
+  properties.policies _array(
+    -> {
+      policy_name 'cfn_access'
+      policy_document.statement _array(
+        -> {
+          effect 'Allow'
+          action 'cloudformation:DescribeStackResource'
+          resource '*' 
+        }
+      )
+    }
+  )
+end
+```
+
+- cfn_key: The IAM keys for the stack IAM user.
+
+```ruby
+resources.cfn_keys do
+  type 'AWS::IAM::AccessKey'
+  properties.user_name ref!(:cfn_user)
+end
+```
+
+- website_asg: The autoscaling group containing website nodes. The
+size of the autoscaling group is set to the value of the web_nodes
+parameter. 
+
+```ruby
+resources.website_autoscale do
+  type 'AWS::AutoScaling::AutoScalingGroup'
+  properties do
+    availability_zones({'Fn::GetAZs' => ''})
+    launch_configuration_name ref!(:website_launch_config)
+    min_size ref!(:web_nodes)
+    max_size ref!(:web_nodes)
+  end
+end
+```
+
+- website_launch_configuration: The launch configuration for
+website_asg nodes. The AMI image ID and instance type (size) are
+required. 
+
+```ruby
+resources.website_launch_config do
+  type 'AWS::AutoScaling::LaunchConfiguration'
+  properties do
+    image_id 'ami-123456'
+    instance_type 'm3.medium'
+  end
+end
+```
+
+- website_elb: The elastic load balancer for the website. The
+listeners array configures port forwarding. The health check
+configures the load balancer health check target and thresholds.
+
+```ruby
+resources.website_elb do
+  type 'AWS::ElasticLoadBalancing::LoadBalancer'
+  properties do
+    availability_zones._set('Fn::GetAZs', '')
+    listeners _array(
+      -> {
+        load_balancer_port '80'
+        protocol 'HTTP'
+        instance_port '80'
+        instance_protocol 'HTTP'
+      }
+    )
+    health_check do
+      target 'HTTP:80/'
+      healthy_threshold '3'
+      unhealthy_threshold '3'
+      interval '5'
+      timeout '15'
+    end
+  end
+end
+```
+
+### Mappings
+Mappings allow you to create key/value pairs which can be referenced
+at runtime. This is useful for things like an image id that differs
+by region or environment. 
+
+Mappings for the 2014.09 Amazon Linux PV Instance Store 64-bit AMIs
+for each US region:
+
+```ruby
+mappings.region_map do
+  set!('us-east-1', :ami => 'ami-8e852ce6')
+  set!('us-west-1', :ami => 'ami-03a8a146')
+  set!('us-west-2', :ami => 'ami-f786c6c7')
+end
+```
+
+These can be referenced, in turn, with the following:
+
+```ruby
+map!(:region_map, ref!('AWS::Region'), :ami)
+```
+
+'AWS::Region' is a psuedo parameter. We could also perform a lookup
+based on a parameter we provide, e.g. an instance size based on the environment:
+
+```ruby
+parameters.environment do
+  type 'String'
+  allowed_values ['development', 'staging', 'production']                
+end
+
+mappings.instance_size do
+  set!('development', :instance => 'm3.small')
+  set!('staging', :instance => 'm3.medium')
+  set!('production', :instance => 'm3.large')
+end
+
+resources.website_launch_config do
+  type 'AWS::AutoScaling::LaunchConfiguration'
+  properties do
+    image_id map!(:region_map, 'AWS::Region', :ami)
+    instance_type map!(:instance_size, ref!(:environment), :instance)
+  end
+end
+```
+
+### Outputs
+Outputs provide metadata for the stack, as key/value pairs within an
+outputs block. Note that this block lives outside the resource
+blocks. This will retrieve the DNSName attribute for our load
+balancer, and provide it as a value for an 'Elb Dns' output:
+
+```ruby
+outputs do
+  elb_dns do
+    value attr!(:website_elb, 'DNSName')
+    description "Website ELB DNS name"
+  end
+end
+```
+
+Outputs are not simply informational. You can interact with them
+during [provisioning](provisioning.md#knife-cloudformation) using the [knife-cloudformation
+plugin](https://rubygems.org/gems/knife-cloudformation). 
+
+

data/docs/building-blocks.md

@@ -0,0 +1,275 @@
+## SparkleFormation Building Blocks
+
+Using SparkleFormation for the above template has already saved us
+many keystrokes, but what about reusing SparkleFormation code between
+similar stacks? This is where SparkleFormation concepts like
+components, dynamics, and registries come into play.
+
+### Components
+
+Components are static configuration which can be reused between many
+stack templates. In our example case we have decided that all our
+stacks will need to make use of IAM credentials, so we will create
+a component which allows us to inserts the two IAM resources into any
+template in a resuable fashion. The component, which we will call
+'base' and put in a file called 'base.rb,' would look like this:
+
+```ruby
+SparkleFormation.build do
+  set!('AWSTemplateFormatVersion', '2010-09-09')
+
+  resources.cfn_user do
+    type 'AWS::IAM::User'
+    properties.path '/'
+    properties.policies _array(
+      -> {
+        policy_name 'cfn_access'
+        policy_document.statement _array(
+          -> {
+            effect 'Allow'
+            action 'cloudformation:DescribeStackResource'
+            resource '*' 
+          }
+        )
+      }
+    )
+  end
+
+  resources.cfn_keys do
+    type 'AWS::IAM::AccessKey'
+    properties.user_name ref!(:cfn_user)
+  end
+end
+```
+
+After moving these resources out of the initial template and into a
+component, we will update the template so that the base component is
+loaded on the first line, and the resources it contains are no longer
+present in the template itself:
+
+```ruby
+SparkleFormation.new(:website).load(:base).overrides do
+
+  description 'Supercool Website'
+
+  parameters.web_nodes do
+    type 'Number'
+    description 'Number of web nodes for ASG.'
+    default '2'
+  end
+
+  resources.website_autoscale do
+    type 'AWS::AutoScaling::AutoScalingGroup'
+    properties do
+      availability_zones({'Fn::GetAZs' => ''})
+      launch_configuration_name ref!(:website_launch_config)
+      min_size ref!(:web_nodes)
+      max_size ref!(:web_nodes)
+    end
+  end
+
+  resources.website_launch_config do
+    type 'AWS::AutoScaling::LaunchConfiguration'
+    properties do
+      image_id 'ami-123456'
+      instance_type 'm3.medium'
+    end
+  end
+
+  resources.website_elb do
+    type 'AWS::ElasticLoadBalancing::LoadBalancer'
+    properties do
+      availability_zones._set('Fn::GetAZs', '')
+      listeners _array(
+        -> {
+          load_balancer_port '80'
+          protocol 'HTTP'
+          instance_port '80'
+          instance_protocol 'HTTP'
+        }
+      )
+      health_check do
+        target 'HTTP:80/'
+        healthy_threshold '3'
+        unhealthy_threshold '3'
+        interval '5'
+        timeout '15'
+      end
+    end
+  end
+end
+```
+
+### Dynamics
+
+Like components, dynamics are another SparkleFormation feature which
+enables code reuse between stack templates. Where components are
+static, dynamics are useful for creating unique resources via
+the passing of arguments.
+
+In our example scenario we have decided that we want to use elastic
+load balancer resources in many of our stack templates, we want to
+create a dynamic which makes inserting ELB resources much easier than
+copying the full resource configuration between templates.
+
+The resulting implementation would look something like this:
+
+```ruby
+SparkleFormation.dynamic(:elb) do |_name, _config={}|
+  resources("#{_name}_elb".to_sym) do
+    type 'AWS::ElasticLoadBalancing::LoadBalancer'
+    properties do
+      availability_zones._set('Fn::GetAZs', '')
+      listeners _array(
+        -> {
+          load_balancer_port _config[:load_balancer_port] || '80'
+          protocol _config[:protocol] || 'HTTP'
+          instance_port _config[:instance_port] || '80'
+          instance_protocol _config[:instance_protocol] || 'HTTP'
+        }
+      )
+      health_check do
+        target _config[:target] || 'HTTP:80/'
+        healthy_threshold _config[:healthy_threshold] || '3'
+        unhealthy_threshold _config[:unhealthy_threshold] || '3'
+        interval _config[:interval] || '5'
+        timeout _config[:timeout] || '15'
+      end
+    end
+  end
+end
+```
+
+This dynamic accepts two arguments: `_name` (a string, required) and `_config`
+(a hash, optional). The dynamic will use the values passed in these
+arguments to generate a new ELB resource, and override the default ELB
+properties wherever a corresponding key/value pair is provided in the
+`_config` hash.
+
+Once updated to make use of the new ELB dynamic, our template looks
+like this:
+
+```ruby
+SparkleFormation.new(:website).load(:base).overrides do
+
+  description 'Supercool Website'
+
+  parameters.web_nodes do
+    type 'Number'
+    description 'Number of web nodes for ASG.'
+    default '2'
+  end
+
+  resources.website_autoscale do
+    type 'AWS::AutoScaling::AutoScalingGroup'
+    properties do
+      availability_zones({'Fn::GetAZs' => ''})
+      launch_configuration_name ref!(:website_launch_config)
+      min_size ref!(:web_nodes)
+      max_size ref!(:web_nodes)
+    end
+  end
+
+  resources.website_launch_config do
+    type 'AWS::AutoScaling::LaunchConfiguration'
+    properties do
+      image_id 'ami-123456'
+      instance_type 'm3.medium'
+    end
+  end
+
+  dynamic!(:elb, 'website')
+end
+```
+
+If we wanted to override the default configuration for the ELB,
+e.g. to configure the ELB to listen on and communicate with back-end
+node on port 8080 instead of 80, we can specify these override values
+in the configuration passed to the ELB dynamic:
+
+```ruby
+  dynamic!(:elb, 'website', :load_balancer_port => 8080, :instance_port => 8080)
+```
+
+We're passing three arguments here:
+
+1. `:elb` is the name of the dynamic to insert, as a ruby symbol
+2. A name string (`'website'`), which is passed to `_name` in the
+dynamic. This is prepended to the resource name in the dynamic,
+resulting in a unique resource name.
+3. The ELB ports to configure as key/value pairs. These are passed into the dynamic as
+the `_config` hash.
+
+### Registries
+
+Similar to dynamics, registries are reusable resource configuration
+code which can be reused inside different resource definitions.
+
+Registries are useful for defining properties that may be reused
+between resources of different types. For example, the
+LaunchConfiguration and Instance resource types make use of metadata
+properties which inform both resource types how to bootstrap one or
+more instances.
+
+In our example scenario we would like our new instances to run
+'sudo apt-get update && sudo apt-get upgrade -y' at first boot,
+regardless of whether or not the instances are members of an
+autoscaling group.
+
+Because these resources are of different types, placing the metadata
+required for this task directly inside a dynamic isn't going to work
+quite the way we need. Instead we can put this inside a registry which
+can be inserted into the resources defined in one or more dynamics:
+
+```ruby
+SparkleFormation::Registry.register(:apt_get_update) do
+  metadata('AWS::CloudFormation::Init') do
+    _camel_keys_set(:auto_disable) do
+    commands('01_apt_get_update') do
+      command 'sudo apt-get update'
+    end
+    commands('02_apt_get_upgrade') do
+      command 'sudo apt-get upgrade -y'
+    end
+  end
+end
+```
+
+Now we can insert this registry entry into our existing template, to
+ensure that apt is updated upon provisioning:
+
+```ruby
+SparkleFormation.new(:website).load(:base).overrides do
+
+  description 'Supercool Website'
+
+  parameters.web_nodes do
+    type 'Number'
+    description 'Number of web nodes for ASG.'
+    default 2
+  end
+
+  resources.website_autoscale do
+    type 'AWS::AutoScaling::AutoScalingGroup'
+    properties do
+      availability_zones({'Fn::GetAZs' => ''})
+      launch_configuration_name ref!(:website_launch_config)
+      min_size ref!(:web_nodes)
+      max_size ref!(:web_nodes)
+    end
+  end
+
+  resources.website_launch_config do
+    type 'AWS::AutoScaling::LaunchConfiguration'
+    registry!(:apt_get_update, 'website')
+    properties do
+      image_id 'ami-123456'
+      instance_type 'm3.medium'
+    end
+  end
+
+  dynamic!(:elb, 'website')
+
+  registry!(:apt_get_update, 'website')
+end
+```