cfoo 0.0.1

data/README.md

@@ -0,0 +1,166 @@
+# Cfoo
+
+[![Build Status](https://travis-ci.org/drrb/cfoo.png?branch=master)](https://travis-ci.org/drrb/cfoo)
+[![Coverage Status](https://coveralls.io/repos/drrb/cfoo/badge.png?branch=master)](https://coveralls.io/r/drrb/cfoo)
+[![Code Climate](https://codeclimate.com/github/drrb/cfoo.png)](https://codeclimate.com/github/drrb/cfoo)
+
+Write your CloudFormation templates in a YAML-based markup language
+
+## Installation
+
+Cfoo can be installed as a Ruby Gem
+
+    $ gem install cfoo
+
+## Usage
+
+Process some Cfoo templates from the command line
+
+    $ cfoo web-server-template.yml database_template.yml
+
+## Templates
+
+### Comparison with standard CloudFormation templates
+
+Snippet from a CloudFormation template (based on [this example](https://s3.amazonaws.com/cloudformation-templates-us-east-1/Rails_Single_Instance.template)):
+
+```json
+"Properties": {
+  "ImageId" : { "Fn::FindInMap" : [ "AWSRegion2AMI", { "Ref" : "AWS::Region" }, "AMI" ] },
+  "InstanceType"   : { "Ref" : "InstanceType" },
+  "SecurityGroups" : [ {"Ref" : "FrontendGroup"} ],
+  "KeyName"        : { "Ref" : "KeyName" },
+  "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
+    "#!/bin/bash -v\n",
+    "yum update -y aws-cfn-bootstrap\n",
+
+    "function error_exit\n",
+    "{\n",
+    "  /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandle" }, "'\n",
+    "  exit 1\n",
+    "}\n",
+
+    "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r WebServer ",
+    "    --region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to run cfn-init'\n",
+
+    "/opt/aws/bin/cfn-signal -e 0 -r \"cfn-init complete\" '", { "Ref" : "WaitHandle" }, "'\n"
+  ]]}}        
+}
+```
+
+Equivalent Cfoo template snippet:
+
+```yaml
+Properties:
+  ImageId : AWSRegion2AMI[$(AWS::Region)][AMI]
+  InstanceType: $(InstanceType)
+  SecurityGroups: 
+     - $(FrontendGroup)
+  KeyName: $(KeyName)
+  UserData: !Base64 |
+    #!/bin/bash -v
+    yum update -y aws-cfn-bootstrap
+
+    function error_exit
+    {
+      /opt/aws/bin/cfn-signal -e 1 -r "$1" '$(WaitHandle)'
+      exit 1
+    }
+
+    /opt/aws/bin/cfn-init -s $(AWS::StackId) -r WebServer --region $(AWS::Region) || error_exit 'Failed to run cfn-init'
+
+    /opt/aws/bin/cfn-signal -e 0 -r "cfn-init completed" '$(WaitHandle)'
+```
+
+### Shortcuts
+
+Cfoo allows you to simplify CloudFormation [intrinsic function](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html)
+references using its own shorthand
+
+##### Reference
+
+CloudFormation: `{ "Ref" : "InstanceType" }`
+
+Cfoo Shortcut: `$(InstanceType)`
+
+##### Mapping Reference
+
+CloudFormation: `{ "FindInMap" : [ "SubnetConfig", "VPC", "CIDR" ] }`
+
+Cfoo Shortcut: `$(SubnetConfig[VPC][CIDR])`
+
+##### Attribute Reference
+
+CloudFormation: `{ "Fn::GetAtt" : [ "Ec2Instance", "PublicIp" ] }`
+
+Cfoo Shortcut: `$(Ec2Instance.PublicIp)`
+
+Other Shortcut: `$(Ec2Instance[PublicIp])`
+
+##### Embedded Reference
+
+CloudFormation: `{ "Fn::Join" : [ "", [{"Ref" : "HostedZone"}, "." ]]}`
+
+Cfoo Shortcut: `$(HostedZone).`
+
+### YAML Types
+
+Cfoo gives you the option of using YAML custom data-types where it helps to make your templates easier to read.
+The table below uses inline YAML lists, but multiline lists can also be used.
+
+##### Reference
+
+CloudFormation: `{ "Ref" : "InstanceType" }`
+
+YAML Type: `!Ref InstanceType`
+
+##### Mapping Reference
+
+CloudFormation: `{ "FindInMap" : [ "SubnetConfig", "VPC", "CIDR" ] }`
+
+YAML Type: `!FindInMap [ SubnetConfig, VPC, CIDR ]`
+
+##### Attribute Reference
+
+CloudFormation: `{ "Fn::GetAtt" : [ "Ec2Instance", "PublicIp" ] }`
+
+YAML Type: `!GetAtt [ Ec2Instance, PublicIp ]`
+
+##### Base64 String
+
+CloudFormation: `{ "Fn::Base64" : "#!/bin/bash\necho 'Running script...'" }`
+
+YAML Type: `!Base64 "#!/bin/bash\necho 'running script...'"`
+
+## Goals
+
+### Primary Goals
+
+Cfoo aims to let developers simplify CloudFormation templates by:
+
+- allowing them to write templates in YAML
+- providing an expression language to simplify CF Ref/Attr/etc expressions
+- allowing templates to be split up into logical components (to simplify and share)
+
+### Secondary Goals
+
+Cfoo aims (subject to Primary Goals) to:
+
+- allow all aspects of CloudFormation templates to be expressed in YAML (so you don't need to use JSON)
+- allow inclusion existing JSON templates (so you don't have to switch all at once)
+
+### Non-goals
+
+Cfoo does not (yet) aim to:
+
+- provide commandline utilities for interacting directly with CloudFormation (it just generates the templates for now)
+- resolve/validate references (the CloudFormation API already does this)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Make your changes (with tests please)
+4. Commit your changes (`git commit -am 'Add some feature'`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create new Pull Request

data/Rakefile

@@ -0,0 +1,10 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+require 'cucumber/rake/task'
+require 'coveralls/rake/task'
+
+Coveralls::RakeTask.new
+RSpec::Core::RakeTask.new(:spec)
+Cucumber::Rake::Task.new(:features)
+
+task :default => [:spec, :features, 'coveralls:push']

data/TODO

@@ -0,0 +1,29 @@
+Features
+----------
+More tests for EL parsing
+Spaces in EL?
+More than [A-Za-z:] in EL identifiers?
+Shorthand for AWS variables (e.g. { "Ref" : "AWS::StackId" } ) 
+Shorthand for Fn::GetAZs
+Better errors when parsing fails
+Split into resources/mappings/parameters/etc by putting templates into directories with those names
+Allow directory for raw templates
+Allow traditional JSON CloudFormation templates (including modules, splitting, etc)
+Vim highlighting
+Use multi-json instead of json gem (spec.add_dependency 'multi_json', '~> 1.0')
+Disablable warnings about naming conventions
+Allow module references to remote templates (inc. Git repositories) (auto-included)
+Allow rendering only part of the project (in case part of the infrastructure already exists)
+Render templates with ERB to allow extra flexibility/substitution
+Resolve EL and complain/fail if targets don't exist
+Unit testing for templates
+
+Tests
+----------
+
+Tasks
+----------
+Refactor EL parser
+
+Bugs
+----------

data/bin/cfoo

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+
+lib_path = File.expand_path('../../lib', __FILE__)
+$LOAD_PATH.unshift lib_path unless $LOAD_PATH.include? lib_path
+require 'cfoo'
+
+if ARGV.include? "--help"
+    STDERR.puts "Usage: #{File.basename $0} [filename...]"
+    exit 1
+end
+
+filenames = ARGV
+
+yaml_parser = Cfoo::YamlParser.new
+file_system = Cfoo::FileSystem.new(".", yaml_parser)
+project = Cfoo::Project.new(file_system)
+parser = Cfoo::Parser.new(file_system)
+processor = Cfoo::Processor.new(parser, project)
+renderer = Cfoo::Renderer.new
+cfoo = Cfoo::Cfoo.new(processor, renderer, STDOUT)
+
+if filenames.empty?
+    cfoo.build_project
+else
+    cfoo.process *filenames
+end

data/cfoo.gemspec

@@ -0,0 +1,35 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'cfoo/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "cfoo"
+  gem.version       = Cfoo::VERSION
+  gem.authors       = ["drrb"]
+  gem.email         = ["drrrrrrrrrrrb@gmail.com"]
+  gem.license       = "GPL-3"
+  gem.description   = "Cfoo: CloudFormation master"
+  gem.summary       = <<-EOF
+    Cfoo (pronounced 'sifu') allows you to write your CloudFormation templates in a
+    YAML-based markup language, and organise it into modules to make it easier to
+    maintain.
+  EOF
+  gem.homepage      = "https://github.com/drrb/cfoo"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(spec|features)/})
+  gem.require_paths = ["lib"]
+
+  gem.add_dependency "json"
+  gem.add_dependency "parslet"
+
+  gem.add_development_dependency "bundler", "~> 1.3"
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "rspec"
+  gem.add_development_dependency "cucumber"
+  gem.add_development_dependency "simplecov"
+  gem.add_development_dependency "coveralls", ">= 0.6.3"
+  gem.add_development_dependency "json" # Coveralls needs it
+end

data/features/attribute_expansion.feature

@@ -0,0 +1,43 @@
+Feature: Expand EL Attribute References
+    As a CloudFormation user
+    I want to use an expression language as a shorthand for references
+    So that I my templates are easier to read
+
+    Scenario: Attribute expansion
+        Given I have a file "outputs.yml" containing
+        """
+        EntryPoint:
+            Description: IP address of the Bastion Host
+            Value: $(BastionHost.PublicIp)
+        """
+        When I process "outputs.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "EntryPoint" : {
+                "Description" : "IP address of the Bastion Host",
+                "Value" :  { "Fn::GetAtt" : [ "BastionHost", "PublicIp" ]}
+            }
+        }
+        """
+
+    Scenario: Embedded attribute expansion
+        Given I have a file "outputs.yml" containing
+        """
+        WebSite:
+            Description: URL of the website
+            Value: http://$(PublicElasticLoadBalancer.DNSName)/index.html
+        """
+        When I process "outputs.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "WebSite" : {                                                                                                                                            
+                "Description" : "URL of the website",
+                "Value" :  { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "PublicElasticLoadBalancer", "DNSName" ]}, "/index.html"]]}
+            }
+        }
+        """
+

data/features/convert_yaml_to_json.feature

@@ -0,0 +1,69 @@
+Feature: Convert YAML to JSON
+    As a JSON provider
+    I want to generate my JSON from YAML
+    So that it's easier to read
+
+    Scenario: Basic array conversion
+        Given I have a file "list.yml" containing
+        """
+        list:
+            - One
+            - 2
+            - "3"
+        """
+        When I process "list.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "list" : ["One", 2, "3"]
+        }
+        """
+
+    Scenario: Basic map conversion
+        Given I have a file "map.yml" containing
+        """
+        1: One
+        2: Two
+        3: Three
+        """
+        When I process "map.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "1" : "One",
+            "2" : "Two",
+            "3" : "Three"
+        }
+        """
+
+    Scenario: Embedded map structure
+        Given I have a file "embeddedmap.yml" containing
+        """
+        Fruit:
+            - Apples: [ red, green ]
+            - Bananas
+            - Grapes: [ seeded, seedless ]
+        Vegetables:
+            - Beans: [ red, black ]
+            - Sweet Corn
+            - Mirleton
+        """
+        When I process "embeddedmap.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "Fruit": [
+                { "Apples": [ "red", "green" ] },
+                "Bananas",
+                { "Grapes": [ "seeded", "seedless" ] }
+            ],
+            "Vegetables": [
+                { "Beans": [ "red", "black" ] },
+                "Sweet Corn",
+                "Mirleton"
+            ]
+        }
+        """

data/features/el_escaping.feature

@@ -0,0 +1,21 @@
+Feature: EL escaping
+    As a CloudFormation user
+    I want to to be able to escape the EL
+    So that I can type whatever text I want
+
+    Scenario: Escape EL
+        Given I have a file "outputs.yml" containing
+        """
+        EntryPoint:
+            Value: \$(BastionHost.PublicIp)
+        """
+        When I process "outputs.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "EntryPoint" : {
+                "Value" : "$(BastionHost.PublicIp)"
+            }
+        }
+        """

data/features/map_reference_expansion.feature

@@ -0,0 +1,68 @@
+Feature: Expand EL Mapping References
+    As a CloudFormation user
+    I want to use an expression language as a shorthand for mapping references
+    So that I my templates are easier to read
+
+    Scenario: Map reference expansion
+        Given I have a file "outputs.yml" containing
+        """
+        EntryPoint:
+            Description: IP address of the Bastion Host
+            Value: $(SubnetConfig[VPC][CIDR])
+        """
+        When I process "outputs.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "EntryPoint" : {
+                "Description" : "IP address of the Bastion Host",
+                "Value" :  { "Fn::FindInMap" : [ "SubnetConfig", "VPC", "CIDR" ]}
+            }
+        }
+        """
+
+    Scenario: Embedded map reference expansion
+        Given I have a file "outputs.yml" containing
+        """
+        WebSite:
+            Description: URL of the website
+            Value: http://$(Network[Dns][LoadBalancerDnsName])/index.html
+        """
+        When I process "outputs.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "WebSite" : {                                                                                                                                            
+                "Description" : "URL of the website",
+                "Value" :  { "Fn::Join" : [ "", [ "http://", { "Fn::FindInMap" : [ "Network", "Dns", "LoadBalancerDnsName" ]}, "/index.html"]]}
+            }
+        }
+        """
+
+    Scenario: Map key is reference
+        Given I have a file "nat.yml" containing
+        """
+        Resources:
+            NATDevice:
+                Type: AWS::EC2::Instance
+                Properties:
+                    ImageId: $(AWSNATAMI[$(AWS::Region)][AMI])
+        """
+        When I process "nat.yml"
+        Then the output should match JSON
+        """
+        {
+            "AWSTemplateFormatVersion" : "2010-09-09",
+            "Resources" : {
+                "NATDevice" : {
+                    "Type" : "AWS::EC2::Instance",
+                    "Properties" : {
+                        "ImageId" : { "Fn::FindInMap" : [ "AWSNATAMI" , { "Ref" : "AWS::Region" } , "AMI" ] }
+                    }
+                }
+            }
+        }
+        """
+