aws-must 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ec263031a8d6a6619934045ec286e400ddf00a72
+  data.tar.gz: f1b69d4f681dc1551986be9f505540dd17337f1b
+SHA512:
+  metadata.gz: ce14b0078feac91d303e3b313ce9f33eab39afb966bdf11ee52efdee6f6e9e4380ce330b7daa8ea203b7651d580e41d3d7409c1494cf09139ed4be390a1c1bed
+  data.tar.gz: 47a828370761ca394473cea460a2b337d2f9c2a3990ca96ae701317dde6ebf467b523a4408a4632b2b60859b683470567259d428fcd8366c475feb98e1d539b8

data/README.md

@@ -0,0 +1,235 @@
+# aws-must - Minimum Viable Solution to Manage CloudFormation Templates - $Release:0.0.2$
+
+`aws-must` is a tool, which allows separating infrastructure
+configuration and Amazon related syntax using
+[YAML](http://learnxinyminutes.com/docs/yaml) and
+[Mustache templates](https://mustache.github.io/). 
+
+
+## The Problem
+
+[Amazon CloudFormation](http://aws.amazon.com/cloudformation/) gives
+developers and systems administrators an easy way to create and manage
+a collection of related AWS resources.  It uses JSON -formatted
+[templates](http://aws.amazon.com/cloudformation/aws-cloudformation-templates)
+for defining services to be managed together as a "stack".
+
+However, the CF -templates soon become quite convoluted as the
+complexity of the infrastructure stack increases. In addition, the
+JSON format adds more to the management difficulties:
+
+1. each stack is represented as a monolithic JSON document.
+
+2. templates mix infrastructure specific configuration with the syntax
+   stipulated by
+   [Amazon CloudFormation Template Reference](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html)
+
+3. the JSON format doesn't support expressing references, and
+   CloudFormation needs to use a
+   [special syntax](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html)
+   to define relations across template elements.
+
+4. the JSON format doesn't support comments, which could be used to
+   add documentation into the templates.
+
+5. JSON documents repeat similar sections as resource instances are
+   defined individually as sub-documents.
+
+6.  JSON adds it own markup overhead to the configuration.
+
+## The solution
+
+`aws-tool` helps mitigating the problems presented in the chapter
+above.  With the tool users may
+
+1.  split stack configuration first in half, using YAML/mustache
+    templates, and then further down using Mustache partials
+
+2. separate infrastructure specific configuration from syntax
+   stipulated by
+   [Amazon CloudFormation Template Reference](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html)
+   using YAML and Mustache templates.
+
+3. use YAML anchors to express dependencies between infrastructure
+   elements.
+
+4. add comments in YAML and Mustache templates. The tool supports
+   simple tag syntax (**++start++**
+   **++close++** -tags) to add documentation to
+   Mustache templates, which can be extracted to a separate document.
+
+5. use Mustache partials to get rid of repeating similar sections in
+   JSON configuration.
+
+6. concentrate on configurations expressed in YAML (with minimal markdown).
+
+
+## Installation
+
+Add following line to `Gemfile`
+
+	gem 'aws-must'
+
+
+and run
+
+	bundle install
+	
+
+## Usage
+
+### CLI  Usage
+
+To get help 	
+
+	aws-must.rb help
+
+To create CloudFormation JSON template for `yaml_file` using default
+template `./mustache/root.mustache` issue the command
+
+	aws-must.rb gen yaml_file
+
+To extract documentation between **++start++**
+**++close++** -tags from template
+`./mustache/root.mustache` issue the command
+
+	aws-must.rb doc 
+
+To dump YAML `yaml_file` in JSON format
+
+	aws-must.rb json yaml_file
+
+
+### Demo Usage
+
+Add the following lines to `Rakefile`
+
+
+	spec = Gem::Specification.find_by_name 'aws-must'
+	load "#{spec.gem_dir}/lib/tasks/demo.rake"
+
+
+and verify that `demo` namespace is added to rake tasks
+
+	rake -T demo
+
+#### Demo Usage Locally
+
+A list of demo cases is shown with the command
+
+	rake -T demo:template
+	
+To show the CloudFormation JSON rendered for a demo case `i`, where
+`i` ranges from `1,2,3, ...`, run rake task `dev:template-i`. For
+example, for demo case `1` run
+
+	rake demo:template-1
+
+To show the difference of running case `i` vs. running the previous
+demo case, run rake task `demo:diff-i`. For example, for demo case `2`
+run
+
+	rake demo:diff-2
+	
+**NOTICE:**: The [jq](http://stedolan.github.io/jq/) must be installed
+for diff target to work.
+
+To show html documentation extracted from demo case `i` templates, run
+rake task `demo:html-i`. For example, for demo case `3` run
+
+	rake demo:html-3
+
+**NOTICE:**: Uses `markdown` command to convert
+[markdown](http://daringfireball.net/projects/markdown) documentation
+into html.
+
+**NOTICE:**: Default browser `firefox` can be changed using command
+line argument `browser`, e.g. `rake demo:html-3[chromium-browser]`.
+
+#### Use Demo to Bootstrap Own Configuration
+
+To extract your own copy of demo case `i` configuration, run rake task
+`demo:bootstrap-i` with command line arguments defining template
+directory and configuration directory. For example, to copy demo case
+`3` templates to directory `tmp/tmpl` and configurations to
+`tmp/conf`, run
+
+    rake demo:bootstrap-3[tmp/tmpl,tmp/conf]
+	
+
+After this command `tmp/tmpl` contains `root.mustache`, which is the
+starting point of mustache template rendering, and `tmp/conf` contains
+a YAML file `conf.yaml` for demo configuration data.
+
+
+#### Demo Usage on Amazon platform
+
+Demo stacks can be provisioned on Amazon platform.
+
+**WARNING** Provisioning CloudFormation templates on Amazon will be
+**charged according to Amazon pricing policies**.
+
+Prerequisites:
+
+* [Install Amazon AWS Command Line Interface](http://docs.aws.amazon.com/cli/latest/userguide/installing.html)
+* [Configure AWS Command Line Interface](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
+
+To demo targets, which provision Amazon, run
+
+	rake -T demo:stack-create
+
+**NOTICE**: Please, notice the region constraint in the output. Some
+of the templates use fixed AMI configuration, which means that they
+work only if aws -tool defines correct region (typically in set in
+`~/.aws/config` file).
+
+**NOTICE**: Demo-case `7,8, ...` make use of public ssh-key
+`demo-key`. See
+[Amazon EC2 Key Pairs](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html),
+which explains how to upload your own key to Amazon platform.
+
+To create a `demo` stack for demo case `i`, for example demo case 7
+run
+
+	rake  demo:stack-create-7
+	
+To show events generated when `demo` stack is created run
+
+	rake  demo:stack-events
+	
+To show status of `demo` stack run
+
+	rake  demo:stack-status
+
+Instances created in demo cases `7,8,...` can be connected using ssh.
+Demo target `demo:stack-ssh` locates an ip address from stack output
+variable, and launches ssh command with the ip and the identity given
+as a parameter.
+
+For example to use ip address in demo stack output variable `IP1`, and
+with the identity `~/.ssh/demo-key/demo-key`, run
+
+    rake demo:stack-ssh[IP1,~/.ssh/demo-key/demo-key]
+	
+where
+
+* `IP1` : is the name of stack output variable holding IP address
+  (cf. Outputs array in `rake demo:stack-status`)
+* `~/.ssh/demo-key/demo-key` is the file path of ssh private key for
+  [Amazon SSH-key pair](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html)
+  `demo-key`.
+
+
+To delete `demo` stack
+
+	rake  demo:stack-delete
+
+
+	
+	
+
+
+
+	 
+
+

data/bin/aws-must.rb

@@ -0,0 +1,92 @@
+#!/usr/bin/env ruby
+
+
+require 'thor'
+require_relative '../lib/aws-must'
+
+class App < Thor
+
+  include Utils::MyLogger     # mix logger
+  PROGNAME = "main"           # logger progname
+  DEAFAULT_TEMPLATE="root"    # name of template in :template_path -directory
+
+
+  # ------------------------------------------------------------------
+
+  class_option :log, :aliases => "-l", :type =>:string, :default => nil, 
+  :enum => [ "DEBUG", "INFO", "WARN", "ERROR" ],
+  :desc => "Set debug level "
+
+  # ------------------------------------------------------------------
+  # action 'doc'
+
+  desc "doc", "Extract template documentation"
+
+  option :template_path, :aliases => "-t", :type => :string, 
+  :default => "mustache", 
+  :desc => "Directory holding mustache templates"
+
+  long_desc <<-LONGDESC
+
+  Extract documation from <template_name> in ':template_path' -direcotory.
+
+  <template_name> defaults to '#{DEAFAULT_TEMPLATE}'
+
+LONGDESC
+
+  def doc( template_name=DEAFAULT_TEMPLATE )
+
+    app = ::AwsMust::AwsMust.new( options )
+    app.doc( template_name )
+
+  end
+
+  # ------------------------------------------------------------------
+  # action 'json'
+
+  desc "json <yaml_file>", "Dump configuration in JSON format"
+
+  long_desc <<-LONGDESC
+
+  Reads <yaml_file> and dumps it to stdout in JSON format.
+
+LONGDESC
+
+
+  def json( yaml_file )
+
+    app = ::AwsMust::AwsMust.new( options )
+    app.json( yaml_file )
+
+  end
+
+
+  # ------------------------------------------------------------------
+  # action 'gen'
+
+  desc "gen <yaml_file> [<template_name>]", "Generate CloudFormation JSON template"
+
+  option :template_path, :aliases => "-t", :type => :string, 
+  :default => "mustache", 
+  :desc => "Directory holding mustache templates"
+
+  long_desc <<-LONGDESC
+
+  Generate Amazon CloudFormation JSON file for <yaml_file> using
+  <template_name> in ':template_path' -direcotory.
+
+  <template_name> defaults to '#{DEAFAULT_TEMPLATE}'
+
+LONGDESC
+
+
+  def gen( yaml_file, template_name=DEAFAULT_TEMPLATE )
+
+    app = ::AwsMust::AwsMust.new( options )
+    app.generate( template_name, yaml_file, options )
+
+  end
+
+end
+
+App.start

data/demo/0/conf.yaml

@@ -0,0 +1 @@
+-- demo 0

data/demo/0/root.mustache

@@ -0,0 +1 @@
+{{! empty }}

data/demo/1/conf.yaml

@@ -0,0 +1 @@
+-- empty

data/demo/1/root.mustache

@@ -0,0 +1,22 @@
+{{!
++++start+++
+
+# <a id="top">aws-must demo 1 templates</a>
+
+**No documentation yet**
+
++++close+++
+}}
+{
+  "AWSTemplateFormatVersion" : "2010-09-09",
+  "Description" : "A simple Amazon EC2 instance. Initial copy",
+  "Resources" : {
+    "MyEC2Instance" : {
+      "Type" : "AWS::EC2::Instance",
+      "Properties" : {
+        "ImageId" : "ami-00dae61d",
+        "InstanceType" : "t2.micro"
+      }
+    }
+  }
+}

data/demo/2/conf.yaml

@@ -0,0 +1,6 @@
+-- demo/2 : added description property
+
+
+description: "demo/2: a simple Amazon EC2 instance created using
+             aws-must tool.  Added 'description' tag in
+             `root.mustache`"

data/demo/2/root.mustache

@@ -0,0 +1,54 @@
+{{!
+
+++start++
+
+<style>
+h1 {
+    color:blue; 
+    font-size: 2.5em;
+   }
+h2 {
+      color:blue;
+      font-size: 1.5em;
+   }
+body {
+    background-color: #b0c4de;
+} 
+</style>
+
+
+# <a id="top">aws-must demo 2 templates</a>
+
+## root.mustache
+
+Create an EC2 with following fixed parameters
+
+* `ImageId` : "ami-00dae61d",
+* `InstanceType` : "t2.micro"
+
+**Attributes**:
+
+* `description`: description for the CF template
+
+**Actions**:
+
+* **Description**: `description`
+
+
+++close++
+
+}}
+
+{
+  "AWSTemplateFormatVersion" : "2010-09-09",
+  "Description" : "{{description}}",
+  "Resources" : {
+    "MyEC2Instance" : {
+      "Type" : "AWS::EC2::Instance",
+      "Properties" : {
+        "ImageId" : "ami-00dae61d",
+        "InstanceType" : "t2.micro"
+      }
+    }
+  }
+}

data/demo/3/conf.yaml

@@ -0,0 +1,3 @@
+
+description: "demo/3: a simple Amazon EC2 instance created using aws-must tool. 
+              Resources section created using partial resource.mustache"

data/demo/3/resources.mustache

@@ -0,0 +1,26 @@
+{{! 
+
+    resources.mustache - fixed resources
+
+++start++
+
+## <a id="resources.mustache"></a>resources.mustache <a class='navigator' href='#top'">[top]</a>
+
+Create an EC2 with following fixed parameters
+
+* `ImageId` : "ami-00dae61d",
+* `InstanceType` : "t2.micro"
+
+++close++
+
+
+}}
+
+
+    "MyEC2Instance" : {
+      "Type" : "AWS::EC2::Instance",
+      "Properties" : {
+        "ImageId" : "ami-00dae61d",
+        "InstanceType" : "t2.micro"
+      }
+    }

data/demo/3/root.mustache

@@ -0,0 +1,85 @@
+{{!
+
+
+==================================================================
+STYLE section
+==================================================================
+
+++start++
+<style>
+h1 {
+    color:blue; 
+    font-size: 2.5em;
+   }
+h2 {
+      color:blue;
+      font-size: 1.5em;
+   }
+body {
+    background-color: #b0c4de;
+} 
+</style>
+++close++
+
+==================================================================
+CONTENT section
+==================================================================
+
+++start++
+
+# <a id="top">aws-must demo 3 templates</a>
+
+This document is generated automically from `aws-must` -demo
+templates. 
+
+
+Output contains markdown syntax between
+**&plus;&plus;start&plus;&plus;** and
+**&plus;&plus;close&plus;&plus;** -tags from the template files. For
+each template, it
+
+* gives a general description of the template
+
+* documents the attribute context e.g. `.` or `./resources/Instance`,
+  and attributes references of the template
+
+* lists template actions, i.e.  output from template, or template
+  inclusion
+
+
+
+## <a id="root.mustache"></a>root.mustache <a class='navigator' href='#top'">[top]</a>
+
+Create an EC2 instance
+
+**Attributes**: context= `.`
+
+* `description`: description for the CF template
+
+**Actions**:
+
+* **Description**: `description`
+* **include** <a href="#resources.mustache">resources.mustache</a>
+
+> resources
+
+++close++
+
+==================================================================
+
+}}
+
+{
+  "AWSTemplateFormatVersion" : "2010-09-09",
+
+  "Description" : "{{description}}",
+
+
+
+  "Resources" : {
+
+    {{> resources }}
+
+  }
+
+}

data/demo/4/conf.yaml

@@ -0,0 +1,22 @@
+-- demo/2 : added description property
+
+
+description: "demo/4: a simple Amazon EC2 instance created using aws-must tool.
+	     The resource created are in defined in YAML configuration, and
+	     the JSON template should not change."
+
+
+parameters:
+
+        -  Name: InstanceType
+           Type: String
+           Description: EC2 reousrce instance type
+           Value: &Param_InstanceType t2.micro
+
+
+resources:
+
+        -  Instance: 
+                 Name: &Resource_1 MyEC2Instance
+                 ImageId: ami-00dae61d
+                 InstanceType: *Param_InstanceType

data/demo/4/resource.mustache

@@ -0,0 +1,27 @@
+{{!
+
+    resource.mustache
+
+++start++
+
+## <a id="resource.mustache"></a>resource.mustache <a class='navigator' href='#top'">[top]</a>
+
+Dispatches resource sub-type templates based resource Type propertys
+
+**Attributes**: context=  `./resources`
+
+* `Instance`: sub-document defining an EC instance
+
+
+**Actions**:
+
+* include <a href="#resourceInstance.mustache">resourceInstance.mustache</a> if `Instance` 
+
+> resourceInstance
+
+++close++
+
+
+}}
+
+{{# Instance }}{{> resourceInstance}}{{/ Instance }}

data/demo/4/resourceInstance.mustache

@@ -0,0 +1,32 @@
+{{!
+
+
+++start++
+
+## <a id="resourceInstance.mustache"></a>resourceInstance.mustache <a class='navigator' href='#top'">[top]</a>
+
+Create an EC2 instance
+
+**Attributes**: context=  `./resources/Instance`
+
+* `Name`: name of the EC2 instance to create
+* `ImageId`: AIM id to create
+
+**Actions**:
+
+* **Name**:  `Name`
+* **Type**: 'AWS::EC2::Instance'
+* **ImageId**: `ImageId`
+
+++close++
+
+}}
+
+
+    "{{Name}}" : {
+      "Type" : "AWS::EC2::Instance",
+      "Properties" : {
+        "ImageId" : "{{ImageId}}",
+        "InstanceType" : "{{InstanceType}}"
+      } {{! Propertites }}
+    }{{_comma}}

data/demo/4/resources.mustache

@@ -0,0 +1,25 @@
+{{! 
+
+++start++
+
+## resources.mustache
+
+Nothing created here
+
+++close++
+
+
+}}
+{{! 
+
+    Moved to YAML
+
+    "MyEC2Instance" : {
+      "Type" : "AWS::EC2::Instance",
+      "Properties" : {
+        "ImageId" : "ami-00dae61d",
+        "InstanceType" : "t2.micro"
+      }
+    }
+
+}}