regentanz 0.3.3 → 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7dd89f8efd4c50cc933e407205c64dd257895ae6
+  data.tar.gz: 57910386c3c39c3c7de7aedd66dc2e133a4cd6a1
+SHA512:
+  metadata.gz: 2147f883198e82a80475c32f2610267793fc54a31968974041be14e0d35eb0c4f86bde50c7ec16b99512db6f360ca64e180c96f8d295adc2dbdd27aa56a6cab9
+  data.tar.gz: 61c4ed079798cad6ae1e71d1838fafc4838c714a43e66d7056085eab1857940aadab2e0700052ed114b9f365d51a9e02f8890337cdf5767e52571ffd4a8196d9

data/README.md

@@ -0,0 +1,268 @@
+# Regentanz
+
+[![Build Status](https://travis-ci.org/burtcorp/regentanz.png?branch=master)](https://travis-ci.org/burtcorp/regentanz)
+
+_If you're reading this on GitHub, please note that this is the readme for the development version and that some features described here might not yet have been released. You can find the readme for a specific version via the release tags ([here is an example](https://github.com/burtcorp/regentanz/tree/1.0.0))._
+
+Regentanz is a compiler and preprocessor for CloudFormation templates. It allows you to split up a template into one file per resource, and also build custom resources that can decrease the complexity of templates.
+
+> _Looking for Regentanz, Carsten Zimmermann's "Library to access the Google Weather API"? It can still be found at https://github.com/carpodaster/regentanz and if you install v0.3.3 from Rubygems, you'll get that gem._
+
+## Installation
+
+Install it on the command line:
+
+```shell
+$ gem install regentanz --version '~> 1.0.0'
+```
+
+or add it to your `Gemfile`:
+
+```ruby
+gem 'regentanz', '~> 1.0.0'
+```
+
+## How to build and run the tests
+
+The best place to see how to build and run the tests is to look at the `.travis.yml` file, but if you just want to get going run:
+
+```shell
+$ bundle
+$ rake
+```
+
+## Examples
+
+The `examples` directory contains runnable example templates.
+
+## Usage
+
+To compile a template you use the `regentanz` command like this:
+
+```shell
+$ regentanz compile path/to/template > path/to/compiled/template.json
+```
+
+### Template validation
+
+The compiler will validate the final template with CloudFormation, so you will need to run it with AWS credentials that permit `cloudformation:ValidateTemplate`.
+
+For the validation to work with templates larger than 51200 bytes you need to specify an S3 bucket in a config file, and need AWS credentials that permit `s3:PutObject` in that bucket.
+
+To configure a bucket to use for validation of large templates create a file called `.regentanz.yml` in the directory where you will run Regentanz, and add the following:
+
+```yaml
+template_url: 's3://some-bucket-name/a_prefix/${TEMPLATE_NAME}.json'
+```
+
+You can use `${AWS_REGION}` in the bucket portion, and `${TEMPLATE_NAME}` and `${TIMESTAMP}` in the key portion to create unique URLs. You can, for example, use the buckets that CloudFormation creates for template uploads, like this:
+
+```yaml
+template_url: 's3://cf-templates-xyz-${AWS_REGION}/regentanz/${TEMPLATE_NAME}-${TIMESTAMP}.json'
+```
+
+Where "xyz" is the unique letter combination for your account.
+
+### Anatomy of a template
+
+Just like a CloudFormation template, a Regentanz template consists of conditions, mappings, outputs, parameters, and resources. In contrast with CloudFormation these are not properties of one big JSON or YAML document, but exists as separate files in a directory structure. Each resource has its own file, and there is one file each for conditions, mappings, and parameters. If the template doesn't need conditions, mappings, or parameters these files can be left out.
+
+The Regentanz compiler will take a directory and output a CloudFormation template in JSON format that can be used with CloudFormation.
+
+#### Example
+
+Say you have an application with an auto scaling group, launch configuration, security group, an instance profile, and an IAM role for the instance profile. In a regular CloudFormation template you would declare five different resources in one big JSON or YAML file, but in Regentanz you would instead keep these in five separate files, something like this:
+
+```
+my_application/
+  resources/
+    auto_scaling_group.yml
+    iam_role.yml
+    instance_profile.yml
+    launch_configuration.yml
+    security_group.yml
+```
+
+You're free to name the files whatever you want, and you can put them in subdirectories too. This is an alternative way to structure the same template:
+
+```
+my_application/
+  resources/
+    iam/
+      role.yml
+      instance_profile.yml
+    asg.yml
+    lc.yml
+    sg.yml
+```
+
+If you have conditions, mappings, outputs, or parameters you put these in files at the top level:
+
+my_application/
+  resources/
+    iam/
+      role.yml
+      instance_profile.yml
+    asg.yml
+    lc.yml
+    sg.yml
+  conditions.yml
+  mappings.yml
+  outputs.yml
+  parameters.yml
+```
+
+You can use JSON or YAML for your files, and mix between these in the same template.
+
+The contents of the files is the same thing you would put in a CloudFormation template. In other words, if your CloudFormation looks something like this:
+
+```yaml
+Parameters:
+  ServerCount:
+    Type: Number
+    Default: 2
+    MinValue: 0
+
+Resources:
+  Asg:
+    Type: AWS::EC2::AutoScalingGroup
+    Properties:
+      DesiredCapacity: !Ref ServerCount
+      # …
+  Lc:
+    Type: AWS::EC2::LaunchConfiguration
+    Properties:
+      # …
+```
+
+You would put the parameters in `parameters.yml`:
+
+```yaml
+Parameters:
+  ServerCount:
+    Type: Number
+    Default: 2
+    MinValue: 0
+```
+
+The auto scaling group resource in `resources/asg.yml`:
+
+```yaml
+Type: AWS::EC2::AutoScalingGroup
+Properties:
+  DesiredCapacity: {Ref: DesiredCapacity}
+  # …
+```
+
+Unfortunately CloudFormation's YAML syntax for intrinsic functions (e.g. `!Ref`) is not supported, you have to convert it to YAML/JSON (e.g. `{Ref: …}`), like above.
+
+The launch configuration resource would go in a file called `resources/lc.yml`:
+
+```yaml
+Type: AWS::EC2::LaunchConfiguration
+Properties:
+  # …
+```
+
+And when you compile this with the Regentanz compiler you would get the same CloudFormation template back, but in JSON format.
+
+### Resource references
+
+The Regentanz compiler will generate resource names based on the relative paths of the files in the template. Currently the scheme is to take the path relative to the root of the template and create a `CamelCase` with no underscores or slashes. You should however not rely on this convention since it could change in the future. Instead you should use two macros provided by Regentanz that work similar to CloudFormation's `Ref` function.
+
+Wherever you would have used `Ref` in CloudFormation you should use `ResolveRef`, and use the relative path, minus file ending as argument. In the template in the example above you could for example refer to the IAM role with `{ResolveRef: iam/role}`, and the auto scaling group as `{ResolveRef: asg}`.
+
+In places where you in CloudFormation would have used the name of a resource directly you should use `ResolveName`. For example `{ResolveName: iam/role}` and `{ResolveName: asg}`. Almost the only place you will need `ResolveName` is in `GetAtt`.
+
+Continuing on the example above the auto scaling group needs to refer to the launch configuration. Using `ResolveRef` that would look like this:
+
+```yaml
+Type: AWS::EC2::AutoScalingGroup
+Properties:
+  LaunchConfigurationName: {ResolveRef: asg}
+  # …
+```
+
+## Custom Resources
+
+If you've written a lot of CloudFormation templates you probably feel like you're constantly repeating yourself. The same patterns appear again and again in multiple templates. To help with this Regentanz lets you write custom resources that can be used in templates and that generate other resources, parameters, mappings, etc. when compiled.
+
+A custom resource is a Ruby class that lives in the `Regentanz::Resources` module that has a `#compile` method that returns a partial template.
+
+### Anatomy of a custom resource
+
+Custom resources will be instantiated by the template compiler and their `#compile` method will be called with the name of the resource and the resource template. The result of this call must be a "template fragment", which is a hash with a `:resources` key, and optionally `:conditions`, `:mappings`, `:outputs`, and `:parameters`. Each of these must be (when specified) a hash that looks like the corresponding CloudFormation structure
+
+Say you used a custom resource like the following, in a file with the relative path `app/sg.yml`
+
+```yaml
+Type: Regentanz::Resources::MySpecialSecurityGroup
+Properties:
+  Name: special-sg
+  PortsOpenToEveryone:
+    - 22
+    - 80
+```
+
+The compiler will, conceptually, do this (`template_fragment` is a the contents of the file):
+
+```ruby
+resource = Regentanz::Resources::MyCustomResource
+result = resource.compile('AppSg', template_fragment)
+```
+
+It will then take the result and merge it with the rest of the template.
+
+This is how you could implement `MySpecialSecurityGroup`:
+
+```ruby
+class Regentanz::Resources::MyCustomResource
+  def compile(name, template)
+    ingress_rules = template['Properties']['PortsOpenToEveryone'].map do |port|
+      {'IpProtocol' => 'tcp', 'FromPort' => port, 'ToPort' => port, 'CidrIp' => '0.0.0.0/0'}
+    end
+    {
+      :resources => {
+        name => {
+          'Type' => 'AWS::EC2::SecurityGroup',
+          'Properties' => {
+            'GroupName' => properties['Name'],
+            'SecurityGroupIngress' => ingress_rules
+          }
+        }
+      }
+    }
+  end
+end
+```
+
+You are free to ignore the `name` parameter, but it is strongly recommended that you use it. It is the name Regentanz has generated from the relative path of the file the resource is declared in, so ignoring it will make it harder and more confusing to refer to the resource the custom resource generates. If you generate more than one resource in your template it is recommended that you use `name` as a prefix. For example if you generate an auto scaling group and a launch configuration from the a custom resource generating resources with names like `"#{name}Asg"`, and `"#{name}Lc"` will make it possible to do `{ResolveRef: my_resource/asg}` in another resource.
+
+If the template fragment returned by `#compile` contains `ResolveRef` or `ResolveName` these will be resolved as expected.
+
+### Adding custom resources to the load path
+
+Custom resources must be available on the load path when the template compiler runs. The name of the file must also follow the standard Ruby naming convention, i.e. a resource called `Regentanz::Resources::MyCustomResource` must be declared in a file with the path `regentanz/resources/my_custom_resource.rb`, so that the compiler knows which file to load.
+
+This can be achieved by manipulating environment variables like `RUBYLIB`, or by packaging your custom resources as a gem, but in most cases the easiest way is to add a config file that tells the template compiler how to modify the load path to be able to load your custom resources.
+
+You can add a file called `.regentanz.yml` in any parent directory of the directory where you run the compiler. In the file you put a list of paths to add to the Ruby load path:
+
+```yaml
+load_path:
+  - path/to/resources
+  - /an/absolute/path
+```
+
+Relative paths will be resolved relative to the config file.
+
+The config file above will allow the template compiler to find custom resources in files such as `path/to/resources/regentanz/resources/my_custom_resource.rb` (relative to the config file) and `/an/absolute/path/regentanz/resources/my_custom_resource.rb`.
+
+## Limitations
+
+* Regentanz unfortunately does not support CloudFormation's YAML syntax for intrinsic functions.
+* The support for validating large templates is currently broken.
+
+# Copyright
+
+© 2015-2018 Burt AB, see LICENSE.txt (BSD 3-Clause).

data/bin/regentanz

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH << File.expand_path('../../lib', __FILE__)
+
+require 'regentanz/yaml-ext'
+
+case ARGV.first
+when 'compare'
+  ARGV.shift
+  require 'regentanz/cli/compare'
+  exit(Regentanz::Cli::Compare.new.run(ARGV))
+else
+  ARGV.shift if ARGV.first == 'compile'
+  require 'regentanz/cli/compile'
+  exit(Regentanz::Cli::Compile.new.run(ARGV))
+end

data/lib/regentanz.rb

@@ -1,11 +1,10 @@
-require 'active_support'
-require 'active_support/core_ext'
-require 'regentanz/version'
-require 'regentanz/cache'
-require 'regentanz/configuration'
-require 'regentanz/configurator'
-require 'regentanz/callbacks'
-require 'regentanz/astronomy'
-require 'regentanz/conditions'
-require 'regentanz/parser'
-require 'regentanz/google_weather'
+require 'json'
+require 'yaml'
+require 'aws-sdk-s3'
+require 'aws-sdk-cloudformation'
+
+module Regentanz
+  Error = Class.new(StandardError)
+end
+
+require 'regentanz/template_compiler'

data/lib/regentanz/cli/common.rb

@@ -0,0 +1,35 @@
+require 'yaml'
+
+module Regentanz
+  module Cli
+    module Common
+      private def load_config
+        if (path = find_config('.'))
+          config = YAML.load_file(path)
+          Array(config['load_path']).each do |extra_load_path|
+            if extra_load_path.start_with?('/')
+              $LOAD_PATH << extra_load_path
+            else
+              $LOAD_PATH << File.absolute_path(extra_load_path, File.dirname(path))
+            end
+          end
+          if config['default_region'].nil? && (region = ENV['AWS_REGION'] || ENV['AWS_DEFAULT_REGION'])
+            config['default_region'] = region
+          end
+          config
+        end
+      end
+
+      private def find_config(path)
+        candidates = Dir[File.join(path, '.regentanz.{yaml,yml,json}')]
+        if candidates.first
+          candidates.first
+        elsif path != '/'
+          find_config(File.expand_path(File.join(path, '..')))
+        else
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/regentanz/cli/compare.rb

@@ -0,0 +1,85 @@
+require 'regentanz'
+require 'regentanz/cli/common'
+
+module Regentanz
+  module Cli
+    class Diff
+      def initialize(new_object, old_object)
+        @new_object = new_object
+        @old_object = old_object
+      end
+
+      def to_json(*args)
+        %(\e[31m#{@old_object.to_json(*args)}\e[m\e[32m#{@new_object.to_json(*args)}\e[m)
+      end
+    end
+
+    class Compare
+      include Common
+
+      def run(args)
+        config = load_config
+        stack_name, stack_path, _ = *args
+        compiler = TemplateCompiler.new(config)
+        new_template = compiler.compile_from_path(stack_path)
+        compiler.validate_template(stack_path, new_template.to_json)
+        old_template = get_template(config, stack_name)
+        diff = compare(new_template, old_template)
+        if diff.to_json != new_template.to_json
+          output = JSON.pretty_generate(diff)
+          puts(output)
+          1
+        else
+          0
+        end
+      rescue Regentanz::Error => e
+        $stderr.puts(e.message)
+        2
+      end
+
+      private
+
+      def get_template(config, stack_name)
+        cf_client = Aws::CloudFormation::Client.new(region: config['default_region'])
+        YAML.load(cf_client.get_template(stack_name: stack_name)[:template_body])
+      rescue Aws::Errors::MissingCredentialsError => e
+        raise Regentanz::Error, 'Retrieving template requires AWS credentials', e.backtrace
+      end
+
+      def compare(new_template, old_template)
+        if new_template.is_a?(Date) && old_template.is_a?(String)
+          new_template = new_template.to_s
+        end
+        if new_template.class != old_template.class
+          Diff.new(new_template, old_template)
+        else
+          case new_template
+          when Hash
+            result = {}
+            new_template.each do |(key, value)|
+              result[key] = compare(value, old_template[key])
+            end
+            old_template.each do |(key, value)|
+              result[key] = compare(nil, value) unless new_template.key?(key)
+            end
+            result
+          when Array
+            if new_template.size != old_template.size
+              Diff.new(new_template, old_template)
+            else
+              new_template.zip(old_template).map do |(left, right)|
+                compare(left, right)
+              end
+            end
+          else
+            if new_template == old_template
+              new_template
+            else
+              Diff.new(new_template, old_template)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/regentanz/cli/compile.rb

@@ -0,0 +1,27 @@
+require 'regentanz'
+require 'regentanz/cli/common'
+
+module Regentanz
+  module Cli
+    class Compile
+      include Common
+
+      def run(args)
+        config = load_config
+        stack_path = args.first
+        compiler = Regentanz::TemplateCompiler.new(config)
+        template = compiler.compile_from_path(stack_path)
+        template_json = JSON.pretty_generate(template)
+        if template_json.bytesize >= 51200
+          template_json = JSON.generate(template)
+        end
+        compiler.validate_template(stack_path, template_json)
+        puts(template_json)
+        0
+      rescue Regentanz::Error => e
+        $stderr.puts(e.message)
+        1
+      end
+    end
+  end
+end