terraframe 0.0.5

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 69c298bb5fae6f55bd0926b9f6ee7d51317a932c
+  data.tar.gz: 1e27ef8d075df5f45a8f32b0bc6790b303c575d3
+SHA512:
+  metadata.gz: 494647a04c86977f4652244a39984e201693628303619122bd50584a73e80c19c350b0e3c6df1392318de2de5829e8247f114e7b83bffd3dcb652986468b1da2
+  data.tar.gz: 4a5b5829ce32c10cf1f39e4e72b77246ada761a612ec5fe2da3f86fe5fa153e532f42d152c49bb1984639fe11839b3ea56f68ccd7fe0067e32dd2c21d140ea92

data/.gitignore

@@ -0,0 +1,15 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log
+*.gem

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in terraframe.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 Ed Ropple
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,60 @@
+# Terraframe #
+Terraframe is a processor for a domain-specific language that outputs to the Terraform specification format.
+
+[Terraform][1] is a cool infrastructure-as-code system with a boatload of functionality and a really awesome core that makes building out entire clouds on AWS super easy and super fun. I am a fan.
+
+_But_. But but but. It's not perfect. And personally, ever since HashiCorp went away from Ruby DSLs as configuration, I have been sad. And worse than sad, I have been unproductive as all hell. I'm just not satisfied with the state of the Terraform description language. It's hard to write, its built-in assumptions are gross (the handling of variables is a minor crime in progress), and it's just a huge step back from being able to deploy, say, a Vagrant instance--or, as I have done, [a configurable number of Vagrant instances][2]--in a nice, comprehensible manner.
+
+The configuration, while not CloudFormation-bad, is pretty bad, and I came close to ditching Terraform because it was really hard to write. I got halfway through some ERB templating monstrosity before I learned of a better way: Terraform supports JSON as a declaration notation, and that makes it really easy to build an external DSL that can be exported for use in Terraform.
+
+## Installation ##
+Terraframe is distributed as a RubyGem.
+
+```bash
+gem install terraframe
+```
+
+## Usage ##
+1. You'll need to have [Terraform][1] installed to actually use the output of Terraframe.
+2. Check `terraframe --help` for exhaustive usage details.
+
+## Syntax ##
+For the most part, the Terraframe syntax directly parallels the Terraform syntax, but has been Rubified. At present, most of this is a wrapper around `method_missing`, and so it's a _little_ rocky when dealing with nested data types.
+
+You can easily wire up outputs via `id_of(resource_type, resource_name)` and `output_of(resource_type, resource_name, output_name)`, to make things a little less crazy.
+
+Variables can be passed in via the `-v` flag in as many YAML files (which will be deep-merged) as you would like. They are exposed to scripts via the `vars` hash.
+
+Examples: TBD. To get you started, here's the supported invocations:
+
+- `provider :provider_type {}`
+- `resource :resource_type, "resource_name" {}`
+- `resource_type "resource_name {}`
+
+The latter deserves special attention, because any attempted invocation other than `provider`, `variable`, `resource`, or `provisioner` will be interpreted as a resource. (`resource :resource_type, "resource_name"` is supported for people who like the Terraform syntax.)
+
+## Future Work ##
+Right now, Terraframe is being extended as I need it. Pull requests very gratefully accepted for
+
+- Variable interpolation in YAML variable files (or possibly Ruby variable files that must emit a hash, a la Chef). This will be soon, I'll need it.
+- Supporting `variable` and `provisioner` blocks, neither are hard but they have to get done.
+- Script linting (checking for id existence, only allowing valid keys, etc.) before starting up Terraform. Very low priority, as that's what Terraform itself does.
+- Test coverage.
+
+## Contributing ##
+1. Fork it ( https://github.com/eropple/terraframe/fork )
+2. Create your feature branch under the `feature/` prefix, as with [git-flow][3] (`git flow feature start my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin feature/my-new-feature`). Please don't change the version number and please make README changes in a separate commit to make everything comprehensible for me.
+5. Make a pull request!
+
+## Thanks ##
+- My employer, [Leaf][4], for encouraging the development and use of Terraframe. ([We're hiring!][5])
+- William Lee, my platform engineering co-conspirator at Leaf.
+- HashiCorp, 'cause Terraform is fundamentally an awesome project and we're all better because you guys made this.
+
+[1]: https://www.terraform.io
+[2]: http://edcanhack.com/2014/07/a-virtual-mesos-cluster-with-vagrant-and-chef/
+[3]: https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow
+[4]: http://engineering.leaf.me
+[5]: mailto:eropple+hiring@leaf.me

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/bin/terraframe

@@ -0,0 +1,32 @@
+#! /usr/bin/env ruby
+
+lib = File.expand_path(File.dirname(__FILE__) + '/../lib')
+$LOAD_PATH.unshift(lib) if File.directory?(lib) && !$LOAD_PATH.include?(lib)
+
+require 'logger'
+require 'trollop'
+require 'terraframe'
+require 'awesome_print'
+
+opts = Trollop::options do
+  opt :input_file,    "Input file(s) to process into a Terraform script.",
+                      :short => "f", :type => :string, :multi => true, :required => true
+  opt :variable_file, "Variable file (YAML or JSON, not tfvars!).",
+                      :short => "v", :type => :string, :multi => true
+  opt :pretty_print,  "Pretty-prints the Terraform output.", :default => true
+  opt :verbose,       "Increases logging verbosity.", :default => false
+end
+
+processor = Terraframe::Processor.new
+if opts[:verbose]
+  processor.logger.level = Logger::DEBUG
+end
+processor.logger.debug opts.inspect
+
+output = processor.process_files(opts[:input_file], opts[:variable_file] || [])
+if opts[:pretty_print]
+  output = JSON.pretty_generate(JSON.parse(output))
+end
+
+processor.logger.info "Writing output to stdout."
+puts output

data/lib/terraframe.rb

@@ -0,0 +1,2 @@
+require "terraframe/version"
+require "terraframe/processor"

data/lib/terraframe/context.rb

@@ -0,0 +1,10 @@
+require 'terraframe/provider'
+require 'terraframe/resource'
+
+module Terraframe
+  class Context
+    def provider_type
+      raise "provider_type be implemented in subclassed contexts."
+    end
+  end
+end

data/lib/terraframe/contexts/aws_context.rb

@@ -0,0 +1,20 @@
+require 'terraframe/context'
+require 'terraframe/provider'
+
+module Terraframe
+  module Contexts
+    class AWSContext < Context
+      RESOURCES = [
+        :aws_instance
+      ]
+
+      def provider_type
+        Terraframe::Provider
+      end
+
+      def resources
+        RESOURCES
+      end
+    end
+  end
+end

data/lib/terraframe/processor.rb

@@ -0,0 +1,64 @@
+require 'logger'
+require 'digest/sha1'
+require 'yaml'
+
+require 'active_support/core_ext/hash'
+require 'awesome_print'
+
+require 'terraframe/state'
+require 'terraframe/contexts/aws_context'
+
+module Terraframe
+  class Processor
+    attr_reader :logger
+    attr_reader :contexts
+
+
+    def initialize
+      @logger = Logger.new($stderr)
+      logger.level = Logger::INFO
+
+      logger.debug "Logger initialized."
+
+      @contexts = {}
+      register_context(:aws, Terraframe::Contexts::AWSContext.new)
+    end
+
+    def register_context(name, context)
+      name = name.to_sym
+      if @contexts[name]
+        logger.warn "A context with the name '#{name}' has been registered more than once."
+      end
+      @contexts[name] = context
+    end
+
+    def process_files(scripts, variables)
+      scripts = scripts.map { |f| File.expand_path(f) }
+      variables = variables.map { |f| File.expand_path(f) }
+
+      missing_scripts = scripts.reject { |f| File.exist?(f) }
+      missing_variables = variables.reject { |f| File.exist?(f) }
+      unless missing_scripts.empty? && missing_variables.empty?
+        missing_scripts.each { |f| logger.fatal "Script file not found: #{f}" }
+        missing_variables.each { |f| logger.fatal "Variable file not found: #{f}" }
+        raise "One or more specified files were missing."
+      end
+
+      apply(Hash[scripts.zip(scripts.map { |f| IO.read(f) })], load_variables(variables))
+    end
+
+    def load_variables(variables)
+      vars = {}
+      variables.each { |f| vars = vars.deep_merge(YAML::load_file(f)) }
+      vars
+    end
+
+    def apply(inputs, vars)
+      logger.info "Beginning state execution."
+
+      state = State.new(logger, vars, @contexts)
+      inputs.each { |input| state.__apply_script(input[0], input[1])}
+      state.__build()
+    end
+  end
+end

data/lib/terraframe/provider.rb

@@ -0,0 +1,6 @@
+require 'terraframe/script_item'
+
+module Terraframe
+  class Provider < Terraframe::ScriptItem
+  end
+end

data/lib/terraframe/resource.rb

@@ -0,0 +1,9 @@
+require 'terraframe/script_item'
+
+module Terraframe
+  class Resource < Terraframe::ScriptItem
+    def resource_type
+      raise "resource_type must be overridden in inheriting classes."
+    end
+  end
+end

data/lib/terraframe/script_item.rb

@@ -0,0 +1,36 @@
+require 'json'
+
+module Terraframe
+  class ScriptItem
+    attr_reader :fields
+    attr_reader :vars
+
+    def initialize(vars, &block)
+      @fields = {}
+      @vars = vars
+
+      instance_eval &block
+    end
+
+    def to_json(*a)
+      @fields.to_json(*a)
+    end
+
+    ## DSL FUNCTIONS BELOW
+    def method_missing(method_name, *args, &block)
+      if args.length == 1
+        @fields[method_name.to_sym] = args[0]
+      else
+        raise "Multiple fields passed to a scalar auto-argument '#{method_name}'."
+      end
+    end
+
+
+    def output_of(resource_type, resource_name, output_type)
+      "${#{resource_type}.#{resource_name}.#{output_type}}"
+    end
+    def id_of(resource_type, resource_name)
+      output_of(resource_type, resource_name, :id)
+    end
+  end
+end

data/lib/terraframe/state.rb

@@ -0,0 +1,96 @@
+require 'json'
+require 'recursive_open_struct'
+
+module Terraframe
+  class State
+    attr_reader :vars
+    attr_reader :logger
+
+    def initialize(logger, vars, contexts)
+      @logger = logger
+      logger.info "Initializing state."
+
+      @vars = RecursiveOpenStruct.new(vars, :recurse_over_arrays => true )
+      logger.debug "State variables:"
+      logger.ap vars, :debug
+
+      @__contexts = contexts
+
+      @__output = {
+        :provider => {},
+        :variable => {},
+        :resource => {}
+      }
+    end
+
+    def __build()
+      logger.info "Building Terraform script from state."
+      logger.debug "Contexts:"
+      @__contexts.each { |c| logger.debug " - #{c}" }
+      @__output.to_json
+    end
+
+    def __apply_script(script_name, script)
+      logger.info "Applying script '#{script_name}' to state."
+      instance_eval(script, script_name, 0)
+      logger.info "Script '#{script_name}' applied successfully."
+    end
+
+    ## DSL FUNCTIONS BELOW ##
+    def provider(type, &block)
+      if !@__contexts[type]
+        msg = "Unknown provider type: '#{type}'."
+        logger.fatal msg
+        raise msg
+      end
+
+      if @__output[:provider][type]
+        msg = "Duplicate provider type (sorry, blame Terraform): '#{type}'"
+        logger.fatal msg
+        raise msg
+      end
+
+      provider = @__contexts[type].provider_type.new(vars, &block)
+      logger.debug "Provider of type '#{type}': #{provider.inspect}"
+      @__output[:provider][type] = provider
+
+      provider
+    end
+
+    def variable
+      msg = "TODO: implement tfvar support."
+      logger.fatal msg
+      raise msg
+    end
+
+    def provisioner
+      msg = "TODO: implement provisioner support."
+      logger.fatal msg
+      raise msg
+    end
+
+    def resource(resource_type, resource_name, &block)
+      unless @__contexts.any? { |k, v| v.resources.include?(resource_type) }
+        logger.warn "Could not find a context that supports resource type '#{resource_type}'. Continuing, but you've been warned."
+      end
+
+      @__output[:resource][resource_type] ||= {}
+      @__output[:resource][resource_type][resource_name.to_s] = Resource.new(vars, &block)
+    end
+
+    # anything that is not a provider or a variable should be interpreted 
+    def method_missing(method_name, *args, &block)
+      case method_name
+        when "vars"
+          @vars
+        else
+          if (args.length != 1)
+            msg = "Too many arguments for resource invocation '#{method_name}'."
+            logger.fatal(msg)
+            raise msg
+          end
+          resource(method_name.to_sym, args[0], &block)
+      end
+    end
+  end
+end

data/lib/terraframe/version.rb

@@ -0,0 +1,3 @@
+module Terraframe
+  VERSION = "0.0.5"
+end

data/terraframe.gemspec

@@ -0,0 +1,28 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'terraframe/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "terraframe"
+  spec.version       = Terraframe::VERSION
+  spec.authors       = ["Ed Ropple"]
+  spec.email         = ["ed@edropple.com"]
+  spec.summary       = "A sane Ruby-based DSL for emitting Terraform scripts."
+  # spec.description   = %q{TODO: Write a longer description. Optional.}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler",                "~> 1.7"
+  spec.add_development_dependency "rake",                   "~> 10.0"
+
+  spec.add_runtime_dependency     "trollop",                "~> 2.0"
+  spec.add_runtime_dependency     "activesupport",          "~> 4.2.0"
+  spec.add_runtime_dependency     "awesome_print",          "~> 1.6.1"
+  spec.add_runtime_dependency     "recursive-open-struct",  "~> 0.5.0"
+end

data/test/scripts/aws.terraframe

@@ -0,0 +1,12 @@
+provider :aws do
+  access_key "boop"
+  secret_key "bop"
+
+  region "us-east-1"
+end
+
+(0...vars.test_vars.num_jimmies).each do |i|
+  aws_instance "jimmy-#{i}" do
+    vpc_id id_of(:aws_vpc, "main-vpc")
+  end
+end

data/test/scripts/aws_vars.yaml

@@ -0,0 +1,2 @@
+aws:
+    thing: 5

data/test/scripts/basic_vars.yaml

@@ -0,0 +1,2 @@
+test_vars:
+    num_jimmies: 6

metadata

@@ -0,0 +1,157 @@
+--- !ruby/object:Gem::Specification
+name: terraframe
+version: !ruby/object:Gem::Version
+  version: 0.0.5
+platform: ruby
+authors:
+- Ed Ropple
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-01-10 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: trollop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 4.2.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 4.2.0
+- !ruby/object:Gem::Dependency
+  name: awesome_print
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.6.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.6.1
+- !ruby/object:Gem::Dependency
+  name: recursive-open-struct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.5.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.5.0
+description: 
+email:
+- ed@edropple.com
+executables:
+- terraframe
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/terraframe
+- lib/terraframe.rb
+- lib/terraframe/context.rb
+- lib/terraframe/contexts/aws_context.rb
+- lib/terraframe/processor.rb
+- lib/terraframe/provider.rb
+- lib/terraframe/resource.rb
+- lib/terraframe/script_item.rb
+- lib/terraframe/state.rb
+- lib/terraframe/version.rb
+- terraframe.gemspec
+- test/aws_tests.rb
+- test/basic_tests.rb
+- test/scripts/aws.terraframe
+- test/scripts/aws_vars.yaml
+- test/scripts/basic.terraframe
+- test/scripts/basic_vars.yaml
+homepage: ''
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: A sane Ruby-based DSL for emitting Terraform scripts.
+test_files:
+- test/aws_tests.rb
+- test/basic_tests.rb
+- test/scripts/aws.terraframe
+- test/scripts/aws_vars.yaml
+- test/scripts/basic.terraframe
+- test/scripts/basic_vars.yaml