terraspace 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6cfa0c4e6aa1aa9871bc8b1902b283f2b91f5392738a066fdc58efb56153bed1
-  data.tar.gz: 2219a5bbbd4dbad228e406d7ce586cf5bfdb8392868478e59e3adbd6dc4adda5
+  metadata.gz: b6c5d0afc0bdc18ff6f10c79369d60f8a722ecede426ce37880993e5f9676860
+  data.tar.gz: 3abc9b03ef0a7766491116747fa0d0b6a0fe7d41fc67098e8d2d08a8c30e4a0f
 SHA512:
-  metadata.gz: f3bb558668074f2e278359a887e369c5182c95fd306973a9cae6c4bd7f95d9be653ce41b923e41e5470350c35234443a0ae3d4e16d5c8b35763feab9393a9c12
-  data.tar.gz: 24acaffec5e568152133e014697404cc9d80a2bd79d54b700584db93a5884ca184508544519d349e833a17f84ac1cc4f2d095a787ed5cdb5216f25b37f04fb50
+  metadata.gz: 680b8fed8c8487b85b27007750106944aad12606aa5a3bcffef40c249312b03a957867e8bf4c42a6ba4f332b498db031d48ae63875623c436b0fd89ab73fe8f7
+  data.tar.gz: 7fc9f978acf6d5759a892530e57ea42839f10201a5cbf14d5db46256d382a1b1b2c86744f4b47f930ee20a2944dff30c58d7d9c406d810e93fc9f73535f1bc9c

data/CHANGELOG.md

@@ -3,6 +3,36 @@
 All notable changes to this project will be documented in this file.
 This project *loosely tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.3.0]
+* All commands: Dependency graph calculated and deployed in proper order
+* All commands: `terraspace all up`, `terraspace all build`, `terraspace all down`, etc. `terraspace all -h`
+* Terraspace log: view and tail log files
+* Terraspace logs management commands: `terraspace logs truncate` and `terraspace logs remove`
+* TFC/TFE: Improve support. `config.cloud.vars`, `config.cloud.workspace.attrs`
+* TFC commands: terraspace cloud runs list, terraspace cloud runs prune
+* TFC VCS also sync as part of deploy. Also separate `terraspace cloud sync` command
+* Logger improvements: configurable formatter, log to stderr by default
+* Rename: `cloud.relative_root` to `cloud.working_dir_prefix`
+* Run a plan to capture the diff as part of `-y` option. IE: `terraspace up demo -y`
+* Run plan -destroy as part of down -y
+* Rename update cli to up. Only support the shorthand.
+* Config options: config.all.concurrency, config.all.exit_on_fail, etc
+* TFC: cloud.auto_sync option
+* Improve terraform version check
+* Terraform init: auto mode will retry initializing up to 3 times
+* Terraspace seed: fix instance option
+
+## [0.2.4]
+- fix version check for some versions of terraform
+## [0.2.3]
+* #37 config.clean_cache option
+
+## [0.2.2]
+* #36 cloud.relative_root setting
+
+## [0.2.1]
+* #35 fix summary bug when ran multiple times with different envs
+
 ## [0.2.0]
 * #34 Terraform Cloud and Terraform Enterprise support added.
 * TFC Vars support: JSON and DSL. config.overwrite and config.overwrite_sensitive configs
@@ -25,10 +55,10 @@ This project *loosely tries* to adhere to [Semantic Versioning](http://semver.or
 * terraspace up: --reconfigure option. This is useful if upgrading Terraform version.
 
 ## [0.1.2]
-- #33 rspec-terraspace dependency added
+* #33 rspec-terraspace dependency added
 
 ## [0.1.1]
-- #32 terraspace summary --short option
+* #32 terraspace summary --short option
 
 ## [0.1.0]
-- Initial release
+* Initial release

data/README.md

@@ -1,3 +1,7 @@
+<div align="center">
+  <a href="https://terraspace.cloud"><img src="https://img.boltops.com/boltops/logos/terraspace-dark-v2.png" /></a>
+</div>
+
 # Terraspace
 
 [![Gem Version](https://badge.fury.io/rb/terraspace.png)](http://badge.fury.io/rb/terraspace)
@@ -8,6 +12,10 @@ The Terraform Framework.
 
 Official Docs Site: [terraspace.cloud](https://terraspace.cloud)
 
+Introduction Video:
+
+[![Watch the video](https://img.boltops.com/boltops/tools/terraspace/terraspace-youtube.png)](https://www.youtube.com/watch?v=O87t5q22YNc)
+
 ## Quick Start
 
 Here are commands to get started:
@@ -17,8 +25,8 @@ Here are commands to get started:
     terraspace up demo
     terraspace down demo
 
-* The `new` command generates a starter project. 
-* The `up` command creates an s3 bucket. 
+* The `new` command generates a starter project.
+* The `up` command creates an s3 bucket.
 * The `down` command cleans up and deletes the bucket.
 
 The default plugin is aws. Other plugins also supported are: google and azurerm.
@@ -28,10 +36,10 @@ The default plugin is aws. Other plugins also supported are: google and azurerm.
 Create infrastructure:
 
     $ terraspace up demo -y
-    Building .terraspace-cache/dev/stacks/demo
-    Current directory: .terraspace-cache/dev/stacks/demo
-    => terraform init -get > /dev/null
-    Built in .terraspace-cache/dev/stacks/demo
+    Building .terraspace-cache/us-west-2/dev/stacks/demo
+    Current directory: .terraspace-cache/us-west-2/dev/stacks/demo
+    => terraform init -get > /tmp/terraspace/out/terraform-init20200824-21379-bkfvnh.out
+    Built in .terraspace-cache/us-west-2/dev/stacks/demo
     => terraform apply -auto-approve
     random_pet.bucket: Creating...
     random_pet.bucket: Creation complete after 0s [id=amusing-mouse]
@@ -48,10 +56,10 @@ Create infrastructure:
 Destroy infrastructure:
 
     $ terraspace down demo -y
-    Building .terraspace-cache/dev/stacks/demo
-    Current directory: .terraspace-cache/dev/stacks/demo
-    => terraform init -get > /dev/null
-    Built in .terraspace-cache/dev/stacks/demo
+    Building .terraspace-cache/us-west-2/dev/stacks/demo
+    Current directory: .terraspace-cache/us-west-2/dev/stacks/demo
+    => terraform init -get > /tmp/terraspace/out/terraform-init20200824-21379-bkfvnh.out
+    Built in .terraspace-cache/us-west-2/dev/stacks/demo
     => terraform destroy -auto-approve
     random_pet.bucket: Refreshing state... [id=amusing-mouse]
     module.bucket.aws_s3_bucket.this: Refreshing state... [id=bucket-amusing-mouse]
@@ -63,13 +71,26 @@ Destroy infrastructure:
     Destroy complete! Resources: 2 destroyed.
     $
 
+## Deploy Multiple Stacks
+
+To deploy all the infrastructure stacks:
+
+    terraspace all up
+
+To choose multiple stacks to deploy
+
+    terraspace all up instance vpc
+
+When you use the all command, the dependency graph is calculated and the stacks are deployed in the right order. To learn more: [Deploy Multiple Stacks](https://terraspace.cloud/docs/dependencies/deploy-all/).
+
 ## Features
 
 * [Config Structure](https://terraspace.cloud/docs/config/): A common config structure that gets materializes with the deployed module. Configs can be dynamically controlled to keep your code DRY. You can override the settings if needed, like for using existing backends. See: [Existing Backends](https://terraspace.cloud/docs/state/existing/).
 * [Generators](https://terraspace.cloud/docs/generators/): Built-in generators to quickly create the starter module. Focus on code instead of boilerplate structure.
-* [Tfvars](https://terraspace.cloud/docs/tfvars/): Use the same code with different tfvars to create multiple environments. Terraspace conventionally loads tfvars from the `tfvars` folder.
-* [Layering](https://terraspace.cloud/docs/tfvars/layering/): Rich layering support. This allows you to build different environments like dev and prod with the same code.
-* [Testing](https://terraspace.cloud/docs/testing/): A testing framework that allows you to create test harnesses, deploy real-resources, and have higher confidence that your code works.
+* [Tfvars](https://terraspace.cloud/docs/tfvars/) & [Layering](https://terraspace.cloud/docs/tfvars/layering/): Use the same code with different tfvars to create multiple environments. Terraspace conventionally loads tfvars from the `tfvars` folder. Rich layering support allows you to build different environments like dev and prod with the same code.  Examples are in [Full Layering](https://terraspace.cloud/docs/tfvars/full-layering/).
+* [Deploy Multiple Stacks](https://terraspace.cloud/docs/dependencies/deploy-all/): The ability to deploy multiple stacks with a single command. Terraspace calculates the [dependency graph](https://terraspace.cloud/docs/dependencies/) and deploys stacks in the right order. You can also target specific stacks and deploy [subgraphs](https://terraspace.cloud/docs/dependencies/subgraphs/).
 * [Configurable CLI](https://terraspace.cloud/docs/cli/): Configurable [CLI Hooks](https://terraspace.cloud/docs/cli/hooks/) and [CLI Args](https://terraspace.cloud/docs/cli/args/) allow you to adjust the underlying terraform command.
+* [Testing](https://terraspace.cloud/docs/testing/): A testing framework that allows you to create test harnesses, deploy real-resources, and have higher confidence that your code works.
+* [Terraform Cloud and Terraform Enterprise Support](https://terraspace.cloud/docs/cloud/): TFC and TFE are both supported. Terraspace adds additional conveniences to make working with Terraform Cloud Workspaces easier.
 
 For more info: [terraspace.cloud](https://terraspace.cloud)

data/lib/templates/base/git_hook/hook.sh

@@ -1,5 +1,5 @@
 #!/bin/bash -ex
-echo "Running: terraspace build placeholder"
+echo "Running: terraspace build"
 <%= terraspace_build_commands %>
 git add .terraspace-cache
 git commit -m 'update .terraspace-cache' || true

data/lib/templates/base/project/.gitignore

@@ -23,6 +23,7 @@ terraform.tfvars
 /Gemfile.lock
 /InstalledFiles
 /lib/bundler/man
+/log
 /output
 /pkg
 /rdoc

data/lib/templates/base/project/README.md

@@ -0,0 +1,17 @@
+# Terraspace Project
+
+This is a Terraspace project. It contains code to provision Cloud infrastructure built with [Terraform](https://www.terraform.io/) and the [Terraspace Framework](https://terraspace.cloud/).
+
+## Deploy
+
+To deploy all the infrastructure stacks:
+
+    terraspace all up
+
+To deploy individual stacks:
+
+    terraspace up demo # where demo is app/stacks/demo
+
+## Terrafile
+
+To add more modules for use, you can add them to the [Terrafile](https://terraspace.cloud/docs/terrafile/).

data/lib/terraspace.rb

@@ -1,16 +1,19 @@
 $stdout.sync = true unless ENV["TS_STDOUT_SYNC"] == "0"
 
 $:.unshift(File.expand_path("../", __FILE__))
+require "active_support/concern"
 require "active_support/core_ext/class"
 require "active_support/core_ext/hash"
 require "active_support/core_ext/string"
 require "active_support/ordered_options"
+require "deep_merge/rails_compat"
 require "dsl_evaluator"
 require "fileutils"
 require "json"
 require "memoist"
 require "rainbow/ext/string"
 require "render_me_pretty"
+require "set"
 require "terraspace/ext"
 require "terraspace/version"
 
@@ -22,6 +25,7 @@ DslEvaluator.backtrace_reject = "lib/terraspace"
 module Terraspace
   extend Core # for Terraspace.root
   class Error < StandardError; end
+  class InitRequiredError < Error; end
 end
 
 Terraspace::Booter.boot

data/lib/terraspace/all/base.rb

@@ -0,0 +1,8 @@
+module Terraspace::All
+  class Base
+    def initialize(options={})
+      @options = options
+      Terraspace.check_project! unless ENV['TS_TEST']
+    end
+  end
+end

data/lib/terraspace/all/grapher.rb

@@ -0,0 +1,129 @@
+require "graph"
+require "tty-tree"
+
+module Terraspace::All
+  class Grapher < Base
+    include Terraspace::Util::Logging
+
+    def run
+      check_graphviz!
+      logger.info "Building graph..."
+      builder = Terraspace::Builder.new(@options.merge(mod: "placeholder", quiet: true, draw_full_graph: draw_full_graph))
+      builder.run
+      graph = builder.graph
+      if @options[:format] == "text"
+        text(graph.top_nodes)
+      else
+        draw(graph.nodes)
+      end
+    end
+
+    def text(nodes)
+      Rainbow.enabled = false unless @options[:full]
+      data = build_tree_data(nodes)
+      Rainbow.enabled = true unless @options[:full]
+      tree = TTY::Tree.new(data)
+      logger.info tree.render
+    end
+
+    def build_tree_data(nodes)
+      if nodes.size == 1
+        tree_data(nodes.first)
+      else
+        root = Terraspace::Dependency::Node.new('.')
+        nodes.each { |node| node.parent!(root) }
+        tree_data(root)
+      end
+    end
+
+    def tree_data(parent, data={})
+      parent_name = text_name(parent)
+      data[parent_name] ||= []
+      parent.children.each do |child|
+        child_name = text_name(child)
+        if child.children.empty? # leaf node
+          data[parent_name] << child_name
+        else
+          next_data = { child_name => [] }
+          data[parent_name] << tree_data(child, next_data)
+        end
+      end
+      data
+    end
+
+    def text_name(node)
+      node.highlighted? ? node.name.bright : node.name
+    end
+
+    def draw(nodes)
+      path, filename = nil, filename() # outside block to capture value
+      digraph do
+        node_attribs << color('"#b6d7a8"') << filled << fontcolor("white")
+        edge_attribs << color('"#999999"') << filled
+        nodes.each do |parent|
+          if parent.highlighted?
+            node(parent.name)
+          else
+            node(parent.name).attributes << color('"#A4C2F4"')
+          end
+          parent.children.each do |child|
+            edge(parent.name, child.name)
+          end
+        end
+        FileUtils.mkdir_p(File.dirname(filename))
+        save(filename, "png")
+        path = "#{filename}.png"
+      end
+
+      logger.info "Graph saved to #{Terraspace::Util.pretty_path(path)}"
+      open(path)
+    end
+
+  private
+    def draw_full_graph
+      if @options[:format] == "text"
+        @options[:full].nil? ? false : @options[:full]
+      else
+        @options[:full].nil? ? true : @options[:full]
+      end
+    end
+
+
+    def filename
+      name = "#{Terraspace.cache_root}/graph/dependencies" # dont include extension
+      unless ENV['TS_GRAPH_TS'] == '0'
+        @@timestamp ||= Time.now.utc.strftime("%Y%m%d%H%M%S")
+        name += "-#{@@timestamp}"
+      end
+      name
+    end
+
+    def open(path)
+      command = command("c9") || command("open")
+      system("#{command} #{path}") if command
+    end
+
+    def command(name)
+      name if system("type #{name} > /dev/null 2>&1") # c9 = cloud9, open = macosx
+    end
+
+    # Check if Graphiz is installed and prints a user friendly message if it is not installed.
+    def check_graphviz!
+      installed = system("type dot > /dev/null 2>&1") # dot is a command that is part of the graphviz package
+      return if installed
+      logger.error "ERROR: It appears that the Graphviz is not installed.  Please install it to use the graph command.".color(:red)
+      if RUBY_PLATFORM =~ /darwin/
+        logger.error "You can install Graphviz with homebrew:"
+        logger.error "    brew install graphviz"
+      end
+      logger.info <<~EOL
+        Also consider:
+
+            terraspace all graph --format text
+
+        Which will print out the graph in text form.
+      EOL
+      exit 1
+    end
+  end
+end

data/lib/terraspace/all/preview.rb

@@ -0,0 +1,43 @@
+module Terraspace::All
+  class Preview
+    extend Memoist
+    include Terraspace::Util::Logging
+    include Terraspace::Compiler::DirsConcern
+
+    def initialize(command, batches, options={})
+      @command, @batches, @options = command, batches, options
+    end
+
+    def show
+      logger.info header
+      logger.info preview
+    end
+
+    def header
+      %w[up down].include?(@command) && !@options[:yes] ? "Will run:" : "Running:"
+    end
+
+    def preview
+      i = 0
+      @batches.map do |batch|
+        i += 1
+        batch.map do |stack|
+          command = "   terraspace #{@command}"
+          ljust = command.size + max_name_size + 1
+          command = "#{command} #{stack.name}"
+          command.ljust(ljust, ' ') + " # batch #{i}"
+        end
+      end.join("\n")
+    end
+
+    def max_name_size
+      @batches.inject(0) do |max,batch|
+        batch.each do |node|
+          max = node.name.size if node.name.size > max
+        end
+        max
+      end
+    end
+    memoize :max_name_size
+  end
+end

data/lib/terraspace/all/runner.rb

@@ -0,0 +1,169 @@
+module Terraspace::All
+  class Runner < Base
+    include Terraspace::Util
+
+    def initialize(command, options={})
+      @command, @options = command, options
+      super(options)
+    end
+
+    def run
+      time_took do
+        @batches = build_batches
+        preview
+        are_you_sure?
+        deploy_batches
+      end
+    end
+
+    def preview
+      Preview.new(@command, @batches, @options).show
+    end
+
+    def build_batches
+      @batches = run_builder(quiet: false)
+      @batches.reverse! if @command == "down"
+      @batches
+    end
+
+    def deploy_batches
+      truncate_logs if ENV['TS_TRUNCATE_LOGS']
+      @batches.each_with_index do |batch,i|
+        logger.info "Batch Run #{i+1}:"
+        run_builder unless i == 0 # already handled by build_batches the first time
+        deploy_batch(batch)
+      end
+    end
+
+    # Should run after each batch run. run_builder also calls replace_outputs.
+    # Important: rebuild from source so placeholders are in place.
+    def run_builder(quiet: true)
+      Terraspace::Builder.new(@options.merge(mod: "placeholder", quiet: quiet)).run
+    end
+
+    def deploy_batch(batch)
+      @pids = {} # stores child processes pids. map of pid to mod_name, reset this list on each batch run
+      concurrency = Terraspace.config.all.concurrency
+      batch.sort_by(&:name).each_slice(concurrency) do |slice|
+        slice.each do |node|
+          pid = fork do
+            run_terraspace(node.name)
+          end
+          @pids[pid] = node.name # store mod_name mapping
+        end
+      end
+      wait_for_child_proccess
+      summarize     # also reports lower-level error info
+      report_errors # reports finall errors and possibly exit
+    end
+
+    def wait_for_child_proccess
+      @errors = [] # stores child processes pids that errored
+      @pids.each do |pid, _|
+        pid, status = Process.wait2(pid)
+        success = status.exitstatus == 0
+        @errors << pid unless success
+      end
+    end
+
+    def report_errors
+      @errors.each do |pid|
+        mod_name = @pids[pid]
+        terraspace_command = terraspace_command(mod_name)
+        logger.error "Error running: #{terraspace_command}. Check logs and fix the error.".color(:red)
+      end
+      unless @errors.empty?
+        exit 2 if exit_on_fail?
+      end
+    end
+
+    # Precendence:
+    # 1. env var
+    # 2. cli
+    # 3. config/app.rb setting
+    def exit_on_fail?
+      return false if ENV['TS_EXIT_ON_FAIL'] == '0'
+      if @options[:exit_on_fail].nil?
+        Terraspace.config.all.exit_on_fail[@command]
+      else
+        @options[:exit_on_fail]
+      end
+    end
+
+    def summarize
+      @pids.each do |_, mod_name|
+        data = {
+          command: @command,
+          log_path: log_path(mod_name),
+          terraspace_command: terraspace_command(mod_name),
+        }
+        Summary.new(data).run
+      end
+    end
+
+    def run_terraspace(mod_name)
+      set_log_path!(mod_name)
+      name = command_map(@command)
+      o = @options.merge(mod: mod_name, yes: true, build: false, input: false)
+      case @command
+      when "up"
+        Terraspace::CLI::Up.new(o).run
+      when "down"
+        Terraspace::CLI::Down.new(o).run
+      else
+        Terraspace::CLI::Commander.new(name, o).run
+      end
+    end
+
+    def set_log_path!(mod_name)
+      command = terraspace_command(mod_name)
+      path = log_path(mod_name)
+      pretty_path = Terraspace::Util.pretty_path(path)
+      logger.info "Running: #{command.bright} Logs: #{pretty_path}"
+
+      FileUtils.mkdir_p(File.dirname(path))
+      logger = Terraspace::Logger.new(path)
+      logger.level = Terraspace.config.logger.level # match the level that user configured
+      logger.formatter = Terraspace.config.logger.formatter # match the level that user configured
+      logger.progname = command
+      Terraspace.logger = logger
+    end
+
+    def log_path(mod_name)
+      "#{Terraspace.config.log.root}/#{@command}/#{mod_name}.log"
+    end
+
+    def truncate_logs
+      logs = Terraspace::CLI::Log::Tasks.new(mute: true)
+      logs.truncate
+    end
+
+    def terraspace_command(name)
+      "terraspace #{@command} #{name}"
+    end
+
+    def command_map(name)
+      map = {
+        up:   "apply",
+        down: "destroy",
+      }.stringify_keys
+      map[name] || name
+    end
+
+    def are_you_sure?
+      return true unless sure_command?
+      sure? # from Util
+    end
+
+    def sure_command?
+      %w[up down].include?(@command)
+    end
+
+    def time_took
+      t1 = Time.now
+      yield
+      t2 = Time.now
+      logger.info "Time took: #{pretty_time(t2-t1)}"
+    end
+  end
+end