terraspace 0.3.6 → 0.4.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 28bdb53824befd6a98263c63612db913c5496546ef751ed408500b4140ac57c4
-  data.tar.gz: fe844bdeb2dacfbecfa954a912bfa4ef3e5b4e6265d2eea697d072d8533b2fd5
+  metadata.gz: bae908fcf2dc448dc5b0d2bba463eedd9487c428a296b74955975b66e2faa7c3
+  data.tar.gz: bef0979ecaeab8eca50387da95a119719696a759ecff7de55659ff0d8cbac322
 SHA512:
-  metadata.gz: 7e01f4b80088de2c7a3f3f1c8ac81a401b4a999cbb3fbc4af78651ee8e1788246dcb8b889172b94ea5ff7a7932a4874f27e225f66dedac2603a5752a20fb383c
-  data.tar.gz: c0ca89b09e3d8ca3f36f6d2f513fedbaa29aad071af4c4c6e99c38438b8c2002582adaa49b2f7105becde5e2ac21785a7f5617b9c5fd1fd60ea25fbaadda7a54
+  metadata.gz: 882b9a94cf48af9399a66540385dcd7dbcdbb8118df046c6af833e053f265e57dfeb0a3692f32e315b6a13bb59334c51cfc856db0fc68a067cee4800bc1a3a32
+  data.tar.gz: 720aa71881469e415445a1ff83e853c1ece6d71873830111eb1de59c121e9408e478ddaabdd30e6c29e2d48d39b1ec1c96c739e16ca682b3e13e129253594d44

data/CHANGELOG.md

@@ -3,6 +3,38 @@
 All notable changes to this project will be documented in this file.
 This project *loosely tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.4.4]
+- #50 retry logic for shared cache error
+- #51 fix cloud sync: call build first
+
+## [0.4.3]
+* #49 add info --path option
+
+## [0.4.2]
+* #48 add logs pid option
+* default to input false, encourage set tfvars
+
+## [0.4.1]
+* #47 `terraspace list` fix and `terraspace all init` help updates
+
+## [0.4.0]
+* #46 improve hooks, bundle, auto init and bug fixes
+* improve hooks: allow multiple hooks of same type, change path hooks/terraform.rb
+* improve hooks: introduce terraspace-level as well as terraform-level hooks
+* improve hooks: can take Ruby block or shell script
+* improve auto init: reinit when module source changed, improve auto init: generalize retry check
+* fix already_init? detection for case when stack does use module
+* terraspace bundler options: can set any option now
+* add `terraspace all init` command
+* improve terraspace clean, prompt user and add `-y` option
+* bundle: check if in terraspace project
+* Rename NullObject to Unresolved
+* improve error message when stacks not found. give `terraspace list` hint
+* terraspace list. change default to `--type stack`
+* change summary option to --details
+* include terraspace version in generated Gemfile
+* logs command to handle viewing, and clean logs to clean
+
 ## [0.3.6]
 * #44 improve logs management commands: `terraspace logs remove` and `terraspace logs truncate`
 

data/README.md

@@ -29,59 +29,77 @@ Here are commands to get started:
 * The `up` command creates an s3 bucket.
 * The `down` command cleans up and deletes the bucket.
 
-The default plugin is aws. Other plugins also supported are: google and azurerm.
+The default plugin is aws. Major cloud providers are supported: [aws](https://terraspace.cloud/docs/learn/aws/), [azurerm](https://terraspace.cloud/docs/learn/azure/), [google](https://terraspace.cloud/docs/learn/gcp/).
 
 ## Usage
 
 Create infrastructure:
 
-    $ terraspace up demo -y
+    $ terraspace up demo
     Building .terraspace-cache/us-west-2/dev/stacks/demo
     Current directory: .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform init -get > /tmp/terraspace/out/terraform-init20200824-21379-bkfvnh.out
-    Built in .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform apply -auto-approve
-    random_pet.bucket: Creating...
-    random_pet.bucket: Creation complete after 0s [id=amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Creating...
-    module.bucket.aws_s3_bucket.this: Creation complete after 1s [id=bucket-amusing-mouse]
-
+    => terraform init -get >> /tmp/terraspace/log/init/demo.log
+    => terraform apply
     Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
-
-    Outputs:
-
-    bucket_name = bucket-amusing-mouse
     $
 
 Destroy infrastructure:
 
-    $ terraspace down demo -y
+    $ terraspace down demo
     Building .terraspace-cache/us-west-2/dev/stacks/demo
     Current directory: .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform init -get > /tmp/terraspace/out/terraform-init20200824-21379-bkfvnh.out
-    Built in .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform destroy -auto-approve
-    random_pet.bucket: Refreshing state... [id=amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Refreshing state... [id=bucket-amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Destroying... [id=bucket-amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Destruction complete after 1s
-    random_pet.bucket: Destroying... [id=amusing-mouse]
-    random_pet.bucket: Destruction complete after 0s
-
+    => terraform destroy
     Destroy complete! Resources: 2 destroyed.
     $
 
+* Blog: [Introducing Terraspace: The Terraform Framework](https://blog.boltops.com/2020/08/22/introducing-terraspace-the-terraform-framework)
+* Docs: [Terraspace Intro](https://terraspace.cloud/docs/intro/)
+
 ## Deploy Multiple Stacks
 
 To deploy all the infrastructure stacks:
 
-    terraspace all up
+    $ terraspace all up
+    Will run:
+        terraspace up vpc      # batch 1
+        terraspace up mysql    # batch 2
+        terraspace up redis    # batch 2
+        terraspace up instance # batch 3
+    Are you sure? (y/N)
 
 To choose multiple stacks to deploy
 
-    terraspace all up instance vpc
+    $ terraspace all up mysql redis
+    Will run:
+        terraspace up vpc   # batch 1
+        terraspace up mysql # batch 2
+        terraspace up redis # batch 2
+    Are you sure? (y/N)
+
+When you use the all command, the dependency graph is calculated and the stacks are deployed in the right order.
+
+* Blog: [Terraspace All: Deploy Multiple Stacks or Terraform Modules At Once](https://blog.boltops.com/2020/09/19/terraspace-all-deploy-multiple-stacks-at-once)
+* Docs: [Deploy Multiple Stacks](https://terraspace.cloud/docs/intro/deploy-all/).
+
+## Terrafile
 
-When you use the all command, the dependency graph is calculated and the stacks are deployed in the right order. To learn more: [Deploy Multiple Stacks](https://terraspace.cloud/docs/intro/deploy-all/).
+Terraspace makes it easy to use Terraform modules sourced from your own git repositories, other git repositories, or the Terraform Registry. Use any module you want:
+
+Terrafile:
+
+```ruby
+# GitHub repo
+mod "s3", source: "boltops-tools/terraform-aws-s3", tag: "v0.1.0"
+# Terraform registry
+mod "sg", source: "terraform-aws-modules/security-group/aws", version: "3.10.0"
+```
+
+To install modules:
+
+    terraspace bundle
+
+* Blog: [Terraspace Terrafile: Using Git and Terraform Registry Modules](https://blog.boltops.com/2020/10/18/terraspace-terrafile-using-git-repos-and-terraform-registry-modules)
+* Docs: [Terrafile](https://terraspace.cloud/docs/terrafile/)
 
 ## Features
 
@@ -89,8 +107,17 @@ When you use the all command, the dependency graph is calculated and the stacks
 * [Generators](https://terraspace.cloud/docs/generators/): Built-in generators to quickly create the starter module. Focus on code instead of boilerplate structure.
 * [Tfvars](https://terraspace.cloud/docs/tfvars/) & [Layering](https://terraspace.cloud/docs/tfvars/layering/): Use the same code with different tfvars to create multiple environments. Terraspace conventionally loads tfvars from the `tfvars` folder. Rich layering support allows you to build different environments like dev and prod with the same code.  Examples are in [Full Layering](https://terraspace.cloud/docs/tfvars/full-layering/).
 * [Deploy Multiple Stacks](https://terraspace.cloud/docs/intro/deploy-all/): The ability to deploy multiple stacks with a single command. Terraspace calculates the [dependency graph](https://terraspace.cloud/docs/dependencies/) and deploys stacks in the right order. You can also target specific stacks and deploy [subgraphs](https://terraspace.cloud/docs/dependencies/subgraphs/).
-* [Configurable CLI](https://terraspace.cloud/docs/cli/): Configurable [CLI Hooks](https://terraspace.cloud/docs/cli/hooks/) and [CLI Args](https://terraspace.cloud/docs/cli/args/) allow you to adjust the underlying terraform command.
+* [Terrafile](https://terraspace.cloud/docs/terrafile/): Terraspace makes it easy to use Terraform modules sourced from your own git repositories, other git repositories, or the Terraform Registry. The git repos can be private or public. This is an incredibly powerful feature of Terraspace because it opens up a world of modules for you to use.  Use any module you want.
+* [Configurable CLI](https://terraspace.cloud/docs/config/args/): Configurable [CLI Hooks](https://terraspace.cloud/docs/config/hooks/) and [CLI Args](https://terraspace.cloud/docs/config/args/) allow you to adjust the underlying terraform command.
 * [Testing](https://terraspace.cloud/docs/testing/): A testing framework that allows you to create test harnesses, deploy real-resources, and have higher confidence that your code works.
 * [Terraform Cloud and Terraform Enterprise Support](https://terraspace.cloud/docs/cloud/): TFC and TFE are both supported. Terraspace adds additional conveniences to make working with Terraform Cloud Workspaces easier.
 
-For more info: [terraspace.cloud](https://terraspace.cloud)
+## Comparison
+
+Here are some useful comparisons to help you compare Terraspace vs other tools in the ecosystem:
+
+* [Terraspace vs Custom Solution](https://terraspace.cloud/docs/vs/custom/)
+* [Terraspace vs Terraform](https://terraspace.cloud/docs/vs/terraform/)
+* [Terraform vs Terragrunt vs Terraspace](https://blog.boltops.com/2020/09/28/terraform-vs-terragrunt-vs-terraspace)
+
+More info: [terraspace.cloud](https://terraspace.cloud)

data/lib/templates/base/project/README.md

@@ -14,4 +14,4 @@ To deploy individual stacks:
 
 ## Terrafile
 
-To add more modules for use, you can add them to the [Terrafile](https://terraspace.cloud/docs/terrafile/).
+To use more modules, add them to the [Terrafile](https://terraspace.cloud/docs/terrafile/).

data/lib/terraspace.rb

@@ -26,8 +26,10 @@ DslEvaluator.backtrace_reject = "lib/terraspace"
 module Terraspace
   extend Core # for Terraspace.root
   class Error < StandardError; end
-  class InitRequiredError < Error; end
+
   class BucketNotFoundError < Error; end
+  class InitRequiredError < Error; end
+  class SharedCacheError < Error; end
 end
 
 Terraspace::Booter.boot

data/lib/terraspace/all/runner.rb

@@ -105,6 +105,7 @@ module Terraspace::All
       set_log_path!(mod_name)
       name = command_map(@command)
       o = @options.merge(mod: mod_name, yes: true, build: false, input: false)
+      o.merge!(quiet: false) if @command == "init" # noisy so can filter and summarize output
       case @command
       when "up"
         Terraspace::CLI::Up.new(o).run

data/lib/terraspace/all/summary.rb

@@ -1,6 +1,6 @@
 module Terraspace::All
   class Summary
-    include Terraspace::CLI::Log::Concern
+    include Terraspace::CLI::Logs::Concern
     include Terraspace::Util::Logging
 
     def initialize(data={})
@@ -35,6 +35,13 @@ module Terraspace::All
       end
     end
 
+    def init
+      @lines.select! do |line|
+        line.include?("successfully initialized") || # success
+        line.include?("Error: ")  # error
+      end
+    end
+
     # Example 1:
     #     [2020-09-07T14:45:14 #23340 terraspace plan b1]: No changes. Infrastructure is up-to-date.
     # Example 2:

data/lib/terraspace/app.rb

@@ -9,6 +9,8 @@ module Terraspace
     end
 
     def defaults
+      ts_logger = Logger.new(ENV['TS_LOG_PATH'] || $stderr)
+
       config = ActiveSupport::OrderedOptions.new
       config.all = ActiveSupport::OrderedOptions.new
       config.all.concurrency = 5
@@ -21,14 +23,16 @@ module Terraspace
       config.build.cache_dir = ":CACHE_ROOT/:REGION/:ENV/:BUILD_DIR"
       config.build.cache_root = nil # defaults to /full/path/to/.terraspace-cache
       config.build.clean_cache = nil # defaults to /full/path/to/.terraspace-cache
+      config.bundle = ActiveSupport::OrderedOptions.new
+      config.bundle.logger = ts_logger
       config.cloud = ActiveSupport::OrderedOptions.new
       config.cloud.auto_sync = true
-      config.cloud.working_dir_prefix = nil
       config.cloud.hostname = nil
       config.cloud.vars = ActiveSupport::OrderedOptions.new
       config.cloud.vars.overwrite = true
       config.cloud.vars.overwrite_sensitive = true
       config.cloud.vars.show_message = "create"
+      config.cloud.working_dir_prefix = nil
       config.cloud.workspace = ActiveSupport::OrderedOptions.new
       config.cloud.workspace.attrs = ActiveSupport::OrderedOptions.new
       config.hooks = Hooks.new
@@ -36,14 +40,14 @@ module Terraspace
       config.init.mode = "auto" # auto, never, always
       config.log = ActiveSupport::OrderedOptions.new
       config.log.root = Terraspace.log_root
-      config.logger = Logger.new(ENV['TS_LOG_PATH'] || $stderr)
-      config.logger.level = ENV['TS_LOG_LEVEL'] || :info
+      config.logger = ts_logger
       config.logger.formatter = Logger::Formatter.new
-      config.test_framework = "rspec"
+      config.logger.level = ENV['TS_LOG_LEVEL'] || :info
       config.terraform = ActiveSupport::OrderedOptions.new
       config.terraform.plugin_cache = ActiveSupport::OrderedOptions.new
-      config.terraform.plugin_cache.enabled = true
       config.terraform.plugin_cache.dir = ENV['TF_PLUGIN_CACHE_DIR'] || "#{Terraspace.tmp_root}/plugin_cache"
+      config.terraform.plugin_cache.enabled = true
+      config.test_framework = "rspec"
       config
     end
 

data/lib/terraspace/builder.rb

@@ -1,7 +1,8 @@
 module Terraspace
   class Builder < Terraspace::CLI::Base
-    include Compiler::DirsConcern
     include Compiler::CommandsConcern
+    include Compiler::DirsConcern
+    include Hooks::Concern
 
     attr_reader :graph
 
@@ -14,11 +15,14 @@ module Terraspace
       placeholder_stack_message
       logger.info "Building #{build_dir}" unless @options[:quiet] # from terraspace all
 
-      build_unresolved
-      auto_create_backend
-      batches = build_batches
-      build_all
-      logger.info "Built in #{build_dir}" unless @options[:quiet] # from terraspace all
+      batches = nil
+      run_hooks("terraspace.rb", "build") do
+        build_unresolved
+        auto_create_backend
+        batches = build_batches
+        build_all
+        logger.info "Built in #{build_dir}" unless @options[:quiet] # from terraspace all
+      end
       batches
     end
 

data/lib/terraspace/cli.rb

@@ -32,14 +32,14 @@ module Terraspace
     long_desc Help.text(:all)
     subcommand "all", All
 
+    desc "clean SUBCOMMAND", "clean subcommands"
+    long_desc Help.text(:clean)
+    subcommand "clean", Clean
+
     desc "cloud SUBCOMMAND", "cloud subcommands"
     long_desc Help.text(:cloud)
     subcommand "cloud", Cloud
 
-    desc "logs SUBCOMMAND", "logs management subcommands"
-    long_desc Help.text(:logs)
-    subcommand "logs", Logs
-
     desc "new SUBCOMMAND", "new subcommands"
     long_desc Help.text(:new)
     subcommand "new", New
@@ -65,12 +65,6 @@ module Terraspace
       CheckSetup.new(options).run
     end
 
-    desc "clean", "Removes .terraspace-cache dir."
-    long_desc Help.text(:clean)
-    def clean
-      Clean.new(options).run
-    end
-
     desc "console STACK", "Run console in built terraform project."
     long_desc Help.text(:console)
     instance_option.call
@@ -92,6 +86,7 @@ module Terraspace
     long_desc Help.text(:info)
     instance_option.call
     option :format, desc: "Output formats: #{CliFormat.formats.join(', ')}"
+    option :path, desc: "Print path to built path"
     def info(mod)
       Info.new(options.merge(mod: mod)).run
     end
@@ -105,19 +100,20 @@ module Terraspace
 
     desc "list", "List stacks and modules."
     long_desc Help.text(:list)
-    option :type, aliases: %w[t], desc: "Type: stack or module. Default all"
+    option :type, default: "stack", aliases: %w[t], desc: "Type: stack, module, or all"
     def list
       List.new(options).run
     end
 
-    desc "log [ACTION] [STACK]", "View and tail logs."
-    long_desc Help.text("log")
+    desc "logs [ACTION] [STACK]", "View and tail logs."
+    long_desc Help.text("logs")
     option :timestamps, aliases: %w[t], type: :boolean, desc: "Whether or not to show the leading timestamp. Defaults to timestamps for multiple logs, and no timestamp if a single log is specified. Note: In follow mode, timestamp always shown"
     option :follow, aliases: %w[f], type: :boolean, desc: "Follow the log in live tail fashion. Must specify a stack if using this option."
     option :limit, aliases: %w[n], default: 10, type: :numeric, desc: "Number of lines to limit showing. Only applies in no-follow mode."
     option :all, aliases: %w[a], type: :boolean, desc: "All mode turns off the limit. Defaults to all if a single log is specified. Only applies in no-follow mode."
-    def log(action=nil, stack=nil)
-      Log.new(@options.merge(action: action, stack: stack)).run
+    option :pid, aliases: %w[p], desc: "Filter by pid. Defaults to the last pid at the bottom of the log file."
+    def logs(action=nil, stack=nil)
+      Logs.new(@options.merge(action: action, stack: stack)).run
     end
 
     desc "plan STACK", "Plan stack."
@@ -159,7 +155,7 @@ module Terraspace
     long_desc Help.text(:summary)
     option :mod, desc: "Module to build to generate a backend file for discovery. By default the last module is used. Usually, it wont matter."
     init_option.call
-    option :short, aliases: %w[s], type: :boolean, desc: "Only show statefiles"
+    option :details, type: :boolean, desc: "Show details of the listed resources"
     def summary
       Summary.new(options).run
     end

data/lib/terraspace/cli/all.rb

@@ -18,6 +18,12 @@ class Terraspace::CLI
       Terraspace::All::Grapher.new(@options.merge(stacks: stacks)).run
     end
 
+    desc "init", "Init all or multiple stacks."
+    long_desc Help.text("all/init")
+    def init(*stacks)
+      Terraspace::All::Runner.new("init", @options.merge(stacks: stacks)).run
+    end
+
     desc "refresh", "Refresh all or multiple stacks."
     long_desc Help.text("all/refresh")
     def refresh(*stacks)

data/lib/terraspace/cli/bundle.rb

@@ -1,5 +1,4 @@
 require "terraspace-bundler"
-TerraspaceBundler.config.logger = Terraspace.logger
 
 class Terraspace::CLI
   class Bundle
@@ -8,6 +7,8 @@ class Terraspace::CLI
     end
 
     def run
+      Terraspace.check_project!
+      TerraspaceBundler.config.deep_merge!(Terraspace.config.bundle)
       TerraspaceBundler::CLI.start(args)
     end
 

data/lib/terraspace/cli/clean.rb

@@ -1,12 +1,24 @@
 class Terraspace::CLI
-  class Clean
-    def initialize(options={})
-      @options = options
+  class Clean < Terraspace::Command
+    class_option :yes, aliases: :y, type: :boolean, desc: "bypass are you sure prompt"
+
+    desc "all", "Runs all clean operations."
+    long_desc Help.text("clean/all")
+    def all
+      All.new(options).run
+    end
+
+    desc "cache", "Removes cache dirs."
+    long_desc Help.text("clean/cache")
+    def cache
+      Cache.new(options).run
     end
 
-    def run
-      FileUtils.rm_rf(Terraspace.cache_root)
-      puts "Removed #{Terraspace::Util.pretty_path(Terraspace.cache_root)}"
+    desc "logs", "Removes or truncate logs."
+    long_desc Help.text("clean/logs")
+    option :truncate, aliases: :t, type: :boolean, desc: "Truncate instead of remove logs"
+    def logs
+      Logs.new(options).run
     end
   end
 end

data/lib/terraspace/cli/clean/all.rb

@@ -0,0 +1,18 @@
+class Terraspace::CLI::Clean
+  class All < Base
+    def run
+      are_you_sure?
+      o = @options.merge(yes: true) # override to avoid double prompt
+      Cache.new(o).run
+      Logs.new(o).run
+    end
+
+    def are_you_sure?
+      message = <<~EOL.chomp
+        Will remove Terraspace cache and logs.
+        Are you sure?
+      EOL
+      sure?(message) # from Util::Sure
+    end
+  end
+end