terraspace 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6b99a86e9b9b829fa0ac1b04a6774b02a43e67642219ea3f22aeb663869bfec6
-  data.tar.gz: f2b265d5e13a86949ddb753e991fc8784c77361909162123dfd7d06d0c1c3555
+  metadata.gz: bbf29f1226e7dc336c0eaf99892e158aff3b804bd56bbd1dc744fd17a65c6ff0
+  data.tar.gz: 6c32f1fe94e9dc53f6ddbfca3cef7c1178a11d788b32ad6ad3fdc88d64e268fa
 SHA512:
-  metadata.gz: 7fd4d65aac675d439e5e23089db75f937e3f50c72158099d1088112db67ed97c93afef6c47e0555cdde5a6dfe3e175ede017fc9e509c5419128c53ee114de91d
-  data.tar.gz: 4ef8b0354c0e8dd14ccd074bc6ecd2cda96f80c846bfd3dc790dce4cf20f36efbe236020eff587555032ec1dd869ec405cde649a348e232e747ba7900ad79393
+  metadata.gz: b5e45213f23aaed8b4348d4a68179b2706577d6cdf7cf2588f979feb6433e742851392331a8d2909be4a745b6ef9089c419d22d4dd6140ec981c536e809c29df
+  data.tar.gz: f9e436d5ee96f410c66d673819a789ffdca814f8d53c8f7c97b2b005c9d2bd5cd48fc9a6619407ec7da42e0d3ca503666a5e3d2f2124f30b211a22ba87ec7075

data/CHANGELOG.md

@@ -3,7 +3,37 @@
 All notable changes to this project will be documented in this file.
 This project *loosely tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.5.0] - 2020-11-15
+- [#55](https://github.com/boltops-tools/terraspace/pull/55) custom helpers support
+- plugin helpers support: aws_secret, aws_ssm, google_secret, etc
+- introduce stack-level test concept and change project-level test concept
+- generators: standardize and unifiy new test
+- new generators: arg, hook, helper
+- clean up Terraspace check project
+- setup up autoloader and bundler/setup earlier. removes need for shim
+- test generator plugin autodetection
+- remove internal run_generator_hook_script
+- stack-level args customization support
+- improve test output noise-level
+- new setting: config.terraform.plugin_cache.purge_on_error
+- quiet option for project generator
+
+## [0.4.4]
+- #50 retry logic for shared cache error
+- #51 fix cloud sync: call build first
+
+## [0.4.3]
+* #49 add info --path option
+
+## [0.4.2]
+* #48 add logs pid option
+* default to input false, encourage set tfvars
+
+## [0.4.1]
+* #47 `terraspace list` fix and `terraspace all init` help updates
+
 ## [0.4.0]
+* #46 improve hooks, bundle, auto init and bug fixes
 * improve hooks: allow multiple hooks of same type, change path hooks/terraform.rb
 * improve hooks: introduce terraspace-level as well as terraform-level hooks
 * improve hooks: can take Ruby block or shell script

data/README.md

@@ -29,59 +29,77 @@ Here are commands to get started:
 * The `up` command creates an s3 bucket.
 * The `down` command cleans up and deletes the bucket.
 
-The default plugin is aws. Other plugins also supported are: google and azurerm.
+The default plugin is aws. Major cloud providers are supported: [aws](https://terraspace.cloud/docs/learn/aws/), [azurerm](https://terraspace.cloud/docs/learn/azure/), [google](https://terraspace.cloud/docs/learn/gcp/).
 
 ## Usage
 
 Create infrastructure:
 
-    $ terraspace up demo -y
+    $ terraspace up demo
     Building .terraspace-cache/us-west-2/dev/stacks/demo
     Current directory: .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform init -get > /tmp/terraspace/out/terraform-init20200824-21379-bkfvnh.out
-    Built in .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform apply -auto-approve
-    random_pet.bucket: Creating...
-    random_pet.bucket: Creation complete after 0s [id=amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Creating...
-    module.bucket.aws_s3_bucket.this: Creation complete after 1s [id=bucket-amusing-mouse]
-
+    => terraform init -get >> /tmp/terraspace/log/init/demo.log
+    => terraform apply
     Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
-
-    Outputs:
-
-    bucket_name = bucket-amusing-mouse
     $
 
 Destroy infrastructure:
 
-    $ terraspace down demo -y
+    $ terraspace down demo
     Building .terraspace-cache/us-west-2/dev/stacks/demo
     Current directory: .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform init -get > /tmp/terraspace/out/terraform-init20200824-21379-bkfvnh.out
-    Built in .terraspace-cache/us-west-2/dev/stacks/demo
-    => terraform destroy -auto-approve
-    random_pet.bucket: Refreshing state... [id=amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Refreshing state... [id=bucket-amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Destroying... [id=bucket-amusing-mouse]
-    module.bucket.aws_s3_bucket.this: Destruction complete after 1s
-    random_pet.bucket: Destroying... [id=amusing-mouse]
-    random_pet.bucket: Destruction complete after 0s
-
+    => terraform destroy
     Destroy complete! Resources: 2 destroyed.
     $
 
+* Blog: [Introducing Terraspace: The Terraform Framework](https://blog.boltops.com/2020/08/22/introducing-terraspace-the-terraform-framework)
+* Docs: [Terraspace Intro](https://terraspace.cloud/docs/intro/)
+
 ## Deploy Multiple Stacks
 
 To deploy all the infrastructure stacks:
 
-    terraspace all up
+    $ terraspace all up
+    Will run:
+        terraspace up vpc      # batch 1
+        terraspace up mysql    # batch 2
+        terraspace up redis    # batch 2
+        terraspace up instance # batch 3
+    Are you sure? (y/N)
 
 To choose multiple stacks to deploy
 
-    terraspace all up instance vpc
+    $ terraspace all up mysql redis
+    Will run:
+        terraspace up vpc   # batch 1
+        terraspace up mysql # batch 2
+        terraspace up redis # batch 2
+    Are you sure? (y/N)
+
+When you use the all command, the dependency graph is calculated and the stacks are deployed in the right order.
+
+* Blog: [Terraspace All: Deploy Multiple Stacks or Terraform Modules At Once](https://blog.boltops.com/2020/09/19/terraspace-all-deploy-multiple-stacks-at-once)
+* Docs: [Deploy Multiple Stacks](https://terraspace.cloud/docs/intro/deploy-all/).
+
+## Terrafile
 
-When you use the all command, the dependency graph is calculated and the stacks are deployed in the right order. To learn more: [Deploy Multiple Stacks](https://terraspace.cloud/docs/intro/deploy-all/).
+Terraspace makes it easy to use Terraform modules sourced from your own git repositories, other git repositories, or the Terraform Registry. Use any module you want:
+
+Terrafile:
+
+```ruby
+# GitHub repo
+mod "s3", source: "boltops-tools/terraform-aws-s3", tag: "v0.1.0"
+# Terraform registry
+mod "sg", source: "terraform-aws-modules/security-group/aws", version: "3.10.0"
+```
+
+To install modules:
+
+    terraspace bundle
+
+* Blog: [Terraspace Terrafile: Using Git and Terraform Registry Modules](https://blog.boltops.com/2020/10/18/terraspace-terrafile-using-git-repos-and-terraform-registry-modules)
+* Docs: [Terrafile](https://terraspace.cloud/docs/terrafile/)
 
 ## Features
 
@@ -89,8 +107,17 @@ When you use the all command, the dependency graph is calculated and the stacks
 * [Generators](https://terraspace.cloud/docs/generators/): Built-in generators to quickly create the starter module. Focus on code instead of boilerplate structure.
 * [Tfvars](https://terraspace.cloud/docs/tfvars/) & [Layering](https://terraspace.cloud/docs/tfvars/layering/): Use the same code with different tfvars to create multiple environments. Terraspace conventionally loads tfvars from the `tfvars` folder. Rich layering support allows you to build different environments like dev and prod with the same code.  Examples are in [Full Layering](https://terraspace.cloud/docs/tfvars/full-layering/).
 * [Deploy Multiple Stacks](https://terraspace.cloud/docs/intro/deploy-all/): The ability to deploy multiple stacks with a single command. Terraspace calculates the [dependency graph](https://terraspace.cloud/docs/dependencies/) and deploys stacks in the right order. You can also target specific stacks and deploy [subgraphs](https://terraspace.cloud/docs/dependencies/subgraphs/).
-* [Configurable CLI](https://terraspace.cloud/docs/cli/): Configurable [CLI Hooks](https://terraspace.cloud/docs/cli/hooks/) and [CLI Args](https://terraspace.cloud/docs/cli/args/) allow you to adjust the underlying terraform command.
+* [Terrafile](https://terraspace.cloud/docs/terrafile/): Terraspace makes it easy to use Terraform modules sourced from your own git repositories, other git repositories, or the Terraform Registry. The git repos can be private or public. This is an incredibly powerful feature of Terraspace because it opens up a world of modules for you to use.  Use any module you want.
+* [Configurable CLI](https://terraspace.cloud/docs/config/args/): Configurable [CLI Hooks](https://terraspace.cloud/docs/config/hooks/) and [CLI Args](https://terraspace.cloud/docs/config/args/) allow you to adjust the underlying terraform command.
 * [Testing](https://terraspace.cloud/docs/testing/): A testing framework that allows you to create test harnesses, deploy real-resources, and have higher confidence that your code works.
 * [Terraform Cloud and Terraform Enterprise Support](https://terraspace.cloud/docs/cloud/): TFC and TFE are both supported. Terraspace adds additional conveniences to make working with Terraform Cloud Workspaces easier.
 
-For more info: [terraspace.cloud](https://terraspace.cloud)
+## Comparison
+
+Here are some useful comparisons to help you compare Terraspace vs other tools in the ecosystem:
+
+* [Terraspace vs Custom Solution](https://terraspace.cloud/docs/vs/custom/)
+* [Terraspace vs Terraform](https://terraspace.cloud/docs/vs/terraform/)
+* [Terraform vs Terragrunt vs Terraspace](https://blog.boltops.com/2020/09/28/terraform-vs-terragrunt-vs-terraspace)
+
+More info: [terraspace.cloud](https://terraspace.cloud)

data/lib/templates/base/arg/terraform.rb.tt

@@ -0,0 +1,3 @@
+command("<%= arg_name %>",
+  args: ["-lock-timeout=22m"],
+)

data/lib/templates/base/helper/%name%_helper.rb.tt

@@ -0,0 +1,2 @@
+module <%= helper_class %>
+end

data/lib/templates/base/hook/%kind%.rb.tt

@@ -0,0 +1,7 @@
+before("<%= hook_name %>",
+  execute: "echo '<%= hook_path %>: test <%= type %> before hook for <%= kind %> <%= hook_name %>'",
+)
+
+after("<%= hook_name %>",
+  execute: "echo '<%= hook_path %>: test <%= type %> after hook for <%= kind %> <%= hook_name %>'"
+)

data/lib/terraspace.rb

@@ -1,6 +1,10 @@
 $stdout.sync = true unless ENV["TS_STDOUT_SYNC"] == "0"
 
 $:.unshift(File.expand_path("../", __FILE__))
+
+require "terraspace/autoloader"
+Terraspace::Autoloader.setup
+
 require "active_support/concern"
 require "active_support/core_ext/class"
 require "active_support/core_ext/hash"
@@ -18,16 +22,15 @@ require "set"
 require "terraspace/ext"
 require "terraspace/version"
 
-require "terraspace/autoloader"
-Terraspace::Autoloader.setup
-
 DslEvaluator.backtrace_reject = "lib/terraspace"
 
 module Terraspace
   extend Core # for Terraspace.root
   class Error < StandardError; end
-  class InitRequiredError < Error; end
+
   class BucketNotFoundError < Error; end
+  class InitRequiredError < Error; end
+  class SharedCacheError < Error; end
 end
 
 Terraspace::Booter.boot

data/lib/terraspace/all/base.rb

@@ -2,7 +2,6 @@ module Terraspace::All
   class Base
     def initialize(options={})
       @options = options
-      Terraspace.check_project! unless ENV['TS_TEST']
     end
   end
 end

data/lib/terraspace/app.rb

@@ -47,6 +47,7 @@ module Terraspace
       config.terraform.plugin_cache = ActiveSupport::OrderedOptions.new
       config.terraform.plugin_cache.dir = ENV['TF_PLUGIN_CACHE_DIR'] || "#{Terraspace.tmp_root}/plugin_cache"
       config.terraform.plugin_cache.enabled = true
+      config.terraform.plugin_cache.purge_on_error = true
       config.test_framework = "rspec"
       config
     end

data/lib/terraspace/autoloader.rb

@@ -1,8 +1,15 @@
-require "terraspace/bundle"
-Terraspace::Bundle.setup
+if File.exist?("config/app.rb")
+  require "terraspace/bundle"
+  Terraspace::Bundle.setup
+end
 require "zeitwerk"
 
 module Terraspace
+  # These modules are namespaces for user-defined custom helpers
+  module Module; end
+  module Project; end
+  module Stack; end
+
   class Autoloader
     class Inflector < Zeitwerk::Inflector
       def camelize(basename, _abspath)
@@ -16,10 +23,20 @@ module Terraspace
         loader = Zeitwerk::Loader.new
         loader.inflector = Inflector.new
         loader.push_dir(File.dirname(__dir__)) # lib
+        loader.push_dir(project_helpers, namespace: Terraspace::Project) if File.exist?(project_helpers)
         loader.log! if ENV["TS_AUTOLOAD_LOG"]
-        loader.ignore("#{__dir__}/core_ext.rb")
+        loader.ignore("#{__dir__}/ext.rb")
         loader.setup
       end
+
+      def project_helpers
+        "#{ts_root}/config/helpers"
+      end
+
+      # Duplicate definition because autoloader logic runs very early and doesnt have access to core methods yet
+      def ts_root
+        ENV['TS_ROOT'] || Dir.pwd
+      end
     end
   end
 end

data/lib/terraspace/cli.rb

@@ -86,6 +86,7 @@ module Terraspace
     long_desc Help.text(:info)
     instance_option.call
     option :format, desc: "Output formats: #{CliFormat.formats.join(', ')}"
+    option :path, desc: "Print path to built path"
     def info(mod)
       Info.new(options.merge(mod: mod)).run
     end
@@ -110,6 +111,7 @@ module Terraspace
     option :follow, aliases: %w[f], type: :boolean, desc: "Follow the log in live tail fashion. Must specify a stack if using this option."
     option :limit, aliases: %w[n], default: 10, type: :numeric, desc: "Number of lines to limit showing. Only applies in no-follow mode."
     option :all, aliases: %w[a], type: :boolean, desc: "All mode turns off the limit. Defaults to all if a single log is specified. Only applies in no-follow mode."
+    option :pid, aliases: %w[p], desc: "Filter by pid. Defaults to the last pid at the bottom of the log file."
     def logs(action=nil, stack=nil)
       Logs.new(@options.merge(action: action, stack: stack)).run
     end

data/lib/terraspace/cli/all.rb

@@ -18,7 +18,7 @@ class Terraspace::CLI
       Terraspace::All::Grapher.new(@options.merge(stacks: stacks)).run
     end
 
-    desc "init", "Init."
+    desc "init", "Init all or multiple stacks."
     long_desc Help.text("all/init")
     def init(*stacks)
       Terraspace::All::Runner.new("init", @options.merge(stacks: stacks)).run

data/lib/terraspace/cli/bundle.rb

@@ -7,7 +7,6 @@ class Terraspace::CLI
     end
 
     def run
-      Terraspace.check_project!
       TerraspaceBundler.config.deep_merge!(Terraspace.config.bundle)
       TerraspaceBundler::CLI.start(args)
     end

data/lib/terraspace/cli/clean/base.rb

@@ -5,7 +5,6 @@ class Terraspace::CLI::Clean
 
     def initialize(options={})
       @options = options
-      Terraspace.check_project!
     end
 
     def pretty(path)

data/lib/terraspace/cli/clean/cache.rb

@@ -1,7 +1,6 @@
 class Terraspace::CLI::Clean
   class Cache < Base
     def run
-      Terraspace.check_project!
       paths = [Terraspace.cache_root, Terraspace.tmp_root]
       are_you_sure?(paths)
       paths.each do |path|

data/lib/terraspace/cli/cloud.rb

@@ -16,6 +16,7 @@ class Terraspace::CLI
 
     desc "destroy STACK", "Destroy workspace by specifying the stack"
     long_desc Help.text("cloud:destroy")
+    yes_option.call
     def destroy(mod)
       Workspace.new(options.merge(mod: mod)).destroy
     end
@@ -24,6 +25,7 @@ class Terraspace::CLI
     long_desc Help.text("cloud:sync")
     yes_option.call
     def sync(*stacks)
+      Terraspace::CLI::Build::Placeholder.new(options).build
       Syncer.new(options.merge(stacks: stacks, override_auto_sync: true)).run
     end
 

data/lib/terraspace/cli/help/all/init.md

@@ -0,0 +1,33 @@
+## Example
+
+    $ terraspace all init
+    Building one stack to build all stacks
+    Building .terraspace-cache/us-west-2/dev/stacks/c1
+    Downloading tfstate files for dependencies defined in tfvars...
+    Built in .terraspace-cache/us-west-2/dev/stacks/c1
+    Running:
+        terraspace init c1 # batch 1
+        terraspace init b1 # batch 2
+        terraspace init b2 # batch 2
+        terraspace init a1 # batch 3
+    Batch Run 1:
+    Running: terraspace init c1 Logs: log/init/c1.log
+    terraspace init c1:  Terraform has been successfully initialized!
+    Batch Run 2:
+    Running: terraspace init b1 Logs: log/init/b1.log
+    Running: terraspace init b2 Logs: log/init/b2.log
+    terraspace init b1:  Terraform has been successfully initialized!
+    terraspace init b2:  Terraform has been successfully initialized!
+    Batch Run 3:
+    Running: terraspace init a1 Logs: log/init/a1.log
+    terraspace init a1:  Terraform has been successfully initialized!
+    Time took: 6s
+    $
+
+If Terraform is having trouble initializing, clearing the cache may help:
+
+    $ terraspace clean cache -y
+    Removed .terraspace-cache
+    Removed /tmp/terraspace
+
+Also consider disabling the [terraform.plugin_cache.enabled](https://terraspace.cloud/docs/config/reference/).

data/lib/terraspace/cli/help/logs.md

@@ -12,12 +12,12 @@ Note, Terraspace automatically checks every second for new logs and adds them to
 
 View last 10 lines of each log file.
 
-    terraspace logs up network # view up log on specific stack
+    terraspace logs up network # view up log on a specific stack
     terraspace logs up         # view all up logs
     terraspace logs down       # view all down logs
     terraspace logs            # view all logs: up, down, etc
 
-By default, the log command shows the last 10 lines of the logs for each log file. You can use the `-n` option to adjust this.
+By default, the logs command shows the last 10 lines for each log file. You can use the `-n` option to adjust this.
 
     terraspace logs -n 2       # view last 2 lines of all logs: up, down, etc
 
@@ -31,14 +31,14 @@ Note, if both an action and stack is specified, then it defaults to showing all
 
 To tail logs, use the `-f` option.
 
-    terraspace logs up network -f # view up log on specific stack
+    terraspace logs up network -f # view up log on a specific stack
     terraspace logs up -f         # view all up logs
     terraspace logs down -f       # view all down logs
     terraspace logs -f            # view all logs: up, down, etc
 
 ## Timestamps
 
-The timestamps are shown by default when you are looking for multiple files.  When you specify both the action and stack for a single log file, then timestamps are not shown.
+The timestamps are shown by default when you are looking for multiple files.  When you specify both the action and stack for a single log file, the timestamps are not shown.
 
     terraspace logs up         # timestamps will be shown in this case
     terraspace logs up network # timestamps not be shown in this case

data/lib/terraspace/cli/help/new/arg.md

@@ -0,0 +1,19 @@
+## Examples
+
+Project-level arg:
+
+    $ terraspace new arg --type project
+          create  config/args
+          create  config/args/terraform.rb
+
+Stack-level arg:
+
+    $ terraspace new arg demo --type stack
+          create  app/stacks/demo/config/args
+          create  app/stacks/demo/config/args/terraform.rb
+
+Module-level arg:
+
+    $ terraspace new arg example --type module
+          create  app/modules/example/config/args
+          create  app/modules/example/config/args/terraform.rb

data/lib/terraspace/cli/help/new/helper.md

@@ -0,0 +1,39 @@
+## Project Examples
+
+Conventionally name the helper custom.
+
+    $ terraspace new helper custom --type project
+          create  config/helpers
+          create  config/helpers/custom_helper.rb
+
+## Stack Examples
+
+Conventionally name the helper to be the same as the stack name.
+
+    $ terraspace new helper demo --type stack
+          create  app/stacks/demo/config/helpers
+          create  app/stacks/demo/config/helpers/demo_helper.rb
+    $
+
+Explictly name helper method. IE: custom here.
+
+    $ terraspace new helper demo --type stack --name custom
+          create  app/stacks/demo/config/helpers
+          create  app/stacks/demo/config/helpers/custom_helper.rb
+    $
+
+## Module Examples
+
+Conventionally name the helper to be the same as the module name.
+
+    $ terraspace new helper example --type module
+          create  app/module/example/config/helpers
+          create  app/module/example/config/helpers/example_helper.rb
+    $
+
+Explictly name helper method. IE: custom here.
+
+    $ terraspace new helper example --type module --name custom
+          create  app/module/example/config/helpers
+          create  app/module/example/config/helpers/custom_helper.rb
+    $