terraspace-bundler 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 500271ff2f8e5799a55febd3ec25261bcf4126bff456b83360ae0e2a96bb6357
-  data.tar.gz: ee45f81805b893d5287fdea6777c9b28ac39936273806c3d97275f6c314a8cd1
+  metadata.gz: b27c1a0f9a31f2e9d7e32c8c098c7be4b677f5109a0283063928a724c42cd54c
+  data.tar.gz: cab661fe23294051ae93b5686bf1a7ee36b453795a422a3c47a9c4154af732eb
 SHA512:
-  metadata.gz: 94af4d341ed3e43445bf2eb0eae1e3383fe064a6df708c449658b8eb5a958163a225be2ddae121356573ae7e099e87669c5cb9c54808423b9db685faf27d7837
-  data.tar.gz: d68d1825f89b91ac4d9ad7525a4b4b95cc709ce1e2f3c10d83aa6ad5c6b27a89b3c6ae1ad9bc1ee3179b08345ac73fc551d1467769f702cc43d69b6f3b38b577
+  metadata.gz: 52f69be90c325355378fb47ce8f75873a13ade4b228494177bdc770d996b4b86f35b156c96a77fa2195c7f1f5d0432f38bde1b9330398ce611d0c471b2646610
+  data.tar.gz: eaad3baf724bf7a0da132257e3e3d59dd9b6305b194c61bdbde0a9d90cf741b47cf163b1524ec05ee83e1f8dad81cd5ca9356b78635ba096eaec1ac5570737f5

data/CHANGELOG.md

@@ -3,6 +3,12 @@
 All notable changes to this project will be documented in this file.
 This project *loosely tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.3.0]
+- change `base_clone_url` default from `git@github.com:` to `https://github.com/`
+- mod-level stack option
+- terrafile-level stack_options
+- rename terrafile-level option to export_to
+
 ## [0.2.0]
 - #1 Rework implementation. Better Terrafile syncing
 - clearer implementation: Terrafile and Lockfile classes build standard mods object.

data/README.md

@@ -1,5 +1,9 @@
 # terraspace-bundler
 
+[![Gem Version](https://badge.fury.io/rb/terraspace-bundler.png)](http://badge.fury.io/rb/terraspace-bundler)
+
+[![BoltOps Badge](https://img.boltops.com/boltops/badges/boltops-badge.png)](https://www.boltops.com)
+
 Bundles terraform modules based on a `Terrafile` to the `vendor/modules` folder. Used by the [Terraspace Terraform Framework](https://terraspace.cloud/).
 
 ## Usage
@@ -7,14 +11,13 @@ Bundles terraform modules based on a `Terrafile` to the `vendor/modules` folder.
 Create a `Terrafile` file:
 
 ```ruby
-org "boltopspro" # set default org
+org "boltops-tools" # set default org
 
 # GitHub repo with default org
-mod "elb", source: "terraform-aws-elb", version: "master"
-mod "ec2", source: "terraform-aws-ec2", version: "branch-name"
+mod "s3", source: "terraform-aws-s3", version: "master"
 
 # GitHub repo with explicit org
-mod "rds", source: "boltopspro/terraform-aws-rds", version: "v0.1.0"
+# mod "elasticache", source: "boltopspro/terraform-aws-elasticache"
 
 # Terraform registry
 mod "sg", source: "terraform-aws-modules/security-group/aws", version: "3.10.0"
@@ -26,90 +29,7 @@ Running `terraspace bundle` creates the `Terrafile.lock` file, which locks the v
 
     terraspace bundle
 
-## Updating
-
-To update all the locked versions in `Terrafile.lock` run `terraspace bundle update`. You can also delete the `Terrafile.lock` file first.
-
-    terraspace bundle update
-
-You can selectively update multiple modules:
-
-    terraspace bundle update MODS
-    terraspace bundle update mod1 mod2
-
-## List & Info
-
-    terraspace bundle list
-    terraspace bundle info mod1
-
-## DSL Methods
-
-Some of these are methods that apply globally at the Terrafile level. Some of these are options that apply the at the mod method level.
-
-### Terrafile level
-
-Name | Description | Default
---- |  --- | ---
-base_clone_url | Base clone url to use | git@gihtub.com:
-export_path |  Where the modules get exported to saved to. | vendor/modules
-export_purge | Where or not to clean up all existing exported modules folder first. | true
-
-### Mod Method level
-
-Name | Description | Default
---- |  --- | ---
-subfolder | The subfolder where the module lives within the repo. | nil
-export_to | Overrides the export_path Terrafile level option. With this one-off option, other modules in this folder will not be purged. | nil
-
-Examples of some of the options are below:
-
-### base url
-
-The base url used for clone is `git@github.com:`. You can change it with `base_url`, example:
-
-```ruby
-base_url "https://github.com/"  # default base url for git clone
-# ...
-```
-
-### export_path
-
-The default export path is `vendor/modules`, you can change it:
-
-```ruby
-export_path "app/modules"
-# ...
-```
-
-## prune
-
-The `terraspace bundle` commands removes all files in the export_path, `vendor/modules`, by default. You can disable this behavior with:
-
-```ruby
-export_purge false
-# ...
-```
-
-### subfolder
-
-The default export path is `vendor/modules`, you can change it:
-
-```ruby
-# ...
-mod "s3", source: "terraform-aws-s3", subfolder: "path/to/module/s3"
-```
-
-## Config Options
-
-You can also configure some behavior with `TB.config`. IE: `TB.config.terrafile = "/path/to/Terrafile"`
-
-Name | Description | Default
---- | --- | ---
-base_clone_url | Base git url to use for cloning | git@github.com:
-export_path | Where to export the modules to. Can also be set with the env var TB_EXPORT_PATH | vendor/modules
-export_purge | Whether or not to prune all the modules | true
-logger | Logger instance
-terrafile | The Terrafile path. Can also be set with the env TB_TERRAFILE || "Terrafile"
+For more detailed usage instructions refer to the [Terraspace Terrafile docs](https://terraspace.cloud/docs/terrafile/options/)
 
 ## Installation
 
@@ -123,7 +43,7 @@ To install:
 * Handles updating the `Terrafile.lock` based on the `Terrafile`
 * Others running the `terraspace bundle install` will install the exact same module versions based the `Terrafile.lock`.
 * To update `Terraform.lock` run `terraspace bundle update`.
-* The repos are downloaded to `/tmp/terraspace-bundler` area as a cache. Delete the cache by running `terraspace bundle purge_cache`.
+* The repos are downloaded to the `/tmp/terraspace-bundler` area as a cache. Delete the cache by running `terraspace bundle purge_cache`.
 
 ## Contributing
 

data/lib/terraspace_bundler/cli/bundle.rb

@@ -22,7 +22,7 @@ class TerraspaceBundler::CLI
     long_desc Help.text("bundle/install")
     terrafile_option.call
     def install
-      Runner.new(options).run
+      TB::Runner.new(options).run
     end
 
     desc "purge_cache", "Purge cache."
@@ -35,7 +35,7 @@ class TerraspaceBundler::CLI
     long_desc Help.text("bundle/update")
     terrafile_option.call
     def update(*mods)
-      Runner.new(options.merge(mods: mods, mode: "update")).run
+      TB::Runner.new(options.merge(mods: mods, mode: "update")).run
     end
   end
 end

data/lib/terraspace_bundler/cli/help/bundle/info.md

@@ -5,4 +5,4 @@
         sha: bad676a426f5d3ddb48e674bebd6fd52f063a8b4
         source: org/terraform-random-pet
         type: git
-        url: git@github.com:org/terraform-random-pet
+        url: https://github.com/org/terraform-random-pet

data/lib/terraspace_bundler/config.rb

@@ -5,11 +5,12 @@ module TerraspaceBundler
 
     def config
       config = ActiveSupport::OrderedOptions.new
-      config.base_clone_url = "git@github.com:"
-      config.export_path = ENV['TB_EXPORT_PATH'] || "vendor/modules"
+      config.base_clone_url = "https://github.com/"
+      config.export_to = ENV['TB_EXPORT_TO'] || "vendor/modules"
       config.export_purge = ENV['TB_EXPORT_PRUNE'] == '0' ? false : true
       config.lockfile = "#{config.terrafile}.lock"
       config.logger = new_logger
+      config.stack_options = {dest: "app/stacks", purge: nil, examples: "examples"}  # Note: Important purge is nil not false so can fallback to Terrafile-level stack_options
       config.terrafile = ENV['TB_TERRAFILE'] || "Terrafile"
       config
     end

data/lib/terraspace_bundler/dsl/syntax.rb

@@ -1,28 +1,32 @@
 class TerraspaceBundler::Dsl
   module Syntax
     def org(url)
-      global[:org] = url
+      config.org = url
     end
     alias_method :user, :org
 
     def base_clone_url(value)
-      TB.config.base_clone_url = value
+      config.base_clone_url = value
     end
 
-    def export_path(path)
-      global[:export_path] = path
+    def export_to(path)
+      config.export_to = path
     end
 
     def export_purge(value)
-      TB.config.export_purge = value
+      config.export_purge = value
     end
 
-    def mod(*args, **options)
-      meta[:mods] << {args: args, options: options}
+    def stack_options(value={})
+      config.stack_options.merge!(value)
+    end
+
+    def config
+      TB.config
     end
 
-    def global
-      meta[:global]
+    def mod(*args, **options)
+      meta[:mods] << {args: args, options: options}
     end
   end
 end

data/lib/terraspace_bundler/exporter.rb

@@ -1,6 +1,5 @@
 module TerraspaceBundler
   class Exporter
-    include TB::Mod::PathConcern
     include TB::Util::Logging
 
     def initialize(options={})
@@ -10,34 +9,25 @@ module TerraspaceBundler
     def run
       purge
       lockfile.mods.each do |mod|
+        logger.info "Exporting #{mod.name}"
         export(mod)
       end
     end
 
     def export(mod)
       downloader = Mod::Downloader.new(mod)
+      downloader.run
       downloader.switch_version(mod.sha)
-
-      stage_path = stage_path(mod.full_repo)
-      stage_path = "#{stage_path}/#{mod.subfolder}" if mod.subfolder
-      mod_path = mod_path(mod)
-      FileUtils.rm_rf(mod_path)
-      FileUtils.mkdir_p(File.dirname(mod_path))
-      FileUtils.cp_r(stage_path, mod_path)
-      FileUtils.rm_rf("#{mod_path}/.git")
-      logger.debug "Exported: #{mod_path}"
-    end
-
-    def mod_path(mod)
-      name = mod.name
-      export_to = mod.export_to || TB.config.export_path
-      "#{export_to}/#{name}"
+      copy = Copy.new(mod)
+      copy.mod
+      copy.stacks
+      logger.debug "Exported: #{copy.mod_path}"
     end
 
   private
     def purge
       return unless TB.config.export_purge
-      FileUtils.rm_rf(TB.config.export_path)
+      FileUtils.rm_rf(TB.config.export_to)
     end
 
     def lockfile

data/lib/terraspace_bundler/exporter/base.rb

@@ -0,0 +1,6 @@
+class TerraspaceBundler::Exporter
+  class Base
+    include TB::Mod::PathConcern
+    include TB::Util::Logging
+  end
+end

data/lib/terraspace_bundler/exporter/copy.rb

@@ -0,0 +1,25 @@
+class TerraspaceBundler::Exporter
+  class Copy < Base
+    def initialize(mod)
+      @mod = mod
+    end
+
+    def mod
+      FileUtils.rm_rf(mod_path)
+      FileUtils.mkdir_p(File.dirname(mod_path))
+      FileUtils.cp_r(src_path, mod_path)
+      FileUtils.rm_rf("#{mod_path}/.git")
+    end
+
+    def stacks
+      Stacks.new(@mod).export
+    end
+
+    # src path is from the stage area
+    def src_path
+      path = stage_path(@mod.full_repo) # from PathConcern
+      path = "#{path}/#{@mod.subfolder}" if @mod.subfolder
+      path
+    end
+  end
+end

data/lib/terraspace_bundler/exporter/stacks.rb

@@ -0,0 +1,14 @@
+class TerraspaceBundler::Exporter
+  class Stacks < Base
+    def initialize(mod)
+      @mod = mod
+      @stacks = mod.stacks || []
+    end
+
+    def export
+      @stacks.each do |options|
+        Stack.new(@mod, options).export
+      end
+    end
+  end
+end

data/lib/terraspace_bundler/exporter/stacks/base.rb

@@ -0,0 +1,5 @@
+class TerraspaceBundler::Exporter::Stacks
+  class Base < TerraspaceBundler::Exporter::Base
+    include TB::Util::Logging
+  end
+end

data/lib/terraspace_bundler/exporter/stacks/rewrite.rb

@@ -0,0 +1,57 @@
+class TerraspaceBundler::Exporter::Stacks
+  class Rewrite < Base
+    def initialize(stack)
+      @stack = stack
+    end
+
+    def run
+      expr = "#{@stack.dest}/*.tf"
+      Dir.glob(expr).each do |path|
+        next unless File.file?(path) # skip symlinks and dirs
+        replace(path)
+      end
+    end
+
+    def replace(path)
+      lines = IO.readlines(path)
+      new_lines = new_lines(lines)
+      text = new_lines.join('')
+      IO.write(path, text)
+    end
+
+    def new_lines(lines)
+      lines.map do |line|
+        # leading spaces or no space. but cannot have any other chars
+        # can have space betweens source and =
+        line =~ /^\s*source\s*=/ ? new_line(line) : line
+      end
+    end
+
+    def new_line(line)
+      # marker comment so files dont get updated twice when purge is false
+      terraspace_comment = "# updated by terraspace"
+      return line unless line.include?("../")
+      return line if line.include?(terraspace_comment)
+
+      md = line.match(/^\s*source\s*=\s*"(.*)?"/)
+      unless md
+        logger.error "ERROR: unable to find the module source".color(:red)
+        logger.error line
+        exit 1
+      end
+      # Lots of cases covered by specs
+      #
+      #     ../..
+      #     ../../
+      #     ../../modules/iam-user
+      #     ../../../modules/compute_instance
+      #
+      source_value = md[1] # IE: ""../../../modules/compute_instance"
+      dirs = source_value.split('/')
+      count = dirs.count('..')
+      mod_dir = dirs[count..-1].join('/')
+      full_dir = [@stack.mod.name, mod_dir].reject(&:empty?).join('/')
+      %Q|  source = "../../modules/#{full_dir}" #{terraspace_comment}\n|
+    end
+  end
+end

data/lib/terraspace_bundler/exporter/stacks/stack.rb

@@ -0,0 +1,77 @@
+class TerraspaceBundler::Exporter::Stacks
+  class Stack < Base
+    attr_reader :mod
+    def initialize(mod, options={})
+      @mod, @options = mod, options
+    end
+
+    def export
+      copy
+      rewrite
+    end
+
+    def copy
+      return unless @options
+
+      FileUtils.rm_rf(dest) if purge?
+      return if File.exist?(dest)
+
+      FileUtils.mkdir_p(File.dirname(dest))
+      FileUtils.cp_r(src, dest)
+    end
+
+    def rewrite
+      Rewrite.new(self).run
+    end
+
+    def src
+      src = @options[:src]
+      without_examples = [mod_path, src].compact.join('/')
+      with_examples = [examples_folder, src].compact.join('/')
+      paths = [with_examples, without_examples]
+      found = paths.find do |path|
+        File.exist?(path)
+      end
+
+      unless found
+        searched = paths.map { |p| pretty_path(p) }.map { |p| "    #{p}" }.join("\n")
+        logger.error "ERROR: Example not found. stack src: #{src}. Searched:".color(:red)
+        logger.error searched
+        exit 1
+      end
+      found
+    end
+
+    def pretty_path(path)
+      path.sub("#{Dir.pwd}/",'')
+    end
+
+    # public method used by StackConcern#all_stacks
+    def examples_folder
+      [mod_path, examples].join('/')
+    end
+
+    def examples
+      @options[:examples] || TB.config.stack_options[:examples]
+    end
+
+    def dest
+      dest = @options[:dest] || TB.config.stack_options[:dest]
+      name = @options[:name] || @mod.name # falls back to mod name by default
+      "#{dest}/#{name}"
+    end
+
+    # purge precedence:
+    #
+    #     1. Terrafile mod level stack option
+    #     2. Terrafile-level stack_options
+    #
+    def purge?
+      # config.stack_options is set from Terrafile-level stack_options to TB.config.stack
+      # relevant source code: dsl/syntax.rb: def stack_options
+      config = TB.config.stack_options[:purge]
+      config = config.nil? ? false : config
+      @options[:purge].nil? ? config : @options[:purge]
+    end
+  end
+end

data/lib/terraspace_bundler/lockfile.rb

@@ -8,11 +8,11 @@ module TerraspaceBundler
     # {"vpc"=>
     #   {"sha"=>"52328b2b5197f9b95e3005cfcfb99595040ee45b",
     #   "source"=>"org/terraform-aws-vpc",
-    #   "url"=>"git@github.com:org/terraform-aws-vpc"},
+    #   "url"=>"https://github.com/org/terraform-aws-vpc"},
     # "instance"=>
     #   {"sha"=>"570cca3ea7b25e3af1961dc57b27ca2c129b934a",
     #   "source"=>"org/terraform-aws-instance",
-    #   "url"=>"git@github.com:org/terraform-aws-instance"}}
+    #   "url"=>"https://github.com/org/terraform-aws-instance"}}
     @@mods = nil
     def mods
       return @@mods if @@mods
@@ -29,9 +29,7 @@ module TerraspaceBundler
 
     # update (if version mismatch) or create (if missing)
     def sync(mod)
-      changed = changed?(mod)
-      logger.debug "Detecting change for mod #{mod.name} changed #{changed.inspect}"
-      replace!(mod) if changed
+      replace!(mod) if changed?(mod)
     end
 
     # mod built from Terrafile
@@ -45,7 +43,7 @@ module TerraspaceBundler
 
       comparer = VersionComparer.new(found, mod)
       comparer.run
-      logger.debug(comparer.reason) if comparer.reason
+      logger.debug("REASON: #{comparer.reason}") if comparer.reason
       comparer.changed?
     end
 

data/lib/terraspace_bundler/lockfile/version_comparer.rb

@@ -10,9 +10,24 @@ class TerraspaceBundler::Lockfile
       @changed
     end
 
+    # Tricky logic, maybe spec this.
+    #
+    #   no mods specified:
+    #     terraspace bundle update  # no mods specified => update all
+    #     terraspace bundle install # no Terrafile.lock => update all
+    #   mods specified:
+    #     terraspace bundle update s3  # explicit mod => update s3
+    #     terraspace bundle install s3 # errors: not possible to specify module for install command
+    #
+    # Note: Install with specific mods wipes existing mods. Not worth it to support.
+    #
     def run
       @changed = false
-      strict_versions = %w[subfolder ref tag export_to]
+
+      # Most props are "strict" version checks. So if user changes options generally in the mod line
+      # the Terrafile.lock will get updated, which is expected behavior.
+      props = @locked.props.keys + @current.props.keys
+      strict_versions = props.uniq.sort - [:sha]
       strict_versions.each do |version|
         @changed = @locked.send(version) != @current.send(version)
         if @changed
@@ -21,23 +36,27 @@ class TerraspaceBundler::Lockfile
         end
       end
 
-      # Loose version checks work a little different. If not set it explicitly, they will not be checked.
-      # Will use locked version in Terrafile.lock in this case.
-      # Note: Also, check the sha last since it triggers a git fetch.
-      loose_versions = %w[branch sha]
-      loose_versions.each do |version|
-        @changed = @current.send(version) && @current.send(version) != @locked.send(version)
-        if @changed
-          @reason = reason_message(version)
-          return @changed
-        end
+      # Lots of nuance with the sha check that works differently
+      # Only check when set.
+      # Also in update mode then always check it.
+      @changed = @current.sha && !@locked.sha.include?(@current.sha) ||
+                 update_mode? && !@current.latest_sha.include?(@locked.sha)
+      if @changed
+        @reason = reason_message("sha")
+        return @changed
       end
 
       @changed
     end
 
     def reason_message(version)
-      "Replacing mod: #{@current.name}. #{version} is different in Terrafile and Terrafile.lock"
+      "Replacing mod #{@current.name} because #{version} is different in Terrafile and Terrafile.lock"
+    end
+
+    def update_mode?
+      self.class.update_mode
     end
+
+    class_attribute :update_mode
   end
 end

data/lib/terraspace_bundler/lockfile/yamler.rb

@@ -19,7 +19,7 @@ class TerraspaceBundler::Lockfile
     def item(mod)
       props = mod.props.dup # passthrough: name, url, version, ref, tag, branch etc
       props.delete(:name) # different structure in Terrafile.lock YAML
-      props[:sha] = mod.sha
+      props[:sha] ||= mod.latest_sha
       props.delete_if { |k,v| v.nil? }
       { mod.name => props }
     end

data/lib/terraspace_bundler/mod.rb

@@ -1,7 +1,9 @@
 module TerraspaceBundler
   class Mod
     extend PropsExtension
-    props :name, :source, :url, :subfolder, :type, :export_to
+    props :export_to, :name, :sha, :source, :subfolder, :type, :url
+
+    include StackConcern
 
     attr_reader :props, :version, :ref, :tag, :branch
     def initialize(props={})
@@ -10,12 +12,6 @@ module TerraspaceBundler
       @version, @ref, @tag, @branch = @props[:version], @props[:ref], @props[:tag], @props[:branch]
     end
 
-    def sha
-      # sha will already be set if coming from Terrafile.lock
-      # sha will be lazily fetch set if coming from Terrafile
-      @props[:sha] ||= fetch_sha
-    end
-
     def checkout_version
       v = detected_version
       v = "v#{v}" if type == "registry" && @version && !v.starts_with?("v")
@@ -35,12 +31,13 @@ module TerraspaceBundler
       "#{org}/#{repo}"
     end
 
-  private
-    def fetch_sha
+    def latest_sha
       downloader = Downloader.new(self)
       downloader.run
       downloader.sha
     end
+
+  private
     # support variety of options, prefer version
     def detected_version
       @version || @ref || @tag || @branch

data/lib/terraspace_bundler/mod/downloader.rb

@@ -16,9 +16,7 @@ class TerraspaceBundler::Mod
       FileUtils.mkdir_p(org_path)
 
       Dir.chdir(org_path) do
-        unless File.exist?(@mod.repo)
-          sh "git clone #{@mod.url}"
-        end
+        clone unless File.exist?(@mod.repo)
 
         Dir.chdir(@mod.repo) do
           git "pull"
@@ -27,14 +25,35 @@ class TerraspaceBundler::Mod
         end
       end
     end
+    memoize :run
+
+    def clone
+      sh "git clone #{@mod.url}"
+    rescue TB::GitError => e
+      logger.error "ERROR: #{e.message}".color(:red)
+      exit 1
+    end
 
     def stage
       copy_to_stage
-      # TODO: if there's no master, need to checkout if its the default branch
-      checkout = @mod.checkout_version || "master"
+      checkout = @mod.checkout_version || default_branch
       switch_version(checkout)
     end
 
+    # Note: if not in a git repo, will get this error message in stderr
+    #
+    #    fatal: Not a git repository (or any of the parent directories): .git
+    #
+    def default_branch
+      origin = `git remote show origin`.strip
+      found = origin.split("\n").find { |l| l.include?("HEAD") }
+      if found
+        found.split(':').last.strip
+      else
+        'master'
+      end
+    end
+
     def switch_version(version)
       stage_path = stage_path("#{@mod.org}/#{@mod.repo}")
       logger.debug "Within: #{stage_path}"

data/lib/terraspace_bundler/mod/path_concern.rb

@@ -24,5 +24,10 @@ class TerraspaceBundler::Mod
     def stage_path(name)
       "#{stage_root}/#{name}"
     end
+
+    def mod_path
+      export_to = @mod.export_to || TB.config.export_to
+      "#{export_to}/#{@mod.name}"
+    end
   end
 end

data/lib/terraspace_bundler/mod/props_builder.rb

@@ -3,7 +3,6 @@
 class TerraspaceBundler::Mod
   class PropsBuilder
     extend Memoist
-    include TerraspaceBundler::Dsl::Concern
 
     def initialize(params={})
       @params = params
@@ -28,7 +27,7 @@ class TerraspaceBundler::Mod
       if registry?
         @source
       else
-        @source.include?('/') ? @source : "#{dsl.global[:org]}/#{@source}"
+        @source.include?('/') ? @source : "#{TB.config.org}/#{@source}"
       end
     end
 

data/lib/terraspace_bundler/mod/registry.rb

@@ -3,6 +3,8 @@ require 'open-uri'
 
 class TerraspaceBundler::Mod
   class Registry
+    include TB::Util::Logging
+
     def initialize(source, version)
       @source, @version = source, version
     end
@@ -41,7 +43,8 @@ class TerraspaceBundler::Mod
         resp = http_request(next_url)
         download_url = resp.header["x-terraform-get"]
       else
-        raise "Unable to lookup up module in Terraform Registry: #{resp}"
+        logger.error "ERROR: Unable to lookup up module in Terraform Registry: #{@source}".color(:red)
+        exit 1
       end
 
       url = download_url.sub(%r{/archive/.*},'')

data/lib/terraspace_bundler/mod/stack_concern.rb

@@ -0,0 +1,44 @@
+class TerraspaceBundler::Mod
+  module StackConcern
+    def stacks
+      stacks = @props[:stacks] || @props[:stack]
+      return unless stacks
+      if all_stacks?(stacks)
+        stacks = all_stacks
+      end
+      normalize_stacks(stacks)
+    end
+    alias_method :stack, :stacks # important to have alias for VersionCheck
+
+    def all_stacks?(*stacks)
+      stacks.flatten == [:all]
+    end
+
+    def all_stacks
+      stack = TerraspaceBundler::Exporter::Stacks::Stack.new(self) # to get the mod src path
+      expr = "#{stack.examples_folder}/*"
+      Dir.glob(expr).map do |path|
+        example = File.basename(path)
+        # Set name so multiple app/stacks are created instead of just one app/stack/MOD
+        {
+          name: example,
+          src: example,
+        }
+      end
+    end
+
+    # Normalizes stack options to an Array of Hashes or just a single Hash
+    def normalize_stacks(option)
+      defaults = TB.config.stack_options.dup
+      result = case option
+      when String
+        [defaults.merge(src: option)]
+      when Array
+        option.map! {|s| normalize_stacks(s) }
+      else # Hash
+        [defaults.merge!(option)]
+      end
+      result.flatten
+    end
+  end
+end

data/lib/terraspace_bundler/{installer.rb → runner.rb}

@@ -1,5 +1,5 @@
 module TerraspaceBundler
-  class Installer < CLI::Base
+  class Runner < CLI::Base
     def run
       Syncer.new(@options).run
       Exporter.new(@options).run
@@ -8,10 +8,10 @@ module TerraspaceBundler
 
     def finish_message
       no_modules_found = true
-      export_paths.each do |export_path|
-        found = Dir.exist?(export_path) && !Dir.empty?(export_path)
+      export_paths.each do |path|
+        found = Dir.exist?(path) && !Dir.empty?(path)
         next unless found
-        logger.info  "Modules saved to #{export_path}"
+        logger.info  "Modules saved to #{path}"
         no_modules_found = false
       end
 
@@ -20,7 +20,7 @@ module TerraspaceBundler
 
     def export_paths
       export_paths = Terrafile.instance.mods.map(&:export_to).compact.uniq
-      export_paths << TB.config.export_path
+      export_paths << TB.config.export_to
       export_paths
     end
   end

data/lib/terraspace_bundler/syncer.rb

@@ -1,7 +1,6 @@
 module TerraspaceBundler
   class Syncer
     include TB::Util::Logging
-    include Dsl::Concern
 
     def initialize(options={})
       @options = options
@@ -31,17 +30,18 @@ module TerraspaceBundler
     end
 
     def sync_mods
+      # VersionComparer is used in lockfile.sync and does heavy lifting to check if mod should be updated and replaced
+      TB::Lockfile::VersionComparer.update_mode = @options[:mode] == "update"
       terrafile.mods.each do |mod|
-        update = update?(mod)
-        next unless update
+        next unless sync?(mod)
+        logger.debug "Syncing #{mod.name}"
         lockfile.sync(mod) # update (if version mismatch) or create (if missing)
       end
     end
 
-    def update?(mod)
+    def sync?(mod)
       names = @options[:mods]
-      return true if names.nil? || names.empty? # when empty never skip update
-      names.include?(mod.name)
+      names.blank? or names.include?(mod.name)
     end
 
     def validate!

data/lib/terraspace_bundler/terrafile.rb

@@ -3,7 +3,6 @@ module TerraspaceBundler
     include Singleton
     extend Memoist
     include TB::Util::Logging
-    include Dsl::Concern
 
     # dsl meta example:
     # {:global=>{:org=>"boltopspro"},
@@ -11,7 +10,7 @@ module TerraspaceBundler
     #   [{:args=>["eks"], :options=>{:source=>"terraform-aws-eks"}},
     #   {:args=>["vpc"], :options=>{:source=>"terraform-aws-vpc"}}]}
     def mods
-      dsl.meta[:mods].map do |params|
+      TB.dsl.meta[:mods].map do |params|
         new_mod(params)
       end
     end

data/lib/terraspace_bundler/version.rb

@@ -1,3 +1,3 @@
 module TerraspaceBundler
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/spec/fixtures/rewrite/example-subfolder.tf

@@ -0,0 +1,3 @@
+module "compute_instance" {
+  source            = "../../../modules/compute_instance"
+}

data/spec/fixtures/rewrite/module-subfolder.tf

@@ -0,0 +1,3 @@
+module "iam-user" {
+  source = "../../modules/iam-user"
+}

data/spec/fixtures/rewrite/no-leading.tf

@@ -0,0 +1,3 @@
+module "consul" {
+source="../.." # no leading space
+}

data/spec/fixtures/rewrite/no-path.tf

@@ -0,0 +1,8 @@
+module "consul" {
+  source = "hashicorp/consul/aws"
+  version = "0.1.0"
+}
+
+module "example" {
+  source = "github.com/hashicorp/example"
+}

data/spec/fixtures/rewrite/no-spaces.tf

@@ -0,0 +1,3 @@
+module "consul" {
+  source="../.." # no spaces
+}

data/spec/fixtures/rewrite/normal.tf

@@ -0,0 +1,3 @@
+module "consul" {
+  source = "../.."
+}

data/spec/fixtures/rewrite/spaces.tf

@@ -0,0 +1,3 @@
+module "consul" {
+  source     =   "../.." # lots of extra spaces
+}

data/spec/fixtures/rewrite/trailing-slash.tf

@@ -0,0 +1,3 @@
+module "consul" {
+  source = "../../"
+}

data/spec/spec_helper.rb

@@ -22,6 +22,10 @@ module Helper
   def show_command?
     ENV['DEBUG'] || ENV['SHOW_COMMAND']
   end
+
+  def fixture(path)
+    "spec/fixtures/#{path}"
+  end
 end
 
 RSpec.configure do |c|

data/spec/terraform_bundler/exporter/stacks/rewrite_spec.rb

@@ -0,0 +1,86 @@
+describe TB::Exporter::Stacks::Rewrite do
+  let(:rewrite) { described_class.new(stack) }
+  let(:stack) do
+    stack = double(:stack).as_null_object
+    allow(stack).to receive(:name).and_return("stack_name")
+    stack
+  end
+
+  before(:each) do
+    allow(IO).to receive(:write) do |path, text|
+      @text = text
+    end
+  end
+
+  context "has path-based module" do
+    let(:path) { fixture("rewrite/normal.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "has path-based module with lots of extra spaces in between" do
+    let(:path) { fixture("rewrite/spaces.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "has path-based module with lots of no spaces inbetween" do
+    let(:path) { fixture("rewrite/no-spaces.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "has path-based module with no leading spaces" do
+    let(:path) { fixture("rewrite/no-leading.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "module is within subfolder and not at root" do
+    let(:path) { fixture("rewrite/module-subfolder.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name/modules/iam-user"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "example is within subfolder of module examples" do
+    let(:path) { fixture("rewrite/example-subfolder.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name/modules/compute_instance"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "trailing slash" do
+    let(:path) { fixture("rewrite/trailing-slash.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      replaced = @text.include?('source = "../../modules/stack_name"')
+      expect(replaced).to be true
+    end
+  end
+
+  context "not a path-based module" do
+    let(:path) { fixture("rewrite/no-path.tf") }
+    it "replace" do
+      rewrite.replace(path)
+      kept = @text.include?('source = "hashicorp/consul/aws"') # keeps current source link
+      expect(kept).to be true
+    end
+  end
+end

data/spec/terraform_bundler/mod_spec.rb

@@ -0,0 +1,75 @@
+describe TB::Mod do
+  let(:mod) { described_class.new(props) }
+  let(:props) do
+    {
+      name: "example",
+      source: "org/terraform-aws-vpc",
+    }
+  end
+
+  context "normalize_stacks" do
+    context "single String" do
+      let(:stacks) { "vpc" }
+      it "return List of Hashes" do
+        options = mod.normalize_stacks(stacks)
+        expect(options).to eq(
+          [{:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"vpc"}]
+        )
+      end
+    end
+
+    context "single Hash" do
+      let(:stacks) { {src: "vpc"} }
+      it "return List of Hashes" do
+        options = mod.normalize_stacks(stacks)
+        expect(options).to eq(
+          [{:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"vpc"}]
+        )
+      end
+    end
+
+    context "multiple Strings" do
+      let(:stacks) { ["ec2", "vpc"] }
+      it "return List of Hashes" do
+        options = mod.normalize_stacks(stacks)
+        expect(options).to eq(
+          [{:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"ec2"},
+           {:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"vpc"}]
+        )
+      end
+    end
+
+    context "multiple Hash" do
+      let(:stacks) { [{src: "ec2"}, {src: "vpc"}] }
+      it "return List of Hashes" do
+        options = mod.normalize_stacks(stacks)
+        expect(options).to eq(
+          [{:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"ec2"},
+           {:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"vpc"}]
+        )
+      end
+    end
+
+    context "multiple Objects" do
+      let(:stacks) { ["ec2", {src: "vpc"}] }
+      it "return List of Hashes" do
+        options = mod.normalize_stacks(stacks)
+        expect(options).to eq(
+          [{:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"ec2"},
+           {:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"vpc"}]
+        )
+      end
+    end
+
+    context "multiple Objects with options" do
+      let(:stacks) { ["ec2", {src: "vpc", purge: true, dest: "vendor/stacks"}] }
+      it "return List of Hashes" do
+        options = mod.normalize_stacks(stacks)
+        expect(options).to eq(
+          [{:dest=>"app/stacks", :purge=>nil, :examples=>"examples", :src=>"ec2"},
+           {:dest=>"vendor/stacks", :purge=>true, :examples=>"examples", :src=>"vpc"}]
+        )
+      end
+    end
+  end
+end

data/spec/terraform_bundler/runner_spec.rb

@@ -1,4 +1,4 @@
-describe TB::CLI::Runner do
+describe TB::Runner do
   let(:runner) { described_class.new }
 
   context "runner" do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: terraspace-bundler
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Tung Nguyen
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-09-14 00:00:00.000000000 Z
+date: 2020-10-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -181,7 +181,6 @@ files:
 - lib/terraspace_bundler/cli/help/completion.md
 - lib/terraspace_bundler/cli/help/completion_script.md
 - lib/terraspace_bundler/cli/purge_cache.rb
-- lib/terraspace_bundler/cli/runner.rb
 - lib/terraspace_bundler/command.rb
 - lib/terraspace_bundler/completer.rb
 - lib/terraspace_bundler/completer/script.rb
@@ -189,11 +188,15 @@ files:
 - lib/terraspace_bundler/config.rb
 - lib/terraspace_bundler/core.rb
 - lib/terraspace_bundler/dsl.rb
-- lib/terraspace_bundler/dsl/concern.rb
 - lib/terraspace_bundler/dsl/syntax.rb
 - lib/terraspace_bundler/exporter.rb
+- lib/terraspace_bundler/exporter/base.rb
+- lib/terraspace_bundler/exporter/copy.rb
+- lib/terraspace_bundler/exporter/stacks.rb
+- lib/terraspace_bundler/exporter/stacks/base.rb
+- lib/terraspace_bundler/exporter/stacks/rewrite.rb
+- lib/terraspace_bundler/exporter/stacks/stack.rb
 - lib/terraspace_bundler/info.rb
-- lib/terraspace_bundler/installer.rb
 - lib/terraspace_bundler/list.rb
 - lib/terraspace_bundler/lockfile.rb
 - lib/terraspace_bundler/lockfile/version_comparer.rb
@@ -206,13 +209,25 @@ files:
 - lib/terraspace_bundler/mod/props_builder.rb
 - lib/terraspace_bundler/mod/props_extension.rb
 - lib/terraspace_bundler/mod/registry.rb
+- lib/terraspace_bundler/mod/stack_concern.rb
+- lib/terraspace_bundler/runner.rb
 - lib/terraspace_bundler/syncer.rb
 - lib/terraspace_bundler/terrafile.rb
 - lib/terraspace_bundler/util/git.rb
 - lib/terraspace_bundler/util/logging.rb
 - lib/terraspace_bundler/version.rb
 - spec/fixtures/Terrafile
+- spec/fixtures/rewrite/example-subfolder.tf
+- spec/fixtures/rewrite/module-subfolder.tf
+- spec/fixtures/rewrite/no-leading.tf
+- spec/fixtures/rewrite/no-path.tf
+- spec/fixtures/rewrite/no-spaces.tf
+- spec/fixtures/rewrite/normal.tf
+- spec/fixtures/rewrite/spaces.tf
+- spec/fixtures/rewrite/trailing-slash.tf
 - spec/spec_helper.rb
+- spec/terraform_bundler/exporter/stacks/rewrite_spec.rb
+- spec/terraform_bundler/mod_spec.rb
 - spec/terraform_bundler/runner_spec.rb
 - terraspace-bundler.gemspec
 homepage: https://github.com/boltops-tools/terraspace-bundler
@@ -240,5 +255,15 @@ specification_version: 4
 summary: Bundles terraform modules
 test_files:
 - spec/fixtures/Terrafile
+- spec/fixtures/rewrite/example-subfolder.tf
+- spec/fixtures/rewrite/module-subfolder.tf
+- spec/fixtures/rewrite/no-leading.tf
+- spec/fixtures/rewrite/no-path.tf
+- spec/fixtures/rewrite/no-spaces.tf
+- spec/fixtures/rewrite/normal.tf
+- spec/fixtures/rewrite/spaces.tf
+- spec/fixtures/rewrite/trailing-slash.tf
 - spec/spec_helper.rb
+- spec/terraform_bundler/exporter/stacks/rewrite_spec.rb
+- spec/terraform_bundler/mod_spec.rb
 - spec/terraform_bundler/runner_spec.rb

data/lib/terraspace_bundler/cli/runner.rb

@@ -1,7 +0,0 @@
-class TerraspaceBundler::CLI
-  class Runner < Base
-    def run
-      TB::Installer.new(@options).run
-    end
-  end
-end

data/lib/terraspace_bundler/dsl/concern.rb

@@ -1,7 +0,0 @@
-class TerraspaceBundler::Dsl
-  module Concern
-    def dsl
-      TB.dsl
-    end
-  end
-end