octocatalog-diff 1.0.4 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a5c575bb439d7b065379f5c30bb650d74f3fa09b
-  data.tar.gz: ba4dda3fd34774227a3babac32f981bdbb4fb03b
+  metadata.gz: 86467134c489b13251760bd9ffd23caced1c5222
+  data.tar.gz: 8260b43c4e844b2224dd491018feb8a1da65c4ed
 SHA512:
-  metadata.gz: 3c577fe4b8add2eb722b83ac5b5c33efece70619d6ca7da88287d2a1d49ac24a7091615d33fd7fe80f413ecee268ecdc24af016e53df0d7dc3f57bc491174db9
-  data.tar.gz: bbb4cdd2d011c87f0ae9f688af9e416bad7009f3391bbb01981a831d16ae83ffe46aadec12cbef579b6076e2c52b77e505c242e5196a381a9258dd679c3a1df2
+  metadata.gz: fedd421520a95d4147878d50510d7f2f149cfcba0933c75d7f88c5f3efbeb45f6105b9c5bab53cbc46cf7234db1987cba0b9bdc327cd0b418ce8f69365723282
+  data.tar.gz: 1bc344f71b92ef5154253e4b115f7916e59a9ee49c21c7344adb80ad7dd97d2f164fb951016e8a22d95ba564eab1dbc27a9c1b0cff8a014d327bfde881fb5d21

data/.version

@@ -1 +1 @@
-1.0.4
+1.1.0

data/doc/CHANGELOG.md

@@ -8,6 +8,16 @@
 </tr>
 </thead><tbody>
 <tr valign=top>
+<td>1.1.0</td>
+<td>2017-05-08</td>
+<td>
+<li><a href="https://github.com/github/octocatalog-diff/pull/108">#108</a>: (Bug) Support hiera.yaml backend declared as a string instead of array</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/105">#105</a>: (Bug) Remove legacy exclusion of tags</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/103">#103</a>: (Enhancement) Identify where the broken reference was declared</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/98">#98</a>: (Enhancement) Separate scripts and commands and make override-able</li>
+</td>
+</tr>
+<tr valign=top>
 <td>1.0.4</td>
 <td>2017-03-17</td>
 <td>

data/doc/advanced-dynamic-ignores.md

@@ -2,6 +2,8 @@
 
 Using the `--ignore-tags` command line option, it is possible to ignore all resources with particular Puppet tags. This allows dynamic ignoring of wrappers or other resources that are not of interest.
 
+NOTE: This option is separate and distinct from `--include-tags`, which controls whether differences in tags themselves will appear as a difference. For more on `--include-tags`, consult the [options reference](/doc/optionsref.md).
+
 ## Getting Started
 
 To use ignored tags, you first need to decide what the name of your tag will be. The standard is `ignored_octocatalog_diff`.

data/doc/advanced-script-override.md

@@ -0,0 +1,51 @@
+# Overriding external scripts
+
+## Background
+
+During normal operation, `octocatalog-diff` runs certain scripts or commands from the underlying operating system. For example, it may run `git` to check out a certain code branch, and run `puppet` to build a catalog.
+
+Each external script is found within the [`scripts`](/scripts) directory.
+
+## How to override scripts
+
+### Command line option
+
+It is possible to override these scripts with customized versions. To do this, specify a directory that contains replacement scripts via the command line:
+
+```
+octocatalog-diff [other options] --override-script-path /path/to/scripts ...
+```
+
+### Configuration file
+
+You can also specify this option via a [configuration file](/doc/configuration.md) setting:
+
+```
+settings[:override_script_path] = '/path/to/scripts'
+```
+
+### Writing replacement scripts
+
+Within the override script path you've configured, place a file with the same name as the built-in script. For example, if you wish to override the `git-extract.sh` script with a custom version, also name your script `git-extract.sh`. (Do NOT create subdirectories within the override directory.)
+
+If you specify an override script path but a particular script is not present there, octocatalog-diff will default to the built-in script. This means that you do not need to create unmodified copies of the built-in scripts. Only override the scripts you need to change.
+
+### Notes
+
+Please note that these scripts are considered part of octocatalog-diff, and not part of your Puppet codebase. Therefore, the path to your scripts must be an absolute path, and we do not support (or intend to support) using multiple script directories during the same run of octocatalog-diff.
+
+## Explanation of scripts
+
+This is an explanation of the [existing scripts supplied by octocatalog-diff](/scripts):
+
+- [`env.sh`](/scripts/env)
+
+    Prints out the environment. This is currently only used for spec tests.
+
+- [`git-extract.sh`](/scripts/git-extract)
+
+    Extracts a specified branch from the git repository into a specified target directory.
+
+- [`puppet.sh`](/scripts/puppet)
+
+    Runs puppet (with additional command line arguments), generally used to compile a catalog or determine the Puppet version.

data/doc/advanced.md

@@ -24,6 +24,7 @@ See also:
 - [Using `octocatalog-diff` without git](/doc/advanced-using-without-git.md)
 - [Catalog validation](/doc/advanced-catalog-validation.md)
 - [Environment setup](/doc/advanced-environments.md)
+- [Overriding built-in octocatalog-diff scripts](/doc/advanced-script-override.md)
 
 ### Controlling output
 

data/doc/dev/releasing.md

@@ -4,9 +4,11 @@ The project maintainers are responsible for bumping the version number, regenera
 
 ## Local testing
 
-To test the new version of `octocatalog-diff` in the Puppet repository:
+*This procedure is performed by a GitHubber.*
 
-0. In the Puppet checkout, start a new branch based off master.
+To test the new version of `octocatalog-diff` in the GitHub Puppet repository:
+
+0. In a checkout of the GitHub Puppet repository, start a new branch based off master.
 0. In the `octocatalog-diff` checkout:
   - Ensure that the desired branch is checked out.
   - Choose a unique internal version number which has never been used in CI. A good guideline is that if you're planning to release a version `0.6.0` then for these tests, use `0.6.0a`, `0.6.0b`, ...
@@ -23,25 +25,62 @@ To test the new version of `octocatalog-diff` in the Puppet repository:
 
 0. Back in the Puppet checkout, ensure that the changes are as expected (updates to Gemfile / Gemfile.lock, addition of new gem). Push the change and build appropriate CI job(s) to validate the changes.
 
-## Merging
+## Merging one PR
+
+This section is useful when releasing a new version based on one PR submitted by a contributor. Following this workflow is important so that the contributor gets appropriate credit in the revision history for his or her work.
+
+0. In your local checkout, start a new branch based off master.
+0. Add the contributor's repository as a remote. For example:
+
+  ```
+  git remote add octocat https://github.com/octocat/octocatalog-diff.git
+  ```
+
+0. Merge in the contributor's branch into your own. For example:
+
+  ```
+  git merge octocat/some-branch
+  ```
+
+0. Update `.version` and `doc/CHANGELOG.md` appropriately. In CHANGELOG you should link to the PR submitted by the contributor (not the PR you're creating now).
+0. Commit your changes to `.version` and `doc/CHANGELOG.md`.
+0. If necessary, auto-generate the build documentation, and commit the changes to your branch.
+
+  ```
+  rake doc:build
+  ```
+
+0. Open a Pull Request based on your branch. Confirm that the history is correct, showing the contributor's commits first, and then your commit(s) updating the version file, change log, and/or auto-generated documentation.
+0. Ensure that CI tests are all passing.
+0. Ensure that you've performed "local testing" within GitHub (typically, ~1 day) to confirm the changes.
+0. Merge your PR and delete your branch.
+0. Confirm that the contributor's PR now appears as merged, and any associated issues have been closed.
+
+## Merging multiple PRs
+
+If multiple PRs will constitute a release, it's generally easier to merge each such PR individually, and then create a separate PR afterwards to update the necessary files.
 
-0. If necessary, complete a Pull Request to update the [version file](/.version).
-0. If necessary, auto-generate the build documentation.
+0. Merge all constituent PRs and ensure that any associated issues have been closed.
+0. Create your own branch based off master.
+0. Update `.version` and `doc/CHANGELOG.md` appropriately. In CHANGELOG you should link to the PR submitted by the contributor (not the PR you're creating now).
+0. Commit your changes to `.version` and `doc/CHANGELOG.md`.
+0. If necessary, auto-generate the build documentation, and commit the changes to your branch.
 
   ```
   rake doc:build
   ```
 
+0. Open a Pull Request based on your branch.
 0. Ensure that CI tests are all passing.
-0. Merge and delete the branch.
+0. Ensure that you've performed "local testing" within GitHub (typically, ~1 day) to confirm the changes.
+0. Merge your PR and delete your branch.
 
 ## Releasing
 
 Generally, a new release will correspond to a merge to master of one or more Pull Requests.
 
 0. Ensure that all changes associated with the release have been merged to master.
-  - Merge all Pull Requests associated with release.
-  - If necessary, complete a Pull Request to update the [change log](/doc/CHANGELOG.md).
+  - Merge all Pull Requests associated with release, including the version number bump, change log update, etc.
   - If necessary (for significant changes), complete a Pull Request to update the top-level README file.
 0. Ensure the the master branch is checked out on your system.
 0. Run the release procedure:

data/doc/optionsref.md

@@ -139,6 +139,8 @@ Usage: octocatalog-diff [command line options]
                                      SSL client certificate to connect to PE ENC
         --pe-enc-ssl-client-key FILENAME
                                      SSL client key to connect to PE ENC
+        --override-script-path DIRNAME
+                                     Directory with scripts to override built-ins
         --no-ignore-tags             Disable ignoring based on tags
         --ignore-tags STRING1[,STRING2[,...]]
                                      Specify tags to ignore
@@ -730,6 +732,19 @@ array of differences, where each difference is an array (the octocatalog-diff 0.
     </td>
   </tr>
 
+  <tr>
+    <td valign=top>
+      <pre><code>--override-script-path DIRNAME</code></pre>
+    </td>
+    <td valign=top>
+      Directory with scripts to override built-ins
+    </td>
+    <td valign=top>
+      Provide an optional directory to override default built-in scripts such as git checkout
+and puppet version determination. (<a href="../lib/octocatalog-diff/cli/options/override_script_path.rb">override_script_path.rb</a>)
+    </td>
+  </tr>
+
   <tr>
     <td valign=top>
       <pre><code>--parallel

data/lib/octocatalog-diff/catalog-diff/differ.rb

@@ -266,11 +266,8 @@ module OctocatalogDiff
               hsh[k] = cleansed_param unless cleansed_param.nil? || cleansed_param.empty?
             elsif k == 'tags'
               # The order of tags is unimportant. Sort this array to avoid false diffs if order changes.
-              # Also if tags is empty, don't add. Most uses of catalog diff will want to ignore tags,
-              # and if you're ignoring tags you won't get here anyway. Also, don't add empty array of tags.
-              unless @opts[:ignore_tags]
-                hsh[k] = v.sort if v.is_a?(Array) && v.any?
-              end
+              # Also if tags is empty, don't add.
+              hsh[k] = v.sort if v.is_a?(Array) && v.any?
             elsif k == 'file' || k == 'line'
               # We don't care, for the purposes of catalog-diff, from which manifest and line this resource originated.
               # However, we may report this to the user, so we will keep it in here for now.

data/lib/octocatalog-diff/catalog-util/bootstrap.rb

@@ -82,8 +82,8 @@ module OctocatalogDiff
 
       # Performs the actual bootstrap of a directory. Intended to be called by bootstrap_directory_parallelizer
       # above, or as part of the parallelized catalog build process from util/catalogs.
+      # @param options [Hash] Directory options: branch, path, tag
       # @param logger [Logger] Logger object
-      # @param dir_opts [Hash] Directory options: branch, path, tag
       def self.bootstrap_directory(options, logger)
         raise ArgumentError, ':path must be supplied' unless options[:path]
         FileUtils.mkdir_p(options[:path]) unless Dir.exist?(options[:path])
@@ -96,11 +96,11 @@ module OctocatalogDiff
 
       # Perform git checkout
       # @param logger [Logger] Logger object
-      # @param dir_opts [Hash] Directory options: branch, path, tag
-      def self.git_checkout(logger, dir_opts)
-        logger.debug("Begin git checkout #{dir_opts[:basedir]}:#{dir_opts[:branch]} -> #{dir_opts[:path]}")
-        OctocatalogDiff::CatalogUtil::Git.check_out_git_archive(dir_opts.merge(logger: logger))
-        logger.debug("Success git checkout #{dir_opts[:basedir]}:#{dir_opts[:branch]} -> #{dir_opts[:path]}")
+      # @param options [Hash] Options (need to contain: basedir, branch, path)
+      def self.git_checkout(logger, options)
+        logger.debug("Begin git checkout #{options[:basedir]}:#{options[:branch]} -> #{options[:path]}")
+        OctocatalogDiff::CatalogUtil::Git.check_out_git_archive(options.merge(logger: logger))
+        logger.debug("Success git checkout #{options[:basedir]}:#{options[:branch]} -> #{options[:path]}")
       rescue OctocatalogDiff::Errors::GitCheckoutError => exc
         logger.error("Git checkout error: #{exc}")
         raise OctocatalogDiff::Errors::BootstrapError, exc

data/lib/octocatalog-diff/catalog-util/builddir.rb

@@ -234,7 +234,7 @@ module OctocatalogDiff
 
         # Munge datadir in hiera config file
         obj = YAML.load_file(file_src)
-        (obj[:backends] || %w(yaml json)).each do |key|
+        ([obj[:backends]].flatten || %w(yaml json)).each do |key|
           next unless obj.key?(key.to_sym)
           if options[:hiera_path_strip].is_a?(String)
             next if obj[key.to_sym][:datadir].nil?

data/lib/octocatalog-diff/catalog-util/command.rb

@@ -21,19 +21,41 @@ module OctocatalogDiff
 
         @node = options[:node]
         raise ArgumentError, 'Node must be specified to compile catalog' if @node.nil? || !@node.is_a?(String)
+
+        # To be initialized on-demand
+        @puppet_argv = nil
+        @puppet_binary = nil
+      end
+
+      # Retrieve puppet_command, puppet_binary, puppet_argv
+      def puppet_argv
+        setup
+        @puppet_argv
+      end
+
+      def puppet_binary
+        setup
+        @puppet_binary
       end
 
-      # Build up the command line to run Puppet
       def puppet_command
-        cmdline = []
+        setup
+        [@puppet_binary, @puppet_argv].flatten.join(' ')
+      end
+
+      private
+
+      # Build up the command line to run Puppet
+      def setup
+        return if @puppet_binary && @puppet_argv
 
         # Where is the puppet binary?
-        puppet = @options[:puppet_binary]
-        raise ArgumentError, 'Puppet binary was not supplied' if puppet.nil?
-        raise Errno::ENOENT, "Puppet binary #{puppet} doesn't exist" unless File.file?(puppet)
-        cmdline << puppet
+        @puppet_binary = @options[:puppet_binary]
+        raise ArgumentError, 'Puppet binary was not supplied' if @puppet_binary.nil?
+        raise Errno::ENOENT, "Puppet binary #{@puppet_binary} doesn't exist" unless File.file?(@puppet_binary)
 
         # Node to compile
+        cmdline = []
         cmdline.concat ['master', '--compile', Shellwords.escape(@node)]
 
         # storeconfigs?
@@ -99,11 +121,9 @@ module OctocatalogDiff
         override_and_append_commandline_with_user_supplied_arguments(cmdline)
 
         # Return full command
-        cmdline.join(' ')
+        @puppet_argv = cmdline
       end
 
-      private
-
       # Private: Mutate the command line with arguments that were passed directly from the
       # user. This appends new arguments and overwrites existing arguments.
       # @param cmdline [Array] Existing command line - mutated by this method

data/lib/octocatalog-diff/catalog-util/git.rb

@@ -1,12 +1,9 @@
 # frozen_string_literal: true
 
-require 'fileutils'
-require 'open3'
 require 'rugged'
-require 'shellwords'
-require 'tempfile'
 
 require_relative '../errors'
+require_relative '../util/scriptrunner'
 
 module OctocatalogDiff
   module CatalogUtil
@@ -24,6 +21,7 @@ module OctocatalogDiff
         path = options.fetch(:path)
         dir = options.fetch(:basedir)
         logger = options.fetch(:logger)
+        override_script_path = options.fetch(:override_script_path, nil)
 
         # Validate parameters
         if dir.nil? || !File.directory?(dir)
@@ -34,31 +32,26 @@ module OctocatalogDiff
         end
 
         # Create and execute checkout script
-        script = create_git_checkout_script(branch, path)
-        logger.debug("Begin git archive #{dir}:#{branch} -> #{path}")
-        output, status = Open3.capture2e(script, chdir: dir)
-        unless status.exitstatus.zero?
-          raise OctocatalogDiff::Errors::GitCheckoutError, "Git archive #{branch}->#{path} failed: #{output}"
-        end
-        logger.debug("Success git archive #{dir}:#{branch}")
-      end
+        sr_opts = {
+          logger: logger,
+          default_script: 'git-extract/git-extract.sh',
+          override_script_path: override_script_path
+        }
+        script = OctocatalogDiff::Util::ScriptRunner.new(sr_opts)
 
-      # Create the temporary file used to interact with the git command line.
-      # To get the options working correctly (-o pipefail in particular) this needs to run under
-      # bash. It's just creating a script, rather than figuring out all the shell escapes...
-      # @param branch [String] Branch name
-      # @param path [String] Target directory
-      # @return [String] Name of script
-      def self.create_git_checkout_script(branch, path)
-        tmp_script = Tempfile.new(['git-checkout', '.sh'])
-        tmp_script.write "#!/bin/bash\n"
-        tmp_script.write "set -euf -o pipefail\n"
-        tmp_script.write "git archive --format=tar #{Shellwords.escape(branch)} | \\\n"
-        tmp_script.write "  ( cd #{Shellwords.escape(path)} && tar -xf - )\n"
-        tmp_script.close
-        FileUtils.chmod 0o755, tmp_script.path
-        at_exit { FileUtils.rm_f tmp_script.path if File.exist?(tmp_script.path) }
-        tmp_script.path
+        sr_run_opts = {
+          :working_dir             => dir,
+          :pass_env_vars           => options[:pass_env_vars],
+          'OCD_GIT_EXTRACT_BRANCH' => branch,
+          'OCD_GIT_EXTRACT_TARGET' => path
+        }
+
+        begin
+          script.run(sr_run_opts)
+          logger.debug("Success git archive #{dir}:#{branch}")
+        rescue OctocatalogDiff::Util::ScriptRunner::ScriptException
+          raise OctocatalogDiff::Errors::GitCheckoutError, "Git archive #{branch}->#{path} failed: #{script.output}"
+        end
       end
 
       # Determine the SHA of origin/master (or any other branch really) in the git repo

data/lib/octocatalog-diff/catalog.rb

@@ -201,29 +201,56 @@ module OctocatalogDiff
       end
       return if missing.empty?
 
-      # At this point there is at least one broken/missing reference. Format an error message and
-      # raise. Error message will look like this:
-      # ---
-      # Catalog has broken references: exec[subscribe caller 1] -> subscribe[Exec[subscribe target]];
-      # exec[subscribe caller 2] -> subscribe[Exec[subscribe target]]; exec[subscribe caller 2] ->
-      # subscribe[Exec[subscribe target 2]]
-      # ---
+      # At this point there is at least one broken/missing reference. Format an error message and raise.
+      errors = format_missing_references(missing)
+      plural = errors =~ /;/ ? 's' : ''
+      raise OctocatalogDiff::Errors::ReferenceValidationError, "Catalog has broken reference#{plural}: #{errors}"
+    end
+
+    private
+
+    # Private method: Format the name of the source file and line number, based on compilation directory and
+    # other settings. This is used by format_missing_references.
+    # @param source_file [String] Raw source file name from catalog
+    # @param line_number [Fixnum] Line number from catalog
+    # @return [String] Formatted source file
+    def format_source_file_line(source_file, line_number)
+      return '' if source_file.nil? || source_file.empty?
+      filename = if compilation_dir && source_file.start_with?(compilation_dir)
+        stripped_file = source_file[compilation_dir.length..-1]
+        stripped_file.start_with?('/') ? stripped_file[1..-1] : stripped_file
+      else
+        source_file
+      end
+      "(#{filename.sub(%r{^environments/production/}, '')}:#{line_number})"
+    end
+
+    # Private method: Format the missing references into human-readable text
+    # Error message will look like this:
+    # ---
+    # Catalog has broken references: exec[subscribe caller 1](file:line) -> subscribe[Exec[subscribe target]];
+    # exec[subscribe caller 2](file:line) -> subscribe[Exec[subscribe target]]; exec[subscribe caller 2](file:line) ->
+    # subscribe[Exec[subscribe target 2]]
+    # ---
+    # @param missing [Array] Array of missing references
+    # @return [String] Formatted references
+    def format_missing_references(missing)
+      unless missing.is_a?(Array) && missing.any?
+        raise ArgumentError, 'format_missing_references() requires a non-empty array as input'
+      end
+
       formatted_references = missing.map do |obj|
         # obj[:target_value] can be a string or an array. If it's an array, create a
         # separate error message per element of that array. This allows the total number
         # of errors to be correct.
-        src = "#{obj[:source]['type'].downcase}[#{obj[:source]['title']}]"
+        src_ref = "#{obj[:source]['type'].downcase}[#{obj[:source]['title']}]"
+        src_file = format_source_file_line(obj[:source]['file'], obj[:source]['line'])
         target_val = obj[:target_value].is_a?(Array) ? obj[:target_value] : [obj[:target_value]]
-        target_val.map { |tv| "#{src} -> #{obj[:target_type].downcase}[#{tv}]" }
-      end
-      formatted_references.flatten!
-      plural = formatted_references.size == 1 ? '' : 's'
-      errors = formatted_references.join('; ')
-      raise OctocatalogDiff::Errors::ReferenceValidationError, "Catalog has broken reference#{plural}: #{errors}"
+        target_val.map { |tv| "#{src_ref}#{src_file} -> #{obj[:target_type].downcase}[#{tv}]" }
+      end.flatten
+      formatted_references.join('; ')
     end
 
-    private
-
     # Private method: Given a list of resources to check, return the references from
     # that list that are missing from the catalog. (An empty array returned would indicate
     # all references are present in the catalog.)

data/lib/octocatalog-diff/catalog/computed.rb

@@ -2,14 +2,14 @@
 
 require 'fileutils'
 require 'json'
-require 'open3'
 require 'stringio'
 
 require_relative '../catalog-util/bootstrap'
 require_relative '../catalog-util/builddir'
 require_relative '../catalog-util/command'
-require_relative '../util/puppetversion'
 require_relative '../catalog-util/facts'
+require_relative '../util/puppetversion'
+require_relative '../util/scriptrunner'
 
 module OctocatalogDiff
   class Catalog
@@ -160,37 +160,60 @@ module OctocatalogDiff
 
       # Get the command to compile the catalog
       # @return [String] Puppet command line
-      def puppet_command(options = @opts)
-        return @puppet_command if @puppet_command
-        raise ArgumentError, '"puppet_binary" was not passed to OctocatalogDiff::Catalog::Computed' unless @puppet_binary
-        command_opts = options.merge(
-          node: @node,
-          compilation_dir: @builddir.tempdir,
-          parser: options.fetch(:parser, :default),
-          puppet_binary: @puppet_binary,
-          fact_file: @builddir.fact_file,
-          dir: @builddir.tempdir,
-          enc: @builddir.enc
-        )
-        command = OctocatalogDiff::CatalogUtil::Command.new(command_opts)
-        @puppet_command = command.puppet_command
+      def puppet_command
+        puppet_command_obj.puppet_command
+      end
+
+      def puppet_command_obj
+        @puppet_command_obj ||= begin
+          raise ArgumentError, '"puppet_binary" was not passed to OctocatalogDiff::Catalog::Computed' unless @puppet_binary
+
+          command_opts = @opts.merge(
+            node: @node,
+            compilation_dir: @builddir.tempdir,
+            parser: @opts.fetch(:parser, :default),
+            puppet_binary: @puppet_binary,
+            fact_file: @builddir.fact_file,
+            dir: @builddir.tempdir,
+            enc: @builddir.enc
+          )
+          OctocatalogDiff::CatalogUtil::Command.new(command_opts)
+        end
       end
 
       # Private method: Actually execute puppet
       # @return [Hash] { stdout, stderr, exitcode }
-      def exec_puppet
+      def exec_puppet(logger)
         # This is the environment provided to the puppet command.
-        env = {
-          'HOME' => ENV['HOME'],
-          'PATH' => ENV['PATH'],
-          'PWD' => @builddir.tempdir
-        }
+        env = {}
         @pass_env_vars.each { |var| env[var] ||= ENV[var] }
-        out, err, status = Open3.capture3(env, puppet_command, unsetenv_others: true, chdir: @builddir.tempdir)
+
+        # This is the Puppet command itself
+        env['OCD_PUPPET_BINARY'] = @puppet_command_obj.puppet_binary
+
+        # Additional passed-in options
+        sr_run_opts = env.merge(
+          logger: logger,
+          working_dir: @builddir.tempdir,
+          argv: @puppet_command_obj.puppet_argv
+        )
+
+        # Set up the ScriptRunner
+        scriptrunner = OctocatalogDiff::Util::ScriptRunner.new(
+          default_script: 'puppet/puppet.sh',
+          override_script_path: @opts[:override_script_path]
+        )
+
+        begin
+          scriptrunner.run(sr_run_opts)
+        rescue OctocatalogDiff::Util::ScriptRunner::ScriptException => exc
+          logger.warn "Puppet command failed: #{exc.message}" if logger
+        end
+
         {
-          stdout: out,
-          stderr: err,
-          exitcode: status.exitstatus
+          stdout: scriptrunner.stdout,
+          stderr: scriptrunner.stderr,
+          exitcode: scriptrunner.exitcode
         }
       end
 
@@ -216,7 +239,7 @@ module OctocatalogDiff
           @retries = retry_num
           time_begin = Time.now
           logger.debug("(#{@tag}) Try #{1 + retry_num} executing Puppet #{puppet_version}: #{puppet_command}")
-          result = exec_puppet
+          result = exec_puppet(logger)
 
           # Success
           if (result[:exitcode]).zero?

data/lib/octocatalog-diff/cli/options/override_script_path.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+# Provide an optional directory to override default built-in scripts such as git checkout
+# and puppet version determination.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::Cli::Options::Option.newoption(:override_script_path) do
+  has_weight 385
+
+  def parse(parser, options)
+    parser.on('--override-script-path DIRNAME', 'Directory with scripts to override built-ins') do |dir|
+      unless dir.start_with?('/')
+        raise ArgumentError, 'Absolute path is required for --override-script-path'
+      end
+
+      unless File.directory?(dir)
+        raise Errno::ENOENT, 'Invalid --override-script-path'
+      end
+
+      options[:override_script_path] = dir
+    end
+  end
+end

data/lib/octocatalog-diff/util/puppetversion.rb

@@ -2,9 +2,7 @@
 
 # Helper to determine the version of Puppet
 
-require 'fileutils'
-require 'open3'
-require 'shellwords'
+require_relative 'scriptrunner'
 
 module OctocatalogDiff
   module Util
@@ -17,19 +15,27 @@ module OctocatalogDiff
       def self.puppet_version(puppet, options = {})
         raise ArgumentError, 'Puppet binary was not supplied' if puppet.nil?
         raise Errno::ENOENT, "Puppet binary #{puppet} doesn't exist" unless File.file?(puppet)
-        cmdline = [Shellwords.escape(puppet), '--version'].join(' ')
 
-        # This is the environment provided to the puppet command.
-        env = {
-          'HOME' => ENV['HOME'],
-          'PATH' => ENV['PATH'],
-          'PWD' => File.dirname(puppet)
+        sr_opts = {
+          default_script: 'puppet/puppet.sh',
+          override_script_path: options[:override_script_path]
         }
-        pass_env_vars = options.fetch(:pass_env_vars, [])
-        pass_env_vars.each { |var| env[var] ||= ENV[var] }
-        out, err, _status = Open3.capture3(env, cmdline, unsetenv_others: true, chdir: env['PWD'])
-        return Regexp.last_match(1) if out =~ /^([\d\.]+)\s*$/
-        raise "Unable to determine Puppet version: #{out} #{err}"
+
+        script = OctocatalogDiff::Util::ScriptRunner.new(sr_opts)
+
+        sr_run_opts = {
+          :logger             => options[:logger],
+          :working_dir        => File.dirname(puppet),
+          :pass_env_vars      => options[:pass_env_vars],
+          :argv               => '--version',
+          'OCD_PUPPET_BINARY' => puppet
+        }
+
+        output = script.run(sr_run_opts)
+        return Regexp.last_match(1) if output =~ /^([\d\.]+)\s*$/
+        # :nocov:
+        raise "Unable to determine Puppet version: #{script.output}"
+        # :nocov:
       end
     end
   end

data/lib/octocatalog-diff/util/scriptrunner.rb

@@ -0,0 +1,147 @@
+# frozen_string_literal: true
+
+# Execute a built-in script (which can also be overridden with a user-supplied script)
+
+require 'fileutils'
+require 'open3'
+require 'shellwords'
+
+module OctocatalogDiff
+  module Util
+    # This is a utility class to execute a built-in script.
+    class ScriptRunner
+      # For an exception running the script
+      class ScriptException < RuntimeError; end
+
+      attr_reader :script, :script_src, :logger, :stdout, :stderr, :exitcode
+
+      # Create the object - the object is a configured script, which can be executed multiple
+      # times with different environment varibles.
+      #
+      # @param opts [Hash] Options hash
+      #   opts[:default_script] (Required) Path to script, relative to `scripts` directory
+      #   opts[:logger] (Optional) Logger object
+      #   opts[:override_script_path] (Optional) Directory where a similarly-named script MAY exist
+      def initialize(opts = {})
+        @logger = opts[:logger]
+        @script_src = find_script(opts.fetch(:default_script), opts[:override_script_path])
+        @script = temp_script(@script_src)
+        @stdout = nil
+        @stderr = nil
+        @exitcode = nil
+      end
+
+      # Execute the script from a given working directory, with additional environment variables
+      # specified in the options hash.
+      #
+      # @param opts [Hash] Options hash
+      #   opts[:working_dir] (Required) Directory where script is to be executed
+      #   opts[:argv] (Optional Array) Command line arguments
+      #   opts[:pass_env_vars] (Optional Array) Environment variables to pass (default: HOME, PATH)
+      #   opts[<STRING>] (Optional) Environment variable
+      def run(opts = {})
+        working_dir = opts.fetch(:working_dir)
+        assert_directory_exists(working_dir)
+
+        argv = opts.fetch(:argv, [])
+        logger = opts[:logger] || @logger
+
+        pass_env_vars = [opts[:pass_env_vars], 'HOME', 'PATH'].flatten.compact
+        env = opts.select { |k, _v| k.is_a?(String) }
+        pass_env_vars.each { |var| env[var] ||= ENV[var] }
+        env['PWD'] = working_dir
+
+        cmdline = [script, argv].flatten.compact.map { |x| Shellwords.escape(x) }.join(' ')
+        log(:debug, "Execute: #{cmdline}", opts[:logger])
+
+        @stdout, @stderr, status = Open3.capture3(env, cmdline, unsetenv_others: true, chdir: working_dir)
+        @exitcode = status.exitstatus
+
+        @stderr.split(/\n/).select { |line| line =~ /\S/ }.each { |line| log(:debug, "STDERR: #{line}", logger) }
+        log(:debug, "Exit status: #{@exitcode}", logger)
+        return @stdout if @exitcode.zero?
+        raise ScriptException, output
+      end
+
+      # All output from the latest execution of the command.
+      # @return [String] Combined output of STDOUT and STDERR
+      def output
+        return if @exitcode.nil?
+        [
+          'STDOUT:',
+          @stdout.split(/\n/).map { |line| "  #{line}" },
+          'STDERR:',
+          @stderr.split(/\n/).map { |line| "  #{line}" }
+        ].flatten.compact.join("\n")
+      end
+
+      private
+
+      # PRIVATE: Log a message, if logger is defined. Since this might be called under `parallel`
+      # it's possible that the logger isn't defined, and if so the logged message is skipped.
+      def log(priority, message, logger = @logger)
+        return unless logger
+        logger.send(priority, [message])
+      end
+
+      # PRIVATE: Create a temporary file with the contents of the script and mark the script executable.
+      # This is to avoid changing ownership or permissions on any user-supplied file.
+      #
+      # @param script [String] Path to script
+      # @return [String] Path to tempfile containing script
+      def temp_script(script)
+        raise Errno::ENOENT, "Script '#{script}' not found" unless File.file?(script)
+        temp_dir = Dir.mktmpdir
+        at_exit do
+          begin
+            FileUtils.remove_entry_secure temp_dir
+          rescue Errno::ENOENT # rubocop:disable Lint/HandleExceptions
+            # OK if the directory doesn't exist since we're trying to remove it anyway
+          end
+        end
+        temp_file = File.join(temp_dir, File.basename(script))
+        File.open(temp_file, 'w') { |f| f.write(File.read(script)) }
+        FileUtils.chmod 0o755, temp_file
+        temp_file
+      end
+
+      # PRIVATE: Determine the path to the script to execute, taking into account the default script
+      # location and the optional override script path.
+      #
+      # @param default_script [String] Path to script, relative to `scripts` directory
+      # @param override_script_path [String] Optional directory with override script
+      # @return [String] Full path to script
+      def find_script(default_script, override_script_path = nil)
+        script = find_script_from_override_path(default_script, override_script_path) ||
+                 File.expand_path("../../../scripts/#{default_script}", File.dirname(__FILE__))
+        raise Errno::ENOENT, "Unable to locate script '#{script}'" unless File.file?(script)
+        script
+      end
+
+      # PRIVATE: Find script from override path.
+      #
+      # @param default_script [String] Path to script, relative to `scripts` directory
+      # @param override_script_path [String] Optional directory
+      # @return [String] Override script if found, else nil
+      def find_script_from_override_path(default_script, override_script_path = nil)
+        return unless override_script_path
+        script_test = File.join(override_script_path, File.basename(default_script))
+        if File.file?(script_test)
+          log(:debug, "Selecting #{script_test} from override script path")
+          script_test
+        else
+          log(:debug, "Did not find #{script_test} in override script path")
+          nil
+        end
+      end
+
+      # PRIVATE: Assert that a directory exists (and is a directory). Raise error if not.
+      #
+      # @param dir [String] Directory to test
+      def assert_directory_exists(dir)
+        return if File.directory?(dir)
+        raise Errno::ENOENT, "Invalid directory '#{dir}'"
+      end
+    end
+  end
+end

data/scripts/env/env.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+# This script echoes back the environment. This is used for spec testing
+# and possible debugging.
+
+env

data/scripts/git-extract/git-extract.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+# This script is called from lib/octocatalog-diff/catalog-util/git.rb and is used to
+# archive and extract a certain branch of a git repository into a target directory.
+
+if [ -z "$OCD_GIT_EXTRACT_BRANCH" ]; then
+  echo "Error: Must declare OCD_GIT_EXTRACT_BRANCH"
+  exit 255
+fi
+
+if [ -z "$OCD_GIT_EXTRACT_TARGET" ]; then
+  echo "Error: Must declare OCD_GIT_EXTRACT_TARGET"
+  exit 255
+fi
+
+set -euf -o pipefail
+git archive --format=tar "$OCD_GIT_EXTRACT_BRANCH" | ( cd "$OCD_GIT_EXTRACT_TARGET" && tar -xf - )

data/scripts/puppet/puppet.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# Script to run Puppet. The default implementation here is simply to pass
+# through the command line arguments (which are likely to be numerous when
+# compiling a catalog).
+
+if [ -z "$OCD_PUPPET_BINARY" ]; then
+  echo "Error: PUPPET_BINARY must be set"
+  exit 255
+fi
+
+"$OCD_PUPPET_BINARY" "$@"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: octocatalog-diff
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.1.0
 platform: ruby
 authors:
 - GitHub, Inc.
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-03-17 00:00:00.000000000 Z
+date: 2017-05-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: diffy
@@ -59,14 +59,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.9.0
+        version: 1.11.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.9.0
+        version: 1.11.1
 - !ruby/object:Gem::Dependency
   name: rugged
   requirement: !ruby/object:Gem::Requirement
@@ -141,16 +141,16 @@ dependencies:
   name: rubocop
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0.35'
+        version: 0.48.1
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0.35'
+        version: 0.48.1
 - !ruby/object:Gem::Dependency
   name: puppetdb-terminus
   requirement: !ruby/object:Gem::Requirement
@@ -171,14 +171,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.14.1
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.14.1
 - !ruby/object:Gem::Dependency
   name: simplecov-json
   requirement: !ruby/object:Gem::Requirement
@@ -199,14 +199,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.5.3
+        version: 4.10.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.5.3
+        version: 4.10.0
 description: |
   Octocatalog-Diff assists with Puppet development and testing by enabling the user to
   compile 2 Puppet catalogs and compare them. It is possible to compare different
@@ -241,6 +241,7 @@ files:
 - doc/advanced-pe-enc.md
 - doc/advanced-puppet-master.md
 - doc/advanced-puppet-versions.md
+- doc/advanced-script-override.md
 - doc/advanced-storeconfigs.md
 - doc/advanced-using-without-git.md
 - doc/advanced.md
@@ -351,6 +352,7 @@ files:
 - lib/octocatalog-diff/cli/options/master_cache_branch.rb
 - lib/octocatalog-diff/cli/options/output_file.rb
 - lib/octocatalog-diff/cli/options/output_format.rb
+- lib/octocatalog-diff/cli/options/override_script_path.rb
 - lib/octocatalog-diff/cli/options/parallel.rb
 - lib/octocatalog-diff/cli/options/parser.rb
 - lib/octocatalog-diff/cli/options/pass_env_vars.rb
@@ -402,7 +404,11 @@ files:
 - lib/octocatalog-diff/util/httparty.rb
 - lib/octocatalog-diff/util/parallel.rb
 - lib/octocatalog-diff/util/puppetversion.rb
+- lib/octocatalog-diff/util/scriptrunner.rb
 - lib/octocatalog-diff/version.rb
+- scripts/env/env.sh
+- scripts/git-extract/git-extract.sh
+- scripts/puppet/puppet.sh
 homepage: https://github.com/github/octocatalog-diff
 licenses:
 - MIT