octocatalog-diff 0.5.1

data/doc/requirements.md

@@ -0,0 +1,16 @@
+# Requirements
+
+To run `octocatalog-diff` you will need these basics:
+
+- Ruby 2.0 or higher
+- Mac OS, Linux, or other Unix-line operating system (Windows is not supported)
+- Ability to install gems, e.g. with [rbenv](https://github.com/rbenv/rbenv) or [rvm](https://rvm.io/), or root privileges to install into the system Ruby
+- Puppet agent for [Linux](https://docs.puppet.com/puppet/latest/reference/install_linux.html) or [Mac OS X](https://docs.puppet.com/puppet/latest/reference/install_osx.html), or installed as a gem
+
+We recommend that you also have the following to get the most out of `octocatalog-diff`, but these are not absolute requirements:
+
+- If your Puppet code stored in a git repository, `octocatalog-diff` can check out branches for you as it does its comparisons. Your git repository can be stored on [GitHub.com](https://github.com/), [GitHub Enterprise](https://enterprise.github.com/home), or similar. If your Puppet code is not stored in a git repository, you can still point the tool at "from" and "to" directories, but you'll have to check them out yourself.
+
+- If you have API access (HTTPS) to PuppetDB, `octocatalog-diff` can retrieve facts automatically and also support [exported resources](https://docs.puppet.com/puppet/latest/reference/lang_exported.html) if you use them. If you are not using PuppetDB or don't have access, the tool can still read facts from YAML files.
+
+- If your site uses an [external node classifier](https://docs.puppet.com/guides/external_nodes.html), `octocatalog-diff` can execute the ENC script as part of its catalog compiles. Depending on how your ENC is designed, this may require network access or credentials to some service. If you are not using an ENC, that's fine. If you have an ENC but don't have the requisite access, depending on your setup the tool could produce unexpected results.

data/doc/roadmap.md

@@ -0,0 +1,26 @@
+# Roadmap
+
+This document outlines our philosophy and goals for the continued development of `octocatalog-diff`.
+
+## Goals
+
+- Work on a system without a full Puppet installation
+- Cause no added load on production Puppet masters (unless you specifically choose to do so with non-default options)
+- Offer a command line tool to help make developers more efficient
+- Offer the ability to in a Continuous Integration (CI) environment
+- Be compatible with Puppet 3.8.7, 4.5, and later versions
+- Provide flexibility to build and compare catalogs even in esoteric Puppet codebases
+
+## Areas for future development
+
+We are considering these areas for possible future development:
+
+- Improved display of diffs, perhaps a web interface
+- Additional CI use cases
+- CI output display to summarize a change and a list of affected hosts, rather than listing all changes host-by-host
+
+## Antipatterns
+
+These are ideas we've evaluated and decided not to pursue. (If you are considering a [contribution](/.github/CONTRIBUTING.md) along these lines, please [open an issue](https://github.com/github/octocatalog-diff/issues/new) before start working on it. We would feel badly if you did a bunch of work that we could not accept.)
+
+- Making this into a Puppet module with a "face" so that it can be run with `puppet octocatalog-diff ...` or similar. (We have specifically designed this tool to run without a full Puppet installation. There are [similar projects](/doc/similar.md) that are distributed as Puppet modules.)

data/doc/similar.md

@@ -0,0 +1,17 @@
+# Similar Projects
+
+We are aware of the following projects that do similar things to octocatalog-diff:
+
+- [Puppet's catalog_preview Puppet module](https://forge.puppet.com/puppetlabs/catalog_preview)
+
+  Installs as a module into your Puppet codebase and helps with migration from older Puppet versions to newer ones, or from open source Puppet to Puppet Enterprise. Also provides the ability to compare environments (branches). Requires a full working Puppet installation.
+
+- [Zack Smith's catalog_diff Puppet module](https://forge.puppet.com/zack/catalog_diff)
+
+  Installs as a module into your Puppet codebase, allowing you to diff catalogs created by different versions of Puppet. Requires a full working Puppet installation.
+
+- [camptocamp's puppet-catalog-diff-viewer](https://github.com/camptocamp/puppet-catalog-diff-viewer)
+
+  A viewer for JSON reports produced by the catalog_diff Puppet module.
+
+`octocatalog-diff` differs from the above projects by running on a system without a fully configured Puppet installation (such as a developer workstation or CI server). This approach allows developers to run it without having access to the production Puppet servers, and it does not put any load on production Puppet masters when it compiles and compares catalogs.

data/doc/troubleshooting.md

@@ -0,0 +1,54 @@
+# Troubleshooting
+
+Things not quite working as expected? This section will contain hints to help you get up and running.
+
+### Make sure the tests pass
+
+If you are getting errors from ruby, we'd really like to know if the tests are passing on your platform. Please follow the [installation instructions](/doc/installation.md#installing-from-source) to install octocatalog-diff from source, if you have not already done so. Once the repository is checked out, change into the directory run `rake` to perform the tests.
+
+If you get test failures from a clean checkout of the master branch, please [open an issue](https://github.com/github/octocatalog-diff/issues/new) to let us know.
+
+### Make sure your configuration file is found and error-free
+
+Run the following command to test for the existence and integrity of your configuration file.
+
+```
+octocatalog-diff --config-test
+```
+
+If you get an error indicating that the file can't be found, or you get errors arising from the content of the file, please review the [configuration instructions](/doc/configuration.md) to make sure you've set things up correctly.
+
+### Run the command in debug mode
+
+Supplying `-d` on the command line, in addition to the node name and any other arguments, will provide a substantial amount of debugging information to the terminal window. If you ultimately end up requesting our help, we will need this debugging output.
+
+Example:
+
+```
+octocatalog-diff -d -n SomeNodeName.yourdomain.com
+```
+
+### Run only certain components of the command
+
+To perform the bootstrapping and catalog compilation in separate steps, you can run octocatalog-diff with arguments asking it to do only one or the other. This will help you narrow down whether the problem is in the bootstrapping (first command) or catalog compilation (second command).
+
+Be sure you are in the directory where your Puppet code is checked out when you run these commands.
+
+To run just the bootstrapping code (do this within a checkout of your Puppet repository):
+
+```
+mkdir /tmp/octo-test
+octocatalog-diff -d --bootstrap-then-exit --bootstrapped-from-dir=/tmp/octo-test
+```
+
+To run just the catalog compilation code (do this within a checkout of your Puppet repository):
+
+```
+octocatalog-diff -d -n SomeNodeName.yourdomain.com -o /tmp/catalog.json --bootstrapped-to-dir=$PWD --catalog-only
+```
+
+### Contact us
+
+Still having trouble? Please [open an issue](https://github.com/github/octocatalog-diff/issues/new) and we will do our best to help.
+
+Please follow the provided issue template, which will ask you for certain output that we need to diagnose the problem.

data/lib/octocatalog-diff.rb

@@ -0,0 +1,12 @@
+# These are all the classes we believe people might want to call directly, so load
+# them in response to a 'require octocatalog-diff'.
+
+loads = [
+  'bootstrap',
+  'catalog',
+  'facts',
+  'puppetdb',
+  'version',
+  'catalog-diff/cli'
+]
+loads.each { |f| require_relative "octocatalog-diff/#{f}" }

data/lib/octocatalog-diff/bootstrap.rb

@@ -0,0 +1,53 @@
+require 'open3'
+require 'shellwords'
+
+module OctocatalogDiff
+  # Contains bootstrap function to bootstrap a checked-out Puppet environment.
+  class Bootstrap
+    # Bootstrap a checked-out Puppet environment
+    # @param options [Hash] Options hash:
+    #        :path [String] => Directory to bootstrap
+    #        :bootstrap_script [String] => Bootstrap script, relative to directory
+    # @return [Hash] => [Fixnum] :status_code, [String] :output
+    def self.bootstrap(options = {})
+      # Options validation
+      unless options[:path].is_a?(String)
+        raise ArgumentError, 'Directory to bootstrap (:path) undefined or wrong data type'
+      end
+      unless File.directory?(options[:path])
+        raise Errno::ENOENT, "Non-existent directory '#{options[:path]}' in bootstrap"
+      end
+      unless options[:bootstrap_script].is_a?(String)
+        raise ArgumentError, 'Bootstrap script (:bootstrap_script) undefined or wrong data type'
+      end
+      bootstrap_script = File.join(options[:path], options[:bootstrap_script])
+      unless File.file?(bootstrap_script)
+        raise Errno::ENOENT, "Non-existent bootstrap script '#{options[:bootstrap_script]}'"
+      end
+
+      # 'env' sets up the environment variables that will be passed to the script.
+      # This is a clean environment.
+      env = {
+        'PWD' => options[:path],
+        'HOME' => ENV['HOME'],
+        'PATH' => ENV['PATH'],
+        'BASEDIR' => options[:basedir]
+      }
+      env.merge!(options[:bootstrap_environment]) if options[:bootstrap_environment].is_a?(Hash)
+
+      # 'opts' are options passed to the Open3.capture2e command which are described
+      # here: http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html#method-c-capture2e
+      # Setting { :chdir => dir } means the shelled-out script will execute in the specified directory
+      # This natively avoids the need to shell out to 'cd $dir && script/bootstrap'
+      opts = { chdir: options[:path], unsetenv_others: true }
+
+      # Actually execute the command and capture the output (combined stdout and stderr).
+      cmd = [bootstrap_script, options[:bootstrap_args]].compact.map { |x| Shellwords.escape(x) }.join(' ')
+      output, status = Open3.capture2e(env, cmd, opts)
+      {
+        status_code: status.exitstatus,
+        output: output
+      }
+    end
+  end
+end

data/lib/octocatalog-diff/catalog-diff/cli.rb

@@ -0,0 +1,205 @@
+require_relative 'cli/catalogs'
+require_relative 'cli/diffs'
+require_relative 'cli/options'
+require_relative 'cli/printer'
+require_relative '../catalog-util/cached_master_directory'
+require_relative 'cli/helpers/fact_override'
+require_relative '../version'
+
+require 'logger'
+require 'socket'
+
+module OctocatalogDiff
+  module CatalogDiff
+    # This is the CLI for catalog-diff. It's responsible for parsing the command line
+    # arguments and then handing off to appropriate methods to perform the catalog-diff.
+    class Cli
+      # Version number
+      VERSION = OctocatalogDiff::Version::VERSION
+
+      # Exit codes
+      EXITCODE_SUCCESS_NO_DIFFS = 0
+      EXITCODE_FAILURE = 1
+      EXITCODE_SUCCESS_WITH_DIFFS = 2
+
+      # The default type+title+attribute to ignore in catalog-diff.
+      DEFAULT_IGNORES = [
+        { type: 'Class' } # Don't care about classes themselves, only what they actually do!
+      ].freeze
+
+      # The default options.
+      DEFAULT_OPTIONS = {
+        from_env: 'origin/master',
+        to_env: '.',
+        colors: true,
+        debug: false,
+        quiet: false,
+        format: :color_text,
+        display_source_file_line: false,
+        compare_file_text: true,
+        display_datatype_changes: true,
+        parallel: true,
+        suppress_absent_file_details: true
+      }.freeze
+
+      # This method is the one to call externally. It is possible to specify alternate
+      # command line arguments, for testing.
+      # @param argv [Array] Use specified arguments (defaults to ARGV)
+      # @param logger [Logger] Logger object
+      # @param opts [Hash] Additional options
+      # @return [Fixnum] Exit code: 0=no diffs, 1=something went wrong, 2=worked but there are diffs
+      def self.cli(argv = ARGV, logger = Logger.new(STDERR), opts = {})
+        # Save a copy of argv to print out later in debugging
+        argv_save = argv.dup
+
+        # Are there additional ARGV to munge, e.g. that have been supplied in the options from a
+        # configuration file?
+        if opts.key?(:additional_argv)
+          raise ArgumentError, ':additional_argv must be array!' unless opts[:additional_argv].is_a?(Array)
+          argv.concat opts[:additional_argv]
+        end
+
+        # Parse command line
+        options = parse_opts(argv)
+
+        # Additional options from hard-coded specified options. These are only processed if
+        # there are not already values defined from command line options.
+        # Note: do NOT use 'options[k] ||= v' here because if the value of options[k] is boolean(false)
+        # it will then be overridden. Whereas the intent is to define values only for those keys that don't exist.
+        opts.each { |k, v| options[k] = v unless options.key?(k) }
+        veto_options = %w(enc header hiera_config include_tags)
+        veto_options.each { |x| options.delete(x.to_sym) if options["no_#{x}".to_sym] }
+        options[:ignore].concat opts.fetch(:additional_ignores, [])
+
+        # Incorporate default options where needed.
+        # Note: do NOT use 'options[k] ||= v' here because if the value of options[k] is boolean(false)
+        # it will then be overridden. Whereas the intent is to define values only for those keys that don't exist.
+        DEFAULT_OPTIONS.each { |k, v| options[k] = v unless options.key?(k) }
+
+        # Fact overrides come in here - 'options' is modified
+        setup_fact_overrides(options)
+
+        # Configure the logger and logger.debug initial information
+        # 'logger' is modified and used
+        setup_logger(logger, options, argv_save)
+
+        # --catalog-only is a special case that compiles the catalog for the "to" branch
+        # and then exits, without doing any 'diff' whatsoever. Support that option.
+        return catalog_only(logger, options) if options[:catalog_only]
+
+        # Set up the cached master directory - maintain it, adjust options if needed. However, if we
+        # are getting the 'from' catalog from PuppetDB, then don't do this.
+        unless options[:cached_master_dir].nil? || options[:from_puppetdb]
+          OctocatalogDiff::CatalogUtil::CachedMasterDirectory.run(options, logger)
+        end
+
+        # bootstrap_then_exit is a special case that only prepares directories and does not
+        # depend on facts. This happens within the 'catalogs' object, since bootstrapping and
+        # preparing catalogs are tightly coupled operations. However this does not actually
+        # build catalogs.
+        catalogs_obj = OctocatalogDiff::CatalogDiff::Cli::Catalogs.new(options, logger)
+        return bootstrap_then_exit(logger, catalogs_obj) if options[:bootstrap_then_exit]
+
+        # Compile catalogs
+        catalogs = catalogs_obj.catalogs
+        logger.info "Catalogs compiled for #{options[:node]}"
+
+        # Cache catalogs if master caching is enabled. If a catalog is being read from the cached master
+        # directory, set the compilation directory attribute, so that the "compilation directory dependent"
+        # suppressor will still work.
+        %w(from to).each do |x|
+          next unless options["#{x}_env".to_sym] == options.fetch(:master_cache_branch, 'origin/master')
+          next if options[:cached_master_dir].nil?
+          catalogs[x.to_sym].compilation_dir = options["#{x}_catalog_compilation_dir".to_sym] || options[:cached_master_dir]
+          rc = OctocatalogDiff::CatalogUtil::CachedMasterDirectory.save_catalog_in_cache_dir(
+            options[:node],
+            options[:cached_master_dir],
+            catalogs[x.to_sym]
+          )
+          logger.debug "Cached master catalog for #{options[:node]}" if rc
+        end
+
+        # Compute diffs
+        diffs_obj = OctocatalogDiff::CatalogDiff::Cli::Diffs.new(options, logger)
+        diffs = diffs_obj.diffs(catalogs)
+        logger.info "Diffs computed for #{options[:node]}"
+
+        # Display diffs
+        logger.info 'No differences' if diffs.empty?
+        printer_obj = OctocatalogDiff::CatalogDiff::Cli::Printer.new(options, logger)
+        printer_obj.printer(diffs, catalogs[:from].compilation_dir, catalogs[:to].compilation_dir)
+
+        # Return the diff object if requested (generally for testing) or otherwise return exit code
+        return diffs if opts[:RETURN_DIFFS]
+        diffs.any? ? EXITCODE_SUCCESS_WITH_DIFFS : EXITCODE_SUCCESS_NO_DIFFS
+      end
+
+      # Parse command line options with 'optparse'. Returns a hash with the parsed arguments.
+      # @param argv [Array] Command line arguments (MUST be specified)
+      # @return [Hash] Options
+      def self.parse_opts(argv)
+        options = { ignore: DEFAULT_IGNORES.dup }
+        Options.parse_options(argv, options)
+      end
+
+      # Fact overrides come in here
+      def self.setup_fact_overrides(options)
+        [:from_fact_override, :to_fact_override].each do |key|
+          o = options["#{key}_in".to_sym]
+          next unless o.is_a?(Array)
+          next unless o.any?
+          options[key] ||= []
+          options[key].concat o.map { |x| OctocatalogDiff::CatalogDiff::Cli::Helpers::FactOverride.new(x) }
+        end
+      end
+
+      # Helper method: Configure and setup logger
+      def self.setup_logger(logger, options, argv_save)
+        # Configure the logger
+        logger.level = Logger::INFO
+        logger.level = Logger::DEBUG if options[:debug]
+        logger.level = Logger::ERROR if options[:quiet]
+
+        # Some debugging information up front
+        logger.debug "Running octocatalog-diff #{VERSION} with ruby #{RUBY_VERSION}"
+        logger.debug "Command line arguments: #{argv_save.inspect}"
+        logger.debug "Running on host #{Socket.gethostname} (#{RUBY_PLATFORM})"
+      end
+
+      # Compile the catalog only
+      def self.catalog_only(logger, options)
+        # Indicate where we are
+        logger.debug "Compiling catalog --catalog-only for #{options[:node]}"
+
+        # Compile catalog
+        catalog_opts = options.merge(
+          from_catalog: '-', # Prevents a compile
+          to_catalog: nil, # Forces a compile
+        )
+        cat_obj = OctocatalogDiff::CatalogDiff::Cli::Catalogs.new(catalog_opts, logger)
+        catalogs = cat_obj.catalogs
+
+        # If the catalog compilation failed, an exception would have been thrown. So if
+        # we get here, the catalog succeeded. Dump the catalog to the appropriate place
+        # and exit successfully.
+        if options[:output_file]
+          File.open(options[:output_file], 'w') { |f| f.write(catalogs[:to].catalog_json) }
+          logger.info "Wrote catalog to #{options[:output_file]}"
+        else
+          puts catalogs[:to].catalog_json
+        end
+        return [catalogs[:from], catalogs[:to]] if options[:RETURN_DIFFS] # For integration testing
+        EXITCODE_SUCCESS_NO_DIFFS
+      end
+
+      # --bootstrap-then-exit command
+      def self.bootstrap_then_exit(logger, catalogs_obj)
+        catalogs_obj.bootstrap_then_exit
+        return EXITCODE_SUCCESS_NO_DIFFS
+      rescue OctocatalogDiff::CatalogDiff::Cli::Catalogs::BootstrapError => exc
+        logger.fatal("--bootstrap-then-exit error: bootstrap failed (#{exc})")
+        return EXITCODE_FAILURE
+      end
+    end
+  end
+end

data/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb

@@ -0,0 +1,240 @@
+require 'json'
+require 'open3'
+require 'yaml'
+require_relative '../../catalog-util/bootstrap' # For BootstrapError
+require_relative '../../catalog'
+require_relative '../../util/parallel'
+
+module OctocatalogDiff
+  module CatalogDiff
+    class Cli
+      # Helper class to construct catalogs, performing all necessary steps such as
+      # bootstrapping directories, installing facts, and running puppet.
+      class Catalogs
+        # Exceptions that are anticipated can be caught in the calling class and tested for explicitly in spec tests.
+        class BootstrapError < RuntimeError; end
+        class CatalogError < RuntimeError; end
+
+        # Constructor
+        # @param options [Hash] Options
+        # @param logger [Logger] Logger object
+        def initialize(options, logger)
+          @options = options
+          @logger = logger
+          @catalogs = nil
+          raise '@logger must not be nil' if @logger.nil?
+        end
+
+        # Compile catalogs. This handles building both the old and new catalog (in parallel) and returns
+        # only when both catalogs have been built.
+        # @return [Hash] { :from => [OctocatalogDiff::Catalog], :to => [OctocatalogDiff::Catalog] }
+        def catalogs
+          @catalogs ||= build_catalog_parallelizer
+        end
+
+        # Handles the "bootstrap then exit" option, which bootstraps directories but
+        # exits without compiling catalogs.
+        def bootstrap_then_exit
+          @logger.debug('Begin bootstrap_then_exit')
+          OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory_parallelizer(@options, @logger)
+          @logger.debug('Success bootstrap_then_exit')
+          @logger.info('Successfully completed --bootstrap-then-exit action')
+        rescue OctocatalogDiff::CatalogUtil::Bootstrap::BootstrapError => exc
+          @logger.error("Bootstrap exception: #{exc}")
+          raise BootstrapError, "Bootstrap exception: #{exc}"
+        end
+
+        private
+
+        # Parallelizes bootstrapping of directories and building catalogs.
+        # @return [Hash] { :from => OctocatalogDiff::Catalog, :to => OctocatalogDiff::Catalog }
+        def build_catalog_parallelizer
+          # Construct parallel tasks. The array supplied to OctocatalogDiff::Util::Parallel is the task portion
+          # of each of the tuples in catalog_tasks.
+          catalog_tasks = build_catalog_tasks
+
+          # Update any tasks for catalogs that do not need to be compiled. This is the case when --catalog-only
+          # is specified and only one catalog is to be built. This will change matching catalog tasks to the 'noop' type.
+          catalog_tasks.map! do |x|
+            if @options["#{x[0]}_catalog".to_sym] == '-'
+              x[1].args[:backend] = :noop
+            elsif @options["#{x[0]}_catalog".to_sym].is_a?(String)
+              x[1].args[:json] = File.read(@options["#{x[0]}_catalog".to_sym])
+              x[1].args[:backend] = :json
+            end
+            x
+          end
+
+          # Initialize the objects for each parallel task. Initializing the object is very fast and does not actually
+          # build the catalog.
+          result = {}
+          catalog_tasks.each do |x|
+            result[x[0]] = OctocatalogDiff::Catalog.new(x[1].args)
+            @logger.debug "Initialized #{result[x[0]].builder} for #{x[0]}-catalog"
+          end
+
+          # Disable --compare-file-text if either (or both) of the chosen backends do not support it
+          if @options.fetch(:compare_file_text, false)
+            result.each do |_key, val|
+              next unless val.convert_file_resources == false
+              @logger.debug "Disabling --compare-file-text; not supported by #{val.builder}"
+              @options[:compare_file_text] = false
+              catalog_tasks.map! do |x|
+                x[1].args[:compare_file_text] = false
+                x
+              end
+              break
+            end
+          end
+
+          # Inject the starting object into the catalog tasks
+          catalog_tasks.map! do |x|
+            x[1].args[:object] = result[x[0]]
+            x
+          end
+
+          # Execute the parallelized catalog builds
+          passed_catalog_tasks = catalog_tasks.map { |x| x[1] }
+          parallel_catalogs = OctocatalogDiff::Util::Parallel.run_tasks(passed_catalog_tasks, @logger, @options[:parallel])
+
+          # If the catalogs array is empty at this point, there is an unexpected size mismatch. This should
+          # never happen, but test for it anyway.
+          unless parallel_catalogs.size == catalog_tasks.size
+            # :nocov:
+            raise "BUG: mismatch catalog_result (#{parallel_catalogs.size} vs #{catalog_tasks.size})"
+            # :nocov:
+          end
+
+          # Construct result hash. Will eventually be in the format
+          # { :from => OctocatalogDiff::Catalog, :to => OctocatalogDiff::Catalog }
+
+          # Analyze the results from parallel run.
+          catalog_tasks.each do |x|
+            # The `parallel_catalog_obj` is a OctocatalogDiff::Util::Parallel::Result. Get the first element from
+            # the parallel_catalogs output.
+            parallel_catalog_obj = parallel_catalogs.shift
+
+            # Add the result to the 'result' hash
+            add_parallel_result(result, parallel_catalog_obj, x)
+          end
+
+          # Things have succeeded if the :to and :from catalogs exist at this point. If not, things have
+          # failed, and an exception should be thrown.
+          raise CatalogError, 'One or more catalogs failed to compile.' unless result.key?(:to) && result.key?(:from)
+          result
+        end
+
+        # Get catalog compilation tasks.
+        # @return [Array<[key, task]>] Catalog tasks
+        def build_catalog_tasks
+          [:from, :to].map do |key|
+            # These are arguments to OctocatalogDiff::Util::Parallel::Task. In most cases the arguments
+            # of OctocatalogDiff::Util::Parallel::Task are taken directly from options, but there are
+            # some defaults or otherwise-named options that must be set here.
+            args = @options.merge(
+              tag: key.to_s,
+              branch: @options["#{key}_env".to_sym],
+              bootstrapped_dir: @options["bootstrapped_#{key}_dir".to_sym],
+              basedir: @options[:basedir],
+              compare_file_text: @options.fetch(:compare_file_text, true),
+              retry_failed_catalog: @options.fetch(:retry_failed_catalog, 0),
+              parser: @options["parser_#{key}".to_sym]
+            )
+
+            # If any options are in the form of 'to_SOMETHING' or 'from_SOMETHING', this sets the option to
+            # 'SOMETHING' for the catalog if it matches this key. For example, when compiling the 'to' catalog
+            # when an option of :to_some_arg => 'foo', this sets :some_arg => foo, and deletes :to_some_arg and
+            # :from_some_arg.
+            @options.keys.select { |x| x.to_s =~ /^(to|from)_/ }.each do |opt_key|
+              args[opt_key.to_s.sub(/^(to|from)_/, '').to_sym] = @options[opt_key] if opt_key.to_s.start_with?(key.to_s)
+              args.delete(opt_key)
+            end
+
+            # The task is a OctocatalogDiff::Util::Parallel::Task object that contains the method to execute,
+            # validator method, text description, and arguments to provide when calling the method.
+            task = OctocatalogDiff::Util::Parallel::Task.new(
+              method: method(:build_catalog),
+              validator: method(:catalog_validator),
+              description: "build_catalog for #{@options["#{key}_env".to_sym]}",
+              args: args
+            )
+
+            # The format of `catalog_tasks` will be a tuple, where the first element is the key
+            # (e.g. :to or :from) and the second element is the OctocatalogDiff::Util::Parallel::Task object.
+            [key, task]
+          end.compact
+        end
+
+        # Given a result from the 'parallel' run and a corresponding (key,task) tuple, add valid
+        # catalogs to the 'result' hash and throw errors for invalid catalogs.
+        # @param result [Hash] Result hash for build_catalog_parallelizer (may be modified)
+        # @param parallel_catalog_obj [OctocatalogDiff::Util::Parallel::Result] Parallel catalog result
+        # @param key_task_tuple [Array<key, task>] Key, task tuple
+        def add_parallel_result(result, parallel_catalog_obj, key_task_tuple)
+          # Expand the tuple into variables
+          key, task = key_task_tuple
+
+          # For reporting purposes, get the branch name.
+          branch = task.args[:branch]
+
+          # Check the result of the parallel run on this object.
+          if parallel_catalog_obj.status.nil?
+            # The compile was killed because another task failed.
+            @logger.warn "Catalog compile for #{branch} was aborted due to another failure"
+
+          elsif parallel_catalog_obj.output.is_a?(OctocatalogDiff::Catalog)
+            # The result is a catalog, but we do not know if it was successfully compiled
+            # until we test the validity.
+            catalog = parallel_catalog_obj.output
+            if catalog.valid?
+              # The catalog was successfully compiled.
+              result[key] = parallel_catalog_obj.output
+            else
+              # The catalog failed, but a catalog object was returned so that better error reporting
+              # can take place. In this error reporting, we will replace 'Error:' with '[Puppet Error]'
+              # and remove the compilation directory (which is a tmpdir) to reveal only the relative
+              # path to the files involved.
+              dir = catalog.compilation_dir || ''
+              dir_regex = Regexp.new(Regexp.escape(dir) + '/environments/production/')
+              error_display = catalog.error_message.split("\n").map do |line|
+                line.sub(/^Error:/, '[Puppet Error]').gsub(dir_regex, '')
+              end.join("\n")
+              raise CatalogError, "Catalog for #{branch} failed to compile due to errors:\n#{error_display}"
+            end
+          else
+            # Something unhandled went wrong, and an exception was thrown. Reveal a generic message.
+            msg = parallel_catalog_obj.exception.message
+            message = "Catalog for '#{key}' (#{branch}) failed to compile with #{parallel_catalog_obj.exception.class}: #{msg}"
+            message += "\n" + parallel_catalog_obj.exception.backtrace.map { |x| "   #{x}" }.join("\n") if @options[:debug]
+            raise CatalogError, message
+          end
+        end
+
+        # Performs the steps necessary to build a catalog.
+        # @param opts [Hash] Options hash
+        # @return [Hash] { :rc => exit code, :catalog => Catalog as JSON string }
+        def build_catalog(opts, logger = @logger)
+          logger.debug("Setting up Puppet catalog build for #{opts[:branch]}")
+          catalog = opts[:object]
+          logger.debug("Catalog for #{opts[:branch]} will be built with #{catalog.builder}")
+          time_start = Time.now
+          catalog.build(logger)
+          time_it_took = Time.now - time_start
+          retries_str = " retries = #{catalog.retries}" if catalog.retries.is_a?(Fixnum)
+          time_str = "in #{time_it_took} seconds#{retries_str}"
+          status_str = catalog.valid? ? 'successfully built' : 'failed'
+          logger.debug "Catalog for #{opts[:branch]} #{status_str} with #{catalog.builder} #{time_str}"
+          catalog
+        end
+
+        # Validate a catalog in the parallel execution
+        # @param catalog [OctocatalogDiff::Catalog] Catalog object
+        # @return [Boolean] true if catalog is valid, false otherwise
+        def catalog_validator(catalog = nil, _logger = @logger)
+          return false unless catalog.is_a?(OctocatalogDiff::Catalog)
+          catalog.valid?
+        end
+      end
+    end
+  end
+end