octocatalog-diff 1.3.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 67cc184b1ece4c4982d29bd714a674cc118f649d
-  data.tar.gz: b2f18b550db7705f998cc2facab1525ded036399
+  metadata.gz: 5f0b3862bc04c59a30e41e90cf689f508cb68dd6
+  data.tar.gz: 4e67157b3a0ba173d2d1099d1e0aec79027a8f3b
 SHA512:
-  metadata.gz: a22fdaab74c77e8e195bb7de464330f441444da4abe76729d98ea74a2c2958cc299ec030644f7804e99972ae162272b191062857d19fe47409877a05222ce2f3
-  data.tar.gz: 08f100f0cec01c5821154341ee4bf4c3abe2d2b1350a41391716a605b966cf11948179a3f0b5452ddfd8782a937841bc6e9d53962e6178f3559027ed2239ce4d
+  metadata.gz: aa208efdde5e60a3bfdd1ced747632bc00d0fec7f916fed1765f9c8e7fe8de99160a7ef673e694fa043a90713af3df2c6eec620f9e792d2c0332864d04baf694
+  data.tar.gz: 228113ef4f77899dfe3185b839b9d67f41db4ae9db0901f9aa7aa3307d0c67d8983982118c8e2dff433f37f428cccc8c1ca755f83c41e9a8a9965a6027bd4952

data/.version

@@ -1 +1 @@
-1.3.0
+1.4.0

data/doc/CHANGELOG.md

@@ -8,6 +8,15 @@
 </tr>
 </thead><tbody>
 <tr valign=top>
+<td>1.4.0</td>
+<td>2017-08-03</td>
+<td>
+<li><a href="https://github.com/github/octocatalog-diff/pull/135">#135</a>: (Enhancement) Puppet 5 compatibility</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/140">#140</a>: (Internal) Prefix tmpdirs with ocd-</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/138">#138</a>: (Internal) Refactor catalog class with proper inheritance</li>
+</td>
+</tr>
+<tr valign=top>
 <td>1.3.0</td>
 <td>2017-06-09</td>
 <td>

data/doc/advanced-catalog-validation.md

@@ -4,12 +4,14 @@
 
 Catalog validation features include:
 
-- Validate references: Ensure resources targeted by `before`, `notify`, `require`, and/or `subscribe` exist in the catalog
+- Validate references: Ensure resources targeted by `before`, `notify`, `require`, and/or `subscribe` exist in the catalog for Puppet 4 and below.
 
 ## Validate references
 
 `octocatalog-diff` includes the ability to validate references by ensuring resources targeted by `before`, `notify`, `require`, and/or `subscribe` parameters also exist in the catalog.
 
+Puppet 5 already has this checking built in, so the `--validate-references` option described in this section will be ignored if Puppet 5 is being used. The same exception (`OctocatalogDiff::Errors::CatalogError`) is raised for a missing reference, whether the problem was detected by octocatalog-diff or Puppet 5.
+
 Consider the following Puppet code:
 
 ```

data/doc/dev/api/v1/calls/catalog-diff.md

@@ -194,7 +194,7 @@ The following exceptions may occur during the compilation of a catalog within th
 
 - `OctocatalogDiff::Errors::CatalogError`
 
-  Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false.
+  Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false. It's also possible that the catalog contained broken references -- see [Catalog validation](/doc/advanced-catalog-validation.md).
 
 - `OctocatalogDiff::Errors::GitCheckoutError`
 
@@ -203,7 +203,3 @@ The following exceptions may occur during the compilation of a catalog within th
 - `OctocatalogDiff::Errors::PuppetVersionError`
 
   The version of Puppet could not be determined, generally because the Puppet binary was not found, or does not respond as expected to `puppet version`.
-
-- `OctocatalogDiff::Errors::ReferenceValidationError`
-
-  See [Catalog validation](/doc/advanced-catalog-validation.md).

data/doc/dev/api/v1/calls/catalog.md

@@ -100,7 +100,7 @@ The following exceptions may occur during the compilation of a catalog:
 
 - `OctocatalogDiff::Errors::CatalogError`
 
-  Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false.
+  Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false. It's also possible that the catalog contained broken references -- see [Catalog validation](/doc/advanced-catalog-validation.md).
 
 - `OctocatalogDiff::Errors::GitCheckoutError`
 
@@ -109,7 +109,3 @@ The following exceptions may occur during the compilation of a catalog:
 - `OctocatalogDiff::Errors::PuppetVersionError`
 
   The version of Puppet could not be determined, generally because the Puppet binary was not found, or does not respond as expected to `puppet version`.
-
-- `OctocatalogDiff::Errors::ReferenceValidationError`
-
-  See [Catalog validation](/doc/advanced-catalog-validation.md).

data/lib/octocatalog-diff/catalog-util/builddir.rb

@@ -38,7 +38,7 @@ module OctocatalogDiff
       # @param options [Hash] Options for class; see above description
       def initialize(options = {}, logger = nil)
         @options = options.dup
-        @tempdir = Dir.mktmpdir
+        @tempdir = Dir.mktmpdir('ocd-builddir-')
         at_exit { FileUtils.rm_rf(@tempdir) if File.directory?(@tempdir) }
 
         @factdir = nil

data/lib/octocatalog-diff/catalog.rb

@@ -12,14 +12,42 @@ require_relative 'catalog-util/fileresources'
 require_relative 'errors'
 
 module OctocatalogDiff
-  # This class represents a catalog. Generation of the catalog is handled via one of the
+  # Basic methods for interacting with a catalog. Generation of the catalog is handled via one of the
   # supported backends listed above as 'require_relative'. Usually, the 'computed' backend
   # will build the catalog from the Puppet command.
   class Catalog
-    # Readable
-    attr_reader :built, :catalog, :catalog_json
+    attr_accessor :node
+    attr_reader :built, :catalog, :catalog_json, :options
 
     # Constructor
+    def initialize(options = {})
+      unless options.is_a?(Hash)
+        raise ArgumentError, "#{self.class}.initialize requires hash argument, not #{options.class}"
+      end
+      @options = options
+
+      # Basic settings
+      @node = options[:node]
+      @error_message = nil
+      @catalog = nil
+      @catalog_json = nil
+      @retries = nil
+
+      # The compilation directory can be overridden, e.g. when testing
+      @override_compilation_dir = options[:compilation_dir]
+
+      # Keep track of whether references have been validated yet. Allow this to be fudged for when we do
+      # not desire reference validation to happen (e.g., for the "from" catalog that is otherwise valid).
+      @references_validated = options[:references_validated] || false
+
+      # Keep track of whether file resources have been converted.
+      @file_resources_converted = false
+
+      # Keep track of whether it's built yet
+      @built = false
+    end
+
+    # Guess the backend from the input and return the appropriate catalog object.
     # @param :backend [Symbol] If set, this will force a backend
     # @param :json [String] JSON catalog content (will avoid running Puppet to compile catalog)
     # @param :puppetdb [Object] If set, pull the catalog from PuppetDB rather than building
@@ -30,18 +58,25 @@ module OctocatalogDiff
     # @param :pass_env_vars [Array<String>] OPTIONAL: Additional environment vars to pass
     # @param :convert_file_resources [Boolean] OPTIONAL: Convert file resource source to content
     # @param :storeconfigs [Boolean] OPTIONAL: Pass the '-s' flag, for puppetdb (storeconfigs) integration
-    def initialize(options = {})
-      @options = options
-
-      # Call appropriate backend for catalog generation
-      @catalog_obj = backend(options)
+    # @return [OctocatalogDiff::Catalog::<?>] Catalog object from guessed backend
+    def self.create(options = {})
+      # Hard-coded backend
+      if options[:backend]
+        return OctocatalogDiff::Catalog::JSON.new(options) if options[:backend] == :json
+        return OctocatalogDiff::Catalog::PuppetDB.new(options) if options[:backend] == :puppetdb
+        return OctocatalogDiff::Catalog::PuppetMaster.new(options) if options[:backend] == :puppetmaster
+        return OctocatalogDiff::Catalog::Computed.new(options) if options[:backend] == :computed
+        return OctocatalogDiff::Catalog::Noop.new(options) if options[:backend] == :noop
+        raise ArgumentError, "Unknown backend :#{options[:backend]}"
+      end
 
-      # The catalog is not built yet, except if the backend has no build method
-      @built = false
-      build unless @catalog_obj.respond_to?(:build)
+      # Determine backend based on arguments
+      return OctocatalogDiff::Catalog::JSON.new(options) if options[:json]
+      return OctocatalogDiff::Catalog::PuppetDB.new(options) if options[:puppetdb]
+      return OctocatalogDiff::Catalog::PuppetMaster.new(options) if options[:puppet_master]
 
-      # The compilation directory can be overridden, e.g. when testing
-      @override_compilation_dir = nil
+      # Default is to build catalog ourselves
+      OctocatalogDiff::Catalog::Computed.new(options)
     end
 
     # Build catalog - this method needs to be called to build the catalog. It is separate due to
@@ -49,43 +84,42 @@ module OctocatalogDiff
     # object so it cannot be part of any object that is passed around.
     # @param logger [Logger] Logger object, initialized to a default throwaway value
     def build(logger = Logger.new(StringIO.new))
-      # Only build once
+      # If already built, don't build again
       return if @built
       @built = true
 
-      # Call catalog's build method.
-      if @catalog_obj.respond_to?(:build)
-        logger.debug "Calling build for object #{@catalog_obj.class}"
-        @catalog_obj.build(logger)
-      end
-
-      # These methods must exist in all backends
-      @catalog = @catalog_obj.catalog
-      @catalog_json = @catalog_obj.catalog_json
-      @error_message = @catalog_obj.error_message
-
       # The resource hash is computed the first time it's needed. For now initialize it as nil.
       @resource_hash = nil
 
+      # Invoke the backend's build method, if there is one. There's a stub below in case there's not.
+      logger.debug "Calling build for object #{self.class}"
+      build_catalog(logger)
+
       # Perform post-generation processing of the catalog
       return unless valid?
-      unless @catalog_obj.respond_to?(:convert_file_resources) && @catalog_obj.convert_file_resources == false
-        if @options.fetch(:compare_file_text, false)
-          OctocatalogDiff::CatalogUtil::FileResources.convert_file_resources(self, environment)
-        end
-      end
+
+      validate_references
+      return unless valid?
+
+      convert_file_resources if @options[:compare_file_text]
+
+      true
+    end
+
+    # Stub method if the backend does not contain a build method.
+    def build_catalog(_logger)
     end
 
     # Compilation environment
     # @return [String] Compilation environment (if set), else 'production' by default
     def environment
-      @catalog_obj.respond_to?(:environment) ? @catalog_obj.environment : 'production'
+      @environment ||= 'production'
     end
 
     # For logging we may wish to know the backend being used
     # @return [String] Class of backend used
     def builder
-      @catalog_obj.class.to_s
+      self.class.to_s
     end
 
     # Set the catalog JSON
@@ -98,8 +132,7 @@ module OctocatalogDiff
     # This retrieves the compilation directory from the catalog, or otherwise the passed-in directory.
     # @return [String] Compilation directory
     def compilation_dir
-      return @override_compilation_dir if @override_compilation_dir
-      @catalog_obj.respond_to?(:compilation_dir) ? @catalog_obj.compilation_dir : @options[:basedir]
+      @override_compilation_dir || @options[:basedir]
     end
 
     # The compilation directory can be overridden, e.g. during testing.
@@ -108,16 +141,16 @@ module OctocatalogDiff
       @override_compilation_dir = dir
     end
 
-    # Determine whether the underlying catalog object supports :compare_file_text
-    # @return [Boolean] Whether underlying catalog object supports :compare_file_text
-    def convert_file_resources
-      return true unless @catalog_obj.respond_to?(:convert_file_resources)
-      @catalog_obj.convert_file_resources
+    # Stub method for "convert_file_resources" -- returns false because if the underlying class does
+    # not implement this method, it's not supported.
+    def convert_file_resources(_dry_run = false)
+      false
     end
 
     # Retrieve the error message.
     # @return [String] Error message (maximum 20,000 characters) - nil if no error.
     def error_message
+      build
       return nil if @error_message.nil? || !@error_message.is_a?(String)
       @error_message[0, 20_000]
     end
@@ -133,12 +166,11 @@ module OctocatalogDiff
       @resource_hash = nil
     end
 
-    # This retrieves the version of Puppet used to compile a catalog. If the underlying catalog was not
-    # compiled by running Puppet (e.g., it was read in from JSON or puppetdb), then this returns the
-    # puppet version optionally passed in to the constructor. This can also be nil.
+    # Stub method to return the puppet version if the back end doesn't support this.
     # @return [String] Puppet version
     def puppet_version
-      @catalog_obj.respond_to?(:puppet_version) ? @catalog_obj.puppet_version : @options[:puppet_version]
+      build
+      @options[:puppet_version]
     end
 
     # This allows retrieving a resource by type and title. This is intended for use when a O(1) lookup is required.
@@ -147,6 +179,7 @@ module OctocatalogDiff
     # @return [Hash] Resource item
     def resource(opts = {})
       raise ArgumentError, ':type and :title are required' unless opts[:type] && opts[:title]
+      build
       build_resource_hash if @resource_hash.nil?
       return nil unless @resource_hash[opts[:type]].is_a?(Hash)
       @resource_hash[opts[:type]][opts[:title]]
@@ -155,6 +188,7 @@ module OctocatalogDiff
     # This is a compatibility layer for the resources, which are in a different place in Puppet 3.x and Puppet 4.x
     # @return [Array] Resource array
     def resources
+      build
       raise OctocatalogDiff::Errors::CatalogError, 'Catalog does not appear to have been built' if !valid? && error_message.nil?
       raise OctocatalogDiff::Errors::CatalogError, error_message unless valid?
       return @catalog['data']['resources'] if @catalog['data'].is_a?(Hash) && @catalog['data']['resources'].is_a?(Array)
@@ -165,16 +199,17 @@ module OctocatalogDiff
       # :nocov:
     end
 
-    # This retrieves the number of retries necessary to compile the catalog. If the underlying catalog
+    # Stub method of the the number of retries necessary to compile the catalog. If the underlying catalog
     # generation backend does not support retries, nil is returned.
     # @return [Integer] Retry count
     def retries
-      @retries = @catalog_obj.respond_to?(:retries) ? @catalog_obj.retries : nil
+      nil
     end
 
     # Determine if the catalog build was successful.
     # @return [Boolean] Whether the catalog is valid
     def valid?
+      build
       !@catalog.nil?
     end
 
@@ -182,11 +217,19 @@ module OctocatalogDiff
     # Raise a OctocatalogDiff::Errors::ReferenceValidationError for any found to be missing.
     # Uses @options[:validate_references] to influence which references are checked.
     def validate_references
+      # If we've already done the validation, don't do it again
+      return if @references_validated
+      @references_validated = true
+
       # Skip out early if no reference validation has been requested.
       unless @options[:validate_references].is_a?(Array) && @options[:validate_references].any?
         return
       end
 
+      # Puppet 5 has reference validation built-in and enabled, so there won't even be a valid catalog if
+      # there were invalid references. It's pointless to perform validation of our own.
+      return if puppet_version && puppet_version >= '5.0.0'
+
       # Iterate over all the resources and check each one that has one of the attributes being checked.
       # Keep track of all references that are missing for ultimate inclusion in the error message.
       missing = []
@@ -204,7 +247,7 @@ module OctocatalogDiff
       # At this point there is at least one broken/missing reference. Format an error message and raise.
       errors = format_missing_references(missing)
       plural = errors =~ /;/ ? 's' : ''
-      raise OctocatalogDiff::Errors::ReferenceValidationError, "Catalog has broken reference#{plural}: #{errors}"
+      self.error_message = "Catalog has broken reference#{plural}: #{errors}"
     end
 
     private
@@ -265,29 +308,6 @@ module OctocatalogDiff
       end
     end
 
-    # Private method: Choose backend based on passed-in options
-    # @param options [Hash] Options passed into constructor
-    # @return [Object] OctocatalogDiff::Catalog::<whatever> object
-    def backend(options)
-      # Hard-coded backend
-      if options[:backend]
-        return OctocatalogDiff::Catalog::JSON.new(options) if options[:backend] == :json
-        return OctocatalogDiff::Catalog::PuppetDB.new(options) if options[:backend] == :puppetdb
-        return OctocatalogDiff::Catalog::PuppetMaster.new(options) if options[:backend] == :puppetmaster
-        return OctocatalogDiff::Catalog::Computed.new(options) if options[:backend] == :computed
-        return OctocatalogDiff::Catalog::Noop.new(options) if options[:backend] == :noop
-        raise ArgumentError, "Unknown backend :#{options[:backend]}"
-      end
-
-      # Determine backend based on arguments
-      return OctocatalogDiff::Catalog::JSON.new(options) if options[:json]
-      return OctocatalogDiff::Catalog::PuppetDB.new(options) if options[:puppetdb]
-      return OctocatalogDiff::Catalog::PuppetMaster.new(options) if options[:puppet_master]
-
-      # Default is to build catalog ourselves
-      OctocatalogDiff::Catalog::Computed.new(options)
-    end
-
     # Private method: Build the resource hash to be used used for O(1) lookups by type and title.
     # This method is called the first time the resource hash is accessed.
     def build_resource_hash

data/lib/octocatalog-diff/catalog/computed.rb

@@ -4,6 +4,7 @@ require 'fileutils'
 require 'json'
 require 'stringio'
 
+require_relative '../catalog'
 require_relative '../catalog-util/bootstrap'
 require_relative '../catalog-util/builddir'
 require_relative '../catalog-util/command'
@@ -15,9 +16,7 @@ module OctocatalogDiff
   class Catalog
     # Represents a Puppet catalog that is computed (via `puppet master --compile ...`)
     # By instantiating this class, the catalog is computed.
-    class Computed
-      attr_reader :node, :error_message, :catalog, :catalog_json, :retries
-
+    class Computed < OctocatalogDiff::Catalog
       # Constructor
       # @param :node [String] REQUIRED: Node name
       # @param :basedir [String] Directory in which to compile the catalog
@@ -28,14 +27,10 @@ module OctocatalogDiff
       # @param :puppet_version [String] Puppet version (optional; if not supplied, it is calculated)
       # @param :puppet_command [String] Full command to run Puppet (optional; if not supplied, it is calculated)
       def initialize(options)
-        raise ArgumentError, 'Usage: OctocatalogDiff::Catalog::Computed.initialize(options_hash)' unless options.is_a?(Hash)
-        raise ArgumentError, 'Node name must be passed to OctocatalogDiff::Catalog::Computed' unless options[:node].is_a?(String)
+        super
 
-        # Standard readable variables
-        @node = options[:node]
-        @error_message = nil
-        @catalog = nil
-        @catalog_json = nil
+        raise ArgumentError, 'Node name must be passed to OctocatalogDiff::Catalog::Computed' unless options[:node].is_a?(String)
+        raise ArgumentError, 'Branch is undefined' unless options[:branch]
 
         # Additional class variables
         @pass_env_vars = options.fetch(:pass_env_vars, [])
@@ -44,31 +39,15 @@ module OctocatalogDiff
         @puppet_binary = options[:puppet_binary]
         @puppet_version = options[:puppet_version]
         @puppet_command = options[:puppet_command]
-        @retries = nil
         @builddir = nil
         @facts_terminus = options.fetch(:facts_terminus, 'yaml')
-
-        # Pass through the input for other access
-        @opts = options
-        raise ArgumentError, 'Branch is undefined' unless @opts[:branch]
-      end
-
-      # Actually build the catalog (populate @error_message, @catalog, @catalog_json)
-      def build(logger = Logger.new(StringIO.new))
-        if @facts_terminus != 'facter'
-          facts_obj = OctocatalogDiff::CatalogUtil::Facts.new(@opts, logger)
-          logger.debug "Start retrieving facts for #{@node} from #{self.class}"
-          @opts[:facts] = facts_obj.facts
-          logger.debug "Success retrieving facts for #{@node} from #{self.class}"
-        end
-        build_catalog(logger)
       end
 
       # Get the Puppet version
       # @return [String] Puppet version
       def puppet_version
         raise ArgumentError, '"puppet_binary" was not passed to OctocatalogDiff::Catalog::Computed' unless @puppet_binary
-        @puppet_version ||= OctocatalogDiff::Util::PuppetVersion.puppet_version(@puppet_binary, @opts)
+        @puppet_version ||= OctocatalogDiff::Util::PuppetVersion.puppet_version(@puppet_binary, @options)
       end
 
       # Compilation directory
@@ -80,7 +59,14 @@ module OctocatalogDiff
 
       # Environment used to compile catalog
       def environment
-        @opts.fetch(:environment, 'production')
+        @options.fetch(:environment, 'production')
+      end
+
+      # Convert file resources source => "puppet:///..." to content => "actual content of file".
+      def convert_file_resources(dry_run = false)
+        return @options.key?(:basedir) if dry_run
+        return false unless @options[:basedir]
+        OctocatalogDiff::CatalogUtil::FileResources.convert_file_resources(self, environment)
       end
 
       private
@@ -105,29 +91,29 @@ module OctocatalogDiff
         return if @builddir
 
         # Fill options for creating and populating the temporary directory
-        tmphash = @opts.dup
+        tmphash = @options.dup
 
         # Bootstrap directory if needed
-        if !@opts[:bootstrapped_dir].nil?
-          raise Errno::ENOENT, "Invalid dir #{@opts[:bootstrapped_dir]}" unless File.directory?(@opts[:bootstrapped_dir])
-          tmphash[:basedir] = @opts[:bootstrapped_dir]
-        elsif @opts[:branch] == '.'
-          if @opts[:bootstrap_current]
-            tmphash[:basedir] = Dir.mktmpdir
+        if !@options[:bootstrapped_dir].nil?
+          raise Errno::ENOENT, "Invalid dir #{@options[:bootstrapped_dir]}" unless File.directory?(@options[:bootstrapped_dir])
+          tmphash[:basedir] = @options[:bootstrapped_dir]
+        elsif @options[:branch] == '.'
+          if @options[:bootstrap_current]
+            tmphash[:basedir] =  Dir.mktmpdir('ocd-bootstrap-basedir-')
             at_exit { cleanup_checkout_dir(tmphash[:basedir], logger) }
 
-            FileUtils.cp_r File.join(@opts[:basedir], '.'), tmphash[:basedir]
+            FileUtils.cp_r File.join(@options[:basedir], '.'), tmphash[:basedir]
 
-            o = @opts.reject { |k, _v| k == :branch }.merge(path: tmphash[:basedir])
+            o = @options.reject { |k, _v| k == :branch }.merge(path: tmphash[:basedir])
             OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory(o, logger)
           else
-            tmphash[:basedir] = @opts[:basedir]
+            tmphash[:basedir] = @options[:basedir]
           end
         else
-          checkout_dir = Dir.mktmpdir
+          checkout_dir = Dir.mktmpdir('ocd-bootstrap-checkout-')
           at_exit { cleanup_checkout_dir(checkout_dir, logger) }
           tmphash[:basedir] = checkout_dir
-          OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory(@opts.merge(path: checkout_dir), logger)
+          OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory(@options.merge(path: checkout_dir), logger)
         end
 
         # Create and populate the temporary directory
@@ -136,8 +122,14 @@ module OctocatalogDiff
 
       # Private method: Build catalog by running Puppet
       # @param logger [Logger] Logger object
-      def build_catalog(logger = nil)
-        return nil unless @catalog.nil? && @error_message.nil?
+      def build_catalog(logger)
+        if @facts_terminus != 'facter'
+          facts_obj = OctocatalogDiff::CatalogUtil::Facts.new(@options, logger)
+          logger.debug "Start retrieving facts for #{@node} from #{self.class}"
+          @options[:facts] = facts_obj.facts
+          logger.debug "Success retrieving facts for #{@node} from #{self.class}"
+        end
+
         bootstrap(logger)
         result = run_puppet(logger)
         @retries = result[:retries]
@@ -168,10 +160,10 @@ module OctocatalogDiff
         @puppet_command_obj ||= begin
           raise ArgumentError, '"puppet_binary" was not passed to OctocatalogDiff::Catalog::Computed' unless @puppet_binary
 
-          command_opts = @opts.merge(
+          command_opts = @options.merge(
             node: @node,
             compilation_dir: @builddir.tempdir,
-            parser: @opts.fetch(:parser, :default),
+            parser: @options.fetch(:parser, :default),
             puppet_binary: @puppet_binary,
             fact_file: @builddir.fact_file,
             dir: @builddir.tempdir,
@@ -201,7 +193,7 @@ module OctocatalogDiff
         # Set up the ScriptRunner
         scriptrunner = OctocatalogDiff::Util::ScriptRunner.new(
           default_script: 'puppet/puppet.sh',
-          override_script_path: @opts[:override_script_path]
+          override_script_path: @options[:override_script_path]
         )
 
         begin

data/lib/octocatalog-diff/catalog/json.rb

@@ -1,21 +1,25 @@
 # frozen_string_literal: true
 
+require_relative '../catalog'
+require_relative '../catalog-util/fileresources'
+
 require 'json'
 
 module OctocatalogDiff
   class Catalog
     # Represents a Puppet catalog that is read in directly from a JSON file.
-    class JSON
-      attr_accessor :node
-      attr_reader :error_message, :catalog, :catalog_json
-
+    class JSON < OctocatalogDiff::Catalog
       # Constructor
       # @param :json [String] REQUIRED: Content of catalog, will be parsed as JSON
       # @param :node [String] Node name (if not supplied, will be determined from catalog)
       def initialize(options)
-        raise ArgumentError, 'Usage: OctocatalogDiff::Catalog::JSON.initialize(options_hash)' unless options.is_a?(Hash)
-        raise ArgumentError, "Must supply :json as string in options: #{options[:json].class}" unless options[:json].is_a?(String)
-        @catalog_json = options[:json]
+        super
+
+        unless options[:json].is_a?(String)
+          raise ArgumentError, "Must supply :json as string in options: #{options[:json].class}"
+        end
+
+        @catalog_json = options.fetch(:json)
         begin
           @catalog = ::JSON.parse(@catalog_json)
           @error_message = nil
@@ -27,6 +31,13 @@ module OctocatalogDiff
           @catalog_json = nil
         end
       end
+
+      # Convert file resources source => "puppet:///..." to content => "actual content of file".
+      def convert_file_resources(dry_run = false)
+        return @options.key?(:basedir) if dry_run
+        return false unless @options[:basedir]
+        OctocatalogDiff::CatalogUtil::FileResources.convert_file_resources(self, environment)
+      end
     end
   end
 end

data/lib/octocatalog-diff/catalog/noop.rb

@@ -1,16 +1,16 @@
 # frozen_string_literal: true
 
+require_relative '../catalog'
+
 require 'json'
 
 module OctocatalogDiff
   class Catalog
     # Represents a null Puppet catalog.
-    class Noop
-      attr_accessor :node
-      attr_reader :error_message, :catalog, :catalog_json
-
-      # Constructor
+    class Noop < OctocatalogDiff::Catalog
       def initialize(options)
+        super
+
         @catalog_json = '{"resources":[]}'
         @catalog = { 'resources' => [] }
         @error_message = nil

data/lib/octocatalog-diff/catalog/puppetdb.rb

@@ -3,43 +3,30 @@
 require 'json'
 require 'stringio'
 
+require_relative '../catalog'
 require_relative '../errors'
 require_relative '../puppetdb'
 
 module OctocatalogDiff
   class Catalog
     # Represents a Puppet catalog that is read from PuppetDB.
-    class PuppetDB
-      attr_accessor :node
-      attr_reader :error_message, :catalog, :catalog_json, :retries, :convert_file_resources
-
+    class PuppetDB < OctocatalogDiff::Catalog
       # Constructor - See OctocatalogDiff::PuppetDB for additional parameters
       # @param :node [String] Node name
       # @param :retry [Integer] Number of retries, if fetch fails
       def initialize(options)
-        raise ArgumentError, 'Hash of options must be passed to OctocatalogDiff::Catalog::PuppetDB' unless options.is_a?(Hash)
-        raise ArgumentError, 'node must be a non-empty string' unless options[:node].is_a?(String) && options[:node] != ''
-        @node = options[:node]
-        @catalog = nil
-        @error_message = nil
-        @retries = nil
-
-        # Cannot convert file resources from this type of catalog
-        @convert_file_resources = false
+        super
 
-        # Save options
-        @options = options
-      end
-
-      def build(logger = Logger.new(StringIO.new))
-        fetch_catalog(logger)
+        unless @options[:node].is_a?(String) && @options[:node] != ''
+          raise ArgumentError, 'node must be a non-empty string'
+        end
       end
 
       private
 
       # Private method: Get catalog from PuppetDB. Sets @catalog / @catalog_json or @error_message
       # @param logger [Logger object] Logger object
-      def fetch_catalog(logger)
+      def build_catalog(logger)
         # Use OctocatalogDiff::PuppetDB to interact with puppetdb
         puppetdb_obj = OctocatalogDiff::PuppetDB.new(@options)
 

data/lib/octocatalog-diff/catalog/puppetmaster.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require_relative '../catalog'
 require_relative '../catalog-util/facts'
 require_relative '../external/pson/pure'
 require_relative '../util/httparty'
@@ -11,10 +12,7 @@ require 'stringio'
 module OctocatalogDiff
   class Catalog
     # Represents a Puppet catalog that is obtained by contacting the Puppet Master.
-    class PuppetMaster
-      attr_accessor :node
-      attr_reader :error_message, :catalog, :catalog_json, :convert_file_resources, :options, :retries
-
+    class PuppetMaster < OctocatalogDiff::Catalog
       # Defaults
       DEFAULT_PUPPET_PORT_NUMBER = 8140
       DEFAULT_PUPPET_SERVER_API = 3
@@ -34,32 +32,29 @@ module OctocatalogDiff
       # @param :puppet_master_ssl_client_auth [Boolean] Override the client-auth that is guessed from parameters
       # @param :timeout [Integer] Connection timeout for Puppet master (default=PUPPET_MASTER_TIMEOUT seconds)
       def initialize(options)
-        raise ArgumentError, 'Hash of options must be passed to OctocatalogDiff::Catalog::PuppetMaster' unless options.is_a?(Hash)
-        raise ArgumentError, 'node must be a non-empty string' unless options[:node].is_a?(String) && options[:node] != ''
-        unless options[:branch].is_a?(String) && options[:branch] != ''
+        super
+
+        unless @options[:node].is_a?(String) && @options[:node] != ''
+          raise ArgumentError, 'node must be a non-empty string'
+        end
+
+        unless @options[:branch].is_a?(String) && @options[:branch] != ''
           raise ArgumentError, 'Environment must be a non-empty string'
         end
-        unless options[:puppet_master].is_a?(String) && options[:puppet_master] != ''
+
+        unless @options[:puppet_master].is_a?(String) && @options[:puppet_master] != ''
           raise ArgumentError, 'Puppet Master must be a non-empty string'
         end
 
-        @node = options[:node]
-        @catalog = nil
-        @error_message = nil
-        @retries = nil
         @timeout = options.fetch(:puppet_master_timeout, options.fetch(:timeout, PUPPET_MASTER_TIMEOUT))
         @retry_failed_catalog = options.fetch(:retry_failed_catalog, 0)
-
-        # Cannot convert file resources from this type of catalog right now.
-        # FIXME: This is possible with additional API calls but is current unimplemented.
-        @convert_file_resources = false
-
-        options[:puppet_master] += ":#{DEFAULT_PUPPET_PORT_NUMBER}" unless options[:puppet_master] =~ /\:\d+$/
-        @options = options
+        @options[:puppet_master] += ":#{DEFAULT_PUPPET_PORT_NUMBER}" unless @options[:puppet_master] =~ /\:\d+$/
       end
 
+      private
+
       # Build method
-      def build(logger = Logger.new(StringIO.new))
+      def build_catalog(logger = Logger.new(StringIO.new))
         facts_obj = OctocatalogDiff::CatalogUtil::Facts.new(@options, logger)
         logger.debug "Start retrieving facts for #{@node} from #{self.class}"
         @facts = facts_obj.facts
@@ -67,8 +62,6 @@ module OctocatalogDiff
         fetch_catalog(logger)
       end
 
-      private
-
       # Returns a hash of parameters for each supported version of the Puppet Server Catalog API.
       # @return [Hash] Hash of parameters
       #

data/lib/octocatalog-diff/errors.rb

@@ -11,7 +11,6 @@ module OctocatalogDiff
     class BootstrapError < RuntimeError; end
     class CatalogError < RuntimeError; end
     class PuppetVersionError < RuntimeError; end
-    class ReferenceValidationError < RuntimeError; end
     class GitCheckoutError < RuntimeError; end
 
     # Error classes for retrieving facts

data/lib/octocatalog-diff/util/catalogs.rb

@@ -63,15 +63,15 @@ module OctocatalogDiff
         # build the catalog.
         result = {}
         catalog_tasks.each do |x|
-          result[x[0]] = OctocatalogDiff::Catalog.new(x[1].args)
+          result[x[0]] = OctocatalogDiff::Catalog.create(x[1].args)
           @logger.debug "Initialized #{result[x[0]].builder} for #{x[0]}-catalog"
         end
 
         # Disable --compare-file-text if either (or both) of the chosen backends do not support it
         if @options.fetch(:compare_file_text, false)
-          result.each do |_key, val|
-            next unless val.convert_file_resources == false
-            @logger.debug "Disabling --compare-file-text; not supported by #{val.builder}"
+          result.each do |_key, builder_obj|
+            next if builder_obj.convert_file_resources(true)
+            @logger.debug "Disabling --compare-file-text; not supported by #{builder_obj.builder}"
             @options[:compare_file_text] = false
             catalog_tasks.map! do |x|
               x[1].args[:compare_file_text] = false
@@ -147,6 +147,7 @@ module OctocatalogDiff
             retry_failed_catalog: @options.fetch(:retry_failed_catalog, 0),
             parser: @options["parser_#{key}".to_sym]
           )
+          args[:basedir] ||= args[:bootstrapped_dir]
 
           # If any options are in the form of 'to_SOMETHING' or 'from_SOMETHING', this sets the option to
           # 'SOMETHING' for the catalog if it matches this key. For example, when compiling the 'to' catalog
@@ -157,6 +158,9 @@ module OctocatalogDiff
             args.delete(opt_key)
           end
 
+          # Skip reference validation in the from-catalog by saying we already performed it.
+          args[:references_validated] = (key == :from)
+
           # The task is a OctocatalogDiff::Util::Parallel::Task object that contains the method to execute,
           # validator method, text description, and arguments to provide when calling the method.
           task = OctocatalogDiff::Util::Parallel::Task.new(
@@ -250,10 +254,9 @@ module OctocatalogDiff
       # @param logger [Logger] Logger object (presently unused)
       # @param args [Hash] Additional arguments set specifically for validator
       # @return [Boolean] Return true if catalog is valid, false otherwise
-      def catalog_validator(catalog = nil, _logger = @logger, args = {})
+      def catalog_validator(catalog = nil, _logger = @logger, _args = {})
         raise ArgumentError, "Expects a catalog, got #{catalog.class}" unless catalog.is_a?(OctocatalogDiff::Catalog)
         raise OctocatalogDiff::Errors::CatalogError, "Catalog failed: #{catalog.error_message}" unless catalog.valid?
-        catalog.validate_references if args[:task] == :to # Raises exception for broken references
         true
       end
     end

data/lib/octocatalog-diff/util/parallel.rb

@@ -107,7 +107,7 @@ module OctocatalogDiff
       # @return [Exception] First exception encountered by a child process; returns nil if no exceptions encountered.
       def self.run_tasks_parallel(result, task_array, logger)
         pidmap = {}
-        ipc_tempdir = Dir.mktmpdir
+        ipc_tempdir = Dir.mktmpdir('ocd-ipc-')
 
         # Child process forking
         task_array.each_with_index do |task, index|

data/lib/octocatalog-diff/util/scriptrunner.rb

@@ -91,7 +91,7 @@ module OctocatalogDiff
       # @return [String] Path to tempfile containing script
       def temp_script(script)
         raise Errno::ENOENT, "Script '#{script}' not found" unless File.file?(script)
-        temp_dir = Dir.mktmpdir
+        temp_dir = Dir.mktmpdir('ocd-scriptrunner-')
         at_exit do
           begin
             FileUtils.remove_entry_secure temp_dir

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: octocatalog-diff
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.4.0
 platform: ruby
 authors:
 - GitHub, Inc.
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-06-09 00:00:00.000000000 Z
+date: 2017-08-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: diffy
@@ -137,32 +137,18 @@ dependencies:
     - - '='
       - !ruby/object:Gem::Version
         version: 0.48.1
-- !ruby/object:Gem::Dependency
-  name: puppetdb-terminus
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 3.2.4
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 3.2.4
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.14.1
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.14.1
 - !ruby/object:Gem::Dependency
@@ -193,6 +179,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 4.10.0
+- !ruby/object:Gem::Dependency
+  name: puppetdb-terminus
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 3.2.4
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 3.2.4
 description: |
   Octocatalog-Diff assists with Puppet development and testing by enabling the user to
   compile 2 Puppet catalogs and compare them. It is possible to compare different