octocatalog-diff 1.5.4 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 0ba4dc520ab7e8ae27324beef80edd6c96f601de
-  data.tar.gz: d927873135a73e5f1b8c178e0cf4a54b9a29aa46
+SHA256:
+  metadata.gz: 90eb139ff8685aa8906b164f1d3bc869efce0a9a8b0711c4f5b575bcd7d9d18f
+  data.tar.gz: 92b5fcd9820cff4e19e02d9748774627a3569d4b1775c321a15bfc8d5fb19bda
 SHA512:
-  metadata.gz: e7a23ff7139158b5092765b190a801fbd670fe2931dde206fad3007fdf03995e3e6f62a881f9b3258d2b91880688664c7f43c947f5a2cfdb9b1c9b36516c94be
-  data.tar.gz: 4eae1cff736c29ced07644eac4cd4af812df56ac6c32afade41cf85ba0771d8136d79bf07783bec6a101cec3e5880ac647fbb83db55fe352b1181db6422288d0
+  metadata.gz: ab5c43c7a9de9d437ce4249b7cbe8b4d07b1cfc5b858deb4b26a2f1021245b49279e02ddc53f412e7d17f58bc27c461921f614650c20055cd11442f8b827e42c
+  data.tar.gz: caa52478aa06d0b51ea00686281d21e808afd1c78330fb17a53492c092836f5000365d2ae1880f0aac5f5577c6ba7d9d27708ecd6f4d6c8f6a0440dd4243b398

data/.version

@@ -1 +1 @@
-1.5.4
+1.6.0

data/README.md

@@ -1,4 +1,4 @@
-# octocatalog-diff
+# octocatalog-diff ![CI](https://github.com/github/octocatalog-diff/workflows/CI/badge.svg)
 
 #### Compile Puppet catalogs from 2 branches, versions, etc., and compare them <img src="/doc/images/octocatolog-diff-logo.png" align="right" height=126 width=240>
 
@@ -81,6 +81,6 @@ If you have a problem or suggestion, please [open an issue](https://github.com/g
 
 It requires 3rd party ruby gems found [here](/vendor/cache). It also includes portions of other open source projects [here](/lib/octocatalog-diff/external/pson), [here](/spec/octocatalog-diff/fixtures/repos/default/modules/stdlib), [here](/spec/octocatalog-diff/support/httparty) and [here](/spec/octocatalog-diff/tests/external/pson). All 3rd party code and required gems are licensed either as MIT or Apache 2.0.
 
-## Authors
+## Authors / Owners
 
-`octocatalog-diff` was designed and authored by [Kevin Paulisse](https://github.com/kpaulisse) and is now maintained, reviewed, and tested by Kevin and the rest of the Site Reliability Engineering team at GitHub.
+`octocatalog-diff` was originally designed and authored by [Kevin Paulisse](https://github.com/kpaulisse). It is now maintained by the Site Reliability Engineering team at GitHub.

data/doc/CHANGELOG.md

@@ -8,6 +8,16 @@
 </tr>
 </thead><tbody>
 
+<tr valign=top>
+<td>1.6.0</td>
+<td>2019-10-31</td>
+<li><a href="https://github.com/github/octocatalog-diff/pull/216">#216</a>: (Enhancement) Hide sensitive parameters</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/204">#204</a>: (Enhancement) Add glob support for modulepath</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/206">#206</a>: (Bug Fix) Fix multi-node list with parallel mode</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/215">#215</a>: (Bug Fix) Add Support for Hashdiff 1.0.0</li>
+</td>
+</tr>
+
 <tr valign=top>
 <td>1.5.4</td>
 <td>2018-12-11</td>

data/doc/limitations.md

@@ -2,23 +2,23 @@
 
 Testing of Puppet catalogs is faster than running the agent, but you need to be careful of the following limitations:
 
-0. Facts are not taken from a live agent run
+1. Facts are not taken from a live agent run
 
-  octocatalog-diff by default uses the facts reported from a node's more recent Puppet run. If you have made changes to custom facts, catalog testing will **NOT** be an adequate test of whether your custom facts worked. (You can still use octocatalog-diff to help predict changes to nodes based on changes to facts, by overriding facts on the command line.)
+    octocatalog-diff by default uses the facts reported from a node's more recent Puppet run. If you have made changes to custom facts, catalog testing will **NOT** be an adequate test of whether your custom facts worked. (You can still use octocatalog-diff to help predict changes to nodes based on changes to facts, by overriding facts on the command line.)
 
-0. Agents handle depenency ordering and implementation details
+1. Agents handle depenency ordering and implementation details
 
-  The catalog defines the state of the system, but it's up to the agent to determine how to bring the system to a point that matches the catalog. The agent is responsible for order of operations and actually making the change.
+    The catalog defines the state of the system, but it's up to the agent to determine how to bring the system to a point that matches the catalog. The agent is responsible for order of operations and actually making the change.
 
-  Two specific situations that catalog testing does **NOT** detect are:
+    Two specific situations that catalog testing does **NOT** detect are:
 
-  - Dependency loops (e.g., you have made A require B, B require C, and C require A).
+    - Dependency loops (e.g., you have made A require B, B require C, and C require A).
 
-  - Operations not supported by the provider. For example, assume that in your current Puppet manifests, you set the size of a file system to 100 GB. You change this in your new branch to 50 GB. octocatalog-diff will dutifully report this change to you. However, the agent will fail to make the change, because it is not possible to shrink a file system from 100 GB to 50 GB.
+    - Operations not supported by the provider. For example, assume that in your current Puppet manifests, you set the size of a file system to 100 GB. You change this in your new branch to 50 GB. octocatalog-diff will dutifully report this change to you. However, the agent will fail to make the change, because it is not possible to shrink a file system from 100 GB to 50 GB.
 
-0. Changes in underlying providers may not be noticed
+1. Changes in underlying providers may not be noticed
 
-  Consider that you are using a Puppet module that creates a file system. The current implementation of that module checks to see if *any* file system is present on the device, and creates a new file system there if no file system was present. You upgrade the module, and the new version checks to see if *the specified* file system is present on the device, and reformats the device with the specified file system (regardless of whether there was no file system or if there was an existing file system of a different type). There would be no catalog changes (hence octocatalog-diff would report nothing) because the catalog simply instructs the agent to create a file system of the specified type at the defined location. However, the actual implementation of those instructions has changed dramatically.
+    Consider that you are using a Puppet module that creates a file system. The current implementation of that module checks to see if *any* file system is present on the device, and creates a new file system there if no file system was present. You upgrade the module, and the new version checks to see if *the specified* file system is present on the device, and reformats the device with the specified file system (regardless of whether there was no file system or if there was an existing file system of a different type). There would be no catalog changes (hence octocatalog-diff would report nothing) because the catalog simply instructs the agent to create a file system of the specified type at the defined location. However, the actual implementation of those instructions has changed dramatically.
 
 In general catalog testing is great for:
 

data/lib/octocatalog-diff/catalog-diff/differ.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'diffy'
+require 'digest'
 require 'hashdiff'
 require 'json'
 require 'set'
@@ -11,6 +12,8 @@ require_relative '../errors'
 require_relative '../util/util'
 require_relative 'filter'
 
+HashDiff = Hashdiff unless defined? HashDiff
+
 module OctocatalogDiff
   module CatalogDiff
     # Calculate the difference between two Puppet catalogs.
@@ -263,7 +266,7 @@ module OctocatalogDiff
 
             # Handle parameters
             if k == 'parameters'
-              cleansed_param = cleanse_parameters_hash(v)
+              cleansed_param = cleanse_parameters_hash(v, resource.fetch('sensitive_parameters', []))
               hsh[k] = cleansed_param unless cleansed_param.nil? || cleansed_param.empty?
             elsif k == 'tags'
               # The order of tags is unimportant. Sort this array to avoid false diffs if order changes.
@@ -456,10 +459,18 @@ module OctocatalogDiff
 
       # Cleanse parameters of filtered attributes.
       # @param parameters_hash [Hash] Hash of parameters
+      # @param sensitive_parameters [Array] Array of sensitive parameters
       # @return [Hash] Cleaned parameters hash (original input hash is not altered)
-      def cleanse_parameters_hash(parameters_hash)
+      def cleanse_parameters_hash(parameters_hash, sensitive_parameters)
         result = parameters_hash.dup
 
+        # hides sensitive params. We still need to know if there's a going to
+        # be a diff, so we hash the value.
+        sensitive_parameters.each do |p|
+          md5 = Digest::MD5.hexdigest Marshal.dump(result[p])
+          result[p] = 'Sensitive [md5sum ' + md5 + ']'
+        end
+
         # 'before' and 'require' handle internal Puppet ordering but do not affect what
         # happens on the target machine. Don't consider these for the purpose of catalog diff.
         result.delete('before')

data/lib/octocatalog-diff/catalog-util/fileresources.rb

@@ -59,7 +59,7 @@ module OctocatalogDiff
           result = []
           Regexp.last_match(1).split(/:/).map(&:strip).each do |path|
             next if path.start_with?('$')
-            result << File.expand_path(path, dir)
+            result.concat(Dir.glob(File.expand_path(path, dir)))
           end
           result
         else

data/lib/octocatalog-diff/util/parallel.rb

@@ -129,22 +129,26 @@ module OctocatalogDiff
 
         # Waiting for children and handling results
         while pidmap.any?
-          this_pid, exit_obj = Process.wait2(0)
-          next unless this_pid && pidmap.key?(this_pid)
-          index = pidmap[this_pid][:index]
-          exitstatus = exit_obj.exitstatus
-          raise "PID=#{this_pid} exited abnormally: #{exit_obj.inspect}" if exitstatus.nil?
-          raise "PID=#{this_pid} exited with status #{exitstatus}" unless exitstatus.zero?
-
-          input = File.read(File.join(ipc_tempdir, "#{this_pid}.dat"))
-          result[index] = Marshal.load(input) # rubocop:disable Security/MarshalLoad
-          time_delta = Time.now - pidmap[this_pid][:start_time]
-          pidmap.delete(this_pid)
-
-          logger.debug "PID=#{this_pid} completed in #{time_delta} seconds, #{input.length} bytes"
-
-          next if result[index].status
-          return result[index].exception
+          pidmap.each do |pid|
+            status = Process.waitpid2(pid[0], Process::WNOHANG)
+            next if status.nil?
+            this_pid, exit_obj = status
+            next unless this_pid && pidmap.key?(this_pid)
+            index = pidmap[this_pid][:index]
+            exitstatus = exit_obj.exitstatus
+            raise "PID=#{this_pid} exited abnormally: #{exit_obj.inspect}" if exitstatus.nil?
+            raise "PID=#{this_pid} exited with status #{exitstatus}" unless exitstatus.zero?
+
+            input = File.read(File.join(ipc_tempdir, "#{this_pid}.dat"))
+            result[index] = Marshal.load(input) # rubocop:disable Security/MarshalLoad
+            time_delta = Time.now - pidmap[this_pid][:start_time]
+            pidmap.delete(this_pid)
+
+            logger.debug "PID=#{this_pid} completed in #{time_delta} seconds, #{input.length} bytes"
+
+            next if result[index].status
+            return result[index].exception
+          end
         end
 
         logger.debug 'All child processes completed with no exceptions raised'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: octocatalog-diff
 version: !ruby/object:Gem::Version
-  version: 1.5.4
+  version: 1.6.0
 platform: ruby
 authors:
 - GitHub, Inc.
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-12-13 00:00:00.000000000 Z
+date: 2019-10-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: diffy
@@ -420,8 +420,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 3.0.6
 signing_key: 
 specification_version: 4
 summary: Compile Puppet catalogs from 2 branches, versions, etc., and compare them.