octocatalog-diff 1.5.3 → 1.5.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.version +1 -1
- data/doc/CHANGELOG.md +11 -0
- data/doc/dev/api/v1/calls/catalog-diff.md +6 -2
- data/doc/dev/api/v1/objects/diff.md +3 -3
- data/doc/dev/releasing.md +1 -1
- data/doc/optionsref.md +9 -5
- data/lib/octocatalog-diff/catalog-diff/differ.rb +11 -1
- data/lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb +29 -25
- data/lib/octocatalog-diff/catalog.rb +3 -2
- data/lib/octocatalog-diff/cli.rb +36 -5
- data/lib/octocatalog-diff/cli/options.rb +1 -1
- data/lib/octocatalog-diff/cli/options/hostname.rb +13 -2
- data/scripts/env/env.sh +1 -1
- data/scripts/git-extract/git-extract.sh +1 -1
- data/scripts/puppet/puppet.sh +1 -1
- metadata +20 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 0ba4dc520ab7e8ae27324beef80edd6c96f601de
|
4
|
+
data.tar.gz: d927873135a73e5f1b8c178e0cf4a54b9a29aa46
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e7a23ff7139158b5092765b190a801fbd670fe2931dde206fad3007fdf03995e3e6f62a881f9b3258d2b91880688664c7f43c947f5a2cfdb9b1c9b36516c94be
|
7
|
+
data.tar.gz: 4eae1cff736c29ced07644eac4cd4af812df56ac6c32afade41cf85ba0771d8136d79bf07783bec6a101cec3e5880ac647fbb83db55fe352b1181db6422288d0
|
data/.version
CHANGED
@@ -1 +1 @@
|
|
1
|
-
1.5.
|
1
|
+
1.5.4
|
data/doc/CHANGELOG.md
CHANGED
@@ -8,6 +8,17 @@
|
|
8
8
|
</tr>
|
9
9
|
</thead><tbody>
|
10
10
|
|
11
|
+
<tr valign=top>
|
12
|
+
<td>1.5.4</td>
|
13
|
+
<td>2018-12-11</td>
|
14
|
+
<td>
|
15
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/190">#190</a>: (Enhancement) Additional filtered out cases for compilation directory</li>
|
16
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/195">#195</a>: (Enhancement) Parallel catalog-diff when multiple hostnames are passed</li>
|
17
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/198">#198</a>: (Bug Fix) Portability fixes</li>
|
18
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/200">#200</a>: (Bug Fix) Support name parameter when validating references</li>
|
19
|
+
</td>
|
20
|
+
</tr>
|
21
|
+
|
11
22
|
<tr valign=top>
|
12
23
|
<td>1.5.3</td>
|
13
24
|
<td>2018-03-05</td>
|
@@ -167,12 +167,16 @@ In this case, "owner", "notify", and "content" are nested under "parameters". In
|
|
167
167
|
|
168
168
|
```
|
169
169
|
# Ignore all changes to the `owner` attribute of a file.
|
170
|
-
[ { type: Regexp.new('\AFile\z'), attr: Regexp.new("
|
170
|
+
[ { type: Regexp.new('\AFile\z'), attr: Regexp.new("\\Aparameters\fowner\\z" } ]
|
171
171
|
|
172
172
|
# Ignore changes to `owner` or `group` for a file or an exec.
|
173
|
-
[ { type: Regexp.new('\A(File|Exec)\z'), attr: Regexp.new("
|
173
|
+
[ { type: Regexp.new('\A(File|Exec)\z'), attr: Regexp.new("\\Aparameters\f(owner|group)\\z" } ]
|
174
174
|
```
|
175
175
|
|
176
|
+
When using regular expressions, `\f` (form feed character) is used to separate the structure (e.g. `parameters\fowner` refers to the `parameters` hash, `owner` key).
|
177
|
+
|
178
|
+
:bulb: Note that `\A` in Ruby matches the beginning of the string and `\z` matches the end, but these are not actual characters. Therefore, if you are using `\A` or `\z` in double quotes (`"`), be sure to heed the examples above and write your expression like: `Regexp.new("\\Aparameters\fowner\\z")`.
|
179
|
+
|
176
180
|
#### `:validate_references` (Array<String>, Optional)
|
177
181
|
|
178
182
|
Invoke the [catalog validation](/doc/advanced-catalog-validation.md) feature to ensure resources targeted by `before`, `notify`, `require`, and/or `subscribe` exist in the catalog. If this parameter is not defined, no reference validation occurs.
|
@@ -94,9 +94,9 @@ Returns the value of the resource from the new catalog.
|
|
94
94
|
}
|
95
95
|
}
|
96
96
|
|
97
|
-
# Demonstrates structure and
|
97
|
+
# Demonstrates structure and new_value
|
98
98
|
diff.structure #=> ['parameters', 'content']
|
99
|
-
diff.
|
99
|
+
diff.new_value #=> 'This is the NEW FILE!!!!!'
|
100
100
|
```
|
101
101
|
|
102
102
|
#### `#old_file` (String)
|
@@ -107,7 +107,7 @@ Note that this is a pass-through of information provided in the Puppet catalog,
|
|
107
107
|
|
108
108
|
Note also that if the diff represents addition of a resource, this will return `nil`, because the resource does not exist in the old catalog.
|
109
109
|
|
110
|
-
#### `#
|
110
|
+
#### `#old_line` (String)
|
111
111
|
|
112
112
|
Returns the line number within the Puppet manifest giving rise to the resource as it exists in the old catalog. (See `#old_file` for the filename of the Puppet manifest.)
|
113
113
|
|
data/doc/dev/releasing.md
CHANGED
@@ -9,7 +9,7 @@ The project maintainers are responsible for bumping the version number, regenera
|
|
9
9
|
To test the new version of `octocatalog-diff` in the GitHub Puppet repository, check out `github/puppet` and:
|
10
10
|
|
11
11
|
- Start a new branch based off master
|
12
|
-
- Run `script/update-octocatalog-diff
|
12
|
+
- Run `script/update-octocatalog-diff <ocd_branch_name>`
|
13
13
|
- Confirm and commit the result
|
14
14
|
- Make sure all CI jobs pass
|
15
15
|
- Run the `puppet-catalog-diff` CI job and make sure it passes and shows expected results
|
data/doc/optionsref.md
CHANGED
@@ -9,7 +9,8 @@
|
|
9
9
|
|
10
10
|
```
|
11
11
|
Usage: octocatalog-diff [command line options]
|
12
|
-
-n,
|
12
|
+
-n HOSTNAME1[,HOSTNAME2[,...]], Use PuppetDB facts from last run of a hostname or a comma separated list of multiple hostnames
|
13
|
+
--hostname
|
13
14
|
--basedir DIRNAME Use an alternate base directory (git checkout of puppet repository)
|
14
15
|
-f, --from FROM_BRANCH Branch you are coming from
|
15
16
|
-t, --to TO_BRANCH Branch you are going to
|
@@ -856,14 +857,17 @@ Puppet control repo template, the value of this should be 'hieradata', which is
|
|
856
857
|
|
857
858
|
<tr>
|
858
859
|
<td valign=top>
|
859
|
-
<pre><code>-n
|
860
|
-
--hostname
|
860
|
+
<pre><code>-n HOSTNAME1[,HOSTNAME2[,...]]
|
861
|
+
--hostname HOSTNAME1[,HOSTNAME2[,...]]</code></pre>
|
861
862
|
</td>
|
862
863
|
<td valign=top>
|
863
|
-
Use PuppetDB facts from last run of hostname
|
864
|
+
Use PuppetDB facts from last run of a hostname or a comma separated list of multiple hostnames
|
864
865
|
</td>
|
865
866
|
<td valign=top>
|
866
|
-
Set hostname, which is used to look up facts in PuppetDB, and in the header of diff display.
|
867
|
+
Set hostname, which is used to look up facts in PuppetDB, and in the header of diff display.
|
868
|
+
This option can recieve a single hostname, or a comma separated list of
|
869
|
+
multiple hostnames, which are split into an Array. Multiple hostnames do not
|
870
|
+
work with the `catalog-only` or `bootstrap-then-exit` options. (<a href="../lib/octocatalog-diff/cli/options/hostname.rb">hostname.rb</a>)
|
867
871
|
</td>
|
868
872
|
</tr>
|
869
873
|
|
@@ -294,10 +294,13 @@ module OctocatalogDiff
|
|
294
294
|
# Use diffy to get only the lines that have changed in a text object.
|
295
295
|
# As we iterate through the diff, jump out if we have our answer: either
|
296
296
|
# true if '=~>' finds ANY match, or false if '=&>' fails to find a match.
|
297
|
-
Diffy::Diff.new(old_val, new_val, context: 0)
|
297
|
+
diffy_result = Diffy::Diff.new(old_val, new_val, context: 0)
|
298
|
+
newline_alerts = diffy_result.count { |line| line.strip == '\' }
|
299
|
+
diffy_result.each do |line|
|
298
300
|
if regex.match(line.strip)
|
299
301
|
return true if operator == '=~>'
|
300
302
|
elsif operator == '=&>'
|
303
|
+
next if line.strip == '\' && newline_alerts == 2
|
301
304
|
return false
|
302
305
|
end
|
303
306
|
end
|
@@ -394,6 +397,13 @@ module OctocatalogDiff
|
|
394
397
|
return false unless rule[:title].casecmp(hsh[:title]).zero?
|
395
398
|
end
|
396
399
|
|
400
|
+
# If rule[:attr] is a regular expression, handle that case here.
|
401
|
+
if rule[:attr].is_a?(Regexp)
|
402
|
+
return false unless hsh[:attr].is_a?(String)
|
403
|
+
return false unless rule[:attr].match(hsh[:attr])
|
404
|
+
return ignore_match_true(hsh, rule)
|
405
|
+
end
|
406
|
+
|
397
407
|
# Special 'attributes': Ignore specific diff types (+ add, - remove, ~ and ! change)
|
398
408
|
if rule[:attr] =~ /\A[\-\+~!]+\Z/
|
399
409
|
return ignore_match_true(hsh, rule) if rule[:attr].include?(diff_type)
|
@@ -1,6 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require_relative '../filter'
|
4
|
+
require_relative '../../util/util'
|
4
5
|
|
5
6
|
module OctocatalogDiff
|
6
7
|
module CatalogDiff
|
@@ -35,43 +36,46 @@ module OctocatalogDiff
|
|
35
36
|
|
36
37
|
# Check for a change where the difference in a parameter exactly corresponds to the difference in the
|
37
38
|
# compilation directory.
|
38
|
-
if diff.change?
|
39
|
-
|
40
|
-
|
41
|
-
from_match = false
|
42
|
-
to_before = nil
|
43
|
-
to_after = nil
|
44
|
-
to_match = false
|
39
|
+
if diff.change?
|
40
|
+
o = remove_compilation_dir(diff.old_value, dir2)
|
41
|
+
n = remove_compilation_dir(diff.new_value, dir1)
|
45
42
|
|
46
|
-
if diff.old_value
|
47
|
-
from_before = Regexp.last_match(1) || ''
|
48
|
-
from_after = Regexp.last_match(2) || ''
|
49
|
-
from_match = true
|
50
|
-
end
|
51
|
-
|
52
|
-
if diff.new_value =~ /^(.*)#{dir1}(.*)$/m
|
53
|
-
to_before = Regexp.last_match(1) || ''
|
54
|
-
to_after = Regexp.last_match(2) || ''
|
55
|
-
to_match = true
|
56
|
-
end
|
57
|
-
|
58
|
-
if from_match && to_match && to_before == from_before && to_after == from_after
|
43
|
+
if o != diff.old_value || n != diff.new_value
|
59
44
|
message = "Resource key #{diff.type}[#{diff.title}] #{diff.structure.join(' => ')}"
|
60
|
-
message += '
|
45
|
+
message += ' may depend on catalog compilation directory, but there may be differences.'
|
46
|
+
message += ' This is included in results for now, but please verify.'
|
61
47
|
@logger.warn message
|
62
|
-
return true
|
63
48
|
end
|
64
49
|
|
65
|
-
if
|
50
|
+
if o == n
|
66
51
|
message = "Resource key #{diff.type}[#{diff.title}] #{diff.structure.join(' => ')}"
|
67
|
-
message += '
|
68
|
-
message += ' This is included in results for now, but please verify.'
|
52
|
+
message += ' appears to depend on catalog compilation directory. Suppressed from results.'
|
69
53
|
@logger.warn message
|
54
|
+
return true
|
70
55
|
end
|
71
56
|
end
|
72
57
|
|
73
58
|
false
|
74
59
|
end
|
60
|
+
|
61
|
+
def remove_compilation_dir(v, dir)
|
62
|
+
value = OctocatalogDiff::Util::Util.deep_dup(v)
|
63
|
+
traverse(value) do |e|
|
64
|
+
e.gsub!(dir, '') if e.respond_to?(:gsub!)
|
65
|
+
end
|
66
|
+
value
|
67
|
+
end
|
68
|
+
|
69
|
+
def traverse(a)
|
70
|
+
case a
|
71
|
+
when Array
|
72
|
+
a.map { |v| traverse(v, &Proc.new) }
|
73
|
+
when Hash
|
74
|
+
traverse(a.values, &Proc.new)
|
75
|
+
else
|
76
|
+
yield a
|
77
|
+
end
|
78
|
+
end
|
75
79
|
end
|
76
80
|
end
|
77
81
|
end
|
@@ -331,8 +331,9 @@ module OctocatalogDiff
|
|
331
331
|
title = normalized_title(resource['title'], resource['type'])
|
332
332
|
@resource_hash[resource['type']][title] = resource
|
333
333
|
|
334
|
-
if resource.key?('parameters')
|
335
|
-
@resource_hash[resource['type']][resource['parameters']['alias']] = resource
|
334
|
+
if resource.key?('parameters')
|
335
|
+
@resource_hash[resource['type']][resource['parameters']['alias']] = resource if resource['parameters'].key?('alias')
|
336
|
+
@resource_hash[resource['type']][resource['parameters']['name']] = resource if resource['parameters'].key?('name')
|
336
337
|
end
|
337
338
|
end
|
338
339
|
end
|
data/lib/octocatalog-diff/cli.rb
CHANGED
@@ -11,6 +11,7 @@ require_relative 'util/util'
|
|
11
11
|
require_relative 'version'
|
12
12
|
|
13
13
|
require 'logger'
|
14
|
+
require 'parallel'
|
14
15
|
require 'socket'
|
15
16
|
|
16
17
|
module OctocatalogDiff
|
@@ -116,16 +117,46 @@ module OctocatalogDiff
|
|
116
117
|
end
|
117
118
|
|
118
119
|
# Compile catalogs and do catalog-diff
|
119
|
-
|
120
|
+
node_set = options.delete(:node)
|
121
|
+
node_set = [node_set] unless node_set.is_a?(Array)
|
122
|
+
|
123
|
+
# run multiple node diffs in parallel
|
124
|
+
catalog_diffs = if node_set.size == 1
|
125
|
+
[run_octocatalog_diff(node_set.first, options, logger)]
|
126
|
+
else
|
127
|
+
::Parallel.map(node_set, in_threads: 4) { |node| run_octocatalog_diff(node, options, logger) }
|
128
|
+
end
|
129
|
+
|
130
|
+
# Return the resulting diff object if requested (generally for testing)
|
131
|
+
# or otherwise return exit code
|
132
|
+
return catalog_diffs.first if opts[:INTEGRATION]
|
133
|
+
|
134
|
+
all_diffs = catalog_diffs.map(&:diffs)
|
135
|
+
|
136
|
+
all_diffs.each do |diff|
|
137
|
+
next unless diff.any?
|
138
|
+
return EXITCODE_SUCCESS_WITH_DIFFS
|
139
|
+
end
|
140
|
+
|
141
|
+
EXITCODE_SUCCESS_NO_DIFFS
|
142
|
+
end
|
143
|
+
|
144
|
+
# Run the octocatalog-diff process for a given node. Return the diffs for a contribution to
|
145
|
+
# the final exit status.
|
146
|
+
# node - String with the node
|
147
|
+
# options - All of the currently defined options
|
148
|
+
# logger - Logger object
|
149
|
+
def self.run_octocatalog_diff(node, options, logger)
|
150
|
+
options_copy = options.merge(node: node)
|
151
|
+
catalog_diff = OctocatalogDiff::API::V1.catalog_diff(options_copy.merge(logger: logger))
|
120
152
|
diffs = catalog_diff.diffs
|
121
153
|
|
122
154
|
# Display diffs
|
123
|
-
printer_obj = OctocatalogDiff::Cli::Printer.new(
|
155
|
+
printer_obj = OctocatalogDiff::Cli::Printer.new(options_copy, logger)
|
124
156
|
printer_obj.printer(diffs, catalog_diff.from.compilation_dir, catalog_diff.to.compilation_dir)
|
125
157
|
|
126
|
-
# Return
|
127
|
-
|
128
|
-
diffs.any? ? EXITCODE_SUCCESS_WITH_DIFFS : EXITCODE_SUCCESS_NO_DIFFS
|
158
|
+
# Return catalog-diff object.
|
159
|
+
catalog_diff
|
129
160
|
end
|
130
161
|
|
131
162
|
# Parse command line options with 'optparse'. Returns a hash with the parsed arguments.
|
@@ -11,7 +11,7 @@ module OctocatalogDiff
|
|
11
11
|
# This class contains the option parser. 'parse_options' is the external entry point.
|
12
12
|
class Options
|
13
13
|
# The usage banner.
|
14
|
-
BANNER = 'Usage: catalog-diff -n <hostname> [-f <from environment>] [-t <to environment>]'.freeze
|
14
|
+
BANNER = 'Usage: catalog-diff -n <hostname>[,<hostname>...] [-f <from environment>] [-t <to environment>]'.freeze
|
15
15
|
|
16
16
|
# An error class specifically for passing information to the document build task.
|
17
17
|
class DocBuildError < RuntimeError; end
|
@@ -1,6 +1,9 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
# Set hostname, which is used to look up facts in PuppetDB, and in the header of diff display.
|
4
|
+
# This option can recieve a single hostname, or a comma separated list of
|
5
|
+
# multiple hostnames, which are split into an Array. Multiple hostnames do not
|
6
|
+
# work with the `catalog-only` or `bootstrap-then-exit` options.
|
4
7
|
# @param parser [OptionParser object] The OptionParser argument
|
5
8
|
# @param options [Hash] Options hash being constructed; this is modified in this method.
|
6
9
|
|
@@ -8,8 +11,16 @@ OctocatalogDiff::Cli::Options::Option.newoption(:hostname) do
|
|
8
11
|
has_weight 1
|
9
12
|
|
10
13
|
def parse(parser, options)
|
11
|
-
parser.on(
|
12
|
-
|
14
|
+
parser.on(
|
15
|
+
'--hostname HOSTNAME1[,HOSTNAME2[,...]]',
|
16
|
+
'-n',
|
17
|
+
'Use PuppetDB facts from last run of a hostname or a comma separated list of multiple hostnames'
|
18
|
+
) do |hostname|
|
19
|
+
options[:node] = if hostname.include?(',')
|
20
|
+
hostname.split(',')
|
21
|
+
else
|
22
|
+
hostname
|
23
|
+
end
|
13
24
|
end
|
14
25
|
end
|
15
26
|
end
|
data/scripts/env/env.sh
CHANGED
data/scripts/puppet/puppet.sh
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: octocatalog-diff
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.5.
|
4
|
+
version: 1.5.4
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitHub, Inc.
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2018-
|
12
|
+
date: 2018-12-13 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: diffy
|
@@ -53,6 +53,20 @@ dependencies:
|
|
53
53
|
- - ">="
|
54
54
|
- !ruby/object:Gem::Version
|
55
55
|
version: 0.3.0
|
56
|
+
- !ruby/object:Gem::Dependency
|
57
|
+
name: parallel
|
58
|
+
requirement: !ruby/object:Gem::Requirement
|
59
|
+
requirements:
|
60
|
+
- - ">="
|
61
|
+
- !ruby/object:Gem::Version
|
62
|
+
version: 1.12.0
|
63
|
+
type: :runtime
|
64
|
+
prerelease: false
|
65
|
+
version_requirements: !ruby/object:Gem::Requirement
|
66
|
+
requirements:
|
67
|
+
- - ">="
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: 1.12.0
|
56
70
|
- !ruby/object:Gem::Dependency
|
57
71
|
name: rugged
|
58
72
|
requirement: !ruby/object:Gem::Requirement
|
@@ -87,14 +101,14 @@ dependencies:
|
|
87
101
|
requirements:
|
88
102
|
- - '='
|
89
103
|
- !ruby/object:Gem::Version
|
90
|
-
version:
|
104
|
+
version: 12.3.1
|
91
105
|
type: :development
|
92
106
|
prerelease: false
|
93
107
|
version_requirements: !ruby/object:Gem::Requirement
|
94
108
|
requirements:
|
95
109
|
- - '='
|
96
110
|
- !ruby/object:Gem::Version
|
97
|
-
version:
|
111
|
+
version: 12.3.1
|
98
112
|
- !ruby/object:Gem::Dependency
|
99
113
|
name: parallel_tests
|
100
114
|
requirement: !ruby/object:Gem::Requirement
|
@@ -171,14 +185,14 @@ dependencies:
|
|
171
185
|
requirements:
|
172
186
|
- - "~>"
|
173
187
|
- !ruby/object:Gem::Version
|
174
|
-
version: 5.
|
188
|
+
version: 5.5.8
|
175
189
|
type: :development
|
176
190
|
prerelease: false
|
177
191
|
version_requirements: !ruby/object:Gem::Requirement
|
178
192
|
requirements:
|
179
193
|
- - "~>"
|
180
194
|
- !ruby/object:Gem::Version
|
181
|
-
version: 5.
|
195
|
+
version: 5.5.8
|
182
196
|
description: |
|
183
197
|
Octocatalog-Diff assists with Puppet development and testing by enabling the user to
|
184
198
|
compile 2 Puppet catalogs and compare them. It is possible to compare different
|