octocatalog-diff 2.0.0 → 2.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.version +1 -1
- data/doc/CHANGELOG.md +11 -0
- data/doc/advanced-compare-file-text.md +79 -0
- data/doc/advanced-environments.md +5 -5
- data/doc/advanced-puppet-master.md +4 -4
- data/doc/configuration-puppet.md +19 -19
- data/doc/configuration.md +31 -31
- data/doc/dev/integration-tests.md +2 -2
- data/doc/dev/releasing.md +40 -40
- data/doc/dev/run-from-branch.md +23 -23
- data/doc/installation.md +14 -14
- data/doc/optionsref.md +31 -0
- data/lib/octocatalog-diff/catalog-diff/differ.rb +3 -1
- data/lib/octocatalog-diff/catalog-util/fileresources.rb +38 -15
- data/lib/octocatalog-diff/cli.rb +2 -1
- data/lib/octocatalog-diff/cli/options/compare_file_text.rb +18 -0
- data/lib/octocatalog-diff/cli/options/use_lcs.rb +14 -0
- data/lib/octocatalog-diff/facts/json.rb +13 -2
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 07e53efa197ece3d868ab453f200b49d6fc0718fcd40e8201ab92d04e4787871
|
|
4
|
+
data.tar.gz: e60db7eb374631ec5a4f430b58b16f57772e081aeea3b12d7576e3265df4ae6b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2b6ce06f9d1d91f8bfeb12f731a8b8d901b794396dd34a6b3e2d29b8e4e7e59d02ceb1f1bdce89ac73d6d3ad14beb2883188810c778ed6ccb431a30b05a9d7b3
|
|
7
|
+
data.tar.gz: 2f712011fd71e32caf2bf3e453ce16cae41c2be79a6bdfa12ed0d3bdc1ce62751e86c6f4eea500d243f8a69adf6642f1009250c88b5b85cb1af2ddee67e3a7a0
|
data/.version
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
2.
|
|
1
|
+
2.1.0
|
data/doc/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,17 @@
|
|
|
8
8
|
</tr>
|
|
9
9
|
</thead><tbody>
|
|
10
10
|
|
|
11
|
+
<tr valign=top>
|
|
12
|
+
<td>2.1.0</td>
|
|
13
|
+
<td>2020-02-18</td>
|
|
14
|
+
|
|
15
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/240">#240</a>: (Enhancement) Run CI against Puppet 7 and Ruby 3</li>
|
|
16
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/241">#241</a>: (Bug Fix) Fix indent and numbering in several docs</li>
|
|
17
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/242">#242</a>: (Enhancement) Handle JSON facts structured as name/values</li>
|
|
18
|
+
<li><a href="https://github.com/github/octocatalog-diff/pull/244">#244</a>: (Enhancement) Update capability and docs of --compare-file-text</li>
|
|
19
|
+
</td>
|
|
20
|
+
</tr>
|
|
21
|
+
|
|
11
22
|
<tr valign=top>
|
|
12
23
|
<td>2.0.0</td>
|
|
13
24
|
<td>2020-01-12</td>
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Compare file text
|
|
2
|
+
|
|
3
|
+
`octocatalog-diff` contains functionality to detect changes in file content for file resources that use the `source` attribute like this:
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
file { '/etc/logrotate.d/my-service.conf':
|
|
7
|
+
source => 'puppet:///modules/logrotate/etc/logrotate.d/my-service.conf',
|
|
8
|
+
}
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
When the `source` attribute is used, the catalog contains the value of the attribute (in the example above, `source` = `puppet:///modules/logrotate/etc/logrotate.d/my-service.conf`). However, the catalog does not contain the content of the file. When an agent applies the catalog, the file found on the server or in the code base at `modules/logrotate/file/etc/logrotate.d/my-service.conf` will be installed into `/etc/logrotate.d/my-service.conf`.
|
|
12
|
+
|
|
13
|
+
If a developer creates a change to this file, the catalog will not indicate this change. This means that so long as the `source` location does not change, any tool analyzing just the catalog would not detect a "diff" resulting from changes in the underlying file itself.
|
|
14
|
+
|
|
15
|
+
However, since applying the catalog could change the content of a file on the target node, `octocatalog-diff` has a feature that will do the following for any `source => 'puppet:///modules/...'` files:
|
|
16
|
+
|
|
17
|
+
- Locate the source file in the Puppet code
|
|
18
|
+
- Substitute the content of the file into the generated catalog (remove `source` and populate `content`)
|
|
19
|
+
- Display the "diff" in the now-populated `content` field
|
|
20
|
+
|
|
21
|
+
This feature is available only when the catalogs are being compiled from local code. This feature is not available, and will be automatically disabled, when pulling catalogs from PuppetDB or a Puppet server.
|
|
22
|
+
|
|
23
|
+
Note: In Puppet >= 4.4 there is an option in Puppet itself called "static catalogs" which if enabled will cause the checksum of the file to be included in the catalog. However, the `octocatalog-diff` feature described here is still useful because it can be used to display a "diff" of the change rather than just displaying a "diff" of a checksum.
|
|
24
|
+
## Command line options
|
|
25
|
+
|
|
26
|
+
### `--compare-file-text` and `--no-compare-file-text`
|
|
27
|
+
|
|
28
|
+
The feature described above is enabled by default, and no special setup is required.
|
|
29
|
+
|
|
30
|
+
To disable this feature for a specific run, add `--no-compare-file-text` to the command line.
|
|
31
|
+
|
|
32
|
+
To disable this feature by default, add the following to a [configuration](/doc/configuration.md) file:
|
|
33
|
+
|
|
34
|
+
```ruby
|
|
35
|
+
settings[:compare_file_text] = false
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
If this feature is disabled by default in a configuration file, add `--compare-file-text` to enable the feature for this specific run.
|
|
39
|
+
|
|
40
|
+
Note that the feature will be automatically disabled, regardless of configuration or command line options, if catalogs are being pulled from PuppetDB or a Puppet server.
|
|
41
|
+
|
|
42
|
+
### `--compare-file-text-ignore-tags`
|
|
43
|
+
|
|
44
|
+
To disable this feature for specific `file` resources, set a tag on the resources for which the comparison is undesired. For example:
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
file { '/etc/logrotate.d/my-service.conf':
|
|
48
|
+
source => 'puppet:///modules/logrotate/etc/logrotate.d/my-service.conf',
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
file { '/etc/logrotate.d/other-service.conf':
|
|
52
|
+
tag => ['compare-file-text-disable'],
|
|
53
|
+
source => 'puppet:///modules/logrotate/etc/logrotate.d/other-service.conf',
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Inform `octocatalog-diff` of the name of the tag either via the command line (`--compare-file-text-ignore-tags "compare-file-text-disable"`) or via a [configuration](/doc/configuration.md) file:
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
settings[:compare_file_text_ignore_tags] = %w(compare-file-text-disable)
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
With this example setup, the file text would be compared for `/etc/logrotate.d/my-service.conf` but would NOT be compared for `/etc/logrotate.d/other-service.conf`.
|
|
64
|
+
|
|
65
|
+
Notes:
|
|
66
|
+
|
|
67
|
+
1. `--compare-file-text-ignore-tags` can take comma-separated arguments if there are multiple tags, e.g.: `--compare-file-text-ignore-tags tag1,tag2`.
|
|
68
|
+
|
|
69
|
+
1. When defining values in the configuration file, `settings[:compare_file_text_ignore_tags]` must be an array. (The default value is an empty array, `[]`.)
|
|
70
|
+
|
|
71
|
+
1. "compare-file-text-disable" is used as the name of the tag in the example above, but it is not a magical value. Any valid tag name can be used.
|
|
72
|
+
|
|
73
|
+
## Notes
|
|
74
|
+
|
|
75
|
+
1. If the underlying file source cannot be found when building the "to" catalog, an exception will be raised. This will display the resource type and title, the value of `source` that is invalid, and the file name and line number of the Puppet manifest that declared the resource.
|
|
76
|
+
|
|
77
|
+
1. If the underlying file source cannot be found when building the "from" catalog, no exception will be raised. This behavior allows `octocatalog-diff` to work correctly even if there is a problem in the "from" catalog that is being corrected in the "to" catalog. (Of course, if the underlying file source can't be found in the "to" catalog either, an exception is raised -- see #1.)
|
|
78
|
+
|
|
79
|
+
1. No processing takes place for file `source` whose values do not start with `puppet:///modules/`.
|
|
@@ -15,13 +15,13 @@ If you are using environment names to control the behavior of Puppet, this defau
|
|
|
15
15
|
When you supply the command line argument `--preserve-environments` (or set `settings[:preserve_environments] = true` in your [configuration file](/doc/configuration.md)), `octocatalog-diff` will instead do the following:
|
|
16
16
|
|
|
17
17
|
1. Create a temporary directory
|
|
18
|
-
|
|
18
|
+
1. Create the following symlinks from `<temporary directory>` to the corresponding directories in the checkout of your code:
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
20
|
+
- `environments`
|
|
21
|
+
- `manifests`
|
|
22
|
+
- `modules`
|
|
23
23
|
|
|
24
|
-
|
|
24
|
+
1. Run Puppet using an environment you specify via the command line
|
|
25
25
|
|
|
26
26
|
Note that you must have set `--preserve-environments` in order for the `--environment` and/or `--create-symlinks` options (described below) to have any effect.
|
|
27
27
|
|
|
@@ -4,15 +4,15 @@
|
|
|
4
4
|
|
|
5
5
|
Please note the following caveats:
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
1. This method will put some load on your Puppet Master to build the catalog. Depending on how you use `octocatalog-diff` you should ensure that this extra load will not overwhelm your Puppet Master (especially if you create a "thundering herd" by launching several instances of `octocatalog-diff` simultaneously).
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
1. You will need to deploy your Puppet code to an environment on your Puppet Master prior to running `octocatalog-diff` for that environment. `octocatalog-diff` does not deploy code for you.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
1. You will need to configure authorization for one or more whitelisted certificates on your Puppet Master. The default permissions allow a node to retrieve its own catalog via the API, but you need a certificate for `octocatalog-diff` that permits it to retrieve any catalog. See the [Certificate authorization](#certificate-authorization) section below. If you are using Puppet Enterprise and use
|
|
12
12
|
the Puppet Master v4 API you may also use a Puppet Enterprise RBAC token. The user owning the token will need the "Puppet Server Compile catalogs for remote nodes" permission.
|
|
13
13
|
See the [PE RBAC Token Authorization](#pe-rbac-token-authorization) section below.
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
1. If you are using the v2 or v3 PuppetServer APIs with Octocatalog-Diff to compile catalogs, then those catalogs and facts will be automatically stored in PuppetDB. However, when using the v4 PuppetServer API with Octocatalog-Diff, facts and catalogs are *not* automatically stored in PuppetDB - persistence is optional and may be enabled with the appropriate Octocatalog-Diff CLI flag. If your environment depends on the accuracy of exported resources or facts in PuppetDB, you may wish to upgrade and use the V4 API, to avoid unintentional side-effects.
|
|
16
16
|
|
|
17
17
|
## Command line options
|
|
18
18
|
|
data/doc/configuration-puppet.md
CHANGED
|
@@ -4,33 +4,33 @@ The most common use of `octocatalog-diff` is to use `puppet` locally to compile
|
|
|
4
4
|
|
|
5
5
|
In order to successfully use Puppet to compile catalogs:
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
1. Puppet must be installed on the system.
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
It is the goal of `octocatalog-diff` to support Puppet version 3.8 and higher, installed via any means supported by Puppet. This includes the [All-In-One agent package](https://docs.puppet.com/puppet/4.0/reference/release_notes.html#all-in-one-packaging) or installed as a Ruby gem.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
By default, `octocatalog-diff` will look for the Puppet binary in several common system locations.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
For maximum reliability, you can specify the full path to the Puppet binary in the configuration file. For example:
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
15
|
+
```
|
|
16
|
+
##############################################################################################
|
|
17
|
+
# puppet_binary
|
|
18
|
+
# This is the full path to the puppet binary on your system. If you don't specify this,
|
|
19
|
+
# the tool will just run 'puppet' and hope to find it in your path.
|
|
20
|
+
##############################################################################################
|
|
21
21
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
22
|
+
# settings[:puppet_binary] = '/usr/bin/puppet'
|
|
23
|
+
settings[:puppet_binary] = '/opt/puppetlabs/puppet/bin/puppet'
|
|
24
|
+
```
|
|
25
25
|
|
|
26
|
-
|
|
26
|
+
1. Applies if you are using [exported resources](https://docs.puppet.com/puppet/latest/reference/lang_exported.html) from PuppetDB (i.e., the octocatalog-diff `--storeconfigs` option enabled):
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
Your Puppet installation must have the `puppetdb-termini` feature available. This feature may not be included by default with the Puppet agent package.
|
|
29
29
|
|
|
30
|
-
|
|
30
|
+
Consult the [Connecting Puppet masters to PuppetDB](https://docs.puppet.com/puppetdb/latest/connect_puppet_master.html#step-1-install-plug-ins) documentation for instructions on installing the `puppetdb-termini` gem.
|
|
31
31
|
|
|
32
|
-
|
|
32
|
+
:warning: Attention Mac OS users: the [documentation](https://docs.puppet.com/puppet/latest/reference/puppet_collections.html#os-x-systems) states:
|
|
33
33
|
|
|
34
|
-
|
|
34
|
+
> While the puppet-agent package is the only component of a Puppet Collection available on OS X, you can still use Puppet Collections to ensure the version of package-agent you install is compatible with the Puppet Collection powering your infrastructure.
|
|
35
35
|
|
|
36
|
-
|
|
36
|
+
Unfortunately this means that you won't be able to enable `--storeconfigs` with the All-In-One Puppet Agent on Mac OS X, unless you manually install a gem-packaged version of `puppetdb-terminus`. The procedure for this is beyond the scope of this documentation.
|
data/doc/configuration.md
CHANGED
|
@@ -2,51 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
`octocatalog-diff` may require configuration to work correctly with your Puppet setup.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
1. Download the [sample configuration file](https://raw.githubusercontent.com/github/octocatalog-diff/master/examples/octocatalog-diff.cfg.rb) and save it into one of the following directories.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
- In the base directory of your Puppet repository checkout (i.e., your current working directory):
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
9
|
+
```
|
|
10
|
+
.octocatalog-diff.cfg.rb
|
|
11
|
+
```
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
- In your home directory:
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
15
|
+
```
|
|
16
|
+
$HOME/.octocatalog-diff.cfg.rb
|
|
17
|
+
```
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
- In one of the following system locations:
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
21
|
+
```
|
|
22
|
+
/usr/local/etc/octocatalog-diff.cfg.rb
|
|
23
|
+
/opt/puppetlabs/octocatalog-diff/octocatalog-diff.cfg.rb
|
|
24
|
+
/etc/octocatalog-diff.cfg.rb
|
|
25
|
+
```
|
|
26
26
|
|
|
27
|
-
|
|
27
|
+
Note: If more than one of the above files is present, the first one found will be used (proceeding from top to bottom in that list). If you set an environment variable `OCTOCATALOG_DIFF_CONFIG_FILE` that will supersede all of the above paths, and allow you to specify the configuration file location however you wish.
|
|
28
28
|
|
|
29
|
-
|
|
29
|
+
1. Open the file in a text editor, and follow the comments within the file to guide yourself through configuration. The configuration file is pure ruby, allowing substantial flexibility in the configuration.
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
To be minimally functional, you will almost certainly need to define at least the following settings:
|
|
32
32
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
33
|
+
- `settings[:hiera_config]` as the absolute or relative path to your hiera configuration file
|
|
34
|
+
- `settings[:hiera_path_strip]` as the prefix to strip when munging the hiera configuration file
|
|
35
|
+
- `settings[:puppetdb_url]` as the URL to your PuppetDB instance so facts can be obtained
|
|
36
36
|
|
|
37
|
-
|
|
37
|
+
For more information on these settings:
|
|
38
38
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
39
|
+
- [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md)
|
|
40
|
+
- [Configuring octocatalog-diff to use ENC](/doc/configuration-enc.md)
|
|
41
|
+
- [Configuring octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md)
|
|
42
|
+
- [Configuring octocatalog-diff to use Puppet](/doc/configuration-puppet.md)
|
|
43
43
|
|
|
44
|
-
|
|
44
|
+
1. Test the configuration, which will indicate the location of the configuration file and validate the contents thereof.
|
|
45
45
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
46
|
+
```
|
|
47
|
+
octocatalog-diff --config-test
|
|
48
|
+
```
|
|
49
49
|
|
|
50
|
-
|
|
50
|
+
Note: If you [installed](/doc/installation.md) octocatalog-diff as a gem, the `octocatalog-diff` binary should be in your $PATH and the above command should work correctly. If you installed in a different way, you may need to provide the full path to where the `octocatalog-diff` binary was actually installed.
|
|
51
51
|
|
|
52
52
|
Now that you have entered your configuration and confirmed proper reading of your configuration file, proceed to [Basic usage](/doc/basic.md) to see if it works!
|
|
@@ -58,6 +58,6 @@ end
|
|
|
58
58
|
|
|
59
59
|
## Hints for writing an integration test
|
|
60
60
|
|
|
61
|
-
|
|
61
|
+
1. If your integration test deals only with the calculation or display of differences, and not catalog compilation, and you are compiling catalogs multiple times, prefer `spec_catalog_old` and `spec_catalog_new` to pass in pre-compiled catalogs. This will make the test run faster.
|
|
62
62
|
|
|
63
|
-
|
|
63
|
+
1. It's a good idea to check the exit code in a test of its own, and then `pending` any subsequent tests if that exit code doesn't match. This way only one test, and not all the tests, will fail if the catalog compilation doesn't work.
|
data/doc/dev/releasing.md
CHANGED
|
@@ -18,65 +18,65 @@ To test the new version of `octocatalog-diff` in the GitHub Puppet repository, c
|
|
|
18
18
|
|
|
19
19
|
This section is useful when releasing a new version based on one PR submitted by a contributor. Following this workflow is important so that the contributor gets appropriate credit in the revision history for his or her work.
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
1. In your local checkout, start a new branch based off master.
|
|
22
|
+
1. Add the contributor's repository as a remote. For example:
|
|
23
23
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
24
|
+
```
|
|
25
|
+
git remote add octocat https://github.com/octocat/octocatalog-diff.git
|
|
26
|
+
```
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
1. Merge in the contributor's branch into your own. For example:
|
|
29
29
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
```
|
|
31
|
+
git merge octocat/some-branch
|
|
32
|
+
```
|
|
33
33
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
34
|
+
1. Update `.version` and `doc/CHANGELOG.md` appropriately. In CHANGELOG you should link to the PR submitted by the contributor (not the PR you're creating now).
|
|
35
|
+
1. Commit your changes to `.version` and `doc/CHANGELOG.md`.
|
|
36
|
+
1. If necessary, auto-generate the build documentation, and commit the changes to your branch.
|
|
37
37
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
38
|
+
```
|
|
39
|
+
rake doc:build
|
|
40
|
+
```
|
|
41
41
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
42
|
+
1. Open a Pull Request based on your branch. Confirm that the history is correct, showing the contributor's commits first, and then your commit(s) updating the version file, change log, and/or auto-generated documentation.
|
|
43
|
+
1. Ensure that CI tests are all passing.
|
|
44
|
+
1. Ensure that you've performed "local testing" within GitHub (typically, ~1 day) to confirm the changes.
|
|
45
|
+
1. Merge your PR and delete your branch.
|
|
46
|
+
1. Confirm that the contributor's PR now appears as merged, and any associated issues have been closed.
|
|
47
47
|
|
|
48
48
|
## Merging multiple PRs
|
|
49
49
|
|
|
50
50
|
If multiple PRs will constitute a release, it's generally easier to merge each such PR individually, and then create a separate PR afterwards to update the necessary files.
|
|
51
51
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
52
|
+
1. Merge all constituent PRs and ensure that any associated issues have been closed.
|
|
53
|
+
1. Create your own branch based off master.
|
|
54
|
+
1. Update `.version` and `doc/CHANGELOG.md` appropriately. In CHANGELOG you should link to the PR submitted by the contributor (not the PR you're creating now).
|
|
55
|
+
1. Commit your changes to `.version` and `doc/CHANGELOG.md`.
|
|
56
|
+
1. If necessary, auto-generate the build documentation, and commit the changes to your branch.
|
|
57
57
|
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
58
|
+
```
|
|
59
|
+
rake doc:build
|
|
60
|
+
```
|
|
61
61
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
62
|
+
1. Open a Pull Request based on your branch.
|
|
63
|
+
1. Ensure that CI tests are all passing.
|
|
64
|
+
1. Ensure that you've performed "local testing" within GitHub (typically, ~1 day) to confirm the changes.
|
|
65
|
+
1. Merge your PR and delete your branch.
|
|
66
66
|
|
|
67
67
|
## Releasing
|
|
68
68
|
|
|
69
69
|
Generally, a new release will correspond to a merge to master of one or more Pull Requests.
|
|
70
70
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
71
|
+
1. Ensure that all changes associated with the release have been merged to master.
|
|
72
|
+
- Merge all Pull Requests associated with release, including the version number bump, change log update, etc.
|
|
73
|
+
- If necessary (for significant changes), complete a Pull Request to update the top-level README file.
|
|
74
|
+
1. Ensure the the master branch is checked out on your system.
|
|
75
|
+
1. Run the release procedure:
|
|
76
76
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
77
|
+
```
|
|
78
|
+
rake gem:release
|
|
79
|
+
```
|
|
80
80
|
|
|
81
81
|
This rake task handles the following:
|
|
82
82
|
|
data/doc/dev/run-from-branch.md
CHANGED
|
@@ -8,38 +8,38 @@ This document is intended for people who may not be familiar with git, GitHub, a
|
|
|
8
8
|
|
|
9
9
|
1. Determine the branch name. If there's an open Pull Request, you can see the branch name near the top of the page.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
1. Clone the `octocatalog-diff` repository in your home directory. From the command line:
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
15
|
+
```
|
|
16
|
+
cd $HOME
|
|
17
|
+
git clone https://github.com/github/octocatalog-diff.git
|
|
18
|
+
```
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
1. Change into the directory created by your checkout:
|
|
21
21
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
22
|
+
```
|
|
23
|
+
cd $HOME/octocatalog-diff
|
|
24
|
+
```
|
|
25
25
|
|
|
26
|
-
|
|
26
|
+
1. Check out the branch you wish to use, filling in the branch name you determined in the first step:
|
|
27
27
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
28
|
+
```
|
|
29
|
+
git checkout BRANCH_NAME_FROM_STEP_1
|
|
30
|
+
```
|
|
31
31
|
|
|
32
|
-
|
|
32
|
+
1. Bootstrap the repository to pull in dependencies:
|
|
33
33
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
34
|
+
```
|
|
35
|
+
./script/bootstrap
|
|
36
|
+
```
|
|
37
37
|
|
|
38
|
-
|
|
38
|
+
1. Optional but recommended - run the test suite:
|
|
39
39
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
40
|
+
```
|
|
41
|
+
rake
|
|
42
|
+
```
|
|
43
43
|
|
|
44
44
|
## Use
|
|
45
45
|
|
|
@@ -53,7 +53,7 @@ We've created a wrapper script to make this easier for you.
|
|
|
53
53
|
cd /etc/puppetlabs/code
|
|
54
54
|
```
|
|
55
55
|
|
|
56
|
-
|
|
56
|
+
1. Run the `script/octocatalog-diff-wrapper` script from *this* checkout. For example, if you checked out `octocatalog-diff` to your home directory, you could use:
|
|
57
57
|
|
|
58
58
|
```
|
|
59
59
|
$HOME/octocatalog-diff/script/octocatalog-diff-wrapper <options>
|
data/doc/installation.md
CHANGED
|
@@ -25,26 +25,26 @@ For general information on installing gems, see: [RubyGems Basics](http://guides
|
|
|
25
25
|
|
|
26
26
|
To install from source, you'll need a git client and internet access.
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
1. Clone the repository
|
|
29
29
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
```
|
|
31
|
+
git clone https://github.com/github/octocatalog-diff.git
|
|
32
|
+
```
|
|
33
33
|
|
|
34
|
-
|
|
34
|
+
1. Bootstrap the repository (this will install dependent gems in the project)
|
|
35
35
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
36
|
+
```
|
|
37
|
+
cd octocatalog-diff
|
|
38
|
+
./script/bootstrap
|
|
39
|
+
```
|
|
40
40
|
|
|
41
|
-
|
|
41
|
+
1. RECOMMENDED: Make sure the tests pass on your machine
|
|
42
42
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
43
|
+
```
|
|
44
|
+
rake
|
|
45
|
+
```
|
|
46
46
|
|
|
47
|
-
|
|
47
|
+
Note: If tests fail on your machine with a clean checkout of the master branch, we would definitely appreciate if you would report it. Please [open an issue](https://github.com/github/octocatalog-diff/issues/new) with the output and some information about your system (e.g. OS, ruby version, etc.) to let us know.
|
|
48
48
|
|
|
49
49
|
Once the code is downloaded and bootstrapped, please proceed to [Configuration](/doc/configuration.md).
|
|
50
50
|
|
data/doc/optionsref.md
CHANGED
|
@@ -74,6 +74,7 @@ Usage: octocatalog-diff [command line options]
|
|
|
74
74
|
--from-enc PATH Path to ENC script (for the from catalog only)
|
|
75
75
|
--to-enc PATH Path to ENC script (for the to catalog only)
|
|
76
76
|
--[no-]display-detail-add Display parameters and other details for added resources
|
|
77
|
+
--[no-]use-lcs Use the LCS algorithm to determine differences in arrays
|
|
77
78
|
--[no-]truncate-details Truncate details with --display-detail-add
|
|
78
79
|
--no-header Do not print a header
|
|
79
80
|
--default-header Print default header with output
|
|
@@ -184,6 +185,8 @@ Usage: octocatalog-diff [command line options]
|
|
|
184
185
|
--no-ignore-tags Disable ignoring based on tags
|
|
185
186
|
--ignore-tags STRING1[,STRING2[,...]]
|
|
186
187
|
Specify tags to ignore
|
|
188
|
+
--compare-file-text-ignore-tags STRING1[,STRING2[,...]]
|
|
189
|
+
Tags that exclude a file resource from text comparison
|
|
187
190
|
--[no-]preserve-environments Enable or disable environment preservation
|
|
188
191
|
--environment STRING Environment for catalog compilation globally
|
|
189
192
|
--to-environment STRING Environment for catalog compilation for the to branch
|
|
@@ -373,6 +376,21 @@ what is most often desired. (<a href="../lib/octocatalog-diff/cli/options/compar
|
|
|
373
376
|
</td>
|
|
374
377
|
</tr>
|
|
375
378
|
|
|
379
|
+
<tr>
|
|
380
|
+
<td valign=top>
|
|
381
|
+
<pre><code>--compare-file-text-ignore-tags STRING1[,STRING2[,...]]</code></pre>
|
|
382
|
+
</td>
|
|
383
|
+
<td valign=top>
|
|
384
|
+
Tags that exclude a file resource from text comparison
|
|
385
|
+
</td>
|
|
386
|
+
<td valign=top>
|
|
387
|
+
When a file is specified with `source => 'puppet:///modules/something/foo.txt'`, remove
|
|
388
|
+
the 'source' attribute and populate the 'content' attribute with the text of the file.
|
|
389
|
+
This allows for a diff of the content, rather than a diff of the location, which is
|
|
390
|
+
what is most often desired. (<a href="../lib/octocatalog-diff/cli/options/compare_file_text.rb">compare_file_text.rb</a>)
|
|
391
|
+
</td>
|
|
392
|
+
</tr>
|
|
393
|
+
|
|
376
394
|
<tr>
|
|
377
395
|
<td valign=top>
|
|
378
396
|
<pre><code>--create-symlinks STRING1[,STRING2[,...]]</code></pre>
|
|
@@ -1896,6 +1914,19 @@ has no effect when `--display-detail-add` is not used. (<a href="../lib/octocata
|
|
|
1896
1914
|
</td>
|
|
1897
1915
|
</tr>
|
|
1898
1916
|
|
|
1917
|
+
<tr>
|
|
1918
|
+
<td valign=top>
|
|
1919
|
+
<pre><code>--use-lcs
|
|
1920
|
+
--no-use-lcs </code></pre>
|
|
1921
|
+
</td>
|
|
1922
|
+
<td valign=top>
|
|
1923
|
+
Use the LCS algorithm to determine differences in arrays
|
|
1924
|
+
</td>
|
|
1925
|
+
<td valign=top>
|
|
1926
|
+
Configures using the Longest common subsequence (LCS) algorithm to determine differences in arrays (<a href="../lib/octocatalog-diff/cli/options/use_lcs.rb">use_lcs.rb</a>)
|
|
1927
|
+
</td>
|
|
1928
|
+
</tr>
|
|
1929
|
+
|
|
1899
1930
|
<tr>
|
|
1900
1931
|
<td valign=top>
|
|
1901
1932
|
<pre><code>--validate-references
|
|
@@ -535,9 +535,11 @@ module OctocatalogDiff
|
|
|
535
535
|
catalog2_resources = catalog2_in[:catalog]
|
|
536
536
|
|
|
537
537
|
@logger.debug "Entering hashdiff_initial; catalog sizes: #{catalog1_resources.size}, #{catalog2_resources.size}"
|
|
538
|
+
use_lcs = @opts.fetch(:use_lcs, true)
|
|
539
|
+
@logger.debug "HashDiff configuration: (use_lcs: #{use_lcs})"
|
|
538
540
|
result = []
|
|
539
541
|
hashdiff_add_remove = Set.new
|
|
540
|
-
hashdiff_result = HashDiff.diff(catalog1_resources, catalog2_resources, delimiter: "\f")
|
|
542
|
+
hashdiff_result = HashDiff.diff(catalog1_resources, catalog2_resources, delimiter: "\f", use_lcs: use_lcs)
|
|
541
543
|
hashdiff_result.each do |obj|
|
|
542
544
|
# Regular change
|
|
543
545
|
if obj[0] == '~'
|
|
@@ -13,9 +13,16 @@ module OctocatalogDiff
|
|
|
13
13
|
# Public method: Convert file resources to text. See the description of the class
|
|
14
14
|
# just above for details.
|
|
15
15
|
# @param obj [OctocatalogDiff::Catalog] Catalog object (will be modified)
|
|
16
|
+
# @param environment [String] Environment (defaults to production)
|
|
16
17
|
def self.convert_file_resources(obj, environment = 'production')
|
|
17
18
|
return unless obj.valid? && obj.compilation_dir.is_a?(String) && !obj.compilation_dir.empty?
|
|
18
|
-
_convert_file_resources(
|
|
19
|
+
_convert_file_resources(
|
|
20
|
+
obj.resources,
|
|
21
|
+
obj.compilation_dir,
|
|
22
|
+
environment,
|
|
23
|
+
obj.options[:compare_file_text_ignore_tags],
|
|
24
|
+
obj.options[:tag]
|
|
25
|
+
)
|
|
19
26
|
begin
|
|
20
27
|
obj.catalog_json = ::JSON.generate(obj.catalog)
|
|
21
28
|
rescue ::JSON::GeneratorError => exc
|
|
@@ -70,8 +77,11 @@ module OctocatalogDiff
|
|
|
70
77
|
# Internal method: Static method to convert file resources. The compilation directory is
|
|
71
78
|
# required, or else this is a no-op. The passed-in array of resources is modified by this method.
|
|
72
79
|
# @param resources [Array<Hash>] Array of catalog resources
|
|
73
|
-
# @param compilation_dir [String] Compilation directory
|
|
74
|
-
|
|
80
|
+
# @param compilation_dir [String] Compilation directory
|
|
81
|
+
# @param environment [String] Environment
|
|
82
|
+
# @param ignore_tags [Array<String>] Tags that exempt a resource from conversion
|
|
83
|
+
# @param to_from_tag [String] Either "to" or "from" for catalog type
|
|
84
|
+
def self._convert_file_resources(resources, compilation_dir, environment, ignore_tags, to_from_tag)
|
|
75
85
|
# Calculate compilation directory. There is not explicit error checking here because
|
|
76
86
|
# there is on-demand, explicit error checking for each file within the modification loop.
|
|
77
87
|
return unless compilation_dir.is_a?(String) && compilation_dir != ''
|
|
@@ -88,14 +98,10 @@ module OctocatalogDiff
|
|
|
88
98
|
resources.map! do |resource|
|
|
89
99
|
if resource_convertible?(resource)
|
|
90
100
|
path = file_path(resource['parameters']['source'], modulepaths)
|
|
91
|
-
if path.nil?
|
|
92
|
-
# Pass this through as a wrapped exception, because it's more likely to be something wrong
|
|
93
|
-
# in the catalog itself than it is to be a broken setup of octocatalog-diff.
|
|
94
|
-
message = "Errno::ENOENT: Unable to resolve '#{resource['parameters']['source']}'!"
|
|
95
|
-
raise OctocatalogDiff::Errors::CatalogError, message
|
|
96
|
-
end
|
|
97
101
|
|
|
98
|
-
if
|
|
102
|
+
if resource['tags'] && ignore_tags && (resource['tags'] & ignore_tags).any?
|
|
103
|
+
# Resource tagged not to be converted -- do nothing.
|
|
104
|
+
elsif path && File.file?(path)
|
|
99
105
|
# If the file is found, read its content. If the content is all ASCII, substitute it into
|
|
100
106
|
# the 'content' parameter for easier comparison. If not, instead populate the md5sum.
|
|
101
107
|
# Delete the 'source' attribute as well.
|
|
@@ -103,23 +109,40 @@ module OctocatalogDiff
|
|
|
103
109
|
is_ascii = content.force_encoding('UTF-8').ascii_only?
|
|
104
110
|
resource['parameters']['content'] = is_ascii ? content : '{md5}' + Digest::MD5.hexdigest(content)
|
|
105
111
|
resource['parameters'].delete('source')
|
|
106
|
-
elsif File.exist?(path)
|
|
112
|
+
elsif path && File.exist?(path)
|
|
107
113
|
# We are not handling recursive file installs from a directory or anything else.
|
|
108
114
|
# However, the fact that we found *something* at this location indicates that the catalog
|
|
109
115
|
# is probably correct. Hence, the very general .exist? check.
|
|
116
|
+
elsif to_from_tag == 'from'
|
|
117
|
+
# Don't raise an exception for an invalid source in the "from"
|
|
118
|
+
# catalog, because the developer may be fixing this in the "to"
|
|
119
|
+
# catalog. If it's broken in the "to" catalog as well, the
|
|
120
|
+
# exception will be raised when this code runs on that catalog.
|
|
110
121
|
else
|
|
111
|
-
#
|
|
112
|
-
#
|
|
113
|
-
|
|
114
|
-
# :
|
|
122
|
+
# Pass this through as a wrapped exception, because it's more likely to be something wrong
|
|
123
|
+
# in the catalog itself than it is to be a broken setup of octocatalog-diff.
|
|
124
|
+
#
|
|
125
|
+
# Example error: <OctocatalogDiff::Errors::CatalogError: Unable to resolve
|
|
126
|
+
# source=>'puppet:///modules/test/tmp/bar' in File[/tmp/bar]
|
|
127
|
+
# (/x/modules/test/manifests/init.pp:46)>
|
|
128
|
+
source = resource['parameters']['source']
|
|
129
|
+
type = resource['type']
|
|
130
|
+
title = resource['title']
|
|
131
|
+
file = resource['file'].sub(Regexp.new('^' + Regexp.escape(env_dir) + '/'), '')
|
|
132
|
+
line = resource['line']
|
|
133
|
+
message = "Unable to resolve source=>'#{source}' in #{type}[#{title}] (#{file}:#{line})"
|
|
134
|
+
raise OctocatalogDiff::Errors::CatalogError, message
|
|
115
135
|
end
|
|
116
136
|
end
|
|
137
|
+
|
|
117
138
|
resource
|
|
118
139
|
end
|
|
119
140
|
end
|
|
120
141
|
|
|
121
142
|
# Internal method: Determine if a resource is convertible. It is convertible if it
|
|
122
143
|
# is a file resource with no declared 'content' and with a declared and parseable 'source'.
|
|
144
|
+
# It is not convertible if the resource is tagged with one of the tags declared by
|
|
145
|
+
# the option `--compare-file-text-ignore-tags`.
|
|
123
146
|
# @param resource [Hash] Resource to check
|
|
124
147
|
# @return [Boolean] True of resource is convertible, false if not
|
|
125
148
|
def self.resource_convertible?(resource)
|
data/lib/octocatalog-diff/cli.rb
CHANGED
|
@@ -44,7 +44,8 @@ module OctocatalogDiff
|
|
|
44
44
|
display_datatype_changes: true,
|
|
45
45
|
parallel: true,
|
|
46
46
|
suppress_absent_file_details: true,
|
|
47
|
-
hiera_path: 'hieradata'
|
|
47
|
+
hiera_path: 'hieradata',
|
|
48
|
+
use_lcs: true
|
|
48
49
|
}.freeze
|
|
49
50
|
|
|
50
51
|
# This method is the one to call externally. It is possible to specify alternate
|
|
@@ -15,3 +15,21 @@ OctocatalogDiff::Cli::Options::Option.newoption(:compare_file_text) do
|
|
|
15
15
|
end
|
|
16
16
|
end
|
|
17
17
|
end
|
|
18
|
+
|
|
19
|
+
# Sometimes there is a particular file resource for which the file text
|
|
20
|
+
# comparison is not desired, while not disabling that option globally. Similar
|
|
21
|
+
# to --ignore_tags, it's possible to tag the file resource and exempt it from
|
|
22
|
+
# the --compare_file_text checks.
|
|
23
|
+
# @param parser [OptionParser object] The OptionParser argument
|
|
24
|
+
# @param options [Hash] Options hash being constructed; this is modified in this method.
|
|
25
|
+
OctocatalogDiff::Cli::Options::Option.newoption(:compare_file_text_ignore_tags) do
|
|
26
|
+
has_weight 415
|
|
27
|
+
|
|
28
|
+
def parse(parser, options)
|
|
29
|
+
description = 'Tags that exclude a file resource from text comparison'
|
|
30
|
+
parser.on('--compare-file-text-ignore-tags STRING1[,STRING2[,...]]', Array, description) do |x|
|
|
31
|
+
options[:compare_file_text_ignore_tags] ||= []
|
|
32
|
+
options[:compare_file_text_ignore_tags].concat x
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Configures using the Longest common subsequence (LCS) algorithm to determine differences in arrays
|
|
4
|
+
# @param parser [OptionParser object] The OptionParser argument
|
|
5
|
+
# @param options [Hash] Options hash being constructed; this is modified in this method.
|
|
6
|
+
OctocatalogDiff::Cli::Options::Option.newoption(:use_lcs) do
|
|
7
|
+
has_weight 250
|
|
8
|
+
|
|
9
|
+
def parse(parser, options)
|
|
10
|
+
parser.on('--[no-]use-lcs', 'Use the LCS algorithm to determine differences in arrays') do |x|
|
|
11
|
+
options[:use_lcs] = x
|
|
12
|
+
end
|
|
13
|
+
end
|
|
14
|
+
end
|
|
@@ -14,8 +14,19 @@ module OctocatalogDiff
|
|
|
14
14
|
# @return [Hash] Facts
|
|
15
15
|
def self.fact_retriever(options = {}, node = '')
|
|
16
16
|
facts = ::JSON.parse(options.fetch(:fact_file_string))
|
|
17
|
-
|
|
18
|
-
|
|
17
|
+
|
|
18
|
+
if facts.keys.include?('name') && facts.keys.include?('values') && facts['values'].is_a?(Hash)
|
|
19
|
+
# If you saved the output of something like
|
|
20
|
+
# `puppet facts find $(hostname)` the structure will already be a
|
|
21
|
+
# {'name' => <fqdn>, 'values' => <hash of facts>}. We do nothing
|
|
22
|
+
# here because we don't want to double-encode.
|
|
23
|
+
else
|
|
24
|
+
facts = { 'name' => node, 'values' => facts }
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
facts['name'] = node unless node.empty?
|
|
28
|
+
facts['name'] = facts['values'].fetch('fqdn', 'unknown.node') if facts['name'].empty?
|
|
29
|
+
facts
|
|
19
30
|
end
|
|
20
31
|
end
|
|
21
32
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: octocatalog-diff
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 2.
|
|
4
|
+
version: 2.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- GitHub, Inc.
|
|
@@ -9,7 +9,7 @@ authors:
|
|
|
9
9
|
autorequire:
|
|
10
10
|
bindir: bin
|
|
11
11
|
cert_chain: []
|
|
12
|
-
date: 2021-
|
|
12
|
+
date: 2021-02-18 00:00:00.000000000 Z
|
|
13
13
|
dependencies:
|
|
14
14
|
- !ruby/object:Gem::Dependency
|
|
15
15
|
name: diffy
|
|
@@ -214,6 +214,7 @@ files:
|
|
|
214
214
|
- doc/advanced-catalog-only.md
|
|
215
215
|
- doc/advanced-catalog-validation.md
|
|
216
216
|
- doc/advanced-ci.md
|
|
217
|
+
- doc/advanced-compare-file-text.md
|
|
217
218
|
- doc/advanced-dynamic-ignores.md
|
|
218
219
|
- doc/advanced-environment-variables.md
|
|
219
220
|
- doc/advanced-environments.md
|
|
@@ -380,6 +381,7 @@ files:
|
|
|
380
381
|
- lib/octocatalog-diff/cli/options/suppress_absent_file_details.rb
|
|
381
382
|
- lib/octocatalog-diff/cli/options/to_from_branch.rb
|
|
382
383
|
- lib/octocatalog-diff/cli/options/truncate_details.rb
|
|
384
|
+
- lib/octocatalog-diff/cli/options/use_lcs.rb
|
|
383
385
|
- lib/octocatalog-diff/cli/options/validate_references.rb
|
|
384
386
|
- lib/octocatalog-diff/cli/printer.rb
|
|
385
387
|
- lib/octocatalog-diff/errors.rb
|