puppet-check 1.3.2 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0c73e077873c45fd337c09547f436ad7c0176f1f
-  data.tar.gz: a40a24c8f2e710c7d0e57bff54738003ae9ad7de
+  metadata.gz: a111d0def6afe05d7924d10fe26d46a3c50eef1f
+  data.tar.gz: 17202cf67cac1fae5700d2c332a2c432b753e8a2
 SHA512:
-  metadata.gz: bbf1b361bd6e6ae303e3e242aea111774beb1234b4a9affa45fbaa178af7b3c0ccb4315babab6d937b24a95b025bc339a469aeeefe232b954bd1aaf4d633b535
-  data.tar.gz: 2c537882049dadf1924de9021f0957932f7142a307f1f343598924474d4d11bae034bde0b2e1e1ce5e2fb01ed65a45d1b6025bf8861affaba8462955b1530838
+  metadata.gz: 86af9837fa801c6b5ace0af98e86d3d999f21adfd627eaa4231606f97835655962e485dda91d6312442934718ab53a10d9dc378f391fb7b41a3ccb0931dc8b3a
+  data.tar.gz: 105522b9b0bc64afbdcafdcc7124b9db9cb0455c8b57d37808080b59a29b423459e6eb0c1c4445419258fd5f89af7c23352f02c786daca8a786f22c85e1d1f71

data/CHANGELOG.md

@@ -1,12 +1,30 @@
-### 1.4.0 (Roadmap)
-- optional dep octocatalog-diff interface (minor)
-- rakefile interface with puppet-lint, rubocop, reek (minor)
-- rspec puppet stubbing (minor)
-- acceptance testing with vagrant/docker+serverspec (minor)
+### 1.5.0 (Roadmap)
+- optional octocatalog-diff regression testing (minor)
+- optional octocatalog-diff regression testing options (patch)
+- test-kitchen interface (minor)
 - improved json and yaml output formats support (file arrays become file hashes and output results further handles formatting) (minor; this would be a big refactoring effort)
+- https://blog.fluentz.io/learn-how-to-achieve-parallelism-with-ruby-i-o-bound-threads-a29c92aff58c#.zcqpibirn (parallelism/minor)
+- resolve deps in module fixtures? (minor)
+- rspec puppet stubbing (minor)
+- parallel spec option (minor)
+- multiple node testing with octocatalog-diff (minor)
+
+### 1.4.1 (Roadmap)
+- rakefile interface with puppet-lint, rubocop, reek (patch)
 - split syntax and style checks to separate methods for style (patch)
+- change settings to hash (patch)
+- add in begin/rescue on rspec, octocatalog, and rake loading (patch)
+- svn support for module fixtures (patch)
+- fail on warnings option + change exit code logic/possibilities (patch)
+- look into partition for execute_parsers optimization (patch)
+- handle octocatalog-diff not installed situation better (patch)
+
+### 1.4.0
+- Optimized and fixed checks on dependencies and requirements in `metadata.json`.
+- Optional octocatalog-diff smoke testing
+- Optional octocatalog-diff config file support
 
-### 1.3.2 (Roadmap)
+### 1.3.2
 - For the Puppet Forge method of downloading external module dependencies as spec fixtures, the module is now updated if it is already present. Previously, a fresh forced install was always attempted.
 - A good `spec_helper` for RSpec Puppet is now generated if one is missing. This is instead of the buggy one that `rspec-puppet-init` generates.
 - Fix blocking of hieradata checks on `hiera.yaml`.

data/README.md

@@ -133,21 +133,23 @@ The following files have unrecognized formats and therefore were not processed:
 It is worth nothing that there is no current development objective for Puppet Check to achieve the same advanced level of robustness for spec testing that Puppetlabs Spec Helper enables. If you are performing standard spec testing on your Puppet code and data, then Puppet Check's spec testing is a fantastic lighter and faster alternative to Puppetlabs Spec Helper. If you require advanced and intricate capabilities in your spec testing (e.g. direct interfacing to the `Puppet::Parser::Scope` API), then you will likely prefer Puppetlabs Spec Helper's spec testing in conjunction with Puppet Check's file validation.
 
 ## Usage
-Puppet Check requires `ruby >= 2.0.0`, `puppet >= 3.4`, and `puppet-lint >= 2.0.0`. All other dependencies should be fine with various versions. Puppet Check can be used with a CLI, Rake tasks, or API, from your system, rbenv, rvm, Docker, or Vagrant. Please note all interfaces (API by default, but can be modified) will ignore any directories named `fixtures` or specified paths with that directory during file checks and spec tests.
+Puppet Check requires `ruby >= 2.0.0`, `puppet >= 3.4`, and `puppet-lint >= 1.1.0`. `Octocatalog-diff >= 1.0.0` if you are performing smoke/regression checks. All other dependencies should be fine with various versions. Puppet Check can be used with a CLI, Rake tasks, or API, from your system, rbenv, rvm, Docker, or Vagrant. Please note all interfaces (API by default, but can be modified) will ignore any directories named `fixtures` or specified paths with that directory during file checks and spec tests.
 
-#### Reek
+#### Ruby 2.0 and Reek
 Reek dropped support for Ruby 2.0 when it went to 4.0. Since dependencies by Ruby version are allowed in Gemfiles but not gemspecs, this means that PuppetCheck installed with `bundler` will automatically pick up the correct version of Reek for your Ruby version and install it. If you are installing PuppetCheck via `gem`, then you can install reek normally with `gem` with Ruby >= 2.1, but you will need to specify `gem install reek -v 3.11` if you are using Ruby 2.0.
 
-#### Important Note for Ruby 1.9.3 and PuppetCheck <= 1.2.1
-If you are using Ruby 1.9.3 (and therefore also PuppetCheck <= 1.2.1), there is an issue where `Hiera <= 3.2.0` has an unspecified version dependency on JSonPure. Since JSonPure 2.0.2 requires `ruby >= 2.0.0`, this breaks Hiera installs on Ruby 1.9.3, which breaks Puppet installs, which breaks PuppetCheck installs. Therefore, you will need to either restrict your installed version of JSonPure to something lower than 2.0.2 if you are using Ruby 1.9.3, or use `Hiera >= 3.2.1`.
-
 ### CLI
 ```
 usage: puppet-check [options] paths
     -f, --future                     Enable future parser
     -s, --style                      Enable style checks
+    --smoke                          Enable smoke testing
+    -r, --regression                 Enable regression testing (in progress, do not use)
     -o, --output format              Format for results output (default is text): text, json, or yaml
-        --puppet-lint arg_one,arg_two
+    --octoconfig config_file         Octocatalog-diff configuration file to use.
+    -n, --octonodes node1.example.com,node2.example.com
+                                     Octocatalog-diff nodes to test catalog on.
+    --puppet-lint arg_one,arg_two
                                      Arguments for PuppetLint ignored checks
     -c, --config file                Load PuppetLint options from file.
         --rubocop arg_one,arg_two    Arguments for Rubocop disabled cops
@@ -171,12 +173,16 @@ rake puppetcheck:beaker    # Execute Beaker acceptance tests
 ```
 
 #### puppetcheck:file
-You can add style checks to and select the future parser for the `rake puppetcheck:file`, or change the output format, by adding the following after the require:
+You can add style, smoke, and regression checks to and select the future parser for the `rake puppetcheck:file`, or change the output format, by adding the following after the require:
 
 ```ruby
 PuppetCheck.style_check = true
 PuppetCheck.future_parser = true
+PuppetCheck.smoke_check = true
+PuppetCheck.regression_check = true # in progress, do not use
 PuppetCheck.output_format = 'yaml'
+PuppetCheck.octoconfig = '.octocatalog-diff.cfg.rb'
+PuppetCheck.octonodes = %w(localhost.localdomain)
 ```
 
 Please note that `rspec` does not support yaml output and therefore would still use the default 'progress' formatter even if `yaml` is specified as the format option to Puppet Check.
@@ -245,7 +251,11 @@ require 'puppet-check'
 
 PuppetCheck.future_parser = true # default false
 PuppetCheck.style_check = true # default false
+PuppetCheck.smoke_check = true # default false
+PuppetCheck.regression_check = true # in progress, do not use; default false
 PuppetCheck.output_format = 'yaml' # also 'json'; default 'text'
+PuppetCheck.octoconfig = '$HOME/octocatalog-diff.cfg.rb' # default '.octocatalog-diff.cfg.rb'
+PuppetCheck.octonodes = %w(server.example.com) # default: %w(localhost.localdomain)
 PuppetCheck.puppetlint_args = ['--puppetlint-arg-one', '--puppetlint-arg-two'] # default []
 PuppetCheck.rubocop_args = ['--except', 'rubocop-arg-one,rubocop-arg-two'] # default []
 
@@ -316,6 +326,7 @@ To overcome the lack of convenient portability, you could try spinning up the Va
 - **rake**: install this if you want to use Puppet Check with `rake` tasks in addition to the CLI.
 - **rspec**: install this if you want to use Puppet Check to execute the spec tests for your Ruby files during `rake`.
 - **rspec-puppet**: install this if you want to use Puppet Check to execute the spec tests for your Puppet files during `rake`.
+- **octocatalog-diff**: install this if you want to use Puppet Check to execute smoke or regression tests for your Puppet catalog.
 - **beaker**: install this if you want to use Puppet Check to execute the acceptance tests during `rake`.
 - **git**: install this if you want to use Puppet Check to download external module dependencies with `git` commands during RSpec Puppet testing.
 - **mercurial**: install this if you want to use Puppet Check to download external module dependencies with `hg` commands during RSpec Puppet testing.

data/lib/puppet-check.rb

@@ -2,16 +2,26 @@ require_relative 'puppet-check/puppet_parser'
 require_relative 'puppet-check/ruby_parser'
 require_relative 'puppet-check/data_parser'
 require_relative 'puppet-check/output_results'
+begin
+  require_relative 'puppet-check/regression_check'
+rescue LoadError
+end
 
 # interfaces from CLI/tasks and to individual parsers
 class PuppetCheck
-  # initialize future parser and style check bools
+  # initialize future parser, style check, and regression check bools
   @future_parser = false
   @style_check = false
+  @smoke_check = false
+  @regression_check = false
 
   # initialize output format option
   @output_format = 'text'
 
+  # initialize octocatalog-diff options
+  @octoconfig = '.octocatalog-diff.cfg.rb'
+  @octonodes = %w(localhost.localdomain)
+
   # initialize diagnostic output arrays
   @error_files = []
   @warning_files = []
@@ -24,7 +34,7 @@ class PuppetCheck
 
   # allow the parser methods read user options and append to the file arrays; allow CLI and tasks write to user options
   class << self
-    attr_accessor :future_parser, :style_check, :output_format, :error_files, :warning_files, :clean_files, :ignored_files, :puppetlint_args, :rubocop_args
+    attr_accessor :future_parser, :style_check, :smoke_check, :regression_check, :output_format, :octoconfig, :octonodes, :error_files, :warning_files, :clean_files, :ignored_files, :puppetlint_args, :rubocop_args
   end
 
   # main runner for PuppetCheck
@@ -38,8 +48,24 @@ class PuppetCheck
     # output the diagnostic results
     PuppetCheck.output_format == 'text' ? OutputResults.text : OutputResults.markup
 
-    # exit code
-    self.class.error_files.empty? ? 0 : 2
+    if self.class.error_files.empty?
+      # perform smoke checks if there were no errors and the user desires
+      begin
+        RegressionCheck.smoke(self.class.octonodes, self.class.octoconfig) if PuppetCheck.smoke_check
+      # smoke check failure? output message and return 2
+      rescue OctocatalogDiff::Errors::CatalogError => err
+        puts 'There was a smoke check error:'
+        puts err
+        2
+      end
+      # perform regression checks if there were no errors and the user desires
+      # RegressionCheck.regression(self.class.octonodes, self.class.octoconfig) if PuppetCheck.regression_check
+      # everything passed? return 0
+      0
+    else
+      # error files? return 2
+      2
+    end
   end
 
   # parse the paths and return the array of files

data/lib/puppet-check/cli.rb

@@ -7,7 +7,7 @@ class PuppetCheck::CLI
   def self.run(args)
     # gather the user arguments
     parse(args)
-    raise 'puppet-check: no paths specified' if args.empty?
+    raise 'puppet-check: no paths specified; try using --help' if args.empty?
 
     # run PuppetCheck
     PuppetCheck.new.run(args)
@@ -21,17 +21,23 @@ class PuppetCheck::CLI
 
       # base options
       opts.on('--version', 'Display the current version.') do
-        puts 'puppet-check 1.3.2'
+        puts 'puppet-check 1.4.0'
         exit 0
       end
 
       # bool options
       opts.on('-f', '--future', 'Enable future parser') { PuppetCheck.future_parser = true }
       opts.on('-s', '--style', 'Enable style checks') { PuppetCheck.style_check = true }
+      opts.on('--smoke', 'Enable smoke testing') { PuppetCheck.smoke_check = true }
+      opts.on('-r', '--regression', 'Enable regression testing (in progress, do not use)') { PuppetCheck.regression_check = true }
 
       # formatting options
       opts.on('-o', '--output format', String, 'Format for results output (default is text): text, json, or yaml') { |arg| PuppetCheck.output_format = arg }
 
+      # octocatalog-diff options
+      opts.on('--octoconfig config_file', String, 'Octocatalog-diff configuration file to use.') { |arg| PuppetCheck.octoconfig = arg }
+      opts.on('-n', '--octonodes node1.example.com,node2.example.com', Array, 'Octocatalog-diff nodes to test catalog on.') { |arg| PuppetCheck.octonodes = arg }
+
       # arguments to style checkers
       opts.on('--puppet-lint arg_one,arg_two', Array, 'Arguments for PuppetLint ignored checks') do |puppetlint_args|
         PuppetCheck.puppetlint_args += puppetlint_args.map { |arg| "--#{arg}" }

data/lib/puppet-check/data_parser.rb

@@ -53,16 +53,33 @@ class DataParser
             errors.push("Required field '#{key}' not found.") unless parsed.key?(key)
           end
 
-          # check for duplicate dependencies and requirements, and that both are an array of hashes if they exist
+          # check requirements and dependencies keys
           %w(requirements dependencies).each do |key|
+            # skip if key is missing or or value is an empty string, array, or hash
             next unless parsed.key?(key)
-            next errors.push("Field '#{key}' is not an array of hashes.") unless (parsed[key].is_a? Array) && (parsed[key].empty? || (parsed[key][0].is_a? Hash))
+            next if parsed[key].empty?
+
+            # check that dependencies and requirements are an array of hashes
+            next errors.push("Field '#{key}' is not an array of hashes.") unless (parsed[key].is_a? Array) && (parsed[key][0].is_a? Hash)
 
+            # check dependencies and requirements values
             names = []
             parsed[key].each do |req_dep|
+              # check for duplicate dependencies and requirements
               name = req_dep['name']
-              errors.push("Duplicate #{key} on #{name}.") if names.include?(name)
+              next errors.push("Duplicate #{key} on #{name}.") if names.include?(name)
               names << name
+
+              # warn and skip if key is missing
+              next warnings.push("'#{req_dep['name']}' is missing a 'version_requirement' key.") if req_dep['version_requirement'].class.to_s == 'NilClass'
+
+              # warn and skip if no upper bound
+              next warnings.push("'#{req_dep['name']}' is missing an upper bound.") unless req_dep['version_requirement'].include?('<')
+
+              # check for semantic versioning
+              if key == 'dependencies'
+                warnings.push("'#{req_dep['name']}' has non-semantic versioning in its 'version_requirement' key.") unless req_dep['version_requirement'] =~ /\d\.\d\.\d.*\d\.\d\.\d/
+              end
             end
           end
 
@@ -90,6 +107,7 @@ class DataParser
               else
                 warnings.push('Recommended field \'operatingsystem\' not found.')
               end
+
               # check for operatingsystemrelease string array
               if parsed['operatingsystem_support'][0].key?('operatingsystemrelease')
                 warnings.push('Field \'operatingsystemrelease\' is not a string array.') unless parsed['operatingsystem_support'][0]['operatingsystemrelease'][0].is_a? String
@@ -101,24 +119,6 @@ class DataParser
             warnings.push('Recommended field \'operatingsystem_support\' not found.')
           end
 
-          # check for requirement and dependency upper bounds
-          %w(requirements dependencies).each do |key|
-            # skip if key is missing or not an array
-            next if parsed[key].empty?
-            next warnings.push("#{key}'s value is not an array.") unless parsed[key].is_a? Array
-            parsed[key].each do |req_dep|
-              # warn and skip if key is missing
-              next warnings.push("'#{req_dep['name']}' is missing a 'version_requirement' key.") if req_dep['version_requirement'].class.to_s == 'NilClass'
-              # warn and skip if no upper bound
-              next warnings.push("'#{req_dep['name']}' is missing an upper bound.") unless req_dep['version_requirement'].include?('<')
-
-              # check for semantic versioning
-              if key == 'dependencies'
-                warnings.push("'#{req_dep['name']}' has non-semantic versioning in its 'version_requirement' key.") unless req_dep['version_requirement'] =~ /\d\.\d\.\d.*\d\.\d\.\d/
-              end
-            end
-          end
-
           # check for spdx license (rubygems/util/licenses for rubygems >= 2.5 in the far future)
           if parsed.key?('license') && !SpdxLicenses.exist?(parsed['license']) && parsed['license'] !~ /[pP]roprietary/
             warnings.push("License identifier '#{parsed['license']}' is not in the SPDX list: http://spdx.org/licenses/")

data/lib/puppet-check/regression_check.rb

@@ -0,0 +1,25 @@
+require 'octocatalog-diff'
+
+# executes smoke and regression tests on catalogs
+class RegressionCheck
+  # smoke testing
+  def self.smoke(interface_nodes, octoconfig)
+    options = RegressionCheck.config(octoconfig)
+    nodes = options.key?(:node) ? [options[:node]] : interface_nodes
+    nodes.each do |node|
+      options[:node] = node
+      OctocatalogDiff::API::V1.catalog(options)
+    end
+  end
+
+  # regression testing
+  # def self.regression(nodes, octoconfig)
+    # options = RegressionCheck.config(octoconfig)
+    # nodes.each { |node| stuff }
+  # end
+
+  # config file loading
+  def self.config(octoconfig)
+    OctocatalogDiff::API::V1.config(filename: octoconfig)
+  end
+end

data/spec/octocatalog-diff/facts.yaml

@@ -0,0 +1,4 @@
+---
+name: good.example.com
+values:
+  domain: example.com

data/spec/octocatalog-diff/hiera.yaml

@@ -0,0 +1,7 @@
+---
+:backends:
+  - yaml
+:yaml:
+  :datadir: "hieradata"
+:hierarchy:
+  - "common"

data/spec/octocatalog-diff/manifests/site.pp

@@ -0,0 +1,3 @@
+node good.example.com {
+  notify { 'hello world': }
+}

data/spec/octocatalog-diff/octocatalog-diff.cfg.rb

@@ -0,0 +1,18 @@
+# This is a configuration file for octocatalog-diff (https://github.com/github/octocatalog-diff).
+module OctocatalogDiff
+  # Configuration class.
+  class Config
+    def self.config
+      settings = {}
+      octocatalog_diff_dir = File.dirname(__FILE__) + '/'
+
+      settings[:hiera_config] = octocatalog_diff_dir + 'hiera.yaml'
+      settings[:hiera_path] = octocatalog_diff_dir + 'hieradata'
+      settings[:fact_file] = octocatalog_diff_dir + 'facts.yaml'
+      settings[:puppet_binary] = '/usr/local/bin/puppet'
+      settings[:bootstrapped_to_dir] = octocatalog_diff_dir
+
+      settings
+    end
+  end
+end

data/spec/puppet-check/cli_spec.rb

@@ -4,7 +4,7 @@ require_relative '../../lib/puppet-check/cli'
 describe PuppetCheck::CLI do
   context '.run' do
     it 'raises an error if no paths were specified' do
-      expect { PuppetCheck::CLI.run(%w(-s -f)) }.to raise_error(RuntimeError, 'puppet-check: no paths specified')
+      expect { PuppetCheck::CLI.run(%w(-s -f)) }.to raise_error(RuntimeError, 'puppet-check: no paths specified; try using --help')
     end
   end
 
@@ -13,12 +13,16 @@ describe PuppetCheck::CLI do
       expect { PuppetCheck::CLI.parse(%w(-s -f -asdf foo)) }.to raise_error(OptionParser::InvalidOption)
     end
 
-    it 'allows future parser and style check to be enabled' do
+    it 'allows future parser, style, smoke, and regression checks to be enabled' do
       PuppetCheck.future_parser = false
       PuppetCheck.style_check = false
-      PuppetCheck::CLI.parse(%w(-s -f foo))
+      PuppetCheck.smoke_check = false
+      PuppetCheck.regression_check = false
+      PuppetCheck::CLI.parse(%w(-s -f -r --smoke foo))
       expect(PuppetCheck.future_parser).to eql(true)
       expect(PuppetCheck.style_check).to eql(true)
+      expect(PuppetCheck.smoke_check).to eql(true)
+      expect(PuppetCheck.regression_check).to eql(true)
     end
 
     it 'correctly parses a formatting option' do
@@ -27,6 +31,14 @@ describe PuppetCheck::CLI do
       expect(PuppetCheck.output_format).to eql('text')
     end
 
+    it 'correctly parses octocatalog-diff options' do
+      PuppetCheck.octoconfig = ''
+      PuppetCheck.octonodes = []
+      PuppetCheck::CLI.parse(%w(--octoconfig config.cfg.rb --octonodes server1,server2))
+      expect(PuppetCheck.octoconfig).to eql('config.cfg.rb')
+      expect(PuppetCheck.octonodes).to eql(%w(server1 server2))
+    end
+
     it 'correctly parses PuppetLint arguments' do
       PuppetCheck.puppetlint_args = []
       PuppetCheck::CLI.parse(%w(--puppet-lint puppetlint-arg-one,puppetlint-arg-two foo))

data/spec/puppet-check/data_parser_spec.rb

@@ -45,13 +45,13 @@ describe DataParser do
     it 'puts a bad style metadata json file in the warning files array' do
       DataParser.json([fixtures_dir + 'metadata_style/metadata.json'])
       expect(PuppetCheck.error_files).to eql([])
-      expect(PuppetCheck.warning_files[0]).to match(%r{^#{fixtures_dir}metadata_style/metadata.json:\n.*operatingsystem_support.*\n'pe' is missing an upper bound.\nLicense identifier})
+      expect(PuppetCheck.warning_files[0]).to match(%r{^#{fixtures_dir}metadata_style/metadata.json:\n'pe' is missing an upper bound.\n.*operatingsystem_support.*\nLicense identifier})
       expect(PuppetCheck.clean_files).to eql([])
     end
     it 'puts another bad style metadata json file in the warning files array' do
       DataParser.json([fixtures_dir + 'metadata_style_two/metadata.json'])
       expect(PuppetCheck.error_files).to eql([])
-      expect(PuppetCheck.warning_files[0]).to match(%r{^#{fixtures_dir}metadata_style_two/metadata.json:\n.*operatingsystem.*\n.*operatingsystemrelease.*\n'puppetlabs/one' has non-semantic versioning.*\n'puppetlabs/two' is missing an upper bound})
+      expect(PuppetCheck.warning_files[0]).to match(%r{^#{fixtures_dir}metadata_style_two/metadata.json:\n'puppetlabs/one' has non-semantic versioning.*\n'puppetlabs/two' is missing an upper bound\.\n.*operatingsystem.*\n.*operatingsystemrelease})
       expect(PuppetCheck.clean_files).to eql([])
     end
     it 'puts a good json file in the clean files array' do

data/spec/puppet-check/puppet_parser_spec.rb

@@ -11,8 +11,12 @@ describe PuppetParser do
   context '.manifest' do
     it 'puts a bad syntax Puppet manifest in the error files array' do
       PuppetParser.manifest([fixtures_dir + 'manifests/syntax.pp'], false, false, [])
-      # expect(subject.instance_variable_get(:@error_files)[0]).to match(%r{^#{fixtures_dir}manifests/syntax.pp:.*syntax error})
-      expect(PuppetCheck.error_files[0]).to match(%r{^#{fixtures_dir}manifests/syntax.pp:\nThis Variable has no effect.*\nIllegal variable name})
+      # stupid Puppet deprecation warning
+      if RUBY_VERSION.to_f < 2.1
+        expect(PuppetCheck.error_files[0]).to match(%r{^#{fixtures_dir}manifests/syntax.pp:\nSupport for ruby version.*\n.*\nThis Variable has no effect.*\nIllegal variable name})
+      else
+        expect(PuppetCheck.error_files[0]).to match(%r{^#{fixtures_dir}manifests/syntax.pp:\nThis Variable has no effect.*\nIllegal variable name})
+      end
       expect(PuppetCheck.warning_files).to eql([])
       expect(PuppetCheck.clean_files).to eql([])
     end

data/spec/puppet-check/regression_check_spec.rb

@@ -0,0 +1,36 @@
+require_relative '../spec_helper'
+require_relative '../../lib/puppet-check/regression_check'
+
+# once a good config is loaded, bad configs would have no effect so failures are tested first; config is also tested before compilation so that the good config can be loaded at the end and used for compilation and regression tests
+describe RegressionCheck do
+  context '.config' do
+    it 'raise an appropriate error if the file is malformed' do
+      expect { RegressionCheck.config(fixtures_dir + 'metadata.json') }.to raise_error(OctocatalogDiff::Errors::ConfigurationFileContentError, 'Configuration must define OctocatalogDiff::Config!')
+    end
+    it 'loads in a good octocatalog-diff config file' do
+      expect { RegressionCheck.config(octocatalog_diff_dir + 'octocatalog-diff.cfg.rb') }.not_to raise_exception
+    end
+    it 'loads in the settings from the file correctly' do
+      #
+    end
+  end
+
+  context '.smoke' do
+    # API requires :puppet_binary, which needs an absolute path, which is absurdly difficult to pull off for a matrix of travis ci rvm tests
+    unless File.directory?('/home/travis')
+      it 'returns a pass for a successful catalog compilation' do
+        expect { RegressionCheck.smoke(['good.example.com'], "#{octocatalog_diff_dir}octocatalog-diff.cfg.rb") }.not_to raise_exception
+      end
+    end
+    it 'returns a failure for a catalog with an error' do
+      expect { RegressionCheck.smoke(['does_not_exist.example.com'], "#{octocatalog_diff_dir}octocatalog-diff.cfg.rb") }.to raise_error(OctocatalogDiff::Errors::CatalogError)
+    end
+    it 'returns a failure for a good and bad catalog' do
+      # RegressionCheck.smoke(['good.example.com', 'syntax_error.example.com'], "#{fixtures_dir}octocatalog-diff.cfg.rb")
+    end
+  end
+
+  context '.regression' do
+    #
+  end
+end

data/spec/puppet-check_spec.rb

@@ -3,19 +3,23 @@ require_relative '../lib/puppet-check'
 
 describe PuppetCheck do
   context 'self' do
-    it 'future parser can be altered' do
+    it 'settings can be altered' do
       PuppetCheck.future_parser = true
       expect(PuppetCheck.future_parser).to eql(true)
-    end
-    it 'style check can be altered' do
       PuppetCheck.style_check = true
       expect(PuppetCheck.style_check).to eql(true)
-    end
-    it 'puppet lint arguments can be altered' do
+      PuppetCheck.smoke_check = true
+      expect(PuppetCheck.smoke_check).to eql(true)
+      PuppetCheck.regression_check = true
+      expect(PuppetCheck.regression_check).to eql(true)
+      PuppetCheck.output_format = 'text'
+      expect(PuppetCheck.output_format).to eql('text')
+      PuppetCheck.octoconfig = '.octocatalog-diff.cfg.rb'
+      expect(PuppetCheck.octoconfig).to eql('.octocatalog-diff.cfg.rb')
+      PuppetCheck.octonodes = %w(localhost.localdomain)
+      expect(PuppetCheck.octonodes).to eql(%w(localhost.localdomain))
       PuppetCheck.puppetlint_args = ['--puppetlint-arg-one', '--puppetlint-arg-two']
       expect(PuppetCheck.puppetlint_args).to eql(['--puppetlint-arg-one', '--puppetlint-arg-two'])
-    end
-    it 'rubocop arguments can be altered' do
       PuppetCheck.rubocop_args = ['--rubocop-arg-one', '--rubocop-arg-two']
       expect(PuppetCheck.rubocop_args).to eql(['--rubocop-arg-one', '--rubocop-arg-two'])
     end

data/spec/spec_helper.rb

@@ -5,6 +5,7 @@ module Variables
   extend RSpec::SharedContext
 
   let(:fixtures_dir) { File.dirname(__FILE__) + '/fixtures/' }
+  let(:octocatalog_diff_dir) { File.dirname(__FILE__) + '/octocatalog-diff/' }
 end
 
 RSpec.configure do |config|

data/spec/system/system_spec.rb

@@ -5,7 +5,12 @@ require_relative '../../lib/puppet-check/tasks'
 
 describe PuppetCheck do
   context 'executed as a system from the CLI with arguments and various files to be processed' do
-    let(:cli) { PuppetCheck::CLI.run(%w(-s --puppet-lint no-hard_tabs-check,no-140chars-check --rubocop Metrics/LineLength,Style/Encoding .)) }
+    # see regression_check_spec
+    if File.directory?('/home/travis')
+      let(:cli) { PuppetCheck::CLI.run(%w(-s --puppet-lint no-hard_tabs-check,no-140chars-check --rubocop Metrics/LineLength,Style/Encoding .)) }
+    else
+      let(:cli) { PuppetCheck::CLI.run(%w(-s --puppet-lint no-hard_tabs-check,no-140chars-check --rubocop Metrics/LineLength,Style/Encoding --smoke -n good.example.com --octoconfig spec/octocatalog-diff/octocatalog-diff.cfg.rb .)) }
+    end
 
     it 'outputs diagnostic results correctly after processing all of the files' do
       Dir.chdir(fixtures_dir)
@@ -13,8 +18,14 @@ describe PuppetCheck do
       expect { cli }.not_to raise_exception
 
       expect(PuppetCheck.error_files.length).to eql(8)
-      expect(PuppetCheck.warning_files.length).to eql(9)
-      expect(PuppetCheck.clean_files.length).to eql(11)
+      # stupid Puppet deprecation warning
+      if RUBY_VERSION.to_f < 2.1
+        expect(PuppetCheck.warning_files.length).to eql(10)
+        expect(PuppetCheck.clean_files.length).to eql(10)
+      else
+        expect(PuppetCheck.warning_files.length).to eql(9)
+        expect(PuppetCheck.clean_files.length).to eql(11)
+      end
       expect(PuppetCheck.ignored_files.length).to eql(1)
 
       expect(cli).to eql(2)
@@ -34,6 +45,12 @@ describe PuppetCheck do
       PuppetCheck.clean_files = []
       PuppetCheck.ignored_files = []
       PuppetCheck.style_check = true
+      # see regression_check_spec
+      unless File.directory?('/home/travis')
+        PuppetCheck.smoke_check = true
+        PuppetCheck.octonodes = %w(good.example.com)
+        PuppetCheck.octoconfig = 'spec/octocatalog-diff/octocatalog-diff.cfg.rb'
+      end
 
       expect { tasks }.not_to raise_exception
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: puppet-check
 version: !ruby/object:Gem::Version
-  version: 1.3.2
+  version: 1.4.0
 platform: ruby
 authors:
 - Matt Schuchard
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-01-22 00:00:00.000000000 Z
+date: 2017-05-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: puppet
@@ -48,16 +48,22 @@ dependencies:
   name: puppet-lint
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+    - - "<"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: 1.1.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
 - !ruby/object:Gem::Dependency
   name: spdx-licenses
   requirement: !ruby/object:Gem::Requirement
@@ -106,6 +112,20 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: '13'
+- !ruby/object:Gem::Dependency
+  name: octocatalog-diff
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 description: Puppet Check is a gem that provides a comprehensive, streamlined, and
   efficient analysis of the syntax, style, and validity of your entire Puppet code
   and data.
@@ -124,6 +144,7 @@ files:
 - lib/puppet-check/data_parser.rb
 - lib/puppet-check/output_results.rb
 - lib/puppet-check/puppet_parser.rb
+- lib/puppet-check/regression_check.rb
 - lib/puppet-check/rspec_puppet_support.rb
 - lib/puppet-check/ruby_parser.rb
 - lib/puppet-check/tasks.rb
@@ -158,10 +179,15 @@ files:
 - spec/fixtures/templates/style.erb
 - spec/fixtures/templates/syntax.epp
 - spec/fixtures/templates/syntax.erb
+- spec/octocatalog-diff/facts.yaml
+- spec/octocatalog-diff/hiera.yaml
+- spec/octocatalog-diff/manifests/site.pp
+- spec/octocatalog-diff/octocatalog-diff.cfg.rb
 - spec/puppet-check/cli_spec.rb
 - spec/puppet-check/data_parser_spec.rb
 - spec/puppet-check/output_results_spec.rb
 - spec/puppet-check/puppet_parser_spec.rb
+- spec/puppet-check/regression_check_spec.rb
 - spec/puppet-check/rspec_puppet_support_spec.rb
 - spec/puppet-check/ruby_parser_spec.rb
 - spec/puppet-check/tasks_spec.rb
@@ -225,10 +251,15 @@ test_files:
 - spec/fixtures/templates/style.erb
 - spec/fixtures/templates/syntax.epp
 - spec/fixtures/templates/syntax.erb
+- spec/octocatalog-diff/facts.yaml
+- spec/octocatalog-diff/hiera.yaml
+- spec/octocatalog-diff/manifests/site.pp
+- spec/octocatalog-diff/octocatalog-diff.cfg.rb
 - spec/puppet-check/cli_spec.rb
 - spec/puppet-check/data_parser_spec.rb
 - spec/puppet-check/output_results_spec.rb
 - spec/puppet-check/puppet_parser_spec.rb
+- spec/puppet-check/regression_check_spec.rb
 - spec/puppet-check/rspec_puppet_support_spec.rb
 - spec/puppet-check/ruby_parser_spec.rb
 - spec/puppet-check/tasks_spec.rb