puppet-catalog-test 0.1.0

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Rene Lengwinat
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,176 @@
+# Test all your puppet catalogs for compiler errors
+
+## Installation
+
+    gem install puppet-catalog-tests
+
+## Usage
+```bash
+$ puppet-catalog-test -h
+USAGE: puppet-catalog-test [options]
+    -m, --module-paths PATHS         Location of puppet modules, separated by collon
+    -M, --manifest-path PATH         Location of the puppet manifests (site.pp)
+    -f, --filter PATTERN             Filter test cases by pattern
+    -s, --scenario FILE              Scenario definition to use
+    -h, --help                       Show this message
+```
+## Examples
+
+### CLI - successfull compile run
+```bash
+$ puppet-catalog-test -m test/cases/working/modules -M test/cases/working/site.pp
+[INFO] Using puppet 3.0.2
+[PASSED]  foo (compile time: 0.168182 seconds)
+[PASSED]  default (compile time: 0.003451 seconds)
+
+----------------------------------------
+Compiled 2 catalogs in 0.1717 seconds (avg: 0.0858 seconds)
+```
+
+### CLI - failed compile run
+```bash
+$ puppet-catalog-test -m test/cases/failing/modules -M test/cases/failing/site.pp
+[INFO] Using puppet 3.0.2
+[FAILED]  foo (compile time: 0.17113 seconds)
+[FAILED]  default (compile time: 0.002951 seconds)
+
+----------------------------------------
+Compiled 2 catalogs in 0.1741 seconds (avg: 0.0871 seconds)
+2 test cases failed.
+
+ [F] foo:
+     Duplicate declaration: Package[myapp-pkg] is already declared in file /Users/rlengwin/devel/github/puppet-catalog-test/test/cases/failing/modules/myapp/manifests/init.pp at line 4; cannot redeclare on node foo
+
+ [F] default:
+     Duplicate declaration: Package[myapp-pkg] is already declared in file /Users/rlengwin/devel/github/puppet-catalog-test/test/cases/failing/modules/myapp/manifests/init.pp at line 4; cannot redeclare on node default
+
+2 / 2 FAILED
+```
+
+## Rake integration
+
+### Testing all catalogs with default facts
+
+When not setting any filters or scenarios puppet-catalog-test will test all nodes defined in site.pp.
+
+```ruby
+require 'puppet-catalog-test'
+
+namespace :catalog do
+  PuppetCatalogTest::RakeTask.new(:all) do |t|
+    t.module_paths = ["modules"]
+    t.manifest_path = File.join("scripts", "site.pp")
+
+    t.filter = ENV["filter"] if ENV["filter"]
+  end
+end
+```
+
+In the case above no facts weren't defined so puppet-catalog-test falls back to some basic facts to satisfy the most basic requirements of puppet. Currently these facts are:
+
+```ruby
+{
+  'architecture' => 'x86_64',
+  'ipaddress' => '127.0.0.1',
+  'local_run' => 'true',
+  'disable_asserts' => 'true',
+  'interfaces' => 'eth0'
+}
+```
+
+### Testing all catalogs with custom facts
+
+It is also possible to define a custom set of facts. In this case the fallback facts listed in previous example won't be used.
+
+*NOTE:* When using custom facts the fact 'fqdn' always has to be set!
+
+```ruby
+require 'puppet-catalog-test'
+
+namespace :catalog do
+  PuppetCatalogTest::RakeTask.new(:all) do |t|
+    t.module_paths = ["modules"]
+    t.manifest_path = File.join("scripts", "site.pp")
+    t.facts = {
+      "fqdn" => "foo.local",
+      "operatingsystem" => "RedHat"
+    }
+
+    t.filter = ENV["filter"] if ENV["filter"]
+  end
+end
+```
+
+## Scenario testing
+
+Scenarios allow testing of more complex catalogs, e.g. having conditional branches depending on custom facts.
+
+```ruby
+require 'puppet-catalog-test'
+
+namespace :catalog do
+  PuppetCatalogTest::RakeTask.new(:scenarios) do |t|
+    t.module_paths = [File.join("modules")]
+    t.manifest_path = File.join("scripts", "site.pp")
+
+    t.scenario_yaml = "scenarios.yml"
+
+    t.filter = ENV["filter"] if ENV["filter"]
+  end
+end
+```
+
+```yaml
+__default_facts: &default_facts
+  architecture: x86_64
+  ipaddress: 127.0.0.1
+  operatingsystem: SLES
+  operatingsystemrelease: 11
+  local_run: true
+  disable_asserts: true
+  interfaces: eth0
+
+SLES_tomcat:
+  <<: *default_facts
+  fqdn: tomcat-a001.foo.local
+
+REDHAT_tomcat:
+  <<: *default_facts
+  fqdn: tomcat-a001.foo.local
+  operatingsystem: RedHat
+
+SLES_db-dev:
+  <<: *default_facts
+  fqdn: db-a001.foo.local
+
+REDHAT_db-dev:
+  <<: *default_facts
+  fqdn: db-a001.foo.local
+  operatingsystem: RedHat
+```
+
+*NOTE:* Scenarios starting with two underscores (like __default_facts) will be ignored.
+
+## Reporters (output plugins)
+
+Per default puppet-catalog-test will use the StdoutReporter which will represent the result like in the examples above. Besides this you can use in your own Reporter.
+
+Puppet-Catalog-Test also ships a JunitXmlReporter which creates a junit compatible xml report.
+
+```ruby
+require 'puppet-catalog-test'
+require 'puppet-catalog-test/junit_xml_reporter'
+
+namespace :catalog do
+  PuppetCatalogTest::RakeTask.new(:all) do |t|
+    t.module_paths = [File.join("modules")]
+    t.manifest_path = File.join("scripts", "site.pp")
+
+    t.reporter = PuppetCatalogTest::JunitXmlReporter.new("puppet-vagrant-playground", "reports/puppet_catalog.xml")
+  end
+end
+```
+
+# Credits
+
+Code is based upon the previous work done on https://github.com/oldNoakes/puppetTesting

data/Rakefile

@@ -0,0 +1,13 @@
+require "rake/testtask"
+require "bundler/gem_tasks"
+
+desc "Clean workspace"
+task :clean do
+  sh "rm -vrf *.gem pkg/"
+end
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/*_test.rb']
+  t.verbose = true
+end

data/bin/puppet-catalog-test

@@ -0,0 +1,50 @@
+#!/usr/bin/env ruby
+
+require "puppet-catalog-test"
+require "optparse"
+
+options = {
+  :filter => PuppetCatalogTest::DEFAULT_FILTER
+}
+
+opt_parser = OptionParser.new do |opts|
+  opts.banner = "USAGE: #{File.basename($0)} [options]"
+
+  opts.on("-m", "--module-paths PATHS", "Location of puppet modules, separated by collon") do |arg|
+    options[:module_paths] = arg.split(":")
+  end
+
+  opts.on("-M", "--manifest-path PATH", "Location of the puppet manifests (site.pp)") do |arg|
+    options[:manifest_path] = arg
+  end
+
+  opts.on("-f", "--filter PATTERN", "Filter test cases by pattern") do |arg|
+    options[:filter] = arg
+  end
+
+  opts.on("-s", "--scenario FILE", "Scenario definition to use") do |arg|
+    options[:scenario] = arg
+  end
+
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts
+    exit
+  end
+end
+
+opt_parser.parse!
+
+if options.empty?
+  puts opt_parser.help
+  exit
+end
+
+pct = PuppetCatalogTest::TestRunner.new(options[:manifest_path], options[:module_paths])
+
+if options[:scenario]
+  pct.load_scenario_yaml(options[:scenario], options[:filter])
+else
+  pct.load_all(options[:filter])
+end
+
+pct.run_tests!

data/lib/puppet-catalog-test.rb

@@ -0,0 +1,8 @@
+module PuppetCatalogTest
+  VERSION = "0.1.0"
+
+  DEFAULT_FILTER = /.*/
+
+  require "puppet-catalog-test/test_runner"
+  require "puppet-catalog-test/rake_task"
+end

data/lib/puppet-catalog-test/junit_xml_reporter.rb

@@ -0,0 +1,35 @@
+require "builder"
+
+module PuppetCatalogTest
+  class JunitXmlReporter < PuppetCatalogTest::StdoutReporter
+    def initialize(project_name, report_file)
+      @project_name = project_name
+      @report_file = report_file
+
+      target_dir = File.dirname(report_file)
+
+      FileUtils.mkdir_p(target_dir)
+
+      @out = $stdout
+    end
+
+    def summarize(tr)
+      failed_nodes = tr.test_cases.select { |tc| tc.passed == false }
+      builder = Builder::XmlMarkup.new
+
+      xml = builder.testsuite(:failures => failed_nodes.size, :tests => tr.test_cases.size) do |ts|
+        tr.test_cases.each do |tc|
+          ts.testcase(:classname => @project_name, :name => tc.name, :time => tc.duration) do |tc_node|
+            if tc.error
+              tc_node.failure tc.error
+            end
+          end
+        end
+      end
+
+      File.open(@report_file, "w") do |fp|
+        fp.puts xml
+      end
+    end
+  end
+end

data/lib/puppet-catalog-test/rake_task.rb

@@ -0,0 +1,62 @@
+require 'rake'
+require 'rake/tasklib'
+
+module PuppetCatalogTest
+  class RakeTask < ::Rake::TaskLib
+    include ::Rake::DSL if defined?(::Rake::DSL)
+
+    attr_accessor :module_paths
+    attr_accessor :manifest_path
+    attr_accessor :scenario_yaml
+    attr_accessor :filter
+    attr_accessor :facts
+    attr_accessor :reporter
+
+    def initialize(name, &task_block)
+      desc "Compile all puppet catalogs" unless ::Rake.application.last_comment
+
+      @filter = PuppetCatalogTest::DEFAULT_FILTER
+
+      task name do
+        task_block.call(self) if task_block
+        setup
+      end
+    end
+
+    def setup
+      pct = TestRunner.new(@manifest_path, @module_paths)
+
+      if @scenario_yaml
+        pct.load_scenario_yaml(@scenario_yaml, @filter)
+      else
+        nodes = pct.collect_puppet_nodes(@filter)
+        test_facts = @facts || fallback_facts
+
+        nodes.each do |nodename|
+          facts = test_facts.merge({
+            'hostname' => nodename,
+            'fqdn' => "#{nodename}.localdomain"
+          })
+
+          pct.add_test_case(nodename, facts)
+        end
+      end
+
+      pct.reporter = @reporter if @reporter
+
+      pct.run_tests!
+    end
+
+    private
+
+    def fallback_facts
+      {
+        'architecture' => 'x86_64',
+        'ipaddress' => '127.0.0.1',
+        'local_run' => 'true',
+        'disable_asserts' => 'true',
+        'interfaces' => 'eth0'
+      }
+    end
+  end
+end

data/lib/puppet-catalog-test/stdout_reporter.rb

@@ -0,0 +1,42 @@
+module PuppetCatalogTest
+  class StdoutReporter
+    def initialize(stdout_target = $stdout)
+      @out = stdout_target
+    end
+
+    def report_passed_test_case(tc)
+      @out.puts "[PASSED]  #{tc.name} (compile time: #{tc.duration} seconds)"
+    end
+
+    def report_failed_test_case(tc)
+      @out.puts "[FAILED]  #{tc.name} (compile time: #{tc.duration} seconds)"
+    end
+
+    def summarize(test_run)
+      failed_cases = test_run.test_cases.select { |tc| tc.passed == false }
+      avg_time = test_run.total_duration / test_run.test_cases.size
+
+      @out.puts
+      @out.puts "-" * 40
+
+      @out.puts "Compiled %d catalogs in %.4f seconds (avg: %.4f seconds)" % [
+        test_run.test_cases.size,
+        test_run.total_duration,
+        avg_time
+      ]
+
+      if !failed_cases.empty?
+        @out.puts "#{failed_cases.size} test cases failed."
+        @out.puts
+
+        failed_cases.each do |tc|
+          @out.puts " [F] #{tc.name}:"
+          @out.puts "     #{tc.error}"
+          @out.puts
+        end
+
+        @out.puts "#{failed_cases.size} / #{test_run.test_cases.size} FAILED"
+      end
+    end
+  end
+end

data/lib/puppet-catalog-test/test_case.rb

@@ -0,0 +1,4 @@
+module PuppetCatalogTest
+  class TestCase < Struct.new(:name, :facts, :passed, :error, :duration)
+  end
+end

data/lib/puppet-catalog-test/test_runner.rb

@@ -0,0 +1,131 @@
+require "yaml"
+require "puppet"
+
+require "puppet-catalog-test/test_case"
+require "puppet-catalog-test/stdout_reporter"
+
+module PuppetCatalogTest
+  class TestRunner
+    attr_accessor :exit_on_fail
+    attr_accessor :reporter
+
+    attr_reader :test_cases
+    attr_reader :total_duration
+
+    def initialize(manifest_path, module_paths, stdout_target = $stdout)
+      @test_cases = []
+      @exit_on_fail = true
+      @out = stdout_target
+
+      @reporter = StdoutReporter.new(stdout_target)
+
+      @total_duration = nil
+
+      raise ArgumentError, "[ERROR] manifest_path must be specified" if !manifest_path
+      raise ArgumentError, "[ERROR] manifest_path (#{manifest_path}) does not exist" if !FileTest.exist?(manifest_path)
+
+      raise ArgumentError, "[ERROR] module_path must be specified" if !module_paths
+      module_paths.each do |mp|
+        raise ArgumentError, "[ERROR] module_path (#{mp}) does not exist" if !FileTest.directory?(mp)
+      end
+
+      Puppet.settings.handlearg("--config", ".")
+      Puppet.settings.handlearg("--manifest", manifest_path)
+
+      module_path = module_paths.join(":")
+
+      Puppet.settings.handlearg("--modulepath", module_path)
+
+      Puppet.parse_config
+    end
+
+    def load_scenario_yaml(yaml_file, filter = nil)
+      scenarios = YAML.load_file(yaml_file)
+
+      scenarios.each do |tc_name, facts|
+        next if tc_name =~ /^__/
+        next if filter && !tc_name.match(filter)
+
+        add_test_case(tc_name, facts)
+      end
+    end
+
+    def load_all(filter = PuppetCatalogTest::DEFAULT_FILTER, facts = {})
+      nodes = collect_puppet_nodes(filter)
+
+      nodes.each do |n|
+        node_facts = facts.dup
+
+        if !node_facts.has_key?('fqdn')
+          node_facts['fqdn'] = n
+        end
+
+        add_test_case(n, node_facts)
+      end
+    end
+
+    def add_test_case(tc_name, facts)
+      tc = TestCase.new
+      tc.name = tc_name
+      tc.facts = facts
+
+      @test_cases << tc
+    end
+
+    def compile_catalog(node_fqdn, facts = {})
+      hostname = node_fqdn.split('.').first
+      facts['hostname'] = hostname
+
+      node = Puppet::Node.new(hostname)
+      node.merge(facts)
+
+      Puppet::Parser::Compiler.compile(node)
+    end
+
+    def collect_puppet_nodes(filter)
+      parser = Puppet::Parser::Parser.new("environment")
+      nodes = parser.environment.known_resource_types.nodes.keys
+      nodes.select { |node| node.match(filter) }
+    end
+
+    def run_tests!
+      @out.puts "[INFO] Using puppet #{Puppet::PUPPETVERSION}"
+
+      run_start = Time.now
+
+      @test_cases.each do |tc|
+        begin
+          tc_start_time = Time.now
+
+          if tc.facts['fqdn'].nil?
+            raise "fact 'fqdn' must be defined"
+          else
+            compile_catalog(tc.facts['fqdn'], tc.facts)
+            tc.duration = Time.now - tc_start_time
+
+            tc.passed = true
+
+            @reporter.report_passed_test_case(tc)
+          end
+        rescue => error
+          tc.duration = Time.now - tc_start_time
+          tc.error = error.message
+          tc.passed = false
+
+          @reporter.report_failed_test_case(tc)
+        end
+      end
+
+      @total_duration = Time.now - run_start
+
+      @reporter.summarize(self)
+
+      if test_cases.any? { |tc| tc.passed == false }
+        exit 1 if @exit_on_fail
+        return false
+      end
+
+      true
+    end
+  end
+end

data/puppet-catalog-test.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |s|
+  s.name = 'puppet-catalog-test'
+  s.version = '0.1.0'
+  s.homepage = 'https://github.com/invadersmustdie/puppet-catalog-test/'
+  s.summary = 'Tests all your puppet catalogs for compiler errors'
+  s.description = 'Tests all your puppet catalogs for compiler errors.'
+
+  s.executables = ['puppet-catalog-test']
+
+  s.files = [
+    'bin/puppet-catalog-test',
+    'LICENSE',
+    'Rakefile',
+    'README.md',
+    'puppet-catalog-test.gemspec'
+  ]
+
+  s.files += Dir["lib/**/*"]
+
+  s.add_dependency 'puppet'
+
+  s.authors = ['Rene Lengwinat']
+  s.email = 'rene.lengwinat@googlemail.com'
+end

metadata

@@ -0,0 +1,72 @@
+--- !ruby/object:Gem::Specification
+name: puppet-catalog-test
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Rene Lengwinat
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-01-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: puppet
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Tests all your puppet catalogs for compiler errors.
+email: rene.lengwinat@googlemail.com
+executables:
+- puppet-catalog-test
+extensions: []
+extra_rdoc_files: []
+files:
+- bin/puppet-catalog-test
+- LICENSE
+- Rakefile
+- README.md
+- puppet-catalog-test.gemspec
+- lib/puppet-catalog-test/junit_xml_reporter.rb
+- lib/puppet-catalog-test/rake_task.rb
+- lib/puppet-catalog-test/stdout_reporter.rb
+- lib/puppet-catalog-test/test_case.rb
+- lib/puppet-catalog-test/test_runner.rb
+- lib/puppet-catalog-test.rb
+homepage: https://github.com/invadersmustdie/puppet-catalog-test/
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Tests all your puppet catalogs for compiler errors
+test_files: []