kafo 3.0.0 → 5.0.1

data/README.md

@@ -79,8 +79,8 @@ All configuration related files are to be found in the config directory.
 
 You can supply custom location for your scenario configuration and answer files
 and change configuration and answer files names using options:
-```
-kafofy --help
+```console
+$ kafofy --help
 Usage: kafofy [options]
     -c, --config_dir DIR            location of the scenarios configuration directory [./config/installer-scenarios.d/]
     -s, --scenario SCENARIO          scenario file name (without extension) [default]
@@ -1004,6 +1004,21 @@ The cache will be skipped if the file modification time of the manifest is
 greater than the mtime recorded in the cache. Using `--parser-cache` will force
 the use of an outdated cache, but this should be used with caution.
 
+## Facts
+
+Kafo provides a structured fact describing the state. This fact is only present
+during the Puppet run. Currently it's the scenario id and name where the id is
+the same as passed via --scenario by the user and matches the scenario filename
+with an extension. The name is a human readable version.
+
+```yaml
+---
+kafo:
+  scenario:
+    id: foreman_proxy
+    name: Foreman Proxy
+```
+
 ## Configuring Hiera
 
 Kafo uses Hiera to include classes and pass parameters to classes using data
@@ -1018,22 +1033,20 @@ The contents of this file are as per the
 [hiera.yaml docs](https://docs.puppet.com/hiera/latest/configuring.html).
 Only Hiera version 5 is supported.
 
-When running Puppet, Kafo will copy the Hiera config to a temporary location.
-Relative data directories will be changed to absolute paths. A
-`kafo_answers.yaml` file will be generated containing _all_ default and
-overriden values for parameters managed by Kafo. This may change in the future
-to allow a more complex hierarchy.
+An answers file will be generated containing _all_ default and overriden values
+for parameters managed by Kafo. During the run this is available as a
+`kafo.scenario.answer_file` fact. This may change in the future to allow a more
+complex hierarchy.
 
-The hierarchy can contain a special value with the name `Kafo Answers`. The
-exact values will be rewritten by Kafo, but it can be used to determine when
-the Kafo answers are loaded. Note the name is case sensitive. When it's
-missing, it will be added.
+The hierarchy must contain the path `%{facts.kafo.scenario.answer_file}`. This
+contains all answers in a temporary location.
 
 As an example, a hierarchy could be set up with:
 
 ```yaml
 hierarchy:
   - name: "Kafo Answers"
+    path: "%{facts.kafo.scenario.answer_file}"
   - name: "Other YAML hierarchy levels"
     paths:
       - "family/%{facts.os.family}.yaml"
@@ -1048,6 +1061,7 @@ hierarchy:
     datadir: "custom"
     path: "override.yaml"
   - name: "Kafo Answers"
+    path: "%{facts.kafo.scenario.answer_file}"
   - name: "Other YAML hierarchy levels"
     datadir: "data"
     paths:
@@ -1058,6 +1072,17 @@ hierarchy:
 This would give precedence to all Kafo-managed parameter values, but for any
 others, would check for values per OS family, followed by a `common.yaml` file.
 
+The scenario id is also available as a fact which can be used to provide
+scenario specific overrides for unmanaged modules.
+
+```yaml
+hierarchy:
+  - name: "Kafo Answers"
+    path: "%{facts.kafo.scenario.answer_file}"
+  - name: "Scenario defaults"
+    path: "scenario/%{facts.kafo.scenario.id}.yaml"
+```
+
 [Migration from Hiera version 3](https://puppet.com/docs/puppet/4.9/hiera_migrate_v3_yaml.html)
 is documented by Puppet.
 

data/Rakefile

@@ -2,6 +2,12 @@ require 'rake/testtask'
 require "bundler/gem_tasks"
 load 'tasks/jenkins.rake'
 
+Rake::TestTask.new('test:ruby') do |t|
+  t.libs << 'lib' << 'test'
+  t.test_files = FileList['test/**/*_test.rb']
+  t.verbose = true
+end
+
 Rake::TestTask.new('test:unit') do |t|
   t.libs << 'lib' << 'test'
   t.test_files = FileList['test/kafo/**/*_test.rb']
@@ -27,4 +33,4 @@ end
 
 CLEAN.include 'test/tmp'
 
-task :test => ['test:unit', 'test:acceptance', 'test:puppet_modules']
+task :test => ['test:ruby', 'test:puppet_modules']

data/doc/kafo_run.uml

@@ -68,7 +68,7 @@
   #Greenyellow:**pre_values**|
   :load values from answer file;
   :load values from previous answer file;
-  note right: if previous scenraio != current scenario
+  note right: if previous scenario != current scenario
 
   :update help with default values;
   |execute|
@@ -94,7 +94,7 @@
 
   #Greenyellow:**pre**|
 
-  :setup HieraConfigurer and save configs;
+  :setup ExecutionEnvironment and save configs;
 
   #Gold:run puppet;
   repeat

data/lib/kafo/base_context.rb

@@ -1,3 +1,5 @@
+require 'open3'
+
 module Kafo
   class BaseContext
     def facts
@@ -24,12 +26,18 @@ module Kafo
 
     def self.facts
       @facts ||= begin
-        symbolize(JSON.load(`#{facter_path} --json`) || {})
+        result = run_command("#{facter_path} --json")
+        symbolize(JSON.load(result) || {})
       end
     end
 
     def self.facter_path
       @facter_path ||= PuppetCommand.search_puppet_path('facter')
     end
+
+    def self.run_command(command)
+      stdout, _stderr, _status = Open3.capture3(*PuppetCommand.format_command(command))
+      stdout
+    end
   end
 end

data/lib/kafo/configuration.rb

@@ -4,11 +4,11 @@ require 'tmpdir'
 require 'kafo/puppet_module'
 require 'kafo/color_scheme'
 require 'kafo/data_type_parser'
-require 'kafo/puppet_configurer'
+require 'kafo/execution_environment'
 
 module Kafo
   class Configuration
-    attr_reader :config_file, :answer_file
+    attr_reader :config_file, :answer_file, :scenario_id
 
     DEFAULT = {
         :name                 => '',
@@ -27,11 +27,16 @@ module Kafo
         :color_of_background  => :dark,
         :hook_dirs            => [],
         :custom               => {},
+        :facts                => {},
         :low_priority_modules => [],
         :verbose_log_level    => 'info',
         :skip_puppet_version_check => false
     }
 
+    def self.get_scenario_id(filename)
+      File.basename(filename, '.yaml')
+    end
+
     def initialize(file, persist = true)
       @config_file = file
       @persist     = persist
@@ -47,6 +52,7 @@ module Kafo
       end
 
       @config_dir = File.dirname(@config_file)
+      @scenario_id = Configuration.get_scenario_id(@config_file)
     end
 
     def save_configuration(configuration)
@@ -90,6 +96,14 @@ module Kafo
       custom_storage[key.to_sym] = value
     end
 
+    def get_custom_fact(key)
+      custom_fact_storage[key.to_s]
+    end
+
+    def set_custom_fact(key, value)
+      custom_fact_storage[key.to_s] = value
+    end
+
     def modules
       @modules ||= begin
         register_data_types
@@ -148,8 +162,10 @@ module Kafo
 
     def params_default_values
       @params_default_values ||= begin
-        puppetconf = PuppetConfigurer.new('noop' => true)
-        KafoConfigure.exit_handler.register_cleanup_path puppetconf.config_path
+        execution_env = ExecutionEnvironment.new(self)
+        KafoConfigure.exit_handler.register_cleanup_path(execution_env.directory)
+
+        puppetconf = execution_env.configure_puppet('noop' => true)
 
         dump_manifest = <<EOS
           #{includes}
@@ -160,12 +176,16 @@ module Kafo
 EOS
 
         @logger.info 'Loading default values from puppet modules...'
-        command = PuppetCommand.new(dump_manifest, [], puppetconf, self).append('2>&1').command
-        result = `#{command}`
-        @logger.debug result
-        unless $?.exitstatus == 0
+        command = PuppetCommand.new(dump_manifest, [], puppetconf, self).command
+        stdout, stderr, status = Open3.capture3(*PuppetCommand.format_command(command))
+
+        @logger.debug stdout
+        @logger.debug stderr
+
+        unless status.success?
           log = app[:log_dir] + '/' + app[:log_name]
-          if (version_mismatch = /kafo_configure::puppet_version_failure: (.+?\))/.match(result))
+
+          if (version_mismatch = /kafo_configure::puppet_version_failure: (.+?\))/.match(stderr))
             puts version_mismatch[1]
             puts "Cannot continue due to incompatible version of Puppet. Use --skip-puppet-version-check to disable this check."
             @logger.error version_mismatch[1]
@@ -174,14 +194,15 @@ EOS
           else
             puts "Could not get default values, check log file at #{log} for more information"
             @logger.error command
-            @logger.error result
+            @logger.error stderr
             @logger.error 'Could not get default values, cannot continue'
             KafoConfigure.exit(:defaults_error)
           end
         end
+
         @logger.info "... finished"
 
-        load_yaml_from_output(result.split($/))
+        load_yaml_from_output(stdout.split($/))
       end
     end
 
@@ -305,6 +326,10 @@ EOS
       app[:custom]
     end
 
+    def custom_fact_storage
+      app[:facts]
+    end
+
     def includes
       modules.map do |mod|
         module_dir = module_dirs.find do |dir|

data/lib/kafo/execution_environment.rb

@@ -0,0 +1,84 @@
+require 'tmpdir'
+
+require 'kafo/fact_writer'
+require 'kafo/hiera_configurer'
+require 'kafo/puppet_configurer'
+
+module Kafo
+  class ExecutionEnvironment
+    def initialize(config, logger = KafoConfigure.logger)
+      @config = config
+      @logger = logger
+    end
+
+    def directory
+      @directory ||= begin
+        directory = Dir.mktmpdir('kafo_installation')
+        @logger.debug("Creating execution environment in #{directory}")
+        directory
+      end
+    end
+
+    def store_answers
+      answer_data = HieraConfigurer.generate_data(@config.modules, @config.app[:order])
+      @logger.debug("Writing temporary answers to #{answer_file}")
+      File.open(answer_file, 'w') { |f| f.write(YAML.dump(answer_data)) }
+    end
+
+    def configure_puppet(settings = {})
+      @logger.debug("Configuring Puppet in #{directory}")
+
+      @logger.debug("Writing facts to #{factpath}")
+      FactWriter.write_facts(facts, factpath)
+
+      hiera_config = configure_hiera
+
+      settings = {
+        'environmentpath' => environmentpath,
+        'factpath'        => factpath,
+        'hiera_config'    => hiera_config,
+      }.merge(settings)
+
+      PuppetConfigurer.new(puppet_conf, settings)
+    end
+
+    private
+
+    def environmentpath
+      File.join(directory, 'environments')
+    end
+
+    def factpath
+      File.join(directory, 'facts')
+    end
+
+    def answer_file
+      File.join(directory, 'answers.yaml')
+    end
+
+    def puppet_conf
+      File.join(directory, 'puppet.conf')
+    end
+
+    def configure_hiera
+      if @config.app[:hiera_config]
+        File.realpath(@config.app[:hiera_config])
+      else
+        config_path = File.join(directory, 'hiera.yaml')
+        @logger.debug("Writing default hiera config to #{config_path}")
+        HieraConfigurer.write_default_config(config_path)
+      end
+    end
+
+    def facts
+      {
+        'scenario' => {
+          'id' => @config.scenario_id,
+          'name' => @config.app[:name],
+          'answer_file' => answer_file,
+          'custom' => @config.app[:facts],
+        },
+      }
+    end
+  end
+end

data/lib/kafo/exit_handler.rb

@@ -1,11 +1,10 @@
 module Kafo
   class ExitHandler
-    attr_accessor :cleanup_paths, :exit_code, :logger
+    attr_accessor :cleanup_paths, :exit_code
 
     def initialize
       @cleanup_paths = []
       @exit_code = 0
-      @logger = KafoConfigure.logger
     end
 
     def error_codes
@@ -27,8 +26,8 @@ module Kafo
     def exit(code, &block)
       @exit_code = translate_exit_code(code)
       block.call if block
-      KafoConfigure.logger.debug "Exit with status code: #{@exit_code} (signal was #{code})"
-      KafoConfigure.logger.dump_errors unless KafoConfigure.verbose
+      logger.debug "Exit with status code: #{@exit_code} (signal was #{code})"
+      logger.dump_errors unless KafoConfigure.verbose
       cleanup
       Kernel.exit(@exit_code)
     end
@@ -54,5 +53,11 @@ module Kafo
       self.cleanup_paths<< path
     end
 
+    private
+
+    def logger
+      @logger ||= KafoConfigure.logger
+    end
+
   end
 end

data/lib/kafo/fact_writer.rb

@@ -0,0 +1,24 @@
+module Kafo
+  class FactWriter
+    DATA_FILENAME = 'kafo.yaml'
+    WRAPPER_FILENAME = 'kafo.rb'
+
+    def self.write_facts(facts, directory)
+      Dir.mkdir(directory)
+
+      # Write a data file containing all the facts encoded as YAML
+      File.open(File.join(directory, DATA_FILENAME), 'w') { |f| f.write(YAML.dump(facts)) }
+
+      # Write a Ruby wrapper since only those are executed within puppet
+      File.open(File.join(directory, 'kafo.rb'), 'w') { |f| f.write(wrapper) }
+    end
+
+    def self.wrapper
+      # Ruby 2.0 doesn't have <<~ heredocs
+      <<-WRAPPER
+      require 'yaml'
+      Facter.add(:kafo) { setcode { YAML.load_file(File.join(__dir__, '#{DATA_FILENAME}')) } }
+      WRAPPER
+    end
+  end
+end

data/lib/kafo/hiera_configurer.rb

@@ -1,111 +1,37 @@
-require 'fileutils'
-require 'tmpdir'
-
 module Kafo
   class HieraConfigurer
-    HIERARCHY_NAME = 'Kafo Answers'
-    HIERARCHY_FILENAME = 'kafo_answers.yaml'
-
-    attr_reader :temp_dir, :config_path, :data_dir, :logger
-
-    def initialize(user_config_path, modules, modules_order)
-      @user_config_path = user_config_path
-      @modules = modules
-      @modules_order = modules_order
-      @logger = KafoConfigure.logger
+    def self.default_config
+      {
+        'version' => 5,
+        'hierarchy' => [
+          {
+            'name' => 'Kafo Answers',
+            'path' => '%{facts.kafo.scenario.answer_file}',
+            'data_hash' => 'yaml_data',
+          },
+        ],
+      }
     end
 
-    def write_configs
-      build_temp_dir
-
-      if @user_config_path
-        logger.debug("Merging existing Hiera config file from #{@user_config_path}")
-        user_config = YAML.load(File.read(@user_config_path))
-      else
-        user_config = {}
-      end
-      logger.debug("Writing Hiera config file to #{config_path}")
-      File.open(config_path, 'w') do |f|
-        # merge required config changes into the user's Hiera config
-        f.write(format_yaml_symbols(generate_config(user_config).to_yaml))
-      end
-
-      logger.debug("Creating Hiera data files in #{data_dir}")
-      FileUtils.mkdir(data_dir)
-
-      File.open(File.join(data_dir, HIERARCHY_FILENAME), 'w') do |f|
-        f.write(format_yaml_symbols(generate_data(@modules).to_yaml))
-      end
+    def self.write_default_config(path)
+      File.open(path, 'w') { |f| f.write(YAML.dump(default_config)) }
+      path
     end
 
-    def generate_config(config = {})
-      config ||= {}
-
-      config['version'] = 5
-
-      # ensure there are defaults
-      config['defaults'] ||= {}
-      config['defaults']['datadir'] = determine_data_dir_path(config['defaults']['datadir'])
-      config['defaults']['data_hash'] ||= 'yaml_data'
-
-      # ensure our answers file is present and has the right settings
-      config['hierarchy'] ||= []
-
-      config['hierarchy'].each do |level|
-        if level['datadir']
-          level['datadir'] = determine_data_dir_path(level['datadir'])
-        end
-      end
-
-      kafo_answers = config['hierarchy'].find { |level| level['name'] == HIERARCHY_NAME }
-      if kafo_answers
-        kafo_answers.clear
-      else
-        kafo_answers = {}
-        config['hierarchy'].unshift(kafo_answers)
-      end
-      kafo_answers['name'] = HIERARCHY_NAME
-      kafo_answers['path'] = HIERARCHY_FILENAME
-      kafo_answers['datadir'] = data_dir
-      kafo_answers['data_hash'] = 'yaml_data'
-
-      config
-    end
-
-    def generate_data(modules)
+    def self.generate_data(modules, order = nil)
       classes = []
       data = modules.select(&:enabled?).inject({}) do |config, mod|
         classes << mod.class_name
         config.update(Hash[mod.params_hash.map { |k, v| ["#{mod.class_name}::#{k}", v] }])
       end
-      data['classes'] = @modules_order ? sort_modules(classes, @modules_order) : classes
+      data['classes'] = sort_modules(classes, order)
       data
     end
 
-    def sort_modules(modules, order)
-      (order & modules) + (modules - order)
-    end
-
-    def build_temp_dir
-      @temp_dir ||= Dir.mktmpdir('kafo_hiera')
-      @config_path = File.join(temp_dir, 'hiera.conf')
-      @data_dir = File.join(temp_dir, 'data')
-    end
-
-    private
-
-    def format_yaml_symbols(data)
-      data.gsub('!ruby/sym ', ':')
-    end
+    def self.sort_modules(modules, order)
+      return modules unless order
 
-    def determine_data_dir_path(path)
-      # Relies on data_dir being absolute or having a user config
-      path ||= data_dir
-      Pathname.new(path).relative? ? File.join(original_hiera_directory, path) : path
-    end
-
-    def original_hiera_directory
-      @user_config_path ? File.dirname(@user_config_path) : nil
+      (order & modules) + (modules - order)
     end
   end
 end