kafo 2.1.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: bb73eb4c71ae5aad0d0c3dc4b891a0ef33af1c8a
-  data.tar.gz: a63a799798654d67bc245630d0f95cc52b379b64
+SHA256:
+  metadata.gz: e35b389c721abe577314b02199940ac88fa9777c9a3daa0d0b1bda886082ea0b
+  data.tar.gz: f92d08ac63cd96cf98b6b0f6a60b22c97d4cdd66b7e482ab3556b9654659a50b
 SHA512:
-  metadata.gz: 819bf86e16679ab8fbc15015bf8562cd415ad9f854ceb031503facf9f6dcd0ea1e978071c69de011e7e9b68bb9a8d20a56bc92ac73ed4e96407e2669c253cef1
-  data.tar.gz: 9e82f85dabf51876b18f8306de913ac93342c3b0ce4ad257dbde7f65ada6de27fcaf529abe870c843f66c1fdf28900181bbb3d2e338482d9d07c7c46a3718386
+  metadata.gz: 26426722b8088cd909266123492b328e686829fe798a4f02cabdd2483fc0e1a1360fcc77b535f86fa7f7e6a176b986779089a409ec37c0d4de93134b0861ee1b
+  data.tar.gz: d48254980b31a24d331cf3450d19d19bd0d62a529c8daedc9164dae01327ba4ec058b6171536cb632e811cacedd45c178ceafc388488e434a423f3e36d3bacc2

data/README.md

@@ -160,15 +160,13 @@ As you may have noticed there are several ways how to specify arguments. Here's
 
 ## Requirements
 
-Kafo is supported with Puppet versions 3 and 4. Puppet 2 is no longer supported
-in current versions, use an older version of Kafo or update Puppet.
-
-Puppet may be installed as a gem (add it to Gemfile) or through a package,
-including official AIO packages.
+Kafo is supported with Puppet versions 4.9+, 5 and 6. Puppet may be installed
+as a gem (add it to Gemfile) or through a package, including official AIO
+packages.
 
 ## How do I report bugs or contribute?
 
-You can find our redmine issue tracker [here](http://projects.theforeman.org/projects/kafo),
+You can find our redmine issue tracker [here](https://projects.theforeman.org/projects/kafo),
 you can use your github account for logging in. When reporting new issues please
 don't forget to specify your:
   * puppet version
@@ -179,10 +177,10 @@ don't forget to specify your:
 Since Kafo is a side project of Foreman you can use its IRC channels to
 contact us on freenode. #theforeman is the channel for generic discussions
 and #theforeman-dev is reserved only for technical topics. Likewise you can use the Foreman
-mailing lists on googlegroups. For more information see [this page](http://theforeman.org/support.html)
+mailing lists on googlegroups. For more information see [this page](https://theforeman.org/support.html)
 
 Patches are always welcome. You can use instructions for Foreman, just
-substitute Foreman with Kafo. More details are [here](http://projects.theforeman.org/projects/foreman/wiki/Contribute#New-Way-github)
+substitute Foreman with Kafo. More details are [here](https://projects.theforeman.org/projects/foreman/wiki/Contribute#New-Way-github)
 
 # Advanced topics
 
@@ -445,8 +443,8 @@ the particular parameter belongs.
 
 ## Argument types
 
-When using Puppet 4 or newer, the data type will be read from the parameter
-list and defaults to Puppet's [Any](https://docs.puppet.com/puppet/latest/reference/lang_data_abstract.html#any)
+The data type will be read from the parameter list and defaults to Puppet's
+[Any](https://docs.puppet.com/puppet/latest/reference/lang_data_abstract.html#any)
 data type, which Kafo handles as a basic string with no validation.
 
 If more specific data types, such as `Optional[Array[2]]` or similar are
@@ -460,21 +458,6 @@ class example (
 ) {
 ```
 
-When using Puppet 3, data types can be specified in the manifest documentation
-rather than the parameter list, like this:
-
-```puppet
-# $param::        Some documentation for param
-                  type:Array[String]
-```
-
-For compatibility with older Kafo releases, additional types are supported:
-string, boolean, integer, array, password, hash. These are equivalent to their
-Puppet 4 namesakes, plus wrapped in `Optional[..]` to permit `undef`.
-
-If the data type is given in both the manifest documentation and the parameter
-list, then the manifest documentation will be preferred.
-
 Note that all arguments that are nil (have no value in answers.yaml or you
 set them UNDEF (see below)) are translated to ```undef``` in puppet.
 
@@ -484,48 +467,6 @@ of DataType which implement validation and typecasting. This can be added to a
 
     Kafo::DataType.register_type('YourType', Kafo::DataType::YourType)
 
-## Password arguments
-
-Kafo supports password arguments. It's adding some level of protection for your
-passwords. Usually people generate random strings for passwords. However all
-values are stored in config/answers.yaml which may introduce a security risk.
-
-If this is something to consider for you, you can use the password type (see
-Argument types for more info how to define parameter type). It will
-generate a secure (random) password with a length of 32 chars and encrypts
-it using AES 256 in CBC mode. It uses a passphrase that is stored in
-config/kafo.yaml so if anyone gets an access to this file, he can read all
-the passwords from the answers.yaml, too. A random password is generated and stored
-if there is none in kafo.yaml yet.
-
-When Kafo runs puppet, puppet will read this password from config/kafo.yaml.
-It runs under the same user so it should have read access by default. The Kafo
-puppet module also provides a function that you can use to decrypt such
-parameters. You can use it like this
-
-```erb
-password: <%= scope.function_decrypt([scope.lookupvar("::foreman::db_password"))]) -%>
-```
-
-Also you can take advantage of already encrypted passwords and store since it is
-(encrypted). Your application can decrypt it as long as it knows the
-passphrase. The passphrase can be obtained from $kafo_configure::password.
-
-Note that we use a bit extraordinary form of encrypted passwords. All our
-encrypted passwords look like "$1$base64encodeddata". As you can see we
-use the $1$ as prefix by which we can detect that it is our specially encrypted password.
-The form has nothing common with Modular Crypt Format. Also our AES output
-is base64 encoded. To get a password from this format you can do something
-like this in your application
-
-```ruby
-require 'base64'
-encrypted = "$1$base64encodeddata"
-encrypted = encrypted[3..-1]           # strip $1$ prefix
-encrypted = Base64.decode64(encrypted) # decode base64 string
-result    = aes_decrypt(encrypted)     # for example how to implement aes_decrypt see lib/kafo/password_manager.rb
-```
-
 ## Array arguments
 
 Some arguments may be Arrays. If you want to specify array values you can
@@ -582,9 +523,11 @@ class foreman::params {
 }
 ```
 
-If no inline default is given in the manifest and on Puppet 4.5+, then Kafo will
-attempt to look up a default value using [data stored in the module](https://docs.puppet.com/puppet/latest/lookup_quick_module.html). This can be specified with Hiera data files (or even a data function) in the
-module under `data/`.
+If no inline default is given in the manifest, then Kafo will attempt to look
+up a default value using [data stored in the
+module](https://docs.puppet.com/puppet/latest/lookup_quick_module.html). This
+can be specified with Hiera data files (or even a data function) in the module
+under `data/`.
 
 ## Resetting an argument
 
@@ -681,7 +624,6 @@ $db_type is not set 'mysql'. Let's look at following example
 # == Parameters:
 #
 # $use_db::                  use database?
-#                            type:boolean
 #
 # == Database parameters:    condition: $use_db
 #
@@ -690,7 +632,6 @@ $db_type is not set 'mysql'. Let's look at following example
 # === MySQL:                 condition: $database_type == 'mysql'
 #
 # $remote::                  use remote connection
-#                            type:boolean
 # $host                      server to connect to
 #                            condition: $remote
 # $socket                    server to connect to
@@ -754,18 +695,8 @@ If you use "params_path" for this purpose, "params_name" is ignored.
 
 ## Validations
 
-If you specify validations of parameters in your init.pp manifest they
-will be replaced with your values even before Puppet is run. In order to do this
-you must follow a few rules however:
-
-* you must use standard validation functions (e.g. validate_array, validate_re, ...)
-
-These functions are re-implemented in Kafo from common stdlib functions, so please
-contribute any missing ones.
-
 If class parameters are declared with Puppet 4 data types then Kafo will
-validate user inputs against Puppet's type validation rules, which should
-replace the use of separate validation functions.
+validate user inputs against Puppet's type validation rules.
 
 ## Enabling or disabling module
 
@@ -1083,26 +1014,53 @@ being managed by Kafo. Set a custom Hiera config file in Kafo's config with:
 :hiera_config: /usr/share/kafo/hiera.yaml
 ```
 
-The contents of this file are as per the [hiera.yaml docs](https://docs.puppet.com/hiera/latest/configuring.html),
-but hierarchy should contain the item `kafo_answers` and the `yaml` backend
-should be enabled. Kafo will add these if they're missing.
+The contents of this file are as per the
+[hiera.yaml docs](https://docs.puppet.com/hiera/latest/configuring.html).
+Only Hiera version 5 is supported.
 
-When running Puppet, Kafo will copy the Hiera config and YAML data directory to
-a temporary location to include its data. The file `kafo_answers.yaml` will be
-generated containing _all_ default and overriden values for parameters managed
-by Kafo. This may change in the future to allow a more complex hierarchy. As an
-example, a hierarchy could be set up with:
+When running Puppet, Kafo will copy the Hiera config to a temporary location.
+Relative data directories will be changed to absolute paths. A
+`kafo_answers.yaml` file will be generated containing _all_ default and
+overriden values for parameters managed by Kafo. This may change in the future
+to allow a more complex hierarchy.
+
+The hierarchy can contain a special value with the name `Kafo Answers`. The
+exact values will be rewritten by Kafo, but it can be used to determine when
+the Kafo answers are loaded. Note the name is case sensitive. When it's
+missing, it will be added.
+
+As an example, a hierarchy could be set up with:
 
 ```yaml
-:hierarchy:
-  - kafo_answers
-  - "%{::osfamily}"
-  - common
+hierarchy:
+  - name: "Kafo Answers"
+  - name: "Other YAML hierarchy levels"
+    paths:
+      - "family/%{facts.os.family}.yaml"
+      - "common.yaml"
+```
+
+It is possible to include another file above answers:
+
+```yaml
+hierarchy:
+  - name: "Custom values"
+    datadir: "custom"
+    path: "override.yaml"
+  - name: "Kafo Answers"
+  - name: "Other YAML hierarchy levels"
+    datadir: "data"
+    paths:
+      - "family/%{facts.os.family}.yaml"
+      - "common.yaml"
 ```
 
 This would give precedence to all Kafo-managed parameter values, but for any
 others, would check for values per OS family, followed by a `common.yaml` file.
 
+[Migration from Hiera version 3](https://puppet.com/docs/puppet/4.9/hiera_migrate_v3_yaml.html)
+is documented by Puppet.
+
 ## Exit code
 
 Kafo can terminate either before or after puppet is run. If it is run with
@@ -1130,14 +1088,14 @@ Other exit codes that can be returned:
 
 ## Running Puppet Profiling
 
-As of Puppet 3.2, performance data can be gathered during a puppet run by adding the `--profile` option. See [Tune Puppet for Performance with Profiler](https://puppetlabs.com/blog/tune-puppet-performance-profiler) for more information from the Puppet team. Users who wish to perform a Kafo run and gather this type of profiling data to analyze can pass the same option to their installer. The profiling data will then be present in the normal Kafo logs.
+Performance data can be gathered during a Puppet run by adding the `--profile` option. See [Tune Puppet for Performance with Profiler](https://puppet.com/blog/tune-puppet-for-performance-profiler) for more information from the Puppet team. Users who wish to perform a Kafo run and gather this type of profiling data to analyze can pass the same option to their installer. The profiling data will then be present in the normal Kafo logs.
 
 ## Issue tracker
 
 Issues are tracked in Redmine, see:
 
-* [Open Kafo issues](http://projects.theforeman.org/projects/kafo/issues/)
-* [File new issue](http://projects.theforeman.org/projects/kafo/issues/new)
+* [Open Kafo issues](https://projects.theforeman.org/projects/kafo/issues/)
+* [File new issue](https://projects.theforeman.org/projects/kafo/issues/new)
 
 ## Related projects
 

data/Rakefile

@@ -14,6 +14,17 @@ Rake::TestTask.new('test:acceptance') do |t|
   t.verbose = true
 end
 
+namespace 'test' do
+  desc 'Run Puppet module tests'
+  task :puppet_modules do
+    Dir['modules/*'].each do |mod|
+      Dir.chdir(mod) do
+        `rake release_checks`
+      end
+    end
+  end
+end
+
 CLEAN.include 'test/tmp'
 
-task :test => ['test:unit', 'test:acceptance']
+task :test => ['test:unit', 'test:acceptance', 'test:puppet_modules']

data/lib/kafo/color_scheme.rb

@@ -13,8 +13,8 @@ module Kafo
     end
 
     def setup
+      HighLine.color_scheme = build_color_scheme
       if @colors
-        HighLine.color_scheme = build_color_scheme
         HighLine.use_color = true
       else
         HighLine.use_color = false
@@ -25,8 +25,8 @@ module Kafo
 
     def build_color_scheme
       HighLine::ColorScheme.new do |cs|
-        color_hash.keys.each do |key|
-          cs[key] = color_hash[key]
+        color_hash.each do |key, value|
+          cs[key] = value
         end
       end
     end

data/lib/kafo/configuration.rb

@@ -2,7 +2,6 @@
 require 'yaml'
 require 'tmpdir'
 require 'kafo/puppet_module'
-require 'kafo/password_manager'
 require 'kafo/color_scheme'
 require 'kafo/data_type_parser'
 require 'kafo/puppet_configurer'
@@ -42,7 +41,7 @@ module Kafo
       @answer_file = app[:answer_file]
       begin
         @data = load_yaml_file(@answer_file)
-      rescue Errno::ENOENT => e
+      rescue Errno::ENOENT
         puts "No answer file at #{@answer_file} found, can not continue"
         KafoConfigure.exit(:no_answer_file)
       end
@@ -72,12 +71,11 @@ module Kafo
       @app ||= begin
         begin
           configuration = load_yaml_file(@config_file)
-        rescue => e
+        rescue
           configuration = {}
         end
 
         result            = DEFAULT.merge(configuration || {})
-        result[:password] ||= PasswordManager.new.password
         result[:module_dirs] = result[:modules_dir] || result[:module_dirs]
         result.delete(:modules_dir)
         result
@@ -139,7 +137,7 @@ module Kafo
     def migrate_configuration(from_config, options={})
       keys_to_skip = options.fetch(:skip, [])
       keys = [:log_dir, :log_name, :log_level, :no_prefix,
-        :colors, :color_of_background, :custom, :password, :verbose_log_level]
+        :colors, :color_of_background, :custom, :verbose_log_level]
       keys += options.fetch(:with, [])
       keys.each do |key|
         next if keys_to_skip.include?(key)
@@ -201,7 +199,7 @@ EOS
 
     def config_header
       files          = [app[:config_header_file], File.join(gem_root, '/config/config_header.txt')].compact
-      file           = files.select { |f| File.exists?(f) }.first
+      file           = files.find { |f| File.exist?(f) }
       @config_header ||= file.nil? ? '' : File.read(file)
     end
 
@@ -209,7 +207,7 @@ EOS
       filename = file || answer_file
       FileUtils.touch filename
       File.chmod 0600, filename
-      File.open(filename, 'w') { |file| file.write(config_header + format(YAML.dump(data))) }
+      File.open(filename, 'w') { |f| f.write(config_header + format(YAML.dump(data))) }
     end
 
     def params

data/lib/kafo/data_types/aliases.rb

@@ -1,14 +1,4 @@
-require 'kafo/data_type'
-
 module Kafo
   DataType.register_type('Data', 'Any')
   DataType.register_type('Default', 'Enum["default"]')
-
-  # pre-Puppet 4 Kafo data types
-  DataType.register_type('array', 'Optional[Array]')
-  DataType.register_type('boolean', 'Optional[Boolean]')
-  DataType.register_type('hash', 'Optional[Hash]')
-  DataType.register_type('integer', 'Optional[Integer]')
-  DataType.register_type('password', 'Optional[String]')
-  DataType.register_type('string', 'Optional[String]')
 end

data/lib/kafo/data_types/not_undef.rb

@@ -10,7 +10,9 @@ module Kafo
       def initialize(inner_type_or_value)
         begin
           @inner_type = DataType.new_from_string(inner_type_or_value)
+          @inner_value = nil
         rescue ConfigurationException
+          @inner_type = nil
           @inner_value = inner_type_or_value
         end
       end

data/lib/kafo/data_types/optional.rb

@@ -10,7 +10,9 @@ module Kafo
       def initialize(inner_type_or_value)
         begin
           @inner_type = DataType.new_from_string(inner_type_or_value)
+          @inner_value = nil
         rescue ConfigurationException
+          @inner_type = nil
           @inner_value = inner_type_or_value
         end
       end

data/lib/kafo/hiera_configurer.rb

@@ -3,6 +3,9 @@ require 'tmpdir'
 
 module Kafo
   class HieraConfigurer
+    HIERARCHY_NAME = 'Kafo Answers'
+    HIERARCHY_FILENAME = 'kafo_answers.yaml'
+
     attr_reader :temp_dir, :config_path, :data_dir, :logger
 
     def initialize(user_config_path, modules, modules_order)
@@ -13,17 +16,13 @@ module Kafo
     end
 
     def write_configs
-      @temp_dir = Dir.mktmpdir('kafo_hiera')
-      @config_path = File.join(temp_dir, 'hiera.conf')
-      @data_dir = File.join(temp_dir, 'data')
+      build_temp_dir
 
       if @user_config_path
         logger.debug("Merging existing Hiera config file from #{@user_config_path}")
         user_config = YAML.load(File.read(@user_config_path))
-        user_data_dir = user_config[:yaml][:datadir] if user_config[:yaml]
       else
         user_config = {}
-        user_data_dir = false
       end
       logger.debug("Writing Hiera config file to #{config_path}")
       File.open(config_path, 'w') do |f|
@@ -31,15 +30,10 @@ module Kafo
         f.write(format_yaml_symbols(generate_config(user_config).to_yaml))
       end
 
-      if user_data_dir
-        logger.debug("Copying Hiera data files from #{user_data_dir} to #{data_dir}")
-        FileUtils.cp_r(user_data_dir, data_dir)
-      else
-        logger.debug("Creating Hiera data files in #{data_dir}")
-        FileUtils.mkdir(data_dir)
-      end
+      logger.debug("Creating Hiera data files in #{data_dir}")
+      FileUtils.mkdir(data_dir)
 
-      File.open(File.join(data_dir, 'kafo_answers.yaml'), 'w') do |f|
+      File.open(File.join(data_dir, HIERARCHY_FILENAME), 'w') do |f|
         f.write(format_yaml_symbols(generate_data(@modules).to_yaml))
       end
     end
@@ -47,17 +41,33 @@ module Kafo
     def generate_config(config = {})
       config ||= {}
 
-      # ensure YAML is enabled
-      config[:backends] ||= []
-      config[:backends] << 'yaml' unless config[:backends].include?('yaml')
+      config['version'] = 5
 
-      # ensure kafo_answers is present and most specific
-      config[:hierarchy] ||= []
-      config[:hierarchy].unshift('kafo_answers') unless config[:hierarchy].include?('kafo_answers')
+      # ensure there are defaults
+      config['defaults'] ||= {}
+      config['defaults']['datadir'] = determine_data_dir_path(config['defaults']['datadir'])
+      config['defaults']['data_hash'] ||= 'yaml_data'
 
-      # use our copy of the data dir
-      config[:yaml] ||= {}
-      config[:yaml][:datadir] = data_dir
+      # ensure our answers file is present and has the right settings
+      config['hierarchy'] ||= []
+
+      config['hierarchy'].each do |level|
+        if level['datadir']
+          level['datadir'] = determine_data_dir_path(level['datadir'])
+        end
+      end
+
+      kafo_answers = config['hierarchy'].find { |level| level['name'] == HIERARCHY_NAME }
+      if kafo_answers
+        kafo_answers.clear
+      else
+        kafo_answers = {}
+        config['hierarchy'].unshift(kafo_answers)
+      end
+      kafo_answers['name'] = HIERARCHY_NAME
+      kafo_answers['path'] = HIERARCHY_FILENAME
+      kafo_answers['datadir'] = data_dir
+      kafo_answers['data_hash'] = 'yaml_data'
 
       config
     end
@@ -76,8 +86,26 @@ module Kafo
       (order & modules) + (modules - order)
     end
 
+    def build_temp_dir
+      @temp_dir ||= Dir.mktmpdir('kafo_hiera')
+      @config_path = File.join(temp_dir, 'hiera.conf')
+      @data_dir = File.join(temp_dir, 'data')
+    end
+
+    private
+
     def format_yaml_symbols(data)
       data.gsub('!ruby/sym ', ':')
     end
+
+    def determine_data_dir_path(path)
+      # Relies on data_dir being absolute or having a user config
+      path ||= data_dir
+      Pathname.new(path).relative? ? File.join(original_hiera_directory, path) : path
+    end
+
+    def original_hiera_directory
+      @user_config_path ? File.dirname(@user_config_path) : nil
+    end
   end
 end