puppet 4.0.0 → 4.5.0

data/lib/puppet/parser/functions/reduce.rb

@@ -3,66 +3,103 @@ Puppet::Parser::Functions::newfunction(
   :type => :rvalue,
   :arity => -3,
   :doc => <<-DOC
-Applies a parameterized block to each element in a sequence of entries from the first
-argument (_the enumerable_) and returns the last result of the invocation of the parameterized block.
-
-This function takes two mandatory arguments: the first should be an Array, Hash, or something of
-enumerable type, and the last a parameterized block as produced by the puppet syntax:
-
-      $a.reduce |$memo, $x| { ... }
-      reduce($a) |$memo, $x| { ... }
-
-When the first argument is an Array or someting of an enumerable type, the block is called with each entry in turn.
-When the first argument is a hash each entry is converted to an array with `[key, value]` before being
-fed to the block. An optional 'start memo' value may be supplied as an argument between the array/hash
-and mandatory block.
-
-      $a.reduce(start) |$memo, $x| { ... }
-      reduce($a, start) |$memo, $x| { ... }
-
-If no 'start memo' is given, the first invocation of the parameterized block will be given the first and second
-elements of the enumeration, and if the enumerable has fewer than 2 elements, the first
-element is produced as the result of the reduction without invocation of the block.
-
-On each subsequent invocation, the produced value of the invoked parameterized block is given as the memo in the
-next invocation.
-
-Example Using reduce
-
-      # Reduce an array
-      $a = [1,2,3]
-      $a.reduce |$memo, $entry| { $memo + $entry }
-      #=> 6
-
-      # Reduce hash values
-      $a = {a => 1, b => 2, c => 3}
-      $a.reduce |$memo, $entry| { [sum, $memo[1]+$entry[1]] }
-      #=> [sum, 6]
-
-      # reverse a string
-      "abc".reduce |$memo, $char| { "$char$memo" }
-      #=>"cbe"
-
-It is possible to provide a starting 'memo' as an argument.
-
-Example Using reduce with given start 'memo'
-
-      # Reduce an array
-      $a = [1,2,3]
-      $a.reduce(4) |$memo, $entry| { $memo + $entry }
-      #=> 10
-
-      # Reduce hash values
-      $a = {a => 1, b => 2, c => 3}
-      $a.reduce([na, 4]) |$memo, $entry| { [sum, $memo[1]+$entry[1]] }
-      #=> [sum, 10]
-
-Example Using reduce with an Integer range
-
-      Integer[1,4].reduce |$memo, $x| { $memo + $x }
-      #=> 10
-
-- since 4.0.0
+Applies a [lambda](https://docs.puppetlabs.com/puppet/latest/reference/lang_lambdas.html)
+to every value in a data structure from the first argument, carrying over the returned
+value of each iteration, and returns the result of the lambda's final iteration. This
+lets you create a new value or data structure by combining values from the first
+argument's data structure.
+
+This function takes two mandatory arguments, in this order:
+
+1. An array or hash the function will iterate over.
+2. A lambda, which the function calls for each element in the first argument. It takes
+two mandatory parameters:
+    1. A memo value that is overwritten after each iteration with the iteration's result.
+    2. A second value that is overwritten after each iteration with the next value in the
+    function's first argument.
+
+**Example**: Using the `reduce` function
+
+`$data.reduce |$memo, $value| { ... }`
+
+or
+
+`reduce($data) |$memo, $value| { ... }`
+
+You can also pass an optional "start memo" value as an argument, such as `start` below:
+
+`$data.reduce(start) |$memo, $value| { ... }`
+
+or
+
+`reduce($data, start) |$memo, $value| { ... }`
+
+When the first argument (`$data` in the above example) is an array, Puppet passes each
+of the data structure's values in turn to the lambda's parameters. When the first
+argument is a hash, Puppet converts each of the hash's values to an array in the form
+`[key, value]`.
+
+If you pass a start memo value, Puppet executes the lambda with the provided memo value
+and the data structure's first value. Otherwise, Puppet passes the structure's first two
+values to the lambda.
+
+Puppet calls the lambda for each of the data structure's remaining values. For each
+call, it passes the result of the previous call as the first parameter ($memo in the
+above examples) and the next value from the data structure as the second parameter
+($value).
+
+If the structure has one value, Puppet returns the value and does not call the lambda.
+
+**Example**: Using the `reduce` function
+
+~~~ puppet
+# Reduce the array $data, returning the sum of all values in the array.
+$data = [1, 2, 3]
+$sum = $data.reduce |$memo, $value| { $memo + $value }
+# $sum contains 6
+
+# Reduce the array $data, returning the sum of a start memo value and all values in the
+# array.
+$data = [1, 2, 3]
+$sum = $data.reduce(4) |$memo, $value| { $memo + $value }
+# $sum contains 10
+
+# Reduce the hash $data, returning the sum of all values and concatenated string of all
+# keys.
+$data = {a => 1, b => 2, c => 3}
+$combine = $data.reduce |$memo, $value| {
+  $string = "${memo[0]}${value[0]}"
+  $number = $memo[1] + $value[1]
+  [$string, $number]
+}
+# $combine contains [abc, 6]
+~~~
+
+**Example**: Using the `reduce` function with a start memo and two-parameter lambda
+
+~~~ puppet
+# Reduce the array $data, returning the sum of all values in the array and starting
+# with $memo set to an arbitrary value instead of $data's first value.
+$data = [1, 2, 3]
+$sum = $data.reduce(4) |$memo, $value| { $memo + $value }
+# At the start of the lambda's first iteration, $memo contains 4 and $value contains 1.
+# After all iterations, $sum contains 10.
+
+# Reduce the hash $data, returning the sum of all values and concatenated string of
+# all keys, and starting with $memo set to an arbitrary array instead of $data's first
+# key-value pair.
+$data = {a => 1, b => 2, c => 3}
+$combine = $data.reduce( [d, 4] ) |$memo, $value| {
+  $string = "${memo[0]}${value[0]}"
+  $number = $memo[1] + $value[1]
+  [$string, $number]
+}
+# At the start of the lambda's first iteration, $memo contains [d, 4] and $value
+# contains [a, 1].
+# $combine contains [dabc, 10]
+~~~
+
+- Since 4.0.0
 DOC
 ) do |args|
   function_fail(["reduce() is only available when parser/evaluator future is in effect"])

data/lib/puppet/parser/functions/scanf.rb

@@ -5,37 +5,26 @@ Puppet::Parser::Functions::newfunction(
   :type => :rvalue,
   :arity => 2,
   :doc => <<-DOC
-Scans a string and returns an array of one or more converted values as directed by a given format string.args
-See the documenation of Ruby's String::scanf method for details about the supported formats (which
-are similar but not identical to the formats used in Puppet's `sprintf` function.
+Scans a string and returns an array of one or more converted values based on the given format string.
+See the documenation of Ruby's String#scanf method for details about the supported formats (which
+are similar but not identical to the formats used in Puppet's `sprintf` function.)
 
-This function takes two mandatory arguments: the first is the String to
-convert, and the second the format String. The result of the scan is an Array,
-with each sucessfully scanned and transformed value.args The scanning stops if
-a scan is unsuccesful and the scanned result up to that point is returned. If
-there was no succesful scan at all, the result is an empty Array.
+This function takes two mandatory arguments: the first is the string to convert, and the second is
+the format string. The result of the scan is an array, with each sucessfully scanned and transformed value.
+The scanning stops if a scan is unsuccesful, and the scanned result up to that point is returned. If there
+was no succesful scan, the result is an empty array.
 
-      scanf("42", "%i")[0] == 42
+   "42".scanf("%i")
 
+You can also optionally pass a lambda to scanf, to do additional validation or processing.
 
-When used with the future parser, an optional parameterized block may be given.
-The block is called with the result that is produced by scanf if no block is
-present, the result of the block is then returned by the function.
-
-The optional code block is typically used to assert that the scan was
-succesful, and either produce the same input, or perform unwrapping of
-the result:
-
-      "42".scanf("%i")
-      "42".scanf("%i") |$x| {
-        unless $x[0] =~ Integer {
-          fail "Expected a well formed integer value, got '$x[0]'"
-        }
-        $x[0]
+    "42".scanf("%i") |$x| {
+      unless $x[0] =~ Integer {
+        fail "Expected a well formed integer value, got '$x[0]'"
       }
-
-- since 3.7.4 with `parser = future`
-- since 3.7.5 with classic parser
+      $x[0]
+    }
+- since 4.0.0
 DOC
 ) do |args|
   data = args[0]

data/lib/puppet/parser/functions/slice.rb

@@ -3,42 +3,35 @@ Puppet::Parser::Functions::newfunction(
   :type => :rvalue,
   :arity => -3,
   :doc => <<-DOC
-Applies a parameterized block to each _slice_ of elements in a sequence of selected entries from the first
-argument and returns the first argument, or if no block is given returns a new array with a concatenation of
-the slices.
+This function takes two mandatory arguments: the first should be an array or hash, and the second specifies
+the number of elements to include in each slice.
 
-This function takes two mandatory arguments: the first, `$a`, should be an Array, Hash, or something of
-enumerable type (integer, Integer range, or String), and the second, `$n`, the number of elements to include
-in each slice. The optional third argument should be a a parameterized block as produced by the puppet syntax:
-
-    $a.slice($n) |$x| { ... }
-    slice($a) |$x| { ... }
-
-The parameterized block should have either one parameter (receiving an array with the slice), or the same number
-of parameters as specified by the slice size (each parameter receiving its part of the slice).
-In case there are fewer remaining elements than the slice size for the last slice it will contain the remaining
-elements. When the block has multiple parameters, excess parameters are set to undef for an array or
-enumerable type, and to empty arrays for a Hash.
-
-    $a.slice(2) |$first, $second| { ... }
-
-When the first argument is a Hash, each `key,value` entry is counted as one, e.g, a slice size of 2 will produce
+When the first argument is a hash, each key value pair is counted as one. For example, a slice size of 2 will produce
 an array of two arrays with key, and value.
 
-Example Using slice with Hash
-
     $a.slice(2) |$entry|          { notice "first ${$entry[0]}, second ${$entry[1]}" }
     $a.slice(2) |$first, $second| { notice "first ${first}, second ${second}" }
 
-When called without a block, the function produces a concatenated result of the slices.
-
-Example Using slice without a block
+The function produces a concatenated result of the slices.
 
     slice([1,2,3,4,5,6], 2) # produces [[1,2], [3,4], [5,6]]
     slice(Integer[1,6], 2)  # produces [[1,2], [3,4], [5,6]]
     slice(4,2)              # produces [[0,1], [2,3]]
     slice('hello',2)        # produces [[h, e], [l, l], [o]]
 
+You can also optionally pass a lambda to slice.
+
+    $a.slice($n) |$x| { ... }
+    slice($a) |$x| { ... }
+
+The lambda should have either one parameter (receiving an array with the slice), or the same number
+of parameters as specified by the slice size (each parameter receiving its part of the slice).
+If there are fewer remaining elements than the slice size for the last slice, it will contain the remaining
+elements. If the lambda has multiple parameters, excess parameters are set to undef for an array, or
+to empty arrays for a hash.
+
+    $a.slice(2) |$first, $second| { ... }
+
 - Since 4.0.0
 DOC
 ) do |args|

data/lib/puppet/parser/functions/split.rb

@@ -12,7 +12,7 @@ Split a string variable into an array using the specified split regexp.
     $string     = 'v1.v2:v3.v4'
     $array_var1 = split($string, ':')
     $array_var2 = split($string, '[.]')
-    $array_var3 = split($string, '[.:]')
+    $array_var3 = split($string, Regexp['[.:]'])
 
 `$array_var1` now holds the result `['v1.v2', 'v3.v4']`,
 while `$array_var2` holds `['v1', 'v2:v3', 'v4']`, and

data/lib/puppet/parser/functions/with.rb

@@ -3,17 +3,25 @@ Puppet::Parser::Functions::newfunction(
   :type => :rvalue,
   :arity => -1,
   :doc => <<-DOC
-Call a lambda code block with the given arguments. Since the parameters of the lambda
-are local to the lambda's scope, this can be used to create private sections
-of logic in a class so that the variables are not visible outside of the
-class.
+Call a [lambda](https://docs.puppetlabs.com/puppet/latest/reference/lang_lambdas.html)
+with the given arguments and return the result. Since a lambda's scope is
+[local](https://docs.puppetlabs.com/puppet/latest/reference/lang_lambdas.html#lambda-scope)
+to the lambda, you can use the `with` function to create private blocks of code within a
+class using variables whose values cannot be accessed outside of the lambda.
 
-Example:
+**Example**: Using `with`
 
-     # notices the array [1, 2, 'foo']
-     with(1, 2, 'foo') |$x, $y, $z| { notice [$x, $y, $z] }
+~~~ puppet
+# Concatenate three strings into a single string formatted as a list.
+$fruit = with("apples", "oranges", "bananas") |$x, $y, $z| { 
+  "${x}, ${y}, and ${z}" 
+}
+$check_var = $x
+# $fruit contains "apples, oranges, and bananas"
+# $check_var is undefined, as the value of $x is local to the lambda.
+~~~
 
-- since 4.0.0
+- Since 4.0.0
 DOC
 ) do |args|
   function_fail(["with() is only available when parser/evaluator future is in effect"])

data/lib/puppet/parser/functions.rb

@@ -1,5 +1,6 @@
 require 'puppet/util/autoload'
 require 'puppet/parser/scope'
+require 'puppet/pops/adaptable'
 
 # A module for managing parser functions.  Each specified function
 # is added to a central module that then gets included into the Scope
@@ -17,12 +18,12 @@ module Puppet::Parser::Functions
   #
   # @api private
   def self.reset
-    @modules = {}
-
     # Runs a newfunction to create a function for each of the log levels
+    root_env = Puppet.lookup(:root_environment)
+    AnonymousModuleAdapter.clear(root_env)
     Puppet::Util::Log.levels.each do |level|
       newfunction(level,
-                  :environment => Puppet.lookup(:root_environment),
+                  :environment => root_env,
                   :doc => "Log a message on the server at level #{level.to_s}.") do |vals|
         send(level, vals.join(" "))
       end
@@ -36,26 +37,37 @@ module Puppet::Parser::Functions
     @autoloader ||= Puppet::Util::Autoload.new(self, "puppet/parser/functions")
   end
 
+  # An adapter that ties the anonymous module that acts as the container for all 3x functions to the environment
+  # where the functions are created. This adapter ensures that the life-cycle of those functions doesn't exceed
+  # the life-cycle of the environment.
+  #
+  # @api private
+  class AnonymousModuleAdapter < Puppet::Pops::Adaptable::Adapter
+    attr_accessor :module
+  end
+
   # Get the module that functions are mixed into corresponding to an
   # environment
   #
   # @api private
   def self.environment_module(env)
-    @modules[env.name] ||= Module.new do
-      @metadata = {}
+    AnonymousModuleAdapter.adapt(env) do |a|
+      a.module ||= Module.new do
+        @metadata = {}
 
-      def self.all_function_info
-        @metadata
-      end
+        def self.all_function_info
+          @metadata
+        end
 
-      def self.get_function_info(name)
-        @metadata[name]
-      end
+        def self.get_function_info(name)
+          @metadata[name]
+        end
 
-      def self.add_function_info(name, info)
-        @metadata[name] = info
+        def self.add_function_info(name, info)
+          @metadata[name] = info
+        end
       end
-    end
+    end.module
   end
 
   # Create a new Puppet DSL function.
@@ -70,7 +82,7 @@ module Puppet::Parser::Functions
   # extend the behavior and functionality of Puppet.
   #
   # See also [Docs: Custom
-  # Functions](http://docs.puppetlabs.com/guides/custom_functions.html)
+  # Functions](https://docs.puppetlabs.com/guides/custom_functions.html)
   #
   # @example Define a new Puppet DSL Function
   #     >> Puppet::Parser::Functions.newfunction(:double, :arity => 1,
@@ -113,7 +125,7 @@ module Puppet::Parser::Functions
   #   This string will be extracted by documentation generation tools.
   #
   # @option options [Integer] :arity (-1) the
-  #   [arity](http://en.wikipedia.org/wiki/Arity) of the function.  When
+  #   [arity](https://en.wikipedia.org/wiki/Arity) of the function.  When
   #   specified as a positive integer the function is expected to receive
   #   _exactly_ the specified number of arguments.  When specified as a
   #   negative number, the function is expected to receive _at least_ the

data/lib/puppet/parser/resource.rb

@@ -73,16 +73,16 @@ class Puppet::Parser::Resource < Puppet::Resource
   # Retrieve the associated definition and evaluate it.
   def evaluate
     return if evaluated?
-    @evaluated = true
-    if klass = resource_type and ! builtin_type?
-      finish
-      evaluated_code = klass.evaluate_code(self)
-
-      return evaluated_code
-    elsif builtin?
-      devfail "Cannot evaluate a builtin type (#{type})"
-    else
-      self.fail "Cannot find definition #{type}"
+    Puppet::Util::Profiler.profile("Evaluated resource #{self}", [:compiler, :evaluate_resource, self]) do
+      @evaluated = true
+      if builtin?
+        devfail "Cannot evaluate a builtin type (#{type})"
+      elsif resource_type.nil?
+        self.fail "Cannot find definition #{type}"
+      else
+        finish(false) # Call finish but do not validate
+        resource_type.evaluate_code(self)
+      end
     end
   end
 
@@ -98,13 +98,18 @@ class Puppet::Parser::Resource < Puppet::Resource
   end
 
   # Do any finishing work on this object, called before evaluation or
-  # before storage/translation.
-  def finish
+  # before storage/translation. The method does nothing the second time
+  # it is called on the same resource.
+  #
+  # @param do_validate [Boolean] true if validation should be performed
+  #
+  # @api private
+  def finish(do_validate = true)
     return if finished?
     @finished = true
     add_defaults
     add_scope_tags
-    validate
+    validate if do_validate
   end
 
   # Has this resource already been finished?
@@ -112,9 +117,9 @@ class Puppet::Parser::Resource < Puppet::Resource
     @finished
   end
 
-  def initialize(*args)
-    raise ArgumentError, "Resources require a hash as last argument" unless args.last.is_a? Hash
-    raise ArgumentError, "Resources require a scope" unless args.last[:scope]
+  def initialize(type, title, attributes)
+    raise ArgumentError, 'Resources require a hash as last argument' unless attributes.is_a? Hash
+    raise ArgumentError, 'Resources require a scope' unless attributes[:scope]
     super
 
     @source ||= scope.source
@@ -129,6 +134,15 @@ class Puppet::Parser::Resource < Puppet::Resource
     end
   end
 
+  def is_unevaluated_consumer?
+    # We don't declare a new variable here just to test. Saves memory
+    instance_variable_defined?(:@unevaluated_consumer)
+  end
+
+  def mark_unevaluated_consumer
+    @unevaluated_consumer = true
+  end
+
   # Merge an override resource in.  This will throw exceptions if
   # any overrides aren't allowed.
   def merge(resource)
@@ -168,11 +182,20 @@ class Puppet::Parser::Resource < Puppet::Resource
   alias []= set_parameter
 
   def to_hash
-    @parameters.inject({}) do |hash, ary|
-      param = ary[1]
-      # Skip "undef" and nil values.
-      hash[param.name] = param.value if param.value != :undef && !param.value.nil?
-      hash
+    @parameters.reduce({}) do |result, (_, param)|
+      value = param.value
+      value = (value == :undef) ? nil : value
+
+      unless value.nil?
+        case param.name
+        when :before, :subscribe, :notify, :require
+          value = value.flatten if value.is_a?(Array)
+          result[param.name] = value
+        else
+          result[param.name] = value
+        end
+      end
+      result
     end
   end
 
@@ -199,6 +222,71 @@ class Puppet::Parser::Resource < Puppet::Resource
     super || ((scope_resource = scope.resource) && !scope_resource.equal?(self) && scope_resource.raw_tagged?(tag_array))
   end
 
+  # Fills resource params from a capability
+  #
+  # This backs 'consumes => Sql[one]'
+  # @api private
+  def add_parameters_from_consume
+    return if self[:consume].nil?
+
+    map = {}
+    [ self[:consume] ].flatten.map do |ref|
+      # Assert that the ref really is a resource reference
+      raise Puppet::Error, "Invalid consume in #{self.ref}: #{ref} is not a resource" unless ref.is_a?(Puppet::Resource)
+
+      # Resolve references
+      cap = catalog.resource(ref.type, ref.title)
+      if cap.nil?
+        raise "Resource #{ref} could not be found; it might not have been produced yet"
+      end
+
+      # Ensure that the found resource is a capability resource
+      raise Puppet::Error, "Invalid consume in #{ref}: #{cap} is not a capability resource" unless cap.resource_type.is_capability?
+      cap
+    end.each do |cns|
+      # Establish mappings
+      blueprint = resource_type.consumes.find do |bp|
+        bp[:capability] == cns.type
+      end
+      # @todo lutter 2015-08-03: catch this earlier, can we do this during
+      # static analysis ?
+      raise "Resource #{self} tries to consume #{cns} but no 'consumes' mapping exists for #{self.resource_type} and #{cns.type}" unless blueprint
+
+      # setup scope that has, for each attr of cns, a binding to cns[attr]
+      scope.with_global_scope do |global_scope|
+        cns_scope = global_scope.newscope(:source => self, :resource => self)
+        cns.to_hash.each { |name, value| cns_scope[name.to_s] = value }
+  
+        # evaluate mappings in that scope
+        resource_type.arguments.keys.each do |name|
+          if expr = blueprint[:mappings][name]
+            # Explicit mapping
+            value = expr.safeevaluate(cns_scope)
+          else
+            value = cns[name]
+          end
+          unless value.nil?
+            # @todo lutter 2015-07-01: this should be caught by the checker
+            # much earlier. We consume several capres, at least two of which
+            # want to map to the same parameter (PUP-5080)
+            raise "Attempt to reassign attribute '#{name}' in '#{self}' caused by multiple consumed mappings to the same attribute" if map[name]
+            map[name] = value
+          end
+        end
+      end
+    end
+
+    map.each { |name, value| self[name] = value if self[name].nil? }
+  end
+
+  def offset
+    nil
+  end
+
+  def pos
+    nil
+  end
+
   private
 
   # Add default values from our definition.
@@ -213,7 +301,8 @@ class Puppet::Parser::Resource < Puppet::Resource
   end
 
   def add_scope_tags
-    if scope_resource = scope.resource
+    scope_resource = scope.resource
+    unless scope_resource.nil? || scope_resource.equal?(self)
       merge_tags(scope_resource)
     end
   end
@@ -256,11 +345,15 @@ class Puppet::Parser::Resource < Puppet::Resource
 
   # Make sure the resource's parameters are all valid for the type.
   def validate
-    @parameters.each do |name, param|
-      validate_parameter(name)
+    if builtin_type?
+      begin
+        @parameters.each { |name, value| validate_parameter(name) }
+      rescue => detail
+        self.fail Puppet::ParseError, detail.to_s + " on #{self}", detail
+      end
+    else
+      resource_type.validate_resource(self)
     end
-  rescue => detail
-    self.fail Puppet::ParseError, detail.to_s + " on #{self}", detail
   end
 
   def extract_parameters(params)