puppet 4.0.0 → 4.9.1

data/lib/puppet/parser/functions/lookup.rb

@@ -1,154 +1,132 @@
-Puppet::Parser::Functions.newfunction(:lookup, :type => :rvalue, :arity => -2, :doc => <<-'ENDHEREDOC') do |args|
-Looks up data defined using Data Binding, and Data Providers using different strategies. The lookup searches in
-Data Bindings first (if configured; typically Hiera), then in the environments data provider (if any), and last in
-the module's data provider (if any) of the module the call to lookup originates from. Thus, the global Data Binding
-has higher priority than data provided in the environment, which has higher priority than data provided in a module,
-
-The lookup function can be called in one of these ways:
-
-    lookup(name)
-    lookup(name, value_type)
-    lookup(name, value_type, merge)
-    lookup(name, value_type, merge, default_value)
-    lookup(options_hash)
-    lookup(name, options_hash)
-
-The function may optionally be called with a code block / lambda with the following signature:
-
-    lookup(...) |$name| { ... }
-
-The block, if present, is mutually exclusive to the `default_value` and will be called with the `name` used in the
-lookup when no value is found. The value produced by the block then becomes the value produced by the lookup.
-
-The meaning of the parameters or content of the options hash is:
-
-* `name` - The name or array of names to lookup (first found is returned)
-* `value_type` - The type to assert. Defaults to 'Data' See 'Type Specification' below.
-* `default_value` - The default value if there was no value found (must comply with the data type)
-* `override` - a hash with map from names to values that are used instead of the underlying bindings. If the name
-  is found here it wins. Defaults to an empty hash.
-* `default_values_hash` - a hash with map from names to values that are used as a last resort to obtain a value.
-  Defaults to an empty hash.
-* `merge` - A string of type Enum[unique, hash, merge] or a hash with the key 'strategy' set to that string. See
-  'Merge Strategies' below.
-
-It is not permitted to pass the `name` as both a parameter and in the options hash.
-
-The search will proceed as follows:
-1. For each name given in the `name` array (or once, if it's just one name):
-  - If a matching key is found in the `override` hash, it's value is immediately type checked and returned
-  - Search and optionally merge Data Binding, environment data providers, and module data providers
-  - Type check and return the value if a matching key is found
-2. For each name given in the `name` array (or once, if it's just one name):
-  - Type check and return the value if a matching key is found in the `default_values_hash`
-3. Type check and return either the given `default_value` or the result of calling the code block if either exist
-4. Raise an error indicating that no matching value was found
-
-*Merge Strategies*
-
-The default behavior of the lookup is to return the first value that is found for the given `name`. The optional
-`merge` parameter will change this so that a lookup makes an attempt to find values in all three sources (the Data
-Binder, the environment, and the module scope) and then merge these values according to the given strategy. This
-does not apply to values found in the 'override' hash. Such values are returned immediately without merging.
-Note that `merge` is passed on to allow the underlying provider to return a merged result
-
-The valid strategies are:
-- 'hash' Performs a simple hash-merge by overwriting keys of lower lookup priority. Merged values must be of Hash type
-- 'unique' Appends everything to an array containing no nested arrays and where all duplicates have been removed. Can
-   append values of Scalar or Array[Scalar] type
-- 'deep' Performs a deep merge on values of Array and Hash type. See documentation for the DeepMerge gem's deep_merge
-   operation for details and options.
-
-The 'deep' strategy can use additional options to control its behavior. Options can be passed as top level
-keys in the `merge` parameter when it is a given as a hash. Recognized options are:
-- 'knockout_prefix' Set to string value to signify prefix which deletes elements from existing element. Defaults is _undef_
-- 'sort_merged_arrays' Set to _true_ to sort all arrays that are merged together. Default is _false_
-- 'unpack_arrays' Set to string value used as a deliminator to join all array values and then split them again. Default is _undef_
-- 'merge_hash_arrays' Set to _true_ to merge hashes within arrays. Default is _false_
-
-*Type Specification*
-
-The type specification is a type in the Puppet Type System, e.g.:
-  * `Integer`, an integral value with optional range e.g.:
-    * `Integer[0, default]` - 0 or positive
-    * `Integer[default, -1]` - negative,
-    * `Integer[1,100]` - value between 1 and 100 inclusive
-  * `String`- any string
-  * `Float` - floating point number (same signature as for Integer for `Integer` ranges)
-  * `Boolean` - true of false (strict)
-  * `Array` - an array (of Data by default), or parameterized as `Array[<element_type>]`, where
-    `<element_type>` is the expected type of elements
-  * `Hash`,  - a hash (of default `Literal` keys and `Data` values), or parameterized as
-    `Hash[<value_type>]`, `Hash[<key_type>, <value_type>]`, where `<key_type>`, and
-    `<value_type>` are the types of the keys and values respectively
-    (key is `Literal` by default).
-  * `Data` - abstract type representing any `Literal` (including _undef_), `Array[Data]`, or `Hash[Literal, Data]`
-  * `Pattern[<p1>, <p2>, ..., <pn>]` - an enumeration of valid patterns (one or more) where
-     a pattern is a regular expression string or regular expression,
-     e.g. `Pattern['.com$', '.net$']`, `Pattern[/[a-z]+[0-9]+/]`
-  * `Enum[<s1>, <s2>, ..., <sn>]`, - an enumeration of exact string values (one or more)
-     e.g. `Enum[blue, red, green]`.
-  * `Variant[<t1>, <t2>,...<tn>]` - matches one of the listed types (at least one must be given)
-    e.g. `Variant[Integer[8000,8999], Integer[20000, 99999]]` to accept a value in either range
-  * `Regexp`- a regular expression (i.e. the result is a regular expression, not a string
-     matching a regular expression).
-
-For more options and details about types, see the Puppet Language Reference
-
-*Handling of undef*
-
-When no match is found for the given `name` when searching all sources, (including the `override`and
-`default_values_hash`), then the value used is either the `default_value` or the value produced by the given block.
-If neither is provided, then the lookup will always raise an error. Note that this only applies when there's no
-match for the given `name`. It does not happen when a value is found and that value happens to be _undef_.
-
-*Validation of returned value*
+module Puppet::Parser::Functions
+  newfunction(:lookup, :type => :rvalue, :arity => -2, :doc => <<-'ENDHEREDOC') do |args|
+Uses the Puppet lookup system to retrieve a value for a given key. By default,
+this returns the first value found (and fails compilation if no values are
+available), but you can configure it to merge multiple values into one, fail
+gracefully, and more.
+
+When looking up a key, Puppet will search up to three tiers of data, in the
+following order:
+
+1. Hiera.
+2. The current environment's data provider.
+3. The indicated module's data provider, if the key is of the form
+   `<MODULE NAME>::<SOMETHING>`.
+
+#### Arguments
+
+You must provide the name of a key to look up, and can optionally provide other
+arguments. You can combine these arguments in the following ways:
+
+* `lookup( <NAME>, [<VALUE TYPE>], [<MERGE BEHAVIOR>], [<DEFAULT VALUE>] )`
+* `lookup( [<NAME>], <OPTIONS HASH> )`
+* `lookup( as above ) |$key| { # lambda returns a default value }`
+
+Arguments in `[square brackets]` are optional.
+
+The arguments accepted by `lookup` are as follows:
+
+1. `<NAME>` (string or array) --- The name of the key to look up.
+    * This can also be an array of keys. If Puppet doesn't find anything for the
+    first key, it will try again with the subsequent ones, only resorting to a
+    default value if none of them succeed.
+2. `<VALUE TYPE>` (data type) --- A
+[data type](https://docs.puppetlabs.com/puppet/latest/reference/lang_data_type.html)
+that must match the retrieved value; if not, the lookup (and catalog
+compilation) will fail. Defaults to `Data` (accepts any normal value).
+3. `<MERGE BEHAVIOR>` (string or hash; see **"Merge Behaviors"** below) ---
+Whether (and how) to combine multiple values. If present, this overrides any
+merge behavior specified in the data sources. Defaults to no value; Puppet will
+use merge behavior from the data sources if present, and will otherwise do a
+first-found lookup.
+4. `<DEFAULT VALUE>` (any normal value) --- If present, `lookup` returns this
+when it can't find a normal value. Default values are never merged with found
+values. Like a normal value, the default must match the value type. Defaults to
+no value; if Puppet can't find a normal value, the lookup (and compilation) will
+fail.
+5. `<OPTIONS HASH>` (hash) --- Alternate way to set the arguments above, plus
+some less-common extra options. If you pass an options hash, you can't combine
+it with any regular arguments (except `<NAME>`). An options hash can have the
+following keys:
+    * `'name'` --- Same as `<NAME>` (argument 1). You can pass this as an
+    argument or in the hash, but not both.
+    * `'value_type'` --- Same as `<VALUE TYPE>` (argument 2).
+    * `'merge'` --- Same as `<MERGE BEHAVIOR>` (argument 3).
+    * `'default_value'` --- Same as `<DEFAULT VALUE>` (argument 4).
+    * `'default_values_hash'` (hash) --- A hash of lookup keys and default
+    values. If Puppet can't find a normal value, it will check this hash for the
+    requested key before giving up. You can combine this with `default_value` or
+    a lambda, which will be used if the key isn't present in this hash. Defaults
+    to an empty hash.
+    * `'override'` (hash) --- A hash of lookup keys and override values. Puppet
+    will check for the requested key in the overrides hash _first;_ if found, it
+    returns that value as the _final_ value, ignoring merge behavior. Defaults
+    to an empty hash.
+
+Finally, `lookup` can take a lambda, which must accept a single parameter.
+This is yet another way to set a default value for the lookup; if no results are
+found, Puppet will pass the requested key to the lambda and use its result as
+the default value.
+
+#### Merge Behaviors
+
+Puppet lookup uses a hierarchy of data sources, and a given key might have
+values in multiple sources. By default, Puppet returns the first value it finds,
+but it can also continue searching and merge all the values together.
+
+> **Note:** Data sources can use the special `lookup_options` metadata key to
+request a specific merge behavior for a key. The `lookup` function will use that
+requested behavior unless you explicitly specify one.
+
+The valid merge behaviors are:
+
+* `'first'` --- Returns the first value found, with no merging. Puppet lookup's
+default behavior.
+* `'unique'` (called "array merge" in classic Hiera) --- Combines any number of
+arrays and scalar values to return a merged, flattened array with all duplicate
+values removed. The lookup will fail if any hash values are found.
+* `'hash'` --- Combines the keys and values of any number of hashes to return a
+merged hash. If the same key exists in multiple source hashes, Puppet will use
+the value from the highest-priority data source; it won't recursively merge the
+values.
+* `'deep'` --- Combines the keys and values of any number of hashes to return a
+merged hash. If the same key exists in multiple source hashes, Puppet will
+recursively merge hash or array values (with duplicate values removed from
+arrays). For conflicting scalar values, the highest-priority value will win.
+* `{'strategy' => 'first|unique|hash'}` --- Same as the string versions of these
+merge behaviors.
+* `{'strategy' => 'deep', <DEEP OPTION> => <VALUE>, ...}` --- Same as `'deep'`,
+but can adjust the merge with additional options. The available options are:
+    * `'knockout_prefix'` (string or undef) --- A string prefix to indicate a
+    value should be _removed_ from the final result. Defaults to `undef`, which
+    disables this feature.
+    * `'sort_merged_arrays'` (boolean) --- Whether to sort all arrays that are
+    merged together. Defaults to `false`.
+    * `'merge_hash_arrays'` (boolean) --- Whether to merge hashes within arrays.
+    Defaults to `false`.
+
+#### Examples
+
+Look up a key and return the first value found:
+
+    lookup('ntp::service_name')
+
+Do a unique merge lookup of class names, then add all of those classes to the
+catalog (like `hiera_include`):
+
+    lookup('classes', Array[String], 'unique').include
+
+Do a deep hash merge lookup of user data, but let higher priority sources
+remove values by prefixing them with `--`:
+
+    lookup( { 'name'  => 'users',
+              'merge' => {
+                'strategy'        => 'deep',
+                'knockout_prefix' => '--',
+              },
+    })
 
-The produced value is subject to type validation using the `value_type` (if given) and an error is raised unless
-the resulting value is of correct type.
-
-*Examples*
-
-When called with one argument; **the name**, it
-returns the bound value with the given name after having  asserted it has the default datatype `Data`:
-
-    lookup('the_name')
-
-When called with two arguments; **the name**, and **the expected type**, it
-returns the bound value with the given name after having asserted it has the given data
-type ('String' in the example):
-
-    lookup('the_name', String)
-
-When called with four arguments, **the name**, the **expected type**, the **merge** strategy, and a
-**default value**, it returns the bound value with the given name, or the default after having asserted the value
-has the given data type:
-
-    lookup('the_name', String, undef, 'Fred')
-    lookup('the_name', Array[String], 'unique', [Fred])
-
-Using a lambda to provide a default value by calling a function:
-
-    lookup('the_size', Integer[1,100]) |$name| {
-      obtain_size_default()
-    }
-
-There are two ways to make lookup return undef when no matching key was found instead of raising an error.
-Either call it with four arguments (the `merge` argument must be present even when using the default strategy
-to ensure that the four argument variant is used):
-
-     $are_you_there = lookup('peekaboo', Optional[String], undef, undef)
-
-or call it using an options hash:
-
-     $are_you_there = lookup('peekaboo', { 'default_value' => undef })
-     $are_you_there = lookup({ 'name' => 'peekaboo', 'default_value' => undef })
-
-or with a block that produces an undef value:
-
-     $are_you_there = lookup('peekaboo', Optional[String]) |$name| { undef }
-
-- Since 4.0.0
 ENDHEREDOC
-  function_fail(["lookup() has been converted to 4x API"])
+  Error.is4x('lookup')
+end
 end

data/lib/puppet/parser/functions/map.rb

@@ -3,39 +3,74 @@ Puppet::Parser::Functions::newfunction(
   :type => :rvalue,
   :arity => -3,
   :doc => <<-DOC
-Applies a parameterized block to each element in a sequence of entries from the first
-argument and returns an array with the result of each invocation of the parameterized block.
+Applies a [lambda](https://docs.puppetlabs.com/puppet/latest/reference/lang_lambdas.html)
+to every value in a data structure and returns an array containing the results.
 
-This function takes two mandatory arguments: the first should be an Array, Hash, or of Enumerable type
-(integer, Integer range, or String), and the second a parameterized block as produced by the puppet syntax:
+This function takes two mandatory arguments, in this order:
 
-      $a.map |$x| { ... }
-      map($a) |$x| { ... }
+1. An array or hash the function will iterate over.
+2. A lambda, which the function calls for each element in the first argument. It can
+request one or two parameters.
 
-When the first argument `$a` is an Array or of enumerable type, the block is called with each entry in turn.
-When the first argument is a hash the entry is an array with `[key, value]`.
+**Example**: Using the `map` function
 
-Example Using map with two arguments
+`$transformed_data = $data.map |$parameter| { <PUPPET CODE BLOCK> }`
 
-     # Turns hash into array of values
-     $a.map |$x|{ $x[1] }
+or
 
-     # Turns hash into array of keys
-     $a.map |$x| { $x[0] }
+`$transformed_data = map($data) |$parameter| { <PUPPET CODE BLOCK> }`
 
-When using a block with 2 parameters, the element's index (starting from 0) for an array, and the key for a hash
-is given to the block's first parameter, and the value is given to the block's second parameter.args.
+When the first argument (`$data` in the above example) is an array, Puppet passes each
+value in turn to the lambda.
 
-Example Using map with two arguments
+**Example**: Using the `map` function with an array and a one-parameter lambda
 
-     # Turns hash into array of values
-     $a.map |$key,$val|{ $val }
+~~~ puppet
+# For the array $data, return an array containing each value multiplied by 10
+$data = [1,2,3]
+$transformed_data = $data.map |$items| { $items * 10 }
+# $transformed_data contains [10,20,30]
+~~~
 
-     # Turns hash into array of keys
-     $a.map |$key,$val|{ $key }
+When the first argument is a hash, Puppet passes each key and value pair to the lambda
+as an array in the form `[key, value]`.
+
+**Example**: Using the `map` function with a hash and a one-parameter lambda
+
+~~~ puppet
+# For the hash $data, return an array containing the keys
+$data = {'a'=>1,'b'=>2,'c'=>3}
+$transformed_data = $data.map |$items| { $items[0] }
+# $transformed_data contains ['a','b','c']
+~~~
+
+When the first argument is an array and the lambda has two parameters, Puppet passes the
+array's indexes (enumerated from 0) in the first parameter and its values in the second
+parameter.
+
+**Example**: Using the `map` function with an array and a two-parameter lambda
+
+~~~ puppet
+# For the array $data, return an array containing the indexes
+$data = [1,2,3]
+$transformed_data = $data.map |$index,$value| { $index }
+# $transformed_data contains [0,1,2]
+~~~
+
+When the first argument is a hash, Puppet passes its keys to the first parameter and its
+values to the second parameter.
+
+**Example**: Using the `map` function with a hash and a two-parameter lambda
+
+~~~ puppet
+# For the hash $data, return an array containing each value
+$data = {'a'=>1,'b'=>2,'c'=>3}
+$transformed_data = $data.map |$key,$value| { $value }
+# $transformed_data contains [1,2,3]
+~~~
 
 - Since 4.0.0
 DOC
 ) do |args|
-  function_fail(["map() is only available when parser/evaluator future is in effect"])
+  Error.is4x('map')
 end

data/lib/puppet/parser/functions/match.rb

@@ -2,26 +2,42 @@ Puppet::Parser::Functions::newfunction(
   :match,
   :arity => 2,
   :doc => <<-DOC
-Returns the match result of matching a String or Array[String] with one of:
+Matches a regular expression against a string and returns an array containing the match
+and any matched capturing groups.
 
-* Regexp
-* String - transformed to a Regexp
-* Pattern type
-* Regexp type
+The first argument is a string or array of strings. The second argument is either a
+regular expression, regular expression represented as a string, or Regex or Pattern
+data type that the function matches against the first argument.
 
-Returns An Array with the entire match at index 0, and each subsequent submatch at index 1-n.
-If there was no match `undef` is returned. If the value to match is an Array, a array
-with mapped match results is returned.
+The returned array contains the entire match at index 0, and each captured group at
+subsequent index values. If the value or expression being matched is an array, the
+function returns an array with mapped match results.
 
-Example matching:
+If the function doesn't find a match, it returns 'undef'.
 
-  "abc123".match(/([a-z]+)[1-9]+/)    # => ["abc"]
-  "abc123".match(/([a-z]+)([1-9]+)/)  # => ["abc", "123"]
+**Example**: Matching a regular expression in a string
 
-See the documentation for "The Puppet Type System" for more information about types.
+~~~ ruby
+$matches = "abc123".match(/[a-z]+[1-9]+/)
+# $matches contains [abc123]
+~~~
+
+**Example**: Matching a regular expressions with grouping captures in a string
+
+~~~ ruby
+$matches = "abc123".match(/([a-z]+)([1-9]+)/)
+# $matches contains [abc123, abc, 123]
+~~~
+
+**Example**: Matching a regular expression with grouping captures in an array of strings
+
+~~~ ruby
+$matches = ["abc123","def456"].match(/([a-z]+)([1-9]+)/)
+# $matches contains [[abc123, abc, 123], [def456, def, 456]]
+~~~
 
 - Since 4.0.0
 DOC
 ) do |args|
-  function_fail(["match() is only available when parser/evaluator future is in effect"])
+  Error.is4x('match')
 end