hiera 0.3.0 → 1.0.0rc4

data/CHANGELOG

@@ -0,0 +1,112 @@
+1.0.0rc4
+===
+6887137 (maint) Add Getting Started tutorial
+091fea3 Overhaul Hiera packaging
+7947dd6 Consolidate changelog, update erb templates
+4698cf1 (#15105) Update README YAML examples
+fcd3d2c (maint) Hiera now has a LICENSE file
+ca47463 Remove datadir value in hiera.yaml
+92a148f (maint) Fix failing Hiera::Util specs
+07366d8 Use File::ALT_SEPARATOR to test platform
+1b7f787 Add specs for Hiera::Util
+cdd7364 (#14867) Add windows support
+16c3dd3 (#12122) Merge arrays and hashes across backends
+480d86c (#12122) Correctly fall through backends during loop
+d0fcc57 Add default config to hiera
+3ec4165 match data in puppet ${::fact} style to get rid of puppet deprecation warnings
+
+1.0.0rc3
+===
+a1a885a Adding package task liberally copied from puppet-dashboard.
+3527443 Updating apple pkg task to refer to ext directory.
+445a6f2 ext is more appropriate than conf for a dirname.
+ade5567 Adding debian packaging to hiera.
+3ef4f96 Move redhat spec into conf dir, to standardize packaging.
+b6218b9 (maint) Hiera should raise an error when config is missing
+1df7201 Update mac packaging to dynamically generate preflight
+10f930f Add mac packaging to hiera
+22a98ee Remove Puppet parser functions
+0067cd2 (maint) Additional tests for Hiera array and hash lookups
+7312a95 (maint) Add Hiera acceptance tests
+9808a64 (#14514) Use default config when hiera.yaml is missing
+fd644b6 Add require hiera/backend to test
+fe3d509 Remove empty Puppet class definition
+
+1.0.0rc2
+===
+fa3081c Updating hiera.spec, CHANGELOG for Hiera 1.0.0rc1
+6de3d4c Moving CHANGES.txt to CHANGELOG. Updating CHANGELOG with git log from initial commit to v0.3.0 tag.
+dbf9fe3 Update README to reference deb and rpm packages
+f697b7f Updated hiera.spec to include puppet functions and put in the right place.
+540d2d6 (#14460) Add Puppet parser functions
+29c5ad7 Add hiera.spec for building hiera on redhat
+24ef7b9 (#13600) Monkey patch mktmpdir on ruby 1.8.5
+b7b3280 (maint) Make noop logger work on Ruby 1.9.x
+602c526 (maint) Fix failing spec test on Ruby 1.9.3
+82881e3 (#14150) Use tar_gz instead of tar
+7ef2821 (#14148) Add docs to Hiera packaging
+67f91d6 (#14147) Handle lack of rubygems gracefully
+f6c9f5d (#14124) Load rake tasks directly to fix tests for Ruby 1.9.x
+122b891 (#13641) Fix Hiera::Backend#parse_string to support :undefined in extra_data
+e6dea8e Commit caching on YAML Backend
+ddd5b66 Fix VERSION contant in lib/hiera.rb
+50f9771 hiera working on ruby 1.9.2/1.9.3, issue #10975 (http://projects.puppetlabs.com/issues/10975)
+794265f (maint) Code base cleanup, use 2 space indention.
+
+v0.3.0
+===
+
+d3b9c41 Fixes a quoting string issue
+2b6c3e7 Change Rakefile to autoversion
+1e7affb Rspec 2.7 has introduced some non backwards compatible changes that is exposed by using require with full paths.
+d136fc2 Switch branding to Puppet Labs
+731f2ef Fix warnings for rake
+0a53e96 fix failing test after sort was removed
+15fb3bd Do not sort arrays after building them as array data with complex data in them like hashes will fail to sort and raise exceptions
+f296a18 Test cases for #21
+dcf51ce Fixes: Hiera errors on empty yaml files
+d2d59d7 Revising backend bool spec test that nil /= false
+3cd4cec Fixes #15 Hiera return nil for false value
+17a2a98 Adding inventory_service option to scope.
+6ecab35 (#13) Hierarchy includes sub-hierarchy
+79b0d4b Corrected invalid YAML markup.
+c05823a Add specs to verify datatypes are being retained
+dedeb8c Make hiera handle boolean/numeric values correctly instead of just returning nil
+b78b3e4 Add type mismatch error output and test cases
+95e60ce Added additional spec tests for hash.
+f078414 Revising spec test for hash merge.
+9d72afa Hash was merged wrong direction.
+b4a90ba Adding hashes to CHANGES.txt
+90a23be Adding hash spec unit tests
+d1171d7 Adding hash support to hiera and yaml backend
+aa6af39 Fix logic around handling of defaults in array searches.  It used to always return [nil] for unknown data in array searches now it returns the supplied default.
+51a0a57 Fix versioning and autoload the Puppet logger
+fb88a3b Removed :json type from load_scope when parsing YAML format
+3839126 Fix typo
+8b7e036 More 0.2.0 update information
+93711a9 Fix spacing
+e804daf Update readme with information for 0.2.0
+4c990e1 Release 0.2.0
+1977a1e Improve tests based on recent changes
+3683a62 Simplify plugins that reads serialized data from files by moving more to the Backend module
+9d92041 Update changes file
+61e3574 Add Puppet logger plugin
+98a9525 Add array merge searches to the framework and the yaml backend Add array search to the command line Parse %{var}s in Strings, Arrays and Hashes correctly
+19cbf04 Update doco
+0f0837c Add querying facts via mcollective for scope of the CLI
+aeaef32 Remove some stray whitespace Turn a warn message into a debug message to avoid spamming logs
+27d6fd9 Fix /bin/env to /usr/bin/env (#1) Improve gem file
+dde5997 Release 0.1.0
+f69ebb2 Fix verbose checking in the cli Do not return empty sources Improve tests
+6d8130f Small language fix
+23ef93d Show variable expansion in string results
+362710c Add sample data to the readme
+dcc2ad2 Add a CLI query tool
+e3eb8de More todo items
+ced8d33 Add licence and readme
+e5d0106 avoid some debug noise
+dc41bcd Add comments and tests
+19e27c2 Small changes to the yaml backend, add a pluggable logging system and a console logger
+28cdd56 - simplify language by changing precedence with hierarchy - add gem bits - fall through to multiple backends till one provides an answer - make the backend writing process a bit easier
+19e03a7 Working YAML backend
+ca3efeb Add Readme

data/COPYING

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2010, 2011 R.I.Pienaar, Puppet Labs
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,237 @@
+# Hiera
+
+A simple pluggable Hierarchical Database.
+
+-
+**Tutorials:** Check the docs directory for tutorials.
+
+## Why?
+
+Hierarchical data is a good fit for the representation of infrastructure information.
+Consider the example of a typical company with 2 datacenters and on-site development,
+staging etc.
+
+All machines need:
+
+ - ntp servers
+ - sysadmin contacts
+
+By thinking about the data in a hierarchical manner you can resolve these to the most
+correct answer easily:
+
+<pre>
+     /------------- DC1 -------------\             /------------- DC2 -------------\
+    | ntpserver: ntp1.dc1.example.com |           | ntpserver: ntp1.dc2.example.com |
+    | sysadmin: dc1noc@example.com    |           |                                 |
+    | classes: users::dc1             |           | classes: users::dc2             |
+     \-------------------------------/             \-------------------------------/
+                                \                      /
+                                  \                  /
+                           /------------- COMMON -------------\
+                          | ntpserver: 1.pool.ntp.org          |
+                          | sysadmin: "sysadmin@%{domain}"     |
+                          | classes: users::common             |
+                           \----------------------------------/
+</pre>
+
+In this simple example machines in DC1 and DC2 have their own NTP servers, additionaly
+DC1 has its own sysadmin contact - perhaps because its a remote DR site - while DC2
+and all the other environments would revert to the common contact that would have the
+machines domain fact expanded into the result.
+
+The _classes_ variable can be searched using the array method which would build up a
+list of classes to include on a node based on the hierarchy.  Machines in DC1 would have
+the classes _users::common_ and _users::dc1_.
+
+The other environment like development and staging would all use the public NTP infrastructure.
+
+This is the data model that extlookup() have promoted in Puppet, Hiera has taken this
+data model and extracted it into a standalone project that is pluggable and have a few
+refinements over extlookup.
+
+## Enhancements over Extlookup
+
+Extlookup had just one backend, Hiera can be extended with your own backends and represent
+a few enhancements over the base Extlookup approach thanks to this.
+
+### Multiple backends are queried
+
+If you have a YAML and Puppet backend loaded and your users provide module defaults in the
+Puppet backend you can use your YAML data to override the Puppet data.  If the YAML doesnt
+provide an answer the Puppet backend will get an oppertunity to provide an answer.
+
+### More scope based variable expansion
+
+Extlookup could parse data like %{foo} into a scope lookup for the variable foo.  Hiera
+retains this ability and any Arrays or Hashes will be recursively searched for all strings
+that will then be parsed.
+
+The datadir and defaults are now also subject to variable parsing based on scope.
+
+### No CSV support by default
+
+We have not at present provided a backward compatible CSV backend.  A converter to
+YAML or JSON should be written. When the CSV backend was first chosen for Puppet the
+Puppet language only supports strings and arrays of strings which mapped well to CSV.
+Puppet has become (a bit) better wrt data and can now handle hashes and arrays of hashes
+so it's a good time to retire the old data format.
+
+### Array Searches
+
+Hiera can search through all the tiers in a hierarchy and merge the result into a single
+array.  This is used in the hiera-puppet project to replace External Node Classifiers by
+creating a Hiera compatible include function.
+
+## Future Enhancements
+
+ * More backends should be created
+ * A webservice that exposes the data
+ * Tools to help maintain the data files.  Ideally this would be Foreman and Dashboard
+   with their own backends
+
+## Installation
+
+Hiera is available as a Gem called _hiera_ and out of the box it comes with just a single
+YAML backend.
+
+Hiera is also available as a native package via apt (http://apt.puppetlabs.com) and yum (http://yum.puppetlabs.com). Instructions for adding these repositories can be found at http://docs.puppetlabs.com/guides/installation.html#debian-and-ubuntu and http://docs.puppetlabs.com/guides/installation.html#enterprise-linux respectively.
+
+At present JSON (github/ripienaar/hiera-json) and Puppet (hiera-puppet) backends are availble.
+
+## Configuration
+
+You can configure Hiera using a YAML file or by providing it Hash data in your code.  There
+isn't a default config path - the CLI script will probably assume _/etc/hiera.yaml_ though.
+The default data directory for file based storage is _/var/lib/hiera_.
+
+A sample configuration file can be seen here:
+
+<pre>
+---
+:backends:
+  - yaml
+  - puppet
+
+:logger: console
+
+:hierarchy:
+  - "%{location}"
+  - common
+
+:yaml:
+   :datadir: /etc/puppet/hieradata
+
+:puppet:
+   :datasource: data
+</pre>
+
+This configuration will require YAML files in  _/etc/puppet/hieradata_ these need to contain
+Hash data, sample files matching the hierarchy described in the _Why?_ section are below:
+
+_/etc/puppet/hieradata/dc1.yaml_:
+<pre>
+---
+ntpserver: ntp1.dc1.example.com
+sysadmin: dc1noc@example.com
+</pre>
+
+_/etc/puppet/hieradata/dc2.yaml_:
+<pre>
+---
+ntpserver: ntp1.dc2.example.com
+</pre>
+
+_/etc/puppet/hieradata/common.yaml_:
+<pre>
+---
+sysadmin: "sysadmin@%{domain}"
+ntpserver: 1.pool.ntp.org
+</pre>
+
+## Querying from CLI
+
+You can query your data from the CLI.  By default the CLI expects a config file in _/etc/hiera.yaml_
+but you can pass _--config_ to override that.
+
+This example searches Hiera for node data.  Scope is loaded from a Puppet created YAML facts
+store as found on your Puppet Masters.
+
+If no data is found and the facts had a location=dc1 fact the default would be _sites/dc1_
+
+<pre>
+$ hiera acme_version 'sites/%{location}' --yaml /var/lib/puppet/yaml/facts/example.com.yaml
+</pre>
+
+You can also supply extra facts on the CLI, assuming Puppet facts did not have a location fact:
+
+<pre>
+$ hiera acme_version 'sites/%{location}' location=dc1 --yaml /var/lib/puppet/yaml/facts/example.com.yaml
+</pre>
+
+Or if you use MCollective you can fetch the scope from a remote node's facts:
+
+<pre>
+$ hiera acme_version 'sites/%{location}' -m box.example.com
+</pre>
+
+You can also do array merge searches on the CLI:
+
+<pre>
+$ hiera -a classes location=dc1
+["users::common", "users::dc1"]
+</pre>
+
+## Querying from code
+
+This is the same query programatically as in the above CLI example:
+
+<pre>
+require 'rubygems'
+require 'hiera'
+require 'puppet'
+
+# load the facts for example.com
+scope = YAML.load_file("/var/lib/puppet/yaml/facts/example.com.yaml").values
+
+# create a new instance based on config file
+hiera = Hiera.new(:config => "/etc/puppet/hiera.yaml")
+
+# resolve the 'acme_version' variable based on scope
+#
+# given a fact location=dc1 in the facts file this will default to a branch sites/dc1
+# and allow hierarchical overrides based on the hierarchy defined in the config file
+puts "ACME Software Version: %s" % [ hiera.lookup("acme_version", "sites/%{location}", scope) ]
+</pre>
+
+## Extending
+
+There exist 2 backends at present in addition to the bundled YAML one.
+
+### JSON
+
+This can be found on github under _ripienaar/hiera-json_.  This is a good example
+of file based backends as Hiera provides a number of helpers to make writing these
+trivial.
+
+### Puppet
+
+This is much more complex and queries the data from the running Puppet state, it's found
+on GitHub under _ripienaar/hiera-puppet_.
+
+This is a good example to learn how to map your internal program state into what Hiera
+wants as I needed to do with the Puppet Scope.
+
+It includes a Puppet Parser Function to query the data from within Puppet.
+
+When used in Puppet you'd expect Hiera to log using the Puppet infrastructure, this
+plugin includes a Puppet Logger plugin for Hiera that uses the normal Puppet logging
+methods for all logging.
+
+## License
+
+See LICENSE file.
+
+## Support
+
+Please log tickets and issues at our [Projects site](http://projects.puppetlabs.com)
+