puppet 3.0.0.rc5 → 3.0.0.rc7

data/LICENSE

@@ -1,6 +1,6 @@
    Puppet - Automating Configuration Management.
 
-   Copyright (C) 2011 Puppet Labs Inc
+   Copyright (C) 2005-2012 Puppet Labs Inc
 
    Puppet Labs can be contacted at: info@puppetlabs.com
 

data/README_DEVELOPER.md

@@ -25,7 +25,133 @@ When we setup CI nodes, but this is not standard or expected behavior.
 Please consider rbenv instead of rvm.  The default behavior of rvm is difficult
 to maintain with `set -e` shell environments.
 
-# Dependencies #
+# Two Types of Catalog
+
+When working on subsystems of Puppet that deal with the catalog it is important
+to be aware of the two different types of Catalog.  I often ran into this when
+working in Professional Services when I built a small tool to diff two catalogs
+to determine if an upgrade in Puppet produces the same configuration catalogs.
+As a developer I've run into this difference while working on spec tests for
+the static compiler and working on spec tests for types and providers.
+
+The two different types of catalog becomes relevant when writing spec tests
+because we frequently need to wire up a fake catalog so that we can exercise
+types, providers, or terminii that filter the catalog.
+
+The two different types of catalogs are so-called "resource" catalogs and "RAL"
+(resource abstraction layer) catalogs.  At a high level, the resource catalog
+is the in-memory object we serialize and transfer around the network.  The
+compiler terminus is expected to produce a resource catalog.  The agent takes a
+resource catalog and converts it into a RAL catalog.  The RAL catalog is what
+is used to apply the configuration model to the system.
+
+Resource dependency information is most easily obtained from a RAL catalog by
+walking the graph instance produced by the `relationship_graph` method.
+
+## Resource Catalog
+
+If you're writing spec tests for something that deals with a catalog "server
+side," a new catalog terminus for example, then you'll be dealing with a
+resource catalog.  You can produce a resource catalog suitable for spec tests
+using something like this:
+
+    let(:catalog) do
+      catalog = Puppet::Resource::Catalog.new("node-name-val") # NOT certname!
+      rsrc = Puppet::Resource.new("file", "sshd_config",
+        :parameters => {
+          :ensure => 'file',
+          :source => 'puppet:///modules/filetest/sshd_config',
+        }
+      )
+      rsrc.file = 'site.pp'
+      rsrc.line = 21
+      catalog.add_resource(rsrc)
+    end
+
+The resources in this catalog may be accessed using `catalog.resources`.
+Resource dependencies are not easily walked using a resource catalog however.
+To walk the dependency tree convert the catalog to a RAL catalog as described
+in
+
+## RAL Catalog
+
+The resource catalog may be converted to a RAL catalog using `catalog.to_ral`.
+The RAL catalog contains `Puppet::Type` instances instead of `Puppet::Resource`
+instances as is the case with the resource catalog.
+
+One very useful feature of the RAL catalog are the methods to work with
+resource relationships.  For example:
+
+    irb> catalog = catalog.to_ral
+    irb> graph = catalog.relationship_graph
+    irb> pp graph.edges
+    [{ Notify[alpha] => File[/tmp/file_20.txt] },
+     { Notify[alpha] => File[/tmp/file_21.txt] },
+     { Notify[alpha] => File[/tmp/file_22.txt] },
+     { Notify[alpha] => File[/tmp/file_23.txt] },
+     { Notify[alpha] => File[/tmp/file_24.txt] },
+     { Notify[alpha] => File[/tmp/file_25.txt] },
+     { Notify[alpha] => File[/tmp/file_26.txt] },
+     { Notify[alpha] => File[/tmp/file_27.txt] },
+     { Notify[alpha] => File[/tmp/file_28.txt] },
+     { Notify[alpha] => File[/tmp/file_29.txt] },
+     { File[/tmp/file_20.txt] => Notify[omega] },
+     { File[/tmp/file_21.txt] => Notify[omega] },
+     { File[/tmp/file_22.txt] => Notify[omega] },
+     { File[/tmp/file_23.txt] => Notify[omega] },
+     { File[/tmp/file_24.txt] => Notify[omega] },
+     { File[/tmp/file_25.txt] => Notify[omega] },
+     { File[/tmp/file_26.txt] => Notify[omega] },
+     { File[/tmp/file_27.txt] => Notify[omega] },
+     { File[/tmp/file_28.txt] => Notify[omega] },
+     { File[/tmp/file_29.txt] => Notify[omega] }]
+
+If the `relationship_graph` method is throwing exceptions at you, there's a
+good chance the catalog is not a RAL catalog.
+
+## Settings Catalog ##
+
+Be aware that Puppet creates a mini catalog and applies this catalog locally to
+manage file resource from the settings.  This behavior made it difficult and
+time consuming to track down a race condition in
+[2888](http://projects.puppetlabs.com/issues/2888).
+
+Even more surprising, the `File[puppetdlockfile]` resource is only added to the
+settings catalog if the file exists on disk.  This caused the race condition as
+it will exist when a separate process holds the lock while applying the
+catalog.
+
+It may be sufficient to simply be aware of the settings catalog and the
+potential for race conditions it presents.  An effective way to be reasonably
+sure and track down the problem is to wrap the File.open method like so:
+
+    # We're wrapping ourselves around the File.open method.
+    # As described at: http://goo.gl/lDsv6
+    class File
+      WHITELIST = [ /pidlock.rb:39/ ]
+
+      class << self
+        alias xxx_orig_open open
+      end
+
+      def self.open(name, *rest, &block)
+        # Check the whitelist for any "good" File.open calls against the #
+        puppetdlock file
+        white_listed = caller(0).find do |line|
+          JJM_WHITELIST.find { |re| re.match(line) }
+        end
+
+        # If you drop into IRB here, take a look at your caller, it might be
+        # the ghost in the machine you're looking for.
+        binding.pry if name =~ /puppetdlock/ and not white_listed
+        xxx_orig_open(name, *rest, &block)
+      end
+    end
+
+The settings catalog is populated by the `Puppet::Util::Settings#to\_catalog`
+method.
+
+# Ruby Dependencies #
 
 Puppet is considered an Application as it relates to the recommendation of
 adding a Gemfile.lock file to the repository and the information published at
@@ -226,4 +352,86 @@ Please do not monkey patch the constant `Puppet::PUPPETVERSION` or obtain the
 version using the constant.  The only supported way to set and get the Puppet
 version is through the accessor methods.
 
+# Static Compiler
+
+The static compiler was added to Puppet in the 2.7.0 release.
+[1](http://links.puppetlabs.com/static-compiler-announce)
+
+The static compiler is intended to provide a configuration catalog that
+requires a minimal amount of network communication in order to apply the
+catalog to the system.  As implemented in Puppet 2.7.x and Puppet 3.0.x this
+intention takes the form of replacing all of the source parameters of File
+resources with a content parameter containing an address in the form of a
+checksum.  The expected behavior is that the process applying the catalog to
+the node will retrieve the file content from the FileBucket instead of the
+FileServer.
+
+The high level approach can be described as follows.  The `StaticCompiler` is a
+terminus that inserts itself between the "normal" compiler terminus and the
+request.  The static compiler takes the resource catalog produced by the
+compiler and filters all File resources.  Any file resource that contains a
+source parameter with a value starting with 'puppet://' is filtered in the
+following way in a "standard" single master / networked agents deployment
+scenario:
+
+ 1. The content, owner, group, and mode values are retrieved from th
+     FileServer by the master.
+ 2. The file content is stored in the file bucket on the master.
+ 3. The source parameter value is stripped from the File resource.
+ 4. The content parameter value is set in the File resource using the form
+    '{XXX}1234567890' which can be thought of as a content address indexed by
+    checksum.
+ 5. The owner, group and mode values are set in the File resource if they are
+    not already set.
+ 6. The filtered catalog is returned in the response.
+
+In addition to the catalog terminus, the process requesting the catalog needs
+to obtain the file content.  The default behavior of `puppet agent` is to
+obtain file contents from the local client bucket.  The method we expect users
+to employ to reconfigure the agent to use the server bucket is to declare the
+`Filebucket[puppet]` resource with the address of the master. For example:
+
+    node default {
+      filebucket { puppet:
+        server => $server,
+        path   => false,
+      }
+      class { filetest: }
+    }
+
+This special filebucket resource named "puppet" will cause the agent to fetch
+file contents specified by checksum from the remote filebucket instead of the
+default clientbucket.
+
+## Quick start
+
+Create a module that recursively downloads something.  The jeffmccune-filetest
+module will recursively copy the rubygems source tree.
+
+    $ puppet module install jeffmccune-filetest
+
+Start the master with the StaticCompiler turned on:
+
+    $ puppet master \
+        --catalog_terminus=static_compiler \
+        --verbose \
+        --no-daemonize
+
+Add the special Filebucket[puppet] resource:
+
+    # site.pp
+    node default {
+      filebucket { puppet: server => $server, path => false }
+      class { filetest: }
+    }
+
+Get the static catalog:
+
+    $ puppet agent --test
+
+You should expect all file metadata to be contained in the catalog, including a
+checksum representing the content.  When managing an out of sync file resource,
+the real contents should be fetched from the server instead of the
+clientbucket.
+
 EOF

data/README_HIERA.md

@@ -0,0 +1,148 @@
+What?
+=====
+
+A data backend for Hiera that can query the internal Puppet
+scope for data.  The data structure and approach is heavily
+based on work by Nigel Kersten but made more configurable and
+with full hierarchy.
+
+It also includes a Puppet function that works like extlookup()
+but uses the Hiera backends.
+
+Usage?
+======
+
+Hiera supports the concept of chaining backends together in order,
+using this we can create a very solid module author/module user
+experience.
+
+Module Author
+-------------
+
+A module author wants to create a configurable module that has sane
+defaults but want to retain the ability for users to configure it.
+
+We'll use a simple NTP config class as an example.
+
+<pre>
+class ntp::config($ntpservers = hiera("ntpservers")) {
+   file{"/etc/ntp.conf":
+       content => template("ntp.conf.erb")
+   }
+}
+</pre>
+
+We create a class that takes as parameters a list of NTP servers.
+
+The module author wants to create a works-out-of-the-box experience
+so creates a data class for the NTP module:
+
+<pre>
+class ntp::data {
+   $ntpservers = ["1.pool.ntp.org", "2.pool.ntp.org"]
+}
+</pre>
+
+Together this creates a default sane setup.
+
+Module User
+-----------
+
+The module user has a complex multi data center setup, he wants to use
+the NTP module from the forge and configure it for his needs.
+
+The user creates a set of default data for his organization, he can do
+this in data files or in Puppet.  We'll show a Puppet example.
+
+<pre>
+class data::common {
+   $ntpservers = ["ntp1.example.com", "ntp2.example.com"]
+}
+</pre>
+
+Being part of the actual code this data is subject to strict change
+control.  This is needed as its data that can potentially affect all
+machines in all locations.
+
+The user has a fact called _location_ that contains, for example, a name
+of the data center.
+
+He decides to create JSON based data for the data centers, being just data
+that applies to one data center this data is not subject to as strict
+change controls and so does not live with the code:
+
+He creates _/var/lib/hiera/dc1.json_ with the following:
+
+<pre>
+{"ntpservers" : ["ntp1.dc1.example.com", "ntp2.dc1.example.com"]}
+</pre>
+
+Machines in dc1 will now use specific NTP servers while all the rest will
+use the data in _data::common_
+
+The module user can now just declare the class on his nodes:
+
+<pre>
+node "web1" {
+   include ntp::config
+}
+</pre>
+
+For true one-off changes, the user can use the full paramterized class approach
+that will completely disable the Hiera handling of this data item.  He could
+also use an ENC to supply this data.
+
+<pre>
+node "web2" {
+   class{"ntp::config": ntpservers => ["another.example.com"]}
+}
+</pre>
+
+This behavior is thanks to Hiera's ability to search through multiple backends
+for data picking the first match.  We can have the JSON searched before the internal
+Puppet data.
+
+To achieve this setup the module user needs to configure Hiera in _/etc/puppet/hiera.yaml_:
+
+<pre>
+---
+:backends:
+  - json
+  - puppet
+
+:hierarchy:
+  - "%{location}"
+  - common
+
+:json:
+  :datadir: /var/lib/hiera
+
+:puppet:
+	:datasource: data
+</pre>
+
+Converting from extlookup?
+==========================
+
+A simple converter is included called _extlookup2hiera_ and it can convert from CSV to JSON or YAML:
+
+<pre>
+$ extlookup2hiera --in common.csv --out common.json --json
+</pre>
+
+Installation?
+=============
+
+It's not 100% ready for prime time, shortly a simple _gem install hiera-puppet_ on your master will do it.
+
+For the moment the Gem install will place the Puppet Parser Function where Puppet cannot find it, you should
+copy it out and distribute it to your master using Pluginsync or something similar
+
+License
+=======
+
+See LICENSE file.
+
+Support
+=======
+Please log tickets and issues at our [Projects site](http://projects.puppetlabs.com)

data/Rakefile

@@ -20,43 +20,41 @@ end
 require 'rake'
 require 'rspec'
 require "rspec/core/rake_task"
+require 'yaml'
 
+Dir['tasks/**/*.rake'].each { |t| load t }
+Dir['ext/packaging/tasks/**/*'].sort.each { |t| load t }
 
-%x{which git &> /dev/null}
-if $?.success? and File.exist?('.git')
-  # remove the git hash from git describe string
-  Puppet.version = %x{git describe}.chomp.gsub('-','.').split('.')[0..3].join('.')
+begin
+  @build_defaults ||= YAML.load_file('ext/build_defaults.yaml')
+  @packaging_url  = @build_defaults['packaging_url']
+  @packaging_repo = @build_defaults['packaging_repo']
+rescue
+  STDERR.puts 'Unable to read the packaging repo info from ext/build_defaults.yaml'
 end
 
-Dir['tasks/**/*.rake'].each { |t| load t }
-
-FILES = FileList[
-    '[A-Z]*',
-    'install.rb',
-    'bin/**/*',
-    'sbin/**/*',
-    'lib/**/*',
-    'conf/**/*',
-    'man/**/*',
-    'examples/**/*',
-    'ext/**/*',
-    'tasks/**/*',
-    'spec/**/*'
-]
-
-Rake::PackageTask.new("puppet", Puppet.version) do |pkg|
-    pkg.package_dir = 'pkg'
-    pkg.need_tar_gz = true
-    pkg.package_files = FILES.to_a
+namespace :package do
+  desc "Bootstrap packaging automation, e.g. clone into packaging repo"
+  task :bootstrap do
+    if File.exist?("ext/#{@packaging_repo}")
+      puts "It looks like you already have ext/#{@packaging_repo}. If you don't like it, blow it away with package:implode."
+    else
+      cd 'ext' do
+        %x{git clone #{@packaging_url}}
+      end
+    end
+  end
+  desc "Remove all cloned packaging automation"
+  task :implode do
+    rm_rf "ext/#{@packaging_repo}"
+  end
 end
 
+
 task :default do
     sh %{rake -T}
 end
 
-desc "Create the tarball and the gem - use when releasing"
-task :puppetpackages => [:gem, :package]
-
 RSpec::Core::RakeTask.new do |t|
     t.pattern ='spec/{unit,integration}/**/*.rb'
     t.fail_on_error = true

data/bin/extlookup2hiera

@@ -0,0 +1,56 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'csv'
+
+options = {:in => nil, :out => nil, :format => :yaml}
+
+OptionParser.new do |opts|
+  opts.banner = "Converter for extlookup CSV files into Hiera JSON and YAML files"
+
+  opts.on("--in FILE", "-i", "Input CSV file") do |v|
+    options[:in] = v
+  end
+
+  opts.on("--out FILE", "-o", "Output Hiera file") do |v|
+    options[:out] = v
+  end
+
+  opts.on("--json", "-j", "Create JSON format file") do |v|
+    options[:format] = :json
+  end
+end.parse!
+
+if options[:in].nil? || options[:out].nil?
+  STDERR.puts "Please specify an input and output file with --in and --out"
+  exit 1
+end
+
+unless File.exist?(options[:in])
+  STDERR.puts "Cannot find input file #{options[:in]}"
+  exit 1
+end
+
+csvdata = CSV.read(options[:in])
+hieradata = {}
+
+csvdata.each do |d|
+  d = d.map{|item| item.to_s}
+
+  if d.size > 2
+    hieradata[d[0]] = d[1, d.size].flatten
+  else
+    hieradata[d[0]] = d[1]
+  end
+end
+
+case options[:format]
+when :yaml
+  require 'yaml'
+  File.open(options[:out], "w") {|f| f.write hieradata.to_yaml}
+when :json
+  require 'rubygems'
+  require 'json'
+  File.open(options[:out], "w") {|f| f.write JSON.pretty_generate hieradata}
+end
+