puppet 2.7.20.rc1 → 2.7.20

data/ext/packaging/README.md

@@ -4,7 +4,10 @@ This is a repository for packaging automation for Puppet Labs software.
 The goal is to abstract and automate packaging processes beyond individual
 software projects to a level where this repo can be cloned inside any project
 and used to build Debian and Redhat packages, as well as gems, apple packages
-and tarballs.
+and tarballs. This repo is currently under heavy development and in a state
+flux, and it should not be considered to have a formal API. However, every
+effort is being made to ensure existing tasks/behavior are not broken as we
+continue to iterate and improve upon it.
 
 ##Using the Packaging Repo
 
@@ -22,7 +25,8 @@ create rpms and debs, but any build dependencies will need to be satisifed by
 the building host, and any dynamically generated dependencies may result in
 packages that are only suitable for the OS/version of the build host. However,
 for rolling one's own debs and rpms or for use in environments without many
-OSes/versions, this may work just fine.
+OSes/versions, this may work just fine. To build an rpm using the packaging
+repo, do a `rake package:rpm`. To build a deb, use `rake package:deb`.
 
 `pl:` namespaced tasks rely on a slighly more complex toolchain for packaging
 inside clean chroot environments for the various operating systems and
@@ -32,12 +36,48 @@ pdebuild and [cowbuilder](http://wiki.debian.org/cowbuilder). For the most
 part, these tasks are keyed to puppetlabs infrastructure, and are used by the
 Release Engineering team to create release packages. However, they can
 certainly be modified to suit other environments, and much effort went into
-making tasks as modular and reusable as possible.
+making tasks as modular and reusable as possible. Several Puppet Labs-specific
+tasks are only available if the file '~/.packaging/builder\_data.yaml' is present.
+This file is created by the `pl:fetch` task, which pulls down another yaml file
+from a [separate build data repository](https://github.com/puppetlabs/build-data),
+which contains additional settings/data specific to Puppet Labs release
+infrastructure. The goal in separating these data and tasks out is to refrain
+from presenting by default yet more Puppet Labs-specific tasks that aren't
+generally consumable by everyone. To build a deb from a local repository using
+a `pl` task, ssh into a builder (e.g., one stood up using the modules detailed
+below) and clone the source repo, e.g. puppet. Then, run `rake package:bootstrap`
+and `rake pl:deb` to create a deb, and `rake pl:mock` to make an rpm (on a debian
+or redhat host, respectively).
+
+There is also a `pe:` namespace, for the building of Puppet Labs' Puppet
+Enterprise packages that have been converted to using this repo. The `pe:`
+tasks rely heavily on PL internal infrastructure, and are not generally useful
+outside of this environment. To create packages, in the source repository run
+`rake package:bootstrap`, followed by `rake pl:fetch`. These two commands
+bootstrap the packaging environment and pull in the additional data needed for
+PE building (see `pl:fetch` notes above).
+Then, to make a debian package, run `rake pe:deb`, and to make an rpm, run
+`rake pe:mock`. There are also `pe:deb_all` and `pe:mock_all` tasks, which build
+packages against all shipped debian/redhat targets. The `pe:deb_all` task is not
+generally necessary for developer use for building test packages; the `pe:deb`
+task creates a package that will work against virtually all supported PE debian
+versions. The same is generally true for PE internal rpms, but because of variances
+in build macros for rpm, rpms should generally be built with `pe:mock_all`, and
+then the desired version installed, or by building only for a specific target.
+This is accomplished by passing MOCK=<mock> to the rake call, e.g. `rake pe:mock MOCK=<mock>`.
+The available mocks are listed in `ext/build_defaults.yaml` after `final_mocks:`.
+For PE, the mocks are formatted as `pupent-<peversion>-<distversion>-<arch>`, e.g.
+`pupent-2.7-el5-i386`. To build for a specific target, set `MOCK=<mock>` to the mock
+that matches the target. The `pe:deb` and `pe:mock` tasks work by building on a
+remote builder using the current committed state of the source repository. To forego
+remote building and build on the local station (e.g., by ssh-ing into a remote
+builder first), the tasks `pe:local_mock` and `pe:local_deb` build using the
+local host.
 
 A puppet module,
 [puppetlabs-debbuilder](https://github.com/puppetlabs/puppetlabs-debbuilder),
 has been created to stand up a debian build host compatible with the debian
-side this packaging repo. The rpm-side module,
+side of this packaging repo. The rpm-side module,
 [puppetlabs-rpmbuilder](https://github.com/puppetlabs/puppetlabs-rpmbuilder),
 is currently a work in progress and.
 
@@ -47,8 +87,9 @@ package:implode`.
 ##Setting up projects for the Packaging Repo
 
 The packaging repo requires many project-side artifacts inside the ext
-directory at the top level.  It expects the following directory structure in
-the project
+directory at the top level. [facter](https://github.com:puppetlabs/facter) and
+[hiera](https://github.com:puppetlabs/hiera) are good examples.
+It expects the following directory structure in the project
 
 *   ext/{debian,redhat,osx}
 

data/ext/packaging/tasks/00_utils.rake

@@ -234,7 +234,7 @@ def gpg_sign_file(file)
   if gpg
     sh "#{gpg} --armor --detach-sign -u #{@gpg_key} #{file}"
   else
-    STDERR.puts "No gpg available. Cannot sign tarball. Exiting..."
+    STDERR.puts "No gpg available. Cannot sign #{file}. Exiting..."
     exit 1
   end
 end
@@ -268,9 +268,14 @@ def ln(target, name)
   FileUtils.ln(name, target, :force => true, :verbose => true)
 end
 
-def git_commit_file(file)
+def git_commit_file(file, message=nil)
   if has_tool('git') and File.exist?('.git')
-    %x{git commit #{file} -m "Commit changes to #{file}" &> /dev/null}
+    message ||= "changes"
+    puts "Commiting changes:"
+    puts
+    diff = %x{git diff HEAD #{file}}
+    puts diff
+    %x{git commit #{file} -m "Commit #{message} in #{file}" &> /dev/null}
   end
 end
 
@@ -278,11 +283,6 @@ def ship_gem(file)
   %x{gem push #{file}}
 end
 
-def x(v)
-  puts %[#{v}]
-  print %x[#{v}]
-end
-
 def ask_yes_or_no
   return boolean_value(ENV['ANSWER_OVERRIDE']) unless ENV['ANSWER_OVERRIDE'].nil?
   answer = STDIN.gets.downcase.chomp
@@ -345,12 +345,16 @@ def git_bundle(treeish)
   "#{temp}/#{@name}-#{@version}-#{appendix}.tar.gz"
 end
 
-def remote_bootstrap(host, treeish)
+# We take a tar argument for cases where `tar` isn't best, e.g. Solaris
+def remote_bootstrap(host, treeish, tar_cmd=nil)
+  unless tar = tar_cmd
+    tar = 'tar'
+  end
   tarball = git_bundle(treeish)
   tarball_name = File.basename(tarball).gsub('.tar.gz','')
   rsync_to(tarball, host, '/tmp')
   appendix = rand_string
-  sh "ssh -t #{host} 'tar -zxvf /tmp/#{tarball_name}.tar.gz -C /tmp/ ; git clone --recursive /tmp/#{tarball_name} /tmp/#{@name}-#{appendix} ; cd /tmp/#{@name}-#{appendix} ; rake package:bootstrap'"
+  sh "ssh -t #{host} '#{tar} -zxvf /tmp/#{tarball_name}.tar.gz -C /tmp/ ; git clone --recursive /tmp/#{tarball_name} /tmp/#{@name}-#{appendix} ; cd /tmp/#{@name}-#{appendix} ; rake package:bootstrap'"
   "/tmp/#{@name}-#{appendix}"
 end
 
@@ -380,3 +384,4 @@ end
 def empty_dir?(dir)
   File.exist?(dir) and File.directory?(dir) and Dir["#{dir}/**/*"].empty?
 end
+

data/ext/packaging/tasks/10_setupvars.rake

@@ -53,6 +53,7 @@ begin
   @ips_repo        = @pkg_defaults['ips_repo']
   @ips_store       = @pkg_defaults['ips_store']
   @ips_host        = @pkg_defaults['ips_host']
+  @ips_inter_cert  = @pkg_defaults['ips_inter_cert']
 rescue => e
   STDERR.puts "There was an error loading the packaging defaults from the 'ext/build_defaults.yaml' file.\n" + e
   exit 1

data/ext/packaging/tasks/20_setupextravars.rake

@@ -1,21 +1,34 @@
+# The pl:load_extras tasks is intended to load variables
+# from the extra yaml file downloaded by the pl:fetch task.
+# The goal is to be able to augment/override settings in the
+# source project's build_data.yaml and project_data.yaml with
+# Puppet Labs-specific data, rather than having to clutter the
+# generic tasks with data not generally useful outside the
+# PL Release team
 namespace :pl do
   task :load_extras do
     begin
       @build_data = YAML.load_file("#{ENV['HOME']}/.packaging/#{@builder_data_file}")
-      @rpm_build_host   = @build_data['rpm_build_host']  if @build_data['rpm_build_host']
-      @deb_build_host   = @build_data['deb_build_host']  if @build_data['deb_build_host']
-      @osx_build_host   = @build_data['osx_build_host']  if @build_data['osx_build_host']
-      @tarball_path     = @build_data['tarball_path']    if @build_data['tarball_path']
-      @dmg_path         = @build_data['dmg_path']        if @build_data['dmg_path']
-      @pe_version       = @build_data['pe_version']      if @build_data['pe_version']
-      @team             = @build_data['team']            if @build_data['team']
-      @yum_repo_path    = @build_data['yum_repo_path']   if @build_data['yum_repo_path']
-      @apt_repo_path    = @build_data['apt_repo_path']   if @build_data['apt_repo_path']
-      @freight_conf     = @build_data['freight_conf']    if @build_data['freight_conf']
-      @sles_build_host  = @build_data['sles_build_host'] if @build_data['sles_build_host']
-      @sles_arch_repos  = @build_data['sles_arch_repos'] if @build_data['sles_arch_repos']
-      @sles_repo_path   = @build_data['sles_repo_path']  if @build_data['sles_repo_path']
-      @sles_repo_host   = @build_data['sles_repo_host']  if @build_data['sles_repo_host']
+      @rpm_build_host   = @build_data['rpm_build_host']   if @build_data['rpm_build_host']
+      @deb_build_host   = @build_data['deb_build_host']   if @build_data['deb_build_host']
+      @osx_build_host   = @build_data['osx_build_host']   if @build_data['osx_build_host']
+      @ips_build_host   = @build_data['ips_build_host']   if @build_data['ips_build_host']
+      @tarball_path     = @build_data['tarball_path']     if @build_data['tarball_path']
+      @dmg_path         = @build_data['dmg_path']         if @build_data['dmg_path']
+      @pe_version       = @build_data['pe_version']       if @build_data['pe_version']
+      @team             = @build_data['team']             if @build_data['team']
+      @yum_repo_path    = @build_data['yum_repo_path']    if @build_data['yum_repo_path']
+      @apt_repo_path    = @build_data['apt_repo_path']    if @build_data['apt_repo_path']
+      @freight_conf     = @build_data['freight_conf']     if @build_data['freight_conf']
+      @sles_build_host  = @build_data['sles_build_host']  if @build_data['sles_build_host']
+      @sles_arch_repos  = @build_data['sles_arch_repos']  if @build_data['sles_arch_repos']
+      @sles_repo_path   = @build_data['sles_repo_path']   if @build_data['sles_repo_path']
+      @sles_repo_host   = @build_data['sles_repo_host']   if @build_data['sles_repo_host']
+      @ips_path         = @build_data['ips_path']         if @build_data['ips_path']
+      @ips_package_host = @build_data['ips_package_host'] if @build_data['ips_package_host']
+      @certificate_pem  = @build_data['certificate_pem']  if @build_data['certificate_pem']
+      @privatekey_pem   = @build_data['privatekey_pem']   if @build_data['privatekey_pem']
+      @ips_inter_cert   = @build_data['ips_inter_cert']   if @build_data['ips_inter_cert']
     rescue => e
       STDERR.puts "There was an error loading the builder data from #{ENV['HOME']}/.packaging/#{@builder_data_file}. Try rake pl:fetch to download the current extras builder data.\n" + e
       exit 1

data/ext/packaging/tasks/fetch.rake

@@ -4,6 +4,12 @@ else
   pl_packaging_url = "https://raw.github.com/puppetlabs/build-data/#{@name}"
 end
 
+# The pl:fetch task pulls down a file from the build-data repo that contains additional
+# data specific to Puppet Labs release infrastructure intended to augment/override any
+# defaults specified in the source project repo, e.g. in ext/build_defaults.yaml
+#
+# It uses curl to download the file, and places it in a hidden directory in the home
+# directory, e.g. ~/.packaging/@builder_data_file
 namespace :pl do
   task :fetch do
     rm_rf "#{ENV['HOME']}/.packaging"

data/ext/packaging/tasks/ips.rake

@@ -2,24 +2,27 @@ if @build_ips
   require 'erb'
   namespace :package do
     namespace :ips do
-      workdir = "pkg/ips"
+      workdir = "pkg/ips/workdir"
       proto = workdir + '/proto'
       repo = workdir + '/repo'
-      pkgs = workdir + '/pkgs'
-      repouri = "file://#{`pwd`.strip + '/' + repo}"
-      artifact = "#{pkgs}/#{@name}@#{@ipsversion}.p5p"
+      pkgs = 'pkg/ips/pkgs'
+      repouri = 'file://' + Dir.pwd + '/' + repo
+      artifact = pkgs + "/#{@name}@#{@ipsversion}.p5p"
 
       # Create a source repo
       # We dont clean the base pkg directory only ips work dir.
       task :clean do
-        %x[rm -rf #{workdir}]
+        rm_rf workdir
+      end
+
+      task :clean_pkgs do
+        rm_rf pkgs
       end
 
       # Create an installation image at ips/proto
-      task :prepare => :clean do
-        check_tool('pkgsend')
-        x %[mkdir -p #{workdir}]
-        x %[gmake -f ext/ips/rules DESTDIR=#{proto} 2>#{workdir}/build.out ]
+      task :prepare do
+        mkdir_pr workdir, pkgs
+        sh "gmake -f ext/ips/rules DESTDIR=#{proto} 2>#{workdir}/build.out"
       end
 
       # Process templates and write the initial manifest
@@ -29,52 +32,92 @@ if @build_ips
 
       # Update manifest to include the installation image information.
       task :protogen => :prototmpl do
-        x %[pkgsend generate #{proto} >> #{workdir}/#{@name}.p5m.x ]
-        os=%x[uname -p].chomp
+        sh "pkgsend generate #{proto} >> #{workdir}/#{@name}.p5m.x"
       end
 
       # Generate and resolve dependency list
       task :protodeps => :protogen do
-        x %[pkgdepend generate -d #{proto} #{workdir}/#{@name}.p5m.x > #{workdir}/#{@name}.depends ]
-        x %[pkgdepend resolve -m #{workdir}/#{@name}.depends ]
-        x %[cat #{workdir}/#{@name}.depends.res >> #{workdir}/#{@name}.p5m.x]
+        sh "pkgdepend generate -d #{proto} #{workdir}/#{@name}.p5m.x > #{workdir}/#{@name}.depends"
+        sh "pkgdepend resolve -m #{workdir}/#{@name}.depends"
+        sh "cat #{workdir}/#{@name}.depends.res >> #{workdir}/#{@name}.p5m.x"
       end
 
       # Mogrify manifest to remove unncecessary info, and other kinds of transforms.
       task :protomogrify => :protodeps do
-        x %[pkgmogrify ./ext/ips/transforms ./#{workdir}/#{@name}.p5m.x| pkgfmt >> #{workdir}/#{@name}.p5m ]
+        sh "pkgmogrify ./ext/ips/transforms ./#{workdir}/#{@name}.p5m.x| pkgfmt >> #{workdir}/#{@name}.p5m"
       end
 
       # Generate and resolve dependency list
       task :license => :protomogrify do
-        x %[cp LICENSE #{proto}/#{@name}.license]
+        cp 'LICENSE', "#{proto}/#{@name}.license"
       end
 
       # Ensure that our manifest is sane.
       task :lint => :license do
-        x %[pkglint #{workdir}/#{@name}.p5m]
+        print %x{pkglint #{workdir}/#{@name}.p5m}
+      end
+
+      task :package => [:clean_pkgs, :clean, :prepare, :lint] do
+        # the package is actually created via the dependency chain of :lint
+      end
+
+      # Create a local file-based IPS repository
+      task :createrepo do
+        check_tool('pkgrepo')
+        sh "pkgrepo create #{repo}"
+        sh "pkgrepo set -s #{repo} publisher/prefix=puppetlabs.com"
+      end
+
+      # Send a created package to the local IPS repository
+      task :send do
+        check_tool('pkgsend')
+        sh "pkgsend -s #{repouri} publish -d #{proto} --fmri-in-manifest #{workdir}/#{@name}.p5m"
+      end
+
+      # Retrieve the package from the remote repository in .p5p archive format
+      task :receive do
+        check_tool('pkgrecv')
+        sh "pkgrecv -s #{repouri} -a -d #{artifact} #{@name}@#{@ipsversion}"
       end
 
-      task :package => :lint do
-        x %[rm -rf #{pkgs}]
-        x %[mkdir -p #{pkgs}]
-        x %[pkgrepo create #{repo}]
-        x %[pkgrepo set -s #{repo} publisher/prefix=puppetlabs.com]
-        x %[pkgsend -s #{repouri} publish -d #{proto} --fmri-in-manifest #{workdir}/#{@name}.p5m]
-        x %[rm -f #{artifact}]
-        x %[pkgrecv -s #{repouri} -a -d #{artifact} #{@name}@#{@ipsversion}]
-       end
 
       task :dry_install do
-        x %[pkg install -nv -g #{artifact} #{@name}@#{@ipsversion}]
+        sh "pkg install -nv -g #{artifact} #{@name}@#{@ipsversion}"
       end
-    end
 
-    desc "Creates an ips version"
-    task :ips do
-      Rake::Task['package:ips:prepare'].invoke
-      Rake::Task['package:ips:package'].invoke
+      task :p5p, :sign_ips do |t, args|
+        # make sure our system dependencies are met
+        check_tool('pkg')
+        check_tool('pkgdepend')
+        check_tool('pkgsend')
+        check_tool('pkglint')
+        check_tool('pkgmogrify')
+        sign_ips = args.sign_ips
+        # create the package manifest & files (the "package")
+        Rake::Task['package:ips:package'].invoke
+        # create the local repository
+        Rake::Task['package:ips:createrepo'].invoke
+        # publish the package to the repository
+        Rake::Task['package:ips:send'].invoke
+        # signing the package occurs remotely in the repository
+        Rake::Task['pl:sign_ips'].invoke(repouri,"#{@name}@#{@ipsversion}") if sign_ips
+        # retrieve the signed package in a .p5p archive file format
+        Rake::Task['package:ips:receive'].invoke
+        # clean up the workdir area
+        Rake::Task['package:ips:clean'].execute
+        STDOUT.puts "Created #{Dir['pkg/ips/pkgs/*']}"
+      end
     end
 
+    desc "Creates an ips p5p archive package from this repository"
+    task :ips => ['package:ips:p5p']
+  end
+
+  namespace :pl do
+    desc "Create and sign a p5p archive package from this repository"
+    task :ips => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task['package:ips:p5p'].reenable
+      Rake::Task['package:ips:p5p'].invoke(TRUE)
+    end
   end
 end

data/ext/packaging/tasks/mock.rake

@@ -1,3 +1,22 @@
+# The mock methods/tasks are fairly specific to puppetlabs infrastructure, e.g., the mock configs
+# have to be named in a format like the PL mocks, e.g. dist-version-architecture, such as:
+# el-5-i386
+# fedora-17-i386
+# as well as the oddly formatted exception, 'pl-5-i386' which is the default puppetlabs FOSS mock
+# format for 'el-5-i386' (note swap 'pl' for 'el')
+#
+# The mock-built rpms are placed in a directory structure under 'pkg' based on how the Puppet Labs
+# repo structure is laid out in order to facilitate easy shipping from the local repository to the
+# Puppet Labs repos. For open source, the directory structure mirrors that of yum.puppetlabs.com:
+# pkg/<dist>/<version>/{products,devel,dependencies}/<arch>/*.rpm
+# e.g.,
+# pkg/el/5/products/i386/*.rpm
+# pkg/fedora/f15/products/i386/*.rpm
+#
+# For PE, the directory structure is flatter:
+# pkg/<dist>-<version>-<arch>/*.rpm
+# e.g.,
+# pkg/el-5-i386/*.rpm
 
 def mock(mock_config, srpm)
   check_tool('mock')

data/ext/packaging/tasks/pe_deb.rake

@@ -1,3 +1,8 @@
+# For PE, the natural default tasks are the remote tasks, rather than
+# the local ones, in reflection of which will be most ideal for PE devs.
+# e.g., pe:local_deb is the task to build a deb on the local host,
+# while pe:deb is the task for building on the remote builder host
+
 if @build_pe
   namespace :pe do
     desc "Create a PE deb from this repo using the default cow #{@default_cow}."

data/ext/packaging/tasks/pe_remote.rake

@@ -1,3 +1,8 @@
+# For PE, the natural default tasks are the remote tasks, rather than
+# the local ones, in reflection of which will be most ideal for PE devs.
+# e.g., pe:local_deb is the task to build a deb on the local host,
+# while pe:deb is the task for building on the remote builder host
+
 if @build_pe
   namespace :pe do
     desc "Execute remote debian build using default cow on builder and retrieve package"

data/ext/packaging/tasks/pe_ship.rake

@@ -2,8 +2,12 @@ if @build_pe
   namespace :pe do
     desc "ship PE rpms to #{@yum_host}"
     task :ship_rpms => ["pl:load_extras"] do
-      rsync_to('pkg/pe/rpm/', @yum_host, "#{@yum_repo_path}/#{@pe_version}/repos/")
-      Rake::Task["pe:remote_update_yum_repo"].invoke
+      if empty_dir?("pkg/pe/rpm")
+        STDERR.puts "The 'pkg/pe/rpm' directory has no packages. Did you run rake pe:deb?"
+      else
+        rsync_to('pkg/pe/rpm/', @yum_host, "#{@yum_repo_path}/#{@pe_version}/repos/")
+        Rake::Task["pe:remote_update_yum_repo"].invoke
+      end
     end
 
     desc "Update remote rpm repodata for PE on #{@yum_host}"
@@ -14,14 +18,17 @@ if @build_pe
     desc "Ship PE debs to #{@apt_host}"
     task :ship_debs => "pl:load_extras" do
       dist = @default_cow.split('-')[1]
-      if Dir["pkg/pe/deb/#{dist}/*"].empty?
-        STDERR.puts "The pkg/pe/deb/#{dist} directory has no packages. Did you run rake pe:deb?"
+      if empty_dir?("pkg/pe/deb/#{dist}")
+        STDERR.puts "The 'pkg/pe/deb/#{dist}' directory has no packages. Did you run rake pe:deb?"
       else
         rsync_to("pkg/pe/deb/#{dist}/", @apt_host, "#{@apt_repo_path}/#{@pe_version}/repos/incoming/unified/")
         Rake::Task["pe:remote_freight"].invoke
       end
     end
 
+    # This is particularly hacky. The 'pe-the-things' script resides on the @apt_host and takes packages placed
+    # in the directory/structure shown in the rsync target of pe:ship_debs and adds them to the remote PE
+    # freight repository and updates the apt repo metadata
     desc "remote freight PE packages to #{@apt_host}"
     task :remote_freight => "pl:load_extras" do
       remote_ssh_cmd(@apt_host, "sudo pe-the-things #{@pe_version} #{@apt_repo_path} #{@freight_conf}")

data/ext/packaging/tasks/release.rake

@@ -66,8 +66,12 @@ namespace :pl do
         Rake::Task["pl:ship_dmg"].execute
       end
     end if @build_dmg
+
+    desc "Release ips, e.g. pl:ips, pl:ship_ips"
+    task :release_ips => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task['pl:ips'].invoke
+      Rake::Task['pl:ship_ips'].invoke
+    end
   end
 end
 
-
-

data/ext/packaging/tasks/remote_build.rake

@@ -1,11 +1,13 @@
 # Tasks for remote building on builder hosts
+
 if File.exist?("#{ENV['HOME']}/.packaging/#{@builder_data_file}")
   namespace 'pl' do
-    task :remote_build, :host, :treeish, :task do |t, args|
+    task :remote_build, :host, :treeish, :task, :tar do |t, args|
       host                    = args.host
       treeish                 = args.treeish
       task                    = args.task
-      remote_repo             = remote_bootstrap(host, treeish)
+      tar                     = args.tar
+      remote_repo             = remote_bootstrap(host, treeish, tar)
       STDOUT.puts "Beginning package build on #{host}"
       remote_ssh_cmd(host, "cd #{remote_repo} ; rake #{task} ANSWER_OVERRIDE=no")
       rsync_from("#{remote_repo}/pkg/", host, 'pkg/')
@@ -16,27 +18,57 @@ if File.exist?("#{ENV['HOME']}/.packaging/#{@builder_data_file}")
     desc "Execute release_deb_rc full build set on remote debian build host"
     task :remote_deb_rc => ['pl:fetch', 'pl:load_extras'] do
       Rake::Task["pl:remote_build"].reenable
-      Rake::Task["pl:remote_build"].invoke(@deb_build_host, 'HEAD', "pl:release_deb_rc")
+      Rake::Task["pl:remote_build"].invoke(@deb_build_host, 'HEAD', "pl:release_deb_rc COW='#{@cows}'")
+    end
+
+    desc "Execute deb_all_rc build on remote debian build host (no signing)"
+    task :remote_deb_rc_build => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task["pl:remote_build"].reenable
+      Rake::Task["pl:remote_build"].invoke(@deb_build_host, 'HEAD', "pl:deb_all_rc COW='#{@cows}'")
     end
 
     desc "Execute release_deb_final full build set on remote debian build host"
     task :remote_deb_final => ['pl:fetch', 'pl:load_extras'] do
       Rake::Task["pl:remote_build"].reenable
-      Rake::Task["pl:remote_build"].invoke(@deb_build_host, 'HEAD', "pl:release_deb_final")
+      Rake::Task["pl:remote_build"].invoke(@deb_build_host, 'HEAD', "pl:release_deb_final COW='#{@cows}'")
+    end
+
+    desc "Execute deb_all on remote debian build host (no signing)"
+    task :remote_deb_final_build => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task["pl:remote_build"].reenable
+      Rake::Task["pl:remote_build"].invoke(@deb_build_host, 'HEAD', "pl:deb_all COW='#{@cows}'")
     end
 
     desc "Execute release_rpm_rc full build set on remote rpm build host"
     task :remote_rpm_rc => ['pl:fetch', 'pl:load_extras'] do
       Rake::Task["pl:remote_build"].reenable
-      Rake::Task["pl:remote_build"].invoke(@rpm_build_host, 'HEAD', "pl:release_rpm_rc")
+      Rake::Task["pl:remote_build"].invoke(@rpm_build_host, 'HEAD', "pl:release_rpm_rc MOCK='#{@rc_mocks}'")
     end
 
-    desc "Execute release_deb_final full build set on remote rpm build host"
+    desc "Execute mock_rc on remote rpm build host (no signing)"
+    task :remote_rpm_rc_build => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task["pl:remote_build"].reenable
+      Rake::Task["pl:remote_build"].invoke(@rpm_build_host, 'HEAD', "pl:mock_rc MOCK='#{@rc_mocks}'")
+    end
+
+    desc "Execute release_rpm_final full build set on remote rpm build host"
     task :remote_rpm_final => ['pl:fetch', 'pl:load_extras'] do
       Rake::Task["pl:remote_build"].reenable
-      Rake::Task["pl:remote_build"].invoke(@rpm_build_host, 'HEAD', "pl:release_rpm_final")
+      Rake::Task["pl:remote_build"].invoke(@rpm_build_host, 'HEAD', "pl:release_rpm_final MOCK='#{@final_mocks}'")
+    end
+
+    desc "Execute mock_final on remote rpm build host (no signing)"
+    task :remote_mock_final => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task["pl:remote_build"].reenable
+      Rake::Task["pl:remote_build"].invoke(@rpm_build_host, 'HEAD', "pl:mock_final MOCK='#{@final_mocks}'")
     end
 
+    desc "Execute pl:ips on remote ips build host"
+    task :remote_ips => ['pl:fetch', 'pl:load_extras'] do
+      Rake::Task["pl:remote_build"].reenable
+      Rake::Task["pl:remote_build"].invoke(@ips_build_host, 'HEAD', 'pl:ips', 'gtar')
+    end if @build_ips
+
     desc "Execute package:apple on remote apple build host"
     task :remote_dmg => ['pl:fetch', 'pl:load_extras'] do
       # Because we use rvmsudo for apple, we end up replicating the :remote_build task
@@ -61,6 +93,7 @@ if File.exist?("#{ENV['HOME']}/.packaging/#{@builder_data_file}")
       Rake::Task["pl:sign_tar"].invoke
       Rake::Task["pl:uber_ship"].execute
       Rake::Task["pl:remote_freight_devel"].invoke
+      Rake::Task["pl:remote_update_yum_repo"].invoke
     end
 
     desc "UBER FINAL build: build and ship FINAL tar, gem (as applicable), remote dmg, remote deb, remote rpm"
@@ -73,6 +106,7 @@ if File.exist?("#{ENV['HOME']}/.packaging/#{@builder_data_file}")
       Rake::Task["pl:sign_tar"].invoke
       Rake::Task["pl:uber_ship"].execute
       Rake::Task["pl:remote_freight_final"].invoke
+      Rake::Task["pl:remote_update_yum_repo"].invoke
     end
   end
 end

data/ext/packaging/tasks/rpm.rake

@@ -7,7 +7,10 @@ def build_rpm(buildarg = "-bs")
      --define "_binary_payload w9.gzdio" --define "_source_payload w9.gzdio" \
      --define "_default_patch_fuzz 2"'
   args = rpm_define + ' ' + rpm_old_version
-  mkdir_pr temp, 'pkg/rpm', 'pkg/srpm', "#{temp}/SOURCES", "#{temp}/SPECS"
+  mkdir_pr temp, 'pkg/srpm', "#{temp}/SOURCES", "#{temp}/SPECS"
+  if buildarg == '-ba'
+    mkdir_p 'pkg/rpm'
+  end
   if @sign_tar
     Rake::Task["pl:sign_tar"].invoke
     cp_p "pkg/#{@name}-#{@version}.tar.gz.asc", "#{temp}/SOURCES"
@@ -16,7 +19,9 @@ def build_rpm(buildarg = "-bs")
   erb "ext/redhat/#{@name}.spec.erb", "#{temp}/SPECS/#{@name}.spec"
   sh "rpmbuild #{args} #{buildarg} --nodeps #{temp}/SPECS/#{@name}.spec"
   mv FileList["#{temp}/SRPMS/*.rpm"], "pkg/srpm"
-  mv FileList["#{temp}/RPMS/*/*.rpm"], "pkg/rpm"
+  if buildarg == '-ba'
+    mv FileList["#{temp}/RPMS/*/*.rpm"], "pkg/rpm"
+  end
   rm_rf temp
   puts
   output = FileList['pkg/*/*.rpm']

data/ext/packaging/tasks/ship.rake

@@ -18,6 +18,8 @@ namespace :pl do
     rsync_to('pkg/deb/', @apt_host, @apt_repo_path)
   end
 
+  # These hacky bits execute a pre-existing rake task on the @apt_host that adds the debs
+  # shipped with the ship task to the apt repo and updates the repo metadata
   desc "freight RCs to devel repos on #{@apt_host}"
   task :remote_freight_devel do
     STDOUT.puts "Really run remote freight RC command on #{@apt_host}? [y,n]"
@@ -29,6 +31,8 @@ namespace :pl do
     end
   end
 
+  # These similar hacky bits execute the same pre-existing rake task on the @apt_host, but
+  # with a different argument
   desc "remote freight final packages to PRODUCTION repos on #{@apt_host}"
   task :remote_freight_final do
     STDOUT.puts "Really run remote freight final command on #{@apt_host}? [y,n]"
@@ -40,12 +44,23 @@ namespace :pl do
 
   desc "Update remote ips repository on #{@ips_host}"
   task :update_ips_repo do
-    rsync_to('pkg/ips/pkgs', @ips_host, @ips_store)
+    rsync_to('pkg/ips/pkgs/', @ips_host, @ips_store)
     remote_ssh_cmd(@ips_host, "pkgrecv -s #{@ips_store}/pkgs/#{@name}@#{@ipsversion}.p5p -d #{@ips_repo} \\*")
     remote_ssh_cmd(@ips_host, "pkgrepo refresh -s #{@ips_repo}")
     remote_ssh_cmd(@ips_host, "/usr/sbin/svcadm restart svc:/application/pkg/server")
   end if @build_ips
 
+  if File.exist?("#{ENV['HOME']}/.packaging/#{@builder_data_file}")
+    desc "Upload ips p5p packages to downloads"
+    task :ship_ips => [ 'pl:fetch', 'pl:load_extras' ] do
+      if Dir['pkg/ips/pkgs/**/*'].empty?
+        STDOUT.puts "There aren't any p5p packages in pkg/ips/pkgs. Maybe something went wrong?"
+      else
+        rsync_to('pkg/ips/pkgs/', @ips_package_host, @ips_path)
+      end
+    end if @build_ips
+  end
+
   desc "Ship built gem to rubygems"
   task :ship_gem do
     ship_gem("pkg/#{@name}-#{@gemversion}.gem")

data/ext/packaging/tasks/sign.rake

@@ -16,10 +16,12 @@ def sign_deb_changes(file)
 end
 
 # requires atleast a self signed prvate key and certificate pair
-def sign_ips(pkg)
-  %x{pkgsign -s pkg/ips/repo/  -k #{@privatekey_pem} -c #{@certificate_pem} #{@name}@#{@ipsversion}}
-  %x{rm -f #{pkg}}
-  %x{pkgrecv -s pkg/ips/repo -a -d #{pkg} #{@name}@#{@ipsversion}}
+# fmri is the full IPS package name with version, e.g.
+# facter@facter@1.6.15,5.11-0:20121112T042120Z
+# technically this can be any ips-compliant package identifier, e.g. application/facter
+# repo_uri is the path to the repo currently containing the package
+def sign_ips(fmri, repo_uri)
+  %x{pkgsign -s #{repo_uri}  -k #{@privatekey_pem} -c #{@certificate_pem} -i #{@ips_inter_cert} #{fmri}}
 end
 
 namespace :pl do
@@ -49,10 +51,11 @@ namespace :pl do
   end
 
   desc "Sign ips package, Defaults to PL Key, pass KEY to override"
-  task :sign_ips do
-    ips_pkgs    = Dir["pkg/ips/pkgs/*.p5p"].join(' ')
+  task :sign_ips, :repo_uri, :fmri do |t, args|
+    repo_uri  = args.repo_uri
+    fmri      = args.fmri
     puts "Signing ips packages..."
-    sign_ips ips_pkgs
+    sign_ips(fmri, repo_uri)
   end if @build_ips
 
   desc "Check if all rpms are signed"

data/ext/packaging/tasks/version.rake

@@ -1,11 +1,26 @@
+# This is something of a work in progress. Unfortunately,
+# many of the projects that use the packaging repo carry
+# version files with hard-coded versions, and many of these
+# are in completely disparate formats.
+#
+# This task attempts to automate the updating of this file
+# with the version to be packaged, but given the many version
+# file formats in use, doing so cleanly is difficult. With
+# any luck, going forward some of these projects will move
+# away from maintaining hard-coded versions in source.
+# However, if this effort loses momentum, we may end up
+# revisiting this task and improving it substantially,
+# and/or standardizing the expected version file format.
 namespace :package do
   desc "Update the version in #{@version_file} to current and commit."
   task :versionbump  do
+    version = ENV['VERSION'] || @version.to_s.strip
     old_version =  get_version_file_version
     contents = IO.read(@version_file)
-    new_version = '"' + @version.to_s.strip + '"'
+    new_version = '"' + version + '"'
+    puts "Updating #{old_version} to #{new_version} in #{@version_file}"
     if contents.match("@DEVELOPMENT_VERSION@")
-      contents.gsub!("@DEVELOPMENT_VERSION@", @version.to_s.strip)
+      contents.gsub!("@DEVELOPMENT_VERSION@", version)
     elsif contents.match("VERSION = #{old_version}")
       contents.gsub!("VERSION = #{old_version}", "VERSION = #{new_version}")
     elsif contents.match("#{@name.upcase}VERSION = #{old_version}")
@@ -17,5 +32,12 @@ namespace :package do
     file.write contents
     file.close
   end
+
+  desc "Set and commit the version in #{@version_file}, requires VERSION."
+  task :versionset do
+    check_var('VERSION', ENV['VERSION'])
+    Rake::Task["package:versionbump"].invoke
+    git_commit_file(@version_file, "update to #{ENV['VERSION']}")
+  end
 end
 

data/lib/puppet/version.rb

@@ -6,7 +6,7 @@
 # Raketasks and such to set the version based on the output of `git describe`
 #
 module Puppet
-  PUPPETVERSION = '2.7.20-rc1'
+  PUPPETVERSION = '2.7.20'
 
   def self.version
     @puppet_version || PUPPETVERSION

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: puppet
 version: !ruby/object:Gem::Version
-  version: 2.7.20.rc1
-  prerelease: 7
+  version: 2.7.20
+  prerelease: 
 platform: ruby
 authors:
 - Puppet Labs
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-11-10 00:00:00.000000000 Z
+date: 2012-11-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: facter
@@ -1972,9 +1972,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>'
+  - - ! '>='
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubyforge_project: puppet
 rubygems_version: 1.8.24