puppet 2.7.22 → 2.7.23

data/CHANGELOG

@@ -1,3 +1,24 @@
+2.7.23
+===
+4295e3d (maint) Fix module_utils regex tests for module file perms
+a322f34 (Maint) Update test to reflect chown using the gid too
+28bda0f (Maint) Update expectations for older rspec
+be2d182 Improving testing around PMT module install permissions.
+7114934 Ensure that PMT uses the correct group membership.
+64d2637 (#21971) Allow paths that contain .. as part of a name
+d88dd3f (Maint) Confine module tool tests off of windows
+9e5ef53 Fixing broken requires in the acceptance tests.
+878c420 (#21971) Fixes PathPattern's usage of Dir.glob for Windows
+2455fac (#21971) Fix how import_all loads files
+7a13291 (#21971) Create system for safely dealing with path patterns
+3c8dc4a (#21971) Split import and autoloading code paths
+0a35ef4 (#21971) Check for possible directory traversal
+30a72ce (Maint) Clean up specs
+c3d1703 (Maint) Use dirname instead of regexes
+f02f186 Backporting acceptance tests.
+90d4180 (#14333) Ensure module permissions are sane.
+
+
 2.7.22
 ===
 ba8c021 [packaging] Update mocks for rpmbuilder mock format

data/Rakefile

@@ -9,8 +9,6 @@ $LOAD_PATH << File.join(File.dirname(__FILE__), 'tasks')
 begin
   require 'rubygems'
   require 'rubygems/package_task'
-  require 'rspec'
-  require 'rspec/core/rake_task'
 rescue LoadError
   # Users of older versions of Rake (0.8.7 for example) will not necessarily
   # have rubygems installed, or the newer rubygems package_task for that

data/ext/build_defaults.yaml

@@ -2,7 +2,7 @@
 packaging_url: 'git://github.com/puppetlabs/packaging.git --branch=master'
 packaging_repo: 'packaging'
 default_cow: 'base-squeeze-i386.cow'
-cows: 'base-lucid-i386.cow base-oneiric-i386.cow base-precise-i386.cow base-quantal-i386.cow base-sid-i386.cow base-squeeze-i386.cow base-stable-i386.cow base-testing-i386.cow base-unstable-i386.cow base-wheezy-i386.cow'
+cows: 'base-lucid-i386.cow base-precise-i386.cow base-quantal-i386.cow base-sid-i386.cow base-squeeze-i386.cow base-stable-i386.cow base-testing-i386.cow base-unstable-i386.cow base-wheezy-i386.cow'
 pbuild_conf: '/etc/pbuilderrc'
 packager: 'puppetlabs'
 gpg_name: 'info@puppetlabs.com'

data/ext/packaging/README.md

@@ -89,11 +89,7 @@ for developer use for building test packages; the `pe:deb` task creates a
   `ext/build_defaults.yaml` after `final_mocks:`.  For PE, the mocks are
   formatted as `pupent-<peversion>-<distversion>-<arch>`, e.g.
   `pupent-2.7-el5-i386`. To build for a specific target, set `MOCK=<mock>` to
-  the mock that matches the target. The `pe:deb` and `pe:mock` tasks work by
-  using the `:remote` tasks for building on a remote builder using the current
-  committed state of the source repository. To forego remote building and build
-  on the local station (e.g., by ssh-ing into a remote builder first), the
-  tasks `pe:local_mock` and `pe:local_deb` build using the local host.
+  the mock that matches the target.
 
 ## `:remote:` tasks
 There are also sub-namespaces of `:pl` and `:pe` that are
@@ -132,10 +128,12 @@ Jenkins tasks are similar to the `:remote:` tasks, but they do not require ssh
 access to the builders. They do require being on the local network - the
 jenkins instance that performs package builds is an internal server only,
 accessible when connected via VPN or on-site.  The jenkins tasks enable the
-packaging repo to kick off packaging builds on a remote jenkins slave. They
-work in a similar way to the :remote tasks, but with a few key differences. The
-jenkins tasks transmit information to a jenkins coordinator, which handles the
-rest. The data passed are the following:
+packaging repo to kick off packaging builds on a remote jenkins slave.
+There are two workflows of jenkins tasks in the packaging repo. The first
+workflow, which is used for creating individual platform packages on jenkins
+(e.g. creating a deb with pl:jenkins:deb) relies on a job that exists on a
+remote jenkins server. The tasks transmit information to the jenkins job, which
+handles the rest. The data passed are the following:
 
 1) $PROJECT\_BUNDLE - a tar.gz of a git-bundle from HEAD of the current
 project, which is cloned on the builder to set up a duplicate of this
@@ -208,10 +206,63 @@ the task are:
 
 #################
 ```
+The second, more recent, jenkins-based workflow is for initiating the
+"uber_build", or a package build for all of our target platforms. This workflow
+doesn't actually use a static job on the jenkins-server. Instead it _creates_
+the jenkins jobs for you, on-demand. Specifically, it creates two jenkins-jobs,
+and can create an optional third.
+
+The first job is a matrix job, the cells of which are individual package tasks
+for all of the build targets. This job takes three parameters:
+
+1) $PROJECT\_BUNDLE - a tar.gz of a git-bundle from HEAD of the current
+project, which is cloned on the builder to set up a duplicate of this
+environment
+
+2) $BUILD\_PROPERTIES - a build parameters file, containing all information
+about the build
+
+3) $PROJECT - the project we're building, e.g. facter, puppet. This is used
+later in determining the target for the build artifacts on the distribution
+server
+
+This first job clones the git bundle passed in as a parameter, then clones the
+packaging repo (rake package:bootstrap) and for every cell in its matrix
+performs a package build for a specific target (e.g. rake pl:deb
+COW=base-lucid-i386.cow). If all cells in the matrix complete successfully (if
+all packages build), this job automatically triggers the second of the new jobs
+as a downstream job.
+
+To receive an email notification from jenkins about the status of the packaging
+job, pass NOTIFY=<recipient> as an environment variable to the uber_build
+invocation, e.g.:
+
+rake pl:jenkins:uber_build NOTIFY="foo@puppetlabs.com bar@puppetlabs.com"
+
+The second job is an automatic repository creation task for this git repo.
+Specifically, the job copies the git bundle from the packaging job and clones
+it, and uses the git information in the git bundle to clone the packaging repo
+and invoke the repository creation jobs `pl:jenkins:rpm_repos` and
+`pl:jenkins:deb_repos`.
+
+The third job is only created _if_ the environment variable
+`DOWNSTREAM_JOB=<job_url>` was passed to the initial "pl:jenkins:uber_build"
+invocation. This third job takes the value assigned to `DOWNSTREAM_JOB` and
+creates a proxy jenkins job with a single build step, a curl call to this
+value, presumably a url to a jenkins job to trigger programmatically.
+Once the repos have successfully been created by the second job, the third job
+will automatically trigger as a downstream job.
+
+All 3 jobs are configured by default for removal by jenkins after 3 days, to
+avoid clutter.
+
+The goal is to move toward migrating all of the jenkins tasks from the first
+workflow, using a static job that is called many times per package, to this
+second workflow of creating the jobs on demand.
 
 ## Task Explanations
 For a listing of all available tasks and their functions, see the [Task
-Dictionary](https://github.com/MosesMendoza/packaging/tree/more_documentation#task-dictionary)
+Dictionary](https://github.com/puppetlabs/packaging/tree/more_documentation#task-dictionary)
 at the end of this README.
 
 ## Modules
@@ -301,15 +352,17 @@ sign_tar: FALSE
 # a space separated list of mock configs. These are the rpm distributions to package for. If a noarch package, only one arch of each is needed.
 final_mocks: 'pl-el-5-i386 pl-el-5-x86_64 pl-el-6-i386 pl-el-6-x86_64 pl-fedora-16-i386 pl-fedora-16-x86_64 pl-fedora-17-i386 pl-fedora-17-x86_64'
 # The host that contains the yum repository to ship to
-yum_host: 'burji.puppetlabs.com'
+yum_host: 'yum.puppetlabs.com'
 # The remote path the repository on the yum\_host
 yum_repo_path: '/some/repo/'
 # The host that contains the apt repository to ship to
-apt_host: 'burji.puppetlabs.com'
+apt_host: 'apt.puppetlabs.com'
 # The URL to use for the apt dependencies in cow building
 apt_repo_url: 'http://apt.puppetlabs.com'
 # The path on the remote apt host that debs should ship to
 apt_repo_path: '/opt/repository/incoming'
+# The host that stores the tarballs for downloading
+tar_host: 'downloads.puppetlabs.com'
 # Whether to present the gem and apple tasks
 build_gem: TRUE
 build_dmg: TRUE
@@ -678,10 +731,12 @@ files:
 
 * **pl:jenkins:uber_build**
 
-    An aggregate of build tasks. These include `jenkins:deb_all`,
-    `pl:jenkins:mock_all`, `pl:jenkins:tar`, `pl:jenkins:dmg`, and `pl:jenkins:gem`. Each
-    task is a separate job that is posted to the jenkins build server,
-    separated by a 5 second sleep.
+    Create a jenkins job on the fly that performs an aggregate of build tasks.
+    These include all the debian builds using `pl:deb COW=<cow>`, rpm builds
+    with `pl:mock MOCKS=<mock>`, `package:tar`, `package:apple`, and
+    `package:gem` if applicable. See
+    [jenkins-tasks](https://github.com/puppetlabs/packaging#jenkins-tasks)
+    above for more detail.
 
 * **pl:jenkins:uber_ship**
 
@@ -868,16 +923,16 @@ files:
 
 * **pl:uber_release**
 
-    A composite task that performs the following tasks:  
-    `package:gem` (if build_gem is "true" in build_defaults.yaml)  
-    `pl:remote:release_deb`  
-    `pl:remote:release_rpm`  
-    `pl:remote:dmg` (if build_dmg is "true" in build_defaults.yaml)  
-    `package:tar`  
-    `pl:sign_tar`  
-    `pl:uber_ship`  
-    `pl:remote:freight`  
-    `pl:remote:update_yum_repo`  
+    A composite task that performs the following tasks:
+    `package:gem` (if build_gem is "true" in build_defaults.yaml)
+    `pl:remote:release_deb`
+    `pl:remote:release_rpm`
+    `pl:remote:dmg` (if build_dmg is "true" in build_defaults.yaml)
+    `package:tar`
+    `pl:sign_tar`
+    `pl:uber_ship`
+    `pl:remote:freight`
+    `pl:remote:update_yum_repo`
     This is essentially a complete build from start to finish. Gem and tarball
     are generated locally, and other packages (deb, rpm, dmg) are all created
     remotely. Assumes ssh access and appropriate build tool access on all
@@ -885,13 +940,13 @@ files:
 
 * **pl:uber_ship**
 
-    A composite task that performs the following tasks:  
-    `pl:ship_gem`  
-    `pl:ship_rpms`  
-    `pl:ship_debs`  
-    `pl:ship_dmgs`  
-    `pl:ship_tar`  
-    `pl:jenkins:ship`  
+    A composite task that performs the following tasks:
+    `pl:ship_gem`
+    `pl:ship_rpms`
+    `pl:ship_debs`
+    `pl:ship_dmgs`
+    `pl:ship_tar`
+    `pl:jenkins:ship`
     This is essentially a "ship all the things" task, but it is important to
     note that it does not update either yum or apt repo metadata on these
     respective servers - this has to be done via `pl:remote:update_yum_repo`

data/ext/packaging/packaging.rake

@@ -0,0 +1,45 @@
+# Load packaging repo tasks
+
+# These are ordered
+
+PACKAGING_PATH = File.join(File.dirname(__FILE__), 'tasks')
+
+@using_loader = true
+
+[ '00_utils.rake',
+  '10_setupvars.rake',
+  '20_setupextravars.rake',
+  '30_metrics.rake',
+  'apple.rake',
+  'build.rake',
+  'clean.rake',
+  'deb.rake',
+  'deb_repos.rake',
+  'doc.rake',
+  'fetch.rake',
+  'gem.rake',
+  'ips.rake',
+  'jenkins.rake',
+  'jenkins_dynamic.rake',
+  'mock.rake',
+  'pe_deb.rake',
+  'pe_remote.rake',
+  'pe_rpm.rake',
+  'pe_ship.rake',
+  'pe_sign.rake',
+  'pe_tar.rake',
+  'release.rake',
+  'remote_build.rake',
+  'retrieve.rake',
+  'rpm.rake',
+  'rpm_repos.rake',
+  'ship.rake',
+  'sign.rake',
+  'tag.rake',
+  'tar.rake',
+  'template.rake',
+  'update.rake',
+  'vendor_gems.rake',
+  'version.rake',
+  'z_data_dump.rake'].each { |t| load File.join(PACKAGING_PATH, t)}
+

data/ext/packaging/spec/tasks/build_object_spec.rb

@@ -10,6 +10,7 @@ describe Build::BuildInstance do
                   :apt_repo_url,
                   :author,
                   :benchmark,
+                  :build_date,
                   :build_defaults,
                   :build_dmg,
                   :build_doc,
@@ -62,6 +63,7 @@ describe Build::BuildInstance do
                   :jenkins_repo_path,
                   :metrics,
                   :name,
+                  :notify,
                   :project,
                   :origversion,
                   :osx_build_host,

data/ext/packaging/tasks/00_utils.rake

@@ -2,8 +2,7 @@
 
 def check_tool(tool)
   return true if has_tool(tool)
-  STDERR.puts "#{tool} tool not found...exiting"
-  exit 1
+  fail "#{tool} tool not found...exiting"
 end
 
 def find_tool(tool)
@@ -16,31 +15,25 @@ end
 alias :has_tool :find_tool
 
 def check_file(file)
-  unless File.exist?(file)
-    STDERR.puts "#{file} file not found...exiting"
-    exit 2
-  end
+  File.exist?(file) or fail "#{file} file not found!"
 end
 
 def check_var(varname,var=nil)
-  if var.nil?
-    STDERR.puts "Requires #{varname} be set...exiting"
-    exit 3
-  end
+  var.nil? and fail "Requires #{varname} be set!"
 end
 
 def check_host(host)
-  unless host == %x{hostname}.chomp!
-    STDERR.puts "Requires host to be #{host}...exiting"
-    exit 5
-  end
+  host == %x{hostname}.chomp! or fail "Requires host to be #{host}!"
+end
+
+def erb_string(erbfile)
+  template  = File.read(erbfile)
+  message   = ERB.new(template, nil, "-")
+  message.result(binding)
 end
 
 def erb(erbfile,  outfile)
-  template         = File.read(erbfile)
-  message          = ERB.new(template, nil, "-")
-  message.filename = erbfile
-  output           = message.result(binding)
+  output           = erb_string(erbfile)
   File.open(outfile, 'w') { |f| f.write output }
   puts "Generated: #{outfile}"
 end
@@ -56,16 +49,13 @@ def cp_p(src, dest, options={})
 end
 
 def mv_f(src, dest, options={})
-  force = {:force => true}
+  mandatory = {:force => true}
   mv(src, dest, options.merge(mandatory))
 end
 
 def git_co(ref)
   %x{git reset --hard ; git checkout #{ref}}
-  unless $?.success?
-    STDERR.puts "Could not checkout #{ref} git branch to build package from...exiting"
-    exit 1
-  end
+  $?.success? or fail "Could not checkout #{ref} git branch to build package from...exiting"
 end
 
 def git_describe
@@ -92,7 +82,7 @@ def git_sha_or_tag
 end
 
 def get_temp
-  temp = `mktemp -d -t pkgXXXXXX`.strip
+  `mktemp -d -t pkgXXXXXX`.strip
 end
 
 def remote_ssh_cmd target, command
@@ -103,7 +93,7 @@ end
 
 def rsync_to *args
   check_tool('rsync')
-  flags = "-Havxl -O --no-perms --no-owner --no-group"
+  flags = "-rHlv -O --no-perms --no-owner --no-group"
   source  = args[0]
   target  = args[1]
   dest    = args[2]
@@ -113,7 +103,7 @@ end
 
 def rsync_from *args
   check_tool('rsync')
-  flags = "-Havxl -O --no-perms --no-owner --no-group"
+  flags = "-rHlv -O --no-perms --no-owner --no-group"
   source  = args[0]
   target  = args[1]
   dest    = args[2]
@@ -129,8 +119,13 @@ def scp_file_to(host,path,file)
   %x{scp #{@tempdir}/#{file} #{host}:#{path}}
 end
 
-def timestamp
-  Time.now.strftime("%Y-%m-%d %H:%M:%S")
+def timestamp(separator=nil)
+  if s = separator
+    format = "%Y#{s}%m#{s}%d#{s}%H#{s}%M#{s}%S"
+  else
+    format = "%Y-%m-%d %H:%M:%S"
+  end
+  Time.now.strftime(format)
 end
 
 # Return information about the current tree, using `git describe`, ready for
@@ -243,6 +238,14 @@ def source_dirty?
   git_describe_version.include?('dirty')
 end
 
+def fail_on_dirty_source
+  if source_dirty?
+    fail "
+The source tree is dirty, e.g. there are uncommited changes. Please
+commit/discard changes and try again."
+  end
+end
+
 def kill_keychain
   %x{keychain -k mine}
 end
@@ -259,8 +262,7 @@ def gpg_sign_file(file)
   if gpg
     sh "#{gpg} --armor --detach-sign -u #{@build.gpg_key} #{file}"
   else
-    STDERR.puts "No gpg available. Cannot sign #{file}. Exiting..."
-    exit 1
+    fail "No gpg available. Cannot sign #{file}."
   end
 end
 
@@ -273,14 +275,12 @@ end
 def set_cow_envs(cow)
   elements = cow.split('-')
   if elements.size != 3
-    STDERR.puts "Expecting a cow name split on hyphens, e.g. 'base-squeeze-i386'"
-    exit 1
+    fail "Expecting a cow name split on hyphens, e.g. 'base-squeeze-i386'"
   else
     dist = elements[1]
     arch = elements[2]
     if dist.nil? or arch.nil?
-      STDERR.puts "Couldn't get the arg and dist from cow name. Expecting something like 'base-dist-arch'"
-      exit 1
+      fail "Couldn't get the arg and dist from cow name. Expecting something like 'base-dist-arch'"
     end
     arch = arch.split('.')[0] if arch.include?('.')
   end
@@ -352,13 +352,8 @@ def boolean_value(var)
 end
 
 def git_tag(version)
-  begin
-    sh "git tag -s -u #{@build.gpg_key} -m '#{version}' #{version}"
-  rescue Exception => e
-    STDERR.puts e
-    STDERR.puts "Unable to tag repo at #{version}"
-    exit 1
-  end
+  sh "git tag -s -u #{@build.gpg_key} -m '#{version}' #{version}"
+  $?.success or fail "Unable to tag repo at #{version}"
 end
 
 def rand_string
@@ -428,7 +423,7 @@ end
 
 def hostname
   require 'socket'
-  host = Socket.gethostname
+  Socket.gethostname
 end
 
 # Loop a block up to the number of attempts given, exiting when we receive success
@@ -446,9 +441,9 @@ def retry_on_fail(args, &blk)
       end
     end
   else
-    raise "retry_on_fail requires and arg (:times => x) where x is an Integer/Fixnum, and a block to execute"
+    fail "retry_on_fail requires and arg (:times => x) where x is an Integer/Fixnum, and a block to execute"
   end
-  raise "Block failed maximum of #{args[:times]} tries. Exiting.." unless success
+  fail "Block failed maximum of #{args[:times]} tries. Exiting.." unless success
 end
 
 def deprecate(old_cmd, new_cmd=nil)
@@ -497,9 +492,8 @@ def data_from_yaml(file)
   begin
     input_data = YAML.load_file(file) || {}
   rescue => e
-    puts "There was an error loading data from #{file}."
-    puts e.backtrace.join("\n")
-    exit 1
+    STDERR.puts "There was an error loading data from #{file}."
+    fail e.backtrace.join("\n")
   end
   input_data
 end
@@ -517,11 +511,8 @@ end
 # This method takes two arguments
 # 1) String - the URL to post to
 # 2) Array  - Ordered array of name=VALUE curl form parameters
-def curl_form_data(uri, form_data=[])
-  unless curl = find_tool("curl")
-    warn "Couldn't find curl. Curl is required for posting jenkins to trigger a build. Please install curl and try again."
-    exit 1
-  end
+def curl_form_data(uri, form_data=[], options={})
+  curl = find_tool("curl") or fail "Couldn't find curl. Curl is required for posting jenkins to trigger a build. Please install curl and try again."
   #
   # Begin constructing the post string.
   # First, assemble the form_data arguments
@@ -531,10 +522,59 @@ def curl_form_data(uri, form_data=[])
     post_string << "#{param} "
   end
 
-  # Add the uri and we're off
+  # Add the uri
   post_string << "#{uri}"
-  sh "#{curl} #{post_string}"
+
+  # If this is quiet, we're going to silence all output
+  if options[:quiet]
+    post_string << " >/dev/null 2>&1"
+  end
+
+  %x{#{curl} #{post_string}}
   return $?.success?
 end
 
+def random_string length
+  rand(36**length).to_s(36)
+end
+
+# Use the curl to create a jenkins job from a valid XML
+# configuration file.
+# Returns the URL to the job
+def create_jenkins_job(name, xml_file)
+  create_url = "http://#{@build.jenkins_build_host}/createItem?name=#{name}"
+  form_args = ["-H", '"Content-Type: application/xml"', "--data-binary", "@#{xml_file}"]
+  curl_form_data(create_url, form_args)
+  "http://#{@build.jenkins_build_host}/job/#{name}"
+end
 
+# Use the curl to check of a named job is defined on the jenkins server.  We
+# curl the config file rather than just checking if the job exists by curling
+# the job url and passing --head because jenkins will mistakenly return 200 OK
+# if you issue multiple very fast requests just requesting the header.
+def jenkins_job_exists?(name)
+  job_url = "http://#{@build.jenkins_build_host}/job/#{name}/config.xml"
+  form_args = ["--silent", "--fail"]
+  curl_form_data(job_url, form_args, :quiet => true)
+end
+
+def require_library_or_fail(library)
+  begin
+    require library
+  rescue LoadError
+    fail "Could not load #{library}. #{library} is required by the packaging repo for this task"
+  end
+end
+
+# Use the provided URL string to print important information with
+# ASCII emphasis
+def print_url_info(url_string)
+puts "\n////////////////////////////////////////////////////////////////////////////////\n\n
+  Build submitted. To view your build progress, go to\n#{url_string}\n\n
+////////////////////////////////////////////////////////////////////////////////\n\n"
+end
+
+def escape_html(uri)
+  require 'cgi'
+  CGI.escapeHTML(uri)
+end