puppet 3.2.4 → 3.3.0.rc2

data/COMMITTERS.md

@@ -13,6 +13,17 @@ on?"  This is already called out in the [CONTRIBUTING.md](http://goo.gl/XRH2J).
 Therefore, it is the responsibility of the committer to re-base the change set
 on the appropriate branch which should receive the contribution.
 
+It is also the responsibility of the committer to review the change set in an
+effort to make sure the end users must opt-in to new behavior that is
+incompatible with previous behavior.  We employ the use of [feature
+flags](http://stackoverflow.com/questions/7707383/what-is-a-feature-flag) as
+the primary way to achieve this user opt-in behavior.  Finally, it is the
+responsibility of the committer to make sure the `master` and `stable` branches
+are both clean and working at all times.  Clean means that dead code is not
+allowed, everything needs to be usable in some manner at all points in time.
+Stable is not an indication of the build status, but rather an expression of
+our intent that the `stable` branch does not receive new functionality.
+
 The rest of this document addresses the concerns of the committer.  This
 document will help guide the committer decide which branch to base, or re-base
 a contribution on top of.  This document also describes our branch management
@@ -37,8 +48,21 @@ making the decision what base branch to merge the change set into.
 
 **base branch** - A branch in Git that contains an active history of changes
 and will eventually be released using semantic version guidelines.  The branch
-named master will always exist as a base branch.  All other base branches will
-be associated with a specific released version of Puppet, e.g. 2.7.x and 3.0.x.
+named `master` will always exist as a base branch.  The other base branches are
+`stable`, and `security` described below.
+
+**master branch** - The branch where new functionality that are not bug fixes
+is merged.
+
+**stable branch** - The branch where bug fixes against the latest release or
+release candidate are merged.
+
+**security** - Where critical security fixes are merged.  These change sets
+will then be merged into release branches independently from one another. (i.e.
+no merging up).  Please do not submit pull requests against the security branch
+and instead report all security related issues to security@puppetlabs.com as
+per our security policy published at
+[https://puppetlabs.com/security/](https://puppetlabs.com/security/).
 
 Committer Guide
 ====
@@ -53,32 +77,58 @@ This section provides a guide to help a committer decide the specific base
 branch that a change set should be merged into.
 
 The latest minor release of a major release is the only base branch that should
-be patched. Older minor releases in a major release do not get patched. Before
-the switch to [semantic versions](http://semver.org/) committers did not have
-to think about the difference between minor and major releases.  Committing to
-the latest minor release of a major release is a policy intended to limit the
-number of active base branches that must be managed.
+be patched.  These patches will be merged into `master` if they contain new
+functionality.  They will be merged into `stable` and `master` if they fix a
+critical bug.  Older minor releases in a major release do not get patched.
+
+Before the switch to [semantic versions](http://semver.org/) committers did not
+have to think about the difference between minor and major releases.
+Committing to the latest minor release of a major release is a policy intended
+to limit the number of active base branches that must be managed.
 
 Security patches are handled as a special case.  Security patches may be
-applied to earlier minor releases of a major release.
+applied to earlier minor releases of a major release, but the patches should
+first be merged into the `security` branch.  Security patches should be merged
+by Puppet Labs staff members.  Pull requests should not be submitted with the
+security branch as the base branch.  Please send all security related
+information or patches to security@puppetlabs.com as per our [Security
+Policy](https://puppetlabs.com/security/).
+
+The CI systems are configured to run against `master` and `stable`.  Over time,
+these branches will refer to different versions, but their name will remain
+fixed to avoid having to update CI jobs and tasks as new versions are released.
 
 How to commit a change set to multiple base branches
 ---
 
-A change set may apply to multiple releases.  In this situation the change set
-needs to be committed to multiple base branches.  This section provides a guide
-for how to merge patches across releases, e.g. 2.7 is patched, how should the
-changes be applied to 3.0?
-
-First, merge the change set into the lowest numbered base branch, e.g. 2.7.
-Next, merge the changed base branch up through all later base branches by using
-the `--no-ff --log` git merge options.  We commonly refer to this as our "merge
-up process" because we merge in once, then merge up multiple times.
-
-When a new minor release branch is created (e.g. 3.1.x) then the previous one
-is deleted (e.g. 3.0.x). Any security or urgent fixes that might have to be
-applied to the older code line is done by creating an ad-hoc branch from the
-tag of the last patch release of the old minor line.
+A change set may apply to multiple branches, for example a bug fix should be
+applied to the stable release and the development branch.  In this situation
+the change set needs to be committed to multiple base branches.  This section
+provides a guide for how to merge patches into these branches, e.g.
+`stable` is patched, how should the changes be applied to `master`?
+
+First, rebase the change set onto the `stable` branch.  Next, merge the change
+set into the `stable` branch using a merge commit.  Once merged into `stable`,
+merge the same change set into `master` without doing a rebase as to preserve
+the commit identifiers.  This merge strategy follows the [git
+flow](http://nvie.com/posts/a-successful-git-branching-model/) model.  Both of
+these change set merges should have a merge commit which makes it much easier
+to track a set of commits as a logical change set through the history of a
+branch.  Merge commits should be created using the `--no-ff --log` git merge
+options.
+
+Any merge conflicts should be resolved using the merge commit in order to
+preserve the commit identifiers for each individual change.  This ensures `git
+branch --contains` will accurately report all of the base branches which
+contain a specific patch.
+
+Using this strategy, the stable branch need not be reset.  Both `master` and
+`stable` have infinite lifetimes.  Patch versions, also known as bug fix
+releases, will be tagged and released directly from the `stable` branch.  Major
+and minor versions, also known as feature releases, will be tagged and released
+directly from the `master` branch.  Upon release of a new major or minor
+version all of the changes in the `master` branch will be merged into the
+`stable` branch.
 
 Code review checklist
 ---
@@ -100,7 +150,6 @@ branch:
    or modify it.)  HINT: `git diff master --check`
  * Does the change set conform to the contributing guide?
 
-
 Commit citizen guidelines:
 ---
 
@@ -117,7 +166,8 @@ paying attention to our automated build tools.
    backwards compatible change set into master, then the target version should
    be 3.2.0 in the issue tracker.)
  * Ensure the pull request is closed (Hint: amend your merge commit to contain
-   the string `closes: #123` where 123 is the pull request number.
+   the string `closes #123` where 123 is the pull request number and github
+   will automatically close the pull request when the branch is pushed.)
 
 Example Procedure
 ====
@@ -130,7 +180,7 @@ Suppose a contributor submits a pull request based on master.  The change set
 fixes a bug reported against Puppet 3.1.1 which is the most recently released
 version of Puppet.
 
-In this example the committer should rebase the change set onto the 3.1.x
+In this example the committer should rebase the change set onto the `stable`
 branch since this is a bug rather than new functionality.
 
 First, the committer pulls down the branch using the `hub` gem.  This tool
@@ -142,44 +192,53 @@ branch to track the remote branch.
     Switched to a new branch 'jeffmccune-fix_foo_error'
 
 At this point the topic branch is a descendant of master, but we want it to
-descend from 3.1.x.  The committer creates a new branch then re-bases the
-change set:
+descend from `stable`.  The committer rebases the change set onto `stable`.
 
-    $ git branch bug/3.1.x/fix_foo_error
-    $ git rebase --onto 3.1.x master bug/3.1.x/fix_foo_error
+    $ git branch bug/stable/fix_foo_error
+    $ git rebase --onto stable master bug/stable/fix_foo_error
     First, rewinding head to replay your work on top of it...
     Applying: (#23456) Fix FooError that always bites users in 3.1.1
 
 The `git rebase` command may be interpreted as, "First, check out the branch
-named `bug/3.1.x/fix_foo_error`, then take the changes that were previously
-based on `master` and re-base them onto `3.1.x`.
+named `bug/stable/fix_foo_error`, then take the changes that were previously
+based on `master` and re-base them onto `stable`.
 
-Now that we have a topic branch containing the change set based on the correct
+Now that we have a topic branch containing the change set based on the `stable`
 release branch, the committer merges in:
 
-    $ git checkout 3.1.x
-    Switched to branch '3.1.x'
-    $ git merge --no-ff --log bug/3.1.x/fix_foo_error
+    $ git checkout stable
+    Switched to branch 'stable'
+    $ git merge --no-ff --log bug/stable/fix_foo_error
     Merge made by the 'recursive' strategy.
      foo | 0
      1 file changed, 0 insertions(+), 0 deletions(-)
      create mode 100644 foo
 
-Once merged into the first base branch, the committer merges up:
+Once merged into the first base branch, the committer merges the `stable`
+branch into `master`, being careful to preserve the same commit identifiers.
 
     $ git checkout master
     Switched to branch 'master'
-    $ git merge --no-ff --log 3.1.x
+    $ git merge --no-ff --log stable
     Merge made by the 'recursive' strategy.
      foo | 0
      1 file changed, 0 insertions(+), 0 deletions(-)
      create mode 100644 foo
 
-Once the change set has been merged "in and up." the committer pushes.  (Note,
-the checklist should be complete at this point.)  Note that both the 3.1.x and
-master branches are being pushed at the same time.
+Once the change set has been merged into one base branch, the change set should
+not be modified in order to keep the history clean, avoid "double" commits, and
+preserve the usefulness of `git branch --contains`.  If there are any merge
+conflicts, they are to be resolved in the merge commit itself and not by
+re-writing (rebasing) the patches for one base branch, but not another.
+
+Once the change set has been merged into `stable` and into `master`, the
+committer pushes.  Please note, the checklist should be complete at this point.
+It's helpful to make sure your local branches are up to date to avoid one of
+the branches failing to fast forward while the other succeeds.  Both the
+`stable` and `master` branches are being pushed at the same time.
 
-    $ git push puppetlabs master:master 3.1.x:3.1.x
+    $ git push puppetlabs master:master stable:stable
 
 That's it!  The committer then updates the pull request, updates the issue in
-our issue tracker, and keeps an eye on the build status.
+our issue tracker, and keeps an eye on the [build
+status](http://jenkins.puppetlabs.com).

data/Gemfile

@@ -14,7 +14,7 @@ end
 platforms :ruby do
   gem 'pry', :group => :development
   gem 'yard', :group => :development
-  gem 'redcarpet', :group => :development
+  gem 'redcarpet', '~> 2.0', :group => :development
   gem "racc", "~> 1.4", :group => :development
 end
 
@@ -22,11 +22,22 @@ gem "puppet", :path => File.dirname(__FILE__), :require => false
 gem "facter", *location_for(ENV['FACTER_LOCATION'] || '~> 1.6')
 gem "hiera", *location_for(ENV['HIERA_LOCATION'] || '~> 1.0')
 gem "rake", :require => false
-gem "rspec", "~> 2.11.0", :require => false
-gem "mocha", "~> 0.10.5", :require => false
 gem "rgen", "0.6.5", :require => false
 
-gem "yarjuf", "~> 1.0"
+
+group(:development, :test) do
+
+  # Jenkins workers may be using RSpec 2.9, so RSpec 2.11 syntax
+  # (like `expect(value).to eq matcher`) should be avoided.
+  gem "rspec", "~> 2.11.0", :require => false
+
+  # Mocha is not compatible across minor version changes; because of this only
+  # versions matching ~> 0.10.5 are supported. All other versions are unsupported
+  # and can be expected to fail.
+  gem "mocha", "~> 0.10.5", :require => false
+
+  gem "yarjuf", "~> 1.0"
+end
 
 group(:extra) do
   gem "rack", "~> 1.4", :require => false

data/README.md

@@ -33,7 +33,7 @@ Generally, you need the following things installed:
 
 Contributions
 ------
-Please see our [Contibution
+Please see our [Contribution
 Documents](https://github.com/puppetlabs/puppet/blob/master/CONTRIBUTING.md)
 and our [Developer
 Documentation](https://github.com/puppetlabs/puppet/blob/master/README_DEVELOPER.md).
@@ -51,3 +51,7 @@ site](http://projects.puppetlabs.com). A [mailing
 list](https://groups.google.com/forum/?fromgroups#!forum/puppet-users) is
 available for asking questions and getting help from others. In addition there
 is an active #puppet channel on Freenode.
+
+HTTP API
+--------
+{file:api_docs/http_api_index.md HTTP API Index}

data/README_DEVELOPER.md

@@ -13,7 +13,7 @@ this difference while working on the static compiler and types and providers.
 
 The two different types of catalog becomes relevant when writing spec tests
 because we frequently need to wire up a fake catalog so that we can exercise
-types, providers, or terminii that filter the catalog.
+types, providers, or termini that filter the catalog.
 
 The two different types of catalogs are so-called "resource" catalogs and "RAL"
 (resource abstraction layer) catalogs.  At a high level, the resource catalog
@@ -130,67 +130,35 @@ method.
 
 # Ruby Dependencies #
 
-Puppet is considered an Application as it relates to the recommendation of
-adding a Gemfile.lock file to the repository and the information published at
-[Clarifying the Roles of the .gemspec and
-Gemfile](http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/)
+To install the dependencies run:
 
-To install the dependencies run: `bundle install` to install the dependencies.
+    $ bundle install --path .bundle/gems/
 
-A checkout of the source repository should be used in a way that provides
-puppet as a gem rather than a simple Ruby library.  The parent directory should
-be set along the `GEM_PATH`, preferably before other tools such as RVM that
-manage gemsets using `GEM_PATH`.
+Once this is done, you can interact with puppet through bundler using `bundle
+exec <command>` which will ensure that `<command>` is executed in the context
+of puppet's dependencies.
 
-For example, Puppet checked out into `/workspace/src/puppet` using `git
-checkout https://github.com/puppetlabs/puppet` in `/workspace/src` can be used
-with the following actions.  The trick is to symlink `gems` to `src`.
+For example to run the specs:
 
-    $ cd /workspace
-    $ ln -s src gems
-    $ mkdir specifications
-    $ pushd specifications; ln -s ../gems/puppet/puppet.gemspec; ln -s ../gems/puppet/lib; popd
-    $ export GEM_PATH="/workspace:${GEM_PATH}"
-    $ gem list puppet
+    $ bundle exec rake spec
 
-This should list out
+To run puppet itself (for a resource lookup say):
 
-    puppet (2.7.19)
+    $ bundle exec puppet resource host localhost
 
-The final directory structure should look like this:
+which should return something like:
 
-    /workspace/src --- git working directory
-              /gems -> src
-              /specifications/puppet.gemspec -> ../gems/puppet/puppet.gemspec
-                             /lib -> ../gems/puppet/lib
-
-## Bundler ##
-
-With a source checkout of Puppet properly setup as a gem, dependencies can be
-installed using [Bundler](http://gembundler.com/)
-
-    $ bundle install
-    Fetching gem metadata from http://rubygems.org/........
-    Using diff-lcs (1.1.3)
-    Installing facter (1.6.11)
-    Using metaclass (0.0.1)
-    Using mocha (0.10.5)
-    Using puppet (2.7.19) from source at /workspace/puppet-2.7.x/src/puppet
-    Using rack (1.4.1)
-    Using rspec-core (2.10.1)
-    Using rspec-expectations (2.10.0)
-    Using rspec-mocks (2.10.1)
-    Using rspec (2.10.0)
-    Using bundler (1.1.5)
-    Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
+    host { 'localhost':
+      ensure => 'present',
+      ip     => '127.0.0.1',
+      target => '/etc/hosts',
+    }
 
 # Running Tests #
 
 Puppet Labs projects use a common convention of using Rake to run unit tests.
 The tests can be run with the following rake task:
 
-    rake spec
-    # Or if using Bundler
     bundle exec rake spec
 
 This allows the Rakefile to set up the environment beforehand if needed. This
@@ -199,14 +167,20 @@ method is how the unit tests are run in [Jenkins](https://jenkins.puppetlabs.com
 Under the hood Puppet's tests use `rspec`.  To run all of them, you can directly
 use 'rspec':
 
-    rspec
-    # Or if using Bundler
     bundle exec rspec
 
 To run a single file's worth of tests (much faster!), give the filename, and use
 the nested format to see the descriptions:
 
-    rspec spec/unit/ssl/host_spec.rb --format nested
+    bundle exec rspec spec/unit/ssl/host_spec.rb --format nested
+
+## Testing dependency version requirements
+
+Puppet is only compatible with certain versions of RSpec and Mocha. If you are
+not using Bundler to install the required test libraries you must ensure that
+you are using the right library versions. Using unsupported versions of Mocha
+and RSpec will probably display many spurious failures. The supported versions
+of RSpec and Mocha can be found in the project Gemfile.
 
 # A brief introduction to testing in Puppet
 
@@ -468,6 +442,95 @@ describe "mocking an object" do
   end
 end
 ```
+### Writing tests without side effects
+
+When properly written each test should be able to run in isolation, and tests
+should be able to be run in any order. This makes tests more reliable and allows
+a single test to be run if only that test is failing, instead of running all
+17000+ tests each time something is changed. However, there are a number of ways
+that can make tests fail when run in isolation or out of order.
+
+#### Using instance variables
+
+Puppet has a number of older tests that use `before` blocks and instance
+variables to set up fixture data, instead of `let` blocks. These can retain
+state between tests, which can lead to test failures when tests are run out of
+order.
+
+```ruby
+# test.rb
+RSpec.configure do |c|
+  c.mock_framework = :mocha
+end
+
+describe "fixture data" do
+  describe "using instance variables" do
+
+    # BAD
+    before :all do
+      # This fixture will be created only once and will retain the `foo` stub
+      # between tests.
+      @fixture = stub 'test data'
+    end
+
+    it "can be stubbed" do
+      @fixture.stubs(:foo).returns :bar
+      @fixture.foo.should == :bar
+    end
+
+    it "should not keep state between tests" do
+      # The foo stub was added in the previous test and shouldn't be present
+      # in this test.
+      expect { @fixture.foo }.to raise_error
+    end
+  end
+
+  describe "using `let` blocks" do
+
+    # GOOD
+    # This will be recreated between tests so that state isn't retained.
+    let(:fixture) { stub 'test data' }
+
+    it "can be stubbed" do
+      fixture.stubs(:foo).returns :bar
+      fixture.foo.should == :bar
+    end
+
+    it "should not keep state between tests" do
+      # since let blocks are regenerated between tests, the foo stub added in
+      # the previous test will not be present here.
+      expect { fixture.foo }.to raise_error
+    end
+  end
+end
+```
+
+```
+bundle exec rspec test.rb -fd
+
+fixture data
+  using instance variables
+    can be stubbed
+    should not keep state between tests (FAILED - 1)
+  using `let` blocks
+    can be stubbed
+    should not keep state between tests
+
+Failures:
+
+  1) fixture data using instance variables should not keep state between tests
+     Failure/Error: expect { @fixture.foo }.to raise_error
+       expected Exception but nothing was raised
+     # ./test.rb:17:in `block (3 levels) in <top (required)>'
+
+Finished in 0.00248 seconds
+4 examples, 1 failure
+
+Failed examples:
+
+rspec ./test.rb:16 # fixture data using instance variables should not keep state between tests
+```
+
 
 ### RSpec references
 
@@ -703,11 +766,11 @@ default clientbucket.
 Create a module that recursively downloads something.  The jeffmccune-filetest
 module will recursively copy the rubygems source tree.
 
-    $ puppet module install jeffmccune-filetest
+    $ bundle exec puppet module install jeffmccune-filetest
 
 Start the master with the StaticCompiler turned on:
 
-    $ puppet master \
+    $ bundle exec puppet master \
         --catalog_terminus=static_compiler \
         --verbose \
         --no-daemonize
@@ -722,7 +785,7 @@ Add the special Filebucket[puppet] resource:
 
 Get the static catalog:
 
-    $ puppet agent --test
+    $ bundle exec puppet agent --test
 
 You should expect all file metadata to be contained in the catalog, including a
 checksum representing the content.  When managing an out of sync file resource,