r10k 1.4.2 → 1.5.1

data/doc/dynamic-environments/quickstart.mkd

@@ -211,6 +211,12 @@ class profile::base {
 }
 ```
 
+Ensure that the user r10k runs as (typically root) can access the git
+repository. See the [git environment guide]
+(doc/dynamic-environments/git-environments.mkd) for more detail.  You can test
+the access by using su/sudo to perform `git clone yourrepoURL` as the correct
+user.
+
 # Summary
 
 We now have the following functional pieces:

data/doc/faq.mkd

@@ -43,10 +43,10 @@ on your remote repositories:
   * [Gitolite v2][gitolite-v2-default-branch]
   * [Gitolite v3][gitolite-v3-default-branch]
 
-[git-clone]: https://www.kernel.org/pub/software/scm/git/docs/git-clone.html "git-clone(1)"
-[git-symbolic-ref]: https://www.kernel.org/pub/software/scm/git/docs/git-symbolic-ref.html "git-symbolic-ref(1)"
+[git-clone]: https://www.kernel.org/pub/software/scm/git/docs/git-clone.html "Man page for git-clone"
+[git-symbolic-ref]: https://www.kernel.org/pub/software/scm/git/docs/git-symbolic-ref.html "Man page for git-symbolic-ref"
 
-[github-default-branch]: https://help.github.com/articles/setting-the-default-branch "Setting the default branch"
-[bitbucket-default-branch]: https://answers.atlassian.com/questions/280944/how-to-change-main-branch-in-bitbucket "How to change MAIN branch in BitBucket?"
-[gitolite-v2-default-branch]: http://stackoverflow.com/questions/7091599/git-default-remote-branch-with-gitolite "git default remote branch with gitolite"
-[gitolite-v3-default-branch]: http://stackoverflow.com/questions/13949093/git-change-default-branch-gitolite "git change default branch (gitolite)"
+[github-default-branch]: https://help.github.com/articles/setting-the-default-branch "Changing the default branch on GitHub"
+[bitbucket-default-branch]: https://answers.atlassian.com/questions/280944/how-to-change-main-branch-in-bitbucket "Changing the default branch on Bitbucket"
+[gitolite-v2-default-branch]: http://stackoverflow.com/questions/7091599/git-default-remote-branch-with-gitolite "Changing the default branch on Gitolite v2"
+[gitolite-v3-default-branch]: http://stackoverflow.com/questions/13949093/git-change-default-branch-gitolite "Changing the default branch on Gitolite v3"

data/doc/git/cloning-and-mirroring.mkd

@@ -0,0 +1,60 @@
+Git Cloning and Mirroring
+=========================
+
+Most Git operations on a repository require having a full clone of that
+repository locally available. Because this clone is a full copy, the initial
+clone is heavyweight and generally time consuming. For a situation like r10k
+where the same repositories are reused often, constantly cloning and deleting
+repositories is very inefficient.
+
+In order to speed up Git operations r10k takes a number of steps to avoid
+cloning and fetching repositories, as well as deduplicating content across
+multiple clones of the same repositories.
+
+Because r10k tends to reuse the same Git repositories in multiple places, r10k
+avoids repeated, full repository clones by mirroring repositories. When r10k
+starts using a new repository, it first clones that repository into a central
+location for later use. This initial clone is the only time that r10k will
+perform a full clone of that repository.
+
+When r10k creates an actual checkout of a Git repository, it uses the
+corresponding mirrored repository as a [reference](#git-alternates). This allows
+the working checkout to borrow objects from the mirrored repository instead of
+cloning all of the Git objects again. This saves a great deal of time and space
+as the number of copies of a repository increases.
+
+Mirrored git repositories are cloned into the directory specified by the
+r10k.yaml 'cachedir' setting.
+
+The name 'cachedir' is a bit of a misnomer; Git repositories are mirrored to
+speed up access to the remote resource like a cache, but unlike a traditional
+cache the mirrored repositories are persistent and should not be deleted.
+
+Because the mirrored repository contains all of the objects for all of the
+referencing repositories, deleting the mirrored repository is akin to deleting
+the `.git/objects` directory. Doing so effectively cripples the repository, so
+removing mirrored repositories should be done with care to avoid deleting
+repositories that are still in use.
+
+Git alternates
+--------------
+
+A standard git repository stores all content in the `.git/objects` directory,
+either as a zlib compressed file in `.git/objects/[0-9a-f]{2}/[0-9a-f]{38}` or
+as part of a packfile in `.git/objects/pack`. Since content can be both stored
+in and retrieved from this location it's treated like a simple database, and is
+generally referred to as the Git object database.
+
+Git allows a single repository to look in more than just `.git/objects` for
+objects; additional object databases are referred to as **alternates**. If a
+repository has alternate object databases set up, it will check `.git/objects`
+and then each alternate object database when looking for an object. Git stores a
+list of alternate object databases in `.git/objects/info/alternates`. Invoking
+`git clone` with the `--reference <repo>` flag will use that repository as an
+alternate object database.
+
+Links
+-----
+
+  * [man git-clone(1) --reference](https://www.kernel.org/pub/software/scm/git/docs/git-clone.html)
+  * [man gitrepository-layout(5) objects/info/alternates](https://www.kernel.org/pub/software/scm/git/docs/gitrepository-layout.html)

data/doc/git/providers.mkd

@@ -0,0 +1,68 @@
+Git Providers
+=============
+
+As of 1.5.0, r10k can interact with Git repositories using multiple Git
+providers.
+
+Shellgit
+--------
+
+The shellgit provider is the original Git provider that is based on shelling out
+to the `git` binary. It relies on the standard set of Git userland executables
+in order to work.
+
+The shellgit provider is the default Git provider in order to maintain
+compatibility with existing r10k installations.
+
+### Requirements
+
+The shellgit provider requires that `git` can be found on the `PATH` environment
+variable. This can be done by installing the git package via the system package
+manager.
+
+### SSH Configuration
+
+Because the shellgit provider relies on the `git` command which in turn uses the
+`ssh` binary as the SSH transport layer, configuring access to Git repositories
+over SSH is done by configuring the underlying `ssh` command.
+
+Rugged
+------
+
+The rugged provider is based on the [libgit2](https://github.com/libgit2/libgit2)
+library and the Ruby [rugged gem](https://github.com/libgit2/rugged).
+
+### Requirements
+
+The rugged provider relies on the rugged Ruby gem. The rugged gem requires Ruby
+1.9 or greater, which in turn means that r10k must be run with Ruby 1.9 or
+greater to use this provider.
+
+### SSH Configuration
+
+In order to use rugged with SSH based Git repositories, the 'private_key' option
+must be provided. An optional 'username' field can be provided when the Git
+remote URL does not provide a username.
+
+```yaml
+git:
+  private_key: '/root/.ssh/id_rsa'
+  username: 'git'
+```
+
+Configuration
+-------------
+
+R10K will attempt to use the shellgit provider, then fall back to the rugged
+provider, and then hard fail if no Git provider is available.
+
+The Git provider in use can be manually specified by specifying the desired
+provider in r10k.yaml.
+
+```yaml
+git:
+  provider: 'rugged'
+```
+
+Valid values are 'rugged' and 'shellgit'. If an invalid value is used r10k will
+raise an error.

data/integration/Gemfile

@@ -0,0 +1,3 @@
+source ENV['GEM_SOURCE'] || 'https://rubygems.org'
+
+gem 'beaker', '~> 2.7.1'

data/integration/README.mkd

@@ -0,0 +1,29 @@
+Integration Tests
+===========================
+
+This folder contains integration tests for the r10k project. These tests were originally written by the QA team
+at Puppet Labs and is actively maintained by the QA team. Feel free to contribute tests to this folder as long
+as they are written with [Beaker](https://github.com/puppetlabs/beaker) and follow the guidelines
+below.
+
+## Integration?
+
+The r10k project already contains RSpec tests and you might be wondering why there is a need to have a set of
+tests separate from those tests. At Puppet Labs we define an "integration" test as:
+
+> Validating the system state and/or side effects while completing a complete life cycle of user stories using a
+> system. This type of test crosses the boundary of a discrete tool in the process of testing a defined user
+> objective that utilizes a system composed of integrated components.
+
+What this means for this project is that we will install and configure all infrastructure needed in a real-world
+r10k environment.
+
+## Running Tests
+
+Included in this folder under the "test_run_scripts" sub-folder are simple Bash scripts that will run suites of
+Beaker tests. This scripts utilize environment variables for specifying test infrastructure. For security
+reasons we do not provide examples from the Puppet Labs testing environment.
+
+## Documentation
+
+Each sub-folder contains a "README.mdk" that describes the content found in the sub-folder.

data/integration/configs/README.mkd

@@ -0,0 +1,7 @@
+Config
+===========================
+
+This folder contains [Beaker](https://github.com/puppetlabs/beaker) configuration files. All of these
+configurations utilize the ["vmpooler"](https://github.com/puppetlabs/vmpooler) service for templates.
+These configurations are specific to the Puppet Labs environment and will not work in other environments without
+alteration!

data/integration/configs/pe/centos-6-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  centos-6-x86_64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: el-6-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/centos-6-x86_64
+    hypervisor: vcloud
+  centos-6-x86_64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: el-6-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/centos-6-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/centos-7-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  centos-7-x86_64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: el-7-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/centos-7-x86_64
+    hypervisor: vcloud
+  centos-7-x86_64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: el-7-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/centos-7-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/debian-6-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  debian-6-amd64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: debian-6-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/debian-6-x86_64
+    hypervisor: vcloud
+  debian-6-amd64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: debian-6-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/debian-6-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/debian-7-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  debian-7-amd64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: debian-7-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/debian-7-x86_64
+    hypervisor: vcloud
+  debian-7-amd64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: debian-7-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/debian-7-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/redhat-6-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  redhat-6-x86_64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: el-6-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/redhat-6-x86_64
+    hypervisor: vcloud
+  redhat-6-x86_64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: el-6-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/redhat-6-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/redhat-7-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  redhat-7-x86_64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: el-7-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/redhat-7-x86_64
+    hypervisor: vcloud
+  redhat-7-x86_64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: el-7-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/redhat-7-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/sles-11-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  sles-11-x86_64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: sles-11-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/sles-11-x86_64
+    hypervisor: vcloud
+  sles-11-x86_64-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: sles-11-x86_64
+    template: Delivery/Quality Assurance/Templates/vCloud/sles-11-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/ubuntu-1004-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  ubuntu-1004-amd64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: ubuntu-10.04-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/ubuntu-1004-x86_64
+    hypervisor: vcloud
+  ubuntu-1004-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: ubuntu-10.04-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/ubuntu-1004-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/ubuntu-1204-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  ubuntu-1204-amd64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: ubuntu-12.04-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/ubuntu-1204-x86_64
+    hypervisor: vcloud
+  ubuntu-1204-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: ubuntu-12.04-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/ubuntu-1204-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com

data/integration/configs/pe/ubuntu-1404-64mda

@@ -0,0 +1,25 @@
+HOSTS:
+  ubuntu-1404-amd64:
+    roles:
+      - master
+      - dashboard
+      - database
+      - agent
+    platform: ubuntu-14.04-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/ubuntu-1404-x86_64
+    hypervisor: vcloud
+  ubuntu-1404-agent:
+    roles:
+      - agent
+      - frictionless
+    platform: ubuntu-14.04-amd64
+    template: Delivery/Quality Assurance/Templates/vCloud/ubuntu-1404-x86_64
+    hypervisor: vcloud
+CONFIG:
+  nfs_server: none
+  consoleport: 443
+  datastore: instance0
+  folder: Delivery/Quality Assurance/Enterprise/Dynamic
+  resourcepool: delivery/Quality Assurance/Enterprise/Dynamic
+  pooling_api: http://vmpooler.delivery.puppetlabs.net/
+  forge_host: forge-aio01-petest.puppetlabs.com