r10k 3.5.2

data/doc/dynamic-environments/configuration.mkd

@@ -0,0 +1,669 @@
+Dynamic Environment Configuration
+=================================
+
+R10k uses a configuration file to determine how dynamic environments should be
+deployed.
+
+Config file location
+--------------------
+
+### Manual configuration
+
+An explicit configuration file location be specified by providing the `--config`
+option to `r10k deploy`, like so:
+
+    r10k deploy --config /srv/puppet/r10k.yaml [...]
+
+### Automatic configuration
+
+If an explicit configuration file is not given, r10k will search the following
+locations for a configuration file.
+
+  * `{current working directory}/r10k.yaml`
+  * `/etc/puppetlabs/r10k/r10k.yaml` (1.5.0 and later)
+  * `/etc/r10k.yaml` (deprecated in 1.5.0)
+
+In 1.5.0 r10k added `/etc/puppetlabs/r10k/r10k.yaml` to the configuration search
+path. The old location, `/etc/r10k.yaml` has been deprecated in favor of the new
+location. If both `/etc/puppetlabs/r10k/r10k.yaml` and `/etc/r10k.yaml` exist
+and explicit configuration file has not been given, r10k will log a warning and
+use `/etc/puppetlabs/r10k/r10.yaml`.
+
+General options
+---------------
+
+### cachedir
+
+The 'cachedir' setting specifies where r10k should keep cached information.
+Right now this is predominantly used for caching git repositories but will be
+expanded as other subsystems can take advantage of caching.
+
+For example:
+
+```yaml
+---
+# Store all cache information in /var/cache
+cachedir: '/var/cache/r10k'
+```
+
+
+The cachedir setting defaults to `~/.r10k`. If the HOME environment variable is
+unset r10k will assume that r10k is being run with the Puppet [`prerun_command`](https://puppet.com/docs/puppet/latest/configuration.html#preruncommand)
+setting and will set the cachedir default to `/root/.r10k`.
+
+### proxy
+
+The 'proxy' setting configures a proxy server to use for all operations which occur over
+an HTTP(S) transport. You can override this setting for Git or Forge operations only by
+setting the 'proxy' setting under the 'git' or 'forge' settings. You can also override
+for a specific Git repository by setting a proxy in the 'repositories' list of the 'git'
+setting. By default, r10k will look for and use the first environment variable it finds
+in this list: HTTPS\_PROXY, https\_proxy, HTTP\_PROXY, http\_proxy. If no proxy setting
+is found in the environment, this setting will default to use no proxy.
+
+```yaml
+proxy: 'http://proxy.example.com:3128'
+```
+
+r10k also supports using authenticated proxies with either Basic or Digest authentication:
+
+```yaml
+proxy: 'http://user:password@proxy.example.com:3128'
+```
+
+The proxy server being used will be logged at the "debug" level when r10k runs.
+
+### pool_size
+
+The pool_size setting is a number to determine how many threads should be spawn
+while updating modules. The default value is 1, which means the default behaviour
+is to update modules in a serial manner. Increasing this number should bring
+some performance gains.
+
+### git
+
+The 'git' setting is a hash that contains Git specific settings.
+
+#### provider
+
+The provider option determines which Git provider should be used.
+
+```yaml
+git:
+  provider: rugged # one of shellgit, rugged
+```
+
+See the [git provider documentation](../git/providers.mkd) for more information
+regarding Git providers.
+
+#### proxy
+
+The 'proxy' setting allows you to set or override the global proxy setting specifically
+for Git operations that use an HTTP(S) transport. See the global proxy setting documentation
+for more information and examples.
+
+#### username
+
+The username setting is only used by the Rugged git provider.
+
+The username option sets the username for SSH remotes when the SSH URL does not provide
+a username. When used with a Git hosting service this is most sensibly set to 'git'.
+
+The username defaults to the username of the currently logged in user.
+
+```yaml
+git:
+  username: "git"
+```
+
+#### private_key
+
+The private_key setting is only used by the Rugged git provider.
+
+The private_key option specifies the path to the default Git SSH private key for Git SSH remotes.
+The private_key setting must be set if SSH remotes are used.
+
+```yaml
+git:
+  private_key: "/etc/puppetlabs/r10k/ssh/id_rsa"
+```
+
+#### repositories
+
+The repositories option allows configuration to be set on a per-remote basis. Each entry is a map of
+the repository URL and per-repository configuration for that repo.
+
+##### private_key
+
+A repository specific private key to use for SSH connections for the given repository URL. This
+overrides the global private_key setting.
+
+```yaml
+git:
+  repositories:
+    - remote: "ssh://tessier-ashpool.freeside/protected-repo.git"
+      private_key: "/etc/puppetlabs/r10k/ssh/id_rsa-protected-repo-deploy-key"
+```
+
+##### proxy
+
+The 'proxy' setting allows you to set or override the global proxy setting for a single, specific
+repository. See the global proxy setting documentation for more information and examples.
+
+### forge
+
+The 'forge' setting is a hash that contains settings for downloading modules
+from the Puppet Forge.
+
+#### proxy
+
+The 'proxy' setting allows you to set or override the global proxy setting for all Forge
+interactions. See the global proxy setting documentation for more information and examples.
+
+#### baseurl
+
+The 'baseurl' setting indicates where Forge modules should be installed from.
+This defaults to 'https://forgeapi.puppetlabs.com'
+
+```yaml
+forge:
+  baseurl: 'https://private-forge.mysite'
+```
+
+Deployment options
+------------------
+
+The following options configure how r10k deploys dynamic environments.
+
+### postrun
+
+The `postrun` setting specifies an arbitrary command to run after deploying all
+environments. The command must be an array of strings that will be used as an
+argument vector. The exit code of the command is not currently used, but the
+command should exit with a return code of 0 as the exit code may have semantics
+in the future.
+
+```yaml
+---
+postrun: ['/usr/bin/curl', '-F', 'deploy=done', 'http://my-app.site/endpoint']
+```
+
+The postrun setting can only be set once.
+
+Occurrences of the string `$modifiedenvs` in the postrun command will be
+replaced with the current environment(s) being deployed.
+
+### sources
+
+The `sources` setting specifies what repositories should be used for creating
+dynamic environments.  It is a hash where each key is the short name of a
+specific repository (for instance, "qa" or "web" or "ops") and the value is a
+hash of properties for that source.
+
+```yaml
+---
+sources:
+  main:
+    # Source settings follow
+```
+
+### deploy
+
+The `deploy` setting is a top level setting for controlling how r10k deploys
+behave. At this point only new settings are included under this setting, but in
+the long term the current top level deploy settings will be moved under
+`deploy`.
+
+#### purge\_levels
+
+The `purge_levels` setting controls how aggressively r10k will purge unmanaged
+content during a deployment. Given value must be a list of strings indicating at
+what levels unmanaged content should be purged. The valid string options for the
+list are 'deployment', 'environment', and 'puppetfile'.
+
+```yaml
+---
+deploy:
+  purge_levels: [ 'deployment', 'environment', 'puppetfile' ]
+```
+
+This setting currently only impacts the "deploy environment" action.
+
+The default value is `['deployment', 'puppetfile']` to maintain parity with
+existing behavior before this setting was added.
+
+The effect of enabling the various purge levels is as follows:
+
+##### deployment
+
+After each deploy, in the configured basedir, r10k will recursively remove any
+content found which is not managed by one of the sources declared in the r10k.yaml
+configuration. Note that disabling this level of purging may cause the number of
+deployed environments to grow without bound; deleting branches from a control
+repo would no longer cause the matching environment to be purged.
+
+##### environment
+
+After a given environment is deployed, r10k will recursively remove any content
+found which is neither committed to the control repo branch that maps to that
+environment, nor declared in a Puppetfile committed to that branch.
+
+Enabling this purge level will cause r10k to load and parse the Puppetfile for
+the environment even without the `--puppetfile` flag being set. However,
+Puppetfile content will still only be deployed if the environment is new or
+the `--puppetfile` flag is set. Additionally, no environment-level content
+will be purged if any errors are encountered while evaluating the Puppetfile
+or deploying its contents.
+
+Note that the .r10k-deploy.json file is exempt from this purging.
+
+##### puppetfile
+
+After Puppetfile content for a given environment is deployed, r10k will
+recursively remove any content found in a directory managed by the Puppetfile
+which is not also declared in that Puppetfile. Directories considered to be
+managed by a Puppetfile include the configured `moduledir` (which defaults to
+"modules") as well as alternate directories specified as an `install_path`
+option to any Puppetfile content declarations.
+
+#### purge\_whitelist
+
+The `purge_whitelist` setting exempts the specified filename patterns from
+being purged. This setting is currently only considered during `environment`
+level purging. (See above.) Given value must be a list of shell style filename
+patterns in string format.
+
+See the Ruby [documentation for the `fnmatch` method](http://ruby-doc.org/core-2.2.0/File.html#method-c-fnmatch)
+for more details on valid patterns. Note that the `FNM_PATHNAME` and
+`FNM_DOTMATCH` flags are in effect when r10k considers the whitelist.
+
+Patterns are relative to the root of the environment being purged and *do
+not match recursively* by default. For example, a whitelist value of
+`*myfile*` would only preserve a matching file at the root of the
+environment. To preserve the file throughout the deployed environment,
+a recursive pattern such as `**/*myfile*` would be required.
+
+Files matching a whitelist pattern may still be removed if they exist in
+a folder that is otherwise subject to purging. In this case, an additional
+whitelist rule to preserve the containing folder is required.
+
+```yaml
+---
+deploy:
+  purge_whitelist: [ 'custom.json', '**/*.xpp' ]
+```
+
+
+#### write\_lock
+
+The `write_lock` setting allows administrators to temporarily disallow r10k code
+deploys without having to remove the r10k configuration entirely. This can be
+useful to prevent r10k deploys at certain times or prevent r10k from interfering
+with a common set of code that may be touched by multiple r10k configurations.
+
+```yaml
+---
+deploy:
+  write_lock: "Deploying code is disallowed until the next maintenance window (2038-01-19)"
+```
+
+#### generate\_types
+
+The `generate_types` setting controls whether r10k should update generated types
+after a successful environment update. See [Environment isolation](https://puppet.com/docs/puppet/latest/environment\_isolation.html)
+for more information on generated types. Defaults to false.
+
+```yaml
+deploy:
+  generate_types: true
+```
+
+#### puppet\_path
+
+The path to the puppet executable used for generating types. Defaults to `/opt/puppetlabs/bin/puppet`.
+
+```yaml
+deploy:
+  puppet_path: '/usr/local/bin/puppet'
+```
+
+Source options
+--------------
+
+The following options are respected by all source implementations. Sources may
+implement other options in addition to the ones listed below; see the source
+specific documentation for more information.
+
+### remote
+
+The 'remote' setting specifies where the source repository should be fetched
+from. It may be any valid URL that the source may check out or clone. The remote
+must be able to be fetched without any interactive input, eg usernames or
+passwords cannot be prompted for in order to fetch the remote.
+
+```yaml
+---
+sources:
+  mysource:
+    remote: 'git://git-server.site/my-org/main-modules'
+```
+
+### basedir
+
+The 'basedir' setting specifies where environments will be created for this
+source.  This directory will be entirely managed by r10k and any contents that
+r10k did not put there will be _removed_.
+
+```yaml
+---
+sources:
+  mysource:
+    basedir: '/etc/puppet/environments'
+```
+
+If two different sources have the same basedir, it's possible for them to create
+two separate environments with the same name and file path. If this occurs r10k
+will treat this as a fatal error and will abort. To avoid this, use prefixing on one
+or both of the sources to make sure that all environment names are unique. See
+also the [prefix](#prefix) setting.
+
+### prefix
+
+The prefix setting allows environment names to be prefixed with the short name
+of the given source. This prevents collisions when multiple sources are deployed
+into the same directory.
+
+```yaml
+---
+sources:
+  mysource:
+    basedir: '/etc/puppet/environments'
+    prefix: true # All environments will be prefixed with "mysource_"
+```
+#### prefix behaviour
+* if `true` environment folder will be prefixed with the name of the source.
+* if `false` (default) environment folder will not be prefixed
+* if `String` environment folder will be prefixed with the `prefix` value.
+
+### ignore_branch_prefixes
+
+The 'ignore_branch_prefixes' setting causes environments to be ignored which match in part or whole
+to any of the prefixes listed in the setting.
+The setting is a list of strings. Each branch in
+the 'git' repo will have its name tested against all prefixes and, if the prefix
+is found, then an environment will not be deployed for this branch.
+If no 'ignore_branch_prefixes' is specified, then all branches in the 'git' repo will
+be deployed (default behavior).
+
+#### ignore_branch_prefixes behaviour
+* if empty, deploy environments for all branches
+* for each branch in git repo
+** if `branch.name` has a prefix found in `ignore_branch_prefixes`, then do not deploy an environment for branch
+
+Example: do not deploy branches with names starting with (or completely named) 'test' or 'dev'.
+```yaml
+---
+sources:
+  mysource:
+    basedir: '/etc/puppet/environments'
+    ignore_branch_prefixes:
+      - 'test'
+      - 'dev'
+```
+
+Examples
+--------
+
+### Minimal example
+
+The majority of users will only have a single repository where all modules and
+hiera data files are kept. In this case you will specify a single source:
+
+```yaml
+---
+sources:
+  operations:
+    remote: 'git://git-server.site/my-org/org-modules'
+    basedir: '/etc/puppet/environments'
+```
+
+### Separate hiera data
+
+For more complex cases where you want to store hiera data in a different
+repository and your modules in another repository, you can specify two sources:
+
+```yaml
+---
+sources:
+  operations:
+    remote: 'git://git-server.site/my-org/org-modules'
+    basedir: '/etc/puppet/environments'
+  hiera:
+    remote: 'git://git-server.site/my-org/org-hiera-data'
+    basedir: '/etc/puppet/hiera-data'
+```
+
+### Multiple tenancy
+
+Alternately you may want to create separate environments from multiple
+repositories. This is useful when you want two groups to be able to deploy
+Puppet modules but they should only have write access to their own modules and
+not the modules of other groups.
+
+```yaml
+---
+sources:
+  main:
+    remote: 'git://git-server.site/my-org/main-modules'
+    basedir: '/etc/puppet/environments'
+    prefix: false # Prefix defaults to false so this is only here for clarity
+  qa:
+    remote: 'git://git-server.site/my-org/qa-puppet-modules'
+    basedir: '/etc/puppet/environments'
+    prefix: true
+  dev:
+    remote: 'git://git-server.site/my-org/dev-puppet-modules'
+    basedir: '/etc/puppet/environments'
+    prefix: true
+```
+
+This will create the following directory structure:
+
+```
+/etc/puppet/environments
+|-- production       # main-modules repository, production branch
+|-- upgrade_apache   # main-modules repository, upgrade_apache branch
+|-- qa_production    # qa repository, production branch
+|-- qa_jenkins_test  # qa repository, jenkins_test branch
+|-- dev_production   # dev repository, production branch
+`-- dev_loadtest     # dev repository, loadtest branch
+```
+
+#### Multiple tenancy with external hieradata
+
+If hiera data is in a separate repository from your control repository, you
+must override the `prefix` so environment folders line up in both directories:
+
+```yaml
+---
+sources:
+  app1_data:
+    remote: 'git://git-server.site/my-org/app1-hieradata'
+    basedir: '/etc/puppet/hieradata'
+    prefix: "app1"
+  app1_modules:
+    remote: 'git://git-server.site/my-org/app1-puppet-modules'
+    basedir: '/etc/puppet/environments'
+    prefix: "app1"
+```
+
+
+This will create the following directory structure:
+
+```
+/etc/puppet/environments
+|-- app1_production  # app1 modules repository, production branch
+|-- app1_develop     # app1 modules repository, develop branch
+
+/etc/puppet/hieradata
+|-- app1_production  # app1 data repository, production branch
+|-- app1_develop     # app1 data repository, develop branch
+```
+
+Experimental Features
+---------------------
+
+### YAML Environment Source
+
+Dynamically deploying Puppet content based on the state of version control repositories can be powerful and efficient for development workflows. The linkage however is not advantageous when trying to build precision controls over deployment of previously-developed and tested content.
+
+The YAML environment source type allows for a clear separation of tooling between development workflow, and deployment workflow. Development workflow creates new commits in the version control system. Deployment workflow consumes them.
+
+To use the YAML environment source, configure r10k's sources with at least one entry using the yaml type.
+
+```yaml
+# r10k.yaml
+---
+sources:
+  puppet:
+    type: yaml
+    basedir: /etc/puppetlabs/code/environments
+    config: /etc/puppetlabs/r10k/environments.yaml # default
+```
+
+When using the YAML source type, every environment is enumerated in a single yaml file. Each environment specifies a type, source, and version (typically a Git ref) to deploy. In the following example, two environments are defined, which are identical to each other.
+
+```yaml
+---
+production:
+  type: git
+  remote: git@github.com:puppetlabs/control-repo.git
+  ref: 8820892
+
+development:
+  type: git
+  remote: git@github.com:puppetlabs/control-repo.git
+  ref: 8820892
+```
+
+### YAMLdir Environment Source
+
+Like the YAML environment source, but implemented as a conf.d pattern.
+
+```yaml
+# r10k.yaml
+---
+sources:
+  puppet:
+    type: yamldir
+    basedir: /etc/puppetlabs/code/environments
+    config: /etc/puppetlabs/r10k/environments.d # default
+```
+
+Each environment is defined in a yaml file placed in the configuration directory. The filename, without the .yaml extension, will be the name of the environment.
+
+```
+/etc/puppetlabs/r10k/environments.d
+├── production.yaml
+└── development.yaml
+```
+
+The contents of the file should be a hash specifying the enviornment type, and all other applicable environment options.
+
+```yaml
+# production.yaml
+---
+type: git
+remote: git@github.com:puppetlabs/control-repo.git
+ref: 8820892
+```
+
+### Exec environment Source
+
+The exec environment source runs an external command which is expected to return on stdout content compatible with the YAML environment source data format. The command may return the data in JSON or YAML form. The exec environment source is similar in purpose to Puppet's exec node terminus, used to implement external node classifiers (ENCs). R10k's exec source type allows the the implementation of external environment sources.
+
+```yaml
+# r10k.yaml
+---
+sources:
+  puppet:
+    type: exec
+    basedir: /etc/puppetlabs/code/environments
+    command: /usr/local/bin/r10k-environments.sh
+```
+
+### Environment Modules
+
+The environment modules feature allows module content to be attached to an environment at environment definition time. This happens before modules specified in a Puppetfile are attached to an environment, which does not happen until deploy time. Environment module implementation depends on the environment source type.
+
+For the YAML environment source type, attach modules to an environment by specifying a modules key for the environment, and providing a hash of modules to attach. Each module accepts the same arguments accepted by the `mod` method in a Puppetfile.
+
+The example below includes two Forge modules and one module sourced from a Git repository. The two environments are almost identical. However, a new version of the stdlib module has been deployed in development (6.2.0), that has not yet been deployed to production.
+
+```yaml
+---
+production:
+  type: git
+  remote: git@github.com:puppetlabs/control-repo.git
+  ref: 8820892
+  modules:
+    puppetlabs-stdlib: 6.0.0
+    puppetlabs-concat: 6.1.0
+    reidmv-xampl:
+      git: https://github.com/reidmv/reidmv-xampl.git
+      ref: 62d07f2
+
+development:
+  type: git
+  remote: git@github.com:puppetlabs/control-repo.git
+  ref: 8820892
+  modules:
+    puppetlabs-stdlib: 6.2.0
+    puppetlabs-concat: 6.1.0
+    reidmv-xampl:
+      git: https://github.com/reidmv/reidmv-xampl.git
+      ref: 62d07f2
+```
+
+An example of a single environment definition for the YAMLdir environment source type:
+
+```yaml
+# production.yaml
+---
+type: git
+remote: git@github.com:puppetlabs/control-repo.git
+ref: 8820892
+modules:
+  puppetlabs-stdlib: 6.0.0
+  puppetlabs-concat: 6.1.0
+  reidmv-xampl:
+    git: https://github.com/reidmv/reidmv-xampl.git
+    ref: 62d07f2
+```
+
+### Bare Environment Type
+
+A "control repository" typically contains a hiera.yaml, an environment.conf, a manifests/site.pp file, and a few other things. However, none of these are strictly necessary for an environment to be functional if modules can be deployed to it.
+
+The bare environment type allows sources that support environment modules to operate without a control repo being required. Modules can be deployed directly.
+
+```yaml
+---
+production:
+  type: bare
+  modules:
+    puppetlabs-stdlib: 6.0.0
+    puppetlabs-concat: 6.1.0
+    reidmv-xampl:
+      git: https://github.com/reidmv/reidmv-xampl.git
+      ref: 62d07f2
+
+development:
+  type: bare
+  modules:
+    puppetlabs-stdlib: 6.0.0
+    puppetlabs-concat: 6.1.0
+    reidmv-xampl:
+      git: https://github.com/reidmv/reidmv-xampl.git
+      ref: 62d07f2
+```