r10k 3.3.3 → 3.5.2

data/doc/dynamic-environments/git-environments.mkd

@@ -29,7 +29,7 @@ seamlessly reflected in Puppet environments. This means that creating a new Git
 branch creates a new Puppet environment, updating a Git branch will update that
 environment, and deleting a Git branch will remove that environment.
 
-R10k supports both [directory and config file environments](https://docs.puppetlabs.com/puppet/latest/reference/environments.html).
+R10k supports both [directory and config file environments](https://puppet.com/docs/puppet/latest/env_environments.html).
 Ensure that the basedir for your sources and your puppet config align.
 
 How it works

data/doc/dynamic-environments/master-configuration.mkd

@@ -1,70 +1,40 @@
 Puppet master configuration
 ===========================
 
-In order to use dynamic environments, your Puppet masters will need to be
+In order to use environments, your Puppet masters will need to be
 configured to load manifests and modules relative to the requested environment.
-The following settings should be configured in puppet.conf.
 
-## Puppet <= 3.4.x
+This is the default behavior in Puppet 4+
 
-In Puppet 3.4 and earlier, dynamic environments are achieved by interpolating
-the `$environment` variable in the modulepath, manifestdir, and manifest
-settings. When environments is loaded the settings are dynamically set based on
-the name of the environment, thus allowing environments to be created on the
-fly.
+This behavior is controled by the following settings
+(listed with their default values):
 
-    [master]
-        modulepath = /etc/puppet/environments/$environment/modules:/etc/puppet/environments/$environment/dist
+```
+codedir = /etc/puppetlabs/code
+environmentpath = $codedir/environments
+basemodulepath = $codedir/modules:/opt/puppetlabs/puppet/share/modules
+```
 
-        # If you use a top level manifest dir
-        manifestdir = /etc/puppet/environments/$environment/manifests
-        # If you use a specific site wide manifest
-        manifest = /etc/puppet/environments/$environment/manifests/nodes.pp
+The environment requested by the agent or assigned to it by an ENC is looked
+for at `$environmentpath/$environment`.
 
-## Puppet = 3.5.x
+That environment may have an environment.conf file in its base directory that
+specifies its own modulepath. If not, the default computed modulepath for that
+environment is `$environmentpath/$environment/modules:$basemodulepath`.
 
-Puppet 3.5.0 adds initial support for "directory environments", where
-environments are created by enumerating directories in `$environmentpath`.
-This new configuration expect that all modules are in the root of the
-directories in the environment path.
+These configuration variables are documented at
+[Puppet Configuration Reference](https://puppet.com/docs/puppet/latest/configuration.html)
+and their interaction in forming the modulepath is documented at
+[Directories and the Modulepath](https://puppet.com/docs/puppet/latest/dirs_modulepath.html).
+More information can be found about environments in general at
+[Creating Environmnets](https://puppet.com/docs/puppet/latest/environments_creating.html).
 
-    [master]
-    # None of modulepath, manifestdir, or manifest should be enabled
-    # See http://docs.puppetlabs.com/puppet/latest/reference/environments.html#enabling-directory-environments
-    #   for more information on the changes
-        environmentpath = $confdir/environments
+The evolution to using environments in this way was a gradual one from community
+conventions to supported feature and there were several intermediate stages that
+had different configuration requirements within the puppet.conf. Some Puppet
+documentation from the 3.x and 4.x series may no longer be applicable to the
+above which solidified in Puppet 4.x.
 
-Note that these settings go into the `[master]` section; you don't need to
-explicitly configure an environment section for each environment you want to
-use. (But you can if you want.)
-
-## Puppet >= 3.6.0
-
-[environmentconf]: http://docs.puppetlabs.com/puppet/latest/reference/config_file_environment.html
-
-Puppet 3.6.0 adds more fine grained control over how directory environments are
-configured. Each directory based environment can have an
-[`environment.conf`][environmentconf] file in the root of that environment that
-can specify the manifest, modulepath, config_version, and environment_timeout
-for that specific environment.
-
-    # puppet.conf
-    [master]
-    # None of modulepath, manifestdir, or manifest should be enabled
-    # See http://docs.puppetlabs.com/puppet/latest/reference/environments.html#enabling-directory-environments
-    #   for more information on the changes
-        environmentpath = $confdir/environments
-
-- - -
-
-    # environment.conf
-    manifest  = site.pp
-    moduledir = modules:dist
-
-A deployed environment with a Puppetfile will look something like this:
-
-    .
-    ├── Puppetfile
-    ├── environment.conf
-    ├── dist
-    └── modules
+If you need to upgrade away from any intermediate setup see
+[Environments in Puppet 3.8](https://puppet.com/docs/puppet/3.8/environments.html)
+for examples that may help.

data/doc/faq.mkd

@@ -141,7 +141,7 @@ modulepath = modules:external-modules
 
 Lastly, you can simply move your locally versioned modules to a separate
 directory to avoid conflicting over the `/modules` directory entirely. With this
-example as well you can use the `environment.conf file to tell Puppet which
+example as well you can use the `environment.conf` file to tell Puppet which
 directories contain modules.
 
 ```
@@ -149,6 +149,11 @@ directories contain modules.
 modulepath = internal-modules:modules
 ```
 
+#### Does R10K support Local/Private Forge?
+
+Yes. Set the Forge to use _globally_ in `r10k.yaml`. see [Configuration](/doc/dynamic-environments/configuration.mkd#baseurl) for details.
+
+
 #### What does the name mean?
 
 It’s called R10K because I’m terrible at names. When I started working on R10K

data/doc/puppetfile.mkd

@@ -54,6 +54,8 @@ The `forge` setting specifies which server that Forge based modules are fetched
 from. This is currently a noop and is provided for compatibility with
 librarian-puppet.
 
+R10k supports setting the Forge to use _globally_ in `r10k.yaml`. see [Configuration](/doc/dynamic-environments/configuration.mkd#baseurl) for details.
+
 ### moduledir
 
 The `moduledir` setting specifies where modules from the Puppetfile will be

data/docker/Makefile

@@ -1,22 +1,43 @@
 PUPPERWARE_ANALYTICS_STREAM ?= dev
 NAMESPACE ?= puppet
-git_describe = $(shell git describe)
+git_describe = $(shell git describe --tags)
 vcs_ref := $(shell git rev-parse HEAD)
 build_date := $(shell date -u +%FT%T)
 hadolint_available := $(shell hadolint --help > /dev/null 2>&1; echo $$?)
-hadolint_command := hadolint --ignore DL3008 --ignore DL3018 --ignore DL3028 --ignore DL4000 --ignore DL4001
+hadolint_command := hadolint
 hadolint_container := hadolint/hadolint:latest
-pwd := $(shell pwd)
-export BUNDLE_PATH = $(pwd)/.bundle/gems
-export BUNDLE_BIN = $(pwd)/.bundle/bin
-export GEMFILE = $(pwd)/Gemfile
+export BUNDLE_PATH = $(PWD)/.bundle/gems
+export BUNDLE_BIN = $(PWD)/.bundle/bin
+export GEMFILE = $(PWD)/Gemfile
+export DOCKER_BUILDKIT = 1
 
-version := $(shell command grep VERSION ../lib/r10k/version.rb | awk '{print $$3}' | sed "s/'//g")
-dockerfile := Dockerfile
+ifeq ($(IS_RELEASE),true)
+	VERSION ?= $(shell echo $(git_describe) | sed 's/-.*//')
+	PUBLISHED_VERSION ?= $(shell curl --silent 'https://rubygems.org/api/v1/gems/r10k.json' | jq '."version"' | tr -d '"')
+	CONTAINER_EXISTS = $(shell DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect $(NAMESPACE)/r10k:$(VERSION) > /dev/null 2>&1; echo $$?)
+ifeq ($(CONTAINER_EXISTS),0)
+	SKIP_BUILD ?= true
+else ifneq ($(VERSION),$(PUBLISHED_VERSION))
+	SKIP_BUILD ?= true
+endif
+
+	LATEST_VERSION ?= latest
+	dockerfile := release.Dockerfile
+	dockerfile_context := r10k
+else
+	VERSION ?= edge
+	IS_LATEST := false
+	dockerfile := Dockerfile
+	dockerfile_context := $(PWD)/..
+endif
 
 prep:
 	@git fetch --unshallow 2> /dev/null ||:
 	@git fetch origin 'refs/tags/*:refs/tags/*'
+ifeq ($(SKIP_BUILD),true)
+	@echo "SKIP_BUILD is true, exiting with 1"
+	@exit 1
+endif
 
 lint:
 ifeq ($(hadolint_available),0)
@@ -27,37 +48,38 @@ else
 endif
 
 build: prep
-	@docker build \
+	docker build \
+		${DOCKER_BUILD_FLAGS} \
 		--pull \
 		--build-arg vcs_ref=$(vcs_ref) \
 		--build-arg build_date=$(build_date) \
-		--build-arg version=$(version) \
+		--build-arg version=$(VERSION) \
 		--build-arg pupperware_analytics_stream=$(PUPPERWARE_ANALYTICS_STREAM) \
 		--file r10k/$(dockerfile) \
-		--tag $(NAMESPACE)/r10k:$(version) $(pwd)/..
+		--tag $(NAMESPACE)/r10k:$(VERSION) $(dockerfile_context)
 ifeq ($(IS_LATEST),true)
-	@docker tag $(NAMESPACE)/r10k:$(version) puppet/r10k:latest
+	@docker tag $(NAMESPACE)/r10k:$(VERSION) puppet/r10k:$(LATEST_VERSION)
 endif
 
 test: prep
 	@bundle install --path $$BUNDLE_PATH --gemfile $$GEMFILE --with test
 	@bundle update
-	@PUPPET_TEST_DOCKER_IMAGE=$(NAMESPACE)/r10k:$(version) \
+	@PUPPET_TEST_DOCKER_IMAGE=$(NAMESPACE)/r10k:$(VERSION) \
 		bundle exec --gemfile $$GEMFILE \
 		rspec spec
 
 push-image: prep
-	@docker push $(NAMESPACE)/r10k:$(version)
+	@docker push $(NAMESPACE)/r10k:$(VERSION)
 ifeq ($(IS_LATEST),true)
-	@docker push $(NAMESPACE)/r10k:latest
+	@docker push $(NAMESPACE)/r10k:$(LATEST_VERSION)
 endif
 
 push-readme:
 	@docker pull sheogorath/readme-to-dockerhub
 	@docker run --rm \
 		-v $(PWD)/README.md:/data/README.md \
-		-e DOCKERHUB_USERNAME="$(DISTELLI_DOCKER_USERNAME)" \
-		-e DOCKERHUB_PASSWORD="$(DISTELLI_DOCKER_PW)" \
+		-e DOCKERHUB_USERNAME="$(DOCKERHUB_USERNAME)" \
+		-e DOCKERHUB_PASSWORD="$(DOCKERHUB_PASSWORD)" \
 		-e DOCKERHUB_REPO_PREFIX=puppet \
 		-e DOCKERHUB_REPO_NAME=r10k \
 		sheogorath/readme-to-dockerhub

data/docker/r10k/Dockerfile

@@ -1,7 +1,8 @@
 FROM alpine:3.9 as build
 
-RUN apk add --no-cache ruby git
-RUN mkdir /workspace
+# hadolint ignore=DL3018
+RUN apk add --no-cache ruby git && \
+    mkdir /workspace
 WORKDIR /workspace
 COPY . /workspace
 RUN gem build r10k.gemspec && \
@@ -15,32 +16,52 @@ ARG version="3.1.0"
 # Used by entrypoint to submit metrics to Google Analytics.
 # Published images should use "production" for this build_arg.
 ARG pupperware_analytics_stream="dev"
-ENV PUPPERWARE_ANALYTICS_STREAM="$pupperware_analytics_stream"
-
-ENV R10K_VERSION="$version"
+# required to schedule runs of "r10k" in K8s
+ARG supercronic_version="0.1.9"
+ARG supercronic_sha1sum="5ddf8ea26b56d4a7ff6faecdd8966610d5cb9d85"
+ARG supercronic="supercronic-linux-amd64"
+ARG supercronic_url="https://github.com/aptible/supercronic/releases/download/v$supercronic_version/$supercronic"
 
 LABEL org.label-schema.maintainer="Puppet Release Team <release@puppet.com>" \
       org.label-schema.vendor="Puppet" \
       org.label-schema.url="https://github.com/puppetlabs/r10k" \
       org.label-schema.name="r10k" \
       org.label-schema.license="Apache-2.0" \
-      org.label-schema.version="$R10K_VERSION" \
       org.label-schema.vcs-url="https://github.com/puppetlabs/r10k" \
-      org.label-schema.vcs-ref="$vcs_ref" \
-      org.label-schema.build-date="$build_date" \
       org.label-schema.schema-version="1.0" \
       org.label-schema.dockerfile="/Dockerfile"
 
-RUN apk add --no-cache ruby openssh-client git ruby-rugged curl ruby-dev make gcc musl-dev
-COPY --from=build /workspace/r10k.gem /
-RUN gem install --no-doc r10k.gem json etc && \
-    rm -f r10k.gem
-
-COPY docker/r10k/docker-entrypoint.sh /
-RUN chmod +x /docker-entrypoint.sh
+COPY docker/r10k/adduser.sh docker/r10k/docker-entrypoint.sh /
 COPY docker/r10k/docker-entrypoint.d /docker-entrypoint.d
 
 ENTRYPOINT ["/docker-entrypoint.sh"]
 CMD ["help"]
 
+# dynamic LABELs and ENV vars placed lower for the sake of Docker layer caching
+ENV PUPPERWARE_ANALYTICS_STREAM="$pupperware_analytics_stream"
+
+LABEL org.label-schema.version="$version" \
+      org.label-schema.vcs-ref="$vcs_ref" \
+      org.label-schema.build-date="$build_date"
+
+COPY --from=build /workspace/r10k.gem /
+SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
+# ignore apk and gem pinning
+# hadolint ignore=DL3018,DL3028
+RUN chmod a+x /adduser.sh /docker-entrypoint.sh && \
+# Add a puppet user to run r10k as for consistency with puppetserver
+    /adduser.sh && \
+    chown -R puppet: /docker-entrypoint.d /docker-entrypoint.sh && \
+    apk add --no-cache ruby openssh-client git ruby-rugged curl ruby-json ruby-etc && \
+    gem install --no-doc /r10k.gem && \
+    rm -f /r10k.gem && \
+    curl --fail --silent --show-error --location --remote-name "$supercronic_url" && \
+    echo "${supercronic_sha1sum}  ${supercronic}" | sha1sum -c - && \
+    chmod +x "$supercronic" && \
+    mv "$supercronic" "/usr/local/bin/${supercronic}" && \
+    ln -s "/usr/local/bin/${supercronic}" /usr/local/bin/supercronic
+
+USER puppet
+WORKDIR /home/puppet
+
 COPY docker/r10k/Dockerfile /

data/docker/r10k/adduser.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+
+getent_string="$(getent group | grep -e ':999$')"
+exit_code=$?
+
+if [ "$exit_code" = '0' ]; then
+  group="$(echo $getent_string | cut -d ':' -f1)"
+else
+  addgroup -g 999 puppet
+  group='puppet'
+fi
+
+adduser -G $group -D -u 999 puppet

data/docker/r10k/docker-entrypoint.d/10-analytics.sh

@@ -11,7 +11,7 @@ tid=UA-132486246-5
 # Application Name
 an=r10k
 # Application Version
-av=$R10K_VERSION
+av=$(r10k version | cut -d ' ' -f 2)
 # Anonymous Client ID
 _file=/var/tmp/pwclientid
 cid=$(cat $_file 2>/dev/null || (cat /proc/sys/kernel/random/uuid | tee $_file))

data/docker/r10k/release.Dockerfile

@@ -0,0 +1,54 @@
+FROM alpine:3.9
+
+ARG vcs_ref
+ARG build_date
+ARG version="3.1.0"
+# Used by entrypoint to submit metrics to Google Analytics.
+# Published images should use "production" for this build_arg.
+ARG pupperware_analytics_stream="dev"
+# required to schedule runs of "r10k" in K8s
+ARG supercronic_version="0.1.9"
+ARG supercronic_sha1sum="5ddf8ea26b56d4a7ff6faecdd8966610d5cb9d85"
+ARG supercronic="supercronic-linux-amd64"
+ARG supercronic_url="https://github.com/aptible/supercronic/releases/download/v$supercronic_version/$supercronic"
+
+LABEL org.label-schema.maintainer="Puppet Release Team <release@puppet.com>" \
+      org.label-schema.vendor="Puppet" \
+      org.label-schema.url="https://github.com/puppetlabs/r10k" \
+      org.label-schema.name="r10k" \
+      org.label-schema.license="Apache-2.0" \
+      org.label-schema.vcs-url="https://github.com/puppetlabs/r10k" \
+      org.label-schema.schema-version="1.0" \
+      org.label-schema.dockerfile="/release.Dockerfile"
+
+COPY adduser.sh docker-entrypoint.sh /
+COPY docker-entrypoint.d /docker-entrypoint.d
+
+ENTRYPOINT ["/docker-entrypoint.sh"]
+CMD ["help"]
+
+# dyanmic LABELs and ENV vars placed lower for the sake of Docker layer caching
+ENV PUPPERWARE_ANALYTICS_STREAM="$pupperware_analytics_stream"
+
+LABEL org.label-schema.version="$version" \
+      org.label-schema.vcs-ref="$vcs_ref" \
+      org.label-schema.build-date="$build_date"
+
+SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
+# ignore apk and gem pinning
+# hadolint ignore=DL3018,DL3028
+RUN chmod a+x /adduser.sh /docker-entrypoint.sh && \
+    /adduser.sh && \
+    chown -R puppet: /docker-entrypoint.d /docker-entrypoint.sh && \
+    apk add --no-cache ruby openssh-client git ruby-rugged curl ruby-dev make gcc musl-dev && \
+    gem install --no-doc r10k:"$version" json etc && \
+    curl --fail --silent --show-error --location --remote-name "$supercronic_url" && \
+    echo "${supercronic_sha1sum}  ${supercronic}" | sha1sum -c - && \
+    chmod +x "$supercronic" && \
+    mv "$supercronic" "/usr/local/bin/${supercronic}" && \
+    ln -s "/usr/local/bin/${supercronic}" /usr/local/bin/supercronic
+
+USER puppet
+WORKDIR /home/puppet
+
+COPY release.Dockerfile /

data/docker/spec/dockerfile_spec.rb

@@ -8,9 +8,11 @@ describe 'r10k container' do
   include Pupperware::SpecHelpers
   def run_r10k(command)
     run_command("docker run --detach \
-                   --volume #{File.join(SPEC_DIRECTORY, 'fixtures')}:/test \
+                   --volume #{File.join(SPEC_DIRECTORY, 'fixtures')}:/home/puppet/test \
                    #{@image} #{command} \
-                   --puppetfile /test/Puppetfile")
+                   --verbose \
+                   --trace \
+                   --puppetfile test/Puppetfile")
   end
 
   before(:all) do
@@ -35,7 +37,6 @@ describe 'r10k container' do
     container = result[:stdout].chomp
     wait_on_container_exit(container)
     expect(get_container_exit_code(container)).to eq(0)
-    expect(Dir.exist?(File.join(SPEC_DIRECTORY, 'fixtures', 'modules', 'ntp'))).to eq(true)
     emit_log(container)
     teardown_container(container)
   end

data/docker/spec/fixtures/Puppetfile

@@ -1,2 +1,2 @@
-moduledir 'test/modules'
+moduledir '/tmp/modules'
 mod 'puppetlabs/ntp'

data/integration/Rakefile

@@ -44,8 +44,8 @@ EOS
 
   t.add_env do |env|
     env.name = 'pe_dist_dir'
-    env.message = 'The location to download PE from example "http://enterprise.delivery.puppetlabs.net/20XX.X/ci-ready"'
-    ENV['pe_dist_dir'] ||= "http://enterprise.delivery.puppetlabs.net/#{ENV['PE_FAMILY']}/ci-ready"
+    env.message = 'The location to download PE from, for example "https://artifactory.delivery.puppetlabs.net/artifactory/generic_enterprise__local/20XX.X/ci-ready"'
+    ENV['pe_dist_dir'] ||= "https://artifactory.delivery.puppetlabs.net/artifactory/generic_enterprise__local/#{ENV['PE_FAMILY']}/ci-ready"
   end
 
   t.add_env do |env|

data/lib/r10k/action/deploy/environment.rb

@@ -140,14 +140,14 @@ module R10K
         end
 
         def visit_module(mod)
-          logger.info _("Deploying Puppetfile content %{mod_path}") % {mod_path: mod.path}
+          logger.info _("Deploying %{origin} content %{path}") % {origin: mod.origin, path: mod.path}
           mod.sync(force: @force)
         end
 
         def write_environment_info!(environment, started_at, success)
           module_deploys = []
           begin
-            environment.puppetfile.modules.each do |mod|
+            environment.modules.each do |mod|
               name = mod.name
               version = mod.version
               sha = mod.repo.head rescue nil
@@ -157,7 +157,10 @@ module R10K
             logger.debug("Unable to get environment module deploy data for .r10k-deploy.json at #{environment.path}")
           end
 
-          File.open("#{environment.path}/.r10k-deploy.json", 'w') do |f|
+          # make this file write as atomic as possible in pure ruby
+          final   = "#{environment.path}/.r10k-deploy.json"
+          staging = "#{environment.path}/.r10k-deploy.json~"
+          File.open(staging, 'w') do |f|
             deploy_info = environment.info.merge({
               :started_at => started_at,
               :finished_at => Time.new,
@@ -167,6 +170,7 @@ module R10K
 
             f.puts(JSON.pretty_generate(deploy_info))
           end
+          FileUtils.mv(staging, final)
         end
 
         def undeployable_environment_names(environments, expected_names)