vanagon 0.3.18

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0273c4745ecf7c50f369b6ea518caf46b92c8e07
+  data.tar.gz: 06437a1d3f2838cfd97d673fe2f82981ca917df0
+SHA512:
+  metadata.gz: 2664084d24f35b26e7b30b4ab20c0ee910281af1722ac82363696de47af77962eb1f9cd3980f3d6af8a3aa10a0472f890f27eae66557c0c5a0aadeca96007852
+  data.tar.gz: f27159049e709a19879702636296d27ab31d2c40455f0c11675616a75a3c96411a017f9fefc1fec0ee460dbd4c0443a64e5f6549dbe3a48fc009f28f4d0cc551

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright 2014-2015 Puppet Labs
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,175 @@
+![Build Status](https://magnum.travis-ci.com/puppetlabs/vanagon.svg?token=A9NWBM3ogJqUCfos2gVF&branch=master)
+The Vanagon Project
+===
+ * What is vanagon?
+ * Runtime requirements
+ * Configuration and Usage
+ * Overview
+ * Contributing
+ * License
+ * Maintainers
+
+What is vanagon?
+---
+Vanagon is a tool to build a single package out of a project, which can itself
+contain one or more components. This tooling is being used to develop the
+puppet-agent package, which contains components such as openssl, ruby, and
+augeas among others. For a simple example, please see the examples directory.
+
+Vanagon builds up a Makefile and packaging files (specfile for RPM,
+control/rules/etc for DEB) and copies them to a remote host, where make can be
+invoked to build all of the components and make a package of the contents.
+
+Vanagon also provides a devkit command that will prepare a machine as a
+development environment for the entire project, or restricted to individual
+components of the project. The devkit command installs all required build tools,
+creates a master makefile for the project, and configures, builds, and installs
+all components. The result is an environment where you can work on individual
+components, then rebuild the project and test the installed artifacts.
+
+Runtime Requirements
+---
+Vanagon is self-contained. A recent version of ruby should be all that is
+required. Beyond that, ssh, rsync and git are also required on the host, and
+ssh-server and rsync is required on the target (package installation for the
+target can be customized in the platform config for the target).
+
+Configuration and Usage
+---
+Vanagon won't be much use without a project to build. Beyond that, you must
+define any platforms you want to build for. Vanagon ships with some simple
+binaries to use, but the one you probably care about is named 'build'.
+
+### `build` usage
+
+The build command has positional arguments and position independent flags.
+
+#### Arguments (position dependent)
+
+##### project name
+The name of the project to build, and a file named \<project\_name\>.rb must be
+present in configs/projects in the working directory.
+
+##### platform name
+The name of the platform to build against, and a file named
+\<platform\_name\>.rb must be present in configs/platforms in the working
+directory.
+
+Platform can also be a comma separated list of platforms such as platform1,platform2.
+
+##### target host [optional]
+Target host is an optional argument to override the host selection. Instead of using
+a vm collected from the pooler, the build will attempt to ssh to target as the
+root user.
+
+If building on multiple platforms, multiple targets can also be specified using
+a comma separated list such as host1,host2. If less targets are specified than
+platforms, the default engine (the pooler) will be used for platforms without a
+target. If more targets are specified than platforms, the extra will be ignored.
+
+#### Flagged arguments (can be anywhere in the command)
+
+##### -w DIR, --workdir DIR
+Specifies a directory where the sources should be placed and builds performed.
+Defaults to a temporary directory created with Ruby's Dir.mktmpdir.
+
+##### -c DIR, --configdir DIR
+Specifies where project configuration is found. Defaults to $pwd/configs.
+
+##### -e ENGINE, --engine ENGINE
+Choose a different virtualization engine to use to select the build target.
+Currently supported engines are:
+* `base` - Pure ssh backend; no teardown currently defined
+* `local` - Build on the local machine; platform name must match the local machine
+* `docker` - Builds in a docker container
+* `pooler` - Selects a vm from Puppet Labs' vm pooler to build on
+
+#### Flags (can be anywhere in the command)
+
+##### -p, --preserve
+Indicates that the host used for building the project should be left intact
+after the build instead of destroyed. The host is usually destroyed after a
+successful build, or left after a failed build.
+
+##### -v, --verbose (not yet implemented)
+Increase verbosity of output.
+
+##### -h, --help
+Display command-line help.
+
+#### Environment variables
+
+##### VANAGON\_SSH\_KEY
+A full path on disk for a private ssh key to be used in ssh and rsync
+communications. This will be used instead of whatever defaults are configured
+in .ssh/config.
+
+##### VMPOOLER\_TOKEN
+Used in conjunction with the pooler engine, this is a token to pass to the
+vmpooler to access the API. Without this token, the default lifetime of vms
+will be much shorter.
+
+#### Example usage
+`build --preserve puppet-agent el-6-i386` will build the puppet-agent project
+on the el-6-i386 platform and leave the host intact afterward.
+
+`build --engine=docker puppet-agent el-6-i386` will build the puppet-agent
+project on the el-6-i386 platform using the docker engine (the platform must
+have a docker\_image defined in its config).
+
+### `devkit` usage
+
+The devkit command has positional arguments and position independent flagged
+arguments.
+
+#### Arguments (position dependent)
+
+##### project name
+As in `build` arguments.
+
+##### platform name
+As in `build` arguments.
+
+##### component names [optional]
+Specifies specific components that should be built. If components are not
+specified, then all components in the project will be built. If components
+are specified as arguments, then any in the project that aren't specified
+as arguments will be retrieved from packages rather than built from source.
+
+#### Flagged arguments (can be anywhere in the command)
+
+Supports all flagged arguments from the `build` command.
+
+##### -t HOST, --target HOST
+As in the `build` target host optional argument.
+
+#### Flags (can be anywhere in the command)
+
+##### -h, --help
+Display command-line help.
+
+Contributing
+---
+We'd love to get contributions from you! Once you are up and running, take a look at the
+[Contribution Documents](CONTRIBUTING.md) to see how to get your changes merged
+in.
+
+License
+---
+See [LICENSE](LICENSE) file.
+
+Overview
+---
+Vanagon is broken down into three core ideas: the project, the component and
+the platform. The project contains one or more components and is built for a
+platform. As a quick example, if I had a ruby app and wanted to package it, the
+project would probably contain a component for ruby and a component for my app.
+If I wanted to build it for debian wheezy, I would define a platform called
+wheezy and build my project against it.
+
+For more detailed examples of the DSLs available, please see the
+[examples](examples) directory and the YARD documentation for vanagon.
+
+Maintainers
+---
+The Release Engineering team at Puppet Labs

data/bin/build

@@ -0,0 +1,33 @@
+#!/usr/bin/env ruby
+load File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "vanagon.rb"))
+
+optparse = Vanagon::OptParse.new("#{File.basename(__FILE__)} <project-name> <platform-name> [<target>] [options]",
+                                 [:workdir, :configdir, :engine, :preserve, :verbose])
+options = optparse.parse! ARGV
+
+project = ARGV[0]
+platforms = ARGV[1]
+targets = ARGV[2]
+
+if project.nil? or platforms.nil?
+  warn "project and platform are both required arguments."
+  puts optparse
+  exit 1
+end
+
+platform_list = platforms.split(',')
+if targets
+  target_list = targets.split(',')
+else
+  target_list = []
+end
+
+platform_list.zip(target_list).each do |pair|
+  platform, target = pair
+  artifact = Vanagon::Driver.new(platform, project, options.merge({ :target => target }))
+
+  artifact.verbose = true if options[:verbose]
+  artifact.preserve = true if options[:preserve]
+
+  artifact.run
+end

data/bin/devkit

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+load File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "vanagon.rb"))
+
+optparse = Vanagon::OptParse.new("#{File.basename(__FILE__)} <project-name> <platform-name> [<component-name>...] [options]",
+                                 [:workdir, :configdir, :target, :engine])
+options = optparse.parse! ARGV
+
+project = ARGV[0]
+platform = ARGV[1]
+components = ARGV.drop(2)
+
+if project.nil? or platform.nil?
+  warn "project and platform are both required arguments."
+  puts optparse
+  exit 1
+end
+
+artifact = Vanagon::Driver.new(platform, project, options.merge({ :components => components }))
+
+artifact.preserve = true
+
+artifact.prepare(options[:workdir])

data/bin/repo

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+
+ENV["PROJECT_ROOT"] = Dir.pwd
+
+# Begin warning: This ship script is an internal tool.
+# This is not intended to function outside of Puppet Labs' infrastructure.
+# This presumes packages for this ref have already been build and shipped.
+# End of warning.
+
+# repo_target will allow us to be more granular with the selection of repos that will be constructed
+# if we have not built debs or rpms, the packaging repo will fail when creating repos
+if ARGV[0]
+  repo_target = ARGV[0].downcase
+end
+
+require 'packaging'
+Pkg::Util::RakeUtils.load_packaging_tasks
+case repo_target
+when 'rpm'
+  Pkg::Util::RakeUtils.invoke_task('pl:jenkins:rpm_repos')
+when 'deb'
+  Pkg::Util::RakeUtils.invoke_task('pl:jenkins:deb_repos')
+else
+  Pkg::Util::RakeUtils.invoke_task('pl:jenkins:rpm_repos')
+  Pkg::Util::RakeUtils.invoke_task('pl:jenkins:deb_repos')
+end

data/bin/ship

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+ENV["PROJECT_ROOT"] = Dir.pwd
+
+# Begin warning: This ship script is an internal tool.
+# This is not intended to function outside of Puppet Labs' infrastructure.
+# End of warning.
+
+if Dir["output/**/*"].select { |entry| File.file?(entry) }.empty?
+  fail "No packages to ship in the output directory. Maybe you want to build some first?"
+else
+  require 'packaging'
+  Pkg::Util::RakeUtils.load_packaging_tasks
+  Pkg::Util::RakeUtils.invoke_task('pl:jenkins:ship', 'artifacts', 'output')
+end

data/lib/vanagon.rb

@@ -0,0 +1,8 @@
+LIBDIR = File.expand_path(File.dirname(__FILE__))
+VANAGON_ROOT = File.join(File.expand_path(File.dirname(__FILE__)), "..")
+
+$:.unshift(LIBDIR) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(LIBDIR)
+
+require 'vanagon/optparse'
+require 'vanagon/driver'

data/lib/vanagon/common.rb

@@ -0,0 +1,2 @@
+require 'vanagon/common/pathname'
+require 'vanagon/common/user'

data/lib/vanagon/common/pathname.rb

@@ -0,0 +1,87 @@
+class Vanagon
+  class Common
+    class Pathname
+      # @!attribute path
+      #   @return [String] Returns clean pathname of self with consecutive
+      #     slashes and useless dots removed. The filesystem is not accessed.
+      #
+      # @!attribute mode
+      #   @return [String, Integer] Returns an integer representing the
+      #     permission bits of self. The meaning of the bits is platform
+      #     dependent; on Unix systems, see stat(2).
+      #
+      # @!attribute owner
+      #   @return [String, Integer] Returns the numeric user id or string
+      #     representing the user name of the owner of self.
+      #
+      # @!attribute group
+      #   @return [String, Integer] Returns the numeric group id or string
+      #     representing the group name of the owner of self.
+      attr_accessor :path, :mode, :owner, :group
+
+      # Each Pathname requires a filesystem path, and has many optional
+      # properties that may be set at initialization time.
+      # @param [String, Integer] mode the UNIX Octal permission string to use when this file is archived
+      # @param [String, Integer] owner the username or UID to use when this file is archived
+      # @param [String, Integer] group the groupname or GID to use when this file is archived
+      # @param [Boolean] config mark this file as a configuration file, stored as private state
+      #   and exposed through the {#configfile?} method.
+      # @return [Vanagon::Common::Pathname] Returns a new Pathname instance.
+      def initialize(path, mode: nil, owner: nil, group: nil, config: false)
+        @path = File.expand_path(path)
+        @mode ||= mode
+        @owner ||= owner
+        @group ||= group
+        @config ||= config
+      end
+
+      # An alias to {Vanagon::Common::Pathname}'s constructor method,
+      #   which returns a new Vanagon::Common::Pathname, explicitly marked as a file
+      # @see Vanagon::Common::Pathname#initialize
+      #
+      # @example Create a new Vanagon::Common::Pathname, marked as a file.
+      #   Vanagon::Common::Pathname.file('/etc/puppet/puppet/puppet.conf')
+      def self.file(path, **args)
+        new(path, **args.merge!({ config: false}))
+      end
+
+      # An alias to {Vanagon::Common::Pathname}'s constructor method,
+      #   which returns a new Vanagon::Common::Pathname, explicitly marked as a configuration file
+      # @see Vanagon::Common::Pathname#initialize
+      #
+      # @example Create a new configuration file, marked as a configuration file.
+      #   Vanagon::Common::Pathname.configfile('/etc/puppet/puppet/puppet.conf')
+      def self.configfile(path, **args)
+        new(path, **args.merge!({ config: true}))
+      end
+
+      # @return [Boolean] true if a self is marked as a configuration file.
+      def configfile?
+        !!@config
+      end
+
+      # Simple test to see if any of the non-required attributes have been set in this object.
+      #
+      # @return [Boolean] whether or not mode, owner or group has been set for the object
+      def has_overrides?
+        !!(@mode || @owner || @group)
+      end
+
+      # Equality -- Two instances of Vanagon::Common::Pathname are equal if they
+      #   contain the same number attributes and if each attribute is equal to
+      #   (according to {Object#==}) the corresponding attribute in other_pathname.
+      #
+      # @return [Boolean] true if all attributes have equal values, or otherwise false.
+      def ==(other)
+        other.hash == hash
+      end
+      alias :eql? :==
+
+      # @return [Fixnum] Compute a hash-code for self, derived from its attributes;
+      #   two Pathnames with the same content will have the same hash code (and will compare using {#eql?}).
+      def hash
+        instance_variables.map { |v| instance_variable_get(v) }.hash
+      end
+    end
+  end
+end

data/lib/vanagon/common/user.rb

@@ -0,0 +1,25 @@
+class Vanagon
+  class Common
+    class User
+      attr_accessor :name, :group, :shell, :is_system, :homedir
+      def initialize(name, group = nil, shell = nil, is_system = false, homedir = nil)
+        @name = name
+        @group = group ? group : @name
+        @shell = shell if shell
+        @is_system = is_system if is_system
+        @homedir = homedir if homedir
+      end
+
+      # Equality. How does it even work?
+      #
+      # @return [true, false] true if all attributes have equal values. false otherwise.
+      def ==(other)
+        other.name == self.name && \
+          other.group == self.group && \
+          other.shell == self.shell && \
+          other.is_system == self.is_system && \
+          other.homedir == self.homedir
+      end
+    end
+  end
+end

data/lib/vanagon/component.rb

@@ -0,0 +1,157 @@
+require 'vanagon/component/source'
+require 'vanagon/component/dsl'
+
+class Vanagon
+  class Component
+    # @!attribute [r] files
+    #   @return [Set] the list of files marked for installation
+
+    attr_accessor :name, :version, :source, :url, :configure, :build, :install
+    attr_accessor :environment, :extract_with, :dirname, :build_requires
+    attr_accessor :settings, :platform, :patches, :requires, :service, :options
+    attr_accessor :directories, :replaces, :provides, :cleanup_source, :environment
+    attr_accessor :sources, :preinstall_actions, :postinstall_actions
+
+    # Loads a given component from the configdir
+    #
+    # @param name [String] the name of the component
+    # @param configdir [String] the path to the component config file
+    # @param settings [Hash] the settings to be used in the component
+    # @param platform [Vanagon::Platform] the platform to build the component for
+    # @return [Vanagon::Component] the component as specified in the component config
+    # @raise if the instance_eval on Component fails, the exception is reraised
+    def self.load_component(name, configdir, settings, platform)
+      compfile = File.join(configdir, "#{name}.rb")
+      code = File.read(compfile)
+      dsl = Vanagon::Component::DSL.new(name, settings, platform)
+      dsl.instance_eval(code, __FILE__, __LINE__)
+      dsl._component
+    rescue => e
+      puts "Error loading project '#{name}' using '#{compfile}':"
+      puts e
+      puts e.backtrace.join("\n")
+      raise e
+    end
+
+    # Component constructor.
+    #
+    # @param name [String] the name of the component
+    # @param settings [Hash] the settings to be used in the component
+    # @param platform [Vanagon::Platform] the platform to build the component for
+    # @return [Vanagon::Component] the component with the given settings and platform
+    def initialize(name, settings, platform)
+      @name = name
+      @settings = settings
+      @platform = platform
+      @options = {}
+      @build_requires = []
+      @requires = []
+      @configure = []
+      @install = []
+      @build = []
+      @patches = []
+      @files = Set.new
+      @directories = []
+      @replaces = []
+      @provides = []
+      @environment = {}
+      @sources = []
+      @preinstall_actions = []
+      @postinstall_actions = []
+    end
+
+    # Adds the given file to the list of files and returns @files.
+    #
+    # @param file [Vanagon::Common::Pathname] file to add to a component's list of files
+    # @return [Set, nil] Returns @files if file is successfully added to @files
+    #   or nil if file already exists
+    def add_file(file)
+      @files.add file
+    end
+
+    # Deletes the given file from the list of files and returns @files.
+    #
+    # @param file [String] path of file to delete from a component's list of files
+    # @return [Set, nil] Returns @files if file is successfully deleted
+    #   from @files or nil if file doesn't exist; this matches strictly on
+    #   the path of a given file, and ignores other attributes like :mode,
+    #   :owner, or :group.
+    def delete_file(file)
+      @files.delete_if { |this_file| this_file.path == file }
+    end
+
+    # Retrieve all items from @files not marked as configuration files
+    #
+    # @return [Set] all files not marked as configuration files
+    def files
+      @files.reject(&:configfile?)
+    end
+
+    # Retrieve all items from @files explicitly marked as configuration files
+    #
+    # @return [Set] all files explicitly marked as configuration files
+    def configfiles
+      @files.select(&:configfile?)
+    end
+
+    # Fetches the primary source for the component. As a side effect, also sets
+    # \@extract_with, @dirname and @version for the component for use in the
+    # makefile template
+    #
+    # @param workdir [String] working directory to put the source into
+    def get_source(workdir)
+      if @url
+        @source = Vanagon::Component::Source.source(@url, @options, workdir)
+        @source.fetch
+        @source.verify
+        @extract_with = @source.extract(@platform.tar) if @source.respond_to?(:extract)
+        @cleanup_source = @source.cleanup if @source.respond_to?(:cleanup)
+        @dirname = @source.dirname
+
+        # Git based sources probably won't set the version, so we load it if it hasn't been already set
+        @version ||= @source.version
+      else
+        warn "No source given for component '#{@name}'"
+
+        # If there is no source, we don't want to try to change directories, so we just change to the current directory.
+        @dirname = './'
+      end
+    end
+
+
+    # Fetches secondary sources for the component. These are just dumped into the workdir currently.
+    #
+    # @param workdir [String] working directory to put the source into
+    def get_sources(workdir)
+      @sources.each do |source|
+        cur_source = Vanagon::Component::Source.source(source.url, { :ref => source.ref, :sum => source.sum }, workdir)
+        cur_source.fetch
+        cur_source.verify
+      end
+    end
+
+    # Fetches patches if any are provided for the project.
+    #
+    # @param workdir [String] working directory to put the patches into
+    def get_patches(workdir)
+      unless @patches.empty?
+        patchdir = File.join(workdir, "patches")
+        FileUtils.mkdir_p(patchdir)
+        FileUtils.cp(@patches.map(&:path), patchdir)
+      end
+    end
+
+    # Prints the environment in a way suitable for use in a Makefile
+    # or shell script.
+    #
+    # @return [String] environment suitable for inclusion in a Makefile
+    def get_environment
+      if @environment.empty?
+        ":"
+      else
+        env = @environment.map { |key, value| %(#{key}="#{value}") }
+        "export #{env.join(' ')}"
+      end
+    end
+  end
+end