vanagon 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f50d333d7dc08205687fc7058b19a3580a0b5028
-  data.tar.gz: c345a2510f5d6fb4d522a680da59f2aebaa274f3
+  metadata.gz: '07341460977d9693423374862c7a7793998b4008'
+  data.tar.gz: 2a4a0e66a4fb11bdaa96d1d7fb3aa36d7908c541
 SHA512:
-  metadata.gz: 259aac01aa4832879c74a99e3578bc9d59d76e07e283729f78ebd71e4807650c508b306f7a1da13c2d7738b989cb0b40f9583ad629e0fe68ad828a35d74f421e
-  data.tar.gz: 7b72a6e2947e07cc178fd373cfc381e28f84259715bfa260362334dedda3a6e99a4fe56f1f228ead0f7bbc74d8cdcc41b25e2be9086b55e530bc3a6b30e1d5c4
+  metadata.gz: f83beaff1911b965ff21bbaf64ab2087ae9cef078c77b6b5cf3050bde80d59843dbfacb0aef31feb6788241e450e255a06c0728708d3db2152d280bcfd360592
+  data.tar.gz: 370382929914b3cdb7dd16d350c66777e5ea0ccf228507c5c5e0e2961f23a18f11f4dd6649f6118f590ec4e14d3c91f9cb9d4dda61784785bb6a3f18d39211d9

data/README.md

@@ -1,21 +1,32 @@
 ![Build Status](https://travis-ci.org/puppetlabs/vanagon.svg?branch=master)
-The Vanagon Project
-===
- * What is vanagon?
- * Runtime requirements
- * Configuration and Usage
- * Overview
- * Contributing
- * License
- * Maintainers
- * Support
-
-What is vanagon?
----
+# The Vanagon Project
+
+<!-- MarkdownTOC -->
+
+- [What is Vanagon?](#what-is-vanagon)
+- [Runtime Requirements](#runtime-requirements)
+  - [Local Host:](#local-host)
+  - [Remote Build Target:](#remote-build-target)
+- [Configuration and Usage](#configuration-and-usage)
+  - [`build` usage](#build-usage)
+  - [`inspect` usage](#inspect-usage)
+  - [`devkit` usage](#devkit-usage)
+- [Engines](#engines)
+  - [Amazon Ec2](#amazon-ec2)
+- [Contributing](#contributing)
+- [License](#license)
+- [Overview](#overview)
+- [Maintainers](#maintainers)
+
+<!-- /MarkdownTOC -->
+
+
+## What is Vanagon?
+
 Vanagon is a tool to build a single package out of a project, which can itself
 contain one or more components. This tooling is being used to develop the
 puppet-agent package, which contains components such as openssl, ruby, and
-augeas among others. For a simple example, please see the examples directory.
+augeas among others. For a simple example, please see the project in the `examples/` directory.
 
 Vanagon builds up a Makefile and packaging files (specfile for RPM,
 control/rules/etc for DEB) and copies them to a remote host, where make can be
@@ -28,15 +39,37 @@ creates a master makefile for the project, and configures, builds, and installs
 all components. The result is an environment where you can work on individual
 components, then rebuild the project and test the installed artifacts.
 
-Runtime Requirements
----
-Vanagon is self-contained. A recent version of ruby (2.1 or greater) should be
-all that is required. Beyond that, ssh, rsync and git are also required on the
-host, and ssh-server and rsync is required on the target (package installation
-for the target can be customized in the platform config for the target).
+## Runtime Requirements
+
+Vanagon carries two sets of requirements: requirements for the local host where Vanagon is run, and for the remote target where compilation and packaging happens.
+
+Also, Vanagon ships with a number of engines which may include additional optional dependencies if you wish to use them. These engines are currently considered experimental, and receive less attention than the `Hardware` or `vmpooler` engines do. If you find a bug in these engines, [please open a ticket](https://tickets.puppetlabs.com/browse/PA) and let us know.
+
+### Local Host:
+
+- [Ruby](https://www.ruby-lang.org/en/) (Ruby 2.1.x is the miniumum supported version)
+- [fustigit](https://github.com/mckern/fustigit)
+- [ruby-git](https://github.com/schacon/ruby-git)
+- The command line tool `ssh` ([homepage](https://www.openssh.com/)) available on the local `${PATH}` (any modern version should suffice)
+- The command line tool `rsync` ([homepage](https://rsync.samba.org/)) available on the local `${PATH}` (At least rsync 2.6.x)
+- The command line tool `git` ([homepage](https://git-scm.com/)) available on the local `${PATH}` (Vanagon is tested against Git version 1.8.x but should work with any newer version)
+
+#### Optional requirements
+
+- [AWS SDK for Ruby](https://github.com/aws/aws-sdk-ruby), if you're using the EC2 engine
+- [Docker](https://www.docker.com/community-edition), if you're using the Docker engine
+
+### Remote Build Target:
+
+**Note:** package installation & builder configuration for the remote target can be customized in the `Platform` configuration that defines target provisioning instructions.
+
+- GNU Make ([homepage](https://www.gnu.org/software/make/)) (Vanagon specifically targets the feature set provided by [GNU Make 3.81](http://git.savannah.gnu.org/cgit/make.git/tree/NEWS?h=3.81&id=776d8b7bc2ff83f8ebf5d357ec89e3bbe6d83962) but newer versions are known to work -- older versions are specifically known to **not** work!)
+- Bash ([homepage](https://www.gnu.org/software/bash/)) is required by the Makefiles that Vanagon generates
+- An ssh server ([homepage](https://www.openssh.com/)) is required by most engines
+- The command line tool `rsync` ([homepage](https://rsync.samba.org/)) (At least rsync 2.6.x)
+
+## Configuration and Usage
 
-Configuration and Usage
----
 Vanagon won't be much use without a project to build. Beyond that, you must
 define any platforms you want to build for. Vanagon ships with some simple
 binaries to use, but the one you probably care about is named 'build'.
@@ -44,7 +77,6 @@ binaries to use, but the one you probably care about is named 'build'.
 ### `build` usage
 The build command has positional arguments and position independent flags.
 
-
 #### Arguments (position dependent)
 
 ##### project name
@@ -57,7 +89,7 @@ The name of the target platform to build `<project_name>` against; a file named
 directory. This can also be a comma separated list of platforms such as `platform1,platform2`;
 note that there are no spaces after the comma.
 
-##### target host [optional]
+##### target host <optional>
 Target host is an optional argument to override the host selection. Instead of using
 a random VM collected from the pooler (Vanagon's default build engine), the build will
 attempt connect to the target host over SSH as the `root` user.
@@ -70,13 +102,15 @@ than platforms, the extra platforms will be ignored.
 
 Build machines should be cleaned between builds.
 
-#### Flagged arguments (can be anywhere in the command)
+#### Flagged arguments
+
+**Note:** command flags can be used anywhere in the command.
 
 ##### -w DIR, --workdir DIR
 Specifies a directory on the local host where the sources should be placed and builds performed.
 Defaults to a temporary directory created with Ruby's `Dir.mktmpdir` method.
 
-##### --remote_dir
+##### -r DIR, --remote-workdir DIR
 Explicitly specify a directory on the remote target to place sources and perform
 builds. Components can then be rebuilt manually on the build host for faster iteration. Sources may not be correctly updated if this directory already exists.
 Defaults to a temporary directory created by running `mktemp -d` on the remote target.
@@ -94,15 +128,17 @@ Currently supported engines are:
 * `hardware` - Build on a specific taget and lock it in redis
 * `ec2` - Build on a specific AWS instance.
 
-#### Flags (can be anywhere in the command)
+#### Flags
+
+**Note:** command flags can be used anywhere in the command.
 
 ##### -p, --preserve
 Indicates that the host used for building the project should be left intact
 after the build instead of destroyed. The host is usually destroyed after a
 successful build, or left after a failed build.
 
-##### -v, --verbose (not yet implemented)
-Increase verbosity of output.
+##### -v, --verbose
+(Reserved for future implementation) Will increase the verbosity of output, when implemented.
 
 ##### -h, --help
 Display command-line help.
@@ -172,13 +208,15 @@ directory.
 
 Platform can also be a comma separated list of platforms such as platform1,platform2.
 
-#### Flagged arguments (can be anywhere in the command)
+#### Flagged arguments
+
+**Note:** command flags can be used anywhere in the command.
 
 ##### -w DIR, --workdir DIR
 Specifies a directory where the sources should be placed and builds performed.
 Defaults to a temporary directory created with Ruby's Dir.mktmpdir.
 
-##### -c DIR, --configdir DIR
+##### -c DIR, --configdir DIR <optional>
 Specifies where project configuration is found. Defaults to $pwd/configs.
 
 ##### -e ENGINE, --engine ENGINE
@@ -188,7 +226,9 @@ rendered -- the `inspect` command performs no compilation.
 
 Supported engines are the same as the `build` command.
 
-#### Flags (can be anywhere in the command)
+#### Flags
+
+**Note:** command flags can be used anywhere in the command.
 
 ##### -v, --verbose (not yet implemented)
 Increase verbosity of output.
@@ -223,20 +263,24 @@ As in `build` arguments.
 ##### platform name
 As in `build` arguments.
 
-##### component names [optional]
+##### component names <optional>
 Specifies specific components that should be built. If components are not
 specified, then all components in the project will be built. If components
 are specified as arguments, then any in the project that aren't specified
 as arguments will be retrieved from packages rather than built from source.
 
-#### Flagged arguments (can be anywhere in the command)
+#### Flagged arguments
+
+**Note:** command flags can be used anywhere in the command.
 
 Supports all flagged arguments from the `build` command.
 
 ##### -t HOST, --target HOST
 As in the `build` target host optional argument.
 
-#### Flags (can be anywhere in the command)
+#### Flags
+
+**Note:** command flags can be used anywhere in the command.
 
 ##### -h, --help
 Display command-line help.
@@ -248,7 +292,7 @@ Engines
 
 ### Amazon Ec2
 
-Note: If you have the `aws_ami` setup vanagon will default to the ec2 engine.
+Note: If you have the `aws_ami` setup Vanagon will default to the ec2 engine.
 
 To use the ec2 engine you should have your credentials set either via your `~/.aws/credentials` or environment variables.
 After this you can setup your `configs/platforms/<platform>.rb` to use your
@@ -294,7 +338,7 @@ If I wanted to build it for debian wheezy, I would define a platform called
 wheezy and build my project against it.
 
 For more detailed examples of the DSLs available, please see the
-[examples](https://github.com/puppetlabs/vanagon/tree/master/examples) directory and the YARD documentation for vanagon.
+[examples](https://github.com/puppetlabs/vanagon/tree/master/examples) directory and the YARD documentation for Vanagon.
 
 ## Maintainers
 See MAINTAINERS file.

data/bin/build

@@ -2,7 +2,7 @@
 load File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "vanagon.rb"))
 
 optparse = Vanagon::OptParse.new("#{File.basename(__FILE__)} <project-name> <platform-name> [<target>] [options]",
-                                 [:workdir, :configdir, :engine, :preserve, :verbose, :skipcheck, :only_build, :remote_workdir])
+                                 [:workdir, :configdir, :engine, :preserve, :verbose, :skipcheck, :only_build, :"remote-workdir"])
 options = optparse.parse! ARGV
 
 project = ARGV[0]

data/lib/makefile.rb

@@ -1,5 +1,3 @@
-require 'vanagon/environment'
-
 class Makefile
   # The Rule class defines a single Makefile rule.
   #
@@ -13,11 +11,6 @@ class Makefile
     #   @return [Array<String>] A list of dependencies that this rule depends on.
     attr_accessor :dependencies
 
-    # @!attribute [rw] environment
-    #   @return [Array<String>] A list of environment variables that this rule
-    #     will export
-    attr_accessor :environment
-
     # @!attribute [rw] recipe
     #   @return [Array<String>] A list of commands to execute upon invocation of this rule.
     attr_accessor :recipe
@@ -43,10 +36,9 @@ class Makefile
     #         "make cpplint",
     #       ]
     #     end
-    def initialize(target, dependencies: [], environment: Vanagon::Environment.new, recipe: [], &block)
+    def initialize(target, dependencies: [], recipe: [], &block)
       @target = target
       @dependencies = dependencies
-      @environment = environment
       @recipe = recipe
 
       yield(self) if block
@@ -65,25 +57,11 @@ class Makefile
       ["#{target}:", dependencies].flatten.compact.join("\s")
     end
 
-    # @return [String] the Makefile target's name, rendered in a format
-    # suitable for using as a Graphite group -- any periods in the name of
-    # the component being built will be removed.
-    #   e.g. "ruby-2.1.9-unpack" will become "ruby-219.unpack"
-    def tokenize_target_name
-      target_name, _, rule = target.rpartition('-')
-      [target_name.tr('.', ''), rule]
-        .select { |s| !(s.nil? || s.empty?) }
-        .join('.')
-    end
-
-    def tokenized_environment_variable
-      "#{target}: export VANAGON_TARGET := #{tokenize_target_name}"
-    end
-
-    def environment_variables
-      environment.map { |k, v| "#{k} := #{v}" }.map do |env|
-        "#{target}: export #{env}"
-      end
+    def compounded_recipe
+      Array(recipe)
+        .compact
+        .map { |line| "\t" + line.gsub("\n", "\n\t") + "\n" }
+        .join
     end
 
     # Format this rule as a Makefile rule.
@@ -92,37 +70,9 @@ class Makefile
     # newline to ensure that the recipe is parsed as part of a single makefile rule.
     #
     # @return [String]
-    def format # rubocop:disable Metrics/AbcSize
-      # create a base target inside an Array, and construct the rest of
-      # the rule around that.
-      t = [base_target]
-
-      # prepend an environment variable that can be used inside a
-      # given Make rule/target. We have to do it this way instead of
-      # appending it to #environment because for reasons that I cannot
-      # work out, the "sane" way results in previous/incorrect names
-      # being used and objects being recycled. My working theory is
-      # a corner case between metaprogrammed methods in Component::Rules,
-      # and Ruby's preference for pass-by-reference.
-      # Ryan McKern 2017-02-02
-      t.unshift tokenized_environment_variable
-
-      # prepend any environment variables to the existing target,
-      # using the target prefix to identify them as such. they should
-      # end up ahead of the dependencies and the build recipe.
-      environment_variables.each do |env|
-        t.unshift env
-      end
-
-      # finally, append the build recipe after the base target condition.
-      # also, here's a fun edge case: if one were to call #squeeze on
-      # the iterator 'line', basically all of the phony make tasks that
-      # `touch` a file just disapear. Fragility ++.
-      # - Ryan McKern 2017-02-02
-      t.push recipe.compact.map { |line| "\t" + line.gsub("\n", "\n\t") + "\n" }.join
-      t.join("\n")
+    def format
+      [base_target, compounded_recipe].flatten.join("\n")
     end
-
     alias to_s format
   end
 end

data/lib/vanagon/component.rb

@@ -29,7 +29,8 @@ class Vanagon
     # that they were cloned to. For the outlying flat files, it'll
     # end up being defined explicitly as the string './'
     attr_accessor :dirname
-    # what special tool should be used to extract the primary source
+    # The specific tools or command line invocations that
+    # should be used to extract a given component's sources
     attr_accessor :extract_with
     # how should this component be configured?
     attr_accessor :configure
@@ -190,7 +191,7 @@ class Vanagon
     end
 
     # Fetches the primary source for the component. As a side effect, also sets
-    # \@extract_with, @dirname and @version for the component for use in the
+    # @extract_with, @dirname and @version for the component for use in the
     # makefile template
     #
     # @param workdir [String] working directory to put the source into
@@ -200,7 +201,8 @@ class Vanagon
         @source = Vanagon::Component::Source.source(url, opts)
         source.fetch
         source.verify
-        @extract_with = source.respond_to?(:extract) ? source.extract(platform.tar) : nil
+        extract_with << source.extract(platform.tar) if source.respond_to? :extract
+
         @cleanup_source = source.cleanup if source.respond_to?(:cleanup)
         @dirname = source.dirname
 
@@ -215,7 +217,7 @@ class Vanagon
         @dirname = './'
 
         # If there is no source, there is nothing to do to extract
-        @extract_with = ': no source, so nothing to extract'
+        extract_with << ': no source, so nothing to extract'
       end
     end
 
@@ -231,16 +233,25 @@ class Vanagon
     # Fetches secondary sources for the component. These are just dumped into the workdir currently.
     #
     # @param workdir [String] working directory to put the source into
-    def get_sources(workdir)
+    def get_sources(workdir) # rubocop:disable Metrics/AbcSize
       sources.each do |source|
         src = Vanagon::Component::Source.source(
           source.url, workdir: workdir, ref: source.ref, sum: source.sum
         )
         src.fetch
         src.verify
+        # set src.file to only be populated with the basename instead of entire file path
+        src.file = File.basename(src.url)
+        extract_with << src.extract(platform.tar) if src.respond_to? :extract
       end
     end
 
+    # @return [Array] the specific tool or command line invocations that
+    #   should be used to extract a given component's primary source
+    def extract_with
+      @extract_with ||= []
+    end
+
     # Fetches patches if any are provided for the project.
     #
     # @param workdir [String] working directory to put the patches into
@@ -265,14 +276,17 @@ class Vanagon
         The #get_environment method will be removed by Vanagon 1.0.0.
       eos
 
-      if @environment.empty?
+      if environment.empty?
         ": no environment variables defined"
       else
-        env = @environment.map { |key, value| %(#{key}="#{value}") }
-        "export #{env.join(' ')}"
+        environment_variables
       end
     end
 
+    def environment_variables
+      environment.map { |key, value| %(export #{key}="#{value}") }
+    end
+
     def rules(project, platform)
       Vanagon::Component::Rules.new(self, project, platform)
     end

data/lib/vanagon/component/dsl.rb

@@ -230,14 +230,14 @@ class Vanagon
       # upgrade if it has been modified
       #
       # @param file [String] name of the configfile
-      def configfile(file)
+      def configfile(file, mode: nil, owner: nil, group: nil)
         # I AM SO SORRY
         @component.delete_file file
         if @component.platform.name =~ /solaris-10|osx/
           @component.install << "mv '#{file}' '#{file}.pristine'"
-          @component.add_file Vanagon::Common::Pathname.configfile("#{file}.pristine")
+          @component.add_file Vanagon::Common::Pathname.configfile("#{file}.pristine", mode: mode, owner: owner, group: group)
         else
-          @component.add_file Vanagon::Common::Pathname.configfile(file)
+          @component.add_file Vanagon::Common::Pathname.configfile(file, mode: mode, owner: owner, group: group)
         end
       end
 
@@ -245,9 +245,9 @@ class Vanagon
       #
       # @param source [String] path to the configfile to copy
       # @param target [String] path to the desired target of the configfile
-      def install_configfile(source, target)
-        install_file(source, target)
-        configfile(target)
+      def install_configfile(source, target, mode: '0644', owner: nil, group: nil)
+        install_file(source, target, mode: mode, owner: owner, group: group)
+        configfile(target, mode: mode, owner: owner, group: group)
       end
 
       # link will add a command to the install to create a symlink from source to target
@@ -350,7 +350,7 @@ class Vanagon
           eos
           return @component.environment.merge!(env.first)
         elsif env.size == 2
-          return @component.environment[env.key] = env.value
+          return @component.environment[env.first] = env.last
         end
         raise ArgumentError, <<-eos.undent
           component DSL method #environment only accepts a single Hash (deprecated)