vanagon 0.6.1 → 0.6.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0e69b827c87e42df5740b868926ede94d03ffe23
-  data.tar.gz: 48063c4f0fff9db3c8c40b8eaf55930d930ff516
+  metadata.gz: f994fbe98407e4e849e71d44ad56a8f6a0ce4946
+  data.tar.gz: 03f24c4388823870073e63f32a653701658cd0aa
 SHA512:
-  metadata.gz: ba1e2146ea55742e934e53b59807042e569a048d9a8f215f7ee3acbeb9026772659d836cefb0ffb15924c13085c2a676cfe25b3d41200c92826d34cda7b46b70
-  data.tar.gz: dc76822ffe4ea4bcf888ead30e16de669ea021211a6b0fe9d11bbda772f2f35887947a4e3e932f994ffd97e0ec8d62bc00eded1fe02362c852bf5cb0fc2566b1
+  metadata.gz: a65988af487b4e29e440afaab76e67eb5dec9fc231a87ff86688227d9104b7e2a6c1089da1daac5f03692181758f595ecb705686faf271245a8dbd66fa23e7a6
+  data.tar.gz: fffaf35d8867478976545d26c0ccdfd4b3dc55edcd7e7817a4ac9f4ef4b23ef4513bfc173ebf7bf1da8ae8725abf17fc6955ba283f8655db86e05899dd608af6

data/README.md

@@ -42,31 +42,32 @@ define any platforms you want to build for. Vanagon ships with some simple
 binaries to use, but the one you probably care about is named 'build'.
 
 ### `build` usage
-
 The build command has positional arguments and position independent flags.
 
+
 #### Arguments (position dependent)
 
 ##### project name
-The name of the project to build, and a file named \<project\_name\>.rb must be
-present in configs/projects in the working directory.
+The name of the project to build; a file named `<project_name>.rb` must be
+present under `configs/projects` in the working directory.
 
 ##### platform name
-The name of the platform to build against, and a file named
-\<platform\_name\>.rb must be present in configs/platforms in the working
-directory.
-
-Platform can also be a comma separated list of platforms such as platform1,platform2.
+The name of the target platform to build `<project_name>` against; a file named
+`<platform_name>.rb` must be present under `configs/platforms` in the working
+directory. This can also be a comma separated list of platforms such as `platform1,platform2`;
+note that there are no spaces after the comma.
 
 ##### target host [optional]
 Target host is an optional argument to override the host selection. Instead of using
-a vm collected from the pooler, the build will attempt to ssh to target as the
-root user.
+a random VM collected from the pooler (Vanagon's default build engine), the build will
+attempt connect to the target host over SSH as the `root` user.
 
 If building on multiple platforms, multiple targets can also be specified using
-a comma separated list such as host1,host2. If less targets are specified than
-platforms, the default engine (the pooler) will be used for platforms without a
-target. If more targets are specified than platforms, the extra will be ignored.
+a comma separated list such as `host1,host2` (note that there are no spaces after
+the comma). If less targets are specified than platforms, the default engine
+(`pooler`) will be used for platforms without a target. If more targets are specified
+than platforms, the extra platforms will be ignored.
+
 
 #### Flagged arguments (can be anywhere in the command)
 
@@ -85,6 +86,7 @@ Currently supported engines are:
 * `docker` - Builds in a docker container
 * `pooler` - Selects a vm from Puppet Labs' vm pooler to build on
 * `hardware` - Build on a specific taget and lock it in redis
+* `ec2` - Build on a specific AWS instance.
 
 #### Flags (can be anywhere in the command)
 
@@ -99,28 +101,40 @@ Increase verbosity of output.
 ##### -h, --help
 Display command-line help.
 
+
 #### Environment variables
 
-##### VANAGON\_SSH\_KEY
+##### `VANAGON_SSH_KEY`
 A full path on disk for a private ssh key to be used in ssh and rsync
 communications. This will be used instead of whatever defaults are configured
 in .ssh/config.
 
-##### VANAGON\_SSH\_AGENT
-When set, Vanagon will forward the ssh authentication agent connection. 
+##### `VANAGON_SSH_AGENT`
+When set, Vanagon will forward the ssh authentication agent connection.
 
-##### VMPOOLER\_TOKEN
+##### `VMPOOLER_TOKEN`
 Used in conjunction with the pooler engine, this is a token to pass to the
 vmpooler to access the API. Without this token, the default lifetime of vms
 will be much shorter.
 
-##### LOCK\_MANAGER\_HOST
+##### `LOCK_MANAGER_HOST`
 The name of the host where redis is running. Redis is used to handle a lock
 when using the hardware engine. It defaults to *redis*, with no domain.
 
-#### LOCK\_MANAGER\_PORT
+##### `LOCK_MANAGER_PORT`
 Port of the system where redis is running. Defaults to *6379*.
 
+##### `VANAGON_RETRY_COUNT`
+Some phases of compilation support retries. The default value is *1* but
+setting to any integer value greater than 1 will causes these components
+to retry operations on failure until the `VANAGON_RETRY_COUNT` limit is reached.
+
+##### `VANAGON_TIMEOUT`
+Some phases of compilation can take an indeterminate (but substantial) amount of
+time. The default value is *7200* seconds(120 minutes) but setting to any
+integer value these components to fail after the `VANAGON_TIMEOUT` count is reached.
+Note that this value is expected to be in seconds.
+
 #### Example usage
 `build --preserve puppet-agent el-6-i386` will build the puppet-agent project
 on the el-6-i386 platform and leave the host intact afterward.
@@ -129,6 +143,67 @@ on the el-6-i386 platform and leave the host intact afterward.
 project on the el-6-i386 platform using the docker engine (the platform must
 have a docker\_image defined in its config).
 
+---
+
+### `inspect` usage
+
+The `inspect` command has positional arguments and position independent flags. It
+mirrors the `build` command, but exits with success after loading and interpolating
+all of the components in the given project. No attempt is made to actually build
+the given project; instead, a JSON formatted array of hashes is returned and printed
+to `stdout`. This JSON array can be further processed by external tooling, such as `jq`.
+
+#### Arguments (position dependent)
+
+##### project name
+The name of the project to build, and a file named \<project\_name\>.rb must be
+present in configs/projects in the working directory.
+
+##### platform name
+The name of the platform to build against, and a file named
+\<platform\_name\>.rb must be present in configs/platforms in the working
+directory.
+
+Platform can also be a comma separated list of platforms such as platform1,platform2.
+
+#### Flagged arguments (can be anywhere in the command)
+
+##### -w DIR, --workdir DIR
+Specifies a directory where the sources should be placed and builds performed.
+Defaults to a temporary directory created with Ruby's Dir.mktmpdir.
+
+##### -c DIR, --configdir DIR
+Specifies where project configuration is found. Defaults to $pwd/configs.
+
+##### -e ENGINE, --engine ENGINE
+Choose a different virtualization engine to use to select the build target.
+Engines are respected, but only insofar as components and projects are
+rendered -- the `inspect` command performs no compilation.
+
+Supported engines are the same as the `build` command.
+
+#### Flags (can be anywhere in the command)
+
+##### -v, --verbose (not yet implemented)
+Increase verbosity of output.
+
+##### -h, --help
+Display command-line help.
+
+#### Environment variables
+
+Environment variables are respected, but only insofar as components and projects are
+rendered -- the `inspect` command has no behavior to alter.
+
+Supported environment variables are the same as the `build` command.
+
+#### Example usage
+`inspect puppet-agent el-6-i386` will load the puppet-agent project
+on the el-6-i386 platform and print the resulting list of dependencies,
+build-time configuration, environment variables, and expected artifacts.
+
+---
+
 ### `devkit` usage
 
 The devkit command has positional arguments and position independent flagged
@@ -160,10 +235,43 @@ As in the `build` target host optional argument.
 ##### -h, --help
 Display command-line help.
 
+---
+
+Engines
+---
+
+### Amazon Ec2
+
+Note: If you have the `aws_ami` setup vanagon will default to the ec2 engine.
+
+To use the ec2 engine you should have your credentials set either via your `~/.aws/credentials` or environment variables.
+After this you can setup your `configs/platforms/<platform>.rb` to use your
+ami, instance type, and key_name to setup the instance.
+
+A simple one looks like this
+
+```ruby
+# configs/platforms/el-7-x86_64.rb
+platform "el-7-x86_64" do |plat|
+    plat.aws_ami "your-ami-id-here" # You must set this
+    plat.aws_instance_type "t2.small" # Defaults to t1.micro
+    plat.aws_key_name "vanagon" # this is the default but you can use whichever
+    plat.aws_user_data <<-eos
+#cloud-config
+    runcmds:
+        - echo #{my_ssh_key} > /root/.ssh/authorized_keys # Most amis block you from logging in as root.
+    eos
+
+### Rest of your code here
+
+end
+```
+
+
 Contributing
 ---
 We'd love to get contributions from you! Once you are up and running, take a look at the
-[Contribution Documents](https://github.com/puppetlabs/vanagon/blob/master/CONTRIBUTING.md) to see how to get your changes merged
+[Contribution Documents](https://github.com/puppetlabs/vanagon/blob/master/docs/CONTRIBUTING.md) to see how to get your changes merged
 in.
 
 License
@@ -183,11 +291,10 @@ For more detailed examples of the DSLs available, please see the
 [examples](https://github.com/puppetlabs/vanagon/tree/master/examples) directory and the YARD documentation for vanagon.
 
 ## Maintainers
----
 The Release Engineering team at Puppet Labs
 
-Maintainer: Michael Stahnke <stahnma@puppetlabs.com>
+Maintainer: Michael Stahnke <stahnma@puppet.com>
 
-Please log tickets and issues at our [Issue Tracker](https://tickets.puppetlabs.com/browse/CPR). Set compononent to Vanagon.
+Please log tickets and issues at our [Issue Tracker](https://tickets.puppet.com/browse/CPR). Set compononent to Vanagon.
 
 In addition there is an active #puppet-dev channel on Freenode.

data/bin/build

@@ -25,9 +25,5 @@ end
 platform_list.zip(target_list).each do |pair|
   platform, target = pair
   artifact = Vanagon::Driver.new(platform, project, options.merge({ :target => target }))
-
-  artifact.verbose = true if options[:verbose]
-  artifact.preserve = true if options[:preserve]
-
   artifact.run
 end

data/bin/build_host_info

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+require 'json'
+
+load File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "vanagon.rb"))
+
+optparse = Vanagon::OptParse.new("#{File.basename(__FILE__)} <project-name> <platform-name> [options]", [:workdir, :configdir, :engine])
+options = optparse.parse! ARGV
+
+project = ARGV[0]
+platforms = ARGV[1]
+
+if project.nil? or platforms.nil?
+  warn "project and platform are both required arguments."
+  puts optparse
+  exit 1
+end
+
+platforms.split(',').each do |platform|
+  driver = Vanagon::Driver.new(platform, project, options)
+  puts JSON.generate(driver.build_host_info)
+end

data/bin/inspect

@@ -0,0 +1,25 @@
+#!/usr/bin/env ruby
+require 'json'
+require 'vanagon/extensions/ostruct/json'
+require 'vanagon/extensions/set/json'
+require 'vanagon/extensions/hashable'
+
+load File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "vanagon.rb"))
+
+optparse = Vanagon::OptParse.new("#{File.basename(__FILE__)} <project-name> <platform-name> [options]", [:workdir, :configdir, :engine])
+options = optparse.parse! ARGV
+
+project = ARGV[0]
+platforms = ARGV[1]
+
+unless project or platforms
+  warn "project and platform are both required arguments."
+  puts optparse
+  exit 1
+end
+
+platforms.split(',').each do |platform|
+  driver = Vanagon::Driver.new(platform, project, options)
+  components = driver.project.components.map(&:to_hash)
+  puts JSON.pretty_generate(components)
+end

data/lib/vanagon/driver.rb

@@ -13,9 +13,9 @@ class Vanagon
     attr_accessor :platform, :project, :target, :workdir, :verbose, :preserve
     attr_accessor :timeout, :retry_count
 
-    def initialize(platform, project, options = { :configdir => nil, :target => nil, :engine => nil, :components => nil, :skipcheck => false })
-      @verbose = false
-      @preserve = false
+    def initialize(platform, project, options = { :configdir => nil, :target => nil, :engine => nil, :components => nil, :skipcheck => false, :verbose => false, :preserve => false })
+      @verbose = options[:verbose]
+      @preserve = options[:preserve]
 
       @@configdir = options[:configdir] || File.join(Dir.pwd, "configs")
       components = options[:components] || []
@@ -24,6 +24,7 @@ class Vanagon
 
       @platform = Vanagon::Platform.load_platform(platform, File.join(@@configdir, "platforms"))
       @project = Vanagon::Project.load_project(project, File.join(@@configdir, "projects"), @platform, components)
+      @project.settings[:verbose] = options[:verbose]
       @project.settings[:skipcheck] = options[:skipcheck]
       loginit('vanagon_hosts.log')
 
@@ -35,6 +36,10 @@ class Vanagon
     def load_engine(engine_type, platform, target)
       if platform.build_hosts
         engine_type = 'hardware'
+      elsif platform.aws_ami
+        engine_type = 'ec2'
+      elsif platform.docker_image
+        engine_type = 'docker'
       elsif target
         engine_type = 'base'
       end
@@ -60,6 +65,10 @@ class Vanagon
       @@logger
     end
 
+    def build_host_info
+      { "name" => @engine.build_host_name, "engine" => @engine.name }
+    end
+
     # Returns the set difference between the build_requires and the components to get a list of external dependencies that need to be installed.
     def list_build_dependencies
       @project.components.map(&:build_requires).flatten.uniq - @project.components.map(&:name)
@@ -87,12 +96,12 @@ class Vanagon
 
       puts "Target is #{@engine.target}"
       retry_task { install_build_dependencies }
-      @project.fetch_sources(@workdir)
+      retry_task { @project.fetch_sources(@workdir) }
       @project.make_makefile(@workdir)
       @project.make_bill_of_materials(@workdir)
       @project.generate_packaging_artifacts(@workdir)
       @engine.ship_workdir(@workdir)
-      retry_task { @engine.dispatch("(cd #{@engine.remote_workdir}; #{@platform.make})") }
+      @engine.dispatch("(cd #{@engine.remote_workdir}; #{@platform.make})")
       @engine.retrieve_built_artifact
       @engine.teardown unless @preserve
       cleanup_workdir unless @preserve
@@ -101,7 +110,7 @@ class Vanagon
       puts e.backtrace.join("\n")
       raise e
     ensure
-      if @engine.name == "hardware"
+      if ["hardware", "ec2"].include?(@engine.name)
         @engine.teardown
       end
     end
@@ -129,8 +138,8 @@ class Vanagon
     # values from the project, if available, otherwise use some
     # sane defaults.
     def retry_task(&block)
-      @timeout = @project.timeout || 3600
-      @retry_count = @project.retry_count || 3
+      @timeout = @project.timeout || ENV["VANAGON_TIMEOUT"] || 7200
+      @retry_count = @project.retry_count || ENV["VANAGON_RETRY_COUNT"] || 1
       Vanagon::Utilities.retry_with_timeout(@retry_count, @timeout) { yield }
     end
     private :retry_task

data/lib/vanagon/engine/base.rb

@@ -4,14 +4,23 @@ require 'vanagon/errors'
 class Vanagon
   class Engine
     class Base
-      attr_accessor :target, :remote_workdir, :name
+      attr_accessor :target, :remote_workdir
 
       def initialize(platform, target = nil)
         @platform = platform
         @required_attributes = ["ssh_port"]
         @target = target if target
         @target_user = @platform.target_user
-        @name = 'base'
+      end
+
+      # Get the engine name
+      def name
+        'base'
+      end
+
+      # Get the engine specific name of the host to build on
+      def build_host_name
+        raise Vanagon::Error, '#build_host_name has not been implemented for your engine.'
       end
 
       # This method is used to obtain a vm to build upon

data/lib/vanagon/engine/docker.rb

@@ -6,17 +6,32 @@ class Vanagon
       # Both the docker_image and the docker command itself are required for
       # the docker engine to work
       def initialize(platform, target = nil)
-        @docker_cmd = Vanagon::Utilities.find_program_on_path('docker')
-        @name = 'docker'
         super
+
+        @docker_cmd = Vanagon::Utilities.find_program_on_path('docker')
         @required_attributes << "docker_image"
       end
 
+      # Get the engine name
+      def name
+        'docker'
+      end
+
+      # Return the docker image name to build on
+      def build_host_name
+        if @build_host_name.nil?
+          validate_platform
+          @build_host_name = @platform.docker_image
+        end
+
+        @build_host_name
+      end
+
       # This method is used to obtain a vm to build upon using
       # a docker container.
       # @raise [Vanagon::Error] if a target cannot be obtained
       def select_target
-        Vanagon::Utilities.ex("#{@docker_cmd} run -d --name #{@platform.docker_image}-builder -p #{@platform.ssh_port}:22 #{@platform.docker_image}")
+        Vanagon::Utilities.ex("#{@docker_cmd} run -d --name #{build_host_name}-builder -p #{@platform.ssh_port}:22 #{build_host_name}")
         @target = 'localhost'
 
         # Wait for ssh to come up in the container
@@ -30,10 +45,10 @@ class Vanagon
       # This method is used to tell the vmpooler to delete the instance of the
       # vm that was being used so the pool can be replenished.
       def teardown
-        Vanagon::Utilities.ex("#{@docker_cmd} stop #{@platform.docker_image}-builder")
-        Vanagon::Utilities.ex("#{@docker_cmd} rm #{@platform.docker_image}-builder")
+        Vanagon::Utilities.ex("#{@docker_cmd} stop #{build_host_name}-builder")
+        Vanagon::Utilities.ex("#{@docker_cmd} rm #{build_host_name}-builder")
       rescue Vanagon::Error => e
-        warn "There was a problem tearing down the docker container #{@platform.docker_image}-builder (#{e.message})."
+        warn "There was a problem tearing down the docker container #{build_host_name}-builder (#{e.message})."
       end
     end
   end