builderator 0.3.15 → 1.0.0.pre.rc.1

data/README.md

@@ -1,30 +1,84 @@
 # Builderator
-__Tools to make CI Packer builds awesome__
 
-## Installation
+Orchestration and configuration of the code development life-cycle.
 
-Add this line to your application's Gemfile:
+## Commands
 
-```ruby
-gem 'builderator'
-```
+### `local [PROFILE = default]`
+
+Provision a local VM using Vagrant and, by default, VirtualBox. Uses Berkshelf to fetch cookbooks, and Chef to provision the VM.
+
+### `ec2 [PROFILE = default]`
+
+Provision an EC2 VM using Vagrant. Same workflow as `local` using the `vagrant-aws` plugin.
+
+### `release [PROFILE = default]`
+
+Perform release tasks and execute Packer builds with released artifacts.
+
+## Configuration
+
+Configuration can be loaded from DSL files as well as JSON and command line arguments. By default, Builderator searches in your home directory (`$HOME/.builderator/Buildfile`) and the working directory (`./Builderator`) for DSL files. Configuration sources are layered and flattened into a single DSL in the following order:
+
+* Global defaults defined in the Builderator sources
+* `Config.defaults` set by plugins, tasks, etc. in code
+* `$HOME/.builderator/Buildfile`
+* `./Buildfile`
+* `Config.overrides` set by plugins, tasks, etc. in code
+* CLI arguments loaded from Thor
+
+[Additional documentation](docs/configuration.md) describes the configuration DSL interface.
+
+## Integrations
 
-And then execute:
+Builderator integrates with other tools, including [Berkshelf](http://berkshelf.com), [Vagrant](https://www.vagrantup.com/), and [Packer](https://www.packer.io/), to orchestrate workflows by generating `Berksfile`s, `Vagrantfile`s, and JSON strings for Packer. This means that you can replace all of these files in your project with a single `Buildfile`.
 
-    $ bundle
+### Packer
 
-Or install it yourself as:
+The Packer integration generates Packer JSON and passes it to STDIN of `packer build -`.
 
-    $ gem install builderator
+    *NOTE* Currently, we assume that you're building Ubuntu images, as one of the provisioners is hard-coded to chown the Chef data directories to `ubuntu:ubuntu`
 
-## Usage
+## Versioning
 
-Run `bundle exec thor build help`
+Builderator can automatically detect versions from SCM tags, increment the latest version of an SCM branch by a variety of steps, and create new SCM tags for new versions.
+
+[Additional documentation](docs/versioning.md) describes CLI commands, configuration, and detailed behavior.
+
+## Generators
+
+Builderator includes a task to generate common project trees from configuration definitions  and templates.
+
+Each type of project is configurable via the project collection in the `generator` namespace:
+
+```ruby
+generator.project :default do |default|
+  default.ruby.version '2.1.5'
+  default.builderator.version '~> 1.0'
+
+  default.vagrant do |vagrant|
+    vagrant.install false
+    vagrant.version 'v1.8.0'
+
+    vagrant.plugin 'vagrant-aws'
+    vagrant.plugin 'vagrant-omnibus'
+  end
+
+  default.resource :berksfile do |berksfile|
+    berksfile.path 'Berksfile', 'Berksfile.lock'
+    berksfile.action :rm
+  end
+
+  default.resource :buildfile do |buildfile|
+    buildfile.path 'Buildfile'
+    buildfile.action :create
+    buildfile.template 'template/Buildfile.erb'
+  end
+
+  # ...
+end
+```
 
-## Contributing
+Valid actions for resources include `:ignore`, `:create` (update only if missing), `:sync` (create or update with prompt), and `:rm`. `:create` and `:sync` actions require a valid template source.
 
-1. Fork it ( https://github.com/[my-github-username]/builderator/fork )
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create a new Pull Request
+By default, the `generator` subcommand includes a `default` project which removes Vagrant, Berkshelf, and Packer configurations.

data/Rakefile

@@ -1,2 +1 @@
-require "bundler/gem_tasks"
-
+require 'bundler/gem_tasks'

data/VERSION

@@ -1 +1 @@
-0.3.15
+1.0.0-rc.1

data/bin/build-clean

@@ -0,0 +1,102 @@
+#!/usr/bin/env ruby
+
+require 'thor'
+
+require_relative '../lib/builderator/config'
+require_relative '../lib/builderator/control/cleaner'
+
+module Builderator
+  module Tasks
+    ##
+    # Tasks to identify and remove unused EC2 resources
+    ##
+    class Cleaner < Thor
+      class_option :region,
+                   :type => :string,
+                   :aliases => :r,
+                   :desc => 'AWS Region in which to perform tasks'
+      class_option :commit,
+                   :type => :boolean,
+                   :default => false,
+                   :desc => 'Perform mutating API calls to cleanup resources'
+      class_option :filter,
+                   :type => :array,
+                   :aliases => :f,
+                   :desc => 'Key/value pairs to filter resources (--filter name foo owner_id 123456789)'
+      class_option :force,
+                   :type => :boolean,
+                   :default => false,
+                   :desc => 'Disable safety restrictions, including resource limits'
+
+      class_option 'group-by',
+                   :type => :array,
+                   :desc => 'Tags/properties by which to group resources for pruning'
+      class_option 'sort-by',
+                   :type => :string,
+                   :default => 'creation_date',
+                   :desc => 'Tag/property by which to sort grouped resources'
+      class_option :keep,
+                   :type => :numeric,
+                   :default => 5,
+                   :desc => 'Number of resources in each group to keep'
+
+      def initialize(*_)
+        super
+
+        ## Convert array of filter key-values to a hash
+        options['filters'] = Hash[*options['filter']] if options['filter'].is_a?(Array)
+
+        ## Load command flags
+        Config.argv(:cleaner => options, :aws => { :region => options['region'] })
+        Config.load(File.join(ENV['HOME'], '.builderator/Buildfile'))
+        Config.load(Util.relative_path('Buildfile').to_s)
+
+        Config.recompile
+
+        say_status 'dry-run', 'This is a dry-run.' unless Config.cleaner.commit
+      end
+
+      desc 'configs', 'Delete unused launch configurations'
+      def configs
+        Control::Cleaner.configs!(&method(:say_status))
+      end
+
+      desc 'images', 'Deregister unused images'
+      def images
+        Control::Cleaner.images!(&method(:say_status))
+      end
+
+      desc 'snapshots', 'Delete unused snapshots'
+      def snapshots
+        Control::Cleaner.snapshots!(&method(:say_status))
+      end
+
+      desc 'volumes', 'Delete unused volumes'
+      def volumes
+        Control::Cleaner.volumes!(&method(:say_status))
+      end
+
+      desc 'all', 'Cleaner volumes, launch configs, images, and snapshots in order'
+      def all
+        volumes
+        configs
+        images
+        snapshots
+
+        ## TODO Print resource counts here.
+        return if Control::Cleaner.exceptions.empty?
+
+        say_status :fail, 'Not all tasks completed successfully. The following '\
+          'exceptions occured:', :red
+        Control::Cleaner.exceptions.each do |e|
+          say_status(*e.status)
+        end
+
+        ## Mark the Jenkins job as fail if there were errors.
+        exit(1)
+      end
+    end
+  end
+end
+
+Builderator::Tasks::Cleaner.start(ARGV)

data/bin/build-data

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+
+require 'thor'
+
+require_relative '../lib/builderator/config'
+require_relative '../lib/builderator/control/data'
+
+module Builderator
+  module Tasks
+    ##
+    # Tasks to search AWS APIs
+    ##
+    class Data < Thor
+      def initialize(*_)
+        super
+
+        Config.load(File.join(ENV['HOME'], '.builderator/Buildfile'))
+        Config.load(Util.relative_path('Buildfile').to_s)
+
+        Config.recompile
+      end
+
+      desc 'image ', 'Search for AMIs'
+      method_option 'filter', :type => :string, :aliases => :f
+      method_option 'latest', :type => :boolean, :aliases => :l, :default => false
+      def image(*query)
+        query = Hash[*query]
+
+        ## Load a pre-defined filter
+        query['filter'] = options['filter']
+
+        result = Control::Data.image(query)
+
+        if options['latest']
+          puts result.first.image_id
+          return
+        end
+
+        puts result.map(&:image_id).join(', ')
+      end
+    end
+  end
+end
+
+Builderator::Tasks::Data.start(ARGV)

data/builderator.gemspec

@@ -13,18 +13,21 @@ Gem::Specification.new do |spec|
   spec.homepage      = 'https://github.com/rapid7/builderator'
   spec.license       = 'MIT'
 
-  spec.files         = `git ls-files -z`.split("\x0")
-  spec.executables   = Dir[File.join(Builderator::PATH, 'bin/*')].map { |f| File.basename(f) }
-  spec.test_files    = spec.files.grep(/^(test|spec|features)\//)
+  spec.files         = Dir['**/*']
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ['lib']
 
   spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+  spec.add_development_dependency 'rubocop', '~> 0.35'
+  spec.add_development_dependency 'thor-scmversion', '1.7.0'
 
   spec.add_dependency 'aws-sdk', '~> 2.0'
   spec.add_dependency 'bundler', '~> 1.7.0'
   spec.add_dependency 'berkshelf', '~> 3.2'
   spec.add_dependency 'chef', '~> 12.0'
+  spec.add_dependency 'faraday_middleware', '~> 0.10.0'
   spec.add_dependency 'ignorefile'
   spec.add_dependency 'thor', '~> 0.19.0'
-  spec.add_dependency 'thor-scmversion', '1.7.0'
 end

data/docs/configuration.md

@@ -0,0 +1,154 @@
+Configuration DSL
+=================
+
+The configuration DSL is made up of key-value pairs, called `attributes`, which are grouped into `namespaces` and `collections`.
+
+Namespaces can accessed with blocks, or with a fluent interface:
+
+```ruby
+aws do |a|
+  a.region = 'us-west-1'
+end
+
+## Is the same as
+aws.region = 'us-west-1'
+```
+
+Collections are named sets. Like namespaces, they can be accessed with blocks, or a fluent interface:
+
+```ruby
+profile :default do |default_profile|
+  default_profile.chef.run_list 'apt:default', 'redis:server'
+end
+
+profile(:default).chef.environment 'development'
+```
+
+In the example above, the same collection is accessed twice. The final result looks like:
+
+```json
+{
+  "profile": {
+    "default": {
+      "chef": {
+        "run_list": ["apt:default", "redis:server"],
+        "environment": "development"
+      }
+    }
+  }
+}
+```
+
+## Helper Methods
+
+* `lookup(source, query)` - Query an external data-source for a value inline.
+* Source `:image`: Return an array of EC2 instances, sorted by `creation_date` (See http://docs.aws.amazon.com/sdkforruby/api/Aws/EC2/Client.html#describe_images-instance_method)
+
+* `vendored(name, path)` - Return the absolute path to `path` in the named vendor resource. _ Hint: Use this helper to reference Builderator policy files and Chef data_bag and environment sources in an external repository._
+
+## Configuration File DSL
+
+Collections and namespaces may be nested indefinitely.
+
+* [Namespace `cookbook`](configuration/cookbook.md)
+* [Collection `profile`](configuration/profile.md)
+
+* `build_name, required: true` The name of the build
+* `build_number` Optional reference to the CI build number for this release
+* `build_url` Optional link the CI page for this release
+* `description` A short human-readable description of the build
+* `version` The version of this release of the build. Auto-populated by `autoversion` by default
+* `cleanup` Enable post-build cleanup tasks. Default `true`
+
+* `relative(path)` - Return the absolute path to `path` relative to the calling Buildfile _Hint: Use this helper to reference templates included with a vendored policy._
+
+## Namespace `autoversion`
+
+* `create_tags` During a release, automatically create and push new SCM tags
+* `search_tags` Use SCM tags to determine the current version of the build
+
+## Namespace `chef`
+
+Global configurations for chef provisioners in Vagrant and Packer
+
+* `log_level` Chef client/solo log level
+* `staging_directory` the path in VMs and images that Chef artifacts should be mounted/copied to. Defaults to `/var/chef`
+* `version` The version of chef to install with Omnibus
+
+## Namespace `local`
+
+Local paths used for build tasks
+
+* `cookbook_path` Path at which to vendor cookbooks. Default `.builderator/cookbooks`
+* `data_bag_path` and `environment_path` Paths that Chef providers should load data-bag and environment documents from.
+
+## Collection `policy`
+
+Load additional attributes into the parent file from a relative path
+
+* `path` Load a DSL file, relative => true
+* `json` Load a JSON file relative => true
+
+## Namespace `aws`
+
+AWS API configurations. _Hint: Configure these in `$HOME/.builderator/Buildfile`, or use a built-in credential source, e.g. ~/.aws/config!_
+
+* `region` The default AWS region to use
+* `access_key` and `secret_key` A valid IAM key-pair
+
+## Collection `vendor`
+
+Fetch remote artifacts for builds
+
+* Sources:
+  * `path` Link to a local file/directory
+  * `git` Fetch a git repository
+  * `github` Fetch a git repository from a GitHub URI (e.g. `OWNER/REPO`) using the SSH protocol. You must have a valid SSH key configuration for public GitHub.
+* Git-specific parameters:
+  * `branch`
+  * `tag`
+  * `ref`
+  * `rel` Checkout a sub-directory of a git repository
+
+## Namespace `cleaner`
+
+Configuration parameters for `build-clean` tasks
+
+### Namespace `limits`
+
+Maximum number of resources to remove without manual override
+
+* `images`
+* `launch_configs`
+* `snapshots`
+* `volumes`
+
+## Namespace `generator`
+
+Configurations for the `generator` task
+
+### Collection `project`
+
+* `builderator.version` The version of Builderator to install with Bundler
+* `ruby.version` The version of ruby to require for Bundler and `rbenv`/`rvm`
+
+#### Namespace `vagrant`
+
+* `install` Boolean, include the vagrant gem from GitHub `mitchellh/vagrant`
+* `version` The version of Vagrant to use from GitHub, if `install` is true
+
+##### Collection `plugin`
+
+Vagrant plugins to install, either with the `build vagrant plugin` command, for a system-wide installation of Vagrant, or in the generated Gemfile if `install` is true
+
+#### Collection `resource`
+
+Add a managed file to the project definition
+
+* `action` One of
+  * `:create` Add a file from a template if it's missing
+  * `:sync` Create or update a file from a template, stopping to ask for instructions if the file exists and the templated output does not match
+  * `:ignore` Do nothing
+  * `:rm` Delete a file if it exists
+* `path` One or more path in the working directory the this resource manages. Action `:rm` will delete multiple files, while `:create` and `:sync` will only use the first element of the list as their destination.
+* `template` The path to an ERB template. Must be an absolute path: use the [helpers](#helpers) in the Buildfile namespace to extend paths inline.

data/docs/configuration/cookbook.md

@@ -0,0 +1,19 @@
+cookbook
+========
+
+* `path` The path to a local cookbook source, including a valid `metadata.rb` file.
+* `sources, type: list, singular: add_source, unique: true` Supermarket APIs to resolve cookbook dependencies from
+* `metadata` Boolean. Read dependencies from local cookbook metadata.
+
+## `depends name`
+
+Collection of declared cookbook dependencies. Options are passed to [Berkshelf](http://berkshelf.com/). Check out their docs for additional details.
+
+* `version` A version constraint spec for the cookbook
+* `git` A git URI from which to fetch the cookbook
+* `GitHub` A GitHub URL from which to fetch the cookbook
+* `branch` A branch reference from which to fetch the cookbook
+* `tag` A tag reference from which to fetch the cookbook
+* `ref` A comittish reference from which to fetch the cookbook
+* `rel` The sub-directory of a git repository to check out as a cookbook
+* `path` The path to a local cookbook, relative to the build workspace.