wukong-deploy 0.0.2 → 0.1.0

data/.yardopts

@@ -0,0 +1,6 @@
+--readme   README.md
+--markup   markdown
+-
+Gemfile
+LICENSE.md
+README.md

data/Gemfile

@@ -5,4 +5,6 @@ gemspec
 group :development do
   gem 'rake',     '~> 0.9'
   gem 'rspec',    '~> 2'
+  gem 'yard'
+  gem 'redcarpet'
 end

data/LICENSE.md

@@ -0,0 +1,95 @@
+# License for Wukong-Deploy
+
+The wukong code is __Copyright (c) 2011, 2012 Infochimps, Inc__
+
+This code is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+
+__________________________________________________________________________
+
+# Apache License
+
+
+                              Apache License
+                        Version 2.0, January 2004
+                     http://www.apache.org/licenses/
+
+_TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION_
+
+## 1. Definitions.
+
+* **License** shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+
+* **Licensor** shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+* **Legal Entity** shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+* **You** (or **Your**) shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+* **Source** form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+* **Object** form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+* **Work** shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+
+* **Derivative Works** shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+* **Contribution** shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+* **Contributor** shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+## 2. Grant of Copyright License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+## 3. Grant of Patent License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+## 4. Redistribution.
+
+You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+  - (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+  - (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
+  - (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+  - (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+## 5. Submission of Contributions.
+
+Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+## 6. Trademarks.
+
+This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+## 7. Disclaimer of Warranty.
+
+Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+## 8. Limitation of Liability.
+
+In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+## 9. Accepting Warranty or Additional Liability.
+
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+_END OF TERMS AND CONDITIONS_
+
+## APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets `[]` replaced with your own identifying information. (Don't include the brackets!)  The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
+
+> Copyright [yyyy] [name of copyright owner]
+>
+> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+>
+>     http://www.apache.org/licenses/LICENSE-2.0
+>
+> Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

data/bin/wu

@@ -0,0 +1,35 @@
+#!/usr/bin/env ruby
+require 'shellwords'
+now=Time.now.strftime("%Y-%m-%d %H:%M:%S")
+if ARGV.empty?
+    $stderr.puts "ERROR #{now} [wu                  ] -- Must provide a Wukong command to run. Try the --help option."
+else
+    if ARGV.size == 1 && ARGV.first == '--help'
+	$stderr.puts <<EOF
+usage: wu COMMAND [OPTIONS] [ARG] ...
+
+wu is a wrapper for easy use of Wukong's command-line tools.  It takes
+your arguments, constructs the name of the proper wu-tool to call, and
+prepends a call to bundle exec.
+
+  $ wu local ...
+
+is equivalent to
+
+  $ bundle exec wu-local ...
+
+You can run any of the wu-tools like
+	
+  wu-local
+  wu-hadoop
+  wu-deploy
+
+EOF
+    else
+      if ARGV.first =~ /^-/
+        $stderr.puts "ERROR ${now} [wu                  ] -- First argument must be the name of a wu tool to run, got <${1}>"
+      else
+        Kernel.exec "bundle exec wu-#{Shellwords.join(ARGV)}"
+      end
+    end
+end

data/bin/wu-deploy

@@ -1,7 +1,4 @@
 #!/usr/bin/env ruby
 
 require 'wukong-deploy'
-settings = Wukong::Deploy::Configuration
-require 'wukong/boot'
-Wukong.boot!(settings)
-Wukong::Deploy::Driver.run(settings, *settings.rest)
+Wukong::Deploy::DeployRunner.run

data/lib/wukong-deploy.rb

@@ -2,25 +2,59 @@
 # build-in widgets.
 require 'wukong'
 
-# Require just enough of wukong-hadoop to give us changes at the
-# processor level.  The rest is only required when running the
-# 'wu-hadoop' program.  Also grab configuration since we'll need it.
-require 'wukong-hadoop/extensions'
-require 'wukong-hadoop/configuration'
-
-# Grab configuration from Wonderdog.
-require 'wonderdog'
+require_relative 'wukong-deploy/deploy_pack'
+require_relative 'wukong-deploy/deploy_runner'
+require_relative 'wukong-deploy/rake_runner'
 
 module Wukong
   # Provides some helper methods for loading the assets within a
   # deploy pack at runtime and for Wukong plugins to ask questions
   # about the deploy pack they're running in.
   module Deploy
-  end
-end
+    include Plugin
+
+    # Configure the given settings object for use with Wukong::Deploy
+    #
+    # @param [Configliere::Param] settings the settings to configure
+    # @param [String] program the name of the currently executing program
+    def self.configure settings, program
+      settings.define(:environment, :description => "The environment to run in", :default => 'development', :flag => 'e', :env_var => 'ICS_PLATFORM_ENV')
+
+      case program
+      when 'wu-deploy'
+        settings.define(:dry_run, :description => "Don't actually create or modify anything", :type => :boolean, :default => false)
+        settings.define(:skip,    :description => "Skip existing files", :type => :boolean, :default => false)
+        settings.define(:force,   :description => "Overwrite existing files", :type => :boolean, :default => false)
+      when 'wu-hadoop'
+        require_relative('wukong-deploy/extensions')
+        settings[:command_prefix] = 'bundle exec'
+      end
+    end
 
+    # Boot the deploy pack, reading all available +settings+ and
+    # rooting it at the given +path+.
+    #
+    # @param [Configliere::Param] settings
+    # @param [String] path
+    def self.boot settings, path
+# # Require just enough of wukong-hadoop to give us changes at the
+# # processor level.  The rest is only required when running the
+# # 'wu-hadoop' program.  Also grab configuration since we'll need it.
+# require 'wukong-hadoop/extensions'
 
-# Now load the rest of the deploy pack.
-require 'wukong-deploy/deploy_pack'
-require 'wukong-deploy/configuration'
-require 'wukong-deploy/driver'
+# # Grab configuration from Wonderdog.
+# require 'wonderdog'
+
+      
+      @pre_deploy_settings = settings.dup
+      @settings            = settings
+      @root                = Pathname.new(path)
+      read_common_settings
+      read_environment_settings
+      read_deploy_settings
+      read_remote_settings
+      @booted = true
+    end
+
+  end
+end

data/lib/wukong-deploy/console.rb

@@ -1,11 +1,15 @@
 require 'irb'
 module Wukong
   module Deploy
+
+    # A class to wrap a basic IRB session.
     class Console
 
+      # Initialize this Console.
       def initialize
       end
 
+      # Start this Console.
       def run!
         ARGV.clear
         IRB.start

data/lib/wukong-deploy/deploy_pack.rb

@@ -1,27 +1,13 @@
 module Wukong
   module Deploy
-
-    # Boot the deploy pack, reading all available +settings+ and
-    # rooting it at the given +path+.
-    #
-    # @param [Configliere::Param] settings
-    # @param [String] path
-    def self.boot! settings, path
-      @pre_deploy_settings = settings.dup
-      @settings            = settings
-      @root                = Pathname.new(path)
-      read_common_settings
-      read_environment_settings
-      read_remote_settings
-    end
-
+    
     # Return the root directory of this deploy pack.
     #
     # @return [Pathname]
     def self.root
       @root
     end
-
+    
     # Recursively require each Ruby file +dir+.
     #
     # @example Requiring all .rb files anywhere within /lib/my_lib
@@ -29,15 +15,8 @@ module Wukong
     #   Wukong::Deploy.require_recursive("lib/my_lib")
     #
     # @param [String] glob
-    def self.require_recursive glob
-      Dir[root.join("#{glob}/**/*.rb")].each { |path| require(path) }
-    end
-    
-    # Return the name of the executable program currently running.
-    #
-    # @return [String]
-    def self.executable
-      File.basename($0)
+    def self.require_recursive glob, ext='.rb'
+      Dir[root.join("#{glob}/**/*#{ext}")].each { |path| require(path) }
     end
 
     # Return the current environment the deploy pack is in.
@@ -54,6 +33,14 @@ module Wukong
       @settings
     end
 
+    # Has this deploy pack been completely booted, all settings read,
+    # &c.
+    #
+    # @return [true, false]
+    def self.booted?
+      @booted
+    end
+
     # Return the settings the deploy pack had *before* it booted
     # itself -- these are used to pass to other tools when invoking
     # them, since they will read the deploy pack settings themselves
@@ -105,40 +92,58 @@ module Wukong
     def self.app_dir
       root.join('app')
     end
-
-    # The logger for this deploy pack.
-    #
-    # @return [Logger]
-    def self.logger
-      return @logger if @logger
-      # FIXME -- want to use the Wukong logger here...
-      require 'logger'
-      @logger ||= defined?(Log) ? Log : Logger.new
-    end
     
     private
 
     # Read settings common across all environments from
-    # config/settings.yml.
+    # `config/settings.yml`.
+    #
+    # Will also read any YAML files in the `config/settings` directory
+    # -- order is *not* guaranteed.
     def self.read_common_settings
       read_settings_from_file(settings, config_dir.join("settings.yml"))
+      if File.directory?(config_dir.join('settings'))
+        Dir[config_dir.join('settings','*.yml')].each do |path|
+          read_settings_from_file(settings, path)
+        end
+      end
     end
     
     # Read settings unique to this deploy pack's current environment
-    # from config/ENVIRONMENT.yml.
+    # from config/environments/ENVIRONMENT.yml.
+    #
+    # Will also read any YAML files in the
+    # `config/environments/ENVIRONMENT` directory -- order is *not*
+    # guaranteed.
     def self.read_environment_settings
       read_settings_from_file(settings, config_dir.join("environments", "#{environment}.yml"))
+      if File.directory?(config_dir.join('environments', environment))
+        Dir[config_dir.join('environments', environment, '*.yml')].each do |path|
+          read_settings_from_file(settings, path)
+        end
+      end
+    end
+
+    # Read settings created by a deployment tool from
+    # `config/deploy.yml` and
+    # `config/environments/deploy-ENVIRONMENT.yml.
+    def self.read_deploy_settings
+      # pass 'false' here so we don't log an annoying warning message
+      # when not in a deployed environment.
+      read_settings_from_file(settings, config_dir.join("deploy.yml"),                                false)
+      read_settings_from_file(settings, config_dir.join("environments", "deploy-#{environment}.yml"), false)
     end
 
     # Update +settings+ with the configuration at the given +path+.
     #
     # @param [Configliere::Param] settings
     # @param [String, Pathname] path
-    def self.read_settings_from_file settings, path
+    # @param [true, false] warn whether to log a warning message if no config file is found
+    def self.read_settings_from_file settings, path, warn=true
       if File.exist?(path) && File.readable?(path) && File.file?(path)
         settings.read(path)
       else
-        logger.warn("Cannot read settings file at #{path}.")
+        Wukong::Log.warn("Could not read settings from path: #{path}") if warn
       end
     end
 

data/lib/wukong-deploy/deploy_runner.rb

@@ -0,0 +1,72 @@
+module Wukong
+  module Deploy
+
+    # A Runner class that implements wu-deploy.
+    class DeployRunner < Wukong::Runner
+
+      include Logging
+
+      usage 'ACTION'
+
+      description <<-EOF.gsub(/^ {8}/,'')
+        wu-deploy is a tool for creating and interacting with deploy packs.
+
+        You can create a new deploy pack
+
+          $ wu-deploy new my_app
+
+        The `--force' and `--skip' options can be used to control how conflict
+        resolution works when creating files.  The `--dry_run` option can be
+        used to see what happens without doing it.
+
+        If you are within the directory of a deploy pack, you can enter an IRB
+        console with the deploy pack's environment already loaded:
+
+          $ wu-deploy console
+      EOF
+
+      # The action, or first argument given.
+      #
+      # @return [String]
+      def action
+        args.first
+      end
+
+      # The directory, or second argument given.
+      #
+      # @return [String]
+      def directory
+        args[1]
+      end
+
+      # Validate that the requested action is known and that its
+      # arguments are valid.
+      #
+      # @return [true]
+      # @raise [Wukong::Error] if the action or its arguments are invalid
+      def validate
+        case action
+        when 'new'
+          raise Error.new("Must provide a path to the root of the deploy pack you want to create") if directory.nil? || directory.blank?
+        when 'console'
+        else
+          raise Error.new("Invalid action: <#{action}>.")
+        end
+        true
+      end
+
+      # Run the requested action.
+      def run
+        case action
+        when 'new'
+          require_relative('templater')
+          Templater.new(File.expand_path(directory, Dir.pwd), settings).run!
+        when 'console'
+          require_relative('console')
+          Wukong::Deploy::Console.new.run!
+        end
+      end
+      
+    end
+  end
+end

data/lib/wukong-deploy/extensions.rb

@@ -0,0 +1,2 @@
+require_relative('extensions/hadoop_invocation_override')
+Wukong::Hadoop::HadoopRunner.class_eval { include Wukong::Deploy::HadoopInvocationOverride }

data/lib/wukong-deploy/extensions/hadoop_invocation_override.rb

@@ -0,0 +1,13 @@
+module Wukong
+  module Deploy
+    module HadoopInvocationOverride
+      def params_to_pass
+        Deploy.pre_deploy_settings
+      end
+    end
+  end
+end
+
+
+    
+    

data/lib/wukong-deploy/rake_runner.rb

@@ -0,0 +1,42 @@
+module Wukong
+  module Deploy
+
+    # A Runner class for booting up the environment within a Rake
+    # task.
+    class RakeRunner < Wukong::Runner
+
+      include Logging
+
+      # Load tasks that come with Wukong-Deploy.
+      def load
+        super()
+        load_rspec_tasks
+        load_generic_deploy_pack_tasks
+        load_this_deploy_packs_tasks
+      end
+
+      # Run the requested action.
+      def run
+      end
+
+      private
+
+      # :nodoc:
+      def load_rspec_tasks
+        require 'rspec/core/rake_task'
+        RSpec::Core::RakeTask.new(:spec)
+      end
+
+      # :nodoc:
+      def load_generic_deploy_pack_tasks
+        require 'wukong-deploy/tasks'
+      end
+
+      # :nodoc:
+      def load_this_deploy_packs_tasks
+        Wukong::Deploy.require_recursive('lib/tasks')
+      end
+      
+    end
+  end
+end

data/lib/wukong-deploy/repo.rb

@@ -1,11 +1,20 @@
 module Wukong
   module Deploy
+
+    # A class to represent a deploy pack repository as a collection of
+    # files on disk.
     class Repo
 
+      # The names of the top level directories in the deploy pack.
       TOP_LEVEL_DIR_NAMES = %w[app config data lib log script spec tmp]
-      
+
+      # The root directory where this deploy pack repo is or will be
+      # located.
       attr_reader :root
 
+      # Initialize a new deploy pack repo at the given `root`.
+      #
+      # @param [String] root
       def initialize root
         @root = Pathname.new(root)
       end
@@ -16,6 +25,9 @@ module Wukong
         end
       end
 
+      # The directories to create within this repo.
+      #
+      # @return [Array<String>]
       def dirs_to_create
         [
          root,
@@ -42,6 +54,10 @@ module Wukong
         ]
       end
 
+      # The files to template within this repo, relative to the repo's
+      # #root.
+      #
+      # @return [Array<String>]
       def files_to_template
         %w[
          config/boot.rb

data/lib/wukong-deploy/tasks.rb

@@ -1,13 +1,8 @@
 require 'rake'
 
-namespace 'bundler' do
-  desc "Remove local bundle repository"
-  task :clean do
-    Wukong::Deploy.logger.info "Removing local bundle at /.bundle and /bundle"
-    sh "rm -rf .bundle bundle"
-    Wukong::Deploy.logger.info "Make sure to run 'bundle install --standalone' to re-create the bundle"
-  end
-  
-end
+require 'yard'
+YARD::Rake::YardocTask.new(:doc)
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)
 
-  

data/lib/wukong-deploy/templater.rb

@@ -8,24 +8,39 @@ require_relative('templater/conflict_resolution')
 
 module Wukong
   module Deploy
+
+    # Templates out a deploy pack repo in a given directory.
     class Templater
 
+      # The Wukong::Deploy::Repo being templated.
       attr_accessor :repo
+
+      # The Configliere::Param settings.
       attr_accessor :options
 
       include FileUtils
       include Messaging
       include ConflictResolution
 
+      # Initialize a new Templater creating a deploy pack repository
+      # at the given `root` using the given `options`.
+      #
+      # @param [String] root
+      # @param [Hash] options
       def initialize root, options={}
         self.repo    = Repo.new(root)
         self.options = options
       end
 
+      # Is this a dry run in which no files are actually created or
+      # modified on disk?
+      #
+      # @return [true, nil]
       def dry_run?
-        @options[:dry_run]
+        options[:dry_run]
       end
 
+      # Run this templater.
       def run!
         if dry_run?
           puts "Would perform the following actions in #{repo.root}"
@@ -37,26 +52,40 @@ module Wukong
         create_gitkeeps
         create_gitignore
       end
+
+      # The directory containing templates used to generate the
+      # corresponding files in the new deploy pack repository.
+      #
+      # @return [Pathname]
+      def templates_dir
+        @templates_dir ||= Pathname.new(File.expand_path('../../../templates', __FILE__))
+      end
       
+      private
+
+      # :nodoc:
       def create_dirs
         repo.dirs_to_create.each do |dir|
           create_directory(dir)
         end
       end
 
+      # :nodoc:
       def create_templates
         repo.files_to_template.each do |repo_path|
           create_template(templates_dir.join("#{repo_path}.erb"), repo.root.join(repo_path))
         end
       end
 
+      # :nodoc:
       def create_template input_path, output_path, binding={}
         input   = File.read(input_path)
         erb     = Erubis::Eruby.new(input)
         content = erb.result(binding)
         create_file(content, output_path)
       end
-      
+
+      # :nodoc:
       def create_gitkeeps
         repo.dirs_to_create.each do |dir|
           if Dir[File.join(dir, '*')].empty?
@@ -65,20 +94,17 @@ module Wukong
         end
       end
 
+      # :nodoc:
       def create_gitignore
         create_template(templates_dir.join('gitignore'), repo.root.join('.gitignore'))
       end
 
-      def templates_dir
-        @templates_dir ||= Pathname.new(File.expand_path('../../../templates', __FILE__))
-      end
-
-      private
-
+      # :nodoc:
       def empty_file
         ""
       end
-      
+
+      # :nodoc:
       def create_file content, path
         if File.exist?(path)
           handle_conflict(content, path)
@@ -88,12 +114,14 @@ module Wukong
         end
       end
 
+      # :nodoc:
       def create_directory(dir)
         message_create(dir)
         return if dry_run?
         mkdir_p(dir)
       end
 
+      # :nodoc:
       def write_file content, path
         return if dry_run?
         File.open(path, 'w') { |f| f.write(content) }

data/lib/wukong-deploy/version.rb

@@ -2,6 +2,6 @@ module Wukong
   module Deploy
 
     # The current version of this deploy pack framework.
-    VERSION = '0.0.2'
+    VERSION = '0.1.0'
   end
 end

data/spec/wukong-deploy/wu_deploy_spec.rb

@@ -9,7 +9,7 @@ describe 'wu-deploy' do
     subject { command('wu-deploy') }
     it { should exit_with(:non_zero) }
     it "displays a help message" do
-      should have_stderr(/usage: wu-deploy/)
+      should have_stderr(/invalid action/i)
     end
   end
 

data/templates/Gemfile.erb

@@ -17,24 +17,57 @@ source                 :rubygems
 # Here's a link to the [Bundler documentation](http://gembundler.com/)
 # for your convenience.
 
+#
+# = Framework
+#
+# The following libraries are part of the deploy pack framework.  You
+# shouldn't need to change these.
+#
 
+# The core library: processors, dataflows, testing, &c.
+#
+# Provides
+#
+#   - wu-local
+gem 'wukong',                 '3.0.0'
 
+# Model Hadoop jobs locally and run them on a cluster.
+# 
+# Provides
+#
+#   - wu-hadoop
+gem 'wukong-hadoop',          '0.1.0'
 
+# Lets Hadoop streaming work with Elasticsearch.
+gem 'wonderdog',              '0.1.0'
 
-
+# Provides an interface into Wukong dataflows accessible from Storm.
 #
-# = Framework
+# Provides
 #
-# The following libraries are part of the deploy pack framework.  You
-# shouldn't need to change these.
+#   - wu-storm
+gem 'wukong-storm',           '0.1.0'
+
+# Lets you load JSON blobs into data stores.
 #
+# Provides
+#
+#   - wu-load
+#   - wu-source
+gem 'wukong-load',            '0.1.0'
 
-gem 'wukong-deploy',          :git => 'https://github.com/infochimps-labs/wukong-deploy'
-gem 'wukong',                 :git => 'https://github.com/infochimps-labs/wukong', :branch => '3.0.0'
-gem 'wukong-hadoop',          :git => 'https://github.com/infochimps-labs/wukong-hadoop'
-gem 'wonderdog',              :git => 'https://github.com/infochimps-labs/wonderdog'
-gem 'vayacondios-client',     :git => 'https://github.com/infochimps-labs/vayacondios'
+# Framework for gluing code together.
+#
+# Provides
+#
+#   - wu-deploy
+gem 'wukong-deploy',          '0.1.0'
 
 group :development do
+  gem 'yard'
+  gem 'redcarpet'
+end
+
+group :test do
   gem 'rspec'
 end

data/templates/README.md.erb

@@ -1,4 +1,4 @@
-f# Welcome to the Infochimps Platform!
+# Welcome to the Infochimps Platform!
 
 The [Infochimps Platform](http://www.infochimps.com) is an end-to-end,
 managed solution for building Big Data applications.  It integrates
@@ -86,17 +86,11 @@ From within the root of your deploy pack run the following commands
 
 ```
 $ sudo gem install bundler
-$ bundle install --standalone
+$ bundle install
 ```
 
 If you're using [rbenv](https://github.com/sstephenson/rbenv) you may
-want to run `rbenv exec bundle install --standalone`.
-
-Bundler will install all the necessary dependencies locally in a
-directory called `bundle`.  We use a `standalone` installation of your
-application bundle because this makes it easier to connect code in the
-deploy pack to frameworks like Hadoop, Storm, &c. when your code is
-running within the Infochimps Platform.
+want to run `rbenv exec bundle install`.
 
 ### Configuration
 

data/templates/Rakefile.erb

@@ -1,10 +1,13 @@
-require_relative 'config/environment'
-require 'wukong-deploy/tasks'
-require 'rake'
+# Require the environment
+require_relative('config/environment')
+
+# Require the Wukong rake helper which gives us the :environment task.
+require 'wukong/rake_helper'
 
-require 'rspec/core/rake_task'
-RSpec::Core::RakeTask.new(:spec)
+# Load all pre-defined tasks from the deploy pack library.
+require 'wukong-deploy/tasks'
 
-Wukong::Deploy.require_recursive('lib/tasks')
+# Load all tasks within this deploy pack.
+Dir[File.join(File.expand_path('../lib/tasks', __FILE__), '*')].each { |path| require path }
 
 task :default => :spec

data/templates/config/boot.rb.erb

@@ -2,9 +2,10 @@
 #
 # Loads the Ruby language framework and activates Bundler.
 #
-# We use a standalone bundle installation becaues it makes it easier
-# to have non-Ruby code in underlying frameworks interact with the
-# deploy pack.
+# We can use a system-wide bundle installation _or_ a standalone
+# bundle installation.  If a standalone installation exists (at
+# /bundle/bundler/setup.rb), we will detect and use it.  Otherwise, we
+# fall back on the system-wide bundle installation.
 #
 # Make sure you install the bundle locally with the following command:
 #
@@ -12,10 +13,14 @@
 #
 #   $ bundle install --standalone
 
-setup_path = File.expand_path("../../bundle/bundler/setup.rb", __FILE__)
-begin
-  require setup_path
-rescue LoadError => e
-  $stderr.puts("Failed to require \"#{setup_path}\".  Try running\n\n  bundle install --standalone\n\n")
-  exit(1)
+local_setup_path = File.expand_path("../../bundle/bundler/setup.rb", __FILE__)
+if File.exist?(local_setup_path)
+  begin
+    require local_setup_path
+  rescue LoadError => e
+    $stderr.puts("Local Bundle installation at #{local_setup_path} is corrupted.  Try running\n\n  rm -rf bundle .bundle\n  bundle install --standalone\n\n")
+    exit(1)
+  end
+else
+  require 'bundler/setup'
 end

data/templates/config/environment.rb.erb

@@ -4,5 +4,3 @@
 # runtime environment of your application.
 
 require_relative("application")
-
-Wukong::Deploy.boot!(Wukong::Deploy::Configuration, File.expand_path("../../", __FILE__))

data/wukong-deploy.gemspec

@@ -31,15 +31,13 @@ Gem::Specification.new do |gem|
   EOF
 
   gem.files         = `git ls-files`.split("\n")
-  gem.executables   = ['wu-deploy']
+  gem.executables   = ['wu-deploy','wu']
   gem.test_files    = gem.files.grep(/^spec/)
   gem.require_paths = ['lib']
 
-  gem.add_dependency('wukong',        '3.0.0.pre3')
-  gem.add_dependency('wukong-hadoop', '>= 0.0.2')
-  gem.add_dependency('wonderdog',     '>= 0.0.2')
+  gem.add_dependency('wukong',        '3.0.0')
+  gem.add_dependency('rake',          '~> 0.9')
   gem.add_dependency('erubis')
   gem.add_dependency('diffy')
-  gem.add_dependency('rake',     '~> 0.9')
 end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wukong-deploy
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -11,7 +11,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-12-17 00:00:00.000000000 Z
+date: 2013-02-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: wukong
@@ -20,7 +20,7 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 3.0.0.pre3
+        version: 3.0.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -28,39 +28,23 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 3.0.0.pre3
+        version: 3.0.0
 - !ruby/object:Gem::Dependency
-  name: wukong-hadoop
-  requirement: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ! '>='
-      - !ruby/object:Gem::Version
-        version: 0.0.2
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ! '>='
-      - !ruby/object:Gem::Version
-        version: 0.0.2
-- !ruby/object:Gem::Dependency
-  name: wonderdog
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 0.0.2
+        version: '0.9'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 0.0.2
+        version: '0.9'
 - !ruby/object:Gem::Dependency
   name: erubis
   requirement: !ruby/object:Gem::Requirement
@@ -93,22 +77,6 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: '0.9'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: '0.9'
 description: ! "  The Infochimps Platform is an end-to-end, managed solution for\n
   \ building Big Data applications.  It integrates best-of-breed\n  technologies like
   Hadoop, Storm, Kafka, MongoDB, ElasticSearch,\n  HBase, &c. and provides simple
@@ -123,21 +91,27 @@ description: ! "  The Infochimps Platform is an end-to-end, managed solution for
 email: coders@infochimps.com
 executables:
 - wu-deploy
+- wu
 extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
 - .rspec
+- .yardopts
 - Gemfile
+- LICENSE.md
 - README.md
 - Rakefile
+- bin/wu
 - bin/wu-deploy
 - examples/.gitkeep
 - lib/wukong-deploy.rb
-- lib/wukong-deploy/configuration.rb
 - lib/wukong-deploy/console.rb
 - lib/wukong-deploy/deploy_pack.rb
-- lib/wukong-deploy/driver.rb
+- lib/wukong-deploy/deploy_runner.rb
+- lib/wukong-deploy/extensions.rb
+- lib/wukong-deploy/extensions/hadoop_invocation_override.rb
+- lib/wukong-deploy/rake_runner.rb
 - lib/wukong-deploy/repo.rb
 - lib/wukong-deploy/tasks.rb
 - lib/wukong-deploy/templater.rb

data/lib/wukong-deploy/configuration.rb

@@ -1,47 +0,0 @@
-module Wukong
-  module Deploy
-
-    # Pass in +settings+ that need to be modified for deploy-specific
-    # settings *before* they are resolved.
-    #
-    # @param [Configliere::Param] settings
-    # @return [Configliere::Param] the newly configured settings
-    def self.configure settings
-      settings.define(:environment, :description => "The environment to run in", :default => 'development', :flag => 'e', :env_var => 'ICS_PLATFORM_ENV')
-      
-      case executable
-      when 'wu-hadoop'
-        Wukong::Elasticsearch.configure(settings) if executable == 'wu-hadoop'
-      when 'wu-deploy'
-        settings.define(:dry_run, :description => "Don't actually create or modify anything", :type => :boolean, :default => false)
-        settings.define(:skip,    :description => "Skip existing files", :type => :boolean, :default => false)
-        settings.define(:force,   :description => "Overwrite existing files", :type => :boolean, :default => false)
-        def settings.usage
-          "usage: wu-deploy ACTION [ --param=val | --param | -p val | -p ] ..."
-        end
-        settings.use(:commandline)
-        settings.description = <<EOF
-wu-deploy is a tool for creating and interacting with deploy packs.
-
-You can create a new deploy pack
-
-  $ wu-deploy new my_app
-
-The `--force' and `--skip' options can be used to control how conflict
-resolution works when creating files.  The `--dry_run` option can be
-used to see what happens without doing it.
-
-If you are within the directory of a deploy pack, you can enter an IRB
-console with the deploy pack's environment already loaded:
-
-  $ wu-deploy console
-EOF
-      end
-      settings
-    end
-
-    # All configuration for Wukong::Deploy lives in this object.
-    Configuration = configure(Configliere::Param.new) unless defined?(Configuration)
-  end
-
-end

data/lib/wukong-deploy/driver.rb

@@ -1,39 +0,0 @@
-module Wukong
-  module Deploy
-    class Driver
-
-      attr_accessor :settings
-      attr_accessor :args
-
-      def self.run(settings, *args)
-        begin
-          new(settings,*args).run!
-        rescue Wukong::Error => e
-          $stderr.puts(e.message)
-          exit(1)
-        end
-      end
-
-      def run!
-        case args.first
-        when 'new'
-          require_relative('templater')
-          raise Error.new("Must provide a path to the root of the deploy pack you want to create") if args[1].nil? || args[1].blank?
-          Templater.new(File.expand_path(args[1], Dir.pwd), settings).run!
-        when 'console'
-          require_relative('console')
-          Wukong::Deploy::Console.new.run!
-        else
-          settings.dump_help
-          exit(2)
-        end
-      end
-
-      def initialize settings, *args
-        self.settings = settings
-        self.args     = args
-      end
-      
-    end
-  end
-end