wukong-deploy 0.0.1

data/.gitignore

@@ -0,0 +1,59 @@
+## OS
+.DS_Store
+Icon
+nohup.out
+.bak
+
+*.pem
+
+## EDITORS
+\#*
+.\#*
+\#*\#
+*~
+*.swp
+REVISION
+TAGS*
+tmtags
+*_flymake.*
+*_flymake
+*.tmproj
+.project
+.settings
+
+## COMPILED
+a.out
+*.o
+*.pyc
+*.so
+
+## OTHER SCM
+.bzr
+.hg
+.svn
+
+## PROJECT::GENERAL
+
+log/*
+tmp/*
+pkg/*
+
+coverage
+rdoc
+doc
+pkg
+.rake_test_cache
+.bundle
+.yardoc
+
+.vendor
+
+## PROJECT::SPECIFIC
+
+old/*
+docpages
+away
+
+.rbx
+Gemfile.lock
+Backup*of*.numbers

data/.rspec

@@ -0,0 +1,2 @@
+--format=documentation
+--color

data/Gemfile

@@ -0,0 +1,3 @@
+source                 :rubygems
+
+gemspec

data/README.md

@@ -0,0 +1,94 @@
+# Wukong Deploy Pack
+
+The [Infochimps Platform](http://www.infochimps.com) is an end-to-end,
+managed solution for building Big Data applications.  It integrates
+best-of-breed technologies like [Hadoop](http://hadoop.apache.org/),
+[Storm](https://github.com/nathanmarz/storm),
+[Kafka](http://incubator.apache.org/kafka/),
+[MongoDB](http://www.mongodb.org/),
+[ElasticSearch](http://www.elasticsearch.org/),
+[HBase](http://hbase.apache.org/), &c. and provides simple interfaces
+for accessing these powerful tools.
+
+Computation, analytics, scripting, &c. are all handled by
+[Wukong](http://github.com/infochimps-labs/wukong) within the
+platform.  Wukong is an abstract framework for defining computations
+on data.  Wukong processors and flows can run in many different
+execution contexts including:
+
+  * locally on the command-line for testing or development purposes
+  * as a Hadoop mapper or reducer for batch analytics or ETL
+  * within Storm as part of a real-time data flow
+
+The Infochimps Platform uses the concept of a deploy pack for
+developers to develop all their processors, flows, and jobs within.
+The deploy pack can be thought of as a container for all the necessary
+Wukong code and plugins useful in the context of an Infochimps
+Platform application.  It includes the following libraries:
+
+* <a href="http://github.com/infochimps-labs/wukong">wukong</a>: The core framework for writing processors and chaining them together.
+* <a href="http://github.com/infochimps-labs/wukong-hadoop">wukong-hadoop</a>: Run Wukong processors as mappers and reducers within the Hadoop framework.  Model Hadoop jobs locally before you run them.
+* <a href="http://github.com/infochimps-labs/wonderdog">wonderdog</a>: Connect Wukong processors running within Hadoop to Elasticsearch as either a source or sink for data.
+
+## Installation
+
+The deploy pack is installed as a RubyGem:
+
+```
+$ sudo gem install wukong-deploy
+```
+
+## File Structure
+
+A deploy pack is a repository with the following
+[Rails](http://rubyonrails.org/)-like file structure:
+
+```
+├──   app
+│   ├──   models
+│   ├──   processors
+│   ├──   flows
+│   └──   jobs
+├──   config
+│   ├──   environment.rb
+│   ├──   application.rb
+│   ├──   initializers
+│   ├──   settings.yml
+│   └──   environments
+│       ├──   development.yml
+│       ├──   production.yml
+│       └──   test.yml
+├──   data
+├──   Gemfile
+├──   Gemfile.lock
+├──   lib
+├──   log
+├──   Rakefile
+├──   spec
+│   ├──   spec_helper.rb
+│   └──   support
+└──   tmp
+```
+
+Let's look at it piece by piece:
+
+* <b>app</b>: The directory with all the action.  It's where you define:
+  * <b>models</b>: Your domain models or "nouns", which define and wrap the different kinds of data elements in your application.  They are built using whatever framework you like (defaults to [Gorillib](http://github.com/infochimps-labs/gorillib))
+  * <b>processors</b>: Your fundamental operations or "verbs", which are passed records and parse, filter, augment, normalize, or split them.
+  * <b>flows</b>: Chain together processors into streaming flows for ingestion, real-time processing, or [complex event processing](http://en.wikipedia.org/wiki/Complex_event_processing) (CEP)
+  * <b>jobs</b>: Pair processors together to create batch jobs to run in Hadoop
+* <b>config</b>: Where you place all application configuration for all environments
+  * <b>environment.rb</b>: Defines the runtime environment for all code, requiring and configuring all Wukong framework code.  You shouldn't have to edit this file directly.
+  * <b>application.rb</b>: Require and configure libraries specific to your application.  Choose a model framework, pick what application code gets loaded by default (vs. auto-loaded).
+  * <b>initializers</b>: Holds any files you need to load before <b>application.rb</b> here.  Useful for requiring and configuring external libraries.
+  * <b>settings.yml</b>: Defines application-wide settings.
+  * <b>environments</b>: Defines environment-specific settings in YAML files named after the environment.  Overrides <b>config/settings.yml</b>.
+* <b>data</b>: Holds sample data in flat files.  You'll develop and test your application using this data.
+* <b>Gemfile</b> and <b>Gemfile.lock</b>: Defines how libraries are resolved with [Bundler](http://gembundler.com/).
+* <b>lib</b>: Holds any code you want to use in your application but that isn't "part of" your application (like vendored libraries, Rake tasks, &c.).
+* <b>log</b>: A good place to stash logs.
+* <b>Rakefile</b>: Defines [Rake](http://rake.rubyforge.org/) tasks for the development, test, and deploy of your application.
+* <b>spec</b>: Holds all your [RSpec](http://rspec.info/) unit tests.
+  * <b>spec_helper.rb</b>: Loads libraries you'll use during testing, includes spec helper libraries from Wukong.
+  * <b>support</b>: Holds support code for your tests.
+* <b>tmp</b>: A good place to stash temporary files.

data/Rakefile

@@ -0,0 +1,10 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:specs)
+
+require 'yard'
+YARD::Rake::YardocTask.new
+
+task :default => [:specs]

data/bin/wu-deploy

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require 'wukong-deploy'
+settings = Wukong::Deploy::Configuration
+require 'wukong/boot'
+Wukong.boot!(settings)
+Wukong::Deploy::Driver.run(settings, *settings.rest)

data/lib/wukong-deploy.rb

@@ -0,0 +1,26 @@
+# Require Wukong so we can write processors & dataflows and use
+# build-in widgets.
+require 'wukong'
+
+# Require just enough of wukong-hadoop to give us changes at the
+# processor level.  The rest is only required when running the
+# 'wu-hadoop' program.  Also grab configuration since we'll need it.
+require 'wukong-hadoop/extensions'
+require 'wukong-hadoop/configuration'
+
+# Grab configuration from Wonderdog.
+require 'wonderdog'
+
+module Wukong
+  # Provides some helper methods for loading the assets within a
+  # deploy pack at runtime and for Wukong plugins to ask questions
+  # about the deploy pack they're running in.
+  module Deploy
+  end
+end
+
+
+# Now load the rest of the deploy pack.
+require 'wukong-deploy/deploy_pack'
+require 'wukong-deploy/configuration'
+require 'wukong-deploy/driver'

data/lib/wukong-deploy/configuration.rb

@@ -0,0 +1,23 @@
+module Wukong
+  module Deploy
+
+    # Pass in +settings+ that need to be modified for deploy-specific
+    # settings *before* they are resolved.
+    #
+    # @param [Configliere::Param] settings
+    # @return [Configliere::Param] the newly configured settings
+    def self.configure settings
+      settings.define(:environment, :description => "The environment to run in", :default => 'development', :flag => 'e', :env_var => 'ICS_PLATFORM_ENV')
+      
+      case executable
+      when 'wu-hadoop'
+        Wukong::Elasticsearch.configure(settings) if executable == 'wu-hadoop'
+      end
+      settings
+    end
+
+    # All configuration for Wukong::Deploy lives in this object.
+    Configuration = configure(Configliere::Param.new) unless defined?(Configuration)
+  end
+
+end

data/lib/wukong-deploy/console.rb

@@ -0,0 +1,15 @@
+require 'irb'
+module Wukong
+  module Deploy
+    class Console
+
+      def initialize
+      end
+
+      def run!
+        ARGV.clear
+        IRB.start
+      end
+    end
+  end
+end

data/lib/wukong-deploy/deploy_pack.rb

@@ -0,0 +1,151 @@
+module Wukong
+  module Deploy
+
+    # Boot the deploy pack, reading all available +settings+ and
+    # rooting it at the given +path+.
+    #
+    # @param [Configliere::Param] settings
+    # @param [String] path
+    def self.boot! settings, path
+      @pre_deploy_settings = settings.dup
+      @settings            = settings
+      @root                = Pathname.new(path)
+      read_common_settings
+      read_environment_settings
+      read_remote_settings
+    end
+
+    # Return the root directory of this deploy pack.
+    #
+    # @return [Pathname]
+    def self.root
+      @root
+    end
+
+    # Recursively require each Ruby file +dir+.
+    #
+    # @example Requiring all .rb files anywhere within /lib/my_lib
+    #
+    #   Wukong::Deploy.require_recursive("lib/my_lib")
+    #
+    # @param [String] glob
+    def self.require_recursive glob
+      Dir[root.join("#{glob}/**/*.rb")].each { |path| require(path) }
+    end
+    
+    # Return the name of the executable program currently running.
+    #
+    # @return [String]
+    def self.executable
+      File.basename($0)
+    end
+
+    # Return the current environment the deploy pack is in.
+    #
+    # @return [String]
+    def self.environment
+      settings[:environment]
+    end
+
+    # Return the deploy pack's own settings.
+    #
+    # @return [Configliere::Param]
+    def self.settings
+      @settings
+    end
+
+    # Return the settings the deploy pack had *before* it booted
+    # itself -- these are used to pass to other tools when invoking
+    # them, since they will read the deploy pack settings themselves
+    # anyway.
+    #
+    # @return [Configliere::Param]
+    def self.pre_deploy_settings
+      @pre_deploy_settings
+    end
+
+    # The root directory of this deploy pack.
+    #
+    # @return [Pathname]
+    def self.root
+      @root
+    end
+
+    # The directory this deploy pack keeps local, sample data in.
+    #
+    # @return [Pathname]
+    def self.data_dir
+      root.join('data')
+    end
+
+    # The directory this deploy pack uses for configuration files.
+    #
+    # @return [Pathname]
+    def self.config_dir
+      root.join('config')
+    end
+
+    # The directory this deploy pack uses for temporary files.
+    #
+    # @return [Pathname]
+    def self.tmp_dir
+      root.join('tmp')
+    end
+
+    # The directory this deploy pack uses for logs.
+    #
+    # @return [Pathname]
+    def self.log_dir
+      root.join('log')
+    end
+
+    # The diretory this deploy pack puts all its application code in.
+    #
+    # @return [Pathname]
+    def self.app_dir
+      root.join('app')
+    end
+
+    # The logger for this deploy pack.
+    #
+    # @return [Logger]
+    def self.logger
+      return @logger if @logger
+      # FIXME -- want to use the Wukong logger here...
+      require 'logger'
+      @logger ||= defined?(Log) ? Log : Logger.new
+    end
+    
+    private
+
+    # Read settings common across all environments from
+    # config/settings.yml.
+    def self.read_common_settings
+      read_settings_from_file(settings, config_dir.join("settings.yml"))
+    end
+    
+    # Read settings unique to this deploy pack's current environment
+    # from config/ENVIRONMENT.yml.
+    def self.read_environment_settings
+      read_settings_from_file(settings, config_dir.join("environments", "#{environment}.yml"))
+    end
+
+    # Update +settings+ with the configuration at the given +path+.
+    #
+    # @param [Configliere::Param] settings
+    # @param [String, Pathname] path
+    def self.read_settings_from_file settings, path
+      if File.exist?(path) && File.readable?(path) && File.file?(path)
+        settings.read(path)
+      else
+        logger.warn("Cannot read settings file at #{path}.")
+      end
+    end
+
+    # Read remote settings.
+    #
+    # FIXME -- not implemented yet.
+    def self.read_remote_settings
+    end
+  end
+end

data/lib/wukong-deploy/driver.rb

@@ -0,0 +1,41 @@
+
+
+module Wukong
+  module Deploy
+    class Driver
+
+      attr_accessor :settings
+      attr_accessor :args
+
+      def self.run(settings, *args)
+        begin
+          new(settings,*args).run!
+        rescue Wukong::Error => e
+          $stderr.puts(e.message)
+          exit(1)
+        end
+      end
+
+      def run!
+        case args.first
+        when 'new'
+          require_relative('templater')
+          raise Error.new("Must provide a path to the root of the deploy pack you want to create") if args[1].nil? || args[1].blank?
+          Templater.new(File.expand_path(args[1], Dir.pwd)).create
+        when 'console'
+          require_relative('console')
+          Wukong::Deploy::Console.new.run!
+        else
+          settings.dump_help
+          exit(2)
+        end
+      end
+
+      def initialize settings, *args
+        self.settings = settings
+        self.args     = args
+      end
+      
+    end
+  end
+end

data/lib/wukong-deploy/repo.rb

@@ -0,0 +1,66 @@
+module Wukong
+  module Deploy
+    class Repo
+
+      TOP_LEVEL_DIR_NAMES = %w[app config data lib log script spec tmp]
+      
+      attr_reader :root
+
+      def initialize root
+        @root = Pathname.new(root)
+      end
+
+      TOP_LEVEL_DIR_NAMES.each do |name|
+        define_method("#{name}_dir") do |*args|
+          root.join(name, *args)
+        end
+      end
+
+      def dirs_to_create
+        [
+         root,
+         app_dir('models'),
+         app_dir('processors'),
+         app_dir('flows'),
+         app_dir('jobs'),
+         
+         config_dir('environments'),
+         config_dir('initializers'),
+         
+         data_dir,
+         lib_dir,
+         log_dir,
+         script_dir,
+         
+         spec_dir('models'),
+         spec_dir('processors'),
+         spec_dir('flows'),
+         spec_dir('jobs'),
+         spec_dir('support'),
+         
+         tmp_dir
+        ]
+      end
+
+      def files_to_template
+        %w[
+         config/boot.rb
+         config/application.rb
+         config/environment.rb
+         config/settings.yml
+         config/environments/development.yml
+         config/environments/test.yml
+         config/environments/production.yml
+
+         Gemfile
+         Rakefile
+         README.md
+
+         spec/spec_helper.rb
+         ]
+      end
+      
+    end
+  end
+end
+

data/lib/wukong-deploy/tasks.rb

@@ -0,0 +1,13 @@
+require 'rake'
+
+namespace 'bundler' do
+  desc "Remove local bundle repository"
+  task :clean do
+    Wukong::Deploy.logger.info "Removing local bundle at /.bundle and /bundle"
+    sh "rm -rf .bundle bundle"
+    Wukong::Deploy.logger.info "Make sure to run 'bundle install --standalone' to re-create the bundle"
+  end
+  
+end
+
+  

data/lib/wukong-deploy/templater.rb

@@ -0,0 +1,65 @@
+require 'pathname'
+require 'fileutils'
+require 'erubis'
+
+require_relative('repo')
+
+module Wukong
+  module Deploy
+    class Templater
+
+      attr_accessor :repo
+      attr_accessor :options
+
+      include FileUtils::Verbose
+
+      def initialize root, options={}
+        self.repo    = Repo.new(root)
+        self.options = options
+      end
+
+      def create
+        create_dirs
+        create_templates
+        create_gitkeeps
+        create_gitignore
+      end
+
+      def create_dirs
+        repo.dirs_to_create.each { |dir| mkdir_p(dir) }
+      end
+
+      def create_templates
+        repo.files_to_template.each do |repo_path|
+          create_template(templates_dir.join("#{repo_path}.erb"), repo.root.join(repo_path))
+        end
+      end
+
+      def create_template input_path, output_path, binding={}
+        input  = File.read(input_path)
+        erb    = Erubis::Eruby.new(input)
+        output = erb.result(binding)
+        action = File.exist?(output_path) ? 'modify' : 'create'
+        puts "#{action} #{output_path}"
+        File.open(output_path, 'w') { |f| f.puts(output) }
+      end
+      
+      def create_gitkeeps
+        repo.dirs_to_create.each do |dir|
+          if Dir[File.join(dir, '*')].empty?
+            touch(File.join(dir, '.gitkeep'))
+          end
+        end
+      end
+
+      def create_gitignore
+        create_template(templates_dir.join('gitignore'), repo.root.join('.gitignore'))
+      end
+
+      def templates_dir
+        @templates_dir ||= Pathname.new(File.expand_path('../../../templates', __FILE__))
+      end
+      
+    end
+  end
+end

data/lib/wukong-deploy/version.rb

@@ -0,0 +1,7 @@
+module Wukong
+  module Deploy
+
+    # The current version of this deploy pack framework.
+    VERSION = '0.0.1'
+  end
+end

data/templates/Gemfile.erb

@@ -0,0 +1,40 @@
+# Use the public [RubyGems server](http://rubygems.org/)
+source                 :rubygems
+
+# = Application
+# 
+# The following dependencies are for your application.
+#
+# Does one of your processors require the UserAgent library because
+# it's parsing HTTP User-Agent strings? Just insert
+#
+#   gem 'useragent'
+#
+# here and then remember to run
+#
+#   $ bundle install --standalone
+#
+# Here's a link to the [Bundler documentation](http://gembundler.com/)
+# for your convenience.
+
+
+
+
+
+
+#
+# = Framework
+#
+# The following libraries are part of the deploy pack framework.  You
+# shouldn't need to change these.
+#
+
+gem 'wukong-deploy',          :git => 'https://github.com/infochimps-labs/wukong-deploy'
+gem 'wukong',                 :git => 'https://github.com/infochimps-labs/wukong', :branch => '3.0.0'
+gem 'wukong-hadoop',          :git => 'https://github.com/infochimps-labs/wukong-hadoop'
+gem 'wonderdog',              :git => 'https://github.com/infochimps-labs/wonderdog'
+gem 'vayacondios-client',     :git => 'https://github.com/infochimps-labs/vayacondios'
+
+group :development do
+  gem 'rspec'
+end

data/templates/README.md.erb

@@ -0,0 +1 @@
+Welcome to your new deploy pack.

data/templates/Rakefile.erb

@@ -0,0 +1,10 @@
+require_relative 'config/environment'
+require 'wukong-deploy/tasks'
+require 'rake'
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)
+
+Wukong::Deploy.require_recursive('lib/tasks')
+
+task :default => :spec

data/templates/config/application.rb.erb

@@ -0,0 +1,66 @@
+# = config/application.rb
+#
+# Loads the deploy pack application and all supporting libraries
+# including Wukong and any of its plugins.
+#
+# The load order is as follows:
+#
+# * [Wukong-Deploy](http://github.com/infochimps-labs/wukong-deploy) which loads all Wukong and Wukong plugins.
+# * Any code you put in the `config/initializers` directory.  This is a good place to configure additional libraries to support your application.
+# * Your application code itself, in the following order
+#   * models
+#   * processors
+#   * flows
+#   * jobs
+#   * anything else
+# 
+# You can put code you want available but you don't intend to use in
+# your application inside the `lib` directory.
+
+#
+# == Deploy Pack Framework
+#
+
+# Boot up first
+require_relative("boot")
+
+# Require the Wukong-Deploy plugin which will itself require and
+# configure core Wukong and other plugins.
+require 'wukong-deploy'
+
+# Load initializers containing any requires or configuration code
+# needed by the app.
+Dir[File.dirname(__FILE__) + '/initializers/**/*.rb'].each { |path| require(path) }
+
+#
+# == Model Support
+#
+# We're going to use Gorillib, Infochimps' own library of useful
+# methods and Ruby hacks.
+require 'gorillib'
+require 'gorillib/model/serialization'
+require 'gorillib/object/blank'
+
+#
+# But you could use a more Rails-y support stack
+# 
+# require 'active_support'
+
+#
+# Or DataMapper
+# 
+# require 'dm-core'
+
+#
+# == Application Structure
+#
+#
+# By default, we try not to be clever and just load *everything* in
+# app/.  If you have a lot of application code, this may be slow on
+# startup and you may want to only load common classes or define
+# autoloads or whatever you want to do.
+Dir[File.expand_path('../../app/models/**/*.rb', __FILE__)].each     { |path| require(path) }
+Dir[File.expand_path('../../app/processors/**/*.rb', __FILE__)].each { |path| require(path) }
+Dir[File.expand_path('../../app/flows/**/*.rb', __FILE__)].each      { |path| require(path) }
+Dir[File.expand_path('../../app/jobs/**/*.rb', __FILE__)].each       { |path| require(path) }
+Dir[File.expand_path('../../app/**/*.rb', __FILE__)].each            { |path| require(path) }

data/templates/config/boot.rb.erb

@@ -0,0 +1,21 @@
+# = config/boot.rb
+#
+# Loads the Ruby language framework and activates Bundler.
+#
+# We use a standalone bundle installation becaues it makes it easier
+# to have non-Ruby code in underlying frameworks interact with the
+# deploy pack.
+#
+# Make sure you install the bundle locally with the following command:
+#
+# @example Install a standalone bundle of gems
+#
+#   $ bundle install --standalone
+
+setup_path = File.expand_path("../../bundle/bundler/setup.rb", __FILE__)
+begin
+  require setup_path
+rescue LoadError => e
+  $stderr.puts("Failed to require \"#{setup_path}\".  Try running\n\n  bundle install --standalone\n\n")
+  exit(1)
+end

data/templates/config/environment.rb.erb

@@ -0,0 +1,8 @@
+# = config/environment.rb
+#
+# Loads all the Ruby language and framework code that defines the
+# runtime environment of your application.
+
+require_relative("application")
+
+Wukong::Deploy.boot!(Wukong::Deploy::Configuration, File.expand_path("../../", __FILE__))

data/templates/config/environments/development.yml.erb

@@ -0,0 +1,8 @@
+---
+
+#
+# Put any development specific settings here.
+#
+
+notifier:
+  type: log

data/templates/config/environments/production.yml.erb

@@ -0,0 +1,10 @@
+---
+
+#
+# Put any production specific settings here.
+#
+
+notifier:
+  type: http
+  host: api.INSERT_ACCOUNT_NAME_HERE.chimpy.us
+  port: 9000

data/templates/config/environments/test.yml.erb

@@ -0,0 +1,8 @@
+---
+
+#
+# Put any test specific settings here.
+#
+
+notifier:
+  type: none

data/templates/config/settings.yml.erb

@@ -0,0 +1,54 @@
+---
+
+# = config/settings.yml
+#
+# Any settings defined in this file will be merged into a global
+# Settings object within any of the wu-tools which run inside this
+# deploy pack.
+#
+# Settings will additionally be loaded from a file named after the
+# ICS_PLATFORM_ENV environment variable and will override those
+# declared here. The 'development' environment is the default
+# environment loaded.
+#
+# Any settings specified on the command line when invoking wu-tools
+# will take precedence over settings within any of these files.
+#
+
+
+#
+# == Notifier
+#
+# Notifiers are used within a Processor for out of band event
+# processing.
+notifier:
+
+  # The type of notifier to use is better set on a per-environment
+  # basis.  See the correspdoning YAML files in config/environments.
+  # type: log
+
+  #
+  # === Log notifier
+  #
+  # The log notifier will print each event to the log.
+
+  # A path to a log file.
+  # log: log/wu.log
+
+  # The priority level of the log
+  # level: info
+
+  #
+  # === HTTP notifier
+  #
+  # The HTTP notifier will send events over HTTP to an instance of the
+  # Infochimps Platform API
+
+  # The host to send each event to.  Your host is typically determined
+  # by the name of your platform account.
+  # host: api.INSERT_ACCOUNT_NAME_HERE.chimpy.us
+
+  # The port to send each event to.
+  # port: 9000
+
+# You can put processor-specfic settings within this file as well.

data/templates/gitignore

@@ -0,0 +1,39 @@
+# version control
+.svn
+.hg
+.bzr
+
+# editors
+\#*
+.#*
+\.#*
+*~
+*.tmproj
+*._flymake.*
+*.project
+
+# os x
+.DS_Store
+Icon?
+
+# backup
+.bak
+
+# tools
+TAGS*
+nohop.out
+a.out
+*.pyc
+*.so
+*.stackdump
+.yardoc
+
+# commonly ignored files
+/**/*DO_NOT_VERSION*
+/**/log/*
+/**/tmp/*
+/doc/*
+/pkg/*
+
+.bundle
+bundle/*

data/templates/spec/spec_helper.rb.erb

@@ -0,0 +1,13 @@
+ENV["ICS_PLATFORM_ENV"] ||= "test"
+
+require_relative('../config/environment')
+
+# Require spec helpers from Wukong
+require 'wukong/spec_helpers'
+
+Dir[File.expand_path('../support/**/*.rb', __FILE__)].each { |path| require path }
+
+RSpec.configure do |config|
+  config.mock_with :rspec
+  include Wukong::SpecHelpers
+end

data/wukong-deploy.gemspec

@@ -0,0 +1,47 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/wukong-deploy/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.name        = 'wukong-deploy'
+  gem.homepage    = 'https://github.com/infochimps-labs/wukong-deploy'
+  gem.licenses    = ["Apache 2.0"]
+  gem.email       = 'coders@infochimps.com'
+  gem.authors     = ['Infochimps', 'Dhruv Bansal', 'Travis Dempsey']
+  gem.version     = Wukong::Deploy::VERSION
+
+  gem.summary     = 'Defines the deploy pack framework used by the Infochimps Platform'
+  gem.description = <<-EOF
+  The Infochimps Platform is an end-to-end, managed solution for
+  building Big Data applications.  It integrates best-of-breed
+  technologies like Hadoop, Storm, Kafka, MongoDB, ElasticSearch,
+  HBase, &c. and provides simple interfaces for accessing these
+  powerful tools.
+
+  Computation, analytics, scripting, &c. are all handled by Wukong
+  within the platform.  Wukong is an abstract framework for defining
+  computations on data.  Wukong processors and flows can run in many
+  different execution contexts including:
+
+    * locally on the command-line for testing or development purposes
+    * as a Hadoop mapper or reducer for batch analytics or ETL
+    * within Storm as part of a real-time data flow
+
+  The Infochimps Platform uses the concept of a deploy pack for
+  developers to develop all their processors, flows, and jobs within.
+  EOF
+
+  gem.files         = `git ls-files`.split("\n")
+  gem.executables   = ['wu-deploy']
+  gem.test_files    = gem.files.grep(/^spec/)
+  gem.require_paths = ['lib']
+
+  gem.add_dependency('wukong', '3.0.0.pre2')
+  gem.add_dependency('wukong-hadoop')
+  gem.add_dependency('wonderdog')
+  gem.add_dependency('erubis')
+  
+  gem.add_dependency('rake',     '~> 0.9')
+  gem.add_development_dependency 'rspec',    '~> 2'
+  
+end
+

metadata

@@ -0,0 +1,183 @@
+--- !ruby/object:Gem::Specification
+name: wukong-deploy
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Infochimps
+- Dhruv Bansal
+- Travis Dempsey
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-12-01 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: wukong
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 3.0.0.pre2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 3.0.0.pre2
+- !ruby/object:Gem::Dependency
+  name: wukong-hadoop
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: wonderdog
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: erubis
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.9'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2'
+description: ! "  The Infochimps Platform is an end-to-end, managed solution for\n
+  \ building Big Data applications.  It integrates best-of-breed\n  technologies like
+  Hadoop, Storm, Kafka, MongoDB, ElasticSearch,\n  HBase, &c. and provides simple
+  interfaces for accessing these\n  powerful tools.\n\n  Computation, analytics, scripting,
+  &c. are all handled by Wukong\n  within the platform.  Wukong is an abstract framework
+  for defining\n  computations on data.  Wukong processors and flows can run in many\n
+  \ different execution contexts including:\n\n    * locally on the command-line for
+  testing or development purposes\n    * as a Hadoop mapper or reducer for batch analytics
+  or ETL\n    * within Storm as part of a real-time data flow\n\n  The Infochimps
+  Platform uses the concept of a deploy pack for\n  developers to develop all their
+  processors, flows, and jobs within.\n"
+email: coders@infochimps.com
+executables:
+- wu-deploy
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .rspec
+- Gemfile
+- README.md
+- Rakefile
+- bin/wu-deploy
+- lib/wukong-deploy.rb
+- lib/wukong-deploy/configuration.rb
+- lib/wukong-deploy/console.rb
+- lib/wukong-deploy/deploy_pack.rb
+- lib/wukong-deploy/driver.rb
+- lib/wukong-deploy/repo.rb
+- lib/wukong-deploy/tasks.rb
+- lib/wukong-deploy/templater.rb
+- lib/wukong-deploy/version.rb
+- templates/Gemfile.erb
+- templates/README.md.erb
+- templates/Rakefile.erb
+- templates/config/application.rb.erb
+- templates/config/boot.rb.erb
+- templates/config/environment.rb.erb
+- templates/config/environments/development.yml.erb
+- templates/config/environments/production.yml.erb
+- templates/config/environments/test.yml.erb
+- templates/config/settings.yml.erb
+- templates/gitignore
+- templates/spec/spec_helper.rb.erb
+- wukong-deploy.gemspec
+homepage: https://github.com/infochimps-labs/wukong-deploy
+licenses:
+- Apache 2.0
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: Defines the deploy pack framework used by the Infochimps Platform
+test_files: []
+has_rdoc: