hackboxen 0.1.0 → 0.1.1

data/CHANGELOG.textile

@@ -16,7 +16,7 @@ h3. Deprecations/Changes
 
 * Much old code was refactored or removed.
 
-h3. New Functionality
+h3. New Functionality (version 0.1.0)
 
 * Default Hackboxen paths can be accessed by using @path_to(:fixd_dir)@. See HackBoxen::Paths for using/adding others.
 
@@ -30,15 +30,21 @@ h3. New Functionality
 
 * A binary executable has been added, @hb-scaffold@ that can be run from anywhere and is designed to replace the rake task.
 
-h3. Still Needing
+h3. Version 0.1.1
 
-* Make Hackboxen a gem. This will require the separation of the actual hackbox code from the hackboxen library (not done yet) and the creation of a coderoot to connect the hackbox library with the other code (implemented).
+* Make Hackboxen a gem. This will require the separation of the actual hackbox code from the hackboxen library (implemented) and the creation of a coderoot to connect the hackbox library with the other code (implemented).
 
 * When Hackboxen is a gem, its version can be added to the requires hash in a hackbox @config.yaml@ so we can keep track of potentially breaking changes to legacy code.
 
+* The separation (completely) of a @config.yaml@ from an @icss.yaml@. (implemented)
+
+* A hackbox __name__ is now separate from a dataset __protocol__. Namespaces are still shared for organizational purposes, but a protocol directory is no longer created for the output. 
+
+h3. Still Needing
+
 * Full spec coverage for the hackboxen library.
 
 * Implementation of @'hb:mini'@ and @ConfigValidator@. Some of this code is written, but it needs to be fleshed out and decided upon.
 
-* The separation (completely) of a @config.yaml@ from an @icss.yaml@. Would not affect hackbox running much.
+
 

data/README.textile

@@ -14,9 +14,9 @@ h2. Hackbox Engine
 A hackbox engine contains:
 
 * @Rakefile@: **(required)** Used to read and combine all the sources of config metadata and execute @main@.
-* @Gemfile@: **(optional)** A list of gems necessary for thsi hackbox to run. Processed automatically by "Bundler":https://github.com/carlhuda/bundler.
+* @Gemfile@: **(optional)** A list of gems necessary for this hackbox to run. Processed automatically by "Bundler":https://github.com/carlhuda/bundler.
 * @config/@: **(required)** A subdirectory containing:
-** @config.yaml@ **(required)** A dataset specific default configuration YAML file.
+** @config.yaml@ **(required)** A hackbox specific default configuration YAML file.
 ** @protocol.icss.yaml@ **(optional)** An "Icss":http://github.com/infochimps/icss schema file describing the output data and publishing targets.
 * @engine/@: **(required)** A subdirectory containing:
 ** @main@: **(required)** An executable data processing file. This may be written in any language.
@@ -46,28 +46,27 @@ The hackbox output directory is where all of the data that a hackbox acquires, r
 └── language
     └── corpora
         └── word_freq
-            └── bnc
                 ├── fixd
                 │   ├── code
                 │   │   └── bnc_endpoint.rb
-                │   ├── data
-                │   │   └── bnc_fixd_data.tsv
-                │   └── env
-                │       └── working_environment.json         
+                │   └── data
+                │       └── bnc_fixd_data.tsv
                 ├── log
                 │   └── bnc_run_0.log
                 ├── rawd
                 │   └── bnc_data_in_process
                 ├── ripd
                 │   └── bnc_download.zip
-                └── tmp
+                └── env
+                    └── working_environment.json         
 </code></pre>
 
+* @fixd/@: **(required)** See the output interface described below.
 * @log/@:  **(optional)** All logging from a hackbox run goes here.
-* @tmp/@:  **(optional)** If needed, any truly ephemeral output of the workflow should go here.
-* @ripd/@: **(required)** This will contain virginal downloaded source data adhering to the directory structure from which it was pulled.
 * @rawd/@: **(optional)** This will contain all intermediate data processing outputs.
-* @fixd/@: **(required)** See the output interface described below.
+* @ripd/@: **(required)** This will contain virginal downloaded source data adhering to the directory structure from which it was pulled.
+* @env/@:  **(required)** This directory contains a file describing the environment in which the hackbox was run.
+** @working_environment.json@: **(required)** All runtime config metadata used to generate the schema and output data.
 
 Engine and output directories are generally created dynamically and are not meant to be archival.
 
@@ -75,11 +74,9 @@ h3. Output Interface (fixd/)
 
 @fixd/@ is the final output directory and contains the following:
 
-* @env/@: **(required)** This directory contains a file describing the environment in which the hackbox was run.
-** @working_environment.json@: **(required)** All runtime config metadata used to generate the schema and output data.
+* @protocol.icss.json@: **(required)** An "Icss":http://github.com/infochimps/icss schema file describing its respective dataset. 
 * @code/@: **(optional)** A directory containing the code assets described in the icss. 
 * @data/@: **(required)** A directory containing a single dataset or subdirectories named for each dataset. Each contains:
-** @protocol.icss.json@: **(required)** An "Icss":http://github.com/infochimps/icss schema file describing its respective dataset. 
 ** **(required)** One or more data files that collectively adhere to the schema of this dataset.
 
 h2. Hackbox Configuration
@@ -97,27 +94,16 @@ h1. Getting Started
 
 Here are the general guidelines for creating your own hackbox.
 
-h3. Hackboxen Dependencies
-
-Clone the Hackboxen repo:
+h3. Install Hackboxen
 
-<pre><code>git clone git@github.com:infochimps/hackboxen.git
+<pre><code>sudo gem install hackboxen
 </code></pre>
 
-Add Hackboxen to your $RUBYLIB:
+This will create a @.hackbox@ directory with a @hackbox.yaml@ file that contains default values for @coderoot@, @dataroot@, @s3_filesystem@, @os@, and @machine@. The @hb-install@ command has optional arguments @--dataroot=@, @--coderoot=@.
 
-<pre><code>export RUBYLIB=$RUBYLIB:/path/to/hackboxen/lib
+<pre><code>hb-install # optionally: hb-install --dataroot=/data/hb --coderoot=/code/hb
 </code></pre>
 
-Install Hackboxen dependencies:
-
-<pre><code>cd hackboxen
-sudo bundle install
-rake install # optionally: rake install -- --dataroot=/data/hb --coderoot=/code/hb
-</code></pre>
-
-This will install the following gems: "configliere":http://github.com/mrflip/configliere, "icss":http://github.com/infochimps/icss, "swineherd":http://github.com/ganglion/swineherd, and "rake":http://github.com/jimweirich/rake. This will also create a @.hackbox@ directory with a @hackbox.yaml@ file that contains default values for @coderoot@, @dataroot@, @s3_filesystem@, @os@, and @machine@. The @rake install@ command has optional arguments @--dataroot=@, @--coderoot=@.
-
 A default @hackbox.yaml@ file:
 
 <pre><code>---
@@ -132,11 +118,15 @@ requires:
   os: darwin
 </code></pre>
 
+h3. Hackboxen Dependencies
+
+The Hackboxen library depends on the following gems: "configliere":http://github.com/mrflip/configliere, "icss":http://github.com/infochimps/icss, "swineherd":http://github.com/ganglion/swineherd, and "rake":http://github.com/jimweirich/rake. 
+
 h3. Creating a Hackbox
 
-Hackboxen comes with scaffold task that creates a template hackbox for you. Required arguments are @--namespace=@ and @--protocol=@. Optional arguments are @--targets=@, @--s3access=@, and @--s3secret=@.
+Hackboxen comes with scaffold task that creates a template hackbox for you. Required arguments are @--namespace=@ and @--protocol=@. Optional arguments are @--targets=@.
 
-<pre><code>hb-scaffold --namespace=foo.bar --protocol --targets=catalog,mysql
+<pre><code>hb-scaffold --namespace=foo.bar.baz --protocol=qux --targets=catalog,mysql
 </code></pre>
 
 This will create the following directories and files:
@@ -145,13 +135,14 @@ This will create the following directories and files:
 └── foo
     └── bar
         └── baz
-            ├── config
-            │   ├── config.yaml
-            │   └── baz.icss.yaml                 
-            ├── engine
-            │   ├── main
-            │   └── baz_endpoint.rb
-            └── Rakefile
+            └── qux
+                ├── config
+                │   ├── config.yaml
+                │   └── qux.icss.yaml                 
+                ├── engine
+                │   ├── main
+                │   └── qux_endpoint.rb
+                └── Rakefile
 </code></pre>
 
 h3. Running a hackbox

data/VERSION

@@ -1 +1 @@
-0.1.0
+0.1.1

data/hackboxen.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{hackboxen}
-  s.version = "0.1.0"
+  s.version = "0.1.1"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["kornypoet", "Ganglion", "bollacker"]
-  s.date = %q{2011-06-30}
+  s.date = %q{2011-07-06}
   s.description = %q{A simple framework to assist in standardizing the data-munging input/output process.}
   s.email = %q{travis@infochimps.com}
   s.executables = ["hb-install", "hb-scaffold", "hb-runner"]

data/lib/hackboxen/tasks/icss.rb

@@ -6,7 +6,7 @@ module HackBoxen
       icss_yaml   = File.join(path_to(:hb_config), "#{WorkingConfig[:protocol]}.icss.yaml")
       if File.exists? icss_yaml
         icss      = YAML.load(File.read icss_yaml)
-        icss_json = File.join(path_to(:data_dir), "#{WorkingConfig[:protocol]}.icss.json")
+        icss_json = File.join(path_to(:fixd_dir), "#{WorkingConfig[:protocol]}.icss.json")
         HackBoxen.current_fs.open(File.join(icss_json), 'w') { |f| f.puts icss.to_json }
       end
     end

data/lib/hackboxen/tasks/init.rb

@@ -13,6 +13,15 @@ module HackBoxen
 
     desc "Create the required output directories using the filesystem specified by the WorkingConfig"
     task :create_required_paths => [:validate_working_config] do
+      output_dirs = [
+        path_to(:hb_dataroot),
+        path_to(:env_dir),
+        path_to(:fixd_dir),
+        path_to(:log_dir),
+        path_to(:ripd_dir),
+        path_to(:rawd_dir),
+        path_to(:data_dir)
+      ]
       output_dirs.each { |dir| HackBoxen.current_fs.mkpath(dir) unless HackBoxen.current_fs.exists? dir }
     end
 

data/lib/hackboxen/utils.rb

@@ -32,12 +32,25 @@ module HackBoxen
     Swineherd::FileSystem.get fs
   end
 
+  def self.name
+    hackbox_root? ? Dir.pwd.gsub(/#{WorkingConfig[:coderoot]}\/#{current}\//, '') : 'debug'
+  end
+
   def self.current
-    hackbox_root? ? File.join(WorkingConfig[:namespace].gsub('.', '/'), WorkingConfig[:protocol]) : 'debug'
+    hackbox_root? ? WorkingConfig[:namespace].gsub('.', '/') : Dir.pwd
+  end
+
+  def self.coderoot
+    WorkingConfig[:coderoot]
   end
 
+  def self.dataroot
+    WorkingConfig[:dataroot]
+  end
+
+
   def self.verify_dependencies
-    %w[ dataroot namespace protocol ].each do |req|
+    %w[ coderoot dataroot namespace ].each do |req|
       raise "Your hackbox config appears to be missing a [#{req}]" unless WorkingConfig[req.to_sym]
     end
   end

data/lib/hackboxen/utils/paths.rb

@@ -13,26 +13,14 @@ module HackBoxen
       path.absolute? ? File.expand_path(path) : path.to_s
     end
 
-    def output_dirs
-      [
-        path_to(:hb_dataroot),
-        path_to(:env_dir),
-        path_to(:fixd_dir),
-        path_to(:log_dir),
-        path_to(:ripd_dir),
-        path_to(:rawd_dir),
-        path_to(:data_dir)
-      ]
-    end
-
     def default_paths
       {
         # root directories
         :home        => ENV['HOME'],
-        :data_root   => WorkingConfig[:dataroot],
-        :code_root   => WorkingConfig[:coderoot],
+        :data_root   => HackBoxen.dataroot,
+        :code_root   => HackBoxen.coderoot,
         # local hackbox directories
-        :hb_current  => [:code_root, HackBoxen.current],
+        :hb_current  => [:code_root, HackBoxen.current, HackBoxen.name],
         :hb_engine   => [:hb_current, 'engine'],
         :hb_config   => [:hb_current, 'config'],
         # output directories

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: hackboxen
 version: !ruby/object:Gem::Version 
-  hash: 27
+  hash: 25
   prerelease: false
   segments: 
   - 0
   - 1
-  - 0
-  version: 0.1.0
+  - 1
+  version: 0.1.1
 platform: ruby
 authors: 
 - kornypoet
@@ -17,7 +17,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-06-30 00:00:00 -05:00
+date: 2011-07-06 00:00:00 -05:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency