hasta 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: adeafa9f4dae6acb9dd59dc39432d4e4b3d86b79
+  data.tar.gz: 3f15a082f3c72d9a75df32e728cff7cc0ff88787
+SHA512:
+  metadata.gz: 0e0dc797ebb1acab0781324a47ed6184f51797a1b3893da9ac069024dc9cfadd56f6dcb5c1338855e59c1c0a82745c601102fd62e4de58a198ed278332ac5f24
+  data.tar.gz: 84a5056c2c069512a99eb8b4d9b58c9dcb0827732ffe555f7708705da0579706be7659abbfc6f4b97e4e458b02e01d2a25c38489c240e0a166d1e2b316d87132

data/.cane

@@ -0,0 +1 @@
+--style-measure 100

data/.gitignore

@@ -0,0 +1,3 @@
+.bundle
+Gemfile.lock
+vendor/bundle

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in hasta.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2014 Swipely, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,150 @@
+# Hasta
+
+<b>HA</b>doop <b>S</b>treaming <b>T</b>est h<b>A</b>rness for Amazon EMR
+
+A test harness for running [Hadoop Streaming](http://hadoop.apache.org/docs/r1.2.1/streaming.html) jobs written in Ruby without running Hadoop.
+The test harness understands the [Amazon Data Pipeline](http://aws.amazon.com/datapipeline/) and [Elastic Map Reduce](http://aws.amazon.com/elasticmapreduce/) (EMR) JSON definition format and can automatically parse the details of a job out of Data Pipeline configuration file.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'hasta'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install hasta
+
+## Usage
+
+1. Require the Hasta Rake tasks in your project's Rakefile
+
+   ```ruby
+   require 'hasta/tasks'
+   ```
+
+2. Add the `Hasta::Tasks:Runner` task to your project's Rakefile
+
+  ```ruby
+  Hasta::Tasks::Runner.new do |task, opts|
+    task.definition_file = <path-to-AWS-datapipeline-definition-json-file>
+    task.job_id = opts[:job_id]
+    task.scheduled_start_time = Time.parse(opts[:scheduled_start_time])
+    task.project_root = File.dirname(__FILE__)
+  end
+```
+3. Run the test from the command line by calling:
+
+  ```
+% rake runner[<job-id>]
+```
+  or
+  ```
+% rake runner[<job-id>,<scheduled-start-time>]
+```
+  Where `job-id` is the id of the EMR job you are testing and `scheduled-start-time` is an [ISO 8601](http://en.wikipedia.org/wiki/ISO_8601)-formatted time specifying the time to use when interpolating any `@scheduledStartTime` variable references in your pipeline definition file.
+  If your pipeline definition file has no `@scheduledStartTime` variable references, there is no need to include a `scheduled-start-time` argument.
+
+## Configuration
+
+The following code snippet illustrates how to update the global Hasta configuration, which values are mandatory, and which values have defaults.
+
+```ruby
+Hasta.configure do |config|
+  # mandatory
+  config.project_root = nil
+
+  # optional
+  config.local_storage_root = '~/fog'
+  config.cache_storage_root = '~/.hasta'
+  config.project_steps = 'steps'
+  config.logger = Logger.new(STDOUT)
+end
+```
+
+## Data
+
+All of the data read and written by EMR jobs is stored in [S3](http://aws.amazon.com/s3/).
+Hasta uses the S3 URIs contained in the Data Pipeline definition file for all of its reads and writes.
+When reading data, it will first look on the local filesystem.
+If the requested S3 URI is not found locally, it will look for it on S3.
+Hasta never writes data to S3, only to the local filesytem.
+
+Hasta uses [Fog](http://fog.io/)'s [local storage provider](https://github.com/fog/fog/blob/master/lib/fog/local/storage.rb) to read and write data to the local filesystem using S3 URIs.
+The root directory for the local storage is controlled by the `Hasta.local_storage_root` configuration property, which is set to `~/fog` by default.
+
+Hasta reads all of its input data from the S3 paths specified in the AWS datapipline definition file.
+If you wish to use different data for an input, you need to put that data into the local directory that corresponds to the S3 path in the definition file.
+
+To control which credentials Hasta is using when communicating with S3, update your `~/.fog` file or set the `FOG_CREDENTIAL` environment variable to the appropriate credential.
+
+## Data Filtering
+
+Hasta automatically filters input data to minimize execution time.
+By default, Hasta looks for a file in the current directory named `filter_config.txt` for the filtering configuration.
+You can change the filter configuration file by setting the `HASTA_DATA_FILTER_FILE` environment variable.
+If you want to disable data filtering, you can set the `HASTA_DATA_FILTERING` environment variable to `OFF`.
+
+### Filter Configuration
+
+The filter configuration file contains regular expressions, one per line.
+Any line of input data that matches at least one of the regular expressions is included in the test input.
+Lines that do no match any of the regular expressions are excluded from the test input.
+
+### Caching
+
+Hasta caches the filtered input data locally to improve performance.
+The first time a data source is referenced in a test, the filtered results are written locally.
+Subsequent tests that access the same data source with the same filter are read from the local cache.
+This results in a significant speedup on subsequent runs when dealing with aggressively filtered large data sets.
+By default, the files are written to the `~/.hasta/cache` directory, but this can be controlled using the `cache_storage_root` configuration setting.
+
+## Execution
+
+Hasta sets up the environent variables specified by the `-cmdenv` switch in the job definition.
+It also pulls down all cache files from S3 and stores them in local S3 storage.
+For all cache files that do not have a `.rb` file extension, an environment variable is added to the `ENV` that points to the absolute path of the local file.
+
+### Example
+Given the following `cacheFile` parameter:
+```
+-cacheFile s3://my-bucket/path/to/abbreviations.json#abbreviations.json
+```
+
+The following environment variable will be added to the `ENV`:
+```
+ENV['ABBREVIATIONS_FILE_PATH'] #=> "#{Hasta.local_storage_root}/my-bucket/path/to/abbreviations.json"
+```
+
+The parent directory of each cache file that has a `.rb` file extension is added to the `$LOAD_PATH`, so the mapper and reducer can use `require` statements to load the code in these files.
+
+Hasta executes mappers and reducers in subprocesses.
+This isolates each job and prevents the modifications to `ENV` and `$LOAD_PATH` described above from leaking into the parent process.
+
+The output of the mapper is sorted before it is processed by the reducer.
+The mapper output is sorted in ascending order, according to the natural sort order for Ruby Strings.
+
+## Requirements
+
+### Mappers and Reducers
+1. must be stand-alone Ruby scripts
+2. must be defined in the `Hasta.project_root`/`Hasta.project_steps` directory
+3. must read their input lines from stdin and write their output lines to stdout
+4. any data written to stderr will be logged at the error level using `Hasta.logger`
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## License
+
+Copyright (c) 2014 Swipely, Inc. See [LICENSE.txt](https://github.com/swipely/hasta/blob/master/LICENSE.txt) for further details.

data/Rakefile

@@ -0,0 +1,15 @@
+require 'rake'
+require 'cane/rake_task'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new do |t|
+  t.pattern = File.join(File.dirname(__FILE__), 'spec', '**', '*_spec.rb')
+end
+
+Cane::RakeTask.new('quality') do |cane|
+  cane.canefile = '.cane'
+end
+
+task :all => [:quality, :spec]
+
+task :default => :all

data/hasta.gemspec

@@ -0,0 +1,29 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'hasta/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "hasta"
+  spec.version       = Hasta::VERSION
+  spec.authors       = ["danhodge"]
+  spec.email         = ["dan@swipely.com"]
+  spec.summary       = %q{HAdoop Streaming Test hArness}
+  spec.description   = %q{Harness for locally testing streaming Hadoop jobs written in Ruby}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "fog"
+  spec.add_dependency "json"
+
+  spec.add_development_dependency "bundler", "~> 1.5"
+  spec.add_development_dependency "cane"
+  spec.add_development_dependency "pry"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec"
+end

data/lib/hasta.rb

@@ -0,0 +1,46 @@
+# Copyright Swipely, Inc.  All rights reserved.
+
+require "hasta/version"
+require "hasta/configuration"
+
+require "forwardable"
+
+# The HAdoop Streaming Test hArness
+module Hasta
+  extend self
+  extend Forwardable
+
+  Error = Class.new(StandardError)
+  NonExistentPath = Class.new(Error)
+  ClassLoadError = Class.new(Error)
+  ExecutionError = Class.new(Error)
+  ConfigurationError = Class.new(Error)
+
+  DELEGATED_ATTRS = [
+    :combined_storage,
+    :local_storage_root,
+    :logger,
+    :project_root,
+    :project_steps,
+  ]
+
+  def_delegators :config, *DELEGATED_ATTRS
+
+  def configure
+    yield config
+  end
+
+  def tab_separated_line(line)
+    if line.include?("\t")
+      line
+    else
+      "#{line}\t"
+    end
+  end
+
+  private
+
+  def config
+    @config ||= Configuration.new
+  end
+end

data/lib/hasta/cached_s3_file.rb

@@ -0,0 +1,21 @@
+# Copyright Swipely, Inc.  All rights reserved.
+
+require 'delegate'
+
+module Hasta
+  # Implements Hasta's S3File interface for files retrieved from the cache
+  class CachedS3File < SimpleDelegator
+    def initialize(cached_file, s3_uri)
+      super(S3File.wrap(cached_file))
+      @s3_uri = s3_uri
+    end
+
+    def key
+      @s3_uri.path
+    end
+
+    def s3_uri
+      @s3_uri
+    end
+  end
+end

data/lib/hasta/combined_data_source.rb

@@ -0,0 +1,35 @@
+# Copyright Swipely, Inc.  All rights reserved.
+
+module Hasta
+  # Combines multiple data sources so they can be iterated over continuously
+  class CombinedDataSource
+    attr_reader :name
+
+    def initialize(sources, name = nil)
+      @sources = sources
+      @name = name || sources.map(&:name).compact.join(', ')
+    end
+
+    def each_line
+      return enum_for(:each_line) unless block_given?
+
+      sources.each do |source|
+        source.each_line do |line|
+          yield line
+        end
+      end
+    end
+
+    def to_a
+      each_line.to_a
+    end
+
+    def to_s
+      "#<#{self.class.name}:#{name}>"
+    end
+
+    private
+
+    attr_reader :sources
+  end
+end

data/lib/hasta/combined_storage.rb

@@ -0,0 +1,30 @@
+# Copyright Swipely, Inc.  All rights reserved.
+
+require 'hasta/s3_uri'
+require 'hasta/storage'
+
+module Hasta
+  # The file storage interface used by the local map/reduce jobs
+  class CombinedStorage
+    def initialize(s3_storage, local_storage)
+      @s3_storage = s3_storage
+      @local_storage = local_storage
+    end
+
+    def files_for(s3_uri)
+      if local_storage.exists?(s3_uri)
+        local_storage.files_for(s3_uri)
+      else
+        s3_storage.files_for(s3_uri)
+      end
+    end
+
+    def write(s3_uri, data_source)
+      local_storage.write(s3_uri, data_source)
+    end
+
+    private
+
+    attr_reader :s3_storage, :local_storage
+  end
+end

data/lib/hasta/configuration.rb

@@ -0,0 +1,88 @@
+# Copyright Swipely, Inc.  All rights reserved.
+
+require 'fog'
+require 'logger'
+
+require 'hasta/local_storage'
+require 'hasta/s3_storage'
+require 'hasta/combined_storage'
+require 'hasta/filter'
+require 'hasta/resolve_cached_s3_file'
+require 'hasta/resolve_filtered_s3_file'
+
+module Hasta
+  # Global configuration settings
+  class Configuration
+    attr_accessor :project_root
+    attr_writer :local_storage_root, :cache_storage_root, :project_steps, :logger, :filter
+
+    def local_storage_root
+      @local_storage_root ||= '~/fog'
+    end
+
+    def cache_storage_root
+      @cache_storage_root ||= '~/.hasta'
+    end
+
+    def project_steps
+      @project_steps ||= 'steps'
+    end
+
+    def logger
+      @logger ||= Logger.new(STDOUT)
+    end
+
+    def project_steps_dir
+      File.join(project_root, project_steps)
+    end
+
+    def combined_storage
+      @combined_storage ||= CombinedStorage.new(
+        S3Storage.new(fog_s3_storage, resolver),
+        LocalStorage.new(fog_local_storage, resolver)
+      )
+    end
+
+    def filter
+      unless @filter || ENV['HASTA_DATA_FILTERING'] == 'OFF'
+        filter_file = ENV['HASTA_DATA_FILTER_FILE'] || 'filter_config.txt'
+        @filter ||= Filter.from_file(filter_file)
+      end
+
+      @filter
+    end
+
+    private
+
+    def fog_s3_storage
+      # Use FOG_CREDENTIAL env variable to control AWS credentials
+      @fog_s3_storage ||= Fog::Storage::AWS.new
+    end
+
+    def fog_local_storage
+      @fog_local_storage ||= local_fog(local_storage_root)
+    end
+
+    def fog_cache_storage
+      @fog_cache_storage ||= local_fog(cache_storage_root)
+    end
+
+    def local_fog(local_root)
+      Fog::Storage.new(
+        :provider => 'Local',
+        :local_root => local_root,
+        :endpoint => 'http://example.com'
+      )
+    end
+
+    def resolver
+      if filter
+        ResolveCachedS3File.new(
+          S3FileCache.new(fog_cache_storage), ResolveFilteredS3File.new(filter)
+        )
+      else
+        Hasta::Storage::ResolveS3File
+      end
+    end
+  end
+end