hog 0.0.1

data/.gitignore

@@ -0,0 +1,22 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in hog.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 jondot
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,173 @@
+# Hog
+
+Supercharge your Ruby Pig UDFs with Hog.  
+
+
+## Why Use Hog?
+
+Use Hog when you want to process (map) tuples with Hadoop, and generally
+belive that:
+
+* It's too much overhead to write custom M/R code
+* You're too lazy to process with Pig data pipes and sprinkled custom UDFs, and they're not powerful
+  enough anyway.
+* Data processing with Hive queries is wrong
+* Cascading is an overkill for your case
+* Streaming Hadoop (unix processes) is the last resort
+
+And instead want to develop within your familiar Ruby ecosystem but
+also:
+
+* Control and tweak job parameters via Pig
+* Use Pig's job management
+* Use Pig's I/O (serdes) easily
+
+And you also want to use gems, jars, and everything Ruby (JRuby) has to offer for a really quick turn-around and pleasent development experience.
+
+
+## Quickstart
+
+Let's say you want to perform geolocation, detect a device's form
+factor, and shove bits and fields from the nested structure into a Hive table to be able to query
+it efficiently.
+
+This is your raw event:
+
+```javascript
+{
+  "x-forwarded-for":"8.8.8.8",
+  "user-agent": "123",
+  "pageview": {
+    "userid":"ud-123",
+    "timestamp":123123123,
+    "headers":{
+      "user-agent":"...",
+      "cookies":"..."
+    }
+  }
+}
+```
+
+To do this with Pig alone, you would need:
+
+* A pig UDF for geolocation
+* A pig UDF for form-factor detection
+* A way to extract JSON fields from a tuple in Pig
+* Combine everything in Pig code
+
+And you'll give up
+
+* Testability
+* Maintainability
+* Development speed
+* Eventually, sanity
+
+
+#### Using Hog
+
+With Hog, you'll mostly feel you're _describing_ how you want the data to be
+shaped, and it'll do the heavy lifting. 
+
+You always have the option to 'drop' to code with the `prepare`
+block.
+
+
+```ruby
+# gem install hog
+require 'hog'
+
+TupleProcessor = Hog.tuple "mytuple" do
+
+  # Your own custom feature extraction. This can be any free style
+  # Ruby or Java code
+  #
+  prepare do |hsh|
+    loc = ip2location(hsh["x-forwarded-for"])
+    hsh.merge!(loc)
+
+    form_factor = formfactor(hsh["user-agent"])
+    hsh.merge!(form_factor)
+  end
+
+
+  # Describe your data columns (i.e. for Hive), and how you'd like
+  # to pull them out of your nested data.
+  #
+  # You can also put a fixed value, or serialize a sub-nested
+  # structure as-is for later use with Hive's json_object
+  #
+  chararray :id, :path => 'pageview.userid'
+  chararray :created_at, :path => 'pageview.timestamp', :with_nil => ''
+  float :sampling, :value => 1.0
+  chararray :ev_type, :value => 'pageview'
+  chararray :ev_info, :json => 'pageview.headers', :default => {}
+end
+```
+
+As you'll notice - there's very little code, and you specify the bare
+essentials -- which you would have had to specify (types and fields) with Pig in any way you'd try to use it.
+
+
+Hog will generate a `TupleProcessor` that conforms to your description and logic.
+
+
+#### Testing
+
+To test your mapper, just use `TupleProcessor` within your specs and
+give it a raw event. It's plain Ruby.
+
+
+#### Hadoop
+
+To hookup with Hadoop, rig `TupleProcessor` within a Pig JRuby UDF shell like so:
+
+```
+require 'tuple_processor'
+require 'pigudf'
+
+class TupleProcessorUdf < PigUdf
+  # TupleProcessor will automatically generate your UDF schema!, no matter how complex or
+  # convoluted.
+  outputSchema TupleProcessor.schema
+
+  # Use TupleProcessor as the processing logic.
+  def process(line)
+    # since 1 raw json can output several rows, 'process' returns
+    # an array. we pass that to Pig as Pig's own 'DataBag'.
+    # You can also do without and just return whatever TupleProcessor#process returns.
+    databag = DataBag.new
+    TupleProcessor.process(line).each do |res|
+      databag.add(res)
+    end
+
+    databag
+  end
+end
+```
+
+
+## Gems and Jars
+
+So you can't really use gems or jars with Pig Ruby UDFs unless all machines have
+them; and then it becomes ugly to manage really.
+
+The solution is neat: pack everything as you would with a JRuby project,
+for example, use `warbler`; and your job code becomes a standalone-jar
+which you deploy, version, and possibly generate via CI.
+
+
+## Related Projects
+
+* PigPen - https://github.com/Netflix/PigPen
+* datafu - https://github.com/linkedin/datafu
+
+
+# Contributing
+
+Fork, implement, add tests, pull request, get my everlasting thanks and a respectable place here :).
+
+# Copyright
+
+Copyright (c) 2014 [Dotan Nahum](http://gplus.to/dotan) [@jondot](http://twitter.com/jondot). See MIT-LICENSE for further details.
+
+

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/bin/hog

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'hog'
+
+# hog binary is reserved for future use

data/example/runner.pig

@@ -0,0 +1,13 @@
+-- Since most work is in your TupleProcessor, this file serves as a simple shim
+-- that loads data into raw lines, and sends them off to your UDF.
+-- As a result, it will mostly stay the same.
+-- You can tweak any M/R or Pig params here as well.
+
+REGISTER 'your-warbled-dependencies-jar.jar'
+REGISTER 'tuple_processor_udf.rb' USING jruby AS processor;
+
+data = FOREACH ( LOAD '/raw-data' AS (line:CHARARRAY) )
+        GENERATE flatten(process.process(line));
+
+store data into '/out-data' USING PARQUETFILE
+

data/example/tuple_processor.rb

@@ -0,0 +1,29 @@
+require 'hog'
+
+TupleProcessor = Hog.tuple "mytuple" do
+
+  # Your own custom feature extraction. This can be any free style
+  # Ruby or Java code
+  #
+  prepare do |hsh|
+    loc = ip2location(hsh["x-forwarded-for"])
+    hsh.merge!(loc)
+
+    form_factor = formfactor(hsh["user-agent"])
+    hsh.merge!(form_factor)
+  end
+
+
+  # Describe your data columns (i.e. for Hive), and how you'd like
+  # to pull them out of your nested data.
+  #
+  # You can also put a fixed value, or serialize a sub-nested
+  # structure as-is for later use with Hive's json_object
+  #
+  chararray :id, :path => 'pageview.userid'
+  chararray :created_at, :path => 'pageview.timestamp', :with_nil => ''
+  float :sampling, :value => 1.0
+  chararray :ev_type, :value => 'pageview'
+  chararray :ev_info, :json => 'pageview.headers', :default => {}
+end
+

data/example/tuple_processor_udf.rb

@@ -0,0 +1,22 @@
+require 'pigudf'
+require 'tuple_processor'
+
+class TupleProcessorUdf < PigUdf
+  # TupleProcessor will automatically generate your UDF schema!, no matter how complex or
+  # convoluted.
+  outputSchema TupleProcessor.schema
+
+  # Use TupleProcessor as the processing logic.
+  def process(line)
+    # since 1 raw json can output several rows, 'process' returns
+    # an array. we pass that to Pig as Pig's own 'DataBag'.
+    # You can also do without and just return whatever TupleProcessor#process returns.
+    databag = DataBag.new
+    TupleProcessor.process(line).each do |res|
+      databag.add(res)
+    end
+
+    databag
+  end
+end
+

data/hog.gemspec

@@ -0,0 +1,23 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'hog/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "hog"
+  spec.version       = Hog::VERSION
+  spec.authors       = ["jondot"]
+  spec.email         = ["jondotan@gmail.com"]
+  spec.summary       = %q{}
+  spec.description   = %q{}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.6"
+  spec.add_development_dependency "rake"
+end

data/lib/hog/field.rb

@@ -0,0 +1,52 @@
+module Hog
+  class Field
+    attr_reader :opts
+
+    def initialize(type, name, opts={})
+      @name = name
+      @type = type
+      @opts = opts
+      opts[:path] ||= name.to_s
+    end
+
+    def get(hash)
+      val = nil
+      if opts[:value]
+        val = opts[:value]
+      elsif opts[:json]
+        lkval = lookup(opts[:json],hash) || opts[:default]
+        val = JSON.dump(lkval)
+      else
+        val = lookup(opts[:path], hash)
+      end
+      get_value(val)
+    end
+
+    def to_s
+      "#{@name}:#{@type}"
+    end
+
+    private
+
+    def get_value(val)
+      # xxx optimize?
+      if @type == "int" && [TrueClass, FalseClass].include?(val.class)
+        val ? 1 : 0
+      elsif @type == "chararray"
+        val.to_s
+      else
+        val
+      end
+    end
+
+    def lookup(path, hash)
+      begin
+      path.split('.').inject(hash) {|acc, value| acc[value]}
+      rescue
+        return opts[:with_nil] if opts[:with_nil]
+        raise "No path: #{path} in hash: #{hash}"
+      end
+    end
+  end
+end
+

data/lib/hog/testing/pig_stubs.rb

@@ -0,0 +1,16 @@
+class PigUdf
+  def self.outputSchema(str)
+
+  end
+end
+
+class DataBag
+  attr_reader :data
+  def initialize
+    @data = []
+  end
+  def add(el)
+    @data << el
+  end
+end
+

data/lib/hog/tuple.rb

@@ -0,0 +1,41 @@
+# todos
+# - safe hash path lookup + provide default value?
+
+
+module Hog
+  class Tuple
+    def initialize(name, block)
+      @name = name
+      @fields = []
+      self.instance_eval &block
+    end
+
+    def process(hsh)
+      @prepare.call(hsh)
+      res = []
+      @fields.each do |f|
+        res << f.get(hsh)
+      end
+      res
+    end
+
+    def prepare(&block)
+      @prepare = block
+    end
+
+    def to_s
+      "(#{@fields.map{|f| f.to_s }.join(',')})"
+    end
+
+    def method_missing(meth, *args, &block)
+      if [:chararray, :float, :double, :int, :long, :map].include?(meth)
+        f = Field.new(meth.to_s, *args)
+        @fields << f
+        return f
+      else
+        super
+      end
+      end
+  end
+end
+

data/lib/hog/utils.rb

@@ -0,0 +1,12 @@
+module Hog
+  module Utils
+    def explode(hsh, path, into_path)
+      arr = hsh.delete(path)
+      exploded = []
+      arr.each do |item|
+        exploded << ({ into_path => item }.merge(hsh))
+      end
+      exploded
+    end
+  end
+end

data/lib/hog/version.rb

@@ -0,0 +1,3 @@
+module Hog
+  VERSION = "0.0.1"
+end

data/lib/hog.rb

@@ -0,0 +1,19 @@
+require "hog/version"
+require 'json'
+require 'hog/tuple'
+require 'hog/field'
+require 'hog/utils'
+
+module Hog
+  class << self
+    include Hog::Utils
+  end
+
+  def self.tuple(name, &block)
+    Tuple.new(name, block)
+  end
+end
+
+
+
+

metadata

@@ -0,0 +1,95 @@
+--- !ruby/object:Gem::Specification
+name: hog
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- jondot
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-06-18 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: ''
+email:
+- jondotan@gmail.com
+executables:
+- hog
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/hog
+- example/runner.pig
+- example/tuple_processor.rb
+- example/tuple_processor_udf.rb
+- hog.gemspec
+- lib/hog.rb
+- lib/hog/field.rb
+- lib/hog/testing/pig_stubs.rb
+- lib/hog/tuple.rb
+- lib/hog/utils.rb
+- lib/hog/version.rb
+homepage: ''
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: ''
+test_files: []