piglet 0.2.3 → 0.2.4

data/README.rdoc

@@ -213,6 +213,8 @@ Binary conditionals, a.k.a. the ternary operator are supported through +test+ (u
   
 The first argument to +test+ is the test expression, the second is the if-true expression and the third is the if-false expression.
 
+<code>FOREACH { … } GENERATE</code>, a.k.a. nested foreach or foreach-with-inner-bag is currently not supported. Nor is specifying the schema of the resulting relation, or +FLATTEN+ operations.
+
 == +split+
 
 The syntax of +split+ shouldn't be surprising if you've read this far, but there's perhaps some details that aren't obvious. To split a relation into a number of parts you call +split+ on the relation and pass a block in which you specify the expressions describing each shard. Just as with +filter+ and +foreach+ the block receives the relation as an argument. +split+ returns an array containing the relation shards and you can use parallel assignment to make it look really nice:
@@ -238,6 +240,37 @@ I'm not absolutely sure that it is legal to join or cogroup on more than one fie
 
   a.cogroup(b => [:y, :inner], a => [:z, :outer]) # => COGROUP b BY y INNER, a BY z OUTER
 
+== +stream+, +define+ & +register+
+
+The +STREAM+ operator is supported through <code>stream</code>. You can either stream through a command, or through a command reference. 
+
+To stream through a command, use this syntax:
+
+  a.stream(:command => 'cut -f 3') # => STREAM a THROUGH `cut -f 3`
+
+to define a command and then stream a relation through that command, use this syntax:
+
+  define(:reverse, :command => 'reverse.rb') # => DEFINE reverse `reverse.rb`
+  a.stream(:reverse)                         # => STREAM a THROUGH reverse
+
+You can also use +define+ to define function references:
+
+  define(:hello, :function => 'com.example.Hello') # => DEFINE hello com.example.Hello
+
+If you need to register a JAR you can use +register+:
+
+  register('path/to/lib.jar') # => REGISTER path/to/lib.jar
+
+Streaming multiple relations is supported, just pass more relations as an array:
+
+  a.stream([b, c, d], :reverse) # => STREAM a, b, c THROUGH reverse
+
+And finally, this is how you specify the schema of the resulting relation:
+
+  a.stream(:reverse, :schema => [:x, :y]) # => STREAM a THROUGH reverse AS (x, y)
+
+the schema syntax is the same as for +load+, and you can read more about it under “Types & schemas” below.
+
 == <code>:parallel</code>
 
 For some operators in Pig Latin you can specify the <code>PARALLEL</code> keyword to tell Pig how many reducers
@@ -351,6 +384,7 @@ The following Pig operators are supported:
 
 * +COGROUP+
 * +CROSS+
+* +DEFINE+
 * +DESCRIBE+
 * +DISTINCT+
 * +DUMP+
@@ -363,32 +397,39 @@ The following Pig operators are supported:
 * +LIMIT+
 * +LOAD+
 * +ORDER+
+* +REGISTER+
 * +SAMPLE+
 * +SPLIT+
 * +STORE+
+* +STREAM+
 * +UNION+
- 
+
 The following are currently not supported (but will be soon):
 
-* +STREAM+
-* +DEFINE+
-* +DECLARE+
-* +REGISTER+
+* <code>FOREACH { … } GENERATE</code>
+* <code>%DECLARE</code>
+* <code>%DEFAULT</code>
 
 The file commands (+cd+, +cat+, etc.) will probably not be supported for the forseeable future.
 
-All the aggregate functions are supported:
+UDF:s are sort of supported. You can declare a UDF with +declare+ and you can use it in some situations by using a string, but some situations are trickier (like renaming the result of a UDF call in a <code>FOREACH … GENERATE</code> body).
+
+All the aggregate functions except two are supported:
 
 * +AVG+
 * +CONCAT+
 * +COUNT+
-* +DIFF+
 * +IsEmpty+
 * +MAX+
 * +MIN+
 * +SIZE+
 * +SUM+
 * +TOKENIZE+
+
+These are not supported yet:
+
+* +DIFF+
+* +FLATTEN+
   
 Piglet only supports most arithmetic and logic operators (see below) on fields -- but check the output and make sure that it's doing what you expect because some it's tricky to see where Piglet hijacks the operators and when it's Ruby that is running the show. I'm doing the best I can, but there are many things that can't be done, at least not in Ruby 1.8.
 
@@ -442,6 +483,10 @@ When you call +group+, +filter+ or any of the other methods that can be applied
 
 As a side effect of using +store+ and the other output operators as the trigger for creating the needed relational operations any relations that are not ancestors of relations that are outputed will not be included in the Pig Latin output. On the other hand, they would be no-ops when run by Pig anyway.
 
+=== The output is not what I expected!
+
+Please contact me and give me the Piglet code and what you think the output should be. I'll try to either fix your Piglet code, or fix Piglet to do what you expect it to do.
+
 == Copyright
 
 © 2009-2010 Theo Hultberg / Iconara. See LICENSE for details.

data/TODO

@@ -0,0 +1,4 @@
+* STREAM
+* DEFINE + gör så att funktionen går att köra från Piglet i ett foreach-block
+* REGISTER
+* DIFF

data/lib/piglet/interpreter.rb

@@ -4,7 +4,7 @@ require 'set'
 module Piglet
   class Interpreter
     def initialize(&block)
-      @stores = [ ]
+      @top_level_statements = [ ]
       
       interpret(&block) if block_given?
     end
@@ -20,18 +20,18 @@ module Piglet
     def to_pig_latin(&block)
       interpret(&block) if block_given?
       
-      return '' if @stores.empty?
+      return '' if @top_level_statements.empty?
       
       handled_relations = Set.new
       statements = [ ]
       
-      @stores.each do |store|
-        unless store.relation.nil?
-          assignments(store.relation, handled_relations).each do |assignment|
+      @top_level_statements.each do |top_level_statement|
+        if top_level_statement.respond_to?(:relation) && ! top_level_statement.relation.nil?
+          assignments(top_level_statement.relation, handled_relations).each do |assignment|
             statements << assignment
           end
         end
-        statements << store
+        statements << top_level_statement
       end
       
       statements.flatten.map { |s| s.to_s }.join(";\n") + ";\n"
@@ -62,28 +62,28 @@ module Piglet
     #   store(x, 'some/path', :using => 'Xyz') # => STORE x INTO 'some/path' USING Xyz
     #   store(x, 'some/path', :using => :pig_storage) # => STORE x INTO 'some/path' USING PigStorage
     def store(relation, path, options={})
-      @stores << Inout::Store.new(relation, path, options)
+      @top_level_statements << Inout::Store.new(relation, path, options)
     end
   
     # DUMP
     #
     #   dump(x) # => DUMP x
     def dump(relation)
-      @stores << Inout::Dump.new(relation)
+      @top_level_statements << Inout::Dump.new(relation)
     end
   
     # ILLUSTRATE
     #
     #   illustrate(x) # => ILLUSTRATE x
     def illustrate(relation)
-      @stores << Inout::Illustrate.new(relation)
+      @top_level_statements << Inout::Illustrate.new(relation)
     end
   
     # DESCRIBE
     #
     #   describe(x) # => DESCRIBE x
     def describe(relation)
-      @stores << Inout::Describe.new(relation)
+      @top_level_statements << Inout::Describe.new(relation)
     end
   
     # EXPLAIN
@@ -91,7 +91,34 @@ module Piglet
     #   explain    # => EXPLAIN
     #   explain(x) # => EXPLAIN(x)
     def explain(relation=nil)
-      @stores << Inout::Explain.new(relation)
+      @top_level_statements << Inout::Explain.new(relation)
+    end
+    
+    # REGISTER
+    #
+    #   register 'path/to/lib.jar' # => REGISTER path/to/lib.jar
+    def register(path)
+      @top_level_statements << Udf::Register.new(path)
+    end
+    
+    # DEFINE
+    #
+    #   define('test', :function => 'com.example.Test')             # => DEFINE test com.example.Test
+    #   define('test', :command => 'test.rb')                       # => DEFINE test `test.rb`
+    #   define('test', :command => 'test.rb', :input => :stdin)     # => DEFINE test `test.rb` INPUT(stdin)
+    #   define('test', :command => 'test.rb', :input => 'path/x')   # => DEFINE test `test.rb` INPUT('path/x')
+    #   define('test', :command => 'test.rb', :output => :stdout)   # => DEFINE test `test.rb` OUTPUT(stdout)
+    #   define('test', :command => 'test.rb', :ship => 'a/b/c')     # => DEFINE test `test.rb` SHIP('a/b/c')
+    #   define('test', :command => 'test.rb', :cache => ['x', 'y']) # => DEFINE test `test.rb` CACHE('x', 'y')
+    #
+    # The <code>:input</code> and <code>:output</code> options can take pretty
+    # complicated definitions in addition to the examples above:
+    #
+    #   :input => {:from => :stdin, :using => :pig_storage}                  # => INPUT(stdin USING PigStorage)
+    #   :output => {:to => :stdout, :using => 'MySerializer'}                # => OUTPUT(stdout USING MySerializer)
+    #   :output => [{:to => :stdout, :using => 'MySerializer'}, 'some/path'] # => OUTPUT(stdout USING MySerializer, 'some/path')
+    def define(ali4s, options=nil)
+      @top_level_statements << Udf::Define.new(ali4s, options)
     end
     
     # Support for binary conditions, a.k.a. the ternary operator.

data/lib/piglet/relation/relation.rb

@@ -120,15 +120,13 @@ module Piglet
   
       # STREAM
       #
-      #   x.stream(x, 'cut -f 3')                         # => STREAM x THROUGH `cut -f 3`
-      #   x.stream([x, y], 'cut -f 3')                    # => STREAM x, y THROUGH `cut -f 3`
-      #   x.stream(x, 'cut -f 3', :schema => [%w(a int)]) # => STREAM x THROUGH `cut -f 3` AS (a:int)
-      #
-      #--
-      #
-      # TODO: how to handle DEFINE'd commands?
-      def stream(relations, command, options={})
-        raise NotSupportedError
+      #   x.stream(:command => 'cut -f 3')       # => STREAM x THROUGH `cut -f 3`
+      #   x.stream(:cmd)                         # => STREAM x THROUGH cmd
+      #   x.stream(y, :command => 'cut -f 3')    # => STREAM x, y THROUGH `cut -f 3`
+      #   x.stream(:cmd, :schema => [%w(a int)]) # => STREAM x THROUGH cmd AS (a:int)
+      def stream(*args)
+        fields, options = split_at_options(args)
+        Stream.new(self, fields, options)
       end
   
       # UNION

data/lib/piglet/relation/stream.rb

@@ -2,6 +2,40 @@ module Piglet
   module Relation
     class Stream # :nodoc:
       include Relation
+      
+      def initialize(source, args, options=nil)
+        options ||= {}
+        @sources = [source]
+        args.each do |arg|
+          @sources << arg if arg.is_a?(Relation) || arg.is_a?(Array)
+        end
+        @command_reference = (args - @sources).first
+        @sources = @sources.flatten
+        @command = options[:command]
+        @schema = options[:schema]
+      end
+      
+      def schema
+        if @schema
+          Piglet::Schema::Tuple.parse(@schema)
+        else
+          nil
+        end
+      end
+      
+      def to_s
+        source_str = @sources.map { |s| s.alias }.join(', ')
+        str = "STREAM #{source_str} THROUGH"
+        if @command_reference
+          str << " #{@command_reference}" 
+        else
+          str << " `#{@command}`"
+        end
+        if @schema
+          str << " AS #{schema}"
+        end
+        str
+      end
     end
   end
 end

data/lib/piglet/udf/define.rb

@@ -0,0 +1,68 @@
+module Piglet
+  module Udf
+    class Define
+      include Piglet::Inout::StorageTypes
+      
+      def initialize(ali4s, options=nil)
+        options ||= {}
+        @alias = ali4s
+        @command = options[:command]
+        @function = options[:function]
+        @input = options[:input]
+        @output = options[:output]
+        @ship = options[:ship]
+        @cache = options[:cache]
+      end
+      
+      def to_s
+        if @command
+          str = "DEFINE #{@alias} `#{@command}`"
+          str << io_to_s(:input, @input) if @input
+          str << io_to_s(:output, @output) if @output
+          str << paths_to_s(:ship, @ship) if @ship
+          str << paths_to_s(:cache, @cache) if @cache
+          str
+        else
+          "DEFINE #{@alias} #{@function}"
+        end
+      end
+      
+    private
+    
+      def paths_to_s(kind, paths)
+        unless Enumerable === paths
+          paths = [paths]
+        end
+        path_str = paths.map { |p| "'#{p}'" }.join(', ')
+        " #{kind.to_s.upcase}(#{path_str})"
+      end
+            
+      def io_to_s(method, description)
+        case description
+        when Symbol, String
+          if method == :input
+            io_to_s(method, [{:from => description}])
+          else
+            io_to_s(method, [{:to => description}])
+          end
+        when Hash
+          io_to_s(method, [description])
+        when Enumerable
+          str = " #{method.to_s.upcase}("
+          description_strs = description.map do |desc|
+            stream = (method == :input ? desc[:from] : desc[:to])
+            stream = "'#{stream}'" unless Symbol === stream
+            if desc[:using]
+              "#{stream} USING #{resolve_load_store_function(desc[:using])}"
+            else
+              stream.to_s
+            end
+          end
+          str << description_strs.join(', ')
+          str << ')'
+          str
+        end
+      end
+    end
+  end
+end

data/lib/piglet/udf/register.rb

@@ -0,0 +1,13 @@
+module Piglet
+  module Udf
+    class Register
+      def initialize(path)
+        @path = path
+      end
+      
+      def to_s
+        "REGISTER #{@path}"
+      end
+    end
+  end
+end

data/lib/piglet.rb

@@ -1,6 +1,6 @@
 # :main: README.rdoc
 module Piglet # :nodoc:
-  VERSION = '0.2.3'
+  VERSION = '0.2.4'
   
   class PigletError < StandardError; end
   class NotSupportedError < PigletError; end
@@ -53,4 +53,9 @@ module Piglet # :nodoc:
     
     class SchemaError < PigletError; end
   end
+  
+  module Udf
+    autoload :Define, 'piglet/udf/define'
+    autoload :Register, 'piglet/udf/register'
+  end
 end

data/spec/piglet_spec.rb

@@ -359,6 +359,134 @@ describe Piglet do
         @interpreter.to_pig_latin.should match(/\w+ BY y OUTER/)
       end
     end
+    
+    describe 'STREAM' do
+      it 'outputs a STREAM statement with a command reference' do
+        output = @interpreter.to_pig_latin do
+          a = load('in')
+          b = a.stream(:swoosch)
+          store(b, 'out')
+        end
+        output.should match(/STREAM \w+ THROUGH swoosch/)
+      end
+
+      it 'outputs a STREAM statement with a command' do
+        output = @interpreter.to_pig_latin do
+          a = load('in')
+          b = a.stream(:command => 'swoosch')
+          store(b, 'out')
+        end
+        output.should match(/STREAM \w+ THROUGH `swoosch`/)
+      end
+
+      it 'outputs a STREAM statement with a schema' do
+        output = @interpreter.to_pig_latin do
+          a = load('in')
+          b = a.stream(:command => 'swoosch', :schema => [:a, :b])
+          store(b, 'out')
+        end
+        output.should match(/STREAM \w+ THROUGH `swoosch` AS \(a:bytearray, b:bytearray\)/)
+      end
+      
+      it 'outputs a STREAM statement with many relations' do
+        output = @interpreter.to_pig_latin do
+          x = load('in1')
+          y = load('in2')
+          z = load('in3')
+          w = x.stream([x, y], :plink)
+          store(w, 'out')
+        end
+        output.should match(/STREAM \w+, \w+, \w+ THROUGH plink/)
+      end
+    end
+  end
+
+  context 'UDF statements:' do
+    describe 'DEFINE' do
+      it 'outputs a DEFINE with the correct alias and function name' do
+        output = @interpreter.to_pig_latin { define('plunk', :function => 'com.example.Plunk') }
+        output.should include('DEFINE plunk com.example.Plunk')
+      end
+      
+      it 'outputs a DEFINE with the correct alias and command string' do
+        output = @interpreter.to_pig_latin { define('plunk', :command => 'plunk.rb') }
+        output.should include('DEFINE plunk `plunk.rb`')
+      end
+
+      it 'outputs a DEFINE with an INPUT definition' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :input => :stdin)
+        end
+        output.should include('DEFINE plunk `plunk.rb` INPUT(stdin)')
+      end
+
+      it 'outputs a DEFINE with an OUTPUT definition' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :output => :stdout)
+        end
+        output.should include('DEFINE plunk `plunk.rb` OUTPUT(stdout)')
+      end
+
+      it 'outputs a DEFINE with a SHIP definition with one path' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :ship => 'path/to/somewhere')
+        end
+        output.should include('DEFINE plunk `plunk.rb` SHIP(\'path/to/somewhere\')')
+      end
+
+      it 'outputs a DEFINE with a SHIP definition with may paths' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :ship => ['path/to/somewhere', 'and/to/somewhere/else'])
+        end
+        output.should include('DEFINE plunk `plunk.rb` SHIP(\'path/to/somewhere\', \'and/to/somewhere/else\')')
+      end
+
+      it 'outputs a DEFINE with a CACHE definition with one path description' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :cache => '/input/data.gz#data.gz')
+        end
+        output.should include('DEFINE plunk `plunk.rb` CACHE(\'/input/data.gz#data.gz\')')
+      end
+
+      it 'outputs a DEFINE with a CACHE definition with may path descriptions' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :cache => ['/input/data.gz#data.gz', '/mydir/mydata.txt#mydata.txt'])
+        end
+        output.should include('DEFINE plunk `plunk.rb` CACHE(\'/input/data.gz#data.gz\', \'/mydir/mydata.txt#mydata.txt\')')
+      end
+
+      it 'outputs a DEFINE with with a somewhat complex INPUT definition' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', :input => {:from => 'some/path', :using => :pig_storage})
+        end
+        output.should include('DEFINE plunk `plunk.rb` INPUT(\'some/path\' USING PigStorage)')
+      end
+
+      it 'outputs a DEFINE with with really complex options' do
+        output = @interpreter.to_pig_latin do
+          define('plunk', :command => 'plunk.rb', 
+            :input => [
+              {:from => 'some/path', :using => :pig_storage},
+              {:from => :stdin, :using => 'HelloWorld(\'test\')'}
+            ],
+            :output => [
+              {:to => 'some/other/path', :using => :bin_storage},
+              {:to => :stdout, :using => 'SomeOtherMechanism()'}
+            ],
+            :ship => 'to/here',
+            :cache => ['first', 'second', 'third']
+          )
+        end
+        output.should include('DEFINE plunk `plunk.rb` INPUT(\'some/path\' USING PigStorage, stdin USING HelloWorld(\'test\')) OUTPUT(\'some/other/path\' USING BinStorage, stdout USING SomeOtherMechanism()) SHIP(\'to/here\') CACHE(\'first\', \'second\', \'third\')')
+      end
+    end
+    
+    describe 'REGISTER' do
+      it 'outputs a REGISTER statement with the path to the specified JAR' do
+        output = @interpreter.to_pig_latin { register('path/to/lib.jar') }
+        output.should include('REGISTER path/to/lib.jar')
+      end
+    end
   end
 
   context 'aliasing & multiple statements' do
@@ -705,6 +833,18 @@ describe Piglet do
       schema.field_type(0).should eql(:double)
     end
     
+    it 'knows the schema of a relation streamed through a command (if there\'s a schema)' do
+      schema = catch(:schema) do
+        @interpreter.interpret do
+          relation1 = load('in1', :schema => [[:a, :float], [:b, :int]])
+          relation2 = relation1.stream(:command => 'command', :schema => [[:x, :chararray]])
+          throw :schema, relation2.schema
+        end
+      end
+      schema.field_names.should eql([:x])
+      schema.field_type(:x).should eql(:chararray)
+    end
+    
   end
 
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: piglet
 version: !ruby/object:Gem::Version 
-  version: 0.2.3
+  version: 0.2.4
 platform: ruby
 authors: 
 - Theo Hultberg
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-01-12 00:00:00 +01:00
+date: 2010-01-14 00:00:00 +01:00
 default_executable: piglet
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -31,6 +31,7 @@ extensions: []
 extra_rdoc_files: 
 - LICENSE
 - README.rdoc
+- TODO
 files: 
 - .document
 - .gitignore
@@ -73,6 +74,8 @@ files:
 - lib/piglet/relation/union.rb
 - lib/piglet/schema/bag.rb
 - lib/piglet/schema/tuple.rb
+- lib/piglet/udf/define.rb
+- lib/piglet/udf/register.rb
 - spec/piglet/field/binary_conditional_spec.rb
 - spec/piglet/field/field_spec.rb
 - spec/piglet/field/infix_expression_spec.rb
@@ -89,6 +92,7 @@ files:
 - tasks/gem.rake
 - tasks/rdoc.rake
 - tasks/spec.rake
+- TODO
 has_rdoc: true
 homepage: http://github.com/iconara/piglet
 licenses: []