piglet 0.1.0

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,22 @@
+## MAC OS
+.DS_Store
+
+## TEXTMATE
+*.tmproj
+tmtags
+
+## EMACS
+*~
+\#*
+.\#*
+
+## VIM
+*.swp
+
+## PROJECT::GENERAL
+coverage
+rdoc
+doc
+pkg
+
+## PROJECT::SPECIFIC

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Theo Hultberg
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,293 @@
+= Piglet
+
+Piglet is a DSL for writing Pig Latin scripts in Ruby:
+
+  a = load 'input'
+  b = a.group :c
+  store b, 'output'
+  
+The code above will be translated to the following Pig Latin:
+
+  relation_2 = LOAD 'input';
+  relation_1 = GROUP relation_2 BY c;
+  STORE relation_1 INTO 'output';
+
+Piglet aims to look like Pig Latin while allowing for things like loops and control of flow that are missing from Pig. I started working on Piglet out of frustration that my Pig scripts started to be very repetitive. Pig lacks control of flow and mechanisms to apply the same set of operations on multiple relations. Piglet is my way of adding those features.
+
+== Usage
+
+It can be used either as a command line tool for translating a file of Piglet code into Pig Latin, or you can use it inline in a Ruby script:
+
+=== Command line usage
+
+If <code>piggy.rb</code> contains
+
+  store(load('input')), 'output')
+
+then running
+
+  piglet piggy.rb
+  
+will output
+  
+  relation_1 = LOAD 'input';
+  STORE relation_1 INTO 'output';
+  
+=== Programmatic interface
+
+  require 'piglet'
+  
+  @piglet = Piglet::Interpreter.new
+  @piglet.interpret do
+    store(load('input'), 'output')
+  end
+  puts @piglet.to_pig_latin
+  
+or
+
+  Piglet::Interpreter.new { store(load('input'), 'output') }.to_pig_latin
+  
+will print 
+
+  relation_1 = LOAD 'input';
+  STORE relation_1 INTO 'output';
+  
+to standard out.
+  
+== Examples of what it can do
+
+  a = load 'input', :schema => [:a, :b, :c]
+  b = a.group :c
+  c = b.foreach { |r| [r[0], r[1].a.max, r[1].b.max] }
+  store c, 'output'
+
+will result in the following Pig Latin:
+
+  relation_3 = LOAD 'input' AS (a, b, c);
+  relation_2 = GROUP relation_3 BY c;
+  relation_1 = FOREACH relation_2 GENERATE $0, MAX($1.a), MAX($1.b);
+  STORE relation_1 INTO 'output';
+
+== Syntax
+
+There are two kinds of operators in Piglet: load & store operators, and relation operators. Load & store are called as functions with no receiver, like this:
+
+  load('input')
+  store(a, 'out')
+  describe(b)
+  illustrate(c)
+  dump(d)
+  explain(e)
+  
+and those are also all the load & store operators. They mirror the Pig Latin operators +LOAD+, +STORE+, +DESCRIBE+, +ILLUSTRATE+, +DUMP+ and +EXPLAIN+.
+
+Relation operators are called as methods on relations. Relations are created by the +load+ operator, and can be stored in regular variables:
+
+  a = load('input', :schema => [:x, :y, :z])
+  b = a.group(:x)
+  store(b, 'out')
+  
+Unlinke Pig Latin, operators can be chained:
+
+  a = load('input', :schema => [:x, :y, :z])
+  b = a.sample(3).group(:x)
+  store(b, 'out')
+  
+In fact, a whole script can be written without using variables at all:
+
+  store(load('input', :schema => [:x, :y, :z]).sample(3).group(:x))
+  
+The relation operators are meant to be close to the Pig Latin syntax, but there are obvious limitations and tradeoffs, see the documentation for the Piglet::Relation mixin for syntax examples.
+
+=== <code>load</code>
+
+When loading a relation you can specify the schema by passing the <code>:schema</code> option to +load+. The syntax of the schema specification is not perfect at this time: if you don't care about types you can pass an array of symbols or strings, like this:
+
+  load('input', :schema => %w(a b c d))
+  load('input', :schema => [:a, :b, :c, :d])
+  
+But if you want types, then you need to pass an array of arrays, where the inner arrays contain the field name and the field type:
+
+  load('input', :schema => [[:a, :chararray], [:b, :long]])
+  
+This is a bit inconvenient. I would like to use a hash, like this: <code>{:a => :chararray, :b => :long}</code>, but since the order of the keys isn't guaranteed in Ruby 1.8, it's not possible. I'm working on something better.
+
+You can also specify a load function by passing the <code>:using</code> option:
+
+  load('input', :using => :pig_storage)
+  load('input', :using => 'MyOwnFunction')
+  
+Piglet knows to translate <code>:pig_storage</code> to <code>PigStorage</code>, as well as the other pre-defined load and store functions: <code>:binary_serializer</code>, <code>:binary_deserializer</code>, <code>:bin_storage</code>, <code>:pig_dump</code> and <code>:text_loader</code>.
+
+== <code>store</code>, <code>dump</code>, <code>describe</code>, etc.
+
++store+ works similarily to +load+, but it takes a relation as its first argument, and the path to the output as second. It too takes the option <code>:using</code>, with the same values as +load+.
+
++dump+, +describe+, +illustrate+ and +explain+ all take a relation as sole argument. +explain+ can be called without argument (see the Pig Latin manual for what +EXPLAIN+ without argument does).
+
+=== Putting it all together
+
+Let's look at a more complex example:
+
+  students = load('students.txt', :schema => [%w(student chararray), %w(age int), %w(grade int)])
+  top_acheivers = students.filter { |r| r.grade == 5 }
+  name_and_age = top_acheivers.foreach { |r| [r.student.as(:name), r.age] }
+  name_by_age = name_and_age.group(:age)
+  count_by_age = name_by_age.foreach { |r| [r[0].as(:age), r[1].name.count.as(:count)]}
+  store(count_by_age, 'student_counts_by_age.txt', :using => :pig_storage)
+
+We load the file <code>students.txt</code> as a relation with three fields: <code>student</code>, a string, <code>age</code> an integer and <code>grade</code> another integer. Next we filter out the top acheivers with +filter+. +filter+ takes a block and that block gets a referece to the relation (the one +filter+ was called on), the result of the block will be the filter expression, in this case it's <code>grade == 5</code>.
+
+When we have the top acheivers we want to make a projection to remove the grades field, since we will not use it in the next set of calculations. In Pig Latin this is done with <code>FOREACH … GENERATE</code>, which is just +foreach+ in Piglet. Like +filter+, +foreach+ takes a block that gets a reference to the relation. The result of the block should be an array of expressions, and in this case it's <code>[r.student.as(:name), r.age]</code>, which means the student field from the relation, renamed to "name" and the age field. The resulting relation will have two fields: "name" and "age".
+
+On the next line we group the relation by the age field, and following that we do another projection, this time on the grouped relation. Remember that when doing a grouping in Pig you get a relation that in this case looks like this: <code>(group:int, name_by_age:{name:chararray, age:int})</code>. In the block passed to +foreach+ we use <code>r[0]</code> and <code>r[1]</code> to reference the first and second fields of <code>name_by_age</code>, equivalent to <code>$0</code> and <code>$1</code> in Pig Latin. In Pig Latin you could also have used the names <code>group</code> and <code>name_by_age</code>, but for a number of reasons you can't do that in Piglet (<code>r.group</code> unfortunately refers to the <code>group</code> method, and the relation isn't actually called <code>name_by_age</code> after Piglet has translated the code into Pig Latin).
+
+The expression <code>r[1].name.count.as(:count)</code> means take the "name" field from the relation in the second field of the relation (<code>$1.name</code>), run the <code>COUNT</code> operator on it, and rename it <code>count</code>, i.e. <code>COUNT($1.name) AS count</code>.
+
+Finally we store the result in a file called <code>student_counts_by_age.txt</code>, using PigStorage (which isn't strictly necessary to specify since it's the default. If you have a custom method you can pass its name as a string instead of <code>:pig_storage</code>).
+
+Piglet will translate this into the following Pig Latin:
+
+  relation_5 = LOAD 'students.txt' AS (student:chararray, age:int, grade:int);
+  relation_4 = FILTER relation_5 BY grade == 5;
+  relation_3 = FOREACH relation_4 GENERATE student AS name, age;
+  relation_2 = GROUP relation_3 BY age;
+  relation_1 = FOREACH relation_2 GENERATE $0 AS age, COUNT($1.name) AS count;
+  STORE relation_1 INTO 'student_counts_by_age.txt' USING PigStorage;
+
+=== Going beyond Pig Latin
+
+My goal with Piglet was to add control of flow and reuse mechanisms to Pig, so I'd better show some of the things you can do:
+
+  input = load('input', :schema => %w(country browser site visit_duration))
+  %w(country browser site).each do |dimension|
+    grouped = input.group(dimension).foreach do |r|
+      [r[0], r[1].visit_duration.sum]
+    end
+    store(grouped, "output-#{dimension}")
+  end
+
+We load a file that contains an ID field, three dimensions (country, browser and site) and a metric: the duration of a visit. This could be data from a the logs of a set of websites, or an ad server. What we want to do is to sum the the <code>visit_duration</code> field for each of the three dimensions. This would be a big tedious in Pig Latin:
+
+  input = LOAD 'input' AS (country browser site visit_duration);
+  by_country = GROUP input BY country;
+  by_browser = GROUP input BY browser;
+  by_site = GROUP input BY site;
+  sum_by_country = FOREACH by_country GENERATE $0, SUM($1.visit_duration);
+  sum_by_browser = FOREACH by_browser GENERATE $0, SUM($1.visit_duration);
+  sum_by_site = FOREACH by_site GENERATE $0, SUM($1.visit_duration);
+  STORE sum_by_country INTO 'output-country;
+  STORE sum_by_browser INTO 'output-browser;
+  STORE sum_by_site INTO 'output-site;
+  
+But in Piglet it's as simple as looping over the names of the dimensions. You could even define a method that encapsulates the grouping, summing and storing (although in this case it would be a bit overkill):
+
+  def sum_dimension(relation, dimension)
+    grouped = relation.group(dimension).foreach do |r|
+      [r[0], r[1].visit_duration.sum]
+    end
+    store(grouped, "output-#{dimension}")
+  end
+
+  input = load('input', :schema => %w(country browser site visit_duration))
+  %w(country browser site).each do |dimension|
+    sum_dimension(input, dimension)
+  end
+
+You can even define your own relation operations if you want, just add them to Piglet::Relation:
+
+  module Piglet::Relation
+    # Returns a list of sampled relations for each given sample size
+    def samples(*sizes)
+      sizes.map { |s| sample(s) }
+    end
+  end
+  
+and then use them just as any other operator:
+
+  small, medium, large = input.samples(0.01, 0.1, 0.5)
+  
+nifty, huh?
+
+== Limitations
+  
+The aim is to support most of Pig Latin, but currently there are some limitations.
+
+The following Pig operators are supported:
+
+* +COGROUP+
+* +CROSS+
+* +DESCRIBE+
+* +DISTINCT+
+* +DUMP+
+* +EXPLAIN+
+* +FILTER+
+* <code>FOREACH … GENERATE</code>
+* +GROUP+
+* +ILLUSTRATE+
+* +JOIN+
+* +LIMIT+
+* +LOAD+
+* +ORDER+
+* +SAMPLE+
+* +SPLIT+
+* +STORE+
+* +UNION+
+ 
+The following are currently not supported (but will be soon):
+
+* +STREAM+
+* +DEFINE+
+* +DECLARE+
+* +REGISTER+
+
+The file commands (+cd+, +cat+, etc.) will probably not be supported for the forseeable future.
+
+All the aggregate functions are supported:
+
+* +AVG+
+* +CONCAT+
+* +COUNT+
+* +DIFF+
+* +IsEmpty+
+* +MAX+
+* +MIN+
+* +SIZE+
+* +SUM+
+* +TOKENIZE+
+  
+Piglet only supports a limited set of arithmetic, comparison and boolean operators, so in practice the support for +FILTER+ and <code>FOREACH … GENERATE</code> is limited. This will be remedied soon, although the <code>!=</code> operator may never be supported in Ruby 1.8, since the <code>!</code> operator cannot be overridden (so there will be some less good looking solution like <code>.not</code>).
+
+Piglet does not support:
+
+* <code>!=</code> (not equals, you have to use <code>==</code> and a <code>NOT</code>, e.g. <code>(a == b).not</code>, which will be translated as <code>NOT (a == b)</code> or you can use <code>#ne</code>, which will translate to !=, e.g. <code>a.ne(b)</code> will become <code>a != b</code>)
+* <code>? :</code> (the ternary operator)
+* <code>-</code> (negation, but you can use <code>#neg</code> on a field expression to get the same result, e.g. <code>a.neg</code> will be translated as <code>-a</code>)
+* <code>key#value</code> (map dereferencing)
+
+=== Why aren't the aliases in the Pig Latin the same as the variable names in the Piglet script?
+
+When you run +piglet+ on a Piglet script the aliases in the output will be <tt>relation_1</tt>, <tt>relation_2</tt>, <tt>relation_3</tt>, and so on, instead of the names of the variables of the Piglet script -- like in the example at the top of this document.
+  
+The names +a+ and +b+ are lost in translation, this is unfortunate but hard to avoid. Firstly there is no way to discover the names of variables, and secondly there is no correspondence between a statement in a Piglet script and a statement in Pig Latin, <code>a.union(b, c).sample(3).group(:x)</code> is at least three statements in Pig Latin. It simply wouldn't be worth the extra complexity of trying to infer some variable names and reuse them as aliases in the Pig Latin output.
+
+In the future I may add a way of manually suggesting relation aliases, so that the Pig Latin output is more readable.
+
+You may also wonder why the relation aliases aren't in consecutive order. The reason is that they get their names in the order they are evaluated, and the interpreter walks the relation ancestry upwards from a +store+ (and it only evaluates a relation once).
+
+=== Why aren’t all operations included in the output?
+
+If you try this Piglet code:
+
+  a = load 'input'
+  b = a.group :c
+  
+You might be surprised that Piglet will not output anything. In fact, Piglet only creates Pig Latin operations on relations that will somehow be outputed. Unless there is a +store+, +dump+, +describe+, +illustrate+ or +explain+ that outputs a relation, the operations applied to that relation and its ancestors will not be included.
+
+When you call +group+, +filter+ or any of the other methods that can be applied to a relation a datastructure that encodes these operations is created. When a relation is passed to +store+ or one of the other output operators the Piglet interpreter traverses the datastructure backwards, building the Pig Latin operations needed to arrive at the relation that should be passed to the output operator. This is similar to how Pig itself interprets a Pig Latin script.
+
+As a side effect of using +store+ and the other output operators as the trigger for creating the needed relational operations any relations that are not ancestors of relations that are outputed will not be included in the Pig Latin output. On the other hand, they would be no-ops when run by Pig anyway.
+
+== Copyright
+
+© 2009-2010 Theo Hultberg / Iconara. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,50 @@
+require 'rubygems'
+require 'rake'
+require 'lib/piglet'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "piglet"
+    gem.summary = %Q{Piglet is a DSL for Pig scripts}
+    gem.description = %Q{Piglet aims to look like Pig Latin while allowing for things like loops and control of flow that are missing from Pig.}
+    gem.email = "theo@iconara.net"
+    gem.homepage = "http://github.com/iconara/piglet"
+    gem.authors = ["Theo Hultberg"]
+    gem.add_development_dependency "rspec", ">= 1.2.9"
+    gem.version = Piglet::VERSION
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'spec/rake/spectask'
+
+Spec::Rake::SpecTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :spec => :check_dependencies
+
+task :default => :spec
+
+require 'rake/rdoctask'
+
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "piglet #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+  rdoc.options << '--charset' << 'utf-8'
+end

data/bin/piglet

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+$: << File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
+
+
+require 'piglet'
+
+
+puts Piglet::Interpreter.new { ARGV.each { |path| eval(open(path).read) } }.to_pig_latin