cascading.jruby 0.0.10 → 1.0.0

data/LICENSE.txt

@@ -1,165 +1,18 @@
-		   GNU LESSER GENERAL PUBLIC LICENSE
-                       Version 3, 29 June 2007
+License:
+  Project and contact information: http://github.com/mrwalker/cascading.jruby
 
- Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
 
+      http://www.apache.org/licenses/LICENSE-2.0
 
-  This version of the GNU Lesser General Public License incorporates
-the terms and conditions of version 3 of the GNU General Public
-License, supplemented by the additional permissions listed below.
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
 
-  0. Additional Definitions. 
+Third-party Licenses:
 
-  As used herein, "this License" refers to version 3 of the GNU Lesser
-General Public License, and the "GNU GPL" refers to version 3 of the GNU
-General Public License.
-
-  "The Library" refers to a covered work governed by this License,
-other than an Application or a Combined Work as defined below.
-
-  An "Application" is any work that makes use of an interface provided
-by the Library, but which is not otherwise based on the Library.
-Defining a subclass of a class defined by the Library is deemed a mode
-of using an interface provided by the Library.
-
-  A "Combined Work" is a work produced by combining or linking an
-Application with the Library.  The particular version of the Library
-with which the Combined Work was made is also called the "Linked
-Version".
-
-  The "Minimal Corresponding Source" for a Combined Work means the
-Corresponding Source for the Combined Work, excluding any source code
-for portions of the Combined Work that, considered in isolation, are
-based on the Application, and not on the Linked Version.
-
-  The "Corresponding Application Code" for a Combined Work means the
-object code and/or source code for the Application, including any data
-and utility programs needed for reproducing the Combined Work from the
-Application, but excluding the System Libraries of the Combined Work.
-
-  1. Exception to Section 3 of the GNU GPL.
-
-  You may convey a covered work under sections 3 and 4 of this License
-without being bound by section 3 of the GNU GPL.
-
-  2. Conveying Modified Versions.
-
-  If you modify a copy of the Library, and, in your modifications, a
-facility refers to a function or data to be supplied by an Application
-that uses the facility (other than as an argument passed when the
-facility is invoked), then you may convey a copy of the modified
-version:
-
-   a) under this License, provided that you make a good faith effort to
-   ensure that, in the event an Application does not supply the
-   function or data, the facility still operates, and performs
-   whatever part of its purpose remains meaningful, or
-
-   b) under the GNU GPL, with none of the additional permissions of
-   this License applicable to that copy.
-
-  3. Object Code Incorporating Material from Library Header Files.
-
-  The object code form of an Application may incorporate material from
-a header file that is part of the Library.  You may convey such object
-code under terms of your choice, provided that, if the incorporated
-material is not limited to numerical parameters, data structure
-layouts and accessors, or small macros, inline functions and templates
-(ten or fewer lines in length), you do both of the following:
-
-   a) Give prominent notice with each copy of the object code that the
-   Library is used in it and that the Library and its use are
-   covered by this License.
-
-   b) Accompany the object code with a copy of the GNU GPL and this license
-   document.
-
-  4. Combined Works.
-
-  You may convey a Combined Work under terms of your choice that,
-taken together, effectively do not restrict modification of the
-portions of the Library contained in the Combined Work and reverse
-engineering for debugging such modifications, if you also do each of
-the following:
-
-   a) Give prominent notice with each copy of the Combined Work that
-   the Library is used in it and that the Library and its use are
-   covered by this License.
-
-   b) Accompany the Combined Work with a copy of the GNU GPL and this license
-   document.
-
-   c) For a Combined Work that displays copyright notices during
-   execution, include the copyright notice for the Library among
-   these notices, as well as a reference directing the user to the
-   copies of the GNU GPL and this license document.
-
-   d) Do one of the following:
-
-       0) Convey the Minimal Corresponding Source under the terms of this
-       License, and the Corresponding Application Code in a form
-       suitable for, and under terms that permit, the user to
-       recombine or relink the Application with a modified version of
-       the Linked Version to produce a modified Combined Work, in the
-       manner specified by section 6 of the GNU GPL for conveying
-       Corresponding Source.
-
-       1) Use a suitable shared library mechanism for linking with the
-       Library.  A suitable mechanism is one that (a) uses at run time
-       a copy of the Library already present on the user's computer
-       system, and (b) will operate properly with a modified version
-       of the Library that is interface-compatible with the Linked
-       Version. 
-
-   e) Provide Installation Information, but only if you would otherwise
-   be required to provide such information under section 6 of the
-   GNU GPL, and only to the extent that such information is
-   necessary to install and execute a modified version of the
-   Combined Work produced by recombining or relinking the
-   Application with a modified version of the Linked Version. (If
-   you use option 4d0, the Installation Information must accompany
-   the Minimal Corresponding Source and Corresponding Application
-   Code. If you use option 4d1, you must provide the Installation
-   Information in the manner specified by section 6 of the GNU GPL
-   for conveying Corresponding Source.)
-
-  5. Combined Libraries.
-
-  You may place library facilities that are a work based on the
-Library side by side in a single library together with other library
-facilities that are not Applications and are not covered by this
-License, and convey such a combined library under terms of your
-choice, if you do both of the following:
-
-   a) Accompany the combined library with a copy of the same work based
-   on the Library, uncombined with any other library facilities,
-   conveyed under the terms of this License.
-
-   b) Give prominent notice with the combined library that part of it
-   is a work based on the Library, and explaining where to find the
-   accompanying uncombined form of the same work.
-
-  6. Revised Versions of the GNU Lesser General Public License.
-
-  The Free Software Foundation may publish revised and/or new versions
-of the GNU Lesser General Public License from time to time. Such new
-versions will be similar in spirit to the present version, but may
-differ in detail to address new problems or concerns.
-
-  Each version is given a distinguishing version number. If the
-Library as you received it specifies that a certain numbered version
-of the GNU Lesser General Public License "or any later version"
-applies to it, you have the option of following the terms and
-conditions either of that published version or of any later version
-published by the Free Software Foundation. If the Library as you
-received it does not specify a version number of the GNU Lesser
-General Public License, you may choose any version of the GNU Lesser
-General Public License ever published by the Free Software Foundation.
-
-  If the Library as you received it specifies that a proxy can decide
-whether future versions of the GNU Lesser General Public License shall
-apply, that proxy's public statement of acceptance of any version is
-permanent authorization for you to choose that version for the
-Library.
+  All third-party dependencies are listed in ivy.xml.

data/README.md

@@ -0,0 +1,35 @@
+# Cascading.JRuby [![Build Status](https://secure.travis-ci.org/mrwalker/cascading.jruby.png)](http://travis-ci.org/mrwalker/cascading.jruby)
+
+cascading.jruby is a DSL for [Cascading](http://www.cascading.org/), which is a dataflow API written in Java.  With cascading.jruby, Ruby programmers can rapidly script efficient MapReduce jobs for Hadoop.
+
+To give you a quick idea of what a cascading.jruby job looks like, here's word count:
+
+```ruby
+require 'rubygems'
+require 'cascading'
+
+input_path = ARGV.shift || (raise 'input_path required')
+
+cascade 'wordcount', :mode => :local do
+  flow 'wordcount' do
+    source 'input', tap(input_path)
+
+    assembly 'input' do
+      split_rows 'line', /[.,]*\s+/, 'word', :output => 'word'
+      group_by 'word' do
+        count
+      end
+    end
+
+    sink 'input', tap('output/wordcount', :sink_mode => :replace)
+  end
+end.complete
+```
+
+cascading.jruby provides a clean Ruby interface to Cascading, but doesn't attempt to add abstractions on top of it.  Therefore, you should be acquainted with the [Cascading](http://docs.cascading.org/cascading/2.0/userguide/html/) [API](http://docs.cascading.org/cascading/2.0/javadoc/) before you begin.
+
+For operations you can apply to your dataflow within a pipe assembly, see the [Assembly](http://rubydoc.info/gems/cascading.jruby/1.0.0/Cascading/Assembly) class.  For operations available within a block passed to a group_by, union, or join, see the [Aggregations](http://rubydoc.info/gems/cascading.jruby/1.0.0/Cascading/Aggregations) class.
+
+Note that the Ruby code you write merely constructs a Cascading job, so no JRuby runtime is required on your cluster.  This stands in contrast with writing [Hadoop streaming jobs in Ruby](http://www.quora.com/How-do-the-different-options-for-Ruby-on-Hadoop-compare).  To run cascading.jruby applications on a Hadoop cluster, you must use [Jading](https://github.com/mrwalker/jading) to package them into a job jar.
+
+cascading.jruby has been tested on JRuby versions 1.2.0, 1.4.0, 1.5.3, 1.6.5, 1.6.7.2, 1.7.0, and 1.7.3.

data/lib/cascading.rb

@@ -2,59 +2,26 @@ require 'java'
 
 module Cascading
   # :stopdoc:
-  VERSION = '0.0.10'
-  LIBPATH = ::File.expand_path(::File.dirname(__FILE__)) + ::File::SEPARATOR
-  PATH = ::File.dirname(LIBPATH) + ::File::SEPARATOR
-  CASCADING_HOME = ENV['CASCADING_HOME']
-  HADOOP_HOME = ENV['HADOOP_HOME']
-
-  # :startdoc:
-
-  # Returns the version string for the library.
-  #
-  def self.version
-    VERSION
-  end
-
-  # Returns the library path for the module. If any arguments are given,
-  # they will be joined to the end of the libray path using
-  # <tt>File.join</tt>.
-  #
-  def self.libpath( *args )
-    args.empty? ? LIBPATH : ::File.join(LIBPATH, args.flatten)
-  end
-
-  # Returns the lpath for the module. If any arguments are given,
-  # they will be joined to the end of the path using
-  # <tt>File.join</tt>.
-  #
-  def self.path( *args )
-    args.empty? ? PATH : ::File.join(PATH, args.flatten)
-  end
-
-  def self.require_all_jars(from = ::File.join(::File.dirname(__FILE__), "..", "jars"))
-    search_me = ::File.expand_path(
-        ::File.join(from, '**', '*.jar'))
-    Dir.glob(search_me).sort.each do |jar|
-      require jar
-    end
-  end
+  VERSION = '1.0.0'
 end
 
-Cascading.require_all_jars(Cascading::HADOOP_HOME) if Cascading::HADOOP_HOME
-Cascading.require_all_jars(Cascading::CASCADING_HOME) if Cascading::CASCADING_HOME
-
+require 'cascading/aggregations'
 require 'cascading/assembly'
 require 'cascading/base'
 require 'cascading/cascade'
 require 'cascading/cascading'
 require 'cascading/cascading_exception'
 require 'cascading/expr_stub'
+require 'cascading/filter_operations'
 require 'cascading/flow'
+require 'cascading/identity_operations'
 require 'cascading/mode'
 require 'cascading/operations'
+require 'cascading/regex_operations'
 require 'cascading/scope'
+require 'cascading/sub_assembly'
 require 'cascading/tap'
+require 'cascading/text_operations'
 
-# include module to make them available at top package
+# include module to make it available at top level
 include Cascading

data/lib/cascading/aggregations.rb

@@ -1,28 +1,39 @@
-require 'cascading/operations'
 require 'cascading/scope'
 require 'cascading/ext/array'
 
 module Cascading
+  # Aggregations is the context available to you within the block of a group_by,
+  # union, or join that allows you to apply Every pipes to the result of those
+  # operations.  You may apply aggregators and buffers within this context
+  # subject to several rules laid out by Cascading.
+  #
   # Rules enforced by Aggregations:
   # * Contains either 1 Buffer or >= 1 Aggregator (explicitly checked)
-  # * No GroupBys, CoGroups, Joins, or Merges (methods for these pipes do not
-  # exist on Aggregations)
+  # * No GroupBys, CoGroups, Joins, or Merges (methods for these pipes do not exist on Aggregations)
   # * No Eaches (Aggregations#each does not exist)
   # * Aggregations may not branch (Aggregations#branch does not exist)
   #
   # Externally enforced rules:
   # * May be empty (in which case, Aggregations is not instantiated)
-  # * Must follow a GroupBy or CoGroup (not a Join or Merge)
+  # * Must follow a GroupBy or CoGroup (not a HashJoin or Merge)
   #
   # Optimizations:
-  # * If the leading Group is a GroupBy and all subsequent Everies are
-  # Aggregators that have a corresponding AggregateBy, Aggregations can replace
-  # the GroupBy/Aggregator pipe with a single composite AggregateBy
+  # * If the leading Group is a GroupBy and all subsequent Everies are Aggregators that have a corresponding AggregateBy, Aggregations can replace the GroupBy/Aggregator pipe with a single composite AggregateBy
+  #
+  # Aggregator and buffer DSL standard optional parameter names:
+  # [input] c.p.Every argument selector
+  # [into] c.o.Operation field declaration
+  # [output] c.p.Every output selector
   class Aggregations
-    include Operations
-
     attr_reader :assembly, :tail_pipe, :scope, :aggregate_bys
 
+    # Do not use this constructor directly; instead, pass a block containing
+    # the desired aggregations to a group_by, union, or join and it will be
+    # instantiated for you.
+    #
+    # Builds the context in which a sequence of Every aggregations may be
+    # evaluated in the given assembly appended to the given group pipe and with
+    # the given incoming_scopes.
     def initialize(assembly, group, incoming_scopes)
       @assembly = assembly
       @tail_pipe = group
@@ -32,23 +43,14 @@ module Cascading
       @aggregate_bys = tail_pipe.is_group_by ? [] : nil
     end
 
+    # Prints information about the scope of these Aggregations at the point at
+    # which it is called.  This allows you to trace the propagation of field
+    # names through your job and is handy for debugging.  See Scope for
+    # details.
     def debug_scope
       puts "Current scope of aggregations for '#{assembly.name}':\n  #{scope}\n----------\n"
     end
 
-    def make_pipe(type, parameters)
-      pipe = type.new(*parameters)
-
-      # Enforce 1 Buffer or >= 1 Aggregator rule
-      if tail_pipe.kind_of?(Java::CascadingPipe::Every)
-        raise 'Buffer must be sole aggregation' if tail_pipe.buffer? || (tail_pipe.aggregator? && pipe.buffer?)
-      end
-
-      @tail_pipe = pipe
-      @scope = Scope.outgoing_scope(tail_pipe, [scope])
-    end
-    private :make_pipe
-
     # We can replace these aggregations with the corresponding composite
     # AggregateBy if the leading Group was a GroupBy and all subsequent
     # Aggregators had a corresponding AggregateBy (which we've encoded in the
@@ -69,13 +71,27 @@ module Cascading
 
     # Builds an every pipe and adds it to the current list of aggregations.
     # Note that this list may be either exactly 1 Buffer or any number of
-    # Aggregators.
-    def every(*args)
-      options = args.extract_options!
-
-      in_fields = fields(args)
+    # Aggregators.  Exactly one of :aggregator or :buffer must be specified and
+    # :aggregator may be accompanied by a corresponding :aggregate_by.
+    #
+    # The named options are:
+    # [aggregator] A Cascading Aggregator, mutually exclusive with :buffer.
+    # [aggregate_by] A Cascading AggregateBy that corresponds to the given
+    #                :aggregator.  Only makes sense with the :aggregator option
+    #                and does not exist for all Aggregators.  Providing nothing
+    #                or nil will cause all Aggregations to operate as normal,
+    #                without being compiled into a composite AggregateBy.
+    # [buffer] A Cascading Buffer, mutually exclusive with :aggregator.
+    # [output] c.p.Every output selector.
+    #
+    # Example:
+    #    every 'field1', 'field2', :aggregator => sum_aggregator, :aggregate_by => sum_by, :output => all_fields
+    #    every fields(input_fields), :buffer => Java::SomePackage::SomeBuffer.new, :output => all_fields
+    def every(*args_with_options)
+      options, in_fields = args_with_options.extract_options!, fields(args_with_options)
       out_fields = fields(options[:output])
       operation = options[:aggregator] || options[:buffer]
+      raise 'every requires either :aggregator or :buffer' unless operation
 
       if options[:aggregate_by] && aggregate_bys
         aggregate_bys << options[:aggregate_by]
@@ -84,71 +100,152 @@ module Cascading
       end
 
       parameters = [tail_pipe, in_fields, operation, out_fields].compact
-      make_pipe(Java::CascadingPipe::Every, parameters)
-    end
+      every = make_pipe(Java::CascadingPipe::Every, parameters)
+      raise ':aggregator specified but c.o.Buffer provided' if options[:aggregator] && every.is_buffer
+      raise ':buffer specified but c.o.Aggregator provided' if options[:buffer] && every.is_aggregator
 
-    def assert_group(*args)
-      options = args.extract_options!
+      every
+    end
 
-      assertion = args[0]
+    # Builds an every assertion pipe given a c.o.a.Assertion and adds it to the
+    # current list of aggregations.  Note this breaks a chain of AggregateBys.
+    #
+    # The named options are:
+    # [level] The assertion level; defaults to strict.
+    def assert_group(assertion, options = {})
       assertion_level = options[:level] || Java::CascadingOperation::AssertionLevel::STRICT
 
       parameters = [tail_pipe, assertion_level, assertion]
       make_pipe(Java::CascadingPipe::Every, parameters)
     end
 
-    def assert_group_size_equals(*args)
-      options = args.extract_options!
-
-      assertion = Java::CascadingOperationAssertion::AssertGroupSizeEquals.new(args[0])
+    # Builds a pipe that asserts the size of the current group is the specified
+    # size for all groups.
+    def assert_group_size_equals(size, options = {})
+      assertion = Java::CascadingOperationAssertion::AssertGroupSizeEquals.new(size)
       assert_group(assertion, options)
     end
 
-    # Builds a series of every pipes for aggregation.
+    # Computes the minima of the specified fields within each group.  Fields
+    # may be a list or a map for renaming.  Note that fields are sorted by
+    # input name when a map is provided.
     #
-    # Args can either be a list of fields to aggregate and an options hash or
-    # a hash that maps input field name to output field name (similar to
-    # insert) and an options hash.
+    # The named options are:
+    # [ignore] Java Array of Objects of values to be ignored.
     #
-    # Options include:
-    # * <tt>:ignore</tt> a Java Array of Objects (for min and max) or Tuples
-    # (for first and last) of values for the aggregator to ignore
-    # * <tt>function</tt> is a symbol that is the method to call to construct
-    # the Cascading Aggregator.
-    def composite_aggregator(args, function)
-      field_map, options = extract_field_map(args)
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       insert 'const' => 1
+    #       group_by 'const' do
+    #         min 'field1', 'field2'
+    #         min 'field3' => 'fieldA', 'field4' => 'fieldB'
+    #       end
+    #       discard 'const'
+    #     end
+    def min(*args_with_options)
+      composite_aggregator(args_with_options, Java::CascadingOperationAggregator::Min)
+    end
 
-      field_map.each do |in_field, out_field|
-        agg = self.send(function, out_field, options)
-        every(in_field, :aggregator => agg, :output => all_fields)
-      end
-      raise "Composite aggregator '#{function.to_s.gsub('_function', '')}' invoked on 0 fields" if field_map.empty?
+    # Computes the maxima of the specified fields within each group.  Fields
+    # may be a list or a map for renaming.  Note that fields are sorted by
+    # input name when a map is provided.
+    #
+    # The named options are:
+    # [ignore] Java Array of Objects of values to be ignored.
+    #
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       insert 'const' => 1
+    #       group_by 'const' do
+    #         max 'field1', 'field2'
+    #         max 'field3' => 'fieldA', 'field4' => 'fieldB'
+    #       end
+    #       discard 'const'
+    #     end
+    def max(*args_with_options)
+      composite_aggregator(args_with_options, Java::CascadingOperationAggregator::Max)
+    end
+
+    # Returns the first value within each group for the specified fields.
+    # Fields may be a list or a map for renaming.  Note that fields are sorted
+    # by input name when a map is provided.
+    #
+    # The named options are:
+    # [ignore] Java Array of Tuples which should be ignored
+    #
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       group_by 'key1', 'key2' do
+    #         first 'field1', 'field2'
+    #         first 'field3' => 'fieldA', 'field4' => 'fieldB'
+    #       end
+    #     end
+    def first(*args_with_options)
+      composite_aggregator(args_with_options, Java::CascadingOperationAggregator::First)
     end
 
-    def min(*args); composite_aggregator(args, :min_function); end
-    def max(*args); composite_aggregator(args, :max_function); end
-    def first(*args); composite_aggregator(args, :first_function); end
-    def last(*args); composite_aggregator(args, :last_function); end
+    # Returns the last value within each group for the specified fields.
+    # Fields may be a list or a map for renaming.  Note that fields are sorted
+    # by input name when a map is provided.
+    #
+    # The named options are:
+    # [ignore] Java Array of Tuples which should be ignored
+    #
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       group_by 'key1', 'key2' do
+    #         last 'field1', 'field2'
+    #         last 'field3' => 'fieldA', 'field4' => 'fieldB'
+    #       end
+    #     end
+    def last(*args_with_options)
+      composite_aggregator(args_with_options, Java::CascadingOperationAggregator::Last)
+    end
 
-    # Counts elements of a group.  May optionally specify the name of the
-    # output count field (defaults to 'count').
+    # Counts elements of each group.  May optionally specify the name of the
+    # output count field, which defaults to 'count'.
+    #
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       group_by 'key1', 'key2' do
+    #         count
+    #         count 'key1_key2_count'
+    #       end
+    #     end
     def count(name = 'count')
       count_aggregator = Java::CascadingOperationAggregator::Count.new(fields(name))
       count_by = Java::CascadingPipeAssembly::CountBy.new(fields(name))
       every(last_grouping_fields, :aggregator => count_aggregator, :output => all_fields, :aggregate_by => count_by)
     end
 
-    # Sums one or more fields.  Fields to be summed may either be provided as
-    # the arguments to sum (in which case they will be aggregated into a field
-    # of the same name in the given order), or via a hash using the :mapping
-    # parameter (in which case they will be aggregated from the field named by
-    # the key into the field named by the value after being sorted).  The type
-    # of the output sum may be controlled with the :type parameter.
-    def sum(*args)
-      options = args.extract_options!
+    # Sums the specified fields within each group.  Fields may be a list or
+    # provided through the :mapping option for renaming.  Note that fields are
+    # sorted by name when a map is provided.
+    #
+    # The named options are:
+    # [mapping] Map of input to output field names if renaming is desired.
+    #           Results in output fields sorted by input field.
+    # [type] Controls the type of the output, specified using values from the
+    #        Cascading::JAVA_TYPE_MAP as in Janino expressions (:double, :long, etc.)
+    #
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       group_by 'key1', 'key2' do
+    #         sum 'field1', 'field2', :type => :long
+    #         sum :mapping => { 'field3' => 'fieldA', 'field4' => 'fieldB' }, :type => :double
+    #       end
+    #     end
+    def sum(*args_with_options)
+      options, in_fields = args_with_options.extract_options!, args_with_options
       type = JAVA_TYPE_MAP[options[:type]]
 
-      mapping = options[:mapping] ? options[:mapping].sort : args.zip(args)
+      mapping = options[:mapping] ? options[:mapping].sort : in_fields.zip(in_fields)
       mapping.each do |in_field, out_field|
         sum_aggregator = Java::CascadingOperationAggregator::Sum.new(*[fields(out_field), type].compact)
         # NOTE: SumBy requires a type in wip-286, unlike Sum (see Sum.java line 42 for default)
@@ -158,10 +255,22 @@ module Cascading
       raise "sum invoked on 0 fields (note :mapping must be provided to explicitly rename fields)" if mapping.empty?
     end
 
-    # Averages one or more fields.  The contract of average is identical to
-    # that of other composite aggregators, but it accepts no options.
-    def average(*args)
-      field_map, _ = extract_field_map(args)
+    # Averages the specified fields within each group.  Fields may be a list or
+    # a map for renaming.  Note that fields are sorted by input name when a map
+    # is provided.
+    #
+    # Examples:
+    #     assembly 'aggregate' do
+    #       ...
+    #       insert 'const' => 1
+    #       group_by 'const' do
+    #         max 'field1', 'field2'
+    #         max 'field3' => 'fieldA', 'field4' => 'fieldB'
+    #       end
+    #       discard 'const'
+    #     end
+    def average(*fields_or_field_map)
+      field_map, _ = extract_field_map(fields_or_field_map)
 
       field_map.each do |in_field, out_field|
         average_aggregator = Java::CascadingOperationAggregator::Average.new(fields(out_field))
@@ -173,6 +282,42 @@ module Cascading
 
     private
 
+    def make_pipe(type, parameters)
+      pipe = type.new(*parameters)
+
+      # Enforce 1 Buffer or >= 1 Aggregator rule
+      if tail_pipe.kind_of?(Java::CascadingPipe::Every)
+        raise 'Buffer must be sole aggregation' if tail_pipe.buffer? || (tail_pipe.aggregator? && pipe.buffer?)
+      end
+
+      @tail_pipe = pipe
+      @scope = Scope.outgoing_scope(tail_pipe, [scope])
+
+      tail_pipe
+    end
+
+    # Builds a series of every pipes for aggregation.
+    #
+    # Args can either be a list of fields to aggregate and an options hash or
+    # a hash that maps input field name to output field name (similar to
+    # insert) and an options hash.
+    #
+    # The named options are:
+    # [ignore] Java Array of Objects (for min and max) or Tuples (for first and
+    #          last) of values for the aggregator to ignore.
+    def composite_aggregator(args, aggregator)
+      field_map, options = extract_field_map(args)
+
+      field_map.each do |in_field, out_field|
+        every(
+          in_field,
+          :aggregator => aggregator.new(*[fields(out_field), options[:ignore]].compact),
+          :output => all_fields
+        )
+      end
+      raise "Composite aggregator '#{aggregator}' invoked on 0 fields" if field_map.empty?
+    end
+
     # Extracts a field mapping, input field => output field, by accepting a
     # hash in the first argument.  If no hash is provided, then maps arguments
     # onto themselves which names outputs the same as inputs.  Additionally