RubyGems - wukong - Versions diffs - 3.0.0.pre → 3.0.0.pre2 - Mend

wukong 3.0.0.pre → 3.0.0.pre2

Files changed (476) hide show

data/.gitignore +46 -33
data/.gitmodules +3 -0
data/.rspec +1 -1
data/.travis.yml +8 -1
data/.yardopts +0 -13
data/Guardfile +4 -6
data/{LICENSE.textile → LICENSE.md} +43 -55
data/README-old.md +422 -0
data/README.md +279 -418
data/Rakefile +21 -5
data/TODO.md +6 -6
data/bin/wu-clean-encoding +31 -0
data/bin/wu-lign +2 -2
data/bin/wu-local +69 -0
data/bin/wu-server +70 -0
data/examples/Gemfile +38 -0
data/examples/README.md +9 -0
data/examples/dataflow/apache_log_line.rb +64 -25
data/examples/dataflow/fibonacci_series.rb +101 -0
data/examples/dataflow/parse_apache_logs.rb +37 -7
data/examples/{dataflow.rb → dataflow/scraper_macro_flow.rb} +0 -0
data/examples/dataflow/simple.rb +4 -4
data/examples/geo.rb +4 -0
data/examples/geo/geo_grids.numbers +0 -0
data/examples/geo/geolocated.rb +331 -0
data/examples/geo/quadtile.rb +69 -0
data/examples/geo/spec/geolocated_spec.rb +247 -0
data/examples/geo/tile_fetcher.rb +77 -0
data/examples/graph/minimum_spanning_tree.rb +61 -61
data/examples/jabberwocky.txt +36 -0
data/examples/models/wikipedia.rb +20 -0
data/examples/munging/Gemfile +8 -0
data/examples/munging/airline_flights/airline.rb +57 -0
data/examples/munging/airline_flights/airline_flights.rake +83 -0
data/{lib/wukong/settings.rb → examples/munging/airline_flights/airplane.rb} +0 -0
data/examples/munging/airline_flights/airport.rb +211 -0
data/examples/munging/airline_flights/airport_id_unification.rb +129 -0
data/examples/munging/airline_flights/airport_ok_chars.rb +4 -0
data/examples/munging/airline_flights/flight.rb +156 -0
data/examples/munging/airline_flights/models.rb +4 -0
data/examples/munging/airline_flights/parse.rb +26 -0
data/examples/munging/airline_flights/reconcile_airports.rb +142 -0
data/examples/munging/airline_flights/route.rb +35 -0
data/examples/munging/airline_flights/tasks.rake +83 -0
data/examples/munging/airline_flights/timezone_fixup.rb +62 -0
data/examples/munging/airline_flights/topcities.rb +167 -0
data/examples/munging/airports/40_wbans.txt +40 -0
data/examples/munging/airports/filter_weather_reports.rb +37 -0
data/examples/munging/airports/join.pig +31 -0
data/examples/munging/airports/to_tsv.rb +33 -0
data/examples/munging/airports/usa_wbans.pig +19 -0
data/examples/munging/airports/usa_wbans.txt +2157 -0
data/examples/munging/airports/wbans.pig +19 -0
data/examples/munging/airports/wbans.txt +2310 -0
data/examples/munging/geo/geo_json.rb +54 -0
data/examples/munging/geo/geo_models.rb +69 -0
data/examples/munging/geo/geonames_models.rb +78 -0
data/examples/munging/geo/iso_codes.rb +172 -0
data/examples/munging/geo/reconcile_countries.rb +124 -0
data/examples/munging/geo/tasks.rake +71 -0
data/examples/munging/rake_helper.rb +62 -0
data/examples/munging/weather/.gitignore +1 -0
data/examples/munging/weather/Gemfile +4 -0
data/examples/munging/weather/Rakefile +28 -0
data/examples/munging/weather/extract_ish.rb +13 -0
data/examples/munging/weather/models/weather.rb +119 -0
data/examples/munging/weather/utils/noaa_downloader.rb +46 -0
data/examples/munging/wikipedia/README.md +34 -0
data/examples/munging/wikipedia/Rakefile +193 -0
data/examples/munging/wikipedia/articles/extract_articles-parsed.rb +79 -0
data/examples/munging/wikipedia/articles/extract_articles-templated.rb +136 -0
data/examples/munging/wikipedia/articles/textualize_articles.rb +54 -0
data/examples/munging/wikipedia/articles/verify_structure.rb +43 -0
data/examples/munging/wikipedia/articles/wp2txt-LICENSE.txt +22 -0
data/examples/munging/wikipedia/articles/wp2txt_article.rb +259 -0
data/examples/munging/wikipedia/articles/wp2txt_utils.rb +452 -0
data/examples/munging/wikipedia/dbpedia/dbpedia_common.rb +4 -0
data/examples/munging/wikipedia/dbpedia/dbpedia_extract_geocoordinates.rb +78 -0
data/examples/munging/wikipedia/dbpedia/extract_links.rb +193 -0
data/examples/munging/wikipedia/dbpedia/sameas_extractor.rb +20 -0
data/examples/munging/wikipedia/n1_subuniverse/n1_nodes.pig +18 -0
data/examples/munging/wikipedia/page_metadata/extract_page_metadata.rb +21 -0
data/examples/munging/wikipedia/page_metadata/extract_page_metadata.rb.old +27 -0
data/examples/munging/wikipedia/pagelinks/augment_pagelinks.pig +29 -0
data/examples/munging/wikipedia/pagelinks/extract_pagelinks.rb +14 -0
data/examples/munging/wikipedia/pagelinks/extract_pagelinks.rb.old +25 -0
data/examples/munging/wikipedia/pagelinks/undirect_pagelinks.pig +29 -0
data/examples/munging/wikipedia/pageviews/augment_pageviews.pig +32 -0
data/examples/munging/wikipedia/pageviews/extract_pageviews.rb +85 -0
data/examples/munging/wikipedia/pig_style_guide.md +25 -0
data/examples/munging/wikipedia/redirects/redirects_page_metadata.pig +19 -0
data/examples/munging/wikipedia/subuniverse/sub_articles.pig +23 -0
data/examples/munging/wikipedia/subuniverse/sub_page_metadata.pig +24 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_from.pig +22 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_into.pig +22 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_within.pig +26 -0
data/examples/munging/wikipedia/subuniverse/sub_pageviews.pig +29 -0
data/examples/munging/wikipedia/subuniverse/sub_undirected_pagelinks_within.pig +24 -0
data/examples/munging/wikipedia/utils/get_namespaces.rb +86 -0
data/examples/munging/wikipedia/utils/munging_utils.rb +68 -0
data/examples/munging/wikipedia/utils/namespaces.json +1 -0
data/examples/rake_helper.rb +85 -0
data/examples/server_logs/geo_ip_mapping/munge_geolite.rb +82 -0
data/examples/server_logs/logline.rb +95 -0
data/examples/server_logs/models.rb +66 -0
data/examples/server_logs/page_counts.pig +48 -0
data/examples/server_logs/server_logs-01-parse-script.rb +13 -0
data/examples/server_logs/server_logs-02-histograms-full.rb +33 -0
data/examples/server_logs/server_logs-02-histograms-mapper.rb +14 -0
data/{old/examples/server_logs/breadcrumbs.rb → examples/server_logs/server_logs-03-breadcrumbs-full.rb} +26 -30
data/examples/server_logs/server_logs-04-page_page_edges-full.rb +40 -0
data/examples/string_reverser.rb +26 -0
data/examples/text/pig_latin.rb +2 -2
data/examples/text/regional_flavor/README.md +14 -0
data/examples/text/regional_flavor/article_wordbags.pig +39 -0
data/examples/text/regional_flavor/j01-article_wordbags.rb +4 -0
data/examples/text/regional_flavor/simple_pig_script.pig +27 -0
data/examples/word_count/accumulator.rb +26 -0
data/examples/word_count/tokenizer.rb +13 -0
data/examples/word_count/word_count.rb +6 -0
data/examples/workflow/cherry_pie.dot +97 -0
data/examples/workflow/cherry_pie.png +0 -0
data/examples/workflow/cherry_pie.rb +61 -26
data/lib/hanuman.rb +34 -7
data/lib/hanuman/graph.rb +55 -31
data/lib/hanuman/graphvizzer.rb +199 -178
data/lib/hanuman/graphvizzer/gv_models.rb +161 -0
data/lib/hanuman/graphvizzer/gv_presenter.rb +97 -0
data/lib/hanuman/link.rb +35 -0
data/lib/hanuman/registry.rb +46 -0
data/lib/hanuman/stage.rb +76 -32
data/lib/wukong.rb +23 -24
data/lib/wukong/boot.rb +87 -0
data/lib/wukong/configuration.rb +8 -0
data/lib/wukong/dataflow.rb +45 -78
data/lib/wukong/driver.rb +99 -0
data/lib/wukong/emitter.rb +22 -0
data/lib/wukong/model/faker.rb +24 -24
data/lib/wukong/model/flatpack_parser/flat.rb +60 -0
data/lib/wukong/model/flatpack_parser/flatpack.rb +4 -0
data/lib/wukong/model/flatpack_parser/lang.rb +46 -0
data/lib/wukong/model/flatpack_parser/parser.rb +55 -0
data/lib/wukong/model/flatpack_parser/tokens.rb +130 -0
data/lib/wukong/processor.rb +60 -114
data/lib/wukong/spec_helpers.rb +81 -0
data/lib/wukong/spec_helpers/integration_driver.rb +144 -0
data/lib/wukong/spec_helpers/integration_driver_matchers.rb +219 -0
data/lib/wukong/spec_helpers/processor_helpers.rb +95 -0
data/lib/wukong/spec_helpers/processor_methods.rb +108 -0
data/lib/wukong/spec_helpers/shared_examples.rb +15 -0
data/lib/wukong/spec_helpers/spec_driver.rb +28 -0
data/lib/wukong/spec_helpers/spec_driver_matchers.rb +195 -0
data/lib/wukong/version.rb +2 -1
data/lib/wukong/widget/filters.rb +311 -0
data/lib/wukong/widget/processors.rb +156 -0
data/lib/wukong/widget/reducers.rb +7 -0
data/lib/wukong/widget/reducers/accumulator.rb +73 -0
data/lib/wukong/widget/reducers/bin.rb +318 -0
data/lib/wukong/widget/reducers/count.rb +61 -0
data/lib/wukong/widget/reducers/group.rb +85 -0
data/lib/wukong/widget/reducers/group_concat.rb +70 -0
data/lib/wukong/widget/reducers/moments.rb +72 -0
data/lib/wukong/widget/reducers/sort.rb +130 -0
data/lib/wukong/widget/serializers.rb +287 -0
data/lib/wukong/widget/sink.rb +10 -52
data/lib/wukong/widget/source.rb +7 -113
data/lib/wukong/widget/utils.rb +46 -0
data/lib/wukong/widgets.rb +6 -0
data/spec/examples/dataflow/fibonacci_series_spec.rb +18 -0
data/spec/examples/dataflow/parsing_spec.rb +12 -11
data/spec/examples/dataflow/simple_spec.rb +32 -6
data/spec/examples/dataflow/telegram_spec.rb +36 -36
data/spec/examples/graph/minimum_spanning_tree_spec.rb +30 -31
data/spec/examples/munging/airline_flights/identifiers_spec.rb +16 -0
data/spec/examples/munging/airline_flights_spec.rb +202 -0
data/spec/examples/text/pig_latin_spec.rb +13 -16
data/spec/examples/workflow/cherry_pie_spec.rb +34 -4
data/spec/hanuman/graph_spec.rb +27 -2
data/spec/hanuman/hanuman_spec.rb +10 -0
data/spec/hanuman/registry_spec.rb +123 -0
data/spec/hanuman/stage_spec.rb +61 -7
data/spec/spec_helper.rb +29 -19
data/spec/support/hanuman_test_helpers.rb +14 -12
data/spec/support/shared_context_for_reducers.rb +37 -0
data/spec/support/shared_examples_for_builders.rb +101 -0
data/spec/support/shared_examples_for_shortcuts.rb +57 -0
data/spec/support/wukong_test_helpers.rb +37 -11
data/spec/wukong/dataflow_spec.rb +77 -55
data/spec/wukong/local_runner_spec.rb +24 -24
data/spec/wukong/model/faker_spec.rb +132 -131
data/spec/wukong/runner_spec.rb +8 -8
data/spec/wukong/widget/filters_spec.rb +61 -0
data/spec/wukong/widget/processors_spec.rb +126 -0
data/spec/wukong/widget/reducers/bin_spec.rb +92 -0
data/spec/wukong/widget/reducers/count_spec.rb +11 -0
data/spec/wukong/widget/reducers/group_spec.rb +20 -0
data/spec/wukong/widget/reducers/moments_spec.rb +36 -0
data/spec/wukong/widget/reducers/sort_spec.rb +26 -0
data/spec/wukong/widget/serializers_spec.rb +92 -0
data/spec/wukong/widget/sink_spec.rb +15 -15
data/spec/wukong/widget/source_spec.rb +65 -41
data/spec/wukong/wukong_spec.rb +10 -0
data/wukong.gemspec +17 -10
metadata +359 -335
data/.document +0 -5
data/VERSION +0 -1
data/bin/hdp-bin +0 -44
data/bin/hdp-bzip +0 -23
data/bin/hdp-cat +0 -3
data/bin/hdp-catd +0 -3
data/bin/hdp-cp +0 -3
data/bin/hdp-du +0 -86
data/bin/hdp-get +0 -3
data/bin/hdp-kill +0 -3
data/bin/hdp-kill-task +0 -3
data/bin/hdp-ls +0 -11
data/bin/hdp-mkdir +0 -2
data/bin/hdp-mkdirp +0 -12
data/bin/hdp-mv +0 -3
data/bin/hdp-parts_to_keys.rb +0 -77
data/bin/hdp-ps +0 -3
data/bin/hdp-put +0 -3
data/bin/hdp-rm +0 -32
data/bin/hdp-sort +0 -40
data/bin/hdp-stream +0 -40
data/bin/hdp-stream-flat +0 -22
data/bin/hdp-stream2 +0 -39
data/bin/hdp-sync +0 -17
data/bin/hdp-wc +0 -67
data/bin/wu-flow +0 -10
data/bin/wu-map +0 -17
data/bin/wu-red +0 -17
data/bin/wukong +0 -17
data/data/CREDITS.md +0 -355
data/data/graph/airfares.tsv +0 -2174
data/data/text/gift_of_the_magi.txt +0 -225
data/data/text/jabberwocky.txt +0 -36
data/data/text/rectification_of_names.txt +0 -33
data/data/twitter/a_atsigns_b.tsv +0 -64
data/data/twitter/a_follows_b.tsv +0 -53
data/data/twitter/tweet.tsv +0 -167
data/data/twitter/twitter_user.tsv +0 -55
data/data/wikipedia/dbpedia-sentences.tsv +0 -1000
data/docpages/INSTALL.textile +0 -92
data/docpages/LICENSE.textile +0 -107
data/docpages/README-elastic_map_reduce.textile +0 -377
data/docpages/README-performance.textile +0 -90
data/docpages/README-wulign.textile +0 -65
data/docpages/UsingWukong-part1-get_ready.textile +0 -17
data/docpages/UsingWukong-part2-ThinkingBigData.textile +0 -75
data/docpages/UsingWukong-part3-parsing.textile +0 -138
data/docpages/_config.yml +0 -39
data/docpages/avro/avro_notes.textile +0 -56
data/docpages/avro/performance.textile +0 -36
data/docpages/avro/tethering.textile +0 -19
data/docpages/bigdata-tips.textile +0 -143
data/docpages/code/api_response_example.txt +0 -20
data/docpages/code/parser_skeleton.rb +0 -38
data/docpages/diagrams/MapReduceDiagram.graffle +0 -0
data/docpages/favicon.ico +0 -0
data/docpages/gem.css +0 -16
data/docpages/hadoop-tips.textile +0 -83
data/docpages/index.textile +0 -92
data/docpages/intro.textile +0 -8
data/docpages/moreinfo.textile +0 -174
data/docpages/news.html +0 -24
data/docpages/pig/PigLatinExpressionsList.txt +0 -122
data/docpages/pig/PigLatinReferenceManual.txt +0 -1640
data/docpages/pig/commandline_params.txt +0 -26
data/docpages/pig/cookbook.html +0 -481
data/docpages/pig/images/hadoop-logo.jpg +0 -0
data/docpages/pig/images/instruction_arrow.png +0 -0
data/docpages/pig/images/pig-logo.gif +0 -0
data/docpages/pig/piglatin_ref1.html +0 -1103
data/docpages/pig/piglatin_ref2.html +0 -14340
data/docpages/pig/setup.html +0 -505
data/docpages/pig/skin/basic.css +0 -166
data/docpages/pig/skin/breadcrumbs.js +0 -237
data/docpages/pig/skin/fontsize.js +0 -166
data/docpages/pig/skin/getBlank.js +0 -40
data/docpages/pig/skin/getMenu.js +0 -45
data/docpages/pig/skin/images/chapter.gif +0 -0
data/docpages/pig/skin/images/chapter_open.gif +0 -0
data/docpages/pig/skin/images/current.gif +0 -0
data/docpages/pig/skin/images/external-link.gif +0 -0
data/docpages/pig/skin/images/header_white_line.gif +0 -0
data/docpages/pig/skin/images/page.gif +0 -0
data/docpages/pig/skin/images/pdfdoc.gif +0 -0
data/docpages/pig/skin/images/rc-b-l-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-b-r-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-b-r-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2searchbox-3searchbox.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2tab-unselected-3tab-unselected.png +0 -0
data/docpages/pig/skin/images/rc-t-r-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2searchbox-3searchbox.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2tab-unselected-3tab-unselected.png +0 -0
data/docpages/pig/skin/print.css +0 -54
data/docpages/pig/skin/profile.css +0 -181
data/docpages/pig/skin/screen.css +0 -587
data/docpages/pig/tutorial.html +0 -1059
data/docpages/pig/udf.html +0 -1509
data/docpages/tutorial.textile +0 -283
data/docpages/usage.textile +0 -195
data/docpages/wutils.textile +0 -263
data/examples/dataflow/complex.rb +0 -11
data/examples/dataflow/donuts.rb +0 -13
data/examples/tiny_count/jabberwocky_output.tsv +0 -92
data/examples/word_count.rb +0 -48
data/examples/workflow/fiddle.rb +0 -24
data/lib/away/escapement.rb +0 -129
data/lib/away/exe.rb +0 -11
data/lib/away/experimental.rb +0 -5
data/lib/away/from_file.rb +0 -52
data/lib/away/job.rb +0 -56
data/lib/away/job/rake_compat.rb +0 -17
data/lib/away/registry.rb +0 -79
data/lib/away/runner.rb +0 -276
data/lib/away/runner/execute.rb +0 -121
data/lib/away/script.rb +0 -161
data/lib/away/script/hadoop_command.rb +0 -240
data/lib/away/source/file_list_source.rb +0 -15
data/lib/away/source/looper.rb +0 -18
data/lib/away/task.rb +0 -219
data/lib/hanuman/action.rb +0 -21
data/lib/hanuman/chain.rb +0 -4
data/lib/hanuman/graphviz.rb +0 -74
data/lib/hanuman/resource.rb +0 -6
data/lib/hanuman/slot.rb +0 -87
data/lib/hanuman/slottable.rb +0 -220
data/lib/wukong/bad_record.rb +0 -15
data/lib/wukong/event.rb +0 -44
data/lib/wukong/local_runner.rb +0 -55
data/lib/wukong/mapred.rb +0 -3
data/lib/wukong/universe.rb +0 -48
data/lib/wukong/widget/filter.rb +0 -81
data/lib/wukong/widget/gibberish.rb +0 -123
data/lib/wukong/widget/monitor.rb +0 -26
data/lib/wukong/widget/reducer.rb +0 -66
data/lib/wukong/widget/stringifier.rb +0 -50
data/lib/wukong/workflow.rb +0 -22
data/lib/wukong/workflow/command.rb +0 -42
data/old/config/emr-example.yaml +0 -48
data/old/examples/README.txt +0 -17
data/old/examples/contrib/jeans/README.markdown +0 -165
data/old/examples/contrib/jeans/data/normalized_sizes +0 -3
data/old/examples/contrib/jeans/data/orders.tsv +0 -1302
data/old/examples/contrib/jeans/data/sizes +0 -3
data/old/examples/contrib/jeans/normalize.rb +0 -20
data/old/examples/contrib/jeans/sizes.rb +0 -55
data/old/examples/corpus/bnc_word_freq.rb +0 -44
data/old/examples/corpus/bucket_counter.rb +0 -47
data/old/examples/corpus/dbpedia_abstract_to_sentences.rb +0 -86
data/old/examples/corpus/sentence_bigrams.rb +0 -53
data/old/examples/corpus/sentence_coocurrence.rb +0 -66
data/old/examples/corpus/stopwords.rb +0 -138
data/old/examples/corpus/words_to_bigrams.rb +0 -53
data/old/examples/emr/README.textile +0 -110
data/old/examples/emr/dot_wukong_dir/credentials.json +0 -7
data/old/examples/emr/dot_wukong_dir/emr.yaml +0 -69
data/old/examples/emr/dot_wukong_dir/emr_bootstrap.sh +0 -33
data/old/examples/emr/elastic_mapreduce_example.rb +0 -28
data/old/examples/network_graph/adjacency_list.rb +0 -74
data/old/examples/network_graph/breadth_first_search.rb +0 -72
data/old/examples/network_graph/gen_2paths.rb +0 -68
data/old/examples/network_graph/gen_multi_edge.rb +0 -112
data/old/examples/network_graph/gen_symmetric_links.rb +0 -64
data/old/examples/pagerank/README.textile +0 -6
data/old/examples/pagerank/gen_initial_pagerank_graph.pig +0 -57
data/old/examples/pagerank/pagerank.rb +0 -72
data/old/examples/pagerank/pagerank_initialize.rb +0 -42
data/old/examples/pagerank/run_pagerank.sh +0 -21
data/old/examples/sample_records.rb +0 -33
data/old/examples/server_logs/apache_log_parser.rb +0 -15
data/old/examples/server_logs/nook.rb +0 -48
data/old/examples/server_logs/nook/faraday_dummy_adapter.rb +0 -94
data/old/examples/server_logs/user_agent.rb +0 -40
data/old/examples/simple_word_count.rb +0 -82
data/old/examples/size.rb +0 -61
data/old/examples/stats/avg_value_frequency.rb +0 -86
data/old/examples/stats/binning_percentile_estimator.rb +0 -140
data/old/examples/stats/data/avg_value_frequency.tsv +0 -3
data/old/examples/stats/rank_and_bin.rb +0 -173
data/old/examples/stupidly_simple_filter.rb +0 -40
data/old/examples/word_count.rb +0 -75
data/old/graph/graphviz_builder.rb +0 -580
data/old/graph_easy/Attributes.pm +0 -4181
data/old/graph_easy/Graphviz.pm +0 -2232
data/old/wukong.rb +0 -18
data/old/wukong/and_pig.rb +0 -38
data/old/wukong/bad_record.rb +0 -18
data/old/wukong/datatypes.rb +0 -24
data/old/wukong/datatypes/enum.rb +0 -127
data/old/wukong/datatypes/fake_types.rb +0 -17
data/old/wukong/decorator.rb +0 -28
data/old/wukong/encoding/asciize.rb +0 -108
data/old/wukong/extensions.rb +0 -16
data/old/wukong/extensions/array.rb +0 -18
data/old/wukong/extensions/blank.rb +0 -93
data/old/wukong/extensions/class.rb +0 -189
data/old/wukong/extensions/date_time.rb +0 -53
data/old/wukong/extensions/emittable.rb +0 -69
data/old/wukong/extensions/enumerable.rb +0 -79
data/old/wukong/extensions/hash.rb +0 -167
data/old/wukong/extensions/hash_keys.rb +0 -16
data/old/wukong/extensions/hash_like.rb +0 -150
data/old/wukong/extensions/hashlike_class.rb +0 -47
data/old/wukong/extensions/module.rb +0 -2
data/old/wukong/extensions/pathname.rb +0 -27
data/old/wukong/extensions/string.rb +0 -65
data/old/wukong/extensions/struct.rb +0 -17
data/old/wukong/extensions/symbol.rb +0 -11
data/old/wukong/filename_pattern.rb +0 -74
data/old/wukong/helper.rb +0 -7
data/old/wukong/helper/stopwords.rb +0 -195
data/old/wukong/helper/tokenize.rb +0 -35
data/old/wukong/logger.rb +0 -38
data/old/wukong/periodic_monitor.rb +0 -72
data/old/wukong/schema.rb +0 -269
data/old/wukong/script.rb +0 -286
data/old/wukong/script/avro_command.rb +0 -5
data/old/wukong/script/cassandra_loader_script.rb +0 -40
data/old/wukong/script/emr_command.rb +0 -168
data/old/wukong/script/hadoop_command.rb +0 -237
data/old/wukong/script/local_command.rb +0 -41
data/old/wukong/store.rb +0 -10
data/old/wukong/store/base.rb +0 -27
data/old/wukong/store/cassandra.rb +0 -10
data/old/wukong/store/cassandra/streaming.rb +0 -75
data/old/wukong/store/cassandra/struct_loader.rb +0 -21
data/old/wukong/store/cassandra_model.rb +0 -91
data/old/wukong/store/chh_chunked_flat_file_store.rb +0 -37
data/old/wukong/store/chunked_flat_file_store.rb +0 -48
data/old/wukong/store/conditional_store.rb +0 -57
data/old/wukong/store/factory.rb +0 -8
data/old/wukong/store/flat_file_store.rb +0 -89
data/old/wukong/store/key_store.rb +0 -51
data/old/wukong/store/null_store.rb +0 -15
data/old/wukong/store/read_thru_store.rb +0 -22
data/old/wukong/store/tokyo_tdb_key_store.rb +0 -33
data/old/wukong/store/tyrant_rdb_key_store.rb +0 -57
data/old/wukong/store/tyrant_tdb_key_store.rb +0 -20
data/old/wukong/streamer.rb +0 -30
data/old/wukong/streamer/accumulating_reducer.rb +0 -83
data/old/wukong/streamer/base.rb +0 -126
data/old/wukong/streamer/counting_reducer.rb +0 -25
data/old/wukong/streamer/filter.rb +0 -20
data/old/wukong/streamer/instance_streamer.rb +0 -15
data/old/wukong/streamer/json_streamer.rb +0 -21
data/old/wukong/streamer/line_streamer.rb +0 -12
data/old/wukong/streamer/list_reducer.rb +0 -31
data/old/wukong/streamer/rank_and_bin_reducer.rb +0 -145
data/old/wukong/streamer/record_streamer.rb +0 -14
data/old/wukong/streamer/reducer.rb +0 -11
data/old/wukong/streamer/set_reducer.rb +0 -14
data/old/wukong/streamer/struct_streamer.rb +0 -48
data/old/wukong/streamer/summing_reducer.rb +0 -29
data/old/wukong/streamer/uniq_by_last_reducer.rb +0 -51
data/old/wukong/typed_struct.rb +0 -12
data/spec/away/encoding_spec.rb +0 -32
data/spec/away/exe_spec.rb +0 -20
data/spec/away/flow_spec.rb +0 -82
data/spec/away/graph_spec.rb +0 -6
data/spec/away/job_spec.rb +0 -15
data/spec/away/rake_compat_spec.rb +0 -9
data/spec/away/script_spec.rb +0 -81
data/spec/hanuman/graphviz_spec.rb +0 -29
data/spec/hanuman/slot_spec.rb +0 -2
data/spec/support/examples_helper.rb +0 -10
data/spec/support/streamer_test_helpers.rb +0 -6
data/spec/support/wukong_widget_helpers.rb +0 -66
data/spec/wukong/processor_spec.rb +0 -109
data/spec/wukong/widget/filter_spec.rb +0 -99
data/spec/wukong/widget/stringifier_spec.rb +0 -51
data/spec/wukong/workflow/command_spec.rb +0 -5

@@ -1,1509 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
-<html>
-<head>
-<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
-<meta content="Apache Forrest" name="Generator">
-<meta name="Forrest-version" content="0.8">
-<meta name="Forrest-skin-name" content="pelt">
-<title>Pig UDF Manual</title>
-<link type="text/css" href="skin/basic.css" rel="stylesheet">
-<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
-<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
-<link type="text/css" href="skin/profile.css" rel="stylesheet">
-<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
-<link rel="shortcut icon" href="">
-</head>
-<body onload="init()">
-<script type="text/javascript">ndeSetTextSize();</script>
-<div id="top">
-<!--+
-    |breadtrail
-    +-->
-<div class="breadtrail">
-<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/pig/">Pig</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
-</div>
-<!--+
-    |header
-    +-->
-<div class="header">
-<!--+
-    |start group logo
-    +-->
-<div class="grouplogo">
-<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
-</div>
-<!--+
-    |end group logo
-    +-->
-<!--+
-    |start Project Logo
-    +-->
-<div class="projectlogo">
-<a href="http://hadoop.apache.org/pig/"><img class="logoImage" alt="Pig" src="images/pig-logo.gif" title="A platform for analyzing large datasets."></a>
-</div>
-<!--+
-    |end Project Logo
-    +-->
-<!--+
-    |start Search
-    +-->
-<div class="searchbox">
-<form action="http://www.google.com/search" method="get" class="roundtopsmall">
-<input value="" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp;
-                    <input name="Search" value="Search" type="submit">
-</form>
-</div>
-<!--+
-    |end search
-    +-->
-<!--+
-    |start Tabs
-    +-->
-<ul id="tabs">
-<li>
-<a class="unselected" href="http://hadoop.apache.org/pig/">Project</a>
-</li>
-<li>
-<a class="unselected" href="http://wiki.apache.org/pig/">Wiki</a>
-</li>
-<li class="current">
-<a class="selected" href="index.html">Pig 0.7.0 Documentation</a>
-</li>
-</ul>
-<!--+
-    |end Tabs
-    +-->
-</div>
-</div>
-<div id="main">
-<div id="publishedStrip">
-<!--+
-    |start Subtabs
-    +-->
-<div id="level2tabs"></div>
-<!--+
-    |end Endtabs
-    +-->
-<script type="text/javascript"><!--
-document.write("Last Published: " + document.lastModified);
-//  --></script>
-</div>
-<!--+
-    |breadtrail
-    +-->
-<div class="breadtrail">
-             &nbsp;
-           </div>
-<!--+
-    |start Menu, mainarea
-    +-->
-<!--+
-    |start Menu
-    +-->
-<div id="menu">
-<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Pig</div>
-<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
-<div class="menuitem">
-<a href="index.html">Overview</a>
-</div>
-<div class="menuitem">
-<a href="setup.html">Setup</a>
-</div>
-<div class="menuitem">
-<a href="tutorial.html">Tutorial</a>
-</div>
-<div class="menuitem">
-<a href="piglatin_ref1.html">Pig Latin 1</a>
-</div>
-<div class="menuitem">
-<a href="piglatin_ref2.html">Pig Latin 2</a>
-</div>
-<div class="menuitem">
-<a href="cookbook.html">Cookbook</a>
-</div>
-<div class="menupage">
-<div class="menupagetitle">UDFs</div>
-</div>
-</div>
-<div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Zebra</div>
-<div id="menu_1.2" class="menuitemgroup">
-<div class="menuitem">
-<a href="zebra_overview.html">Zebra Overview </a>
-</div>
-<div class="menuitem">
-<a href="zebra_users.html">Zebra Users </a>
-</div>
-<div class="menuitem">
-<a href="zebra_reference.html">Zebra Reference </a>
-</div>
-<div class="menuitem">
-<a href="zebra_mapreduce.html">Zebra MapReduce </a>
-</div>
-<div class="menuitem">
-<a href="zebra_pig.html">Zebra Pig </a>
-</div>
-<div class="menuitem">
-<a href="zebra_stream.html">Zebra Streaming </a>
-</div>
-</div>
-<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">Miscellaneous</div>
-<div id="menu_1.3" class="menuitemgroup">
-<div class="menuitem">
-<a href="api/">API Docs</a>
-</div>
-<div class="menuitem">
-<a href="http://wiki.apache.org/pig/">Wiki</a>
-</div>
-<div class="menuitem">
-<a href="http://wiki.apache.org/pig/FAQ">FAQ</a>
-</div>
-<div class="menuitem">
-<a href="http://hadoop.apache.org/pig/releases.html">Release Notes</a>
-</div>
-</div>
-<div id="credit"></div>
-<div id="roundbottom">
-<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
-<!--+
-  |alternative credits
-  +-->
-<div id="credit2"></div>
-</div>
-<!--+
-    |end Menu
-    +-->
-<!--+
-    |start content
-    +-->
-<div id="content">
-<div title="Portable Document Format" class="pdflink">
-<a class="dida" href="udf.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
-        PDF</a>
-</div>
-<h1>Pig UDF Manual</h1>
-<div id="minitoc-area">
-<ul class="minitoc">
-<li>
-<a href="#Overview">Overview</a>
-</li>
-<li>
-<a href="#Eval+Functions">Eval Functions</a>
-<ul class="minitoc">
-<li>
-<a href="#How+to+Use+a+Simple+Eval+Function">How to Use a Simple Eval Function</a>
-</li>
-<li>
-<a href="#How+to+Write+a+Simple+Eval+Function"> How to Write a Simple Eval Function</a>
-</li>
-<li>
-<a href="#Aggregate+Functions">Aggregate Functions</a>
-</li>
-<li>
-<a href="#Filter+Functions"> Filter Functions</a>
-</li>
-<li>
-<a href="#Pig+Types"> Pig Types</a>
-</li>
-<li>
-<a href="#Schema"> Schema</a>
-</li>
-<li>
-<a href="#Error+Handling"> Error Handling</a>
-</li>
-<li>
-<a href="#Function+Overloading">Function Overloading</a>
-</li>
-<li>
-<a href="#Reporting+Progress">Reporting Progress</a>
-</li>
-<li>
-<a href="#Import+Lists">Import Lists</a>
-</li>
-</ul>
-</li>
-<li>
-<a href="#Load%2FStore+Functions"> Load/Store Functions</a>
-<ul class="minitoc">
-<li>
-<a href="#Load+Functions"> Load Functions</a>
-</li>
-<li>
-<a href="#Store+Functions"> Store Functions</a>
-</li>
-</ul>
-</li>
-<li>
-<a href="#Builtin+Functions+and+Function+Repositories">Builtin Functions and Function Repositories</a>
-</li>
-<li>
-<a href="#Accumulator+Interface">Accumulator Interface</a>
-</li>
-<li>
-<a href="#Advanced+Topics">Advanced Topics</a>
-<ul class="minitoc">
-<li>
-<a href="#Function+Instantiation">Function Instantiation</a>
-</li>
-<li>
-<a href="#Schemas">Schemas</a>
-</li>
-<li>
-<a href="#Passing+Configurations+to+UDFs">Passing Configurations to UDFs</a>
-</li>
-</ul>
-</li>
-</ul>
-</div>
-<a name="N1000D"></a><a name="Overview"></a>
-<h2 class="h3">Overview</h2>
-<div class="section">
-<p>Pig provides extensive support for user-defined functions (UDFs) as a way to specify custom processing.
-Functions can be a part of almost every operator in Pig.
-This document describes how to use existing functions as well as how to write your own functions.</p>
-</div>
-<a name="N10017"></a><a name="Eval+Functions"></a>
-<h2 class="h3">Eval Functions</h2>
-<div class="section">
-<a name="N1001D"></a><a name="How+to+Use+a+Simple+Eval+Function"></a>
-<h3 class="h4">How to Use a Simple Eval Function</h3>
-<p>Eval is the most common type of function. It can be used in <span class="codefrag">FOREACH</span> statements as shown in this script: </p>
-<pre class="code">
--- myscript.pig
-REGISTER myudfs.jar;
-A = LOAD 'student_data' AS (name: chararray, age: int, gpa: float);
-B = FOREACH A GENERATE myudfs.UPPER(name);
-DUMP B;
-</pre>
-<p>The command below can be used to run the script. Note that all examples in this document run in local mode for simplicity
-but the examples can also run in Hadoop mode. For more information on how to run Pig, please see the PigTutorial. </p>
-<pre class="code">
-java -cp pig.jar org.apache.pig.Main -x local myscript.pig
-</pre>
-<p>The first line of the script provides the location of the <span class="codefrag">jar&nbsp;file</span> that contains the UDF.
-(Note that there are no quotes around the jar file. Having quotes would result in a syntax error.)
-To locate the jar file, Pig first checks the <span class="codefrag">classpath</span>. If the jar file can't be found in the classpath,
-Pig assumes that the location is either an absolute path or a path relative to the location from which Pig was invoked.
-If the jar file can't be found, an error will be printed: <span class="codefrag">java.io.IOException:&nbsp;Can't&nbsp;read&nbsp;jar&nbsp;file:&nbsp;myudfs.jar</span>. </p>
-<p>Multiple <span class="codefrag">register</span> commands can be used in the same script. If the same fully-qualified function is present in multiple jars,
-the first occurrence will be used consistently with Java semantics. </p>
-<p>The name of the UDF has to be fully qualified with the package name or an error will be reported:
-<span class="codefrag">java.io.IOException:&nbsp;Cannot&nbsp;instantiate:UPPER</span>. Also, the function name is case sensitive (UPPER and upper are not the same).
-A UDF can take one or more parameters. The exact signature of the function should clear from its documentation. </p>
-<p>The function provided in this example takes an ASCII string and produces its uppercase version. If you are familiar with column transformation functions
-in SQL, you will recognize that UPPER fits this concept. However, as we will see later in the document, eval functions in Pig go beyond column
-transformation functions and include aggregate and filter functions. </p>
-<p>If you are just a user of UDFs, this is most of what you need to know about UDFs to use them in your code.  </p>
-<a name="N10053"></a><a name="How+to+Write+a+Simple+Eval+Function"></a>
-<h3 class="h4"> How to Write a Simple Eval Function</h3>
-<p>Let's now look at the implementation of the <span class="codefrag">UPPER</span> UDF. </p>
-<pre class="code">
-package myudfs;
-import java.io.IOException;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.impl.util.WrappedIOException;
-public class UPPER extends EvalFunc (String)
-{
-    public String exec(Tuple input) throws IOException {
-        if (input == null || input.size() == 0)
-            return null;
-        try{
-            String str = (String)input.get(0);
-            return str.toUpperCase();
-        }catch(Exception e){
-            throw WrappedIOException.wrap("Caught exception processing input row ", e);
-        }
-    }
-}
-</pre>
-<p>The first line indicates that the function is part of the <span class="codefrag">myudfs</span> package. The UDF class extends the <span class="codefrag">EvalFunc</span> class which is the base class for all eval functions. It is parameterized with the return type of the UDF which is a Java <span class="codefrag">String</span> in this case. We will look into the <span class="codefrag">EvalFunc</span> class in more detail later, but for now all we need to do is to implement the <span class="codefrag">exec</span> function. This function is invoked on every input tuple. The input into the function is a tuple with input parameters in the order they are passed to the function in the Pig script. In our example, it will contain a single string field corresponding to the student name. </p>
-<p>The first thing to decide is what to do with invalid data. This depends on the format of the data. If the data is of type <span class="codefrag">bytearray</span> it means that it has not yet been converted to its proper type. In this case, if the format of the data does not match the expected type, a NULL value should be returned. If, on the other hand, the input data is of another type, this means that the conversion has already happened and the data should be in the correct format. This is the case with our example and that's why it throws an error (line 16.) Note that <span class="codefrag">WrappedIOException</span> is a helper class to convert the actual exception to an IOException.  </p>
-<p>Also, note that lines 10-11 check if the input data is null or empty and if so returns null. </p>
-<p>The actual function implementation is on lines 13-14 and is self-explanatory. </p>
-<p>Now that we have the function implemented, it needs to be compiled and included in a jar. You will need to build <span class="codefrag">pig.jar</span> to compile your UDF. You can use the following set of commands to checkout the code from SVN repository and create pig.jar: </p>
-<pre class="code">
-svn co http://svn.apache.org/repos/asf/hadoop/pig/trunk
-cd trunk
-ant
-</pre>
-<p>You should see <span class="codefrag">pig.jar</span> in your current working directory. The set of commands below first compiles the function and then creates a jar file that contains it. </p>
-<pre class="code">
-cd myudfs
-javac -cp pig.jar UPPER.java
-cd ..
-jar -cf myudfs.jar myudfs
-</pre>
-<p>You should now see <span class="codefrag">myudfs.jar</span> in your current working directory. You can use this jar with the script described in the previous section. </p>
-<a name="N1009F"></a><a name="Aggregate+Functions"></a>
-<h3 class="h4">Aggregate Functions</h3>
-<p>Aggregate functions are another common type of eval function. Aggregate functions are usually applied to grouped data, as shown in this script: </p>
-<pre class="code">
--- myscript2.pig
-A = LOAD 'student_data' AS (name: chararray, age: int, gpa: float);
-B = GROUP A BY name;
-C = FOREACH B GENERATE group, COUNT(A);
-DUMP C;
-</pre>
-<p>The script above uses the <span class="codefrag">COUNT</span> function to count the number of students with the same name. There are a couple of things to note about this script. First, even though we are using a function, there is no <span class="codefrag">register</span> command. Second, the function is not qualified with the package name. The reason for both is that <span class="codefrag">COUNT</span> is a <span class="codefrag">builtin</span> function meaning that it comes with the Pig distribution. These are the only two differences between builtins and UDFs. Builtins are discussed in more detail later in this document.  </p>
-<p>An aggregate function is an eval function that takes a bag and returns a scalar value. One interesting and useful property of many aggregate functions is that they can be computed incrementally in a distributed fashion. We call these functions <span class="codefrag">algebraic</span>. <span class="codefrag">COUNT</span> is an example of an algebraic function because we can count the number of elements in a subset of the data and then sum the counts to produce a final output. In the Hadoop world, this means that the partial computations can be done by the map and combiner, and the final result can be computed by the reducer. </p>
-<p>It is very important for performance to make sure that aggregate functions that are algebraic are implemented as such. Let's look at the implementation of the COUNT function to see what this means. (Error handling and some other code is omitted to save space. The full code can be accessed <a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/builtin/COUNT.java?view=markup"> here</a>.</p>
-<pre class="code">
-public class COUNT extends EvalFunc (Long) implements Algebraic{
-    public Long exec(Tuple input) throws IOException {return count(input);}
-    public String getInitial() {return Initial.class.getName();}
-    public String getIntermed() {return Intermed.class.getName();}
-    public String getFinal() {return Final.class.getName();}
-    static public class Initial extends EvalFunc (Tuple) {
-        public Tuple exec(Tuple input) throws IOException {return TupleFactory.getInstance().newTuple(count(input));}
-    }
-    static public class Intermed extends EvalFunc (Tuple) {
-        public Tuple exec(Tuple input) throws IOException {return TupleFactory.getInstance().newTuple(sum(input));}
-    }
-    static public class Final extends EvalFunc (Long) {
-        public Tuple exec(Tuple input) throws IOException {return sum(input);}
-    }
-    static protected Long count(Tuple input) throws ExecException {
-        Object values = input.get(0);
-        if (values instanceof DataBag) return ((DataBag)values).size();
-        else if (values instanceof Map) return new Long(((Map)values).size());
-    }
-    static protected Long sum(Tuple input) throws ExecException, NumberFormatException {
-        DataBag values = (DataBag)input.get(0);
-        long sum = 0;
-        for (Iterator (Tuple) it = values.iterator(); it.hasNext();) {
-            Tuple t = it.next();
-            sum += (Long)t.get(0);
-        }
-        return sum;
-    }
-}
-</pre>
-<p>
-<span class="codefrag">COUNT</span> implements <span class="codefrag">Algebraic</span> interface which looks like this: </p>
-<pre class="code">
-public interface Algebraic{
-    public String getInitial();
-    public String getIntermed();
-    public String getFinal();
-}
-</pre>
-<p>For a function to be algebraic, it needs to implement <span class="codefrag">Algebraic</span> interface that consist of definition of three classes derived from <span class="codefrag">EvalFunc</span>. The contract is that the  <span class="codefrag">exec</span> function of the <span class="codefrag">Initial</span> class is called once and is passed the original input tuple. Its output is a tuple that contains partial results. The <span class="codefrag">exec</span> function of the <span class="codefrag">Intermed</span> class can be called zero or more times and takes as its input a tuple that contains partial results produced by the <span class="codefrag">Initial</span> class or by prior invocations of the <span class="codefrag">Intermed</span> class and produces a tuple with another partial result. Finally, the <span class="codefrag">exec</span> function of the <span class="codefrag">Final</span> class is called and produces the final result as a scalar type. </p>
-<p>Here's the way to think about this in the Hadoop world. The <span class="codefrag">exec</span> function of the <span class="codefrag">Initial</span> class is invoked once by the <span class="codefrag">map</span> process and produces partial results. The <span class="codefrag">exec</span> function of the <span class="codefrag">Intermed</span> class is invoked once by each <span class="codefrag">combiner</span> invocation (which can happen zero or more times) and also produces partial results. The <span class="codefrag">exec</span> function of the <span class="codefrag">Final</span> class is invoked once by the reducer and produces the final result. </p>
-<p>Take a look at the <span class="codefrag">COUNT</span> implementation to see how this is done. Note that the <span class="codefrag">exec</span> function of the <span class="codefrag">Initial</span> and <span class="codefrag">Intermed</span> classes is parameterized with <span class="codefrag">Tuple</span> and the <span class="codefrag">exec</span> of the <span class="codefrag">Final</span> class is parameterized with the real type of the function, which in the case of the <span class="codefrag">COUNT</span> is <span class="codefrag">Long</span>. Also, note that the fully-qualified name of the class needs to be returned from <span class="codefrag">getInitial</span>, <span class="codefrag">getIntermed</span>, and <span class="codefrag">getFinal</span> methods. </p>
-<a name="N1013F"></a><a name="Filter+Functions"></a>
-<h3 class="h4"> Filter Functions</h3>
-<p>Filter functions are eval functions that return a <span class="codefrag">boolean</span> value. Filter functions can be used anywhere a Boolean expression is appropriate, including the <span class="codefrag">FILTER</span> operator or <span class="codefrag">bincond</span> expression. </p>
-<p>The example below uses the <span class="codefrag">IsEmpy</span> builtin filter function to implement joins. </p>
-<pre class="code">
--- inner join
-A = LOAD 'student_data' AS (name: chararray, age: int, gpa: float);
-B = LOAD 'voter_data' AS (name: chararray, age: int, registration: chararay, contributions: float);
-C = COGROUP A BY name, B BY name;
-D = FILTER C BY not IsEmpty(A);
-E = FILTER D BY not IsEmpty(B);
-F = FOREACH E GENERATE flatten(A), flatten(B);
-DUMP F;
-</pre>
-<p>Note that, even if filtering is omitted, the same results will be produced because the <span class="codefrag">foreach</span> results is a cross product and cross products get rid of empty bags. However, doing up-front filtering is more efficient since it reduces the input of the cross product. </p>
-<pre class="code">
--- full outer join
-A = LOAD 'student_data' AS (name: chararray, age: int, gpa: float);
-B = LOAD 'voter_data' AS (name: chararray, age: int, registration: chararay, contributions: float);
-C = COGROUP A BY name, B BY name;
-D = FOREACH C GENERATE group, flatten((IsEmpty(A) ? null : A)), flatten((IsEmpty(B) ? null : B));
-dump D
-</pre>
-<p>The implementation of the <span class="codefrag">IsEmpty</span> function looks like this: </p>
-<pre class="code">
-import java.io.IOException;
-import java.util.Map;
-import org.apache.pig.FilterFunc;
-import org.apache.pig.backend.executionengine.ExecException;
-import org.apache.pig.data.DataBag;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.data.DataType;
-import org.apache.pig.impl.util.WrappedIOException;
-public class IsEmpty extends FilterFunc {
-    public Boolean exec(Tuple input) throws IOException {
-        if (input == null || input.size() == 0)
-            return null;
-        try {
-            Object values = input.get(0);
-            if (values instanceof DataBag)
-                return ((DataBag)values).size() == 0;
-            else if (values instanceof Map)
-                return ((Map)values).size() == 0;
-            else{
-                throw new IOException("Cannot test a " +
-                    DataType.findTypeName(values) + " for emptiness.");
-            }
-        } catch (ExecException ee) {
-            throw WrappedIOException.wrap("Caught exception processing input row ", ee);
-        }
-    }
-}
-</pre>
-<a name="N10170"></a><a name="Pig+Types"></a>
-<h3 class="h4"> Pig Types</h3>
-<p>The main thing to know about Pig's type system is that Pig uses native Java types for almost all of its types, as shown in this table. </p>
-<table class="ForrestTable" cellspacing="1" cellpadding="4">
-<tr>
-<th colspan="1" rowspan="1">
-Pig Type
-</th>
-<th colspan="1" rowspan="1">
-Java Class
-</th>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> bytearray </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> DataByteArray </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> chararray </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> String </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> int </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> Integer </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> long </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> Long </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> float </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> Float </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> double </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> Double </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> tuple </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> Tuple </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> bag </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> DataBag </p>
-</td>
-</tr>
-<tr>
-<td colspan="1" rowspan="1">
-<p> map </p>
-</td>
-<td colspan="1" rowspan="1">
-<p> Map&lt;Object, Object&gt; </p>
-</td>
-</tr>
-</table>
-<p>All Pig-specific classes are available <a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/data/"> here</a>. </p>
-<p>
-<span class="codefrag">Tuple</span> and <span class="codefrag">DataBag</span> are different in that they are not concrete classes but rather interfaces. This enables users to extend Pig with their own versions of tuples and bags. As a result, UDFs cannot directly instantiate bags or tuples; they need to go through factory classes: <span class="codefrag">TupleFactory</span> and <span class="codefrag">BagFactory</span>. </p>
-<p>The builtin <span class="codefrag">TOKENIZE</span> function shows how bags and tuples are created. A function takes a text string as input and returns a bag of words from the text. (Note that currently Pig bags always contain tuples.) </p>
-<pre class="code">
-package org.apache.pig.builtin;
-import java.io.IOException;
-import java.util.StringTokenizer;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.BagFactory;
-import org.apache.pig.data.DataBag;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.data.TupleFactory;
-public class TOKENIZE extends EvalFunc (DataBag) {
-    TupleFactory mTupleFactory = TupleFactory.getInstance();
-    BagFactory mBagFactory = BagFactory.getInstance();
-    public DataBag exec(Tuple input) throws IOException
-        try {
-            DataBag output = mBagFactory.newDefaultBag();
-            Object o = input.get(0);
-            if (!(o instanceof String)) {
-                throw new IOException("Expected input to be chararray, but  got " + o.getClass().getName());
-            }
-            StringTokenizer tok = new StringTokenizer((String)o, " \",()*", false);
-            while (tok.hasMoreTokens()) output.add(mTupleFactory.newTuple(tok.nextToken()));
-            return output;
-        } catch (ExecException ee) {
-            // error handling goes here
-        }
-    }
-}
-</pre>
-<a name="N10254"></a><a name="Schema"></a>
-<h3 class="h4"> Schema</h3>
-<p>The latest version of Pig uses type information for validation and performance. It is important for UDFs to participate in type propagation. Until now, our UDFs made no effort to communicate their output schema to Pig. This is because, most of the time, Pig can figure out this information by using Java's <a href="http://java.sun.com/developer/technicalArticles/ALT/Reflection/"> Reflection</a>. If your UDF returns a scalar or a map, no work is required. However, if your UDF returns a <span class="codefrag">tuple</span> or a <span class="codefrag">bag</span> (of tuples), it needs to help Pig figure out the structure of the tuple. </p>
-<p>If a UDF returns a <span class="codefrag">tuple</span> or a <span class="codefrag">bag</span> and schema information is not provided, Pig assumes that the tuple contains a single field of type <span class="codefrag">bytearray</span>. If this is not the case, then not specifying the schema can cause failures. We look at this next. </p>
-<p>Let's assume that we have UDF <span class="codefrag">Swap</span> that, given a tuple with two fields, swaps their order. Let's assume that the UDF does not specify a schema and look at the scripts below: </p>
-<pre class="code">
-register myudfs.jar;
-A = load 'student_data' as (name: chararray, age: int, gpa: float);
-B = foreach A generate flatten(myudfs.Swap(name, age)), gpa;
-C = foreach B generate $2;
-D = limit B 20;
-dump D;
-</pre>
-<p>This script will result in the following error cause by line 4. </p>
-<pre class="code">
-java.io.IOException: Out of bound access. Trying to access non-existent column: 2. Schema {bytearray,gpa: float} has 2 column(s).
-</pre>
-<p>This is because Pig is only aware of two columns in B while line 4 is requesting the third column of the tuple. (Column indexing in Pig starts with 0.) </p>
-<p>The function, including the schema, looks like this: </p>
-<pre class="code">
-package myudfs;
-import java.io.IOException;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.data.TupleFactory;
-import org.apache.pig.impl.logicalLayer.schema.Schema;
-import org.apache.pig.data.DataType;
-public class Swap extends EvalFunc (Tuple) {
-    public Tuple exec(Tuple input) throws IOException {
-        if (input == null || input.size()   2
-            return null;
-        try{
-            Tuple output = TupleFactory.getInstance().newTuple(2);
-            output.set(0, input.get(1));
-            output.set(1, input.get(0));
-            return output;
-        } catch(Exception e){
-            System.err.println("Failed to process input; error - " + e.getMessage());
-            return null;
-        }
-    }
-    public Schema outputSchema(Schema input) {
-        try{
-            Schema tupleSchema = new Schema();
-            tupleSchema.add(input.getField(1));
-            tupleSchema.add(input.getField(0));
-            return new Schema(new Schema.FieldSchema(getSchemaName(this.getClass().getName().toLowerCase(), input),tupleSchema, DataType.TUPLE));
-        }catch (Exception e){
-                return null;
-        }
-    }
-}
-</pre>
-<p>The function creates a schema with a single field (of type <span class="codefrag">FieldSchema=)&nbsp;of&nbsp;type&nbsp;=tuple</span>. The name of the field is constructed using the <span class="codefrag">getSchemaName</span> function of the <span class="codefrag">EvalFunc</span> class. The name consists of the name of the UDF function, the first parameter passed to it, and a sequence number to guarantee uniqueness. In the previous script, if you replace <span class="codefrag">dump&nbsp;D;</span>  with <span class="codefrag">describe&nbsp;B;</span> , you will see the following output:
-</p>
-<pre class="code">
-B: {myudfs.swap_age_3::age: int,myudfs.swap_age_3::name: chararray,gpa: float}
-</pre>
-<p>The second parameter to the <span class="codefrag">FieldSchema</span> constructor is the schema representing this field, which in this case is a tuple with two fields. The third parameter represents the type of the schema, which in this case is a <span class="codefrag">TUPLE</span>. All supported schema types are defined in the <span class="codefrag">org.apache.pig.data.DataType</span> class. </p>
-<pre class="code">
-public class DataType {
-    public static final byte UNKNOWN   =   0;
-    public static final byte NULL      =   1;
-    public static final byte BOOLEAN   =   5; // internal use only
-    public static final byte BYTE      =   6; // internal use only
-    public static final byte INTEGER   =  10;
-    public static final byte LONG      =  15;
-    public static final byte FLOAT     =  20;
-    public static final byte DOUBLE    =  25;
-    public static final byte BYTEARRAY =  50;
-    public static final byte CHARARRAY =  55;
-    public static final byte MAP       = 100;
-    public static final byte TUPLE     = 110;
-    public static final byte BAG       = 120;
-    public static final byte ERROR     =  -1;
-    // more code here
-}
-</pre>
-<p>You need to import the <span class="codefrag">org.apache.pig.data.DataType</span> class into your code to define schemas. You also need to import the schema class <span class="codefrag">org.apache.pig.impl.logicalLayer.schema.Schema</span>. </p>
-<p>The example above shows how to create an output schema for a tuple. Doing this for a bag is very similar. Let's extend the <span class="codefrag">TOKENIZE</span> function to do that: </p>
-<pre class="code">
-package org.apache.pig.builtin;
-import java.io.IOException;
-import java.util.StringTokenizer;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.BagFactory;
-import org.apache.pig.data.DataBag;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.data.TupleFactory;
-import org.apache.pig.impl.logicalLayer.schema.Schema;
-import org.apache.pig.data.DataType;
-public class TOKENIZE extends EvalFunc (DataBag) {
-    TupleFactory mTupleFactory = TupleFactory.getInstance();
-    BagFactory mBagFactory = BagFactory.getInstance();
-    public DataBag exec(Tuple input) throws IOException {
-        try {
-            DataBag output = mBagFactory.newDefaultBag();
-            Object o = input.get(0);
-            if (!(o instanceof String)) {
-                throw new IOException("Expected input to be chararray, but  got " + o.getClass().getName());
-            }
-            StringTokenizer tok = new StringTokenizer((String)o, " \",()*", false);
-            while (tok.hasMoreTokens()) output.add(mTupleFactory.newTuple(tok.nextToken()));
-            return output;
-        } catch (ExecException ee) {
-            // error handling goes here
-        }
-    }
-    public Schema outputSchema(Schema input) {
-         try{
-             Schema bagSchema = new Schema();
-             bagSchema.add(new Schema.FieldSchema("token", DataType.CHARARRAY));
-             return new Schema(new Schema.FieldSchema(getSchemaName(this.getClass().getName().toLowerCase(), input),
-                                                    bagSchema, DataType.BAG));
-         }catch (Exception e){
-            return null;
-         }
-    }
-}
-</pre>
-<p>As you can see, this is very similar to the output schema definition in the <span class="codefrag">Swap</span> function. One difference is that instead of reusing input schema, we create a brand new field schema to represent the tokens stored in the bag. The other difference is that the type of the schema created is <span class="codefrag">BAG</span> (not =TUPLE=). </p>
-<a name="N102D1"></a><a name="Error+Handling"></a>
-<h3 class="h4"> Error Handling</h3>
-<p>There are several types of errors that can occur in a UDF: </p>
-<ol>
-<li>
-<p>An error that affects a particular row but is not likely to impact other rows. An example of such an error would be a malformed input value or divide by zero problem. A reasonable handling of this situation would be to emit a warning and return a null value. <span class="codefrag">ABS</span> function in the next section demonstrates this approach. The current approach is to write the warning to <span class="codefrag">stderr</span>. Eventually we would like to pass a logger to the UDFs. Note that returning a NULL value only makes sense if the malformed value is of type <span class="codefrag">bytearray</span>. Otherwise the proper type has been already created and should have an appropriate value. If this is not the case, it is an internal error and should cause the system to fail. Both cases can be seen in the implementation of the <span class="codefrag">ABS</span> function in the next section. </p>
-</li>
-<li>
-<p>An error that affects the entire processing but can succeed on retry. An example of such a failure is the inability to open a lookup file because the file could not be found. This could be a temporary environmental issue that can go away on retry. A UDF can signal this to Pig by throwing an <span class="codefrag">IOException</span> as with the case of the <span class="codefrag">ABS</span> function below. </p>
-</li>
-<li>
-<p>An error that affects the entire processing and is not likely to succeed on retry. An example of such a failure is the inability to open a lookup file because of file permission problems. Pig currently does not have a way to handle this case. Hadoop does not have a way to handle this case either. It will be handled the same way as 2 above. </p>
-</li>
-</ol>
-<p>Pig provides a helper class <span class="codefrag">WrappedIOException</span>. The intent here is to allow you to convert any exception into <span class="codefrag">IOException</span>. Its usage can be seen in the <span class="codefrag">UPPER</span> function in our first example. </p>
-<a name="N1030B"></a><a name="Function+Overloading"></a>
-<h3 class="h4">Function Overloading</h3>
-<p>Before the type system was available in Pig, all values for the purpose of arithmetic calculations were assumed to be doubles as the safest choice. However, this is not very efficient if the data is actually of type integer or long. (We saw about a 2x slowdown of a query when using double where integer could be used.) Now that Pig supports types we can take advantage of the type information and choose the function that is most efficient for the provided operands. </p>
-<p>UDF writers are encouraged to provide type-specific versions of a function if this can result in better performance. On the other hand, we don't want the users of the functions to worry about different functions - the right thing should just happen. Pig allows for this via a function table mechanism as shown in the next example. </p>
-<p>This example shows the implementation of the <span class="codefrag">ABS</span> function that returns the absolute value of a numeric value passed to it as input. </p>
-<pre class="code">
-import java.io.IOException;
-import java.util.List;
-import java.util.ArrayList;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.FuncSpec;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.impl.logicalLayer.FrontendException;
-import org.apache.pig.impl.util.WrappedIOException;
-import org.apache.pig.impl.logicalLayer.schema.Schema;
-import org.apache.pig.data.DataType;
-public class ABS extends EvalFunc (Double) {
-    public Double exec(Tuple input) throws IOException {
-        if (input == null || input.size() == 0)
-            return null;
-        Double d;
-        try{
-            d = DataType.toDouble(input.get(0));
-        } catch (NumberFormatException nfe){
-            System.err.println("Failed to process input; error - " + nfe.getMessage());
-            return null;
-        } catch (Exception e){
-            throw WrappedIOException.wrap("Caught exception processing input row ", e);
-        }
-        return Math.abs(d);
-    }
-    public List (FuncSpec) getArgToFuncMapping() throws FrontendException {
-        List (FuncSpec) funcList = new ArrayList (FuncSpec) ();
-        funcList.add(new FuncSpec(this.getClass().getName(), new Schema(new Schema.FieldSchema(null, DataType.BYTEARRAY))));
-        funcList.add(new FuncSpec(DoubleAbs.class.getName(),  new Schema(new Schema.FieldSchema(null, DataType.DOUBLE))));
-        funcList.add(new FuncSpec(FloatAbs.class.getName(),   new Schema(new Schema.FieldSchema(null, DataType.FLOAT))));
-        funcList.add(new FuncSpec(IntAbs.class.getName(),  new Schema(new Schema.FieldSchema(null, DataType.INTEGER))));
-        funcList.add(new FuncSpec(LongAbs.class.getName(),  new Schema(new Schema.FieldSchema(null, DataType.LONG))));
-        return funcList;
-    }
-}
-</pre>
-<p>The main thing to notice in this example is the <span class="codefrag">getArgToFuncMapping()</span> method. This method returns a list that contains a mapping from the input schema to the class that should be used to handle it. In this example the main class handles the <span class="codefrag">bytearray</span> input and outsources the rest of the work to other classes implemented in separate files in the same package. The example of one such class is below. This class handles integer input values. </p>
-<pre class="code">
-import java.io.IOException;
-import org.apache.pig.impl.util.WrappedIOException;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.Tuple;
-public class IntAbs extends EvalFunc (Integer) {
-    public Integer exec(Tuple input) throws IOException {
-        if (input == null || input.size() == 0)
-            return null;
-        Integer d;
-        try{
-            d = (Integer)input.get(0);
-        } catch (Exception e){
-            throw WrappedIOException.wrap("Caught exception processing input row ", e);
-        }
-        return Math.abs(d);
-    }
-}
-</pre>
-<p>A note on error handling. The <span class="codefrag">ABS</span> class covers the case of the <span class="codefrag">bytearray</span> which means the data has not been converted yet to its actual type. This is why a null value is returned when <span class="codefrag">NumberFormatException</span> is encountered. However, the <span class="codefrag">IntAbs</span> function is only called if the data is already of type <span class="codefrag">Integer</span> which means it has already been converted to the real type and bad format has been dealt with. This is why an exception is thrown if the input can't be cast to <span class="codefrag">Integer</span>. </p>
-<p>The example above covers a reasonably simple case where the UDF only takes one parameter and there is a separate function for each parameter type. However, this will not always be the case. If Pig can't find an <span class="codefrag">exact&nbsp;match</span> it tries to do a <span class="codefrag">best&nbsp;match</span>. The rule for the best match is to find the most efficient function that can be used safely. This means that Pig must find the function that, for each input parameter, provides the smallest type that is equal to or greater than the input type. The type progression rules are: <span class="codefrag">int=-&gt;=long=-&gt;=float=-&gt;=double</span>. </p>
-<p>For instance, let's consider function <span class="codefrag">MAX</span> which is part of the <span class="codefrag">piggybank</span> described later in this document. Given two values, the function returns the larger value. The function table for <span class="codefrag">MAX</span> looks like this: </p>
-<pre class="code">
-public List (FuncSpec) getArgToFuncMapping() throws FrontendException {
-    List (FuncSpec) funcList = new ArrayList (FuncSpec) ();
-    Util.addToFunctionList(funcList, IntMax.class.getName(), DataType.INTEGER);
-    Util.addToFunctionList(funcList, DoubleMax.class.getName(), DataType.DOUBLE);
-    Util.addToFunctionList(funcList, FloatMax.class.getName(), DataType.FLOAT);
-    Util.addToFunctionList(funcList, LongMax.class.getName(), DataType.LONG);
-    return funcList;
-}
-</pre>
-<p>The <span class="codefrag">Util.addToFunctionList</span> function is a helper function that adds an entry to the list as the first argument, with the key of the class name passed as the second argument, and the schema containing two fields of the same type as the third argument.  </p>
-<p>Let's now see how this function can be used in a Pig script: </p>
-<pre class="code">
-REGISTER piggybank.jar
-A = LOAD 'student_data' AS (name: chararray, gpa1: float, gpa2: double);
-B = FOREACH A GENERATE name, org.apache.pig.piggybank.evaluation.math.MAX(gpa1, gpa2);
-DUMP B;
-</pre>
-<p>In this example, the function gets one parameter of type <span class="codefrag">float</span> and another of type <span class="codefrag">double</span>. The best fit will be the function that takes two double values. Pig makes this choice on the user's behalf by inserting implicit casts for the parameters. Running the script above is equivalent to running the script below: </p>
-<pre class="code">
-A = LOAD 'student_data' AS (name: chararray, gpa1: float, gpa2: double);
-B = FOREACH A GENERATE name, org.apache.pig.piggybank.evaluation.math.MAX((double)gpa1, gpa2);
-DUMP B;
-</pre>
-<p>A special case of the <span class="codefrag">best&nbsp;fit</span> approach is handling data without a schema specified. The type for this data is interpreted as <span class="codefrag">bytearray</span>. Since the type of the data is not known, there is no way to choose a best fit version. The only time a cast is performed is when the function table contains only a single entry. This works well to maintain backward compatibility.  </p>
-<p>Let's revisit the <span class="codefrag">UPPER</span> function from our first example. As it is written now, it would only work if the data passed to it is of type <span class="codefrag">chararray</span>. To make it work with data whose type is not explicitly set, a function table with a single entry needs to be added: </p>
-<pre class="code">
-package myudfs;
-import java.io.IOException;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.Tuple;
-public class UPPER extends EvalFunc (String)
-{
-    public String exec(Tuple input) throws IOException {
-        if (input == null || input.size() == 0)
-            return null;
-        try{
-            String str = (String)input.get(0);
-            return str.toUpperCase();
-        }catch(Exception e){
-            System.err.println("WARN: UPPER: failed to process input; error - " + e.getMessage());
-            return null;
-        }
-    }
-    public List (FuncSpec) getArgToFuncMapping() throws FrontendException {
-        List (FuncSpec) funcList = new ArrayList (FuncSpec) ();
-        funcList.add(new FuncSpec(this.getClass().getName(), new Schema(new Schema.FieldSchema(null, DataType.CHARARRAY))));
-        return funcList;
-    }
-}
-</pre>
-<p>Now the following script will ran: </p>
-<pre class="code">
--- this is myscript.pig
-REGISTER myudfs.jar;
-A = LOAD 'student_data' AS (name, age, gpa);
-B = FOREACH A GENERATE myudfs.UPPER(name);
-DUMP B;
-</pre>
-<a name="N10397"></a><a name="Reporting+Progress"></a>
-<h3 class="h4">Reporting Progress</h3>
-<p>A challenge of running a large shared system is to make sure system resources are used efficiently. One aspect of this challenge is detecting runaway processes that are no longer making progress. Pig uses a heartbeat mechanism for this purpose. If any of the tasks stops sending a heartbeat, the system assumes that it is dead and kills it.   </p>
-<p>Most of the time, single-tuple processing within a UDF is very short and does not require a UDF to heartbeat. The same is true for aggregate functions that operate on large bags because bag iteration code takes care of it. However, if you have a function that performs a complex computation that can take an order of minutes to execute, you should add a progress indicator to your code. This is very easy to accomplish. The <span class="codefrag">EvalFunc</span> function provides a <span class="codefrag">progress</span> function that you need to call in your <span class="codefrag">exec</span> method. </p>
-<p>For instance, the <span class="codefrag">UPPER</span> function would now look as follows: </p>
-<pre class="code">
-public class UPPER extends EvalFunc (String)
-{
-        public String exec(Tuple input) throws IOException {
-                if (input == null || input.size() == 0)
-                return null;
-                try{
-                        reporter.progress();
-                        String str = (String)input.get(0);
-                        return str.toUpperCase();
-                }catch(Exception e){
-                    throw WrappedIOException.wrap("Caught exception processing input row ", e);
-                }
-        }
-}
-</pre>
-<a name="N103B7"></a><a name="Import+Lists"></a>
-<h3 class="h4">Import Lists</h3>
-<p>An import list allows you to specify the package to which a UDF or a group of UDFs belong,
- eliminating the need to qualify the UDF on every call. An import list can be specified via the udf.import.list Java
- property on the Pig command line: </p>
-<pre class="code">
-pig -Dudf.import.list=com.yahoo.yst.sds.ULT
-</pre>
-<p>You can supply multiple locations as well: </p>
-<pre class="code">
-pig -Dudf.import.list=com.yahoo.yst.sds.ULT:org.apache.pig.piggybank.evaluation
-</pre>
-<p>To make use of import scripts, do the following:</p>
-<pre class="code">
-myscript.pig:
-A = load '/data/SDS/data/searcg_US/20090820' using ULTLoader as (s, m, l);
-....
-command:
-pig -cp sds.jar -Dudf.import.list=com.yahoo.yst.sds.ULT myscript.pig
-</pre>
-</div>
-<!-- BEGIN LOAD/STORE FUNCTIONS -->
-<a name="N103D6"></a><a name="Load%2FStore+Functions"></a>
-<h2 class="h3"> Load/Store Functions</h2>
-<div class="section">
-<p>The load/store user-defined functions control how data goes into Pig and comes out of Pig. Often, the same function handles both input and output but that does not have to be the case. </p>
-<p>
-With Pig 0.7.0, the Pig load/store API moves closer to using Hadoop's InputFormat and OutputFormat classes.
-This enables Pig users/developers to create new LoadFunc and StoreFunc implementation based on existing Hadoop InputFormat and OutputFormat classes with minimal code. The complexity of reading the data and creating a record will now lie in the InputFormat and likewise on the writing end, the complexity of writing will lie in the OutputFormat. This enables Pig to easily read/write data in new storage formats as and when an Hadoop InputFormat and OutputFormat is available for them. </p>
-<p>
-<strong>Note:</strong> Both the LoadFunc and StoreFunc implementations should use the Hadoop 20 API based classes (InputFormat/OutputFormat and related classes) under the <strong>new</strong> org.apache.hadoop.mapreduce package instead of the old org.apache.hadoop.mapred package.
-</p>
-<a name="N103EB"></a><a name="Load+Functions"></a>
-<h3 class="h4"> Load Functions</h3>
-<p>
-<a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup">LoadFunc</a>
-abstract class has the main methods for loading data and for most use cases it would suffice to extend it. There are three other optional interfaces which can be implemented to achieve extended functionality: </p>
-<ul>
-<li>
-<a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/LoadMetadata.java?view=markup">LoadMetadata</a>
-has methods to deal with metadata - most implementation of loaders don't need to implement this unless they interact with some metadata system. The getSchema() method in this interface provides a way for loader implementations to communicate the schema of the data back to pig. If a loader implementation returns data comprised of fields of real types (rather than DataByteArray fields), it should provide the schema describing the data returned through the getSchema() method. The other methods are concerned with other types of metadata like partition keys and statistics. Implementations can return null return values for these methods if they are not applicable for that implementation.</li>
-<li>
-<a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/LoadPushDown.java?view=markup">LoadPushDown</a>
-has methods to push operations from pig runtime into loader implementations - currently only projections .i.e the pushProjection() method is called by Pig to communicate to the loader what exact fields are required in the pig script. The loader implementation can choose to honor the request or respond that it will not honor the request and return all fields in the data. If a loader implementation is able to efficiently return only required fields, it should implement LoadPushDown to improve query performance. (Irrespective of whether the implementation can or cannot return only the required fields, if the implementation also implements getSchema(), the schema returned in getSchema() should be for the entire tuple of data.) </li>
-<li>
-<a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/LoadCaster.java?view=markup">LoadCaster</a>
-has methods to convert byte arrays to specific types. A loader implementation should implement this if casts (implicit or explicit) from DataByteArray fields to other types need to be supported. </li>
-</ul>
-<p>The LoadFunc abstract class is the main class to extend for implementing a loader. The methods which need to be overriden are explained below:</p>
-<ul>
-<li>getInputFormat() :This method is called by Pig to get the InputFormat used by the loader. The methods in the InputFormat (and underlying RecordReader) are called by Pig in the same manner (and in the same context) as by Hadoop in a MapReduce java program. If the InputFormat is a Hadoop packaged one, the implementation should use the new API based one under org.apache.hadoop.mapreduce. If it is a custom InputFormat, it should be implemented using the new API in org.apache.hadoop.mapreduce.<br>
-<br>
- If a custom loader using a text-based InputFormat or a file-based InputFormat would like to read files in all subdirectories under a given input directory recursively, then it should use the PigTextInputFormat and PigFileInputFormat classes provided in org.apache.pig.backend.hadoop.executionengine.mapReduceLayer. The Pig InputFormat classes work around a current limitation in the Hadoop TextInputFormat and FileInputFormat classes which only read one level down from the provided input directory. For example, if the input in the load statement is 'dir1' and there are subdirs 'dir2' and 'dir2/dir3' beneath dir1, the Hadoop TextInputFormat and FileInputFormat classes read the files under 'dir1' only. Using PigTextInputFormat or PigFileInputFormat (or by extending them), the files in all the directories can be read. </li>
-<li>setLocation() :This method is called by Pig to communicate the load location to the loader. The loader should use this method to communicate the same information to the underlying InputFormat. This method is called multiple times by pig - implementations should bear this in mind and should ensure there are no inconsistent side effects due to the multiple calls. </li>
-<li>prepareToRead() : Through this method the RecordReader associated with the InputFormat provided by the LoadFunc is passed to the LoadFunc. The RecordReader can then be used by the implementation in getNext() to return a tuple representing a record of data back to pig. </li>
-<li>getNext() :The meaning of getNext() has not changed and is called by Pig runtime to get the next tuple in the data - in this method the implementation should use the the underlying RecordReader and construct the tuple to return. </li>
-</ul>
-<p>The following methods have default implementations in LoadFunc and should be overridden only if needed: </p>
-<ul>
-<li>setUdfContextSignature():This method will be called by Pig both in the front end and back end to pass a unique signature to the Loader. The signature can be used to store into the UDFContext any information which the Loader needs to store between various method invocations in the front end and back end. A use case is to store RequiredFieldList passed to it in LoadPushDown.pushProjection(RequiredFieldList) for use in the back end before returning tuples in getNext(). The default implementation in LoadFunc has an empty body. This method will be called before other methods. </li>
-<li>relativeToAbsolutePath():Pig runtime will call this method to allow the Loader to convert a relative load location to an absolute location. The default implementation provided in LoadFunc handles this for FileSystem locations. If the load source is something else, loader implementation may choose to override this.</li>
-</ul>
-<p>
-<strong>Example Implementation</strong>
-</p>
-<p>
-The loader implementation in the example is a loader for text data with line delimiter as '\n' and '\t' as default field delimiter (which can be overridden by passing a different field delimiter in the constructor) - this is similar to current PigStorage loader in Pig. The implementation uses an existing Hadoop supported Inputformat - TextInputFormat - as the underlying InputFormat.
-</p>
-<pre class="code">
-public class SimpleTextLoader extends LoadFunc {
-    protected RecordReader in = null;
-    private byte fieldDel = '\t';
-    private ArrayList&lt;Object&gt; mProtoTuple = null;
-    private TupleFactory mTupleFactory = TupleFactory.getInstance();
-    private static final int BUFFER_SIZE = 1024;
-    public SimpleTextLoader() {
-    }
-    /**
-     * Constructs a Pig loader that uses specified character as a field delimiter.
-     *
-     * @param delimiter
-     *            the single byte character that is used to separate fields.
-     *            ("\t" is the default.)
-     */
-    public SimpleTextLoader(String delimiter) {
-        this();
-        if (delimiter.length() == 1) {
-            this.fieldDel = (byte)delimiter.charAt(0);
-        } else if (delimiter.length() &gt;  1 &amp; &amp; delimiter.charAt(0) == '\\') {
-            switch (delimiter.charAt(1)) {
-            case 't':
-                this.fieldDel = (byte)'\t';
-                break;
-            case 'x':
-               fieldDel =
-                    Integer.valueOf(delimiter.substring(2), 16).byteValue();
-               break;
-            case 'u':
-                this.fieldDel =
-                    Integer.valueOf(delimiter.substring(2)).byteValue();
-                break;
-            default:
-                throw new RuntimeException("Unknown delimiter " + delimiter);
-            }
-        } else {
-            throw new RuntimeException("PigStorage delimeter must be a single character");
-        }
-    }
-    @Override
-    public Tuple getNext() throws IOException {
-        try {
-            boolean notDone = in.nextKeyValue();
-            if (!notDone) {
-                return null;
-            }
-            Text value = (Text) in.getCurrentValue();
-            byte[] buf = value.getBytes();
-            int len = value.getLength();
-            int start = 0;
-            for (int i = 0; i &lt; len; i++) {
-                if (buf[i] == fieldDel) {
-                    readField(buf, start, i);
-                    start = i + 1;
-                }
-            }
-            // pick up the last field
-            readField(buf, start, len);
-            Tuple t =  mTupleFactory.newTupleNoCopy(mProtoTuple);
-            mProtoTuple = null;
-            return t;
-        } catch (InterruptedException e) {
-            int errCode = 6018;
-            String errMsg = "Error while reading input";
-            throw new ExecException(errMsg, errCode,
-                    PigException.REMOTE_ENVIRONMENT, e);
-        }
-    }
-    private void readField(byte[] buf, int start, int end) {
-        if (mProtoTuple == null) {
-            mProtoTuple = new ArrayList&lt;Object&gt;();
-        }
-        if (start == end) {
-            // NULL value
-            mProtoTuple.add(null);
-        } else {
-            mProtoTuple.add(new DataByteArray(buf, start, end));
-        }
-    }
-    @Override
-    public InputFormat getInputFormat() {
-        return new TextInputFormat();
-    }
-    @Override
-    public void prepareToRead(RecordReader reader, PigSplit split) {
-        in = reader;
-    }
-    @Override
-    public void setLocation(String location, Job job)
-            throws IOException {
-        FileInputFormat.setInputPaths(job, location);
-    }
-}
-</pre>
-<a name="N1043C"></a><a name="Store+Functions"></a>
-<h3 class="h4"> Store Functions</h3>
-<p>
-<a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/StoreFunc.java?view=markup">StoreFunc</a>
-abstract class has the main methods for storing data and for most use cases it should suffice to extend it. There is an optional interface which can be implemented to achieve extended functionality: </p>
-<ul>
-<li>
-<a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/src/org/apache/pig/StoreMetadata.java?view=markup">StoreMetadata:</a>
-This interface has methods to interact with metadata systems to store schema and store statistics. This interface is truely optional and should only be implemented if metadata needs to stored. </li>
-</ul>
-<p>The methods which need to be overridden in StoreFunc are explained below: </p>
-<ul>
-<li>getOutputFormat(): This method will be called by Pig to get the OutputFormat used by the storer. The methods in the OutputFormat (and underlying RecordWriter and OutputCommitter) will be called by pig in the same manner (and in the same context) as by Hadoop in a map-reduce java program. If the OutputFormat is a hadoop packaged one, the implementation should use the new API based one under org.apache.hadoop.mapreduce. If it is a custom OutputFormat, it should be implemented using the new API under org.apache.hadoop.mapreduce. The checkOutputSpecs() method of the OutputFormat will be called by pig to check the output location up-front. This method will also be called as part of the Hadoop call sequence when the job is launched. So implementations should ensure that this method can be called multiple times without inconsistent side effects. </li>
-<li>setStoreLocation(): This method is called by Pig to communicate the store location to the storer. The storer should use this method to communicate the same information to the underlying OutputFormat. This method is called multiple times by pig - implementations should bear in mind that this method is called multiple times and should ensure there are no inconsistent side effects due to the multiple calls. </li>
-<li>prepareToWrite(): In the new API, writing of the data is through the OutputFormat provided by the StoreFunc. In prepareToWrite() the RecordWriter associated with the OutputFormat provided by the StoreFunc is passed to the StoreFunc. The RecordWriter can then be used by the implementation in putNext() to write a tuple representing a record of data in a manner expected by the RecordWriter. </li>
-<li>putNext(): The meaning of putNext() has not changed and is called by Pig runtime to write the next tuple of data - in the new API, this is the method wherein the implementation will use the the underlying RecordWriter to write the Tuple out.</li>
-</ul>
-<p>The following methods have default implementations in StoreFunc and should be overridden only if necessary: </p>
-<ul>
-<li>setStoreFunc!UDFContextSignature(): This method will be called by Pig both in the front end and back end to pass a unique signature to the Storer. The signature can be used to store into the UDFContext any information which the Storer needs to store between various method invocations in the front end and back end. The default implementation in StoreFunc has an empty body. This method will be called before other methods.
-</li>
-<li>relToAbsPathForStoreLocation(): Pig runtime will call this method to allow the Storer to convert a relative store location to an absolute location. An implementation is provided in StoreFunc which handles this for FileSystem based locations. </li>
-<li>checkSchema(): A Store function should implement this function to check that a given schema describing the data to be written is acceptable to it. The default implementation in StoreFunc has an empty body. This method will be called before any calls to setStoreLocation(). </li>
-</ul>
-<p>
-<strong>Example Implementation</strong>
-</p>
-<p>
-The storer implementation in the example is a storer for text data with line delimiter as '\n' and '\t' as default field delimiter (which can be overridden by passing a different field delimiter in the constructor) - this is similar to current PigStorage storer in Pig. The implementation uses an existing Hadoop supported OutputFormat - TextOutputFormat as the underlying OutputFormat.
-</p>
-<pre class="code">
-public class SimpleTextStorer extends StoreFunc {
-    protected RecordWriter writer = null;
-    private byte fieldDel = '\t';
-    private static final int BUFFER_SIZE = 1024;
-    private static final String UTF8 = "UTF-8";
-    public PigStorage() {
-    }
-    public PigStorage(String delimiter) {
-        this();
-        if (delimiter.length() == 1) {
-            this.fieldDel = (byte)delimiter.charAt(0);
-        } else if (delimiter.length() &gt; 1delimiter.charAt(0) == '\\') {
-            switch (delimiter.charAt(1)) {
-            case 't':
-                this.fieldDel = (byte)'\t';
-                break;
-            case 'x':
-               fieldDel =
-                    Integer.valueOf(delimiter.substring(2), 16).byteValue();
-               break;
-            case 'u':
-                this.fieldDel =
-                    Integer.valueOf(delimiter.substring(2)).byteValue();
-                break;
-            default:
-                throw new RuntimeException("Unknown delimiter " + delimiter);
-            }
-        } else {
-            throw new RuntimeException("PigStorage delimeter must be a single character");
-        }
-    }
-    ByteArrayOutputStream mOut = new ByteArrayOutputStream(BUFFER_SIZE);
-    @Override
-    public void putNext(Tuple f) throws IOException {
-        int sz = f.size();
-        for (int i = 0; i &lt; sz; i++) {
-            Object field;
-            try {
-                field = f.get(i);
-            } catch (ExecException ee) {
-                throw ee;
-            }
-            putField(field);
-            if (i != sz - 1) {
-                mOut.write(fieldDel);
-            }
-        }
-        Text text = new Text(mOut.toByteArray());
-        try {
-            writer.write(null, text);
-            mOut.reset();
-        } catch (InterruptedException e) {
-            throw new IOException(e);
-        }
-    }
-    @SuppressWarnings("unchecked")
-    private void putField(Object field) throws IOException {
-        //string constants for each delimiter
-        String tupleBeginDelim = "(";
-        String tupleEndDelim = ")";
-        String bagBeginDelim = "{";
-        String bagEndDelim = "}";
-        String mapBeginDelim = "[";
-        String mapEndDelim = "]";
-        String fieldDelim = ",";
-        String mapKeyValueDelim = "#";
-        switch (DataType.findType(field)) {
-        case DataType.NULL:
-            break; // just leave it empty
-        case DataType.BOOLEAN:
-            mOut.write(((Boolean)field).toString().getBytes());
-            break;
-        case DataType.INTEGER:
-            mOut.write(((Integer)field).toString().getBytes());
-            break;
-        case DataType.LONG:
-            mOut.write(((Long)field).toString().getBytes());
-            break;
-        case DataType.FLOAT:
-            mOut.write(((Float)field).toString().getBytes());
-            break;
-        case DataType.DOUBLE:
-            mOut.write(((Double)field).toString().getBytes());
-            break;
-        case DataType.BYTEARRAY: {
-            byte[] b = ((DataByteArray)field).get();
-            mOut.write(b, 0, b.length);
-            break;
-                                 }
-        case DataType.CHARARRAY:
-            // oddly enough, writeBytes writes a string
-            mOut.write(((String)field).getBytes(UTF8));
-            break;
-        case DataType.MAP:
-            boolean mapHasNext = false;
-            Map&lt;String, Object&gt; m = (Map&lt;String, Object&gt;)field;
-            mOut.write(mapBeginDelim.getBytes(UTF8));
-            for(Map.Entry&lt;String, Object&gt; e: m.entrySet()) {
-                if(mapHasNext) {
-                    mOut.write(fieldDelim.getBytes(UTF8));
-                } else {
-                    mapHasNext = true;
-                }
-                putField(e.getKey());
-                mOut.write(mapKeyValueDelim.getBytes(UTF8));
-                putField(e.getValue());
-            }
-            mOut.write(mapEndDelim.getBytes(UTF8));
-            break;
-        case DataType.TUPLE:
-            boolean tupleHasNext = false;
-            Tuple t = (Tuple)field;
-            mOut.write(tupleBeginDelim.getBytes(UTF8));
-            for(int i = 0; i &lt; t.size(); ++i) {
-                if(tupleHasNext) {
-                    mOut.write(fieldDelim.getBytes(UTF8));
-                } else {
-                    tupleHasNext = true;
-                }
-                try {
-                    putField(t.get(i));
-                } catch (ExecException ee) {
-                    throw ee;
-                }
-            }
-            mOut.write(tupleEndDelim.getBytes(UTF8));
-            break;
-        case DataType.BAG:
-            boolean bagHasNext = false;
-            mOut.write(bagBeginDelim.getBytes(UTF8));
-            Iterator&lt;Tuple&gt; tupleIter = ((DataBag)field).iterator();
-            while(tupleIter.hasNext()) {
-                if(bagHasNext) {
-                    mOut.write(fieldDelim.getBytes(UTF8));
-                } else {
-                    bagHasNext = true;
-                }
-                putField((Object)tupleIter.next());
-            }
-            mOut.write(bagEndDelim.getBytes(UTF8));
-            break;
-        default: {
-            int errCode = 2108;
-            String msg = "Could not determine data type of field: " + field;
-            throw new ExecException(msg, errCode, PigException.BUG);
-        }
-        }
-    }
-    @Override
-    public OutputFormat getOutputFormat() {
-        return new TextOutputFormat&lt;WritableComparable, Text&gt;();
-    }
-    @Override
-    public void prepareToWrite(RecordWriter writer) {
-        this.writer = writer;
-    }
-    @Override
-    public void setStoreLocation(String location, Job job) throws IOException {
-        job.getConfiguration().set("mapred.textoutputformat.separator", "");
-        FileOutputFormat.setOutputPath(job, new Path(location));
-        if (location.endsWith(".bz2")) {
-            FileOutputFormat.setCompressOutput(job, true);
-            FileOutputFormat.setOutputCompressorClass(job,  BZip2Codec.class);
-        }  else if (location.endsWith(".gz")) {
-            FileOutputFormat.setCompressOutput(job, true);
-            FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);
-        }
-    }
-}
-</pre>
-</div>
-<!-- END LOAD/STORE FUNCTIONS -->
-<a name="N10483"></a><a name="Builtin+Functions+and+Function+Repositories"></a>
-<h2 class="h3">Builtin Functions and Function Repositories</h2>
-<div class="section">
-<p>Pig comes with a set of builtin functions. Two main properties differentiate builtin functions from UDFs. First, they don't need to be registered because Pig knows where they are. Second, they don't need to be qualified when used because Pig knows where to find them. </p>
-<p>Pig also hosts a UDF repository called <span class="codefrag">piggybank</span> that allows users to share UDFs that they have written. The details are described in <a href="http://wiki.apache.org/pig/PiggyBank"> PiggyBank</a>. </p>
-</div>
-<a name="N10497"></a><a name="Accumulator+Interface"></a>
-<h2 class="h3">Accumulator Interface</h2>
-<div class="section">
-<p>In Pig, problems with memory usage can occur when data, which results from a group or cogroup operation, needs to be placed in a bag  and passed in its entirety to a UDF.</p>
-<p>This problem is partially addressed by Algebraic UDFs that use the combiner and can deal with data being passed to them incrementally during different processing phases (map, combiner, and reduce.) However, there are a number of UDFs that are not Algebraic, don't use the combiner, but still don&rsquo;t need to be given all data at once. </p>
-<p>The new Accumulator interface is designed to decrease memory usage by targeting such UDFs. For the functions that implement this interface, Pig guarantees that the data for the same key is passed continuously but in small increments. To work with incremental data, here is the interface a UDF needs to implement:</p>
-<pre class="code">
-public interface Accumulator &lt;T&gt; {
-   /**
-    * Process tuples. Each DataBag may contain 0 to many tuples for current key
-    */
-    public void accumulate(Tuple b) throws IOException;
-    /**
-     * Called when all tuples from current key have been passed to the accumulator.
-     * @return the value for the UDF for this key.
-     */
-    public T getValue();
-    /**
-     * Called after getValue() to prepare processing for next key.
-     */
-    public void cleanup();
-}
-</pre>
-<p>There are several things to note here:</p>
-<ol>
-<li>Each UDF must extend the EvalFunc class and implement all necessary functions there.</li>
-<li>If a function is algebraic but can be used in a FOREACH statement with accumulator functions, it needs to implement the Accumulator interface in addition to the Algebraic interface.</li>
-<li>The interface is parameterized with the return type of the function.</li>
-<li>The accumulate function is guaranteed to be called one or more times, passing one or more tuples in a bag, to the UDF. (Note that the tuple that is passed to the accumulator has the same content as the one passed to exec &ndash; all the parameters passed to the UDF &ndash; one of which should be a bag).</li>
-<li>The getValue function is called after all the tuples for a particular key have been processed to retrieve the final value.</li>
-<li>The cleanup function is called after getValue but before the next value is processed.</li>
-</ol>
-<p>Here us a code snippet of the integer version of the MAX function that implements the interface:</p>
-<pre class="code">
-public class IntMax extends EvalFunc&lt;Integer&gt; implements Algebraic, Accumulator&lt;Integer&gt; {
-    &hellip;&hellip;.
-    /* Accumulator interface */
-    private Integer intermediateMax = null;
-    @Override
-    public void accumulate(Tuple b) throws IOException {
-        try {
-            Integer curMax = max(b);
-            if (curMax == null) {
-                return;
-            }
-            /* if bag is not null, initialize intermediateMax to negative infinity */
-            if (intermediateMax == null) {
-                intermediateMax = Integer.MIN_VALUE;
-            }
-            intermediateMax = java.lang.Math.max(intermediateMax, curMax);
-        } catch (ExecException ee) {
-            throw ee;
-        } catch (Exception e) {
-            int errCode = 2106;
-            String msg = "Error while computing max in " + this.getClass().getSimpleName();
-            throw new ExecException(msg, errCode, PigException.BUG, e);
-        }
-    }
-    @Override
-    public void cleanup() {
-        intermediateMax = null;
-    }
-    @Override
-    public Integer getValue() {
-        return intermediateMax;
-    }
-}
-</pre>
-</div>
-<a name="N104CA"></a><a name="Advanced+Topics"></a>
-<h2 class="h3">Advanced Topics</h2>
-<div class="section">
-<a name="N104D0"></a><a name="Function+Instantiation"></a>
-<h3 class="h4">Function Instantiation</h3>
-<p>One problem that users run into is when they make assumption about how many times a constructor for their UDF is called. For instance, they might be creating side files in the store function and doing it in the constructor seems like a good idea. The problem with this approach is that in most cases Pig instantiates functions on the client side to, for instance, examine the schema of the data.  </p>
-<p>Users should not make assumptions about how many times a function is instantiated; instead, they should make their code resilient to multiple instantiations. For instance, they could check if the files exist before creating them. </p>
-<a name="N104DD"></a><a name="Schemas"></a>
-<h3 class="h4">Schemas</h3>
-<p>One request from users is to have the ability to examine the input schema of the data before processing the data. For example, they would like to know how to convert an input tuple to a map such that the keys in the map are the names of the input columns. The current answer is that there is now way to do this. This is something we would like to support in the future. </p>
-<a name="N104E7"></a><a name="Passing+Configurations+to+UDFs"></a>
-<h3 class="h4">Passing Configurations to UDFs</h3>
-<p>The singleton UDFContext class provides two features to UDF writers. First, on the backend, it allows UDFs to get access to the JobConf object, by calling getJobConf. This is only available on the backend (at run time) as the JobConf has not yet been constructed on the front end (during planning time).</p>
-<p>Second, it allows UDFs to pass configuration information between instantiations of the UDF on the front and backends. UDFs can store information in a configuration object when they are constructed on the front end, or during other front end calls such as describeSchema. They can then read that information on the backend when exec (for EvalFunc) or getNext (for LoadFunc) is called. Note that information will not be passed between instantiations of the function on the backend. The communication channel only works from front end to back end.</p>
-<p>To store information, the UDF calls getUDFProperties. This returns a Properties object which the UDF can record the information in or read the information from. To avoid name space conflicts UDFs are required to provide a signature when obtaining a Properties object. This can be done in two ways. The UDF can provide its Class object (via this.getClass()). In this case, every instantiation of the UDF will be given the same Properties object. The UDF can also provide its Class plus an array of Strings. The UDF can pass its constructor arguments, or some other identifying strings. This allows each instantiation of the UDF to have a different properties object thus avoiding name space collisions between instantiations of the UDF.</p>
-</div>
-</div>
-<!--+
-    |end content
-    +-->
-<div class="clearboth">&nbsp;</div>
-</div>
-<div id="footer">
-<!--+
-    |start bottomstrip
-    +-->
-<div class="lastmodified">
-<script type="text/javascript"><!--
-document.write("Last Published: " + document.lastModified);
-//  --></script>
-</div>
-<div class="copyright">
-        Copyright &copy;
-         2007-2010 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
-</div>
-<!--+
-    |end bottomstrip
-    +-->
-</div>
-</body>
-</html>