RubyGems - wukong - Versions diffs - 3.0.0.pre → 3.0.0.pre2 - Mend

wukong 3.0.0.pre → 3.0.0.pre2

Files changed (476) hide show

data/.gitignore +46 -33
data/.gitmodules +3 -0
data/.rspec +1 -1
data/.travis.yml +8 -1
data/.yardopts +0 -13
data/Guardfile +4 -6
data/{LICENSE.textile → LICENSE.md} +43 -55
data/README-old.md +422 -0
data/README.md +279 -418
data/Rakefile +21 -5
data/TODO.md +6 -6
data/bin/wu-clean-encoding +31 -0
data/bin/wu-lign +2 -2
data/bin/wu-local +69 -0
data/bin/wu-server +70 -0
data/examples/Gemfile +38 -0
data/examples/README.md +9 -0
data/examples/dataflow/apache_log_line.rb +64 -25
data/examples/dataflow/fibonacci_series.rb +101 -0
data/examples/dataflow/parse_apache_logs.rb +37 -7
data/examples/{dataflow.rb → dataflow/scraper_macro_flow.rb} +0 -0
data/examples/dataflow/simple.rb +4 -4
data/examples/geo.rb +4 -0
data/examples/geo/geo_grids.numbers +0 -0
data/examples/geo/geolocated.rb +331 -0
data/examples/geo/quadtile.rb +69 -0
data/examples/geo/spec/geolocated_spec.rb +247 -0
data/examples/geo/tile_fetcher.rb +77 -0
data/examples/graph/minimum_spanning_tree.rb +61 -61
data/examples/jabberwocky.txt +36 -0
data/examples/models/wikipedia.rb +20 -0
data/examples/munging/Gemfile +8 -0
data/examples/munging/airline_flights/airline.rb +57 -0
data/examples/munging/airline_flights/airline_flights.rake +83 -0
data/{lib/wukong/settings.rb → examples/munging/airline_flights/airplane.rb} +0 -0
data/examples/munging/airline_flights/airport.rb +211 -0
data/examples/munging/airline_flights/airport_id_unification.rb +129 -0
data/examples/munging/airline_flights/airport_ok_chars.rb +4 -0
data/examples/munging/airline_flights/flight.rb +156 -0
data/examples/munging/airline_flights/models.rb +4 -0
data/examples/munging/airline_flights/parse.rb +26 -0
data/examples/munging/airline_flights/reconcile_airports.rb +142 -0
data/examples/munging/airline_flights/route.rb +35 -0
data/examples/munging/airline_flights/tasks.rake +83 -0
data/examples/munging/airline_flights/timezone_fixup.rb +62 -0
data/examples/munging/airline_flights/topcities.rb +167 -0
data/examples/munging/airports/40_wbans.txt +40 -0
data/examples/munging/airports/filter_weather_reports.rb +37 -0
data/examples/munging/airports/join.pig +31 -0
data/examples/munging/airports/to_tsv.rb +33 -0
data/examples/munging/airports/usa_wbans.pig +19 -0
data/examples/munging/airports/usa_wbans.txt +2157 -0
data/examples/munging/airports/wbans.pig +19 -0
data/examples/munging/airports/wbans.txt +2310 -0
data/examples/munging/geo/geo_json.rb +54 -0
data/examples/munging/geo/geo_models.rb +69 -0
data/examples/munging/geo/geonames_models.rb +78 -0
data/examples/munging/geo/iso_codes.rb +172 -0
data/examples/munging/geo/reconcile_countries.rb +124 -0
data/examples/munging/geo/tasks.rake +71 -0
data/examples/munging/rake_helper.rb +62 -0
data/examples/munging/weather/.gitignore +1 -0
data/examples/munging/weather/Gemfile +4 -0
data/examples/munging/weather/Rakefile +28 -0
data/examples/munging/weather/extract_ish.rb +13 -0
data/examples/munging/weather/models/weather.rb +119 -0
data/examples/munging/weather/utils/noaa_downloader.rb +46 -0
data/examples/munging/wikipedia/README.md +34 -0
data/examples/munging/wikipedia/Rakefile +193 -0
data/examples/munging/wikipedia/articles/extract_articles-parsed.rb +79 -0
data/examples/munging/wikipedia/articles/extract_articles-templated.rb +136 -0
data/examples/munging/wikipedia/articles/textualize_articles.rb +54 -0
data/examples/munging/wikipedia/articles/verify_structure.rb +43 -0
data/examples/munging/wikipedia/articles/wp2txt-LICENSE.txt +22 -0
data/examples/munging/wikipedia/articles/wp2txt_article.rb +259 -0
data/examples/munging/wikipedia/articles/wp2txt_utils.rb +452 -0
data/examples/munging/wikipedia/dbpedia/dbpedia_common.rb +4 -0
data/examples/munging/wikipedia/dbpedia/dbpedia_extract_geocoordinates.rb +78 -0
data/examples/munging/wikipedia/dbpedia/extract_links.rb +193 -0
data/examples/munging/wikipedia/dbpedia/sameas_extractor.rb +20 -0
data/examples/munging/wikipedia/n1_subuniverse/n1_nodes.pig +18 -0
data/examples/munging/wikipedia/page_metadata/extract_page_metadata.rb +21 -0
data/examples/munging/wikipedia/page_metadata/extract_page_metadata.rb.old +27 -0
data/examples/munging/wikipedia/pagelinks/augment_pagelinks.pig +29 -0
data/examples/munging/wikipedia/pagelinks/extract_pagelinks.rb +14 -0
data/examples/munging/wikipedia/pagelinks/extract_pagelinks.rb.old +25 -0
data/examples/munging/wikipedia/pagelinks/undirect_pagelinks.pig +29 -0
data/examples/munging/wikipedia/pageviews/augment_pageviews.pig +32 -0
data/examples/munging/wikipedia/pageviews/extract_pageviews.rb +85 -0
data/examples/munging/wikipedia/pig_style_guide.md +25 -0
data/examples/munging/wikipedia/redirects/redirects_page_metadata.pig +19 -0
data/examples/munging/wikipedia/subuniverse/sub_articles.pig +23 -0
data/examples/munging/wikipedia/subuniverse/sub_page_metadata.pig +24 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_from.pig +22 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_into.pig +22 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_within.pig +26 -0
data/examples/munging/wikipedia/subuniverse/sub_pageviews.pig +29 -0
data/examples/munging/wikipedia/subuniverse/sub_undirected_pagelinks_within.pig +24 -0
data/examples/munging/wikipedia/utils/get_namespaces.rb +86 -0
data/examples/munging/wikipedia/utils/munging_utils.rb +68 -0
data/examples/munging/wikipedia/utils/namespaces.json +1 -0
data/examples/rake_helper.rb +85 -0
data/examples/server_logs/geo_ip_mapping/munge_geolite.rb +82 -0
data/examples/server_logs/logline.rb +95 -0
data/examples/server_logs/models.rb +66 -0
data/examples/server_logs/page_counts.pig +48 -0
data/examples/server_logs/server_logs-01-parse-script.rb +13 -0
data/examples/server_logs/server_logs-02-histograms-full.rb +33 -0
data/examples/server_logs/server_logs-02-histograms-mapper.rb +14 -0
data/{old/examples/server_logs/breadcrumbs.rb → examples/server_logs/server_logs-03-breadcrumbs-full.rb} +26 -30
data/examples/server_logs/server_logs-04-page_page_edges-full.rb +40 -0
data/examples/string_reverser.rb +26 -0
data/examples/text/pig_latin.rb +2 -2
data/examples/text/regional_flavor/README.md +14 -0
data/examples/text/regional_flavor/article_wordbags.pig +39 -0
data/examples/text/regional_flavor/j01-article_wordbags.rb +4 -0
data/examples/text/regional_flavor/simple_pig_script.pig +27 -0
data/examples/word_count/accumulator.rb +26 -0
data/examples/word_count/tokenizer.rb +13 -0
data/examples/word_count/word_count.rb +6 -0
data/examples/workflow/cherry_pie.dot +97 -0
data/examples/workflow/cherry_pie.png +0 -0
data/examples/workflow/cherry_pie.rb +61 -26
data/lib/hanuman.rb +34 -7
data/lib/hanuman/graph.rb +55 -31
data/lib/hanuman/graphvizzer.rb +199 -178
data/lib/hanuman/graphvizzer/gv_models.rb +161 -0
data/lib/hanuman/graphvizzer/gv_presenter.rb +97 -0
data/lib/hanuman/link.rb +35 -0
data/lib/hanuman/registry.rb +46 -0
data/lib/hanuman/stage.rb +76 -32
data/lib/wukong.rb +23 -24
data/lib/wukong/boot.rb +87 -0
data/lib/wukong/configuration.rb +8 -0
data/lib/wukong/dataflow.rb +45 -78
data/lib/wukong/driver.rb +99 -0
data/lib/wukong/emitter.rb +22 -0
data/lib/wukong/model/faker.rb +24 -24
data/lib/wukong/model/flatpack_parser/flat.rb +60 -0
data/lib/wukong/model/flatpack_parser/flatpack.rb +4 -0
data/lib/wukong/model/flatpack_parser/lang.rb +46 -0
data/lib/wukong/model/flatpack_parser/parser.rb +55 -0
data/lib/wukong/model/flatpack_parser/tokens.rb +130 -0
data/lib/wukong/processor.rb +60 -114
data/lib/wukong/spec_helpers.rb +81 -0
data/lib/wukong/spec_helpers/integration_driver.rb +144 -0
data/lib/wukong/spec_helpers/integration_driver_matchers.rb +219 -0
data/lib/wukong/spec_helpers/processor_helpers.rb +95 -0
data/lib/wukong/spec_helpers/processor_methods.rb +108 -0
data/lib/wukong/spec_helpers/shared_examples.rb +15 -0
data/lib/wukong/spec_helpers/spec_driver.rb +28 -0
data/lib/wukong/spec_helpers/spec_driver_matchers.rb +195 -0
data/lib/wukong/version.rb +2 -1
data/lib/wukong/widget/filters.rb +311 -0
data/lib/wukong/widget/processors.rb +156 -0
data/lib/wukong/widget/reducers.rb +7 -0
data/lib/wukong/widget/reducers/accumulator.rb +73 -0
data/lib/wukong/widget/reducers/bin.rb +318 -0
data/lib/wukong/widget/reducers/count.rb +61 -0
data/lib/wukong/widget/reducers/group.rb +85 -0
data/lib/wukong/widget/reducers/group_concat.rb +70 -0
data/lib/wukong/widget/reducers/moments.rb +72 -0
data/lib/wukong/widget/reducers/sort.rb +130 -0
data/lib/wukong/widget/serializers.rb +287 -0
data/lib/wukong/widget/sink.rb +10 -52
data/lib/wukong/widget/source.rb +7 -113
data/lib/wukong/widget/utils.rb +46 -0
data/lib/wukong/widgets.rb +6 -0
data/spec/examples/dataflow/fibonacci_series_spec.rb +18 -0
data/spec/examples/dataflow/parsing_spec.rb +12 -11
data/spec/examples/dataflow/simple_spec.rb +32 -6
data/spec/examples/dataflow/telegram_spec.rb +36 -36
data/spec/examples/graph/minimum_spanning_tree_spec.rb +30 -31
data/spec/examples/munging/airline_flights/identifiers_spec.rb +16 -0
data/spec/examples/munging/airline_flights_spec.rb +202 -0
data/spec/examples/text/pig_latin_spec.rb +13 -16
data/spec/examples/workflow/cherry_pie_spec.rb +34 -4
data/spec/hanuman/graph_spec.rb +27 -2
data/spec/hanuman/hanuman_spec.rb +10 -0
data/spec/hanuman/registry_spec.rb +123 -0
data/spec/hanuman/stage_spec.rb +61 -7
data/spec/spec_helper.rb +29 -19
data/spec/support/hanuman_test_helpers.rb +14 -12
data/spec/support/shared_context_for_reducers.rb +37 -0
data/spec/support/shared_examples_for_builders.rb +101 -0
data/spec/support/shared_examples_for_shortcuts.rb +57 -0
data/spec/support/wukong_test_helpers.rb +37 -11
data/spec/wukong/dataflow_spec.rb +77 -55
data/spec/wukong/local_runner_spec.rb +24 -24
data/spec/wukong/model/faker_spec.rb +132 -131
data/spec/wukong/runner_spec.rb +8 -8
data/spec/wukong/widget/filters_spec.rb +61 -0
data/spec/wukong/widget/processors_spec.rb +126 -0
data/spec/wukong/widget/reducers/bin_spec.rb +92 -0
data/spec/wukong/widget/reducers/count_spec.rb +11 -0
data/spec/wukong/widget/reducers/group_spec.rb +20 -0
data/spec/wukong/widget/reducers/moments_spec.rb +36 -0
data/spec/wukong/widget/reducers/sort_spec.rb +26 -0
data/spec/wukong/widget/serializers_spec.rb +92 -0
data/spec/wukong/widget/sink_spec.rb +15 -15
data/spec/wukong/widget/source_spec.rb +65 -41
data/spec/wukong/wukong_spec.rb +10 -0
data/wukong.gemspec +17 -10
metadata +359 -335
data/.document +0 -5
data/VERSION +0 -1
data/bin/hdp-bin +0 -44
data/bin/hdp-bzip +0 -23
data/bin/hdp-cat +0 -3
data/bin/hdp-catd +0 -3
data/bin/hdp-cp +0 -3
data/bin/hdp-du +0 -86
data/bin/hdp-get +0 -3
data/bin/hdp-kill +0 -3
data/bin/hdp-kill-task +0 -3
data/bin/hdp-ls +0 -11
data/bin/hdp-mkdir +0 -2
data/bin/hdp-mkdirp +0 -12
data/bin/hdp-mv +0 -3
data/bin/hdp-parts_to_keys.rb +0 -77
data/bin/hdp-ps +0 -3
data/bin/hdp-put +0 -3
data/bin/hdp-rm +0 -32
data/bin/hdp-sort +0 -40
data/bin/hdp-stream +0 -40
data/bin/hdp-stream-flat +0 -22
data/bin/hdp-stream2 +0 -39
data/bin/hdp-sync +0 -17
data/bin/hdp-wc +0 -67
data/bin/wu-flow +0 -10
data/bin/wu-map +0 -17
data/bin/wu-red +0 -17
data/bin/wukong +0 -17
data/data/CREDITS.md +0 -355
data/data/graph/airfares.tsv +0 -2174
data/data/text/gift_of_the_magi.txt +0 -225
data/data/text/jabberwocky.txt +0 -36
data/data/text/rectification_of_names.txt +0 -33
data/data/twitter/a_atsigns_b.tsv +0 -64
data/data/twitter/a_follows_b.tsv +0 -53
data/data/twitter/tweet.tsv +0 -167
data/data/twitter/twitter_user.tsv +0 -55
data/data/wikipedia/dbpedia-sentences.tsv +0 -1000
data/docpages/INSTALL.textile +0 -92
data/docpages/LICENSE.textile +0 -107
data/docpages/README-elastic_map_reduce.textile +0 -377
data/docpages/README-performance.textile +0 -90
data/docpages/README-wulign.textile +0 -65
data/docpages/UsingWukong-part1-get_ready.textile +0 -17
data/docpages/UsingWukong-part2-ThinkingBigData.textile +0 -75
data/docpages/UsingWukong-part3-parsing.textile +0 -138
data/docpages/_config.yml +0 -39
data/docpages/avro/avro_notes.textile +0 -56
data/docpages/avro/performance.textile +0 -36
data/docpages/avro/tethering.textile +0 -19
data/docpages/bigdata-tips.textile +0 -143
data/docpages/code/api_response_example.txt +0 -20
data/docpages/code/parser_skeleton.rb +0 -38
data/docpages/diagrams/MapReduceDiagram.graffle +0 -0
data/docpages/favicon.ico +0 -0
data/docpages/gem.css +0 -16
data/docpages/hadoop-tips.textile +0 -83
data/docpages/index.textile +0 -92
data/docpages/intro.textile +0 -8
data/docpages/moreinfo.textile +0 -174
data/docpages/news.html +0 -24
data/docpages/pig/PigLatinExpressionsList.txt +0 -122
data/docpages/pig/PigLatinReferenceManual.txt +0 -1640
data/docpages/pig/commandline_params.txt +0 -26
data/docpages/pig/cookbook.html +0 -481
data/docpages/pig/images/hadoop-logo.jpg +0 -0
data/docpages/pig/images/instruction_arrow.png +0 -0
data/docpages/pig/images/pig-logo.gif +0 -0
data/docpages/pig/piglatin_ref1.html +0 -1103
data/docpages/pig/piglatin_ref2.html +0 -14340
data/docpages/pig/setup.html +0 -505
data/docpages/pig/skin/basic.css +0 -166
data/docpages/pig/skin/breadcrumbs.js +0 -237
data/docpages/pig/skin/fontsize.js +0 -166
data/docpages/pig/skin/getBlank.js +0 -40
data/docpages/pig/skin/getMenu.js +0 -45
data/docpages/pig/skin/images/chapter.gif +0 -0
data/docpages/pig/skin/images/chapter_open.gif +0 -0
data/docpages/pig/skin/images/current.gif +0 -0
data/docpages/pig/skin/images/external-link.gif +0 -0
data/docpages/pig/skin/images/header_white_line.gif +0 -0
data/docpages/pig/skin/images/page.gif +0 -0
data/docpages/pig/skin/images/pdfdoc.gif +0 -0
data/docpages/pig/skin/images/rc-b-l-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-b-r-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-b-r-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2searchbox-3searchbox.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2tab-unselected-3tab-unselected.png +0 -0
data/docpages/pig/skin/images/rc-t-r-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2searchbox-3searchbox.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2tab-unselected-3tab-unselected.png +0 -0
data/docpages/pig/skin/print.css +0 -54
data/docpages/pig/skin/profile.css +0 -181
data/docpages/pig/skin/screen.css +0 -587
data/docpages/pig/tutorial.html +0 -1059
data/docpages/pig/udf.html +0 -1509
data/docpages/tutorial.textile +0 -283
data/docpages/usage.textile +0 -195
data/docpages/wutils.textile +0 -263
data/examples/dataflow/complex.rb +0 -11
data/examples/dataflow/donuts.rb +0 -13
data/examples/tiny_count/jabberwocky_output.tsv +0 -92
data/examples/word_count.rb +0 -48
data/examples/workflow/fiddle.rb +0 -24
data/lib/away/escapement.rb +0 -129
data/lib/away/exe.rb +0 -11
data/lib/away/experimental.rb +0 -5
data/lib/away/from_file.rb +0 -52
data/lib/away/job.rb +0 -56
data/lib/away/job/rake_compat.rb +0 -17
data/lib/away/registry.rb +0 -79
data/lib/away/runner.rb +0 -276
data/lib/away/runner/execute.rb +0 -121
data/lib/away/script.rb +0 -161
data/lib/away/script/hadoop_command.rb +0 -240
data/lib/away/source/file_list_source.rb +0 -15
data/lib/away/source/looper.rb +0 -18
data/lib/away/task.rb +0 -219
data/lib/hanuman/action.rb +0 -21
data/lib/hanuman/chain.rb +0 -4
data/lib/hanuman/graphviz.rb +0 -74
data/lib/hanuman/resource.rb +0 -6
data/lib/hanuman/slot.rb +0 -87
data/lib/hanuman/slottable.rb +0 -220
data/lib/wukong/bad_record.rb +0 -15
data/lib/wukong/event.rb +0 -44
data/lib/wukong/local_runner.rb +0 -55
data/lib/wukong/mapred.rb +0 -3
data/lib/wukong/universe.rb +0 -48
data/lib/wukong/widget/filter.rb +0 -81
data/lib/wukong/widget/gibberish.rb +0 -123
data/lib/wukong/widget/monitor.rb +0 -26
data/lib/wukong/widget/reducer.rb +0 -66
data/lib/wukong/widget/stringifier.rb +0 -50
data/lib/wukong/workflow.rb +0 -22
data/lib/wukong/workflow/command.rb +0 -42
data/old/config/emr-example.yaml +0 -48
data/old/examples/README.txt +0 -17
data/old/examples/contrib/jeans/README.markdown +0 -165
data/old/examples/contrib/jeans/data/normalized_sizes +0 -3
data/old/examples/contrib/jeans/data/orders.tsv +0 -1302
data/old/examples/contrib/jeans/data/sizes +0 -3
data/old/examples/contrib/jeans/normalize.rb +0 -20
data/old/examples/contrib/jeans/sizes.rb +0 -55
data/old/examples/corpus/bnc_word_freq.rb +0 -44
data/old/examples/corpus/bucket_counter.rb +0 -47
data/old/examples/corpus/dbpedia_abstract_to_sentences.rb +0 -86
data/old/examples/corpus/sentence_bigrams.rb +0 -53
data/old/examples/corpus/sentence_coocurrence.rb +0 -66
data/old/examples/corpus/stopwords.rb +0 -138
data/old/examples/corpus/words_to_bigrams.rb +0 -53
data/old/examples/emr/README.textile +0 -110
data/old/examples/emr/dot_wukong_dir/credentials.json +0 -7
data/old/examples/emr/dot_wukong_dir/emr.yaml +0 -69
data/old/examples/emr/dot_wukong_dir/emr_bootstrap.sh +0 -33
data/old/examples/emr/elastic_mapreduce_example.rb +0 -28
data/old/examples/network_graph/adjacency_list.rb +0 -74
data/old/examples/network_graph/breadth_first_search.rb +0 -72
data/old/examples/network_graph/gen_2paths.rb +0 -68
data/old/examples/network_graph/gen_multi_edge.rb +0 -112
data/old/examples/network_graph/gen_symmetric_links.rb +0 -64
data/old/examples/pagerank/README.textile +0 -6
data/old/examples/pagerank/gen_initial_pagerank_graph.pig +0 -57
data/old/examples/pagerank/pagerank.rb +0 -72
data/old/examples/pagerank/pagerank_initialize.rb +0 -42
data/old/examples/pagerank/run_pagerank.sh +0 -21
data/old/examples/sample_records.rb +0 -33
data/old/examples/server_logs/apache_log_parser.rb +0 -15
data/old/examples/server_logs/nook.rb +0 -48
data/old/examples/server_logs/nook/faraday_dummy_adapter.rb +0 -94
data/old/examples/server_logs/user_agent.rb +0 -40
data/old/examples/simple_word_count.rb +0 -82
data/old/examples/size.rb +0 -61
data/old/examples/stats/avg_value_frequency.rb +0 -86
data/old/examples/stats/binning_percentile_estimator.rb +0 -140
data/old/examples/stats/data/avg_value_frequency.tsv +0 -3
data/old/examples/stats/rank_and_bin.rb +0 -173
data/old/examples/stupidly_simple_filter.rb +0 -40
data/old/examples/word_count.rb +0 -75
data/old/graph/graphviz_builder.rb +0 -580
data/old/graph_easy/Attributes.pm +0 -4181
data/old/graph_easy/Graphviz.pm +0 -2232
data/old/wukong.rb +0 -18
data/old/wukong/and_pig.rb +0 -38
data/old/wukong/bad_record.rb +0 -18
data/old/wukong/datatypes.rb +0 -24
data/old/wukong/datatypes/enum.rb +0 -127
data/old/wukong/datatypes/fake_types.rb +0 -17
data/old/wukong/decorator.rb +0 -28
data/old/wukong/encoding/asciize.rb +0 -108
data/old/wukong/extensions.rb +0 -16
data/old/wukong/extensions/array.rb +0 -18
data/old/wukong/extensions/blank.rb +0 -93
data/old/wukong/extensions/class.rb +0 -189
data/old/wukong/extensions/date_time.rb +0 -53
data/old/wukong/extensions/emittable.rb +0 -69
data/old/wukong/extensions/enumerable.rb +0 -79
data/old/wukong/extensions/hash.rb +0 -167
data/old/wukong/extensions/hash_keys.rb +0 -16
data/old/wukong/extensions/hash_like.rb +0 -150
data/old/wukong/extensions/hashlike_class.rb +0 -47
data/old/wukong/extensions/module.rb +0 -2
data/old/wukong/extensions/pathname.rb +0 -27
data/old/wukong/extensions/string.rb +0 -65
data/old/wukong/extensions/struct.rb +0 -17
data/old/wukong/extensions/symbol.rb +0 -11
data/old/wukong/filename_pattern.rb +0 -74
data/old/wukong/helper.rb +0 -7
data/old/wukong/helper/stopwords.rb +0 -195
data/old/wukong/helper/tokenize.rb +0 -35
data/old/wukong/logger.rb +0 -38
data/old/wukong/periodic_monitor.rb +0 -72
data/old/wukong/schema.rb +0 -269
data/old/wukong/script.rb +0 -286
data/old/wukong/script/avro_command.rb +0 -5
data/old/wukong/script/cassandra_loader_script.rb +0 -40
data/old/wukong/script/emr_command.rb +0 -168
data/old/wukong/script/hadoop_command.rb +0 -237
data/old/wukong/script/local_command.rb +0 -41
data/old/wukong/store.rb +0 -10
data/old/wukong/store/base.rb +0 -27
data/old/wukong/store/cassandra.rb +0 -10
data/old/wukong/store/cassandra/streaming.rb +0 -75
data/old/wukong/store/cassandra/struct_loader.rb +0 -21
data/old/wukong/store/cassandra_model.rb +0 -91
data/old/wukong/store/chh_chunked_flat_file_store.rb +0 -37
data/old/wukong/store/chunked_flat_file_store.rb +0 -48
data/old/wukong/store/conditional_store.rb +0 -57
data/old/wukong/store/factory.rb +0 -8
data/old/wukong/store/flat_file_store.rb +0 -89
data/old/wukong/store/key_store.rb +0 -51
data/old/wukong/store/null_store.rb +0 -15
data/old/wukong/store/read_thru_store.rb +0 -22
data/old/wukong/store/tokyo_tdb_key_store.rb +0 -33
data/old/wukong/store/tyrant_rdb_key_store.rb +0 -57
data/old/wukong/store/tyrant_tdb_key_store.rb +0 -20
data/old/wukong/streamer.rb +0 -30
data/old/wukong/streamer/accumulating_reducer.rb +0 -83
data/old/wukong/streamer/base.rb +0 -126
data/old/wukong/streamer/counting_reducer.rb +0 -25
data/old/wukong/streamer/filter.rb +0 -20
data/old/wukong/streamer/instance_streamer.rb +0 -15
data/old/wukong/streamer/json_streamer.rb +0 -21
data/old/wukong/streamer/line_streamer.rb +0 -12
data/old/wukong/streamer/list_reducer.rb +0 -31
data/old/wukong/streamer/rank_and_bin_reducer.rb +0 -145
data/old/wukong/streamer/record_streamer.rb +0 -14
data/old/wukong/streamer/reducer.rb +0 -11
data/old/wukong/streamer/set_reducer.rb +0 -14
data/old/wukong/streamer/struct_streamer.rb +0 -48
data/old/wukong/streamer/summing_reducer.rb +0 -29
data/old/wukong/streamer/uniq_by_last_reducer.rb +0 -51
data/old/wukong/typed_struct.rb +0 -12
data/spec/away/encoding_spec.rb +0 -32
data/spec/away/exe_spec.rb +0 -20
data/spec/away/flow_spec.rb +0 -82
data/spec/away/graph_spec.rb +0 -6
data/spec/away/job_spec.rb +0 -15
data/spec/away/rake_compat_spec.rb +0 -9
data/spec/away/script_spec.rb +0 -81
data/spec/hanuman/graphviz_spec.rb +0 -29
data/spec/hanuman/slot_spec.rb +0 -2
data/spec/support/examples_helper.rb +0 -10
data/spec/support/streamer_test_helpers.rb +0 -6
data/spec/support/wukong_widget_helpers.rb +0 -66
data/spec/wukong/processor_spec.rb +0 -109
data/spec/wukong/widget/filter_spec.rb +0 -99
data/spec/wukong/widget/stringifier_spec.rb +0 -51
data/spec/wukong/workflow/command_spec.rb +0 -5

data/docpages/pig/images/hadoop-logo.jpg DELETED

Binary file

data/docpages/pig/images/instruction_arrow.png DELETED

Binary file

data/docpages/pig/images/pig-logo.gif DELETED

Binary file

data/docpages/pig/piglatin_ref1.html DELETED

@@ -1,1103 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
-<html>
-<head>
-<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
-<meta content="Apache Forrest" name="Generator">
-<meta name="Forrest-version" content="0.8">
-<meta name="Forrest-skin-name" content="pelt">
-<title>Pig Latin Reference Manual 1</title>
-<link type="text/css" href="skin/basic.css" rel="stylesheet">
-<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
-<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
-<link type="text/css" href="skin/profile.css" rel="stylesheet">
-<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
-<link rel="shortcut icon" href="">
-</head>
-<body onload="init()">
-<script type="text/javascript">ndeSetTextSize();</script>
-<div id="top">
-<!--+
-    |breadtrail
-    +-->
-<div class="breadtrail">
-<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/pig/">Pig</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
-</div>
-<!--+
-    |header
-    +-->
-<div class="header">
-<!--+
-    |start group logo
-    +-->
-<div class="grouplogo">
-<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
-</div>
-<!--+
-    |end group logo
-    +-->
-<!--+
-    |start Project Logo
-    +-->
-<div class="projectlogo">
-<a href="http://hadoop.apache.org/pig/"><img class="logoImage" alt="Pig" src="images/pig-logo.gif" title="A platform for analyzing large datasets."></a>
-</div>
-<!--+
-    |end Project Logo
-    +-->
-<!--+
-    |start Search
-    +-->
-<div class="searchbox">
-<form action="http://www.google.com/search" method="get" class="roundtopsmall">
-<input value="" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp;
-                    <input name="Search" value="Search" type="submit">
-</form>
-</div>
-<!--+
-    |end search
-    +-->
-<!--+
-    |start Tabs
-    +-->
-<ul id="tabs">
-<li>
-<a class="unselected" href="http://hadoop.apache.org/pig/">Project</a>
-</li>
-<li>
-<a class="unselected" href="http://wiki.apache.org/pig/">Wiki</a>
-</li>
-<li class="current">
-<a class="selected" href="index.html">Pig 0.7.0 Documentation</a>
-</li>
-</ul>
-<!--+
-    |end Tabs
-    +-->
-</div>
-</div>
-<div id="main">
-<div id="publishedStrip">
-<!--+
-    |start Subtabs
-    +-->
-<div id="level2tabs"></div>
-<!--+
-    |end Endtabs
-    +-->
-<script type="text/javascript"><!--
-document.write("Last Published: " + document.lastModified);
-//  --></script>
-</div>
-<!--+
-    |breadtrail
-    +-->
-<div class="breadtrail">
-             &nbsp;
-           </div>
-<!--+
-    |start Menu, mainarea
-    +-->
-<!--+
-    |start Menu
-    +-->
-<div id="menu">
-<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Pig</div>
-<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
-<div class="menuitem">
-<a href="index.html">Overview</a>
-</div>
-<div class="menuitem">
-<a href="setup.html">Setup</a>
-</div>
-<div class="menuitem">
-<a href="tutorial.html">Tutorial</a>
-</div>
-<div class="menupage">
-<div class="menupagetitle">Pig Latin 1</div>
-</div>
-<div class="menuitem">
-<a href="piglatin_ref2.html">Pig Latin 2</a>
-</div>
-<div class="menuitem">
-<a href="cookbook.html">Cookbook</a>
-</div>
-<div class="menuitem">
-<a href="udf.html">UDFs</a>
-</div>
-</div>
-<div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Zebra</div>
-<div id="menu_1.2" class="menuitemgroup">
-<div class="menuitem">
-<a href="zebra_overview.html">Zebra Overview </a>
-</div>
-<div class="menuitem">
-<a href="zebra_users.html">Zebra Users </a>
-</div>
-<div class="menuitem">
-<a href="zebra_reference.html">Zebra Reference </a>
-</div>
-<div class="menuitem">
-<a href="zebra_mapreduce.html">Zebra MapReduce </a>
-</div>
-<div class="menuitem">
-<a href="zebra_pig.html">Zebra Pig </a>
-</div>
-<div class="menuitem">
-<a href="zebra_stream.html">Zebra Streaming </a>
-</div>
-</div>
-<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">Miscellaneous</div>
-<div id="menu_1.3" class="menuitemgroup">
-<div class="menuitem">
-<a href="api/">API Docs</a>
-</div>
-<div class="menuitem">
-<a href="http://wiki.apache.org/pig/">Wiki</a>
-</div>
-<div class="menuitem">
-<a href="http://wiki.apache.org/pig/FAQ">FAQ</a>
-</div>
-<div class="menuitem">
-<a href="http://hadoop.apache.org/pig/releases.html">Release Notes</a>
-</div>
-</div>
-<div id="credit"></div>
-<div id="roundbottom">
-<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
-<!--+
-  |alternative credits
-  +-->
-<div id="credit2"></div>
-</div>
-<!--+
-    |end Menu
-    +-->
-<!--+
-    |start content
-    +-->
-<div id="content">
-<div title="Portable Document Format" class="pdflink">
-<a class="dida" href="piglatin_ref1.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
-        PDF</a>
-</div>
-<h1>Pig Latin Reference Manual 1</h1>
-<div id="minitoc-area">
-<ul class="minitoc">
-<li>
-<a href="#Overview">Overview</a>
-</li>
-<li>
-<a href="#Pig+Latin+Statements">Pig Latin Statements</a>
-<ul class="minitoc">
-<li>
-<a href="#Running+Pig+Latin">Running Pig Latin </a>
-</li>
-<li>
-<a href="#Retrieving+Pig+Latin+Results">Retrieving Pig Latin Results</a>
-</li>
-<li>
-<a href="#Debugging+Pig+Latin">Debugging Pig Latin</a>
-</li>
-<li>
-<a href="#Working+with+Data">Working with Data</a>
-</li>
-<li>
-<a href="#Using+Comments+in+Scripts">Using Comments in Scripts</a>
-</li>
-<li>
-<a href="#Case+Sensitivity">Case Sensitivity</a>
-</li>
-</ul>
-</li>
-<li>
-<a href="#Multi-Query+Execution">Multi-Query Execution</a>
-<ul class="minitoc">
-<li>
-<a href="#Turning+it+On+or+Off">Turning it On or Off</a>
-</li>
-<li>
-<a href="#How+it+Works">How it Works</a>
-</li>
-<li>
-<a href="#Store+vs.+Dump">Store vs. Dump</a>
-</li>
-<li>
-<a href="#Error+Handling">Error Handling</a>
-</li>
-<li>
-<a href="#Backward+Compatibility">Backward Compatibility</a>
-</li>
-<li>
-<a href="#Implicit+Dependencies">Implicit Dependencies</a>
-</li>
-</ul>
-</li>
-<li>
-<a href="#Specialized+Joins">Specialized Joins</a>
-<ul class="minitoc">
-<li>
-<a href="#Replicated+Joins">Replicated Joins</a>
-</li>
-<li>
-<a href="#Skewed+Joins">Skewed Joins</a>
-</li>
-<li>
-<a href="#Merge+Joins">Merge Joins</a>
-</li>
-</ul>
-</li>
-<li>
-<a href="#Optimization+Rules">Optimization Rules</a>
-<ul class="minitoc">
-<li>
-<a href="#ImplicitSplitInserter">ImplicitSplitInserter</a>
-</li>
-<li>
-<a href="#TypeCastInserter">TypeCastInserter</a>
-</li>
-<li>
-<a href="#StreamOptimizer">StreamOptimizer</a>
-</li>
-<li>
-<a href="#OpLimitOptimizer">OpLimitOptimizer</a>
-</li>
-<li>
-<a href="#PushUpFilters">PushUpFilters</a>
-</li>
-<li>
-<a href="#PushDownExplodes">PushDownExplodes</a>
-</li>
-</ul>
-</li>
-<li>
-<a href="#Memory+Management">Memory Management</a>
-</li>
-<li>
-<a href="#Zebra+Integration">Zebra Integration</a>
-</li>
-</ul>
-</div>
- <!-- ABOUT PIG LATIN -->
-<a name="N10010"></a><a name="Overview"></a>
-<h2 class="h3">Overview</h2>
-<div class="section">
-<p>Use this manual together with <a href="piglatin_ref2.html">Pig Latin Reference Manual 2</a>. </p>
-<p>Also, be sure to review the information in the <a href="cookbook.html">Pig Cookbook</a>. </p>
-</div>
-   <!-- PIG LATIN STATEMENTS -->
-<a name="N10027"></a><a name="Pig+Latin+Statements"></a>
-<h2 class="h3">Pig Latin Statements</h2>
-<div class="section">
-<p>A Pig Latin statement is an operator that takes a <a href="piglatin_ref2.html#Relations%2C+Bags%2C+Tuples%2C+Fields">relation</a>
-   as input and produces another relation as output.
-   (This definition applies to all Pig Latin operators except LOAD and STORE which read data from and write data to the file system.)
-   Pig Latin statements can span multiple lines and must end with a semi-colon ( ; ).
-   Pig Latin statements are generally organized in the following manner: </p>
-<ol>
-<li>
-<p>A LOAD statement reads data from the file system. </p>
-</li>
-<li>
-<p>A series of "transformation" statements process the data. </p>
-</li>
-<li>
-<p>A STORE statement writes output to the file system; or, a DUMP statement displays output to the screen.</p>
-</li>
-</ol>
-<a name="N10049"></a><a name="Running+Pig+Latin"></a>
-<h3 class="h4">Running Pig Latin </h3>
-<p>You can execute Pig Latin statements: </p>
-<ul>
-<li>Using grunt shell or command line</li>
-<li>In mapreduce mode or local mode</li>
-<li>Either interactively or in batch </li>
-</ul>
-<p></p>
-<p>Note that Pig now uses Hadoop's local mode (rather than Pig's native local mode).</p>
-<p>A few run examples are shown here; see <a href="setup.html">Pig Setup</a> for more examples.</p>
-<p>Grunt Shell - interactive, mapreduce mode (because mapreduce mode is the default you do not need to specify)</p>
-<pre class="code">
-$ pig
-... - Connecting to ...
-grunt&gt; A = load 'data';
-grunt&gt; B = ... ;
-</pre>
-<p>Grunt Shell - batch, local mode (see the <a href="piglatin_ref2.html#exec">exec</a> and <a href="piglatin_ref2.html#run">run</a> commands)</p>
-<pre class="code">
-$ pig -x local
-grunt&gt; exec myscript.pig;
-or
-grunt&gt; run myscript.pig;
-</pre>
-<p>Command Line - batch, mapreduce mode</p>
-<pre class="code">
-$ pig myscript.pig
-</pre>
-<p>Command Line - batch, local mode mode</p>
-<pre class="code">
-$ pig -x local myscript.pig
-</pre>
-<p></p>
-<p>
-<em>In general</em>, Pig processes Pig Latin statements as follows:</p>
-<ol>
-<li>
-<p>First, Pig validates the syntax and semantics of all statements.</p>
-</li>
-<li>
-<p>Next, if Pig encounters a DUMP or STORE, Pig will execute the statements.</p>
-</li>
-</ol>
-<p></p>
-<p>In this example Pig will validate, but not execute, the LOAD and FOREACH statements.</p>
-<pre class="code">
-A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);
-B = FOREACH A GENERATE name;
-</pre>
-<p>In this example, Pig will validate and then execute the LOAD, FOREACH, and DUMP statements.</p>
-<pre class="code">
-A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);
-B = FOREACH A GENERATE name;
-DUMP B;
-(John)
-(Mary)
-(Bill)
-(Joe)
-</pre>
-<p>
-</p>
-<p>See <a href="#Multi-Query+Execution">Multi-Query Execution</a> for more information on how Pig Latin statements are processed.</p>
-<a name="N100BF"></a><a name="Retrieving+Pig+Latin+Results"></a>
-<h3 class="h4">Retrieving Pig Latin Results</h3>
-<p>Pig Latin includes operators you can use to retrieve the results of your Pig Latin statements: </p>
-<ol>
-<li>
-<p>Use the DUMP operator to display results to a screen. </p>
-</li>
-<li>
-<p>Use the STORE operator to write results to a file on the file system.</p>
-</li>
-</ol>
-<a name="N100D8"></a><a name="Debugging+Pig+Latin"></a>
-<h3 class="h4">Debugging Pig Latin</h3>
-<p>Pig Latin includes operators that can help you debug your Pig Latin statements:</p>
-<ol>
-<li>
-<p>Use the DESCRIBE operator to review the schema of a relation.</p>
-</li>
-<li>
-<p>Use the EXPLAIN operator to view the logical, physical, or map reduce execution plans to compute a relation.</p>
-</li>
-<li>
-<p>Use the ILLUSTRATE operator to view the step-by-step execution of a series of statements.</p>
-</li>
-</ol>
-<a name="N100F7"></a><a name="Working+with+Data"></a>
-<h3 class="h4">Working with Data</h3>
-<p>Pig Latin allows you to work with data in many ways. In general, and as a starting point:</p>
-<ol>
-<li>
-<p>Use the FILTER operator to work with tuples or rows of data. Use the FOREACH operator to work with columns of data.</p>
-</li>
-<li>
-<p>Use the GROUP operator to group data in a single relation. Use the COGROUP and JOIN operators to group or join data in two or more relations.</p>
-</li>
-<li>
-<p>Use the UNION operator to merge the contents of two or more relations. Use the SPLIT operator to partition the contents of a relation into multiple relations.</p>
-</li>
-</ol>
-<a name="N10116"></a><a name="Using+Comments+in+Scripts"></a>
-<h3 class="h4">Using Comments in Scripts</h3>
-<p>If you place Pig Latin statements in a script, the script can include comments. </p>
-<ol>
-<li>
-<p>For multi-line comments use /* &hellip;. */</p>
-</li>
-<li>
-<p>For single line comments use --</p>
-</li>
-</ol>
-<pre class="code">
-/* myscript.pig
-My script includes three simple Pig Latin Statements.
-*/
-A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float); -- load statement
-B = FOREACH A GENERATE name;  -- foreach statement
-DUMP B;  --dump statement
-</pre>
-<a name="N10133"></a><a name="Case+Sensitivity"></a>
-<h3 class="h4">Case Sensitivity</h3>
-<p>The names (aliases) of relations and fields are case sensitive. The names of Pig Latin functions are case sensitive.
-   The names of parameters (see Parameter Substitution) and all other Pig Latin keywords are case insensitive.</p>
-<p>In the example below, note the following:</p>
-<ol>
-<li>
-<p>The names (aliases) of relations A, B, and C are case sensitive.</p>
-</li>
-<li>
-<p>The names (aliases) of fields f1, f2, and f3 are case sensitive.</p>
-</li>
-<li>
-<p>Function names PigStorage and COUNT are case sensitive.</p>
-</li>
-<li>
-<p>Keywords LOAD, USING, AS, GROUP, BY, FOREACH, GENERATE, and DUMP are case insensitive.
-         They can also be written as load, using, as, group, by, etc.</p>
-</li>
-<li>
-<p>In the FOREACH statement, the field in relation B is referred to by positional notation ($0).</p>
-</li>
-</ol>
-<p></p>
-<pre class="code">
-grunt&gt; A = LOAD 'data' USING PigStorage() AS (f1:int, f2:int, f3:int);
-grunt&gt; B = GROUP A BY f1;
-grunt&gt; C = FOREACH B GENERATE COUNT ($0);
-grunt&gt; DUMP C;
-</pre>
-</div>
-<!-- END PIG LATIN STATEMENTS -->
-<!-- MULTI-QUERY EXECUTION-->
-<a name="N1016C"></a><a name="Multi-Query+Execution"></a>
-<h2 class="h3">Multi-Query Execution</h2>
-<div class="section">
-<p>With multi-query execution Pig processes an entire script or a batch of statements at once.</p>
-<a name="N10175"></a><a name="Turning+it+On+or+Off"></a>
-<h3 class="h4">Turning it On or Off</h3>
-<p>Multi-query execution is turned on by default.
-	To turn it off and revert to Pig's "execute-on-dump/store" behavior, use the "-M" or "-no_multiquery" options. </p>
-<p>To run script "myscript.pig" without the optimization, execute Pig as follows: </p>
-<pre class="code">
-$ pig -M myscript.pig
-or
-$ pig -no_multiquery myscript.pig
-</pre>
-<a name="N10186"></a><a name="How+it+Works"></a>
-<h3 class="h4">How it Works</h3>
-<p>Multi-query execution introduces some changes:</p>
-<ol>
-<li>
-<p>For batch mode execution, the entire script is first parsed to determine if intermediate tasks
-can be combined to reduce the overall amount of work that needs to be done; execution starts only after the parsing is completed
-(see the <a href="piglatin_ref2.html#EXPLAIN">EXPLAIN</a> operator and the <a href="piglatin_ref2.html#exec">exec</a> and <a href="piglatin_ref2.html#run">run</a> commands). </p>
-</li>
-<li>
-<p>Two run scenarios are optimized, as explained below: explicit and implicit splits, and storing intermediate results.</p>
-</li>
-</ol>
-<a name="N101AA"></a><a name="Explicit+and+Implicit+Splits"></a>
-<h4>Explicit and Implicit Splits</h4>
-<p>There might be cases in which you want different processing on separate parts of the same data stream.</p>
-<p>Example 1:</p>
-<pre class="code">
-A = LOAD ...
-...
-SPLIT A' INTO B IF ..., C IF ...
-...
-STORE B' ...
-STORE C' ...
-</pre>
-<p>Example 2:</p>
-<pre class="code">
-A = LOAD ...
-...
-B = FILTER A' ...
-C = FILTER A' ...
-...
-STORE B' ...
-STORE C' ...
-</pre>
-<p>In prior Pig releases, Example 1 will dump A' to disk and then start jobs for B' and C'.
-Example 2 will execute all the dependencies of B' and store it and then execute all the dependencies of C' and store it.
-Both are equivalent, but the performance will be different. </p>
-<p>Here's what the multi-query execution does to increase the performance: </p>
-<ol>
-<li>
-<p>For Example 2, adds an implicit split to transform the query to Example 1.
-		This eliminates the processing of A' multiple times.</p>
-</li>
-<li>
-<p>Makes the split non-blocking and allows processing to continue.
-		This helps reduce the amount of data that has to be stored right at the split.  </p>
-</li>
-<li>
-<p>Allows multiple outputs from a job. This way some results can be stored as a side-effect of the main job.
-		This is also necessary to make the previous item work.  </p>
-</li>
-<li>
-<p>Allows multiple split branches to be carried on to the combiner/reducer.
-		This reduces the amount of IO again in the case where multiple branches in the split can benefit from a combiner run. </p>
-</li>
-</ol>
-<a name="N101DB"></a><a name="Storing+Intermediate+Results"></a>
-<h4>Storing Intermediate Results</h4>
-<p>Sometimes it is necessary to store intermediate results. </p>
-<pre class="code">
-A = LOAD ...
-...
-STORE A'
-...
-STORE A''
-</pre>
-<p>If the script doesn't re-load A' for the processing of A the steps above A' will be duplicated.
-This is a special case of Example 2 above, so the same steps are recommended.
-With multi-query execution, the script will process A and dump A' as a side-effect.</p>
-<a name="N101ED"></a><a name="Store+vs.+Dump"></a>
-<h3 class="h4">Store vs. Dump</h3>
-<p>With multi-query exection, you want to use <a href="piglatin_ref2.html#STORE">STORE</a> to save (persist) your results.
-	You do not want to use <a href="piglatin_ref2.html#DUMP">DUMP</a> as it will disable multi-query execution and is likely to slow down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.) </p>
-<p>DUMP Example: In this script, because the DUMP command is interactive, the multi-query execution will be disabled and two separate jobs will be created to execute this script. The first job will execute A &gt; B &gt; DUMP while the second job will execute A &gt; B &gt; C &gt; STORE.</p>
-<pre class="code">
-A = LOAD 'input' AS (x, y, z);
-B = FILTER A BY x &gt; 5;
-DUMP B;
-C = FOREACH B GENERATE y, z;
-STORE C INTO 'output';
-</pre>
-<p>STORE Example: In this script, multi-query optimization will kick in allowing the entire script to be executed as a single job. Two outputs are produced: output1 and output2.</p>
-<pre class="code">
-A = LOAD 'input' AS (x, y, z);
-B = FILTER A BY x &gt; 5;
-STORE B INTO 'output1';
-C = FOREACH B GENERATE y, z;
-STORE C INTO 'output2';
-</pre>
-<a name="N1020D"></a><a name="Error+Handling"></a>
-<h3 class="h4">Error Handling</h3>
-<p>With multi-query execution Pig processes an entire script or a batch of statements at once.
-	By default Pig tries to run all the jobs that result from that, regardless of whether some jobs fail during execution.
-	To check which jobs have succeeded or failed use one of these options. </p>
-<p>First, Pig logs all successful and failed store commands. Store commands are identified by output path.
-	At the end of execution a summary line indicates success, partial failure or failure of all store commands. </p>
-<p>Second, Pig returns different code upon completion for these scenarios:</p>
-<ol>
-<li>
-<p>Return code 0: All jobs succeeded</p>
-</li>
-<li>
-<p>Return code 1: <em>Used for retrievable errors</em>
-</p>
-</li>
-<li>
-<p>Return code 2: All jobs have failed </p>
-</li>
-<li>
-<p>Return code 3: Some jobs have failed  </p>
-</li>
-</ol>
-<p></p>
-<p>In some cases it might be desirable to fail the entire script upon detecting the first failed job.
-	This can be achieved with the "-F" or "-stop_on_failure" command line flag.
-	If used, Pig will stop execution when the first failed job is detected and discontinue further processing.
-	This also means that file commands that come after a failed store in the script will not be executed (this can be used to create "done" files). </p>
-<p>This is how the flag is used: </p>
-<pre class="code">
-$ pig -F myscript.pig
-or
-$ pig -stop_on_failure myscript.pig
-</pre>
-<a name="N1023F"></a><a name="Backward+Compatibility"></a>
-<h3 class="h4">Backward Compatibility</h3>
-<p>Most existing Pig scripts will produce the same result with or without the multi-query execution.
-	There are cases though where this is not true. Path names and schemes are discussed here.</p>
-<p>Any script is parsed in it's entirety before it is sent to execution. Since the current directory can change
-	throughout the script any path used in LOAD or STORE statement is translated to a fully qualified and absolute path.</p>
-<p>In map-reduce mode, the following script will load from "hdfs://&lt;host&gt;:&lt;port&gt;/data1" and store into "hdfs://&lt;host&gt;:&lt;port&gt;/tmp/out1". </p>
-<pre class="code">
-cd /;
-A = LOAD 'data1';
-cd tmp;
-STORE A INTO 'out1';
-</pre>
-<p>These expanded paths will be passed to any LoadFunc or Slicer implementation.
-	In some cases this can cause problems, especially when a LoadFunc/Slicer is not used to read from a dfs file or path
-	(for example, loading from an SQL database). </p>
-<p>Solutions are to either: </p>
-<ol>
-<li>
-<p>Specify "-M" or "-no_multiquery" to revert to the old names</p>
-</li>
-<li>
-<p>Specify a custom scheme for the LoadFunc/Slicer </p>
-</li>
-</ol>
-<p>Arguments used in a LOAD statement that have a scheme other than "hdfs" or "file" will not be expanded and passed to the LoadFunc/Slicer unchanged.</p>
-<p>In the SQL case, the SQLLoader function is invoked with 'sql://mytable'. </p>
-<pre class="code">
-A = LOAD 'sql://mytable' USING SQLLoader();
-</pre>
-<a name="N1026E"></a><a name="Implicit+Dependencies"></a>
-<h3 class="h4">Implicit Dependencies</h3>
-<p>If a script has dependencies on the execution order outside of what Pig knows about, execution may fail. </p>
-<a name="N10277"></a><a name="Example"></a>
-<h4>Example</h4>
-<p>In this script, MYUDF might try to read from out1, a file that A was just stored into.
-However, Pig does not know that MYUDF depends on the out1 file and might submit the jobs
-producing the out2 and out1 files at the same time.</p>
-<pre class="code">
-...
-STORE A INTO 'out1';
-B = LOAD 'data2';
-C = FOREACH B GENERATE MYUDF($0,'out1');
-STORE C INTO 'out2';
-</pre>
-<p>To make the script work (to ensure that the right execution order is enforced) add the exec statement.
-The exec statement will trigger the execution of the statements that produce the out1 file. </p>
-<pre class="code">
-...
-STORE A INTO 'out1';
-EXEC;
-B = LOAD 'data2';
-C = FOREACH B GENERATE MYUDF($0,'out1');
-STORE C INTO 'out2';
-</pre>
-<a name="N1028C"></a><a name="Example-N1028C"></a>
-<h4>Example</h4>
-<p>In this script, the STORE/LOAD operators have different file paths; however, the LOAD operator depends on the STORE operator.</p>
-<pre class="code">
-A = LOAD '/user/xxx/firstinput' USING PigStorage();
-B = group ....
-C = .... agrregation function
-STORE C INTO '/user/vxj/firstinputtempresult/days1';
-..
-Atab = LOAD '/user/xxx/secondinput' USING  PigStorage();
-Btab = group ....
-Ctab = .... agrregation function
-STORE Ctab INTO '/user/vxj/secondinputtempresult/days1';
-..
-E = LOAD '/user/vxj/firstinputtempresult/' USING  PigStorage();
-F = group ....
-G = .... aggregation function
-STORE G INTO '/user/vxj/finalresult1';
-Etab =LOAD '/user/vxj/secondinputtempresult/' USING  PigStorage();
-Ftab = group ....
-Gtab = .... aggregation function
-STORE Gtab INTO '/user/vxj/finalresult2';
-</pre>
-<p>To make the script works, add the exec statement.  </p>
-<pre class="code">
-A = LOAD '/user/xxx/firstinput' USING PigStorage();
-B = group ....
-C = .... agrregation function
-STORE C INTO '/user/vxj/firstinputtempresult/days1';
-..
-Atab = LOAD '/user/xxx/secondinput' USING  PigStorage();
-Btab = group ....
-Ctab = .... agrregation function
-STORE Ctab INTO '/user/vxj/secondinputtempresult/days1';
-EXEC;
-E = LOAD '/user/vxj/firstinputtempresult/' USING  PigStorage();
-F = group ....
-G = .... aggregation function
-STORE G INTO '/user/vxj/finalresult1';
-..
-Etab =LOAD '/user/vxj/secondinputtempresult/' USING  PigStorage();
-Ftab = group ....
-Gtab = .... aggregation function
-STORE Gtab INTO '/user/vxj/finalresult2';
-</pre>
-</div>
-<!-- END MULTI-QUERY EXECUTION-->
-<!-- SPECIALIZED JOINS-->
-<a name="N102A7"></a><a name="Specialized+Joins"></a>
-<h2 class="h3">Specialized Joins</h2>
-<div class="section">
-<p>
-Pig Latin includes three "specialized" joins: replicated joins, skewed joins, and merge joins. </p>
-<ul>
-<li>Replicated, skewed, and merge joins can be performed using <a href="piglatin_ref2.html#JOIN+%28inner%29">inner joins</a>.</li>
-<li>Replicated and skewed joins can also be performed using <a href="piglatin_ref2.html#JOIN+%28outer%29">outer joins</a>.</li>
-</ul>
-<a name="N102C3"></a><a name="Replicated+Joins"></a>
-<h3 class="h4">Replicated Joins</h3>
-<p>Fragment replicate join is a special type of join that works well if one or more relations are small enough to fit into main memory.
-In such cases, Pig can perform a very efficient join because all of the hadoop work is done on the map side. In this type of join the
-large relation is followed by one or more small relations. The small relations must be small enough to fit into main memory; if they
-don't, the process fails and an error is generated.</p>
-<a name="N102CC"></a><a name="Usage"></a>
-<h4>Usage</h4>
-<p>Perform a replicated join with the USING clause (see <a href="piglatin_ref2.html#JOIN+%28inner%29">inner joins</a> and <a href="piglatin_ref2.html#JOIN+%28outer%29">outer joins</a>).
-In this example, a large relation is joined with two smaller relations. Note that the large relation comes first followed by the smaller relations;
-and, all small relations together must fit into main memory, otherwise an error is generated. </p>
-<pre class="code">
-big = LOAD 'big_data' AS (b1,b2,b3);
-tiny = LOAD 'tiny_data' AS (t1,t2,t3);
-mini = LOAD 'mini_data' AS (m1,m2,m3);
-C = JOIN big BY b1, tiny BY t1, mini BY m1 USING 'replicated';
-</pre>
-<a name="N102E2"></a><a name="Conditions"></a>
-<h4>Conditions</h4>
-<p>Fragment replicate joins are experimental; we don't have a strong sense of how small the small relation must be to fit
-into memory. In our tests with a simple query that involves just a JOIN, a relation of up to 100 M can be used if the process overall
-gets 1 GB of memory. Please share your observations and experience with us.</p>
-<a name="N102F1"></a><a name="Skewed+Joins"></a>
-<h3 class="h4">Skewed Joins</h3>
-<p>
-Parallel joins are vulnerable to the presence of skew in the underlying data.
-If the underlying data is sufficiently skewed, load imbalances will swamp any of the parallelism gains.
-In order to counteract this problem, skewed join computes a histogram of the key space and uses this
-data to allocate reducers for a given key. Skewed join does not place a restriction on the size of the input keys.
-It accomplishes this by splitting the left input on the join predicate and streaming the right input. The left input is
-sampled to create the histogram.
-</p>
-<p>
-Skewed join can be used when the underlying data is sufficiently skewed and you need a finer
-control over the allocation of reducers to counteract the skew. It should also be used when the data
-associated with a given key is too large to fit in memory.
-</p>
-<a name="N102FD"></a><a name="Usage-N102FD"></a>
-<h4>Usage</h4>
-<p>Perform a skewed join with the USING clause (see <a href="piglatin_ref2.html#JOIN+%28inner%29">inner joins</a> and <a href="piglatin_ref2.html#JOIN+%28outer%29">outer joins</a>). </p>
-<pre class="code">
-big = LOAD 'big_data' AS (b1,b2,b3);
-massive = LOAD 'massive_data' AS (m1,m2,m3);
-C = JOIN big BY b1, massive BY m1 USING 'skewed';
-</pre>
-<a name="N10313"></a><a name="Conditions-N10313"></a>
-<h4>Conditions</h4>
-<p>
-Skewed join will only work under these conditions:
-</p>
-<ul>
-<li>Skewed join works with two-table inner join. Currently we do not support more than two tables for skewed join.
-Specifying three-way (or more) joins will fail validation. For such joins, we rely on you to break them up into two-way joins.</li>
-<li>The pig.skewedjoin.reduce.memusage Java parameter specifies the fraction of heap available for the
-reducer to perform the join. A low fraction forces pig to use more reducers but increases
-copying cost. We have seen good performance when we set this value
-in the range 0.1 - 0.4. However, note that this is hardly an accurate range. Its value
-depends on the amount of heap available for the operation, the number of columns
-in the input and the skew. An appropriate value is best obtained by conducting experiments to achieve
-a good performance. The default value is =0.5=. </li>
-</ul>
-<a name="N1032A"></a><a name="Merge+Joins"></a>
-<h3 class="h4">Merge Joins</h3>
-<p>
-Often user data is stored such that both inputs are already sorted on the join key.
-In this case, it is possible to join the data in the map phase of a MapReduce job.
-This provides a significant performance improvement compared to passing all of the data through
-unneeded sort and shuffle phases.
-</p>
-<p>
-Pig has implemented a merge join algorithm, or sort-merge join, although in this case the sort is already
-assumed to have been done (see the Conditions, below).
-Pig implements the merge join algorithm by selecting the left input of the join to be the input file for the map phase,
-and the right input of the join to be the side file. It then samples records from the right input to build an
- index that contains, for each sampled record, the key(s) the filename and the offset into the file the record
- begins at. This sampling is done in an initial map only job. A second MapReduce job is then initiated,
- with the left input as its input. Each map uses the index to seek to the appropriate record in the right
- input and begin doing the join.
-</p>
-<a name="N10336"></a><a name="Usage-N10336"></a>
-<h4>Usage</h4>
-<p>Perform a merge join with the USING clause (see <a href="piglatin_ref2.html#JOIN+%28inner%29">inner joins</a>).</p>
-<pre class="code">
-C = JOIN A BY a1, B BY b1 USING 'merge';
-</pre>
-<a name="N10348"></a><a name="Conditions-N10348"></a>
-<h4>Conditions</h4>
-<p>
-Merge join will only work under these conditions:
-</p>
-<ul>
-<li>Both inputs are sorted in *ascending* order of join keys. If an input consists of many files, there should be
-a total ordering across the files in the *ascending order of file name*. So for example if one of the inputs to the
-join is a directory called input1 with files a and b under it, the data should be sorted in ascending order of join
-key when read starting at a and ending in b. Likewise if an input directory has part files part-00000, part-00001,
-part-00002 and part-00003, the data should be sorted if the files are read in the sequence part-00000, part-00001,
-part-00002 and part-00003. </li>
-<li>The merge join only has two inputs </li>
-<li>The loadfunc for the right input of the join should implement the OrderedLoadFunc interface (PigStorage does
-implement the OrderedLoadFunc interface). </li>
-<li>Only inner join will be supported </li>
-<li>Between the load of the sorted input and the merge join statement there can only be filter statements and
-foreach statement where the foreach statement should meet the following conditions:
-<ul>
-<li>There should be no UDFs in the foreach statement </li>
-<li>The foreach statement should not change the position of the join keys </li>
-<li>There should not transformation on the join keys which will change the sort order </li>
-</ul>
-</li>
-</ul>
-<p></p>
-<p>
-For optimal performance, each part file of the left (sorted) input of the join should have a size of at least
-1 hdfs block size (for example if the hdfs block size is 128 MB, each part file should be less than 128 MB).
-If the total input size (including all part files) is greater than blocksize, then the part files should be uniform in size
-(without large skews in sizes). The main idea is to eliminate skew in the amount of input the final map
-job performing the merge-join will process.
-</p>
-<p>
-In local mode, merge join will revert to regular join.
-</p>
-</div>
-<!-- END SPECIALIZED JOINS-->
- <!-- OPTIMIZATION RULES -->
-<a name="N1037F"></a><a name="Optimization+Rules"></a>
-<h2 class="h3">Optimization Rules</h2>
-<div class="section">
-<p>Pig supports various optimization rules. By default optimization, and all optimization rules, are turned on.
-To turn off optimiztion, use:</p>
-<pre class="code">
-pig -optimizer_off [opt_rule | all ]
-</pre>
-<p>Note that some rules are mandatory and cannot be turned off.</p>
-<a name="N1038F"></a><a name="ImplicitSplitInserter"></a>
-<h3 class="h4">ImplicitSplitInserter</h3>
-<p>Status: Mandatory</p>
-<p>
-<a href="piglatin_ref2.html#SPLIT">SPLIT</a> is the only operator that models multiple outputs in Pig.
-To ease the process of building logical plans, all operators are allowed to have multiple outputs. As part of the
-optimization, all non-split operators that have multiple outputs are altered to have a SPLIT operator as the output
-and the outputs of the operator are then made outputs of the SPLIT operator. An example will illustrate the point.
-Here, a split will be inserted after the LOAD and the split outputs will be connected to the FILTER (b) and the COGROUP (c).
-</p>
-<pre class="code">
-A = LOAD 'input';
-B = FILTER A BY $1 == 1;
-C = COGROUP A BY $0, B BY $0;
-</pre>
-<a name="N103A4"></a><a name="TypeCastInserter"></a>
-<h3 class="h4">TypeCastInserter</h3>
-<p>Status: Mandatory</p>
-<p>
-If you specify a <a href="piglatin_ref2.html#Schemas">schema</a> with the
-<a href="piglatin_ref2.html#LOAD">LOAD</a> statement, the optimizer will perform a pre-fix projection of the columns
-and <a href="piglatin_ref2.html#Cast+Operators">cast</a> the columns to the appropriate types. An example will illustrate the point.
-The LOAD statement (a) has a schema associated with it. The optimizer will insert a FOREACH operator that will project columns 0, 1 and 2
-and also cast them to chararray, int and float respectively.
-</p>
-<pre class="code">
-A = LOAD 'input' AS (name: chararray, age: int, gpa: float);
-B = FILER A BY $1 == 1;
-C = GROUP A By $0;
-</pre>
-<a name="N103C1"></a><a name="StreamOptimizer"></a>
-<h3 class="h4">StreamOptimizer</h3>
-<p>
-Optimize when <a href="piglatin_ref2.html#LOAD">LOAD</a> precedes <a href="piglatin_ref2.html#STREAM">STREAM</a>
-and the loader class is the same as the serializer for the stream. Similarly, optimize when STREAM is followed by
-<a href="piglatin_ref2.html#STORE">STORE</a> and the deserializer class is same as the storage class.
-For both of these cases the optimization is to replace the loader/serializer with BinaryStorage which just moves bytes
-around and to replace the storer/deserializer with BinaryStorage.
-</p>
-<a name="N103D7"></a><a name="OpLimitOptimizer"></a>
-<h3 class="h4">OpLimitOptimizer</h3>
-<p>
-The objective of this rule is to push the <a href="piglatin_ref2.html#LIMIT">LIMIT</a> operator up the data flow graph
-(or down the tree for database folks). In addition, for top-k (ORDER BY followed by a LIMIT) the LIMIT is pushed into the ORDER BY.
-</p>
-<pre class="code">
-A = LOAD 'input';
-B = ORDER A BY $0;
-C = LIMIT B 10;
-</pre>
-<a name="N103E9"></a><a name="PushUpFilters"></a>
-<h3 class="h4">PushUpFilters</h3>
-<p>
-The objective of this rule is to push the <a href="piglatin_ref2.html#FILTER">FILTER</a> operators up the data flow graph.
-As a result, the number of records that flow through the pipeline is reduced.
-</p>
-<pre class="code">
-A = LOAD 'input';
-B = GROUP A BY $0;
-C = FILTER B BY $0 &lt; 10;
-</pre>
-<a name="N103FB"></a><a name="PushDownExplodes"></a>
-<h3 class="h4">PushDownExplodes</h3>
-<p>
-The objective of this rule is to reduce the number of records that flow through the pipeline by moving
-<a href="piglatin_ref2.html#FOREACH">FOREACH</a> operators with a
-<a href="piglatin_ref2.html#Flatten+Operator">FLATTEN</a> down the data flow graph.
-In the example shown below, it would be more efficient to move the foreach after the join to reduce the cost of the join operation.
-</p>
-<pre class="code">
-A = LOAD 'input' AS (a, b, c);
-B = LOAD 'input2' AS (x, y, z);
-C = FOREACH A GENERATE FLATTEN($0), B, C;
-D = JOIN C BY $1, B BY $1;
-</pre>
-</div> <!-- END OPTIMIZATION RULES -->
- <!-- MEMORY MANAGEMENT -->
-<a name="N10416"></a><a name="Memory+Management"></a>
-<h2 class="h3">Memory Management</h2>
-<div class="section">
-<p>Pig allocates a fix amount of memory to store bags and spills to disk as soon as the memory limit is reached. This is very similar to how Hadoop decides when to spill data accumulated by the combiner. </p>
-<p>The amount of memory allocated to bags is determined by pig.cachedbag.memusage; the default is set to 10% of available memory. Note that this memory is shared across all large bags used by the application.</p>
-</div>
-<!-- END MEMORY MANAGEMENT  -->
-  <!-- ZEBRA INTEGRATION -->
-<a name="N10427"></a><a name="Zebra+Integration"></a>
-<h2 class="h3">Zebra Integration</h2>
-<div class="section">
-<p>For information about how to integrate Zebra with your Pig scripts, see <a href="zebra_pig.html">Zebra and Pig</a>.</p>
-</div>
-<!-- END ZEBRA INTEGRATION  -->
-</div>
-<!--+
-    |end content
-    +-->
-<div class="clearboth">&nbsp;</div>
-</div>
-<div id="footer">
-<!--+
-    |start bottomstrip
-    +-->
-<div class="lastmodified">
-<script type="text/javascript"><!--
-document.write("Last Published: " + document.lastModified);
-//  --></script>
-</div>
-<div class="copyright">
-        Copyright &copy;
-         2007-2010 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
-</div>
-<!--+
-    |end bottomstrip
-    +-->
-</div>
-</body>
-</html>