RubyGems - wukong - Versions diffs - 3.0.0.pre → 3.0.0.pre2 - Mend

wukong 3.0.0.pre → 3.0.0.pre2

Files changed (476) hide show

data/.gitignore +46 -33
data/.gitmodules +3 -0
data/.rspec +1 -1
data/.travis.yml +8 -1
data/.yardopts +0 -13
data/Guardfile +4 -6
data/{LICENSE.textile → LICENSE.md} +43 -55
data/README-old.md +422 -0
data/README.md +279 -418
data/Rakefile +21 -5
data/TODO.md +6 -6
data/bin/wu-clean-encoding +31 -0
data/bin/wu-lign +2 -2
data/bin/wu-local +69 -0
data/bin/wu-server +70 -0
data/examples/Gemfile +38 -0
data/examples/README.md +9 -0
data/examples/dataflow/apache_log_line.rb +64 -25
data/examples/dataflow/fibonacci_series.rb +101 -0
data/examples/dataflow/parse_apache_logs.rb +37 -7
data/examples/{dataflow.rb → dataflow/scraper_macro_flow.rb} +0 -0
data/examples/dataflow/simple.rb +4 -4
data/examples/geo.rb +4 -0
data/examples/geo/geo_grids.numbers +0 -0
data/examples/geo/geolocated.rb +331 -0
data/examples/geo/quadtile.rb +69 -0
data/examples/geo/spec/geolocated_spec.rb +247 -0
data/examples/geo/tile_fetcher.rb +77 -0
data/examples/graph/minimum_spanning_tree.rb +61 -61
data/examples/jabberwocky.txt +36 -0
data/examples/models/wikipedia.rb +20 -0
data/examples/munging/Gemfile +8 -0
data/examples/munging/airline_flights/airline.rb +57 -0
data/examples/munging/airline_flights/airline_flights.rake +83 -0
data/{lib/wukong/settings.rb → examples/munging/airline_flights/airplane.rb} +0 -0
data/examples/munging/airline_flights/airport.rb +211 -0
data/examples/munging/airline_flights/airport_id_unification.rb +129 -0
data/examples/munging/airline_flights/airport_ok_chars.rb +4 -0
data/examples/munging/airline_flights/flight.rb +156 -0
data/examples/munging/airline_flights/models.rb +4 -0
data/examples/munging/airline_flights/parse.rb +26 -0
data/examples/munging/airline_flights/reconcile_airports.rb +142 -0
data/examples/munging/airline_flights/route.rb +35 -0
data/examples/munging/airline_flights/tasks.rake +83 -0
data/examples/munging/airline_flights/timezone_fixup.rb +62 -0
data/examples/munging/airline_flights/topcities.rb +167 -0
data/examples/munging/airports/40_wbans.txt +40 -0
data/examples/munging/airports/filter_weather_reports.rb +37 -0
data/examples/munging/airports/join.pig +31 -0
data/examples/munging/airports/to_tsv.rb +33 -0
data/examples/munging/airports/usa_wbans.pig +19 -0
data/examples/munging/airports/usa_wbans.txt +2157 -0
data/examples/munging/airports/wbans.pig +19 -0
data/examples/munging/airports/wbans.txt +2310 -0
data/examples/munging/geo/geo_json.rb +54 -0
data/examples/munging/geo/geo_models.rb +69 -0
data/examples/munging/geo/geonames_models.rb +78 -0
data/examples/munging/geo/iso_codes.rb +172 -0
data/examples/munging/geo/reconcile_countries.rb +124 -0
data/examples/munging/geo/tasks.rake +71 -0
data/examples/munging/rake_helper.rb +62 -0
data/examples/munging/weather/.gitignore +1 -0
data/examples/munging/weather/Gemfile +4 -0
data/examples/munging/weather/Rakefile +28 -0
data/examples/munging/weather/extract_ish.rb +13 -0
data/examples/munging/weather/models/weather.rb +119 -0
data/examples/munging/weather/utils/noaa_downloader.rb +46 -0
data/examples/munging/wikipedia/README.md +34 -0
data/examples/munging/wikipedia/Rakefile +193 -0
data/examples/munging/wikipedia/articles/extract_articles-parsed.rb +79 -0
data/examples/munging/wikipedia/articles/extract_articles-templated.rb +136 -0
data/examples/munging/wikipedia/articles/textualize_articles.rb +54 -0
data/examples/munging/wikipedia/articles/verify_structure.rb +43 -0
data/examples/munging/wikipedia/articles/wp2txt-LICENSE.txt +22 -0
data/examples/munging/wikipedia/articles/wp2txt_article.rb +259 -0
data/examples/munging/wikipedia/articles/wp2txt_utils.rb +452 -0
data/examples/munging/wikipedia/dbpedia/dbpedia_common.rb +4 -0
data/examples/munging/wikipedia/dbpedia/dbpedia_extract_geocoordinates.rb +78 -0
data/examples/munging/wikipedia/dbpedia/extract_links.rb +193 -0
data/examples/munging/wikipedia/dbpedia/sameas_extractor.rb +20 -0
data/examples/munging/wikipedia/n1_subuniverse/n1_nodes.pig +18 -0
data/examples/munging/wikipedia/page_metadata/extract_page_metadata.rb +21 -0
data/examples/munging/wikipedia/page_metadata/extract_page_metadata.rb.old +27 -0
data/examples/munging/wikipedia/pagelinks/augment_pagelinks.pig +29 -0
data/examples/munging/wikipedia/pagelinks/extract_pagelinks.rb +14 -0
data/examples/munging/wikipedia/pagelinks/extract_pagelinks.rb.old +25 -0
data/examples/munging/wikipedia/pagelinks/undirect_pagelinks.pig +29 -0
data/examples/munging/wikipedia/pageviews/augment_pageviews.pig +32 -0
data/examples/munging/wikipedia/pageviews/extract_pageviews.rb +85 -0
data/examples/munging/wikipedia/pig_style_guide.md +25 -0
data/examples/munging/wikipedia/redirects/redirects_page_metadata.pig +19 -0
data/examples/munging/wikipedia/subuniverse/sub_articles.pig +23 -0
data/examples/munging/wikipedia/subuniverse/sub_page_metadata.pig +24 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_from.pig +22 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_into.pig +22 -0
data/examples/munging/wikipedia/subuniverse/sub_pagelinks_within.pig +26 -0
data/examples/munging/wikipedia/subuniverse/sub_pageviews.pig +29 -0
data/examples/munging/wikipedia/subuniverse/sub_undirected_pagelinks_within.pig +24 -0
data/examples/munging/wikipedia/utils/get_namespaces.rb +86 -0
data/examples/munging/wikipedia/utils/munging_utils.rb +68 -0
data/examples/munging/wikipedia/utils/namespaces.json +1 -0
data/examples/rake_helper.rb +85 -0
data/examples/server_logs/geo_ip_mapping/munge_geolite.rb +82 -0
data/examples/server_logs/logline.rb +95 -0
data/examples/server_logs/models.rb +66 -0
data/examples/server_logs/page_counts.pig +48 -0
data/examples/server_logs/server_logs-01-parse-script.rb +13 -0
data/examples/server_logs/server_logs-02-histograms-full.rb +33 -0
data/examples/server_logs/server_logs-02-histograms-mapper.rb +14 -0
data/{old/examples/server_logs/breadcrumbs.rb → examples/server_logs/server_logs-03-breadcrumbs-full.rb} +26 -30
data/examples/server_logs/server_logs-04-page_page_edges-full.rb +40 -0
data/examples/string_reverser.rb +26 -0
data/examples/text/pig_latin.rb +2 -2
data/examples/text/regional_flavor/README.md +14 -0
data/examples/text/regional_flavor/article_wordbags.pig +39 -0
data/examples/text/regional_flavor/j01-article_wordbags.rb +4 -0
data/examples/text/regional_flavor/simple_pig_script.pig +27 -0
data/examples/word_count/accumulator.rb +26 -0
data/examples/word_count/tokenizer.rb +13 -0
data/examples/word_count/word_count.rb +6 -0
data/examples/workflow/cherry_pie.dot +97 -0
data/examples/workflow/cherry_pie.png +0 -0
data/examples/workflow/cherry_pie.rb +61 -26
data/lib/hanuman.rb +34 -7
data/lib/hanuman/graph.rb +55 -31
data/lib/hanuman/graphvizzer.rb +199 -178
data/lib/hanuman/graphvizzer/gv_models.rb +161 -0
data/lib/hanuman/graphvizzer/gv_presenter.rb +97 -0
data/lib/hanuman/link.rb +35 -0
data/lib/hanuman/registry.rb +46 -0
data/lib/hanuman/stage.rb +76 -32
data/lib/wukong.rb +23 -24
data/lib/wukong/boot.rb +87 -0
data/lib/wukong/configuration.rb +8 -0
data/lib/wukong/dataflow.rb +45 -78
data/lib/wukong/driver.rb +99 -0
data/lib/wukong/emitter.rb +22 -0
data/lib/wukong/model/faker.rb +24 -24
data/lib/wukong/model/flatpack_parser/flat.rb +60 -0
data/lib/wukong/model/flatpack_parser/flatpack.rb +4 -0
data/lib/wukong/model/flatpack_parser/lang.rb +46 -0
data/lib/wukong/model/flatpack_parser/parser.rb +55 -0
data/lib/wukong/model/flatpack_parser/tokens.rb +130 -0
data/lib/wukong/processor.rb +60 -114
data/lib/wukong/spec_helpers.rb +81 -0
data/lib/wukong/spec_helpers/integration_driver.rb +144 -0
data/lib/wukong/spec_helpers/integration_driver_matchers.rb +219 -0
data/lib/wukong/spec_helpers/processor_helpers.rb +95 -0
data/lib/wukong/spec_helpers/processor_methods.rb +108 -0
data/lib/wukong/spec_helpers/shared_examples.rb +15 -0
data/lib/wukong/spec_helpers/spec_driver.rb +28 -0
data/lib/wukong/spec_helpers/spec_driver_matchers.rb +195 -0
data/lib/wukong/version.rb +2 -1
data/lib/wukong/widget/filters.rb +311 -0
data/lib/wukong/widget/processors.rb +156 -0
data/lib/wukong/widget/reducers.rb +7 -0
data/lib/wukong/widget/reducers/accumulator.rb +73 -0
data/lib/wukong/widget/reducers/bin.rb +318 -0
data/lib/wukong/widget/reducers/count.rb +61 -0
data/lib/wukong/widget/reducers/group.rb +85 -0
data/lib/wukong/widget/reducers/group_concat.rb +70 -0
data/lib/wukong/widget/reducers/moments.rb +72 -0
data/lib/wukong/widget/reducers/sort.rb +130 -0
data/lib/wukong/widget/serializers.rb +287 -0
data/lib/wukong/widget/sink.rb +10 -52
data/lib/wukong/widget/source.rb +7 -113
data/lib/wukong/widget/utils.rb +46 -0
data/lib/wukong/widgets.rb +6 -0
data/spec/examples/dataflow/fibonacci_series_spec.rb +18 -0
data/spec/examples/dataflow/parsing_spec.rb +12 -11
data/spec/examples/dataflow/simple_spec.rb +32 -6
data/spec/examples/dataflow/telegram_spec.rb +36 -36
data/spec/examples/graph/minimum_spanning_tree_spec.rb +30 -31
data/spec/examples/munging/airline_flights/identifiers_spec.rb +16 -0
data/spec/examples/munging/airline_flights_spec.rb +202 -0
data/spec/examples/text/pig_latin_spec.rb +13 -16
data/spec/examples/workflow/cherry_pie_spec.rb +34 -4
data/spec/hanuman/graph_spec.rb +27 -2
data/spec/hanuman/hanuman_spec.rb +10 -0
data/spec/hanuman/registry_spec.rb +123 -0
data/spec/hanuman/stage_spec.rb +61 -7
data/spec/spec_helper.rb +29 -19
data/spec/support/hanuman_test_helpers.rb +14 -12
data/spec/support/shared_context_for_reducers.rb +37 -0
data/spec/support/shared_examples_for_builders.rb +101 -0
data/spec/support/shared_examples_for_shortcuts.rb +57 -0
data/spec/support/wukong_test_helpers.rb +37 -11
data/spec/wukong/dataflow_spec.rb +77 -55
data/spec/wukong/local_runner_spec.rb +24 -24
data/spec/wukong/model/faker_spec.rb +132 -131
data/spec/wukong/runner_spec.rb +8 -8
data/spec/wukong/widget/filters_spec.rb +61 -0
data/spec/wukong/widget/processors_spec.rb +126 -0
data/spec/wukong/widget/reducers/bin_spec.rb +92 -0
data/spec/wukong/widget/reducers/count_spec.rb +11 -0
data/spec/wukong/widget/reducers/group_spec.rb +20 -0
data/spec/wukong/widget/reducers/moments_spec.rb +36 -0
data/spec/wukong/widget/reducers/sort_spec.rb +26 -0
data/spec/wukong/widget/serializers_spec.rb +92 -0
data/spec/wukong/widget/sink_spec.rb +15 -15
data/spec/wukong/widget/source_spec.rb +65 -41
data/spec/wukong/wukong_spec.rb +10 -0
data/wukong.gemspec +17 -10
metadata +359 -335
data/.document +0 -5
data/VERSION +0 -1
data/bin/hdp-bin +0 -44
data/bin/hdp-bzip +0 -23
data/bin/hdp-cat +0 -3
data/bin/hdp-catd +0 -3
data/bin/hdp-cp +0 -3
data/bin/hdp-du +0 -86
data/bin/hdp-get +0 -3
data/bin/hdp-kill +0 -3
data/bin/hdp-kill-task +0 -3
data/bin/hdp-ls +0 -11
data/bin/hdp-mkdir +0 -2
data/bin/hdp-mkdirp +0 -12
data/bin/hdp-mv +0 -3
data/bin/hdp-parts_to_keys.rb +0 -77
data/bin/hdp-ps +0 -3
data/bin/hdp-put +0 -3
data/bin/hdp-rm +0 -32
data/bin/hdp-sort +0 -40
data/bin/hdp-stream +0 -40
data/bin/hdp-stream-flat +0 -22
data/bin/hdp-stream2 +0 -39
data/bin/hdp-sync +0 -17
data/bin/hdp-wc +0 -67
data/bin/wu-flow +0 -10
data/bin/wu-map +0 -17
data/bin/wu-red +0 -17
data/bin/wukong +0 -17
data/data/CREDITS.md +0 -355
data/data/graph/airfares.tsv +0 -2174
data/data/text/gift_of_the_magi.txt +0 -225
data/data/text/jabberwocky.txt +0 -36
data/data/text/rectification_of_names.txt +0 -33
data/data/twitter/a_atsigns_b.tsv +0 -64
data/data/twitter/a_follows_b.tsv +0 -53
data/data/twitter/tweet.tsv +0 -167
data/data/twitter/twitter_user.tsv +0 -55
data/data/wikipedia/dbpedia-sentences.tsv +0 -1000
data/docpages/INSTALL.textile +0 -92
data/docpages/LICENSE.textile +0 -107
data/docpages/README-elastic_map_reduce.textile +0 -377
data/docpages/README-performance.textile +0 -90
data/docpages/README-wulign.textile +0 -65
data/docpages/UsingWukong-part1-get_ready.textile +0 -17
data/docpages/UsingWukong-part2-ThinkingBigData.textile +0 -75
data/docpages/UsingWukong-part3-parsing.textile +0 -138
data/docpages/_config.yml +0 -39
data/docpages/avro/avro_notes.textile +0 -56
data/docpages/avro/performance.textile +0 -36
data/docpages/avro/tethering.textile +0 -19
data/docpages/bigdata-tips.textile +0 -143
data/docpages/code/api_response_example.txt +0 -20
data/docpages/code/parser_skeleton.rb +0 -38
data/docpages/diagrams/MapReduceDiagram.graffle +0 -0
data/docpages/favicon.ico +0 -0
data/docpages/gem.css +0 -16
data/docpages/hadoop-tips.textile +0 -83
data/docpages/index.textile +0 -92
data/docpages/intro.textile +0 -8
data/docpages/moreinfo.textile +0 -174
data/docpages/news.html +0 -24
data/docpages/pig/PigLatinExpressionsList.txt +0 -122
data/docpages/pig/PigLatinReferenceManual.txt +0 -1640
data/docpages/pig/commandline_params.txt +0 -26
data/docpages/pig/cookbook.html +0 -481
data/docpages/pig/images/hadoop-logo.jpg +0 -0
data/docpages/pig/images/instruction_arrow.png +0 -0
data/docpages/pig/images/pig-logo.gif +0 -0
data/docpages/pig/piglatin_ref1.html +0 -1103
data/docpages/pig/piglatin_ref2.html +0 -14340
data/docpages/pig/setup.html +0 -505
data/docpages/pig/skin/basic.css +0 -166
data/docpages/pig/skin/breadcrumbs.js +0 -237
data/docpages/pig/skin/fontsize.js +0 -166
data/docpages/pig/skin/getBlank.js +0 -40
data/docpages/pig/skin/getMenu.js +0 -45
data/docpages/pig/skin/images/chapter.gif +0 -0
data/docpages/pig/skin/images/chapter_open.gif +0 -0
data/docpages/pig/skin/images/current.gif +0 -0
data/docpages/pig/skin/images/external-link.gif +0 -0
data/docpages/pig/skin/images/header_white_line.gif +0 -0
data/docpages/pig/skin/images/page.gif +0 -0
data/docpages/pig/skin/images/pdfdoc.gif +0 -0
data/docpages/pig/skin/images/rc-b-l-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-b-r-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-b-r-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2searchbox-3searchbox.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-l-5-1header-2tab-unselected-3tab-unselected.png +0 -0
data/docpages/pig/skin/images/rc-t-r-15-1body-2menu-3menu.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2searchbox-3searchbox.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2tab-selected-3tab-selected.png +0 -0
data/docpages/pig/skin/images/rc-t-r-5-1header-2tab-unselected-3tab-unselected.png +0 -0
data/docpages/pig/skin/print.css +0 -54
data/docpages/pig/skin/profile.css +0 -181
data/docpages/pig/skin/screen.css +0 -587
data/docpages/pig/tutorial.html +0 -1059
data/docpages/pig/udf.html +0 -1509
data/docpages/tutorial.textile +0 -283
data/docpages/usage.textile +0 -195
data/docpages/wutils.textile +0 -263
data/examples/dataflow/complex.rb +0 -11
data/examples/dataflow/donuts.rb +0 -13
data/examples/tiny_count/jabberwocky_output.tsv +0 -92
data/examples/word_count.rb +0 -48
data/examples/workflow/fiddle.rb +0 -24
data/lib/away/escapement.rb +0 -129
data/lib/away/exe.rb +0 -11
data/lib/away/experimental.rb +0 -5
data/lib/away/from_file.rb +0 -52
data/lib/away/job.rb +0 -56
data/lib/away/job/rake_compat.rb +0 -17
data/lib/away/registry.rb +0 -79
data/lib/away/runner.rb +0 -276
data/lib/away/runner/execute.rb +0 -121
data/lib/away/script.rb +0 -161
data/lib/away/script/hadoop_command.rb +0 -240
data/lib/away/source/file_list_source.rb +0 -15
data/lib/away/source/looper.rb +0 -18
data/lib/away/task.rb +0 -219
data/lib/hanuman/action.rb +0 -21
data/lib/hanuman/chain.rb +0 -4
data/lib/hanuman/graphviz.rb +0 -74
data/lib/hanuman/resource.rb +0 -6
data/lib/hanuman/slot.rb +0 -87
data/lib/hanuman/slottable.rb +0 -220
data/lib/wukong/bad_record.rb +0 -15
data/lib/wukong/event.rb +0 -44
data/lib/wukong/local_runner.rb +0 -55
data/lib/wukong/mapred.rb +0 -3
data/lib/wukong/universe.rb +0 -48
data/lib/wukong/widget/filter.rb +0 -81
data/lib/wukong/widget/gibberish.rb +0 -123
data/lib/wukong/widget/monitor.rb +0 -26
data/lib/wukong/widget/reducer.rb +0 -66
data/lib/wukong/widget/stringifier.rb +0 -50
data/lib/wukong/workflow.rb +0 -22
data/lib/wukong/workflow/command.rb +0 -42
data/old/config/emr-example.yaml +0 -48
data/old/examples/README.txt +0 -17
data/old/examples/contrib/jeans/README.markdown +0 -165
data/old/examples/contrib/jeans/data/normalized_sizes +0 -3
data/old/examples/contrib/jeans/data/orders.tsv +0 -1302
data/old/examples/contrib/jeans/data/sizes +0 -3
data/old/examples/contrib/jeans/normalize.rb +0 -20
data/old/examples/contrib/jeans/sizes.rb +0 -55
data/old/examples/corpus/bnc_word_freq.rb +0 -44
data/old/examples/corpus/bucket_counter.rb +0 -47
data/old/examples/corpus/dbpedia_abstract_to_sentences.rb +0 -86
data/old/examples/corpus/sentence_bigrams.rb +0 -53
data/old/examples/corpus/sentence_coocurrence.rb +0 -66
data/old/examples/corpus/stopwords.rb +0 -138
data/old/examples/corpus/words_to_bigrams.rb +0 -53
data/old/examples/emr/README.textile +0 -110
data/old/examples/emr/dot_wukong_dir/credentials.json +0 -7
data/old/examples/emr/dot_wukong_dir/emr.yaml +0 -69
data/old/examples/emr/dot_wukong_dir/emr_bootstrap.sh +0 -33
data/old/examples/emr/elastic_mapreduce_example.rb +0 -28
data/old/examples/network_graph/adjacency_list.rb +0 -74
data/old/examples/network_graph/breadth_first_search.rb +0 -72
data/old/examples/network_graph/gen_2paths.rb +0 -68
data/old/examples/network_graph/gen_multi_edge.rb +0 -112
data/old/examples/network_graph/gen_symmetric_links.rb +0 -64
data/old/examples/pagerank/README.textile +0 -6
data/old/examples/pagerank/gen_initial_pagerank_graph.pig +0 -57
data/old/examples/pagerank/pagerank.rb +0 -72
data/old/examples/pagerank/pagerank_initialize.rb +0 -42
data/old/examples/pagerank/run_pagerank.sh +0 -21
data/old/examples/sample_records.rb +0 -33
data/old/examples/server_logs/apache_log_parser.rb +0 -15
data/old/examples/server_logs/nook.rb +0 -48
data/old/examples/server_logs/nook/faraday_dummy_adapter.rb +0 -94
data/old/examples/server_logs/user_agent.rb +0 -40
data/old/examples/simple_word_count.rb +0 -82
data/old/examples/size.rb +0 -61
data/old/examples/stats/avg_value_frequency.rb +0 -86
data/old/examples/stats/binning_percentile_estimator.rb +0 -140
data/old/examples/stats/data/avg_value_frequency.tsv +0 -3
data/old/examples/stats/rank_and_bin.rb +0 -173
data/old/examples/stupidly_simple_filter.rb +0 -40
data/old/examples/word_count.rb +0 -75
data/old/graph/graphviz_builder.rb +0 -580
data/old/graph_easy/Attributes.pm +0 -4181
data/old/graph_easy/Graphviz.pm +0 -2232
data/old/wukong.rb +0 -18
data/old/wukong/and_pig.rb +0 -38
data/old/wukong/bad_record.rb +0 -18
data/old/wukong/datatypes.rb +0 -24
data/old/wukong/datatypes/enum.rb +0 -127
data/old/wukong/datatypes/fake_types.rb +0 -17
data/old/wukong/decorator.rb +0 -28
data/old/wukong/encoding/asciize.rb +0 -108
data/old/wukong/extensions.rb +0 -16
data/old/wukong/extensions/array.rb +0 -18
data/old/wukong/extensions/blank.rb +0 -93
data/old/wukong/extensions/class.rb +0 -189
data/old/wukong/extensions/date_time.rb +0 -53
data/old/wukong/extensions/emittable.rb +0 -69
data/old/wukong/extensions/enumerable.rb +0 -79
data/old/wukong/extensions/hash.rb +0 -167
data/old/wukong/extensions/hash_keys.rb +0 -16
data/old/wukong/extensions/hash_like.rb +0 -150
data/old/wukong/extensions/hashlike_class.rb +0 -47
data/old/wukong/extensions/module.rb +0 -2
data/old/wukong/extensions/pathname.rb +0 -27
data/old/wukong/extensions/string.rb +0 -65
data/old/wukong/extensions/struct.rb +0 -17
data/old/wukong/extensions/symbol.rb +0 -11
data/old/wukong/filename_pattern.rb +0 -74
data/old/wukong/helper.rb +0 -7
data/old/wukong/helper/stopwords.rb +0 -195
data/old/wukong/helper/tokenize.rb +0 -35
data/old/wukong/logger.rb +0 -38
data/old/wukong/periodic_monitor.rb +0 -72
data/old/wukong/schema.rb +0 -269
data/old/wukong/script.rb +0 -286
data/old/wukong/script/avro_command.rb +0 -5
data/old/wukong/script/cassandra_loader_script.rb +0 -40
data/old/wukong/script/emr_command.rb +0 -168
data/old/wukong/script/hadoop_command.rb +0 -237
data/old/wukong/script/local_command.rb +0 -41
data/old/wukong/store.rb +0 -10
data/old/wukong/store/base.rb +0 -27
data/old/wukong/store/cassandra.rb +0 -10
data/old/wukong/store/cassandra/streaming.rb +0 -75
data/old/wukong/store/cassandra/struct_loader.rb +0 -21
data/old/wukong/store/cassandra_model.rb +0 -91
data/old/wukong/store/chh_chunked_flat_file_store.rb +0 -37
data/old/wukong/store/chunked_flat_file_store.rb +0 -48
data/old/wukong/store/conditional_store.rb +0 -57
data/old/wukong/store/factory.rb +0 -8
data/old/wukong/store/flat_file_store.rb +0 -89
data/old/wukong/store/key_store.rb +0 -51
data/old/wukong/store/null_store.rb +0 -15
data/old/wukong/store/read_thru_store.rb +0 -22
data/old/wukong/store/tokyo_tdb_key_store.rb +0 -33
data/old/wukong/store/tyrant_rdb_key_store.rb +0 -57
data/old/wukong/store/tyrant_tdb_key_store.rb +0 -20
data/old/wukong/streamer.rb +0 -30
data/old/wukong/streamer/accumulating_reducer.rb +0 -83
data/old/wukong/streamer/base.rb +0 -126
data/old/wukong/streamer/counting_reducer.rb +0 -25
data/old/wukong/streamer/filter.rb +0 -20
data/old/wukong/streamer/instance_streamer.rb +0 -15
data/old/wukong/streamer/json_streamer.rb +0 -21
data/old/wukong/streamer/line_streamer.rb +0 -12
data/old/wukong/streamer/list_reducer.rb +0 -31
data/old/wukong/streamer/rank_and_bin_reducer.rb +0 -145
data/old/wukong/streamer/record_streamer.rb +0 -14
data/old/wukong/streamer/reducer.rb +0 -11
data/old/wukong/streamer/set_reducer.rb +0 -14
data/old/wukong/streamer/struct_streamer.rb +0 -48
data/old/wukong/streamer/summing_reducer.rb +0 -29
data/old/wukong/streamer/uniq_by_last_reducer.rb +0 -51
data/old/wukong/typed_struct.rb +0 -12
data/spec/away/encoding_spec.rb +0 -32
data/spec/away/exe_spec.rb +0 -20
data/spec/away/flow_spec.rb +0 -82
data/spec/away/graph_spec.rb +0 -6
data/spec/away/job_spec.rb +0 -15
data/spec/away/rake_compat_spec.rb +0 -9
data/spec/away/script_spec.rb +0 -81
data/spec/hanuman/graphviz_spec.rb +0 -29
data/spec/hanuman/slot_spec.rb +0 -2
data/spec/support/examples_helper.rb +0 -10
data/spec/support/streamer_test_helpers.rb +0 -6
data/spec/support/wukong_widget_helpers.rb +0 -66
data/spec/wukong/processor_spec.rb +0 -109
data/spec/wukong/widget/filter_spec.rb +0 -99
data/spec/wukong/widget/stringifier_spec.rb +0 -51
data/spec/wukong/workflow/command_spec.rb +0 -5

data/docpages/pig/commandline_params.txt DELETED

@@ -1,26 +0,0 @@
-Apache Pig version 0.7.0 (r941408)
-compiled May 05 2010, 11:15:55
-USAGE: Pig [options] [-] : Run interactively in grunt shell.
-       Pig [options] -e[xecute] cmd [cmd ...] : Run cmd(s).
-       Pig [options] [-f[ile]] file : Run cmds found in file.
-  options include:
-    -4, -log4jconf log4j configuration file, overrides log conf
-    -b, -brief brief logging (no timestamps)
-    -c, -cluster clustername, kryptonite is default
-    -d, -debug debug level, INFO is default
-    -e, -execute commands to execute (within quotes)
-    -f, -file path to the script to execute
-    -h, -help display this message
-    -i, -version display version information
-    -j, -jar jarfile load jarfile
-    -l, -logfile path to client side log file; current working directory is default
-    -m, -param_file path to the parameter file
-    -p, -param key value pair of the form param=val
-    -r, -dryrun
-    -t, -optimizer_off optimizer rule name, turn optimizer off for this rule; use all to turn all rules off, optimizer is turned on by default
-    -v, -verbose print all error messages to screen
-    -w, -warning turn warning on; also turns warning aggregation off
-    -x, -exectype local|mapreduce, mapreduce is default
-    -F, -stop_on_failure aborts execution on the first failed job; off by default
-    -M, -no_multiquery turn multiquery optimization off; Multiquery is on by default

data/docpages/pig/cookbook.html DELETED

@@ -1,481 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
-<html>
-<head>
-<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
-<meta content="Apache Forrest" name="Generator">
-<meta name="Forrest-version" content="0.8">
-<meta name="Forrest-skin-name" content="pelt">
-<title>Pig Cookbook</title>
-<link type="text/css" href="skin/basic.css" rel="stylesheet">
-<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
-<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
-<link type="text/css" href="skin/profile.css" rel="stylesheet">
-<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
-<link rel="shortcut icon" href="">
-</head>
-<body onload="init()">
-<script type="text/javascript">ndeSetTextSize();</script>
-<div id="top">
-<!--+
-    |breadtrail
-    +-->
-<div class="breadtrail">
-<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/pig/">Pig</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
-</div>
-<!--+
-    |header
-    +-->
-<div class="header">
-<!--+
-    |start group logo
-    +-->
-<div class="grouplogo">
-<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
-</div>
-<!--+
-    |end group logo
-    +-->
-<!--+
-    |start Project Logo
-    +-->
-<div class="projectlogo">
-<a href="http://hadoop.apache.org/pig/"><img class="logoImage" alt="Pig" src="images/pig-logo.gif" title="A platform for analyzing large datasets."></a>
-</div>
-<!--+
-    |end Project Logo
-    +-->
-<!--+
-    |start Search
-    +-->
-<div class="searchbox">
-<form action="http://www.google.com/search" method="get" class="roundtopsmall">
-<input value="" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp;
-                    <input name="Search" value="Search" type="submit">
-</form>
-</div>
-<!--+
-    |end search
-    +-->
-<!--+
-    |start Tabs
-    +-->
-<ul id="tabs">
-<li>
-<a class="unselected" href="http://hadoop.apache.org/pig/">Project</a>
-</li>
-<li>
-<a class="unselected" href="http://wiki.apache.org/pig/">Wiki</a>
-</li>
-<li class="current">
-<a class="selected" href="index.html">Pig 0.7.0 Documentation</a>
-</li>
-</ul>
-<!--+
-    |end Tabs
-    +-->
-</div>
-</div>
-<div id="main">
-<div id="publishedStrip">
-<!--+
-    |start Subtabs
-    +-->
-<div id="level2tabs"></div>
-<!--+
-    |end Endtabs
-    +-->
-<script type="text/javascript"><!--
-document.write("Last Published: " + document.lastModified);
-//  --></script>
-</div>
-<!--+
-    |breadtrail
-    +-->
-<div class="breadtrail">
-             &nbsp;
-           </div>
-<!--+
-    |start Menu, mainarea
-    +-->
-<!--+
-    |start Menu
-    +-->
-<div id="menu">
-<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Pig</div>
-<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
-<div class="menuitem">
-<a href="index.html">Overview</a>
-</div>
-<div class="menuitem">
-<a href="setup.html">Setup</a>
-</div>
-<div class="menuitem">
-<a href="tutorial.html">Tutorial</a>
-</div>
-<div class="menuitem">
-<a href="piglatin_ref1.html">Pig Latin 1</a>
-</div>
-<div class="menuitem">
-<a href="piglatin_ref2.html">Pig Latin 2</a>
-</div>
-<div class="menupage">
-<div class="menupagetitle">Cookbook</div>
-</div>
-<div class="menuitem">
-<a href="udf.html">UDFs</a>
-</div>
-</div>
-<div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Zebra</div>
-<div id="menu_1.2" class="menuitemgroup">
-<div class="menuitem">
-<a href="zebra_overview.html">Zebra Overview </a>
-</div>
-<div class="menuitem">
-<a href="zebra_users.html">Zebra Users </a>
-</div>
-<div class="menuitem">
-<a href="zebra_reference.html">Zebra Reference </a>
-</div>
-<div class="menuitem">
-<a href="zebra_mapreduce.html">Zebra MapReduce </a>
-</div>
-<div class="menuitem">
-<a href="zebra_pig.html">Zebra Pig </a>
-</div>
-<div class="menuitem">
-<a href="zebra_stream.html">Zebra Streaming </a>
-</div>
-</div>
-<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">Miscellaneous</div>
-<div id="menu_1.3" class="menuitemgroup">
-<div class="menuitem">
-<a href="api/">API Docs</a>
-</div>
-<div class="menuitem">
-<a href="http://wiki.apache.org/pig/">Wiki</a>
-</div>
-<div class="menuitem">
-<a href="http://wiki.apache.org/pig/FAQ">FAQ</a>
-</div>
-<div class="menuitem">
-<a href="http://hadoop.apache.org/pig/releases.html">Release Notes</a>
-</div>
-</div>
-<div id="credit"></div>
-<div id="roundbottom">
-<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
-<!--+
-  |alternative credits
-  +-->
-<div id="credit2"></div>
-</div>
-<!--+
-    |end Menu
-    +-->
-<!--+
-    |start content
-    +-->
-<div id="content">
-<div title="Portable Document Format" class="pdflink">
-<a class="dida" href="cookbook.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
-        PDF</a>
-</div>
-<h1>Pig Cookbook</h1>
-<div id="minitoc-area">
-<ul class="minitoc">
-<li>
-<a href="#Overview">Overview</a>
-</li>
-<li>
-<a href="#Performance+Enhancers">Performance Enhancers</a>
-<ul class="minitoc">
-<li>
-<a href="#Use+Optimization">Use Optimization</a>
-</li>
-<li>
-<a href="#Use+Types">Use Types</a>
-</li>
-<li>
-<a href="#Project+Early+and+Often">Project Early and Often </a>
-</li>
-<li>
-<a href="#Filter+Early+and+Often">Filter Early and Often</a>
-</li>
-<li>
-<a href="#Reduce+Your+Operator+Pipeline">Reduce Your Operator Pipeline</a>
-</li>
-<li>
-<a href="#Make+Your+UDFs+Algebraic">Make Your UDFs Algebraic</a>
-</li>
-<li>
-<a href="#Implement+the+Aggregator+Interface">Implement the Aggregator Interface</a>
-</li>
-<li>
-<a href="#Drop+Nulls+Before+a+Join">Drop Nulls Before a Join</a>
-</li>
-<li>
-<a href="#Take+Advantage+of+Join+Optimizations">Take Advantage of Join Optimizations</a>
-</li>
-<li>
-<a href="#Use+the+PARALLEL+Clause">Use the PARALLEL Clause</a>
-</li>
-<li>
-<a href="#Use+the+LIMIT+Operator">Use the LIMIT Operator</a>
-</li>
-<li>
-<a href="#Prefer+DISTINCT+over+GROUP+BY+-+GENERATE">Prefer DISTINCT over GROUP BY - GENERATE</a>
-</li>
-</ul>
-</li>
-</ul>
-</div>
-<a name="N1000D"></a><a name="Overview"></a>
-<h2 class="h3">Overview</h2>
-<div class="section">
-<p>This document provides hints and tips for pig users. </p>
-</div>
-<a name="N10017"></a><a name="Performance+Enhancers"></a>
-<h2 class="h3">Performance Enhancers</h2>
-<div class="section">
-<a name="N1001D"></a><a name="Use+Optimization"></a>
-<h3 class="h4">Use Optimization</h3>
-<p>Pig supports various <a href="piglatin_ref1.html#Optimization+Rules">optimization rules</a> which are turned on by default.
-Become familiar with these rules.</p>
-<a name="N1002B"></a><a name="Use+Types"></a>
-<h3 class="h4">Use Types</h3>
-<p>If types are not specified in the load statement, Pig assumes the type of =double= for numeric computations.
-A lot of the time, your data would be much smaller, maybe, integer or long. Specifying the real type will help with
-speed of arithmetic computation. It has an additional advantage of early error detection. </p>
-<pre class="code">
---Query 1
-A = load 'myfile' as (t, u, v);
-B = foreach A generate t + u;
---Query 2
-A = load 'myfile' as (t: int, u: int, v);
-B = foreach A generate t + u;
-</pre>
-<p>The second query will run more efficiently than the first. In some of our queries with see 2x speedup. </p>
-<a name="N1003C"></a><a name="Project+Early+and+Often"></a>
-<h3 class="h4">Project Early and Often </h3>
-<p>Pig does not (yet) determine when a field is no longer needed and drop the field from the row. For example, say you have a query like: </p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = load 'myotherfile' as (x, y, z);
-C = join A by t, B by x;
-D = group C by u;
-E = foreach D generate group, COUNT($1);
-</pre>
-<p>There is no need for v, y, or z to participate in this query.  And there is no need to carry both t and x past the join, just one will suffice. Changing the query above to the query below will greatly reduce the amount of data being carried through the map and reduce phases by pig. </p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-A1 = foreach A generate t, u;
-B = load 'myotherfile' as (x, y, z);
-B1 = foreach B generate x;
-C = join A1 by t, B1 by x;
-C1 = foreach C generate t, u;
-D = group C1 by u;
-E = foreach D generate group, COUNT($1);
-</pre>
-<p>Depending on your data, this can produce significant time savings. In queries similar to the example shown here we have seen total time drop by 50%.</p>
-<a name="N10054"></a><a name="Filter+Early+and+Often"></a>
-<h3 class="h4">Filter Early and Often</h3>
-<p>As with early projection, in most cases it is beneficial to apply filters as early as possible to reduce the amount of data flowing through the pipeline. </p>
-<pre class="code">
--- Query 1
-A = load 'myfile' as (t, u, v);
-B = load 'myotherfile' as (x, y, z);
-C = filter A by t == 1;
-D = join C by t, B by x;
-E = group D by u;
-F = foreach E generate group, COUNT($1);
--- Query 2
-A = load 'myfile' as (t, u, v);
-B = load 'myotherfile' as (x, y, z);
-C = join A by t, B by x;
-D = group C by u;
-E = foreach D generate group, COUNT($1);
-F = filter E by C.t == 1;
-</pre>
-<p>The first query is clearly more efficient than the second one because it reduces the amount of data going into the join. </p>
-<p>One case where pushing filters up might not be a good idea is if the cost of applying filter is very high and only a small amount of data is filtered out. </p>
-<a name="N10068"></a><a name="Reduce+Your+Operator+Pipeline"></a>
-<h3 class="h4">Reduce Your Operator Pipeline</h3>
-<p>For clarity of your script, you might choose to split your projects into several steps for instance: </p>
-<pre class="code">
-A = load 'data' as (in: map[]);
--- get key out of the map
-B = foreach A generate in#k1 as k1, in#k2 as k2;
--- concatenate the keys
-C = foreach B generate CONCAT(k1, k2);
-.......
-</pre>
-<p>While the example above is easier to read, you might want to consider combining the two foreach statements to improve your query performance: </p>
-<pre class="code">
-A = load 'data' as (in: map[]);
--- concatenate the keys from the map
-B = foreach A generate CONCAT(in#k1, in#k2);
-....
-</pre>
-<p>The same goes for filters. </p>
-<a name="N10080"></a><a name="Make+Your+UDFs+Algebraic"></a>
-<h3 class="h4">Make Your UDFs Algebraic</h3>
-<p>Queries that can take advantage of the combiner generally ran much faster (sometimes several times faster) than the versions that don't. The latest code significantly improves combiner usage; however, you need to make sure you do your part. If you have a UDF that works on grouped data and is, by nature, algebraic (meaning their computation can be decomposed into multiple steps) make sure you implement it as such. For details on how to write algebraic UDFs, see the Pig UDF Manual and <a href="udf.html#Aggregate+Functions">Aggregate Functions</a>.</p>
-<pre class="code">
-A = load 'data' as (x, y, z)
-B = group A by x;
-C = foreach B generate group, MyUDF(A);
-....
-</pre>
-<p>If <span class="codefrag">MyUDF</span> is algebraic, the query will use combiner and run much faster. You can run <span class="codefrag">explain</span> command on your query to make sure that combiner is used. </p>
-<a name="N1009B"></a><a name="Implement+the+Aggregator+Interface"></a>
-<h3 class="h4">Implement the Aggregator Interface</h3>
-<p>
-If your UDF can't be made Algebraic but is able to deal with getting input in chunks rather than all at once, consider implementing the Aggregator interface to reduce the amount of memory used by your script. If your function <em>is</em> Algebraic and can be used on conjunction with Accumulator functions, you will need to implement the Accumulator interface as well as the Algebraic interface. For more information, see the Pig UDF Manual and <a href="udf.html#Accumulator+Interface">Accumulator Interface</a>.
-</p>
-<a name="N100AC"></a><a name="Drop+Nulls+Before+a+Join"></a>
-<h3 class="h4">Drop Nulls Before a Join</h3>
-<p>With the introduction of nulls, join and cogroup semantics were altered to work with nulls.  The semantic for cogrouping with nulls is that nulls from a given input are grouped together, but nulls across inputs are not grouped together.  This preserves the semantics of grouping (nulls are collected together from a single input to be passed to aggregate functions like COUNT) and the semantics of join (nulls are not joined across inputs).  Since flattening an empty bag results in an empty row, in a standard join the rows with a null key will always be dropped.  The join:  </p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = load 'myotherfile' as (x, y, z);
-C = join A by t, B by x;
-</pre>
-<p>is rewritten by pig to </p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = load 'myotherfile' as (x, y, z);
-C1 = cogroup A by t INNER, B by x INNER;
-C = foreach C1 generate flatten(A), flatten(B);
-</pre>
-<p>Since the nulls from A and B won't be collected together, when the nulls are flattened we're guaranteed to have an empty bag, which will result in no output.  So the null keys will be dropped.  But they will not be dropped until the last possible moment.  If the query is rewritten to </p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = load 'myotherfile' as (x, y, z);
-A1 = filter A by t is not null;
-B1 = filter B by x is not null;
-C = join A1 by t, B1 by x;
-</pre>
-<p>then the nulls will be dropped before the join.  Since all null keys go to a single reducer, if your key is null even a small percentage of the time the gain can be significant.  In one test where the key was null 7% of the time and the data was spread across 200 reducers, we saw a about a 10x speed up in the query by adding the early filters. </p>
-<a name="N100CB"></a><a name="Take+Advantage+of+Join+Optimizations"></a>
-<h3 class="h4">Take Advantage of Join Optimizations</h3>
-<a name="N100D1"></a><a name="Regular+Join+Optimizations"></a>
-<h4>Regular Join Optimizations</h4>
-<p>Optimization for regular joins ensures that the last table in the join is not brought into memory but streamed through instead. Optimization reduces the amount of memory used which means you can avoid spilling the data and also should be able to scale your query to larger data volumes. </p>
-<p>To take advantage of this optimization, make sure that the table with the largest number of tuples per key is the last table in your query.
-In some of our tests we saw 10x performance improvement as the result of this optimization.</p>
-<pre class="code">
-small = load 'small_file' as (t, u, v);
-large = load 'large_file' as (x, y, z);
-C = join small by t, large by x;
-</pre>
-<a name="N100E2"></a><a name="Specialized+Join+Optimizations"></a>
-<h4>Specialized Join Optimizations</h4>
-<p>Optimization can also be achieved using fragment replicate joins, skewed joins, and merge joins.
-For more information see <a href="piglatin_ref1.html#Specialized+Joins">Specialized Joins</a>.</p>
-<a name="N100F1"></a><a name="Use+the+PARALLEL+Clause"></a>
-<h3 class="h4">Use the PARALLEL Clause</h3>
-<p>Use the PARALLEL clause to increase the parallelism of a job:</p>
-<ul>
-<li>PARALLEL sets the number of reduce tasks for the MapReduce jobs generated by Pig. The default value is 1 (one reduce task).</li>
-<li>PARALLEL only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </li>
-<li>If you don&rsquo;t specify PARALLEL, you still get the same map parallelism but only one reduce task.</li>
-</ul>
-<p></p>
-<p>As noted, the default value for PARALLEL is 1 (one reduce task). However, the number of reducers you need for a particular construct in Pig that forms a MapReduce boundary depends entirely on (1) your data and the number of intermediate keys you are generating in your mappers  and (2) the partitioner and distribution of map (combiner) output keys. In the best cases we have seen that a reducer processing about 500 MB of data behaves efficiently.</p>
-<p>You can include the PARALLEL clause with any operator that starts a reduce phase (see the example below). This includes
-<a href="piglatin_ref2.html#COGROUP">COGROUP</a>,
-<a href="piglatin_ref2.html#CROSS">CROSS</a>,
-<a href="piglatin_ref2.html#DISTINCT">DISTINCT</a>,
-<a href="piglatin_ref2.html#GROUP">GROUP</a>,
-<a href="piglatin_ref2.html#JOIN+%28inner%29">JOIN (inner)</a>,
-<a href="piglatin_ref2.html#JOIN+%28outer%29">JOIN (outer)</a>, and
-<a href="piglatin_ref2.html#ORDER">ORDER</a>.
-</p>
-<p>You can also set the value of PARALLEL for all Pig scripts using the <a href="piglatin_ref2.html#set">set default parallel</a> command.</p>
-<p>In this example PARALLEL is used with the GROUP operator. </p>
-<pre class="code">
-A = LOAD 'myfile' AS (t, u, v);
-B = GROUP A BY t PARALLEL 18;
-.....
-</pre>
-<p>In this example all the MapReduce jobs that get launched use 20 reducers.</p>
-<pre class="code">
-SET DEFAULT_PARALLEL 20;
-A = LOAD &lsquo;myfile.txt&rsquo; USING PigStorage() AS (t, u, v);
-B = GROUP A BY t;
-C = FOREACH B GENERATE group, COUNT(A.t) as mycount;
-D = ORDER C BY mycount;
-STORE D INTO &lsquo;mysortedcount&rsquo; USING PigStorage();
-</pre>
-<a name="N10140"></a><a name="Use+the+LIMIT+Operator"></a>
-<h3 class="h4">Use the LIMIT Operator</h3>
-<p>Often you are not interested in the entire output but rather a sample or top results. In such cases, using LIMIT can yield a much better performance as we push the limit as high as possible to minimize the amount of data travelling through the pipeline. </p>
-<p>Sample:
-</p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = limit A 500;
-</pre>
-<p>Top results: </p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = order A by t;
-C = limit B 500;
-</pre>
-<a name="N10158"></a><a name="Prefer+DISTINCT+over+GROUP+BY+-+GENERATE"></a>
-<h3 class="h4">Prefer DISTINCT over GROUP BY - GENERATE</h3>
-<p>When it comes to extracting the unique values from a column in a relation, one of two approaches can be used: </p>
-<p>Example Using GROUP BY - GENERATE</p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = foreach A generate u;
-C = group B by u;
-D = foreach C generate group as uniquekey;
-dump D;
-</pre>
-<p>Example Using DISTINCT</p>
-<pre class="code">
-A = load 'myfile' as (t, u, v);
-B = foreach A generate u;
-C = distinct B;
-dump C;
-</pre>
-<p>In pig 0.1.x, DISTINCT is just GROUP BY/PROJECT under the hood. In pig 0.2.0 it is not, and it is much faster and more efficient (depending on your key cardinality, up to 20x faster in pig team's tests). Therefore, the use of DISTINCT is recommended over GROUP BY - GENERATE.  </p>
-</div>
-</div>
-<!--+
-    |end content
-    +-->
-<div class="clearboth">&nbsp;</div>
-</div>
-<div id="footer">
-<!--+
-    |start bottomstrip
-    +-->
-<div class="lastmodified">
-<script type="text/javascript"><!--
-document.write("Last Published: " + document.lastModified);
-//  --></script>
-</div>
-<div class="copyright">
-        Copyright &copy;
-         2007-2010 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
-</div>
-<!--+
-    |end bottomstrip
-    +-->
-</div>
-</body>
-</html>