ul-wukong 4.1.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (261) hide show
  1. checksums.yaml +15 -0
  2. data/.gitignore +60 -0
  3. data/.gitmodules +6 -0
  4. data/.rspec +2 -0
  5. data/.travis.yml +19 -0
  6. data/.yardopts +6 -0
  7. data/CHANGELOG.md +7 -0
  8. data/Gemfile +17 -0
  9. data/Guardfile +12 -0
  10. data/LICENSE.md +95 -0
  11. data/NOTES-travis.md +31 -0
  12. data/README-old.md +422 -0
  13. data/README.md +1308 -0
  14. data/Rakefile +28 -0
  15. data/TODO.md +99 -0
  16. data/bin/cutc +30 -0
  17. data/bin/cuttab +5 -0
  18. data/bin/greptrue +6 -0
  19. data/bin/md5sort +20 -0
  20. data/bin/setcat +11 -0
  21. data/bin/tabchar +5 -0
  22. data/bin/uniq-ord +59 -0
  23. data/bin/uniqc +3 -0
  24. data/bin/wu +34 -0
  25. data/bin/wu-clean-encoding +31 -0
  26. data/bin/wu-date +13 -0
  27. data/bin/wu-datetime +13 -0
  28. data/bin/wu-hist +3 -0
  29. data/bin/wu-lign +186 -0
  30. data/bin/wu-local +4 -0
  31. data/bin/wu-plus +9 -0
  32. data/bin/wu-source +5 -0
  33. data/bin/wu-sum +31 -0
  34. data/diagrams/wu_local.dot +39 -0
  35. data/diagrams/wu_local.dot.png +0 -0
  36. data/examples/Gemfile +38 -0
  37. data/examples/README.md +9 -0
  38. data/examples/basic/string_reverser.rb +23 -0
  39. data/examples/basic/tiny_count.rb +8 -0
  40. data/examples/basic/word_count/accumulator.rb +26 -0
  41. data/examples/basic/word_count/tokenizer.rb +13 -0
  42. data/examples/basic/word_count/word_count.rb +6 -0
  43. data/examples/dataflow/scraper_macro_flow.rb +28 -0
  44. data/examples/deploy_pack/Gemfile +6 -0
  45. data/examples/deploy_pack/README.md +6 -0
  46. data/examples/deploy_pack/a/b/c/.gitkeep +0 -0
  47. data/examples/deploy_pack/app/processors/string_reverser.rb +5 -0
  48. data/examples/deploy_pack/config/environment.rb +1 -0
  49. data/examples/dsl/dataflow/fibonacci_series.rb +101 -0
  50. data/examples/dsl/dataflow/scraper_macro_flow.rb +28 -0
  51. data/examples/dsl/dataflow/simple.rb +12 -0
  52. data/examples/dsl/dataflow/telegram.rb +45 -0
  53. data/examples/dsl/workflow/cherry_pie.dot +97 -0
  54. data/examples/dsl/workflow/cherry_pie.md +104 -0
  55. data/examples/dsl/workflow/cherry_pie.png +0 -0
  56. data/examples/dsl/workflow/cherry_pie.rb +101 -0
  57. data/examples/empty/.gitkeep +0 -0
  58. data/examples/examples_helper.rb +9 -0
  59. data/examples/geo.rb +4 -0
  60. data/examples/geo/geo_grids.numbers +0 -0
  61. data/examples/geo/geolocated.rb +331 -0
  62. data/examples/geo/quadtile.rb +69 -0
  63. data/examples/geo/spec/geolocated_spec.rb +247 -0
  64. data/examples/geo/tile_fetcher.rb +77 -0
  65. data/examples/graph/implied_geolocation/README.md +63 -0
  66. data/examples/graph/minimum_spanning_tree/airfares_graphviz.rb +73 -0
  67. data/examples/improver/tweet_summary.rb +73 -0
  68. data/examples/loadable.rb +2 -0
  69. data/examples/munging/airline_flights/airline_flights.rake +83 -0
  70. data/examples/munging/airline_flights/airplane.rb +0 -0
  71. data/examples/munging/airline_flights/airport_id_unification.rb +129 -0
  72. data/examples/munging/airline_flights/airport_ok_chars.rb +4 -0
  73. data/examples/munging/airline_flights/indexable.rb +75 -0
  74. data/examples/munging/airline_flights/indexable_spec.rb +90 -0
  75. data/examples/munging/airline_flights/reconcile_airports.rb +142 -0
  76. data/examples/munging/airline_flights/tasks.rake +83 -0
  77. data/examples/munging/airline_flights/topcities.rb +167 -0
  78. data/examples/munging/geo/geo_json.rb +54 -0
  79. data/examples/munging/geo/geo_models.rb +69 -0
  80. data/examples/munging/geo/geonames_models.rb +107 -0
  81. data/examples/munging/geo/iso_codes.rb +172 -0
  82. data/examples/munging/geo/reconcile_countries.rb +124 -0
  83. data/examples/munging/geo/tasks.rake +71 -0
  84. data/examples/munging/wikipedia/articles/extract_articles-parsed.rb +79 -0
  85. data/examples/munging/wikipedia/articles/extract_articles-templated.rb +136 -0
  86. data/examples/munging/wikipedia/articles/textualize_articles.rb +54 -0
  87. data/examples/munging/wikipedia/articles/verify_structure.rb +43 -0
  88. data/examples/munging/wikipedia/articles/wp2txt-LICENSE.txt +22 -0
  89. data/examples/munging/wikipedia/articles/wp2txt_article.rb +259 -0
  90. data/examples/munging/wikipedia/articles/wp2txt_utils.rb +452 -0
  91. data/examples/munging/wikipedia/dbpedia/dbpedia_common.rb +5 -0
  92. data/examples/munging/wikipedia/dbpedia/dbpedia_extract_geocoordinates.rb +78 -0
  93. data/examples/munging/wikipedia/dbpedia/extract_links-cruft.rb +66 -0
  94. data/examples/munging/wikipedia/dbpedia/extract_links.rb +260 -0
  95. data/examples/munging/wikipedia/dbpedia/sameas_extractor.rb +20 -0
  96. data/examples/rake_helper.rb +97 -0
  97. data/examples/ruby_project/Gemfile +6 -0
  98. data/examples/ruby_project/README.md +6 -0
  99. data/examples/ruby_project/a/b/c/.gitkeep +0 -0
  100. data/examples/server_logs/geo_ip_mapping/munge_geolite.rb +82 -0
  101. data/examples/server_logs/logline.rb +95 -0
  102. data/examples/server_logs/models.rb +66 -0
  103. data/examples/server_logs/page_counts.pig +48 -0
  104. data/examples/server_logs/server_logs-01-parse-script.rb +13 -0
  105. data/examples/server_logs/server_logs-02-histograms-full.rb +33 -0
  106. data/examples/server_logs/server_logs-02-histograms-mapper.rb +14 -0
  107. data/examples/server_logs/server_logs-03-breadcrumbs-full.rb +71 -0
  108. data/examples/server_logs/server_logs-04-page_page_edges-full.rb +40 -0
  109. data/examples/serverlogs/geo_ip_mapping/munge_geolite.rb +82 -0
  110. data/examples/serverlogs/models/logline.rb +102 -0
  111. data/examples/serverlogs/parser/apache_parser_widget.rb +46 -0
  112. data/examples/serverlogs/visit_paths/common.rb +4 -0
  113. data/examples/serverlogs/visit_paths/page_counts.pig +48 -0
  114. data/examples/serverlogs/visit_paths/serverlogs-01-parse-script.rb +11 -0
  115. data/examples/serverlogs/visit_paths/serverlogs-02-histograms-full.rb +31 -0
  116. data/examples/serverlogs/visit_paths/serverlogs-02-histograms-mapper.rb +12 -0
  117. data/examples/serverlogs/visit_paths/serverlogs-03-breadcrumbs-full.rb +67 -0
  118. data/examples/serverlogs/visit_paths/serverlogs-04-page_page_edges-full.rb +38 -0
  119. data/examples/splitter.rb +94 -0
  120. data/examples/string_reverser.rb +7 -0
  121. data/examples/text/pig_latin/pig_latinizer.rb +35 -0
  122. data/examples/text/pig_latin/pig_latinizer_widget.rb +16 -0
  123. data/examples/text/regional_flavor/README.md +14 -0
  124. data/examples/text/regional_flavor/article_wordbags.pig +39 -0
  125. data/examples/text/regional_flavor/j01-article_wordbags.rb +4 -0
  126. data/examples/text/regional_flavor/simple_pig_script.pig +27 -0
  127. data/examples/twitter.rb +5 -0
  128. data/lib/hanuman.rb +36 -0
  129. data/lib/hanuman/graph.rb +97 -0
  130. data/lib/hanuman/graphvizzer.rb +206 -0
  131. data/lib/hanuman/graphvizzer/gv_models.rb +161 -0
  132. data/lib/hanuman/graphvizzer/gv_presenter.rb +97 -0
  133. data/lib/hanuman/link.rb +35 -0
  134. data/lib/hanuman/registry.rb +46 -0
  135. data/lib/hanuman/stage.rb +128 -0
  136. data/lib/hanuman/tree.rb +67 -0
  137. data/lib/wu/geo.rb +4 -0
  138. data/lib/wu/geo/geo_grids.numbers +0 -0
  139. data/lib/wu/geo/geolocated.rb +331 -0
  140. data/lib/wu/geo/quadtile.rb +69 -0
  141. data/lib/wu/graph/union_find.rb +62 -0
  142. data/lib/wu/model/reconcilable.rb +63 -0
  143. data/lib/wu/munging.rb +71 -0
  144. data/lib/wu/social/models/twitter.rb +31 -0
  145. data/lib/wu/wikipedia/models.rb +20 -0
  146. data/lib/wukong.rb +54 -0
  147. data/lib/wukong/dataflow.rb +43 -0
  148. data/lib/wukong/doc_helpers.rb +14 -0
  149. data/lib/wukong/doc_helpers/dataflow_handler.rb +29 -0
  150. data/lib/wukong/doc_helpers/field_handler.rb +91 -0
  151. data/lib/wukong/doc_helpers/processor_handler.rb +29 -0
  152. data/lib/wukong/driver.rb +214 -0
  153. data/lib/wukong/driver/event_machine_driver.rb +15 -0
  154. data/lib/wukong/driver/wiring.rb +68 -0
  155. data/lib/wukong/local.rb +42 -0
  156. data/lib/wukong/local/runner.rb +96 -0
  157. data/lib/wukong/local/stdio_driver.rb +104 -0
  158. data/lib/wukong/logger.rb +102 -0
  159. data/lib/wukong/model/faker.rb +136 -0
  160. data/lib/wukong/model/flatpack_parser/flat.rb +60 -0
  161. data/lib/wukong/model/flatpack_parser/flatpack.rb +4 -0
  162. data/lib/wukong/model/flatpack_parser/lang.rb +46 -0
  163. data/lib/wukong/model/flatpack_parser/parser.rb +55 -0
  164. data/lib/wukong/model/flatpack_parser/tokens.rb +130 -0
  165. data/lib/wukong/plugin.rb +48 -0
  166. data/lib/wukong/processor.rb +110 -0
  167. data/lib/wukong/rake_helper.rb +6 -0
  168. data/lib/wukong/runner.rb +169 -0
  169. data/lib/wukong/runner/boot_sequence.rb +123 -0
  170. data/lib/wukong/runner/code_loader.rb +52 -0
  171. data/lib/wukong/runner/command_runner.rb +44 -0
  172. data/lib/wukong/runner/deploy_pack_loader.rb +75 -0
  173. data/lib/wukong/runner/help_message.rb +42 -0
  174. data/lib/wukong/source.rb +33 -0
  175. data/lib/wukong/source/source_driver.rb +74 -0
  176. data/lib/wukong/source/source_runner.rb +38 -0
  177. data/lib/wukong/spec_helpers.rb +74 -0
  178. data/lib/wukong/spec_helpers/integration_tests.rb +150 -0
  179. data/lib/wukong/spec_helpers/integration_tests/integration_test_matchers.rb +207 -0
  180. data/lib/wukong/spec_helpers/integration_tests/integration_test_runner.rb +97 -0
  181. data/lib/wukong/spec_helpers/shared_examples.rb +22 -0
  182. data/lib/wukong/spec_helpers/unit_tests.rb +135 -0
  183. data/lib/wukong/spec_helpers/unit_tests/unit_test_driver.rb +132 -0
  184. data/lib/wukong/spec_helpers/unit_tests/unit_test_matchers.rb +169 -0
  185. data/lib/wukong/spec_helpers/unit_tests/unit_test_runner.rb +60 -0
  186. data/lib/wukong/version.rb +3 -0
  187. data/lib/wukong/widget/echo.rb +55 -0
  188. data/lib/wukong/widget/extract.rb +122 -0
  189. data/lib/wukong/widget/filters.rb +452 -0
  190. data/lib/wukong/widget/logger.rb +56 -0
  191. data/lib/wukong/widget/operators.rb +82 -0
  192. data/lib/wukong/widget/reducers.rb +10 -0
  193. data/lib/wukong/widget/reducers/accumulator.rb +73 -0
  194. data/lib/wukong/widget/reducers/bin.rb +368 -0
  195. data/lib/wukong/widget/reducers/count.rb +73 -0
  196. data/lib/wukong/widget/reducers/group.rb +128 -0
  197. data/lib/wukong/widget/reducers/group_concat.rb +98 -0
  198. data/lib/wukong/widget/reducers/improver.rb +71 -0
  199. data/lib/wukong/widget/reducers/join_xml.rb +37 -0
  200. data/lib/wukong/widget/reducers/moments.rb +72 -0
  201. data/lib/wukong/widget/reducers/sort.rb +180 -0
  202. data/lib/wukong/widget/reducers/uniq.rb +91 -0
  203. data/lib/wukong/widget/serializers.rb +317 -0
  204. data/lib/wukong/widget/utils.rb +46 -0
  205. data/lib/wukong/widgets.rb +7 -0
  206. data/spec/examples/dataflow/fibonacci_series_spec.rb +18 -0
  207. data/spec/examples/dataflow/parse_apache_logs_spec.rb +8 -0
  208. data/spec/examples/dataflow/parsing_spec.rb +14 -0
  209. data/spec/examples/dataflow/simple_spec.rb +34 -0
  210. data/spec/examples/dataflow/telegram_spec.rb +43 -0
  211. data/spec/examples/graph/minimum_spanning_tree_spec.rb +34 -0
  212. data/spec/examples/munging/airline_flights/identifiers_spec.rb +16 -0
  213. data/spec/examples/munging/airline_flights_spec.rb +202 -0
  214. data/spec/examples/text/pig_latin_spec.rb +18 -0
  215. data/spec/examples/workflow/cherry_pie_spec.rb +36 -0
  216. data/spec/hanuman/graph_spec.rb +119 -0
  217. data/spec/hanuman/hanuman_spec.rb +10 -0
  218. data/spec/hanuman/registry_spec.rb +123 -0
  219. data/spec/hanuman/stage_spec.rb +81 -0
  220. data/spec/hanuman/tree_spec.rb +119 -0
  221. data/spec/spec.opts +1 -0
  222. data/spec/spec_helper.rb +43 -0
  223. data/spec/support/example_test_helpers.rb +95 -0
  224. data/spec/support/hanuman_test_helpers.rb +92 -0
  225. data/spec/support/integration_helper.rb +38 -0
  226. data/spec/support/model_test_helpers.rb +115 -0
  227. data/spec/support/shared_context_for_graphs.rb +57 -0
  228. data/spec/support/shared_context_for_reducers.rb +37 -0
  229. data/spec/support/shared_examples_for_builders.rb +94 -0
  230. data/spec/support/shared_examples_for_shortcuts.rb +57 -0
  231. data/spec/wu/model/reconcilable_spec.rb +152 -0
  232. data/spec/wukong/dataflow_spec.rb +87 -0
  233. data/spec/wukong/driver_spec.rb +154 -0
  234. data/spec/wukong/local/runner_spec.rb +29 -0
  235. data/spec/wukong/local/stdio_driver_spec.rb +73 -0
  236. data/spec/wukong/local_spec.rb +6 -0
  237. data/spec/wukong/logger_spec.rb +49 -0
  238. data/spec/wukong/model/faker_spec.rb +132 -0
  239. data/spec/wukong/processor_spec.rb +21 -0
  240. data/spec/wukong/runner_spec.rb +132 -0
  241. data/spec/wukong/source_spec.rb +6 -0
  242. data/spec/wukong/widget/extract_spec.rb +101 -0
  243. data/spec/wukong/widget/filters_spec.rb +79 -0
  244. data/spec/wukong/widget/logger_spec.rb +23 -0
  245. data/spec/wukong/widget/operators_spec.rb +25 -0
  246. data/spec/wukong/widget/reducers/bin_spec.rb +92 -0
  247. data/spec/wukong/widget/reducers/count_spec.rb +11 -0
  248. data/spec/wukong/widget/reducers/group_spec.rb +21 -0
  249. data/spec/wukong/widget/reducers/join_xml_spec.rb +25 -0
  250. data/spec/wukong/widget/reducers/moments_spec.rb +36 -0
  251. data/spec/wukong/widget/reducers/sort_spec.rb +26 -0
  252. data/spec/wukong/widget/reducers/uniq_spec.rb +14 -0
  253. data/spec/wukong/widget/serializers_spec.rb +114 -0
  254. data/spec/wukong/widget/sink_spec.rb +19 -0
  255. data/spec/wukong/widget/source_spec.rb +65 -0
  256. data/spec/wukong/wu-local_spec.rb +109 -0
  257. data/spec/wukong/wu-source_spec.rb +32 -0
  258. data/spec/wukong/wu_spec.rb +14 -0
  259. data/spec/wukong/wukong_spec.rb +10 -0
  260. data/wukong.gemspec +35 -0
  261. metadata +465 -0
@@ -0,0 +1,15 @@
1
+ ---
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ Njc1YWQyMGMxZWRkZDZmY2M5NTAxZGFmNGU1YTliMDBjZWRmYjQ3YQ==
5
+ data.tar.gz: !binary |-
6
+ YjRlOTFhM2QyOWY1N2EyYmRmNDE2ZTE5MzU3ZDQ0NDg4YjcxZmVjZQ==
7
+ SHA512:
8
+ metadata.gz: !binary |-
9
+ NTEyYWM0MmNhYmJiYjhkYjE3MGNkNjk2MzQ1NWE3Nzg3YzYyMTk0YmIyM2E0
10
+ NTA2MWI0NTA1NjBhOWYzOGJiNWFkNmJlNjQxNjY5YTgxYzA2NGFhNWVhOTUy
11
+ NDA2NGI1ZDgxMzM1MGNkNTdhOWFlOTNjOWJlMDM0ZjU3NGEzZWU=
12
+ data.tar.gz: !binary |-
13
+ ZmE4YTA1N2JmNGE2MmM2MzU0MzQ0Yzg1YjE5ZGFkMDYwNDMyNzRiZWJmNzdh
14
+ MzY4YjMzYmIxYmViY2U0MTExOTVmYjY4OTcxODg2MDA0NWEwZmRkNGYyMTRl
15
+ ZjFjYjVkMTY5Mzc3MjU1Mzk5ZjE5ZTVkM2ZkZGI2MWZjOTExZTE=
@@ -0,0 +1,60 @@
1
+ ## OS
2
+ .DS_Store
3
+ Icon
4
+ nohup.out
5
+ .bak
6
+
7
+ *.pem
8
+
9
+ ## EDITORS
10
+ \#*
11
+ .\#*
12
+ \#*\#
13
+ *~
14
+ *.swp
15
+ REVISION
16
+ TAGS*
17
+ tmtags
18
+ *_flymake.*
19
+ *_flymake
20
+ *.tmproj
21
+ .project
22
+ .settings
23
+
24
+ ## COMPILED
25
+ a.out
26
+ *.o
27
+ *.pyc
28
+ *.so
29
+
30
+ ## OTHER SCM
31
+ .bzr
32
+ .hg
33
+ .svn
34
+
35
+ ## PROJECT::GENERAL
36
+
37
+ log/*
38
+ tmp/*
39
+ pkg/*
40
+
41
+ coverage
42
+ rdoc
43
+ doc
44
+ pkg
45
+ .rake_test_cache
46
+ .bundle
47
+ .yardoc
48
+
49
+ .vendor
50
+
51
+ ## PROJECT::SPECIFIC
52
+
53
+ old/*
54
+ docpages
55
+ away
56
+
57
+ .rbx
58
+ Gemfile.lock
59
+ Backup*of*.numbers
60
+ *.gem
@@ -0,0 +1,6 @@
1
+ [submodule "notes"]
2
+ path = notes
3
+ url = git://github.com/infochimps-labs/wukong.wiki.git
4
+ [submodule "data"]
5
+ path = data
6
+ url = http://github.com/infochimps-labs/wukong-example-data.git
data/.rspec ADDED
@@ -0,0 +1,2 @@
1
+ --format=progress
2
+ --color
@@ -0,0 +1,19 @@
1
+ language: ruby
2
+ rvm:
3
+ - 1.9.3
4
+ - 1.9.2
5
+ - jruby-19mode # JRuby in 1.9 mode
6
+ - rbx-19mode
7
+
8
+ before_install: "git clone -b version_1 git://github.com/infochimps-labs/gorillib.git ~/builds/infochimps-labs/gorillib"
9
+
10
+ bundler_args: --without docs support
11
+
12
+ branches:
13
+ only:
14
+ - master
15
+ - wukong_ng
16
+ - vanilla_2
17
+
18
+ notifications:
19
+ email: false
@@ -0,0 +1,6 @@
1
+ --readme README.md
2
+ --markup markdown
3
+ -
4
+ CHANGELOG.md
5
+ LICENSE.md
6
+ README.md
@@ -0,0 +1,7 @@
1
+ ## Version 3: Complete rewrite
2
+
3
+ Version 3 is a complete refresh of Wukong. There will probably be a compatibility layer.
4
+
5
+ The new version is highly modularized, and built on top of the Hanuman dataflow toolkit.
6
+
7
+ The central idea is to assemble your jobs as a stack of decoupled stages. These stages are agnostic to whether they are running in a hadoop batch job, from the command line, in a Flume decorator, or as middleware in a Hanuman request stack.
data/Gemfile ADDED
@@ -0,0 +1,17 @@
1
+ source 'https://rubygems.org'
2
+
3
+ gemspec
4
+
5
+ group :development do
6
+ gem 'rake', '>= 0.9'
7
+ gem 'rspec', '~> 2.8'
8
+ gem 'spork', '0.9.2'
9
+ gem 'guard', '>= 1.0'
10
+ gem 'guard-rspec', '>= 0.6'
11
+ gem 'simplecov', '>= 0.5'
12
+ gem 'pry'
13
+ gem 'yard'
14
+ gem 'redcarpet'
15
+ gem 'addressable'
16
+ gem 'htmlentities'
17
+ end
@@ -0,0 +1,12 @@
1
+ # -*- ruby -*-
2
+
3
+ format = 'doc' # 'doc' for more verbose, 'progress' for less
4
+ tags = %w[ only ] # builder_spec model_spec
5
+
6
+ guard 'rspec', :version => 2, :cli => "--format #{format} #{ tags.map{|tag| "--tag #{tag}"}.join(' ') }" do
7
+ watch(%r{^spec/.+_spec\.rb$})
8
+ watch(%r{^examples/(.+)\.rb$}) { |m| "spec/examples/#{m[1]}_spec.rb" }
9
+ watch(%r{^lib/(.+)\.rb$}) { |m| "spec/#{m[1]}}_spec.rb" }
10
+ watch('spec/spec_helper.rb') { 'spec' }
11
+ watch(/spec\/support\/(.+)\.rb/) { 'spec' }
12
+ end
@@ -0,0 +1,95 @@
1
+ # License for Wukong
2
+
3
+ The wukong code is __Copyright (c) 2011, 2012 Infochimps, Inc__
4
+
5
+ This code is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
10
+
11
+ __________________________________________________________________________
12
+
13
+ # Apache License
14
+
15
+
16
+ Apache License
17
+ Version 2.0, January 2004
18
+ http://www.apache.org/licenses/
19
+
20
+ _TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION_
21
+
22
+ ## 1. Definitions.
23
+
24
+ * **License** shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
25
+
26
+ * **Licensor** shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
27
+
28
+ * **Legal Entity** shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
29
+
30
+ * **You** (or **Your**) shall mean an individual or Legal Entity exercising permissions granted by this License.
31
+
32
+ * **Source** form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
33
+
34
+ * **Object** form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
35
+
36
+ * **Work** shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
37
+
38
+ * **Derivative Works** shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
39
+
40
+ * **Contribution** shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
41
+
42
+ * **Contributor** shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
43
+
44
+ ## 2. Grant of Copyright License.
45
+
46
+ Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
47
+
48
+ ## 3. Grant of Patent License.
49
+
50
+ Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
51
+
52
+ ## 4. Redistribution.
53
+
54
+ You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
55
+
56
+ - (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
57
+ - (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
58
+ - (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
59
+ - (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
60
+
61
+ You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
62
+
63
+ ## 5. Submission of Contributions.
64
+
65
+ Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
66
+
67
+ ## 6. Trademarks.
68
+
69
+ This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
70
+
71
+ ## 7. Disclaimer of Warranty.
72
+
73
+ Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
74
+
75
+ ## 8. Limitation of Liability.
76
+
77
+ In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
78
+
79
+ ## 9. Accepting Warranty or Additional Liability.
80
+
81
+ While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
82
+
83
+ _END OF TERMS AND CONDITIONS_
84
+
85
+ ## APPENDIX: How to apply the Apache License to your work.
86
+
87
+ To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets `[]` replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
88
+
89
+ > Copyright [yyyy] [name of copyright owner]
90
+ >
91
+ > Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
92
+ >
93
+ > http://www.apache.org/licenses/LICENSE-2.0
94
+ >
95
+ > Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
@@ -0,0 +1,31 @@
1
+ # General
2
+
3
+ * There is this awkward and unstated need that a dataflow set its output using `set_output(sink)`.
4
+ ** `Wukong::LocalRunner` has this hard-coded to `sink(:test_sink)` - broke.
5
+
6
+ * The `map` method used in dataflow; from whence does it come? Found it in `wukong/processor.rb`; does this fit better in `wukong/widget`?
7
+
8
+ * A "null" processor AND an "as_is" processor don't both make sense. I think, conceptually, that `Wukong::Processor::Null` === `Wukong::Processor::AsIs` and we should choose only one. A "null" sink might be better suited for the purporse of "rejecting all records." And this already exists in `widget/sink.rb`.
9
+ ** There are also "all" and "none" filters in `widget/filter.rb`.
10
+
11
+ * `Wukong::Filter::All` and `Wukong::Filter::None` are not registered. Should they be? Do they work as they should, because it appears not...
12
+ ** Are these necessary as `Wukong::Processor::Null` and `Wukong::Processor::AsIs` already exist?
13
+
14
+ * In `Wukong::Runner` why do I have to specify sinks/sources wih a name? are these ever referenced/used in any context later? Seems that the runner might not need names...
15
+
16
+ * Is there a reason wukong has a config directory with a very outdated yaml file?
17
+
18
+ * DONE ~~The Guardfile has a lot of debuggy cruft. Fixit.~~
19
+
20
+ * DONE ~~Let's use Bundler to manage the gem instead of Jeweler.~~
21
+
22
+ * So many rad 80's references.
23
+
24
+ # Specs
25
+
26
+ * DONE ~~The graphviz spec could be made into an argument in spec helper, whether to run the file or not.~~
27
+
28
+ * There should be a cleanup phase after specs have run to delete artifacts.
29
+
30
+ * A lot of examples. Most don't work. Fixit.
31
+
@@ -0,0 +1,422 @@
1
+ # Wukong [![Build Status](https://secure.travis-ci.org/infochimps-labs/wukong.png)](http://travis-ci.org/infochimps-labs/wukong)
2
+
3
+ Wukong is a toolkit for rapid, agile development of dataflows at any scale.
4
+
5
+ (note: the syntax below is mostly false)
6
+
7
+
8
+ <a name="design"></a>
9
+ ## Design Overview
10
+
11
+ The fundamental principle of Wukong/Hanuman is *powerful black boxes, beautiful glue*. In general, they don't do the thing -- they coordinate the boxes that do the thing, to let you implement rapidly, nimbly and readably. Hanuman elegantly describes high-level data flows; Wukong is a pragmatic collection of dataflow primitives. They both emphasize scalability, readability and rapid development over performance or universality.
12
+
13
+ Wukong/Hanuman are chiefly concerned with these specific types of graphs:
14
+
15
+ * **dataflow** -- chains of simple modules to handle continuous data processing -- coordinates Flume, Unix pipes, ZeroMQ, Esper, Storm.
16
+ * **workflows** -- episodic jobs sequences, joined by dependency links -- comparable to Rake, Azkaban or Oozie.
17
+ * **map/reduce** -- Hadoop's standard *disordered/partitioned stream > partition, sort & group > process groups* workflow. Comparable to MRJob and Dumbo.
18
+ * **queue workers** -- pub/sub asynchronously triggered jobs -- comparable Resque, RabbitMQ/AMQP, Amazon Simple Worker, Heroku workers.
19
+
20
+ In addition, wukong stages may be deployed into **http middlware**: lightweight distributed API handlers -- comparable to Rack, Goliath or Twisted.
21
+
22
+ When you're describing a Wukong/Hanuman flow, you're writing pure expressive ruby, not some hokey interpreted language or clumsy XML format. Thanks to JRuby, it can speak directly to Java-based components like Hadoop, Flume, Storm or Spark.
23
+
24
+ ## What's where
25
+
26
+ * Configliere -- Manage settings
27
+ - Layer - Project settings through a late-resolved stack of config objects.
28
+ * Gorillib
29
+ - Type, RecordType
30
+ - TypeConversion
31
+ - Model
32
+ - PathHelpers
33
+ * Wukong
34
+ - fs - Abstracts file hdfs s3n s3hdfs scp
35
+ - streamer - Black-box data transform
36
+ - job - Workflow definition
37
+ - flow - Dataflow definition
38
+ - widgets - Common data transforms
39
+ - RubyHadoop - Hadoop jobs using streamers
40
+ - RubyFlume - Flume decorators using streamers
41
+ * Hanuman -- Elegant small graph assembly
42
+ * Swineherd -- Common interface on ugly tools
43
+ - Turn readable hash into safe commandline (param conv, escaping)
44
+ - Execute command, capture stdin/stderr
45
+ - Summarize execution with a broham-able hash
46
+ - Common modules: Input/output, Java, gnu, configliere
47
+ - Template
48
+ - Hadoop, pig, flume
49
+ - ?? Cp, mv, rm, zip, tar, bz2, gz, ssh, scp
50
+ - ?? Remotely execute command
51
+
52
+ <a name="design-rules"></a>
53
+ ### Story
54
+
55
+ [Narrative Method Structure](http://avdi.org/talks/confident-code-rubymidwest-2011/confident-code.html)
56
+
57
+ * Gather input
58
+ * Perform work
59
+ * Deliver results
60
+ * Handle failure
61
+
62
+
63
+ <a name="design-rules"></a>
64
+ ### Design Rules
65
+
66
+ * **whiteboard rule**: the user-facing conceptual model should match the picture you would draw on the whiteboard in an engineering discussion. The fundamental goal is to abstract the necessary messiness surrounding the industrial-strength components it orchestrates while still providing their essential power.
67
+ * **common cases are simple, complex cases are always possible**: The code should be as simple as the story it tells. For the things you do all the time, you only need to describe how this data flow is different from all other data flows. However, at no point in the project lifecycle should Wukong/Hanuman hit a brick wall or peat bog requiring its total replacement. A complex production system may, for example, require that you replace a critical path with custom Java code -- but that's a small set of substitutions in an otherwise stable, scalable graph. In the world of web programming, Ruby on Rails passes this test; Sinatra and Drupal do not.
68
+ * **petabyte rule**: Wukong/Hanuman coordinate industrial-strength components that wort at terabyte- and petabyte-scale. Conceptual simplicity makes it an excellent tool even for small jobs, but scalability is key. All components must assume an asynchronous, unreliable and distributed system.
69
+ * **laptop rule**:
70
+ * **no dark magick**: the core libraries provide *elegant, predictable magic or no magic at all*. We use metaprogramming heavily, but always predictably, and only in service of making common cases simple.
71
+ - Soupy multi-option `case` statements are a smell.
72
+ - Complex tasks will require code that is more explicit, but readable and organically connected to the typical usage. For example, many data flows will require a custom `Wukong::Streamer` class; but that class is no more complex than the built-in streamer models and receives all the same sugar methods they do.
73
+ * **get shit done**: sometimes ugly tasks require ugly solutions. Shelling out to the hadoop process monitor and parsing its output is acceptable if it is robust and obviates the need for a native protocol handler.
74
+ * **be clever early, boring late**: magic in service of having a terse language for assembling a graph is great. However, the assembled graph should be stomic and largely free of any conditional logic or dependencies.
75
+ - for example, the data flow `split` statement allows you to set a condition on each branch. The assembled graph, however, is typically a `fanout` stage followed by `filter` stages.
76
+ - the graph language has some helpers to refer to graph stages. The compiled graph uses explicit mostly-readable but unambiguous static handles.
77
+ - some stages offer light polymorphism -- for example, `select` accepts either a regexp or block. This is handled at the factory level, and the resulting stage is free of conditional logic.
78
+ * **no lock-in**: needless to say, Wukong works seamlessly with the Infochimps platform, making robust, reliable massive-scale dataflows amazingly simple. However, wukong flows are not tied to the cloud: they project to Hadoop, Flume or any of the other open-source components that power our platform.
79
+
80
+ __________________________________________________________________________
81
+
82
+ <a name="stage"></a>
83
+ ## Stage
84
+
85
+ A graph is composed of `stage`s.
86
+
87
+ * *desc* (alias `description`)
88
+
89
+ #### Actions
90
+
91
+ each action
92
+
93
+ * the default action is `call`
94
+ * all stages respond to `nothing`, and like ze goggles, do `nothing`.
95
+
96
+ __________________________________________________________________________
97
+
98
+ <a name="dataflows"></a>
99
+ ## Workflows
100
+
101
+ Wukong workflows work somewhat differently than you may be familiar with Rake and such.
102
+
103
+ In wukong, a stage corresponds to a product; you can then act on that product.
104
+
105
+ Consider first compiling a c program:
106
+
107
+ to build the executable, run `cc -o cake eggs.o milk.o flour.o sugar.o -I./include -L./lib`
108
+ to build files like '{file}.o', run `cc -c -o {file}.o {file}.c -I./include`
109
+
110
+ In this case, you define the *steps*, implying the products.
111
+
112
+
113
+ Something rake can't do (but we should be able to): make it so I can define a dependency that runs **last**
114
+
115
+ ### Defining jobs
116
+
117
+ Wukong.job(:launch) do
118
+ task :aim do
119
+ #...
120
+ end
121
+ task :enter do
122
+ end
123
+ task :commit do
124
+ # ...
125
+ end
126
+ end
127
+
128
+ Wukong.job(:recall) do
129
+ task :smash_with_rock do
130
+ #...
131
+ end
132
+ task :reprogram do
133
+ # ...
134
+ end
135
+ end
136
+
137
+ * stages construct products
138
+ - these have default actions
139
+ * hanuman tracks defined order
140
+
141
+ * do steps run in order, or is dependency explicit?
142
+ * what about idempotency?
143
+
144
+ * `task` vs `action` vs `product`; `job`, `task`, `group`, `namespace`.
145
+
146
+ ### documenting
147
+
148
+ Inline option (`:desc` or `:description`?)
149
+
150
+ ```ruby
151
+ task :foo, :description => "pity the foo" do
152
+ # ...
153
+ end
154
+ ```
155
+
156
+ DSL method option
157
+
158
+ ```ruby
159
+ task :foo do
160
+ description "pity the foo"
161
+ # ...
162
+ end
163
+ ```
164
+
165
+ ### actions
166
+
167
+ default action:
168
+
169
+ ```ruby
170
+ script 'nukes/launch_codes.rb' do
171
+ # ...
172
+ end
173
+ ```
174
+
175
+ define the `:undo` action:
176
+
177
+ ```ruby
178
+ script 'nukes/launch_codes.rb', :undo do
179
+ # ...
180
+ end
181
+ ```
182
+
183
+ <a name="file-name-templates"></a>
184
+ ### File name templates
185
+
186
+ * *timestamp*: timestamp of run. everything in this invocation will have the same timestamp.
187
+ * *user*: username; `ENV['USER']` by default
188
+ * *sources*: basenames of job inputs, minus extension, non-`\w` replaced with '_', joined by '-', max 50 chars.
189
+ * *job*: job flow name
190
+
191
+ <a name="job-versioning-of-clobbered"></a>
192
+ ### versioning of clobbered files
193
+
194
+ * when files are generated or removed, relocate to a timestamped location
195
+ - a file `/path/to/file.txt` is relocated to `~/.wukong/backups/path/to/file.txt.wukong-20110102120011` where `20110102120011` is the [job timestamp](#file-naming)
196
+ - accepts a `max_size` param
197
+ - raises if it can't write to directory -- must explicitly say `--safe_file_ops=false`
198
+
199
+ <a name="job-running"></a>
200
+ ### running
201
+
202
+ * `clobber` -- run, but clear all dependencies
203
+ * `undo` --
204
+ * `clean` --
205
+
206
+
207
+ ### Utility and Filesystem tasks
208
+
209
+ The primitives correspond heavily with Rake and Chef. However, they extend them in many ways, don't cover all their functionality in many ways, and incompatible in several ways.
210
+
211
+ ### Configuration
212
+
213
+
214
+ #### Commandline args
215
+
216
+ * handled by configliere: `nukes launch --launch_code=GLG20`
217
+
218
+ * TODO: configliere needs context-specific config vars, so I only get information about the `launch` action in the `nukes` job when I run `nukes launch --help`
219
+
220
+
221
+
222
+ __________________________________________________________________________
223
+
224
+ <a name="dataflows"></a>
225
+ ## Dataflows
226
+
227
+
228
+ Data flows
229
+
230
+ * you can have a consumer connect to a provider, or vice versa
231
+ - producer binds to a port, consumers connect to it: pub/sub
232
+ - consumers open a port, producer connects to many: megaphone
233
+
234
+ * you can bring the provider on line first, and the consumers later, or vice versa.
235
+
236
+
237
+ <a name="dataflow-syntax"></a>
238
+ ## Syntax
239
+
240
+ **note: this is a scratch pad; actual syntax evolving rapidly and currently looks not much like the following**
241
+
242
+ read('/foo/bar') # source( FileSource.new('/foo/bar') )
243
+ writes('/foo/bar') # sink( FileSink.new('/foo/bar') )
244
+
245
+ ... | file('/foo/bar') # this we know is a source
246
+ file('/foo/bar') | ... # this we know is a sink
247
+ file('/foo/bar') # don't know; maybe we can guess later
248
+
249
+ Here is an example Wukong script, `count_followers.rb`:
250
+
251
+ from :json
252
+
253
+ mapper do |user|
254
+ year_month = Time.parse(user[:created_at]).strftime("%Y%M")
255
+ emit [ user[:followers_count], year_month ]
256
+ end
257
+
258
+ reducer do
259
+ start{ @count = 0 }
260
+
261
+ each do |followers_count, year_month|
262
+ @count += 1
263
+ end
264
+
265
+ finally{ emit [*@group_key, @count] }
266
+ end
267
+
268
+ You can run this from the commandline:
269
+
270
+ wukong count_followers.rb users.json followers_histogram.tsv
271
+
272
+ It will run in local mode, effectively doing
273
+
274
+ cat users.json | {the map block} | sort | {the reduce block} > followers_histogram.tsv
275
+
276
+ You can instead run it in Hadoop mode, and it will launch the job across a distributed Hadoop cluster
277
+
278
+ wukong --run=hadoop count_followers.rb users.json followers_histogram.tsv
279
+
280
+ <a name="formatters"></a>
281
+ #### Data Formats (Serialization / Deserialization)
282
+
283
+ * tsv/csv
284
+ * json
285
+ * xml
286
+ * avro
287
+ * apache_log
288
+ * flat
289
+ * regexp
290
+ * [Tagged Netstrings](http://tnetstrings.org/)
291
+ * [ZeroMQ Property Language](http://rfc.zeromq.org/spec:4)
292
+
293
+ * gz/bz2/zip/snappy
294
+
295
+ <a name="data-packets"></a>
296
+ #### Data Packets
297
+
298
+ Data consists of
299
+
300
+ - record
301
+ - schema
302
+ - metadata
303
+
304
+ ## Delivery Guarantees
305
+
306
+ Most messaging systems keep metadata about what messages have been consumed on the broker. That is, as a message is handed out to a consumer, the broker records that fact locally. This is a fairly intuitive choice, and indeed for a single machine server it is not clear where else it could go. Since the data structure used for storage in many messaging systems scale poorly, this is also a pragmatic choice--since the broker knows what is consumed it can immediately delete it, keeping the data size small.
307
+
308
+ What is perhaps not obvious, is that getting the broker and consumer to come into agreement about what has been consumed is not a trivial problem. If the broker records a message as consumed immediately every time it is handed out over the network, then if the consumer fails to process the message (say because it crashes or the request times out or whatever) then that message will be lost. To solve this problem, many messaging systems add an acknowledgement feature which means that messages are only marked as sent not consumed when they are sent; the broker waits for a specific acknowledgement from the consumer to record the message as consumed. This strategy fixes the problem of losing messages, but creates new problems. First of all, if the consumer processes the message but fails before it can send an acknowledgement then the message will be consumed twice. The second problem is around performance, now the broker must keep multiple states about every single message (first to lock it so it is not given out a second time, and then to mark it as permanently consumed so that it can be removed). Tricky problems must be dealt with, like what to do with messages that are sent but never acknowledged.
309
+
310
+ So clearly there are multiple possible message delivery guarantees that could be provided:
311
+
312
+ * At most once—this handles the first case described. Messages are immediately marked as consumed, so they can't be given out twice, but many failure scenarios may lead to losing messages.
313
+ * At least once—this is the second case where we guarantee each message will be delivered at least once, but in failure cases may be delivered twice.
314
+ * Exactly once—this is what people actually want, each message is delivered once and only once.
315
+
316
+
317
+ __________________________________________________________________________
318
+
319
+ <a name="design-questions"></a>
320
+ ## Design Questions
321
+
322
+ * **filename helpers**:
323
+ - `':data_dir:/this/that/:script:-:user:-:timestamp:.:ext:'`?
324
+ - `path_to(:data_dir, 'this/that', "???")`?
325
+
326
+ * `class Wukong::Foo::Base` vs `class Wukong::Foo`
327
+ - the latter is more natural, and still allows
328
+ - I'd like
329
+
330
+
331
+
332
+ __________________________________________________________________________
333
+ __________________________________________________________________________
334
+ __________________________________________________________________________
335
+
336
+
337
+ <a name="references"></a>
338
+ ## References
339
+
340
+ <a name="refs-workflow"></a>
341
+ ### Workflow
342
+
343
+ * **Rake**
344
+
345
+ - [Rake Docs](http://rdoc.info/gems/rake/file/README.rdoc)
346
+ - [Rake Tutorial](http://jasonseifer.com/2010/04/06/rake-tutorial) by Jason Seifer -- 2010, with a good overview of why Rake is useful
347
+ - [Rake Tutorial](http://martinfowler.com/articles/rake.html) by Martin Fowler -- from 2005, so may lack some modernities
348
+ - [Rake Tutorial](http://onestepback.org/index.cgi/Tech/Rake/Tutorial/RakeTutorialRules.red) -- from 2005, so may lack some modernities
349
+
350
+ * **Rake Examples**
351
+
352
+ - [resque's redis.rake](https://github.com/defunkt/resque/blob/master/lib/tasks/redis.rake) and [resque/tasks](https://github.com/defunkt/resque/blob/master/lib/resque/tasks.rb)
353
+ - [rails' Rails Ties](https://github.com/rails/rails/tree/master/railties/lib/rails/tasks)
354
+
355
+ * **Thor**
356
+
357
+ - [Thor Wiki](https://github.com/wycats/thor/wiki)
358
+ -
359
+
360
+ * **Chef**
361
+
362
+ - [Chef Wiki](http://wiki.opscode.com/display/chef/Home)
363
+ - specifically, [Chef Resources](http://wiki.opscode.com/display/chef/Resources)
364
+
365
+ * **Other**
366
+
367
+ - [**Gradle**](http://gradle.org/) -- a modern take on `ant` + `maven`. The [Gradle overview](http://gradle.org/overview) states its case.
368
+
369
+ <a name="refs-dataflow"></a>
370
+ ### Dataflow
371
+
372
+ * **Esper**
373
+
374
+ - Must read: [StreamSQL Event Processing with Esper](http://www.igvita.com/2011/05/27/streamsql-event-processing-with-esper/)
375
+ - [Esper docs](http://esper.codehaus.org/esper-4.5.0/doc/reference/en/html_single/index.html#epl_clauses)
376
+ - [Esper EPL Reference](http://esper.codehaus.org/esper-4.5.0/doc/reference/en/html_single/index.html#epl_clauses)
377
+
378
+ * **Storm**
379
+
380
+ - [A Storm is coming: more details and plans for release](http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html)
381
+ - [Storm: distributed and fault-tolerant realtime computation](http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolerant-realtime-computation) -- slideshare presentation
382
+ - [Storm: the Hadoop of Realtime Processing](http://tech.backtype.com/preview-of-storm-the-hadoop-of-realtime-proce)
383
+
384
+ * **Kafka**: LinkedIn's high-throughput messaging queue
385
+
386
+ - [Kafka's Design: Why we built this](http://incubator.apache.org/kafka/design.html)
387
+
388
+ * **ZeroMQ**: tcp sockets like you think they should work
389
+
390
+ - [ZeroMQ: A Modern & Fast Networking Stack](http://www.igvita.com/2010/09/03/zeromq-modern-fast-networking-stack/)
391
+ - [ZeroMQ Guide](http://zguide.zeromq.org/page:all)
392
+ - [ZeroMQ: An Introduction](http://nichol.as/zeromq-an-introduction)
393
+ - [Routing with Ruby & ZeroMQ Devices](http://www.igvita.com/2010/11/17/routing-with-ruby-zeromq-devices/)
394
+ - [Ruby bindings for ZeroMQ](http://zeromq.github.com/rbzmq/) and the [Ruby-FFI bindings](http://www.zeromq.org/bindings:ruby-ffi)
395
+ - [Learn ruby ZeroMQ](https://github.com/andrewvc/learn-ruby-zeromq) by @andrewvc
396
+
397
+ * **Other**
398
+
399
+ - [Infopipes: An abstraction for multimedia streamin](http://web.cecs.pdx.edu/~black/publications/Mms062%203rd%20try.pdf) Black et al 2002
400
+ - [Yahoo Pipes](http://pipes.yahoo.com/pipes/)
401
+ - [Yahoo Pipes wikipedia page](http://en.wikipedia.org/wiki/Yahoo_Pipes)
402
+ - [Streambase](http://www.streambase.com/products/streambasecep/faqs/) -- Why is is so goddamn hard to find out anything real about a project once it gets an enterprise version? Seriously, the consistent fundamental brokenness of enterprise product is astonishing. It's like they take inspiration from shitty major-label band websites but layer a whiteout of [web jargon bullshit](http://www.dack.com/web/bullshit.html) in place of inessential flash animation. Anyway I think Streambase is kinda similar but who the hell can tell.
403
+ - [Scribe](http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/)
404
+ - [Splunk Case Study](http://www.igvita.com/2008/10/22/distributed-logging-syslog-ng-splunk/)
405
+
406
+ <a name="refs-dataflow"></a>
407
+ ### Messaging Queue
408
+
409
+ - [DripDrop](https://github.com/andrewvc/dripdrop) - a message passing library with a unified API abstracting HTTP, zeroMQ and websockets.
410
+
411
+
412
+ <a name="refs-dataflow"></a>
413
+ ### Data Processing
414
+
415
+ * **Hadoop**
416
+
417
+ - [Hadoop]()
418
+
419
+
420
+ * **Spark/Mesos**
421
+
422
+ - [Mesos](http://www.mesosproject.org/)