wayfarer 0.0.3 → 0.4.3

Sign up to get free protection for your applications and to get access to all the features.
Files changed (369) hide show
  1. checksums.yaml +5 -5
  2. data/.github/workflows/ci.yaml +32 -0
  3. data/.gitignore +3 -4
  4. data/.rubocop.yml +25 -9
  5. data/.ruby-version +1 -1
  6. data/Dockerfile +5 -0
  7. data/Gemfile +1 -7
  8. data/Gemfile.lock +221 -0
  9. data/RELEASING.md +17 -0
  10. data/Rakefile +38 -90
  11. data/bin/wayfarer +1 -111
  12. data/docker-compose.yml +32 -0
  13. data/docs/cookbook/batch_routing.md +22 -0
  14. data/docs/cookbook/consent_screen.md +36 -0
  15. data/docs/cookbook/executing_javascript.md +41 -0
  16. data/docs/cookbook/querying_html.md +42 -0
  17. data/docs/cookbook/screenshots.md +27 -0
  18. data/docs/cookbook/user_agent.md +7 -0
  19. data/docs/guides/browser_automation/capybara.md +69 -0
  20. data/docs/guides/browser_automation/custom_adapters.md +100 -0
  21. data/docs/guides/browser_automation/ferrum.md +39 -0
  22. data/docs/guides/browser_automation/selenium.md +63 -0
  23. data/docs/guides/callbacks.md +131 -31
  24. data/docs/guides/configuration.md +24 -169
  25. data/docs/guides/debugging.md +17 -0
  26. data/docs/guides/error_handling.md +30 -45
  27. data/docs/guides/jobs.md +101 -0
  28. data/docs/guides/navigation.md +73 -0
  29. data/docs/guides/networking.md +94 -0
  30. data/docs/guides/pages.md +52 -0
  31. data/docs/guides/performance.md +130 -0
  32. data/docs/guides/reliability.md +41 -0
  33. data/docs/guides/routing/steering.md +30 -0
  34. data/docs/guides/tasks.md +14 -0
  35. data/docs/index.md +40 -66
  36. data/docs/reference/api/base.md +48 -0
  37. data/docs/reference/api/route.md +182 -0
  38. data/docs/reference/cli.md +61 -0
  39. data/docs/reference/configuration_keys.md +42 -0
  40. data/docs/reference/environment_variables.md +83 -0
  41. data/lib/wayfarer/base.rb +50 -0
  42. data/lib/wayfarer/callbacks.rb +71 -0
  43. data/lib/wayfarer/cli/base.rb +27 -0
  44. data/lib/wayfarer/cli/generate.rb +17 -0
  45. data/lib/wayfarer/cli/job.rb +60 -0
  46. data/lib/wayfarer/cli/route.rb +29 -0
  47. data/lib/wayfarer/cli/route_printer.rb +116 -0
  48. data/lib/wayfarer/cli/runner.rb +34 -0
  49. data/lib/wayfarer/cli/templates/Gemfile.tt +5 -0
  50. data/lib/wayfarer/cli/templates/job.rb.tt +10 -0
  51. data/lib/wayfarer/config/capybara.rb +10 -0
  52. data/lib/wayfarer/config/ferrum.rb +11 -0
  53. data/lib/wayfarer/config/networking.rb +26 -0
  54. data/lib/wayfarer/config/redis.rb +14 -0
  55. data/lib/wayfarer/config/root.rb +11 -0
  56. data/lib/wayfarer/config/selenium.rb +21 -0
  57. data/lib/wayfarer/config/strconv.rb +45 -0
  58. data/lib/wayfarer/config/struct.rb +72 -0
  59. data/lib/wayfarer/gc.rb +15 -0
  60. data/lib/wayfarer/middleware/chain.rb +19 -0
  61. data/lib/wayfarer/middleware/dedup.rb +25 -0
  62. data/lib/wayfarer/middleware/fetch.rb +47 -0
  63. data/lib/wayfarer/middleware/normalize.rb +25 -0
  64. data/lib/wayfarer/middleware/router.rb +53 -0
  65. data/lib/wayfarer/middleware/stage.rb +23 -0
  66. data/lib/wayfarer/middleware/worker.rb +30 -0
  67. data/lib/wayfarer/networking/capybara.rb +28 -0
  68. data/lib/wayfarer/networking/context.rb +36 -0
  69. data/lib/wayfarer/networking/ferrum.rb +35 -0
  70. data/lib/wayfarer/networking/http.rb +34 -0
  71. data/lib/wayfarer/networking/pool.rb +40 -0
  72. data/lib/wayfarer/networking/result.rb +18 -0
  73. data/lib/wayfarer/networking/selenium.rb +43 -0
  74. data/lib/wayfarer/networking/strategy.rb +38 -0
  75. data/lib/wayfarer/page.rb +17 -74
  76. data/lib/wayfarer/parsing/json.rb +17 -0
  77. data/lib/wayfarer/parsing/xml.rb +17 -0
  78. data/lib/wayfarer/redis/.#barrier.rb +1 -0
  79. data/lib/wayfarer/redis/barrier.rb +36 -0
  80. data/lib/wayfarer/redis/connection.rb +13 -0
  81. data/lib/wayfarer/redis/counter.rb +29 -0
  82. data/lib/wayfarer/redis/pool.rb +20 -0
  83. data/lib/wayfarer/redis/version.rb +19 -0
  84. data/lib/wayfarer/routing/dsl.rb +57 -0
  85. data/lib/wayfarer/routing/matchers/custom.rb +25 -0
  86. data/lib/wayfarer/routing/matchers/host.rb +19 -0
  87. data/lib/wayfarer/routing/matchers/path.rb +49 -0
  88. data/lib/wayfarer/routing/matchers/query.rb +63 -0
  89. data/lib/wayfarer/routing/matchers/scheme.rb +17 -0
  90. data/lib/wayfarer/routing/matchers/suffix.rb +17 -0
  91. data/lib/wayfarer/routing/matchers/url.rb +17 -0
  92. data/lib/wayfarer/routing/path_finder.rb +46 -0
  93. data/lib/wayfarer/routing/result.rb +15 -0
  94. data/lib/wayfarer/routing/root_route.rb +7 -0
  95. data/lib/wayfarer/routing/route.rb +47 -0
  96. data/lib/wayfarer/routing/router.rb +10 -54
  97. data/lib/wayfarer/routing/target_route.rb +7 -0
  98. data/lib/wayfarer/serializer.rb +17 -0
  99. data/lib/wayfarer/stringify.rb +47 -0
  100. data/lib/wayfarer/task.rb +34 -0
  101. data/lib/wayfarer.rb +48 -57
  102. data/mkdocs.yml +47 -0
  103. data/requirements.txt +1 -0
  104. data/spec/base_spec.rb +233 -0
  105. data/spec/callbacks_spec.rb +102 -0
  106. data/spec/cli/generate_spec.rb +39 -0
  107. data/spec/cli/job_spec.rb +74 -0
  108. data/spec/cli/version_spec.rb +13 -0
  109. data/spec/config/capybara_spec.rb +18 -0
  110. data/spec/config/ferrum_spec.rb +24 -0
  111. data/spec/config/networking_spec.rb +73 -0
  112. data/spec/config/redis_spec.rb +32 -0
  113. data/spec/config/root_spec.rb +31 -0
  114. data/spec/config/selenium_spec.rb +56 -0
  115. data/spec/config/strconv_spec.rb +58 -0
  116. data/spec/config/struct_spec.rb +66 -0
  117. data/spec/factories/middleware.rb +15 -0
  118. data/spec/factories/page.rb +78 -0
  119. data/spec/factories/task.rb +12 -0
  120. data/spec/fixtures/dummy_job.rb +7 -0
  121. data/spec/gc_spec.rb +63 -0
  122. data/spec/middleware/chain_spec.rb +96 -0
  123. data/spec/middleware/dedup_spec.rb +76 -0
  124. data/spec/middleware/fetch_spec.rb +100 -0
  125. data/spec/middleware/normalize_spec.rb +28 -0
  126. data/spec/middleware/router_spec.rb +80 -0
  127. data/spec/middleware/stage_spec.rb +39 -0
  128. data/spec/middleware/worker_spec.rb +117 -0
  129. data/spec/networking/capybara_spec.rb +12 -0
  130. data/spec/networking/context_spec.rb +127 -0
  131. data/spec/networking/ferrum_spec.rb +12 -0
  132. data/spec/networking/http_spec.rb +12 -0
  133. data/spec/networking/pool_spec.rb +67 -0
  134. data/spec/networking/selenium_spec.rb +12 -0
  135. data/spec/networking/strategy.rb +170 -0
  136. data/spec/page_spec.rb +21 -12
  137. data/spec/{parsers/json_parser_spec.rb → parsing/json_spec.rb} +5 -4
  138. data/spec/{parsers/xml_parser_spec.rb → parsing/xml_spec.rb} +3 -2
  139. data/spec/redis/barrier_spec.rb +78 -0
  140. data/spec/redis/counter_spec.rb +32 -0
  141. data/spec/redis/pool_spec.rb +18 -0
  142. data/spec/redis/version_spec.rb +13 -0
  143. data/spec/routing/dsl_spec.rb +98 -0
  144. data/spec/routing/integration_spec.rb +110 -0
  145. data/spec/routing/matchers/custom_spec.rb +31 -0
  146. data/spec/routing/matchers/host_spec.rb +49 -0
  147. data/spec/routing/matchers/path_spec.rb +43 -0
  148. data/spec/routing/matchers/query_spec.rb +137 -0
  149. data/spec/routing/matchers/scheme_spec.rb +25 -0
  150. data/spec/routing/{filetypes_rule_spec.rb → matchers/suffix_spec.rb} +14 -13
  151. data/spec/routing/matchers/uri_spec.rb +27 -0
  152. data/spec/routing/path_finder_spec.rb +33 -0
  153. data/spec/routing/root_route_spec.rb +29 -0
  154. data/spec/routing/route_spec.rb +43 -0
  155. data/spec/routing/router_spec.rb +13 -56
  156. data/spec/spec_helpers.rb +73 -38
  157. data/spec/stringify_spec.rb +23 -0
  158. data/{support → spec/support}/static/finders.html +0 -0
  159. data/{support → spec/support}/static/graph/details/a.html +0 -0
  160. data/{support → spec/support}/static/graph/details/b.html +0 -0
  161. data/{support → spec/support}/static/graph/index.html +0 -0
  162. data/{support → spec/support}/static/json/dummy.json +0 -0
  163. data/{support → spec/support}/static/links/links.html +0 -0
  164. data/{support → spec/support}/static/xml/dummy.xml +0 -0
  165. data/{support → spec/support}/test_app.rb +9 -2
  166. data/spec/task_spec.rb +27 -0
  167. data/spec/wayfarer_spec.rb +2 -13
  168. data/wayfarer.gemspec +40 -42
  169. metadata +234 -361
  170. data/.travis.yml +0 -5
  171. data/Changelog.md +0 -10
  172. data/README.md +0 -21
  173. data/benchmark/frontiers.rb +0 -143
  174. data/docs/.gitignore +0 -2
  175. data/docs/_config.yml +0 -15
  176. data/docs/_includes/base.html +0 -7
  177. data/docs/_includes/head.html +0 -10
  178. data/docs/_includes/navigation.html +0 -187
  179. data/docs/_layouts/default.html +0 -42
  180. data/docs/_sass/base.scss +0 -439
  181. data/docs/_sass/variables.scss +0 -24
  182. data/docs/_sass/vendor/bourbon/_bourbon-deprecate.scss +0 -19
  183. data/docs/_sass/vendor/bourbon/_bourbon-deprecated-upcoming.scss +0 -425
  184. data/docs/_sass/vendor/bourbon/_bourbon.scss +0 -90
  185. data/docs/_sass/vendor/bourbon/addons/_border-color.scss +0 -29
  186. data/docs/_sass/vendor/bourbon/addons/_border-radius.scss +0 -48
  187. data/docs/_sass/vendor/bourbon/addons/_border-style.scss +0 -28
  188. data/docs/_sass/vendor/bourbon/addons/_border-width.scss +0 -28
  189. data/docs/_sass/vendor/bourbon/addons/_buttons.scss +0 -69
  190. data/docs/_sass/vendor/bourbon/addons/_clearfix.scss +0 -25
  191. data/docs/_sass/vendor/bourbon/addons/_ellipsis.scss +0 -30
  192. data/docs/_sass/vendor/bourbon/addons/_font-stacks.scss +0 -31
  193. data/docs/_sass/vendor/bourbon/addons/_hide-text.scss +0 -27
  194. data/docs/_sass/vendor/bourbon/addons/_margin.scss +0 -29
  195. data/docs/_sass/vendor/bourbon/addons/_padding.scss +0 -29
  196. data/docs/_sass/vendor/bourbon/addons/_position.scss +0 -51
  197. data/docs/_sass/vendor/bourbon/addons/_prefixer.scss +0 -66
  198. data/docs/_sass/vendor/bourbon/addons/_retina-image.scss +0 -27
  199. data/docs/_sass/vendor/bourbon/addons/_size.scss +0 -56
  200. data/docs/_sass/vendor/bourbon/addons/_text-inputs.scss +0 -118
  201. data/docs/_sass/vendor/bourbon/addons/_timing-functions.scss +0 -34
  202. data/docs/_sass/vendor/bourbon/addons/_triangle.scss +0 -63
  203. data/docs/_sass/vendor/bourbon/addons/_word-wrap.scss +0 -29
  204. data/docs/_sass/vendor/bourbon/css3/_animation.scss +0 -61
  205. data/docs/_sass/vendor/bourbon/css3/_appearance.scss +0 -5
  206. data/docs/_sass/vendor/bourbon/css3/_backface-visibility.scss +0 -5
  207. data/docs/_sass/vendor/bourbon/css3/_background-image.scss +0 -44
  208. data/docs/_sass/vendor/bourbon/css3/_background.scss +0 -57
  209. data/docs/_sass/vendor/bourbon/css3/_border-image.scss +0 -61
  210. data/docs/_sass/vendor/bourbon/css3/_calc.scss +0 -6
  211. data/docs/_sass/vendor/bourbon/css3/_columns.scss +0 -67
  212. data/docs/_sass/vendor/bourbon/css3/_filter.scss +0 -6
  213. data/docs/_sass/vendor/bourbon/css3/_flex-box.scss +0 -327
  214. data/docs/_sass/vendor/bourbon/css3/_font-face.scss +0 -29
  215. data/docs/_sass/vendor/bourbon/css3/_font-feature-settings.scss +0 -6
  216. data/docs/_sass/vendor/bourbon/css3/_hidpi-media-query.scss +0 -12
  217. data/docs/_sass/vendor/bourbon/css3/_hyphens.scss +0 -6
  218. data/docs/_sass/vendor/bourbon/css3/_image-rendering.scss +0 -15
  219. data/docs/_sass/vendor/bourbon/css3/_keyframes.scss +0 -38
  220. data/docs/_sass/vendor/bourbon/css3/_linear-gradient.scss +0 -40
  221. data/docs/_sass/vendor/bourbon/css3/_perspective.scss +0 -12
  222. data/docs/_sass/vendor/bourbon/css3/_placeholder.scss +0 -10
  223. data/docs/_sass/vendor/bourbon/css3/_radial-gradient.scss +0 -40
  224. data/docs/_sass/vendor/bourbon/css3/_selection.scss +0 -44
  225. data/docs/_sass/vendor/bourbon/css3/_text-decoration.scss +0 -27
  226. data/docs/_sass/vendor/bourbon/css3/_transform.scss +0 -21
  227. data/docs/_sass/vendor/bourbon/css3/_transition.scss +0 -81
  228. data/docs/_sass/vendor/bourbon/css3/_user-select.scss +0 -5
  229. data/docs/_sass/vendor/bourbon/functions/_assign-inputs.scss +0 -16
  230. data/docs/_sass/vendor/bourbon/functions/_contains-falsy.scss +0 -25
  231. data/docs/_sass/vendor/bourbon/functions/_contains.scss +0 -31
  232. data/docs/_sass/vendor/bourbon/functions/_is-length.scss +0 -16
  233. data/docs/_sass/vendor/bourbon/functions/_is-light.scss +0 -26
  234. data/docs/_sass/vendor/bourbon/functions/_is-number.scss +0 -16
  235. data/docs/_sass/vendor/bourbon/functions/_is-size.scss +0 -23
  236. data/docs/_sass/vendor/bourbon/functions/_modular-scale.scss +0 -74
  237. data/docs/_sass/vendor/bourbon/functions/_px-to-em.scss +0 -24
  238. data/docs/_sass/vendor/bourbon/functions/_px-to-rem.scss +0 -26
  239. data/docs/_sass/vendor/bourbon/functions/_shade.scss +0 -24
  240. data/docs/_sass/vendor/bourbon/functions/_strip-units.scss +0 -22
  241. data/docs/_sass/vendor/bourbon/functions/_tint.scss +0 -24
  242. data/docs/_sass/vendor/bourbon/functions/_transition-property-name.scss +0 -37
  243. data/docs/_sass/vendor/bourbon/functions/_unpack.scss +0 -32
  244. data/docs/_sass/vendor/bourbon/helpers/_convert-units.scss +0 -26
  245. data/docs/_sass/vendor/bourbon/helpers/_directional-values.scss +0 -108
  246. data/docs/_sass/vendor/bourbon/helpers/_font-source-declaration.scss +0 -53
  247. data/docs/_sass/vendor/bourbon/helpers/_gradient-positions-parser.scss +0 -24
  248. data/docs/_sass/vendor/bourbon/helpers/_linear-angle-parser.scss +0 -35
  249. data/docs/_sass/vendor/bourbon/helpers/_linear-gradient-parser.scss +0 -51
  250. data/docs/_sass/vendor/bourbon/helpers/_linear-positions-parser.scss +0 -77
  251. data/docs/_sass/vendor/bourbon/helpers/_linear-side-corner-parser.scss +0 -41
  252. data/docs/_sass/vendor/bourbon/helpers/_radial-arg-parser.scss +0 -74
  253. data/docs/_sass/vendor/bourbon/helpers/_radial-gradient-parser.scss +0 -55
  254. data/docs/_sass/vendor/bourbon/helpers/_radial-positions-parser.scss +0 -28
  255. data/docs/_sass/vendor/bourbon/helpers/_render-gradients.scss +0 -31
  256. data/docs/_sass/vendor/bourbon/helpers/_shape-size-stripper.scss +0 -15
  257. data/docs/_sass/vendor/bourbon/helpers/_str-to-num.scss +0 -55
  258. data/docs/_sass/vendor/bourbon/settings/_asset-pipeline.scss +0 -7
  259. data/docs/_sass/vendor/bourbon/settings/_deprecation-warnings.scss +0 -8
  260. data/docs/_sass/vendor/bourbon/settings/_prefixer.scss +0 -9
  261. data/docs/_sass/vendor/bourbon/settings/_px-to-em.scss +0 -1
  262. data/docs/_sass/vendor/neat/_neat-helpers.scss +0 -11
  263. data/docs/_sass/vendor/neat/_neat.scss +0 -23
  264. data/docs/_sass/vendor/neat/functions/_new-breakpoint.scss +0 -49
  265. data/docs/_sass/vendor/neat/functions/_private.scss +0 -114
  266. data/docs/_sass/vendor/neat/grid/_box-sizing.scss +0 -15
  267. data/docs/_sass/vendor/neat/grid/_direction-context.scss +0 -33
  268. data/docs/_sass/vendor/neat/grid/_display-context.scss +0 -28
  269. data/docs/_sass/vendor/neat/grid/_fill-parent.scss +0 -22
  270. data/docs/_sass/vendor/neat/grid/_media.scss +0 -92
  271. data/docs/_sass/vendor/neat/grid/_omega.scss +0 -87
  272. data/docs/_sass/vendor/neat/grid/_outer-container.scss +0 -34
  273. data/docs/_sass/vendor/neat/grid/_pad.scss +0 -25
  274. data/docs/_sass/vendor/neat/grid/_private.scss +0 -35
  275. data/docs/_sass/vendor/neat/grid/_row.scss +0 -52
  276. data/docs/_sass/vendor/neat/grid/_shift.scss +0 -50
  277. data/docs/_sass/vendor/neat/grid/_span-columns.scss +0 -94
  278. data/docs/_sass/vendor/neat/grid/_to-deprecate.scss +0 -97
  279. data/docs/_sass/vendor/neat/grid/_visual-grid.scss +0 -42
  280. data/docs/_sass/vendor/neat/mixins/_clearfix.scss +0 -25
  281. data/docs/_sass/vendor/neat/settings/_disable-warnings.scss +0 -13
  282. data/docs/_sass/vendor/neat/settings/_grid.scss +0 -51
  283. data/docs/_sass/vendor/neat/settings/_visual-grid.scss +0 -27
  284. data/docs/_sass/vendor/normalize-3.0.2.scss +0 -427
  285. data/docs/_sass/vendor/pygments.scss +0 -356
  286. data/docs/automating_browsers/capybara.md +0 -70
  287. data/docs/css/screen.scss +0 -7
  288. data/docs/guides/cli.md +0 -52
  289. data/docs/guides/frontiers.md +0 -93
  290. data/docs/guides/halting.md +0 -23
  291. data/docs/guides/job_queues.md +0 -26
  292. data/docs/guides/locals.md +0 -36
  293. data/docs/guides/logging.md +0 -22
  294. data/docs/guides/page_objects.md +0 -67
  295. data/docs/guides/peeking.md +0 -46
  296. data/docs/guides/selenium_capybara.md +0 -100
  297. data/docs/guides/tutorial.md +0 -452
  298. data/docs/js/navigation.js +0 -11
  299. data/docs/misc/contributing.md +0 -20
  300. data/docs/misc/testing.md +0 -11
  301. data/docs/recipes/authentication.md +0 -23
  302. data/docs/recipes/csv.md +0 -29
  303. data/docs/recipes/javascript.md +0 -20
  304. data/docs/recipes/multiple_uris.md +0 -18
  305. data/docs/recipes/screenshots.md +0 -20
  306. data/docs/routing/custom_rules.md +0 -16
  307. data/docs/routing/filetypes_rules.md +0 -21
  308. data/docs/routing/host_rules.md +0 -24
  309. data/docs/routing/path_rules.md +0 -33
  310. data/docs/routing/protocol_rules.md +0 -17
  311. data/docs/routing/query_rules.md +0 -69
  312. data/docs/routing/routes.md +0 -96
  313. data/docs/routing/uri_rules.md +0 -18
  314. data/examples/collect_github_issues.rb +0 -65
  315. data/examples/find_foobar_on_wikipedia.rb +0 -23
  316. data/lib/wayfarer/configuration.rb +0 -86
  317. data/lib/wayfarer/crawl.rb +0 -79
  318. data/lib/wayfarer/crawl_observer.rb +0 -103
  319. data/lib/wayfarer/dispatcher.rb +0 -104
  320. data/lib/wayfarer/finders.rb +0 -61
  321. data/lib/wayfarer/frontiers/frontier.rb +0 -79
  322. data/lib/wayfarer/frontiers/memory_bloomfilter.rb +0 -32
  323. data/lib/wayfarer/frontiers/memory_frontier.rb +0 -76
  324. data/lib/wayfarer/frontiers/memory_trie_frontier.rb +0 -39
  325. data/lib/wayfarer/frontiers/normalize_uris.rb +0 -48
  326. data/lib/wayfarer/frontiers/redis_bloomfilter.rb +0 -34
  327. data/lib/wayfarer/frontiers/redis_frontier.rb +0 -83
  328. data/lib/wayfarer/http_adapters/adapter_pool.rb +0 -62
  329. data/lib/wayfarer/http_adapters/net_http_adapter.rb +0 -77
  330. data/lib/wayfarer/http_adapters/selenium_adapter.rb +0 -80
  331. data/lib/wayfarer/job.rb +0 -211
  332. data/lib/wayfarer/locals.rb +0 -40
  333. data/lib/wayfarer/parsers/json_parser.rb +0 -20
  334. data/lib/wayfarer/parsers/xml_parser.rb +0 -27
  335. data/lib/wayfarer/processor.rb +0 -103
  336. data/lib/wayfarer/routing/custom_rule.rb +0 -21
  337. data/lib/wayfarer/routing/filetypes_rule.rb +0 -20
  338. data/lib/wayfarer/routing/host_rule.rb +0 -19
  339. data/lib/wayfarer/routing/path_rule.rb +0 -54
  340. data/lib/wayfarer/routing/protocol_rule.rb +0 -21
  341. data/lib/wayfarer/routing/query_rule.rb +0 -59
  342. data/lib/wayfarer/routing/rule.rb +0 -114
  343. data/lib/wayfarer/routing/uri_rule.rb +0 -21
  344. data/spec/configuration_spec.rb +0 -26
  345. data/spec/crawl_spec.rb +0 -48
  346. data/spec/finders_spec.rb +0 -49
  347. data/spec/frontiers/memory_bloomfilter_spec.rb +0 -6
  348. data/spec/frontiers/memory_frontier_spec.rb +0 -6
  349. data/spec/frontiers/memory_trie_frontier_spec.rb +0 -6
  350. data/spec/frontiers/normalize_uris_spec.rb +0 -59
  351. data/spec/frontiers/redis_bloomfilter_spec.rb +0 -6
  352. data/spec/frontiers/redis_frontier_spec.rb +0 -6
  353. data/spec/http_adapters/adapter_pool_spec.rb +0 -33
  354. data/spec/http_adapters/net_http_adapter_spec.rb +0 -83
  355. data/spec/http_adapters/selenium_adapter_spec.rb +0 -53
  356. data/spec/integration/callbacks_spec.rb +0 -42
  357. data/spec/integration/locals_spec.rb +0 -106
  358. data/spec/integration/peeking_spec.rb +0 -61
  359. data/spec/job_spec.rb +0 -122
  360. data/spec/processor_spec.rb +0 -31
  361. data/spec/routing/custom_rule_spec.rb +0 -26
  362. data/spec/routing/host_rule_spec.rb +0 -48
  363. data/spec/routing/path_rule_spec.rb +0 -66
  364. data/spec/routing/protocol_rule_spec.rb +0 -26
  365. data/spec/routing/query_rule_spec.rb +0 -124
  366. data/spec/routing/rule_spec.rb +0 -251
  367. data/spec/routing/uri_rule_spec.rb +0 -24
  368. data/spec/shared/frontier.rb +0 -96
  369. data/wayfarer-jruby.gemspec +0 -49
@@ -0,0 +1,130 @@
1
+ # Performance
2
+
3
+ How to write performant crawlers with Wayfarer.
4
+
5
+ ## Use a sufficiently sized user agent pool
6
+
7
+ Automated browser processes or HTTP clients are kept in a [connection pool]() of
8
+ static size. This avoids having to re-establish browser processes and enables
9
+ their reuse.
10
+
11
+ If the size of the pool is too small, the pool is a
12
+ bottleneck. For example, if your message queue adapter uses 8 threads, but the
13
+ pool only contains 1 user agent, the remaining 7 threads block until the agent
14
+ is checked back in to the pool for use by one of the blocked threads.
15
+
16
+ There is no reliable way to detect the number of threads of the underlying
17
+ message queue adapter. The pool size should equal the number of threads;
18
+
19
+ ```ruby
20
+ Wayfarer.config.network.pool_size = 8 # defaults to 1
21
+ ```
22
+
23
+ ### Job shedding
24
+
25
+ There is a maximum number of seconds that jobs wait when checking out a user
26
+ agent from the pool. Once this time is exceeded,
27
+ a `Wayfarer::UserAgentTimeoutError` is raised. By default, the timeout is 10
28
+ seconds.
29
+
30
+ This hints there are more threads in use than user agents in the pool.
31
+
32
+ ## Stage less URLs
33
+
34
+ Staging less URLs saves space and time:
35
+
36
+ * Less tasks written to the message queue
37
+ * Less time spent consuming tasks
38
+ * Less time spent filtering URLs with Redis
39
+
40
+ Wayfarer maintains a set of processed URLs for a batch in Redis. Every staged
41
+ URL is checked for inclusion in this set before it gets appended as a task to
42
+ the message queue.
43
+
44
+ A common pattern is to stage all links of a page, and rely on routing to fetch
45
+ only the relevant ones:
46
+
47
+ ```ruby
48
+ class DummyJob < Wayfarer::Base
49
+ route { to: index, host: "example.com" }
50
+
51
+ def index
52
+ stage page.meta.links.all
53
+ end
54
+ end
55
+ ```
56
+
57
+ Pages commonly contain a large number of URLs.
58
+
59
+ Every staged URL is:
60
+
61
+ 1. Normalized to a canonical form, for example by sorting query parameters
62
+ alphabetically.
63
+ 2. Checked for inclusion in the batch Redis set or discarded.
64
+ 3. Written to the message queue.
65
+ 4. Consumed from the queue and matched against the router.
66
+ 5. Fetched, if a route matches.
67
+
68
+ Narrowing down the links in the document to follow speeds up the process.
69
+ For example using Nokogiri, interesting links can be identified with a CSS
70
+ selector:
71
+
72
+ ```ruby
73
+ class DummyJob < Wayfarer::Base
74
+ route { to: index, host: "example.com" }
75
+
76
+ def index
77
+ stage interesting_links
78
+ end
79
+
80
+ private
81
+
82
+ def interesting_links
83
+ page.doc.css("a.interesting").map { |elem| elem["href"] }
84
+ end
85
+ end
86
+ ```
87
+
88
+ Because the router only accepts the single hostname `example.com`, the job can
89
+ also ensure it stages only internal URLs by intersecting them with the
90
+ interesting ones:
91
+
92
+ ```ruby
93
+ class DummyJob < Wayfarer::Base
94
+ route { to: index, host: "example.com" }
95
+
96
+ def index
97
+ stage interesting_internal_links
98
+ end
99
+
100
+ private
101
+
102
+ def interesting_internal_links
103
+ page.meta.links.internal & interesting_links
104
+ end
105
+
106
+ def interesting_links
107
+ page.doc.css("a.interesting").map { |elem| elem["href"] }
108
+ end
109
+ end
110
+ ```
111
+
112
+
113
+ ## Use Redis >= 6.2.0
114
+
115
+ Redis 6.2.0 introduced the
116
+ [`SMISMEMBER`](https://redis.io/commands/smismember) command which enables
117
+ Wayfarer to check whether multiple URLs have been processed in a batch with a
118
+ single command. With earlier versions, one command per URL is required.
119
+
120
+ Wayfarer detects the Redis server version and uses `SMISMEMBER` without user
121
+ configuration when supported.
122
+
123
+ ## Use Oj for JSON parsing
124
+
125
+ Wayfarer uses [Oj](https://github.com/ohler55/oj) for JSON parsing if the gem
126
+ has been required at runtime:
127
+
128
+ ```ruby
129
+ require "oj"
130
+ ```
@@ -0,0 +1,41 @@
1
+ # Reliablity
2
+
3
+ ## Durability
4
+
5
+ Wayfarer executes atop reliable messages queues such as Sidekiq, Resque,
6
+ RabbitMQ, etc. Its configuration is independent of the underlying queue
7
+ infrastructure it reads from and writes to.
8
+
9
+ ## Self-healing user agents
10
+
11
+ Wayfarer handles the scenario where a remote browser process has crashed and
12
+ must be replaced by a fresh browser process.
13
+
14
+ This can be tested locally by automating a browser with headless mode turned
15
+ off, and then closing the opened browser window: The current job fails, but the
16
+ next job has access to a newly established browser session again.
17
+
18
+ For example Ferrum might raise `Ferrum::DeadBrowserError`. Wayfarer's
19
+ user agents are self-healing and react to these kinds of errors internally. When
20
+ a browser window is closed, the Ferrum user agent attempts to establish a new
21
+ browser process as a replacement, for the next job to use.
22
+
23
+ [Wayfarer never swallows exceptions](/guides/error_handling). This means
24
+ that even though the user agent might heal itself, jobs still need to explicitly
25
+ retry browser errors:
26
+
27
+ ```ruby
28
+ class Foobar < Wayfarer::Base
29
+ route { to: :index }
30
+
31
+ retry_on Ferrum::DeadBrowserError, attempts: 3, wait: :exponentially_longer
32
+
33
+ # ...
34
+ end
35
+ ```
36
+
37
+ This leads to log entries like:
38
+
39
+ ```
40
+ Retrying DummyJob in 3 seconds, due to a Ferrum::DeadBrowserError.
41
+ ```
@@ -0,0 +1,30 @@
1
+ # Steering
2
+
3
+ A job's router can receive arguments computed dynamically by `::steer`.
4
+ Steering enables [batch routing](/cookbook/batch_routing).
5
+
6
+ For example, the following router has hostname and path hard-coded:
7
+
8
+ ```ruby
9
+ class DummyJob < Wayfarer::Base
10
+ route do
11
+ host "example.com", path: "/contact", to: :index
12
+ end
13
+ end
14
+ ```
15
+
16
+ Instead, hostname and path could be provided by `::steer`, too:
17
+
18
+ ```ruby
19
+ class DummyJob < Wayfarer::Base
20
+ route do |hostname, path|
21
+ host hostname, path: path, to: :index
22
+ end
23
+
24
+ steer do |_task|
25
+ ["example.com", "/contact"]
26
+ end
27
+ end
28
+ ```
29
+
30
+ Note that `steer` yields the current [task](/guides/tasks).
@@ -0,0 +1,14 @@
1
+ # Tasks
2
+
3
+ Tasks are the immutable units of work processed by [jobs](/guides/jobs). A task
4
+ consists of:
5
+
6
+ 1. The __URL__ to process
7
+ * Within a batch, every URL gets processed at most once.
8
+
9
+ 2. The __batch__ the task belongs to
10
+ * Like URLs, batches are strings.
11
+
12
+ Tasks get appended to the end of a message queue, and consumed from the
13
+ beginning. Because jobs can enqueue other tasks, jobs are both consumers
14
+ and producers of tasks.
data/docs/index.md CHANGED
@@ -1,82 +1,56 @@
1
1
  ---
2
- layout: default
3
- title: Versatile web crawling with (J)Ruby
2
+ hide:
3
+ - navigation
4
4
  ---
5
5
 
6
- # Versatile web crawling with (J)Ruby
6
+ # Wayfarer
7
7
 
8
- Wayfarer is the swiss army knife for web crawling.
8
+ ![CI status](https://github.com/actions/starter-workflows/workflows/CI/badge.svg)
9
+ [![RubyGem](https://badge.fury.io/rb/wayfarer.svg)](https://rubygems.org/gems/wayfarer)
9
10
 
10
- MRI:
11
- ```
12
- % [sudo] gem install wayfarer
13
- ```
14
-
15
- JRuby:
16
- ```
17
- % [sudo] gem install wayfarer-jruby
18
- ```
19
-
20
- If you …
21
-
22
- * __Need to crawl page graphs breadth-first__
23
- * __Need to extract whatever data__
24
- * Do it multi-threaded
25
- * Integrate with Rails seamlessly
26
- * Want to automate a web browser
27
- * Need to execute arbitrary JavaScript
28
- * Need URI normalization
29
- * Need to take screenshots
30
- * Want to use a job queue and make work happen later
31
- * Want in-memory and Redis-backed frontiers, tries and Bloom filters
11
+ ## Versatile web crawling with Ruby
32
12
 
33
- then you might like Wayfarer!
13
+ * Web scraping
14
+ * Data extraction
15
+ * Browser automation
34
16
 
35
- ## What it looks like
17
+ !!! attention "Unstable software"
36
18
 
37
- Say you want to
19
+ Wayfarer is under development and releases should be considered unstable.
38
20
 
39
- * Automate Google Chrome
40
- * Start off with a random Wikipedia article
41
- * Follow all links until you find a page with the word "Foobar"
42
- * Take a screenshot of the page containing "Foobar"
43
- * Extract keywords from every page you encounter
44
- * Use 4 threads and Chrome processes to do so
21
+ Wayfarer complies to
22
+ [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html) in
23
+ which v0.x means that there could be backward-incompatible changes for every
24
+ release:
45
25
 
46
- This amounts to 16 lines of code with Wayfarer:
26
+ >Major version zero (0.y.z) is for initial development. Anything MAY change
27
+ at any time. The public API SHOULD NOT be considered stable.
47
28
 
48
- {% highlight ruby %}
49
- require "wayfarer"
29
+ ### Installation
50
30
 
51
- class FindFoobarOnWikipedia < Wayfarer::Job
52
- config.http_adapter = :selenium
53
- config.selenium_argv = [:chrome]
54
- config.connection_count = 4
31
+ Install the RubyGem:
55
32
 
56
- let(:keywords) { [] }
57
-
58
- route.host "en.wikipedia.org", to: :article
59
-
60
- def article
61
- if page.body =~ /Foobar/
62
- driver.save_screenshot("/tmp/foobar.png")
63
- return halt
64
- end
65
-
66
- keywords << page.keywords
67
- stage page.links
68
- end
69
- end
70
-
71
- FindFoobarOnWikipedia.perform_now("https://en.wikipedia.org/wiki/Special:Random")
72
- {% endhighlight %}
33
+ ```
34
+ gem install wayfarer
35
+ ```
73
36
 
74
- Wayfarer integrates with [ActiveJob]() and supports your favorite job queue out of the box. Your job is ready to be enqueued:
75
- {% highlight ruby %}
76
- FindFoobarOnWikipedia.perform_later("https://en.wikipedia.org/wiki/Special:Random")
77
- {% endhighlight %}
37
+ Or add it to Bundler's Gemfile:
78
38
 
79
- ### Where to go from here
39
+ ```ruby
40
+ gem "wayfarer"
41
+ ```
80
42
 
81
- * [The tutorial]() shows how to collect all open issues from a GitHub repository
82
- * Read the [API documentation]()
43
+ ### Features
44
+
45
+ * Breadth-first, acyclic, multi-threaded graph traversal
46
+ * Executes atop a variety of message queues thanks to [ActiveJob](https://edgeguides.rubyonrails.org/active_job_basics.html)
47
+ * Browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
48
+ (<abbr title="Chrome DevTools Protocol">CDP</abbr>),
49
+ [Selenium](https://www.selenium.dev) or plain HTTP via `net/http`
50
+ * Declarative routing DSL
51
+ * URI normalization and deduplication
52
+ * XML, HTML, JSON parsing
53
+ * HTTP redirect handling
54
+ * Storage-agnostic
55
+ * Small footprint: <500 LoC
56
+ * Open Source (MIT)
@@ -0,0 +1,48 @@
1
+ ---
2
+ title: Wayfarer::Base
3
+ ---
4
+
5
+ # `Wayfarer::Base`
6
+
7
+ Wayfarer's complete job API.
8
+
9
+ ---
10
+
11
+ ### `::route`
12
+ : Draw routes to instance methods.
13
+
14
+ ---
15
+
16
+ ### `::steer { (Wayfarer::Task) -> [any] }`
17
+ : Provide router arguments.
18
+
19
+ ---
20
+
21
+ ### `#task -> Wayfarer::Task`
22
+ : The currently processing task.
23
+
24
+ ---
25
+
26
+ ### `#params -> Hash`
27
+ : URL parameters collected from the matching route.
28
+
29
+ ---
30
+
31
+ ### `#stage(String | [String]) -> void`
32
+ : Add URLs to a processing set. URLs already processed within the
33
+ current batch get discarded are not enqueued. Every staged URL gets
34
+ normalized.
35
+
36
+ ---
37
+
38
+ ### `#browser -> Object`
39
+ : The user agent that retrieved the current page.
40
+
41
+ ---
42
+
43
+ ### `#page(live: true | false) -> Page`
44
+ : The page representing the response retrieved from the currently
45
+ processing URL.
46
+
47
+ With `live: true` called, a fresh `Page` is returned that reflects the
48
+ current browser DOM. Calls to `#page` return the most recent page.
@@ -0,0 +1,182 @@
1
+ ---
2
+ title: Wayfarer::Routing::DSL
3
+ ---
4
+
5
+ # `Wayfarer::Route`
6
+
7
+ All public functionality documented here is exposed via
8
+ `Wayfarer::Routing::DSL`.
9
+
10
+ ---
11
+
12
+ ### `#host(String | Regexp)`
13
+ : Match hosts against strings or regular expressions.
14
+
15
+ ##### Examples
16
+
17
+ !!! example "Exact hostname string matching"
18
+
19
+ ```ruby
20
+ route.host "example.com"
21
+ ```
22
+
23
+ - [x] https://{==example.com==}
24
+ - [x] https://{==example.com==}/foo/bar
25
+ - [ ] https://sub.example.com
26
+ - [ ] https://nasa.gov
27
+
28
+ !!! example "Hostname matching with regular expression"
29
+
30
+ ```ruby
31
+ route.host /example.com$/
32
+ ```
33
+
34
+ - [x] https://{==example.com==}
35
+ - [x] https://{==example.com==}/foo/bar
36
+ - [x] https://{==sub.example.com==}
37
+ - [ ] https://nasa.gov
38
+
39
+ ---
40
+
41
+ ### `#suffix(String)`
42
+ : Match URL paths against exact suffixes.
43
+
44
+ ##### Examples
45
+
46
+ !!! example "Match only URLs with paths ending with \*.png"
47
+
48
+ ```ruby
49
+ route.suffix ".png"
50
+ ```
51
+
52
+ - [x] https://example.com/image{==.png==}
53
+ - [ ] https://example.com/index.html
54
+ - [ ] https://example.com
55
+
56
+ ---
57
+
58
+ ### `#path(String)`
59
+ : Match URLs against their path.
60
+
61
+ ##### Examples
62
+
63
+ !!! example "Exact path string matching"
64
+
65
+ ```ruby
66
+ route.path "/users/123"
67
+ ```
68
+
69
+ - [x] https://example.com/{==users/123==}
70
+ - [x] https://example.com/{==users/123==}
71
+ - [ ] https://example.com/users
72
+ - [ ] https://example.com
73
+
74
+ !!! example "Root path matching"
75
+
76
+ ```ruby
77
+ route.path "/"
78
+ ```
79
+
80
+ - [x] https://example.com{==/==}
81
+ - [ ] https://example.com
82
+ - [ ] https://example.com/users
83
+
84
+ !!! example "Pattern matching"
85
+
86
+ ```ruby
87
+ route.path "/users/:id", to: :with_params
88
+ route.path "/contact", to: :without_params
89
+
90
+ def with_params
91
+ params # => { "id" => ... }
92
+ end
93
+
94
+ def without_params
95
+ params # => {}
96
+ end
97
+ ```
98
+
99
+ - [x] https://example.com{==/users/123==}
100
+ - [x] https://example.com{==/contact==}
101
+ - [ ] https://example.com
102
+ - [ ] https://example.com/users/123/photos
103
+ - [ ] https://example.com/contact/
104
+
105
+ ---
106
+
107
+ ### `#query(Hash)`
108
+ : Match URLs against query parameters.
109
+
110
+ ##### Examples
111
+
112
+ !!! example "Exact parameter string matching"
113
+
114
+ ```ruby
115
+ route.query arg: "foo"
116
+ ```
117
+
118
+ - [x] https://example.com?{==arg=foo==}
119
+ - [x] https://example.com?{==arg=foo==}&page=2
120
+ - [ ] https://example.com?arg=123
121
+ - [ ] https://example.com
122
+
123
+ !!! example "Parameter regular expression matching"
124
+
125
+ ```ruby
126
+ route.query arg: /foo/
127
+ ```
128
+
129
+ - [x] https://example.com?{==arg=foo==}
130
+ - [x] https://example.com?{==arg=foobar==}
131
+ - [x] https://example.com?{==arg=barfoo==}&page=2
132
+ - [ ] https://example.com?arg=123
133
+ - [ ] https://example.com
134
+
135
+ !!! example "Parameter integer matching"
136
+
137
+ ```ruby
138
+ route.query page: 5
139
+ ```
140
+
141
+ - [x] https://example.com?{==page=5==}
142
+ - [x] https://example.com?arg=foo&{==page=5==}
143
+ - [ ] https://example.com?page=3
144
+ - [ ] https://example.com
145
+
146
+ !!! example "Parameter range matching"
147
+
148
+ ```ruby
149
+ route.query page: 1..10
150
+ ```
151
+
152
+ - [x] https://example.com?{==page=1==}
153
+ - [x] https://example.com?arg=foo&{==page=10==}
154
+ - [ ] https://example.com?page=11
155
+ - [ ] https://example.com
156
+
157
+ ### `#scheme(Symbol)`
158
+ : Match URLs by their scheme.
159
+
160
+ ##### Examples
161
+
162
+ !!! example "Scheme matching"
163
+
164
+ ```ruby
165
+ route.scheme :https
166
+ ```
167
+
168
+ - [x] {==https==}://example.com
169
+ - [ ] http://example.com
170
+
171
+ ### `#uri(String)`
172
+ : Match URLs exactly.
173
+
174
+ ##### Examples
175
+
176
+ !!! example "URL matching"
177
+
178
+ ```ruby
179
+ route.uri "https://example.com/foo/bar?page=3"
180
+ ```
181
+
182
+ - [x] {==https://example.com/foo/bar?page=3==}
@@ -0,0 +1,61 @@
1
+ # Command Line Interface
2
+
3
+ ## Usage
4
+
5
+ ```
6
+ wayfarer [OPTIONS] [generate|job|route|version]
7
+ ```
8
+
9
+ All [environment variables](../environment_variables) are respected.
10
+
11
+ ## `wayfarer generate`
12
+
13
+ ### `wayfarer generate project NAME`
14
+
15
+ : Generates a new project directory `NAME`.
16
+
17
+ ## `wayfarer job`
18
+
19
+ ### `wayfarer job perform JOB URL`
20
+
21
+ : Performs `JOB` with `URL`. The job does not reach any Active Job backend.
22
+ Staged jobs will not be processed.
23
+
24
+ ##### Options
25
+
26
+ * `--mock-redis`: Use an in-memory implementation of Redis instead of
27
+ talking to an actual server.
28
+ * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
29
+
30
+ ### `wayfarer job enqueue JOB URL`
31
+
32
+ : Enqueues `JOB` with `URL` to the configured Active Job backend.
33
+
34
+ ##### Options
35
+
36
+ * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
37
+
38
+ ### `wayfarer job execute JOB URL`
39
+
40
+ : Execute `JOB` with `URL` by using the
41
+ [Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html).
42
+ The job does not reach any other Active Job backend. Blocks until the batch
43
+ completes.
44
+
45
+ ##### Options
46
+
47
+ * `--mock-redis`: Use an in-memory implementation of Redis instead of
48
+ talking to an actual server.
49
+ * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
50
+ * `--min-threads`: Minimum number of threads to use. Default: 1
51
+ * `--max-threads`: Maximum number of threads to use. Default: 1
52
+
53
+ ## `wayfarer route`
54
+
55
+ ### `wayfarer route result JOB URL`
56
+
57
+ : Prints the result of invoking `JOB`'s router with `URL`.
58
+
59
+ ### `wayfarer route tree JOB URL`
60
+
61
+ : Visualises the routing tree result of invoking `JOB`'s router with `URL`.
@@ -0,0 +1,42 @@
1
+ ---
2
+ hide:
3
+ - toc
4
+ ---
5
+
6
+ # Configuration Keys
7
+
8
+ ## `Wayfarer.config.network`
9
+
10
+ | Runtime config key | Environment variable | Description | Default | Supported values |
11
+ | ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
12
+ | `network.agent` | `WAYFARER_NETWORK_AGENT` | The user agent to use. | `:http` | `:http`, `:ferrum`, `:selenium` |
13
+ | `network.pool_size` | `WAYFARER_NETWORK_POOL_SIZE` | How many user agents to spawn. | 1 | Integers |
14
+ | `network.pool_timeout` | `WAYFARER_NETWORK_POOL_TIMEOUT` | How long jobs may use an agent in seconds. | 10 | Integers |
15
+ | `network.http_headers` | `WAYFARER_NETWORK_HTTP_HEADERS` | HTTP headers to append to requests. | `{}` | Hashes |
16
+
17
+ ## `Wayfarer.config.ferrum`
18
+
19
+ | Runtime config key | Environment variable | Description | Default | Supported values |
20
+ | ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
21
+ | `ferrum.options` | `WAYFARER_FERRUM_OPTIONS` | Ferrum options. | `{}` | Hashes |
22
+
23
+ ## `Wayfarer.config.selenium`
24
+
25
+ | Runtime config key | Environment variable | Description | Default | Supported values |
26
+ | ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
27
+ | `selenium.driver` | `WAYFARER_SELENIUM_DRIVER` | Selenium driver to use. | `:chrome` | Symbols |
28
+ | `selenium.options` | `WAYFARER_SELENIUM_OPTIONS` | Selenium options. | `{}` | Hashes |
29
+ | `selenium.client_timeout` | `WAYFARER_SELENIUM_CLIENT_TIMEOUT` | Selenium client timeout in seconds. | 60 | Integers |
30
+
31
+ ## `Wayfarer.config.redis`
32
+
33
+ | Runtime config key | Environment variable | Description | Default | Supported values |
34
+ | ---------------------- | ------------------------------------ | ------------------------------------------- | ------------------------------------------ | ----------------------------------- |
35
+ | `redis.url` | `WAYFARER_REDIS_URL` | Redis URL to connect to. | http://localhost:6379 | Strings |
36
+ | `redis.factory` | n/a | Redis factory lambda. | ` ->(redis) { ::Redis.new(url: redis.url)` | Lambdas |
37
+
38
+ ## `Wayfarer.config.capybara`
39
+
40
+ | Runtime config key | Environment variable | Description | Default | Supported values |
41
+ | ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
42
+ | `capybara.driver` | `WAYFARER_CAPYBARA_DRIVER` | The Capybara driver to use. | n/a | Symbols |