wayfarer 0.0.3 → 0.4.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.github/workflows/ci.yaml +32 -0
- data/.gitignore +3 -4
- data/.rubocop.yml +25 -9
- data/.ruby-version +1 -1
- data/Dockerfile +5 -0
- data/Gemfile +1 -7
- data/Gemfile.lock +221 -0
- data/RELEASING.md +17 -0
- data/Rakefile +38 -90
- data/bin/wayfarer +1 -111
- data/docker-compose.yml +32 -0
- data/docs/cookbook/batch_routing.md +22 -0
- data/docs/cookbook/consent_screen.md +36 -0
- data/docs/cookbook/executing_javascript.md +41 -0
- data/docs/cookbook/querying_html.md +42 -0
- data/docs/cookbook/screenshots.md +27 -0
- data/docs/cookbook/user_agent.md +7 -0
- data/docs/guides/browser_automation/capybara.md +69 -0
- data/docs/guides/browser_automation/custom_adapters.md +100 -0
- data/docs/guides/browser_automation/ferrum.md +39 -0
- data/docs/guides/browser_automation/selenium.md +63 -0
- data/docs/guides/callbacks.md +131 -31
- data/docs/guides/configuration.md +24 -169
- data/docs/guides/debugging.md +17 -0
- data/docs/guides/error_handling.md +30 -45
- data/docs/guides/jobs.md +101 -0
- data/docs/guides/navigation.md +73 -0
- data/docs/guides/networking.md +94 -0
- data/docs/guides/pages.md +52 -0
- data/docs/guides/performance.md +130 -0
- data/docs/guides/reliability.md +41 -0
- data/docs/guides/routing/steering.md +30 -0
- data/docs/guides/tasks.md +14 -0
- data/docs/index.md +40 -66
- data/docs/reference/api/base.md +48 -0
- data/docs/reference/api/route.md +182 -0
- data/docs/reference/cli.md +61 -0
- data/docs/reference/configuration_keys.md +42 -0
- data/docs/reference/environment_variables.md +83 -0
- data/lib/wayfarer/base.rb +50 -0
- data/lib/wayfarer/callbacks.rb +71 -0
- data/lib/wayfarer/cli/base.rb +27 -0
- data/lib/wayfarer/cli/generate.rb +17 -0
- data/lib/wayfarer/cli/job.rb +60 -0
- data/lib/wayfarer/cli/route.rb +29 -0
- data/lib/wayfarer/cli/route_printer.rb +116 -0
- data/lib/wayfarer/cli/runner.rb +34 -0
- data/lib/wayfarer/cli/templates/Gemfile.tt +5 -0
- data/lib/wayfarer/cli/templates/job.rb.tt +10 -0
- data/lib/wayfarer/config/capybara.rb +10 -0
- data/lib/wayfarer/config/ferrum.rb +11 -0
- data/lib/wayfarer/config/networking.rb +26 -0
- data/lib/wayfarer/config/redis.rb +14 -0
- data/lib/wayfarer/config/root.rb +11 -0
- data/lib/wayfarer/config/selenium.rb +21 -0
- data/lib/wayfarer/config/strconv.rb +45 -0
- data/lib/wayfarer/config/struct.rb +72 -0
- data/lib/wayfarer/gc.rb +15 -0
- data/lib/wayfarer/middleware/chain.rb +19 -0
- data/lib/wayfarer/middleware/dedup.rb +25 -0
- data/lib/wayfarer/middleware/fetch.rb +47 -0
- data/lib/wayfarer/middleware/normalize.rb +25 -0
- data/lib/wayfarer/middleware/router.rb +53 -0
- data/lib/wayfarer/middleware/stage.rb +23 -0
- data/lib/wayfarer/middleware/worker.rb +30 -0
- data/lib/wayfarer/networking/capybara.rb +28 -0
- data/lib/wayfarer/networking/context.rb +36 -0
- data/lib/wayfarer/networking/ferrum.rb +35 -0
- data/lib/wayfarer/networking/http.rb +34 -0
- data/lib/wayfarer/networking/pool.rb +40 -0
- data/lib/wayfarer/networking/result.rb +18 -0
- data/lib/wayfarer/networking/selenium.rb +43 -0
- data/lib/wayfarer/networking/strategy.rb +38 -0
- data/lib/wayfarer/page.rb +17 -74
- data/lib/wayfarer/parsing/json.rb +17 -0
- data/lib/wayfarer/parsing/xml.rb +17 -0
- data/lib/wayfarer/redis/.#barrier.rb +1 -0
- data/lib/wayfarer/redis/barrier.rb +36 -0
- data/lib/wayfarer/redis/connection.rb +13 -0
- data/lib/wayfarer/redis/counter.rb +29 -0
- data/lib/wayfarer/redis/pool.rb +20 -0
- data/lib/wayfarer/redis/version.rb +19 -0
- data/lib/wayfarer/routing/dsl.rb +57 -0
- data/lib/wayfarer/routing/matchers/custom.rb +25 -0
- data/lib/wayfarer/routing/matchers/host.rb +19 -0
- data/lib/wayfarer/routing/matchers/path.rb +49 -0
- data/lib/wayfarer/routing/matchers/query.rb +63 -0
- data/lib/wayfarer/routing/matchers/scheme.rb +17 -0
- data/lib/wayfarer/routing/matchers/suffix.rb +17 -0
- data/lib/wayfarer/routing/matchers/url.rb +17 -0
- data/lib/wayfarer/routing/path_finder.rb +46 -0
- data/lib/wayfarer/routing/result.rb +15 -0
- data/lib/wayfarer/routing/root_route.rb +7 -0
- data/lib/wayfarer/routing/route.rb +47 -0
- data/lib/wayfarer/routing/router.rb +10 -54
- data/lib/wayfarer/routing/target_route.rb +7 -0
- data/lib/wayfarer/serializer.rb +17 -0
- data/lib/wayfarer/stringify.rb +47 -0
- data/lib/wayfarer/task.rb +34 -0
- data/lib/wayfarer.rb +48 -57
- data/mkdocs.yml +47 -0
- data/requirements.txt +1 -0
- data/spec/base_spec.rb +233 -0
- data/spec/callbacks_spec.rb +102 -0
- data/spec/cli/generate_spec.rb +39 -0
- data/spec/cli/job_spec.rb +74 -0
- data/spec/cli/version_spec.rb +13 -0
- data/spec/config/capybara_spec.rb +18 -0
- data/spec/config/ferrum_spec.rb +24 -0
- data/spec/config/networking_spec.rb +73 -0
- data/spec/config/redis_spec.rb +32 -0
- data/spec/config/root_spec.rb +31 -0
- data/spec/config/selenium_spec.rb +56 -0
- data/spec/config/strconv_spec.rb +58 -0
- data/spec/config/struct_spec.rb +66 -0
- data/spec/factories/middleware.rb +15 -0
- data/spec/factories/page.rb +78 -0
- data/spec/factories/task.rb +12 -0
- data/spec/fixtures/dummy_job.rb +7 -0
- data/spec/gc_spec.rb +63 -0
- data/spec/middleware/chain_spec.rb +96 -0
- data/spec/middleware/dedup_spec.rb +76 -0
- data/spec/middleware/fetch_spec.rb +100 -0
- data/spec/middleware/normalize_spec.rb +28 -0
- data/spec/middleware/router_spec.rb +80 -0
- data/spec/middleware/stage_spec.rb +39 -0
- data/spec/middleware/worker_spec.rb +117 -0
- data/spec/networking/capybara_spec.rb +12 -0
- data/spec/networking/context_spec.rb +127 -0
- data/spec/networking/ferrum_spec.rb +12 -0
- data/spec/networking/http_spec.rb +12 -0
- data/spec/networking/pool_spec.rb +67 -0
- data/spec/networking/selenium_spec.rb +12 -0
- data/spec/networking/strategy.rb +170 -0
- data/spec/page_spec.rb +21 -12
- data/spec/{parsers/json_parser_spec.rb → parsing/json_spec.rb} +5 -4
- data/spec/{parsers/xml_parser_spec.rb → parsing/xml_spec.rb} +3 -2
- data/spec/redis/barrier_spec.rb +78 -0
- data/spec/redis/counter_spec.rb +32 -0
- data/spec/redis/pool_spec.rb +18 -0
- data/spec/redis/version_spec.rb +13 -0
- data/spec/routing/dsl_spec.rb +98 -0
- data/spec/routing/integration_spec.rb +110 -0
- data/spec/routing/matchers/custom_spec.rb +31 -0
- data/spec/routing/matchers/host_spec.rb +49 -0
- data/spec/routing/matchers/path_spec.rb +43 -0
- data/spec/routing/matchers/query_spec.rb +137 -0
- data/spec/routing/matchers/scheme_spec.rb +25 -0
- data/spec/routing/{filetypes_rule_spec.rb → matchers/suffix_spec.rb} +14 -13
- data/spec/routing/matchers/uri_spec.rb +27 -0
- data/spec/routing/path_finder_spec.rb +33 -0
- data/spec/routing/root_route_spec.rb +29 -0
- data/spec/routing/route_spec.rb +43 -0
- data/spec/routing/router_spec.rb +13 -56
- data/spec/spec_helpers.rb +73 -38
- data/spec/stringify_spec.rb +23 -0
- data/{support → spec/support}/static/finders.html +0 -0
- data/{support → spec/support}/static/graph/details/a.html +0 -0
- data/{support → spec/support}/static/graph/details/b.html +0 -0
- data/{support → spec/support}/static/graph/index.html +0 -0
- data/{support → spec/support}/static/json/dummy.json +0 -0
- data/{support → spec/support}/static/links/links.html +0 -0
- data/{support → spec/support}/static/xml/dummy.xml +0 -0
- data/{support → spec/support}/test_app.rb +9 -2
- data/spec/task_spec.rb +27 -0
- data/spec/wayfarer_spec.rb +2 -13
- data/wayfarer.gemspec +40 -42
- metadata +234 -361
- data/.travis.yml +0 -5
- data/Changelog.md +0 -10
- data/README.md +0 -21
- data/benchmark/frontiers.rb +0 -143
- data/docs/.gitignore +0 -2
- data/docs/_config.yml +0 -15
- data/docs/_includes/base.html +0 -7
- data/docs/_includes/head.html +0 -10
- data/docs/_includes/navigation.html +0 -187
- data/docs/_layouts/default.html +0 -42
- data/docs/_sass/base.scss +0 -439
- data/docs/_sass/variables.scss +0 -24
- data/docs/_sass/vendor/bourbon/_bourbon-deprecate.scss +0 -19
- data/docs/_sass/vendor/bourbon/_bourbon-deprecated-upcoming.scss +0 -425
- data/docs/_sass/vendor/bourbon/_bourbon.scss +0 -90
- data/docs/_sass/vendor/bourbon/addons/_border-color.scss +0 -29
- data/docs/_sass/vendor/bourbon/addons/_border-radius.scss +0 -48
- data/docs/_sass/vendor/bourbon/addons/_border-style.scss +0 -28
- data/docs/_sass/vendor/bourbon/addons/_border-width.scss +0 -28
- data/docs/_sass/vendor/bourbon/addons/_buttons.scss +0 -69
- data/docs/_sass/vendor/bourbon/addons/_clearfix.scss +0 -25
- data/docs/_sass/vendor/bourbon/addons/_ellipsis.scss +0 -30
- data/docs/_sass/vendor/bourbon/addons/_font-stacks.scss +0 -31
- data/docs/_sass/vendor/bourbon/addons/_hide-text.scss +0 -27
- data/docs/_sass/vendor/bourbon/addons/_margin.scss +0 -29
- data/docs/_sass/vendor/bourbon/addons/_padding.scss +0 -29
- data/docs/_sass/vendor/bourbon/addons/_position.scss +0 -51
- data/docs/_sass/vendor/bourbon/addons/_prefixer.scss +0 -66
- data/docs/_sass/vendor/bourbon/addons/_retina-image.scss +0 -27
- data/docs/_sass/vendor/bourbon/addons/_size.scss +0 -56
- data/docs/_sass/vendor/bourbon/addons/_text-inputs.scss +0 -118
- data/docs/_sass/vendor/bourbon/addons/_timing-functions.scss +0 -34
- data/docs/_sass/vendor/bourbon/addons/_triangle.scss +0 -63
- data/docs/_sass/vendor/bourbon/addons/_word-wrap.scss +0 -29
- data/docs/_sass/vendor/bourbon/css3/_animation.scss +0 -61
- data/docs/_sass/vendor/bourbon/css3/_appearance.scss +0 -5
- data/docs/_sass/vendor/bourbon/css3/_backface-visibility.scss +0 -5
- data/docs/_sass/vendor/bourbon/css3/_background-image.scss +0 -44
- data/docs/_sass/vendor/bourbon/css3/_background.scss +0 -57
- data/docs/_sass/vendor/bourbon/css3/_border-image.scss +0 -61
- data/docs/_sass/vendor/bourbon/css3/_calc.scss +0 -6
- data/docs/_sass/vendor/bourbon/css3/_columns.scss +0 -67
- data/docs/_sass/vendor/bourbon/css3/_filter.scss +0 -6
- data/docs/_sass/vendor/bourbon/css3/_flex-box.scss +0 -327
- data/docs/_sass/vendor/bourbon/css3/_font-face.scss +0 -29
- data/docs/_sass/vendor/bourbon/css3/_font-feature-settings.scss +0 -6
- data/docs/_sass/vendor/bourbon/css3/_hidpi-media-query.scss +0 -12
- data/docs/_sass/vendor/bourbon/css3/_hyphens.scss +0 -6
- data/docs/_sass/vendor/bourbon/css3/_image-rendering.scss +0 -15
- data/docs/_sass/vendor/bourbon/css3/_keyframes.scss +0 -38
- data/docs/_sass/vendor/bourbon/css3/_linear-gradient.scss +0 -40
- data/docs/_sass/vendor/bourbon/css3/_perspective.scss +0 -12
- data/docs/_sass/vendor/bourbon/css3/_placeholder.scss +0 -10
- data/docs/_sass/vendor/bourbon/css3/_radial-gradient.scss +0 -40
- data/docs/_sass/vendor/bourbon/css3/_selection.scss +0 -44
- data/docs/_sass/vendor/bourbon/css3/_text-decoration.scss +0 -27
- data/docs/_sass/vendor/bourbon/css3/_transform.scss +0 -21
- data/docs/_sass/vendor/bourbon/css3/_transition.scss +0 -81
- data/docs/_sass/vendor/bourbon/css3/_user-select.scss +0 -5
- data/docs/_sass/vendor/bourbon/functions/_assign-inputs.scss +0 -16
- data/docs/_sass/vendor/bourbon/functions/_contains-falsy.scss +0 -25
- data/docs/_sass/vendor/bourbon/functions/_contains.scss +0 -31
- data/docs/_sass/vendor/bourbon/functions/_is-length.scss +0 -16
- data/docs/_sass/vendor/bourbon/functions/_is-light.scss +0 -26
- data/docs/_sass/vendor/bourbon/functions/_is-number.scss +0 -16
- data/docs/_sass/vendor/bourbon/functions/_is-size.scss +0 -23
- data/docs/_sass/vendor/bourbon/functions/_modular-scale.scss +0 -74
- data/docs/_sass/vendor/bourbon/functions/_px-to-em.scss +0 -24
- data/docs/_sass/vendor/bourbon/functions/_px-to-rem.scss +0 -26
- data/docs/_sass/vendor/bourbon/functions/_shade.scss +0 -24
- data/docs/_sass/vendor/bourbon/functions/_strip-units.scss +0 -22
- data/docs/_sass/vendor/bourbon/functions/_tint.scss +0 -24
- data/docs/_sass/vendor/bourbon/functions/_transition-property-name.scss +0 -37
- data/docs/_sass/vendor/bourbon/functions/_unpack.scss +0 -32
- data/docs/_sass/vendor/bourbon/helpers/_convert-units.scss +0 -26
- data/docs/_sass/vendor/bourbon/helpers/_directional-values.scss +0 -108
- data/docs/_sass/vendor/bourbon/helpers/_font-source-declaration.scss +0 -53
- data/docs/_sass/vendor/bourbon/helpers/_gradient-positions-parser.scss +0 -24
- data/docs/_sass/vendor/bourbon/helpers/_linear-angle-parser.scss +0 -35
- data/docs/_sass/vendor/bourbon/helpers/_linear-gradient-parser.scss +0 -51
- data/docs/_sass/vendor/bourbon/helpers/_linear-positions-parser.scss +0 -77
- data/docs/_sass/vendor/bourbon/helpers/_linear-side-corner-parser.scss +0 -41
- data/docs/_sass/vendor/bourbon/helpers/_radial-arg-parser.scss +0 -74
- data/docs/_sass/vendor/bourbon/helpers/_radial-gradient-parser.scss +0 -55
- data/docs/_sass/vendor/bourbon/helpers/_radial-positions-parser.scss +0 -28
- data/docs/_sass/vendor/bourbon/helpers/_render-gradients.scss +0 -31
- data/docs/_sass/vendor/bourbon/helpers/_shape-size-stripper.scss +0 -15
- data/docs/_sass/vendor/bourbon/helpers/_str-to-num.scss +0 -55
- data/docs/_sass/vendor/bourbon/settings/_asset-pipeline.scss +0 -7
- data/docs/_sass/vendor/bourbon/settings/_deprecation-warnings.scss +0 -8
- data/docs/_sass/vendor/bourbon/settings/_prefixer.scss +0 -9
- data/docs/_sass/vendor/bourbon/settings/_px-to-em.scss +0 -1
- data/docs/_sass/vendor/neat/_neat-helpers.scss +0 -11
- data/docs/_sass/vendor/neat/_neat.scss +0 -23
- data/docs/_sass/vendor/neat/functions/_new-breakpoint.scss +0 -49
- data/docs/_sass/vendor/neat/functions/_private.scss +0 -114
- data/docs/_sass/vendor/neat/grid/_box-sizing.scss +0 -15
- data/docs/_sass/vendor/neat/grid/_direction-context.scss +0 -33
- data/docs/_sass/vendor/neat/grid/_display-context.scss +0 -28
- data/docs/_sass/vendor/neat/grid/_fill-parent.scss +0 -22
- data/docs/_sass/vendor/neat/grid/_media.scss +0 -92
- data/docs/_sass/vendor/neat/grid/_omega.scss +0 -87
- data/docs/_sass/vendor/neat/grid/_outer-container.scss +0 -34
- data/docs/_sass/vendor/neat/grid/_pad.scss +0 -25
- data/docs/_sass/vendor/neat/grid/_private.scss +0 -35
- data/docs/_sass/vendor/neat/grid/_row.scss +0 -52
- data/docs/_sass/vendor/neat/grid/_shift.scss +0 -50
- data/docs/_sass/vendor/neat/grid/_span-columns.scss +0 -94
- data/docs/_sass/vendor/neat/grid/_to-deprecate.scss +0 -97
- data/docs/_sass/vendor/neat/grid/_visual-grid.scss +0 -42
- data/docs/_sass/vendor/neat/mixins/_clearfix.scss +0 -25
- data/docs/_sass/vendor/neat/settings/_disable-warnings.scss +0 -13
- data/docs/_sass/vendor/neat/settings/_grid.scss +0 -51
- data/docs/_sass/vendor/neat/settings/_visual-grid.scss +0 -27
- data/docs/_sass/vendor/normalize-3.0.2.scss +0 -427
- data/docs/_sass/vendor/pygments.scss +0 -356
- data/docs/automating_browsers/capybara.md +0 -70
- data/docs/css/screen.scss +0 -7
- data/docs/guides/cli.md +0 -52
- data/docs/guides/frontiers.md +0 -93
- data/docs/guides/halting.md +0 -23
- data/docs/guides/job_queues.md +0 -26
- data/docs/guides/locals.md +0 -36
- data/docs/guides/logging.md +0 -22
- data/docs/guides/page_objects.md +0 -67
- data/docs/guides/peeking.md +0 -46
- data/docs/guides/selenium_capybara.md +0 -100
- data/docs/guides/tutorial.md +0 -452
- data/docs/js/navigation.js +0 -11
- data/docs/misc/contributing.md +0 -20
- data/docs/misc/testing.md +0 -11
- data/docs/recipes/authentication.md +0 -23
- data/docs/recipes/csv.md +0 -29
- data/docs/recipes/javascript.md +0 -20
- data/docs/recipes/multiple_uris.md +0 -18
- data/docs/recipes/screenshots.md +0 -20
- data/docs/routing/custom_rules.md +0 -16
- data/docs/routing/filetypes_rules.md +0 -21
- data/docs/routing/host_rules.md +0 -24
- data/docs/routing/path_rules.md +0 -33
- data/docs/routing/protocol_rules.md +0 -17
- data/docs/routing/query_rules.md +0 -69
- data/docs/routing/routes.md +0 -96
- data/docs/routing/uri_rules.md +0 -18
- data/examples/collect_github_issues.rb +0 -65
- data/examples/find_foobar_on_wikipedia.rb +0 -23
- data/lib/wayfarer/configuration.rb +0 -86
- data/lib/wayfarer/crawl.rb +0 -79
- data/lib/wayfarer/crawl_observer.rb +0 -103
- data/lib/wayfarer/dispatcher.rb +0 -104
- data/lib/wayfarer/finders.rb +0 -61
- data/lib/wayfarer/frontiers/frontier.rb +0 -79
- data/lib/wayfarer/frontiers/memory_bloomfilter.rb +0 -32
- data/lib/wayfarer/frontiers/memory_frontier.rb +0 -76
- data/lib/wayfarer/frontiers/memory_trie_frontier.rb +0 -39
- data/lib/wayfarer/frontiers/normalize_uris.rb +0 -48
- data/lib/wayfarer/frontiers/redis_bloomfilter.rb +0 -34
- data/lib/wayfarer/frontiers/redis_frontier.rb +0 -83
- data/lib/wayfarer/http_adapters/adapter_pool.rb +0 -62
- data/lib/wayfarer/http_adapters/net_http_adapter.rb +0 -77
- data/lib/wayfarer/http_adapters/selenium_adapter.rb +0 -80
- data/lib/wayfarer/job.rb +0 -211
- data/lib/wayfarer/locals.rb +0 -40
- data/lib/wayfarer/parsers/json_parser.rb +0 -20
- data/lib/wayfarer/parsers/xml_parser.rb +0 -27
- data/lib/wayfarer/processor.rb +0 -103
- data/lib/wayfarer/routing/custom_rule.rb +0 -21
- data/lib/wayfarer/routing/filetypes_rule.rb +0 -20
- data/lib/wayfarer/routing/host_rule.rb +0 -19
- data/lib/wayfarer/routing/path_rule.rb +0 -54
- data/lib/wayfarer/routing/protocol_rule.rb +0 -21
- data/lib/wayfarer/routing/query_rule.rb +0 -59
- data/lib/wayfarer/routing/rule.rb +0 -114
- data/lib/wayfarer/routing/uri_rule.rb +0 -21
- data/spec/configuration_spec.rb +0 -26
- data/spec/crawl_spec.rb +0 -48
- data/spec/finders_spec.rb +0 -49
- data/spec/frontiers/memory_bloomfilter_spec.rb +0 -6
- data/spec/frontiers/memory_frontier_spec.rb +0 -6
- data/spec/frontiers/memory_trie_frontier_spec.rb +0 -6
- data/spec/frontiers/normalize_uris_spec.rb +0 -59
- data/spec/frontiers/redis_bloomfilter_spec.rb +0 -6
- data/spec/frontiers/redis_frontier_spec.rb +0 -6
- data/spec/http_adapters/adapter_pool_spec.rb +0 -33
- data/spec/http_adapters/net_http_adapter_spec.rb +0 -83
- data/spec/http_adapters/selenium_adapter_spec.rb +0 -53
- data/spec/integration/callbacks_spec.rb +0 -42
- data/spec/integration/locals_spec.rb +0 -106
- data/spec/integration/peeking_spec.rb +0 -61
- data/spec/job_spec.rb +0 -122
- data/spec/processor_spec.rb +0 -31
- data/spec/routing/custom_rule_spec.rb +0 -26
- data/spec/routing/host_rule_spec.rb +0 -48
- data/spec/routing/path_rule_spec.rb +0 -66
- data/spec/routing/protocol_rule_spec.rb +0 -26
- data/spec/routing/query_rule_spec.rb +0 -124
- data/spec/routing/rule_spec.rb +0 -251
- data/spec/routing/uri_rule_spec.rb +0 -24
- data/spec/shared/frontier.rb +0 -96
- data/wayfarer-jruby.gemspec +0 -49
@@ -0,0 +1,130 @@
|
|
1
|
+
# Performance
|
2
|
+
|
3
|
+
How to write performant crawlers with Wayfarer.
|
4
|
+
|
5
|
+
## Use a sufficiently sized user agent pool
|
6
|
+
|
7
|
+
Automated browser processes or HTTP clients are kept in a [connection pool]() of
|
8
|
+
static size. This avoids having to re-establish browser processes and enables
|
9
|
+
their reuse.
|
10
|
+
|
11
|
+
If the size of the pool is too small, the pool is a
|
12
|
+
bottleneck. For example, if your message queue adapter uses 8 threads, but the
|
13
|
+
pool only contains 1 user agent, the remaining 7 threads block until the agent
|
14
|
+
is checked back in to the pool for use by one of the blocked threads.
|
15
|
+
|
16
|
+
There is no reliable way to detect the number of threads of the underlying
|
17
|
+
message queue adapter. The pool size should equal the number of threads;
|
18
|
+
|
19
|
+
```ruby
|
20
|
+
Wayfarer.config.network.pool_size = 8 # defaults to 1
|
21
|
+
```
|
22
|
+
|
23
|
+
### Job shedding
|
24
|
+
|
25
|
+
There is a maximum number of seconds that jobs wait when checking out a user
|
26
|
+
agent from the pool. Once this time is exceeded,
|
27
|
+
a `Wayfarer::UserAgentTimeoutError` is raised. By default, the timeout is 10
|
28
|
+
seconds.
|
29
|
+
|
30
|
+
This hints there are more threads in use than user agents in the pool.
|
31
|
+
|
32
|
+
## Stage less URLs
|
33
|
+
|
34
|
+
Staging less URLs saves space and time:
|
35
|
+
|
36
|
+
* Less tasks written to the message queue
|
37
|
+
* Less time spent consuming tasks
|
38
|
+
* Less time spent filtering URLs with Redis
|
39
|
+
|
40
|
+
Wayfarer maintains a set of processed URLs for a batch in Redis. Every staged
|
41
|
+
URL is checked for inclusion in this set before it gets appended as a task to
|
42
|
+
the message queue.
|
43
|
+
|
44
|
+
A common pattern is to stage all links of a page, and rely on routing to fetch
|
45
|
+
only the relevant ones:
|
46
|
+
|
47
|
+
```ruby
|
48
|
+
class DummyJob < Wayfarer::Base
|
49
|
+
route { to: index, host: "example.com" }
|
50
|
+
|
51
|
+
def index
|
52
|
+
stage page.meta.links.all
|
53
|
+
end
|
54
|
+
end
|
55
|
+
```
|
56
|
+
|
57
|
+
Pages commonly contain a large number of URLs.
|
58
|
+
|
59
|
+
Every staged URL is:
|
60
|
+
|
61
|
+
1. Normalized to a canonical form, for example by sorting query parameters
|
62
|
+
alphabetically.
|
63
|
+
2. Checked for inclusion in the batch Redis set or discarded.
|
64
|
+
3. Written to the message queue.
|
65
|
+
4. Consumed from the queue and matched against the router.
|
66
|
+
5. Fetched, if a route matches.
|
67
|
+
|
68
|
+
Narrowing down the links in the document to follow speeds up the process.
|
69
|
+
For example using Nokogiri, interesting links can be identified with a CSS
|
70
|
+
selector:
|
71
|
+
|
72
|
+
```ruby
|
73
|
+
class DummyJob < Wayfarer::Base
|
74
|
+
route { to: index, host: "example.com" }
|
75
|
+
|
76
|
+
def index
|
77
|
+
stage interesting_links
|
78
|
+
end
|
79
|
+
|
80
|
+
private
|
81
|
+
|
82
|
+
def interesting_links
|
83
|
+
page.doc.css("a.interesting").map { |elem| elem["href"] }
|
84
|
+
end
|
85
|
+
end
|
86
|
+
```
|
87
|
+
|
88
|
+
Because the router only accepts the single hostname `example.com`, the job can
|
89
|
+
also ensure it stages only internal URLs by intersecting them with the
|
90
|
+
interesting ones:
|
91
|
+
|
92
|
+
```ruby
|
93
|
+
class DummyJob < Wayfarer::Base
|
94
|
+
route { to: index, host: "example.com" }
|
95
|
+
|
96
|
+
def index
|
97
|
+
stage interesting_internal_links
|
98
|
+
end
|
99
|
+
|
100
|
+
private
|
101
|
+
|
102
|
+
def interesting_internal_links
|
103
|
+
page.meta.links.internal & interesting_links
|
104
|
+
end
|
105
|
+
|
106
|
+
def interesting_links
|
107
|
+
page.doc.css("a.interesting").map { |elem| elem["href"] }
|
108
|
+
end
|
109
|
+
end
|
110
|
+
```
|
111
|
+
|
112
|
+
|
113
|
+
## Use Redis >= 6.2.0
|
114
|
+
|
115
|
+
Redis 6.2.0 introduced the
|
116
|
+
[`SMISMEMBER`](https://redis.io/commands/smismember) command which enables
|
117
|
+
Wayfarer to check whether multiple URLs have been processed in a batch with a
|
118
|
+
single command. With earlier versions, one command per URL is required.
|
119
|
+
|
120
|
+
Wayfarer detects the Redis server version and uses `SMISMEMBER` without user
|
121
|
+
configuration when supported.
|
122
|
+
|
123
|
+
## Use Oj for JSON parsing
|
124
|
+
|
125
|
+
Wayfarer uses [Oj](https://github.com/ohler55/oj) for JSON parsing if the gem
|
126
|
+
has been required at runtime:
|
127
|
+
|
128
|
+
```ruby
|
129
|
+
require "oj"
|
130
|
+
```
|
@@ -0,0 +1,41 @@
|
|
1
|
+
# Reliablity
|
2
|
+
|
3
|
+
## Durability
|
4
|
+
|
5
|
+
Wayfarer executes atop reliable messages queues such as Sidekiq, Resque,
|
6
|
+
RabbitMQ, etc. Its configuration is independent of the underlying queue
|
7
|
+
infrastructure it reads from and writes to.
|
8
|
+
|
9
|
+
## Self-healing user agents
|
10
|
+
|
11
|
+
Wayfarer handles the scenario where a remote browser process has crashed and
|
12
|
+
must be replaced by a fresh browser process.
|
13
|
+
|
14
|
+
This can be tested locally by automating a browser with headless mode turned
|
15
|
+
off, and then closing the opened browser window: The current job fails, but the
|
16
|
+
next job has access to a newly established browser session again.
|
17
|
+
|
18
|
+
For example Ferrum might raise `Ferrum::DeadBrowserError`. Wayfarer's
|
19
|
+
user agents are self-healing and react to these kinds of errors internally. When
|
20
|
+
a browser window is closed, the Ferrum user agent attempts to establish a new
|
21
|
+
browser process as a replacement, for the next job to use.
|
22
|
+
|
23
|
+
[Wayfarer never swallows exceptions](/guides/error_handling). This means
|
24
|
+
that even though the user agent might heal itself, jobs still need to explicitly
|
25
|
+
retry browser errors:
|
26
|
+
|
27
|
+
```ruby
|
28
|
+
class Foobar < Wayfarer::Base
|
29
|
+
route { to: :index }
|
30
|
+
|
31
|
+
retry_on Ferrum::DeadBrowserError, attempts: 3, wait: :exponentially_longer
|
32
|
+
|
33
|
+
# ...
|
34
|
+
end
|
35
|
+
```
|
36
|
+
|
37
|
+
This leads to log entries like:
|
38
|
+
|
39
|
+
```
|
40
|
+
Retrying DummyJob in 3 seconds, due to a Ferrum::DeadBrowserError.
|
41
|
+
```
|
@@ -0,0 +1,30 @@
|
|
1
|
+
# Steering
|
2
|
+
|
3
|
+
A job's router can receive arguments computed dynamically by `::steer`.
|
4
|
+
Steering enables [batch routing](/cookbook/batch_routing).
|
5
|
+
|
6
|
+
For example, the following router has hostname and path hard-coded:
|
7
|
+
|
8
|
+
```ruby
|
9
|
+
class DummyJob < Wayfarer::Base
|
10
|
+
route do
|
11
|
+
host "example.com", path: "/contact", to: :index
|
12
|
+
end
|
13
|
+
end
|
14
|
+
```
|
15
|
+
|
16
|
+
Instead, hostname and path could be provided by `::steer`, too:
|
17
|
+
|
18
|
+
```ruby
|
19
|
+
class DummyJob < Wayfarer::Base
|
20
|
+
route do |hostname, path|
|
21
|
+
host hostname, path: path, to: :index
|
22
|
+
end
|
23
|
+
|
24
|
+
steer do |_task|
|
25
|
+
["example.com", "/contact"]
|
26
|
+
end
|
27
|
+
end
|
28
|
+
```
|
29
|
+
|
30
|
+
Note that `steer` yields the current [task](/guides/tasks).
|
@@ -0,0 +1,14 @@
|
|
1
|
+
# Tasks
|
2
|
+
|
3
|
+
Tasks are the immutable units of work processed by [jobs](/guides/jobs). A task
|
4
|
+
consists of:
|
5
|
+
|
6
|
+
1. The __URL__ to process
|
7
|
+
* Within a batch, every URL gets processed at most once.
|
8
|
+
|
9
|
+
2. The __batch__ the task belongs to
|
10
|
+
* Like URLs, batches are strings.
|
11
|
+
|
12
|
+
Tasks get appended to the end of a message queue, and consumed from the
|
13
|
+
beginning. Because jobs can enqueue other tasks, jobs are both consumers
|
14
|
+
and producers of tasks.
|
data/docs/index.md
CHANGED
@@ -1,82 +1,56 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
|
2
|
+
hide:
|
3
|
+
- navigation
|
4
4
|
---
|
5
5
|
|
6
|
-
#
|
6
|
+
# Wayfarer
|
7
7
|
|
8
|
-
|
8
|
+
![CI status](https://github.com/actions/starter-workflows/workflows/CI/badge.svg)
|
9
|
+
[![RubyGem](https://badge.fury.io/rb/wayfarer.svg)](https://rubygems.org/gems/wayfarer)
|
9
10
|
|
10
|
-
|
11
|
-
```
|
12
|
-
% [sudo] gem install wayfarer
|
13
|
-
```
|
14
|
-
|
15
|
-
JRuby:
|
16
|
-
```
|
17
|
-
% [sudo] gem install wayfarer-jruby
|
18
|
-
```
|
19
|
-
|
20
|
-
If you …
|
21
|
-
|
22
|
-
* __Need to crawl page graphs breadth-first__
|
23
|
-
* __Need to extract whatever data__
|
24
|
-
* Do it multi-threaded
|
25
|
-
* Integrate with Rails seamlessly
|
26
|
-
* Want to automate a web browser
|
27
|
-
* Need to execute arbitrary JavaScript
|
28
|
-
* Need URI normalization
|
29
|
-
* Need to take screenshots
|
30
|
-
* Want to use a job queue and make work happen later
|
31
|
-
* Want in-memory and Redis-backed frontiers, tries and Bloom filters
|
11
|
+
## Versatile web crawling with Ruby
|
32
12
|
|
33
|
-
|
13
|
+
* Web scraping
|
14
|
+
* Data extraction
|
15
|
+
* Browser automation
|
34
16
|
|
35
|
-
|
17
|
+
!!! attention "Unstable software"
|
36
18
|
|
37
|
-
|
19
|
+
Wayfarer is under development and releases should be considered unstable.
|
38
20
|
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
* Extract keywords from every page you encounter
|
44
|
-
* Use 4 threads and Chrome processes to do so
|
21
|
+
Wayfarer complies to
|
22
|
+
[Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html) in
|
23
|
+
which v0.x means that there could be backward-incompatible changes for every
|
24
|
+
release:
|
45
25
|
|
46
|
-
|
26
|
+
>Major version zero (0.y.z) is for initial development. Anything MAY change
|
27
|
+
at any time. The public API SHOULD NOT be considered stable.
|
47
28
|
|
48
|
-
|
49
|
-
require "wayfarer"
|
29
|
+
### Installation
|
50
30
|
|
51
|
-
|
52
|
-
config.http_adapter = :selenium
|
53
|
-
config.selenium_argv = [:chrome]
|
54
|
-
config.connection_count = 4
|
31
|
+
Install the RubyGem:
|
55
32
|
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
def article
|
61
|
-
if page.body =~ /Foobar/
|
62
|
-
driver.save_screenshot("/tmp/foobar.png")
|
63
|
-
return halt
|
64
|
-
end
|
65
|
-
|
66
|
-
keywords << page.keywords
|
67
|
-
stage page.links
|
68
|
-
end
|
69
|
-
end
|
70
|
-
|
71
|
-
FindFoobarOnWikipedia.perform_now("https://en.wikipedia.org/wiki/Special:Random")
|
72
|
-
{% endhighlight %}
|
33
|
+
```
|
34
|
+
gem install wayfarer
|
35
|
+
```
|
73
36
|
|
74
|
-
|
75
|
-
{% highlight ruby %}
|
76
|
-
FindFoobarOnWikipedia.perform_later("https://en.wikipedia.org/wiki/Special:Random")
|
77
|
-
{% endhighlight %}
|
37
|
+
Or add it to Bundler's Gemfile:
|
78
38
|
|
79
|
-
|
39
|
+
```ruby
|
40
|
+
gem "wayfarer"
|
41
|
+
```
|
80
42
|
|
81
|
-
|
82
|
-
|
43
|
+
### Features
|
44
|
+
|
45
|
+
* Breadth-first, acyclic, multi-threaded graph traversal
|
46
|
+
* Executes atop a variety of message queues thanks to [ActiveJob](https://edgeguides.rubyonrails.org/active_job_basics.html)
|
47
|
+
* Browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
|
48
|
+
(<abbr title="Chrome DevTools Protocol">CDP</abbr>),
|
49
|
+
[Selenium](https://www.selenium.dev) or plain HTTP via `net/http`
|
50
|
+
* Declarative routing DSL
|
51
|
+
* URI normalization and deduplication
|
52
|
+
* XML, HTML, JSON parsing
|
53
|
+
* HTTP redirect handling
|
54
|
+
* Storage-agnostic
|
55
|
+
* Small footprint: <500 LoC
|
56
|
+
* Open Source (MIT)
|
@@ -0,0 +1,48 @@
|
|
1
|
+
---
|
2
|
+
title: Wayfarer::Base
|
3
|
+
---
|
4
|
+
|
5
|
+
# `Wayfarer::Base`
|
6
|
+
|
7
|
+
Wayfarer's complete job API.
|
8
|
+
|
9
|
+
---
|
10
|
+
|
11
|
+
### `::route`
|
12
|
+
: Draw routes to instance methods.
|
13
|
+
|
14
|
+
---
|
15
|
+
|
16
|
+
### `::steer { (Wayfarer::Task) -> [any] }`
|
17
|
+
: Provide router arguments.
|
18
|
+
|
19
|
+
---
|
20
|
+
|
21
|
+
### `#task -> Wayfarer::Task`
|
22
|
+
: The currently processing task.
|
23
|
+
|
24
|
+
---
|
25
|
+
|
26
|
+
### `#params -> Hash`
|
27
|
+
: URL parameters collected from the matching route.
|
28
|
+
|
29
|
+
---
|
30
|
+
|
31
|
+
### `#stage(String | [String]) -> void`
|
32
|
+
: Add URLs to a processing set. URLs already processed within the
|
33
|
+
current batch get discarded are not enqueued. Every staged URL gets
|
34
|
+
normalized.
|
35
|
+
|
36
|
+
---
|
37
|
+
|
38
|
+
### `#browser -> Object`
|
39
|
+
: The user agent that retrieved the current page.
|
40
|
+
|
41
|
+
---
|
42
|
+
|
43
|
+
### `#page(live: true | false) -> Page`
|
44
|
+
: The page representing the response retrieved from the currently
|
45
|
+
processing URL.
|
46
|
+
|
47
|
+
With `live: true` called, a fresh `Page` is returned that reflects the
|
48
|
+
current browser DOM. Calls to `#page` return the most recent page.
|
@@ -0,0 +1,182 @@
|
|
1
|
+
---
|
2
|
+
title: Wayfarer::Routing::DSL
|
3
|
+
---
|
4
|
+
|
5
|
+
# `Wayfarer::Route`
|
6
|
+
|
7
|
+
All public functionality documented here is exposed via
|
8
|
+
`Wayfarer::Routing::DSL`.
|
9
|
+
|
10
|
+
---
|
11
|
+
|
12
|
+
### `#host(String | Regexp)`
|
13
|
+
: Match hosts against strings or regular expressions.
|
14
|
+
|
15
|
+
##### Examples
|
16
|
+
|
17
|
+
!!! example "Exact hostname string matching"
|
18
|
+
|
19
|
+
```ruby
|
20
|
+
route.host "example.com"
|
21
|
+
```
|
22
|
+
|
23
|
+
- [x] https://{==example.com==}
|
24
|
+
- [x] https://{==example.com==}/foo/bar
|
25
|
+
- [ ] https://sub.example.com
|
26
|
+
- [ ] https://nasa.gov
|
27
|
+
|
28
|
+
!!! example "Hostname matching with regular expression"
|
29
|
+
|
30
|
+
```ruby
|
31
|
+
route.host /example.com$/
|
32
|
+
```
|
33
|
+
|
34
|
+
- [x] https://{==example.com==}
|
35
|
+
- [x] https://{==example.com==}/foo/bar
|
36
|
+
- [x] https://{==sub.example.com==}
|
37
|
+
- [ ] https://nasa.gov
|
38
|
+
|
39
|
+
---
|
40
|
+
|
41
|
+
### `#suffix(String)`
|
42
|
+
: Match URL paths against exact suffixes.
|
43
|
+
|
44
|
+
##### Examples
|
45
|
+
|
46
|
+
!!! example "Match only URLs with paths ending with \*.png"
|
47
|
+
|
48
|
+
```ruby
|
49
|
+
route.suffix ".png"
|
50
|
+
```
|
51
|
+
|
52
|
+
- [x] https://example.com/image{==.png==}
|
53
|
+
- [ ] https://example.com/index.html
|
54
|
+
- [ ] https://example.com
|
55
|
+
|
56
|
+
---
|
57
|
+
|
58
|
+
### `#path(String)`
|
59
|
+
: Match URLs against their path.
|
60
|
+
|
61
|
+
##### Examples
|
62
|
+
|
63
|
+
!!! example "Exact path string matching"
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
route.path "/users/123"
|
67
|
+
```
|
68
|
+
|
69
|
+
- [x] https://example.com/{==users/123==}
|
70
|
+
- [x] https://example.com/{==users/123==}
|
71
|
+
- [ ] https://example.com/users
|
72
|
+
- [ ] https://example.com
|
73
|
+
|
74
|
+
!!! example "Root path matching"
|
75
|
+
|
76
|
+
```ruby
|
77
|
+
route.path "/"
|
78
|
+
```
|
79
|
+
|
80
|
+
- [x] https://example.com{==/==}
|
81
|
+
- [ ] https://example.com
|
82
|
+
- [ ] https://example.com/users
|
83
|
+
|
84
|
+
!!! example "Pattern matching"
|
85
|
+
|
86
|
+
```ruby
|
87
|
+
route.path "/users/:id", to: :with_params
|
88
|
+
route.path "/contact", to: :without_params
|
89
|
+
|
90
|
+
def with_params
|
91
|
+
params # => { "id" => ... }
|
92
|
+
end
|
93
|
+
|
94
|
+
def without_params
|
95
|
+
params # => {}
|
96
|
+
end
|
97
|
+
```
|
98
|
+
|
99
|
+
- [x] https://example.com{==/users/123==}
|
100
|
+
- [x] https://example.com{==/contact==}
|
101
|
+
- [ ] https://example.com
|
102
|
+
- [ ] https://example.com/users/123/photos
|
103
|
+
- [ ] https://example.com/contact/
|
104
|
+
|
105
|
+
---
|
106
|
+
|
107
|
+
### `#query(Hash)`
|
108
|
+
: Match URLs against query parameters.
|
109
|
+
|
110
|
+
##### Examples
|
111
|
+
|
112
|
+
!!! example "Exact parameter string matching"
|
113
|
+
|
114
|
+
```ruby
|
115
|
+
route.query arg: "foo"
|
116
|
+
```
|
117
|
+
|
118
|
+
- [x] https://example.com?{==arg=foo==}
|
119
|
+
- [x] https://example.com?{==arg=foo==}&page=2
|
120
|
+
- [ ] https://example.com?arg=123
|
121
|
+
- [ ] https://example.com
|
122
|
+
|
123
|
+
!!! example "Parameter regular expression matching"
|
124
|
+
|
125
|
+
```ruby
|
126
|
+
route.query arg: /foo/
|
127
|
+
```
|
128
|
+
|
129
|
+
- [x] https://example.com?{==arg=foo==}
|
130
|
+
- [x] https://example.com?{==arg=foobar==}
|
131
|
+
- [x] https://example.com?{==arg=barfoo==}&page=2
|
132
|
+
- [ ] https://example.com?arg=123
|
133
|
+
- [ ] https://example.com
|
134
|
+
|
135
|
+
!!! example "Parameter integer matching"
|
136
|
+
|
137
|
+
```ruby
|
138
|
+
route.query page: 5
|
139
|
+
```
|
140
|
+
|
141
|
+
- [x] https://example.com?{==page=5==}
|
142
|
+
- [x] https://example.com?arg=foo&{==page=5==}
|
143
|
+
- [ ] https://example.com?page=3
|
144
|
+
- [ ] https://example.com
|
145
|
+
|
146
|
+
!!! example "Parameter range matching"
|
147
|
+
|
148
|
+
```ruby
|
149
|
+
route.query page: 1..10
|
150
|
+
```
|
151
|
+
|
152
|
+
- [x] https://example.com?{==page=1==}
|
153
|
+
- [x] https://example.com?arg=foo&{==page=10==}
|
154
|
+
- [ ] https://example.com?page=11
|
155
|
+
- [ ] https://example.com
|
156
|
+
|
157
|
+
### `#scheme(Symbol)`
|
158
|
+
: Match URLs by their scheme.
|
159
|
+
|
160
|
+
##### Examples
|
161
|
+
|
162
|
+
!!! example "Scheme matching"
|
163
|
+
|
164
|
+
```ruby
|
165
|
+
route.scheme :https
|
166
|
+
```
|
167
|
+
|
168
|
+
- [x] {==https==}://example.com
|
169
|
+
- [ ] http://example.com
|
170
|
+
|
171
|
+
### `#uri(String)`
|
172
|
+
: Match URLs exactly.
|
173
|
+
|
174
|
+
##### Examples
|
175
|
+
|
176
|
+
!!! example "URL matching"
|
177
|
+
|
178
|
+
```ruby
|
179
|
+
route.uri "https://example.com/foo/bar?page=3"
|
180
|
+
```
|
181
|
+
|
182
|
+
- [x] {==https://example.com/foo/bar?page=3==}
|
@@ -0,0 +1,61 @@
|
|
1
|
+
# Command Line Interface
|
2
|
+
|
3
|
+
## Usage
|
4
|
+
|
5
|
+
```
|
6
|
+
wayfarer [OPTIONS] [generate|job|route|version]
|
7
|
+
```
|
8
|
+
|
9
|
+
All [environment variables](../environment_variables) are respected.
|
10
|
+
|
11
|
+
## `wayfarer generate`
|
12
|
+
|
13
|
+
### `wayfarer generate project NAME`
|
14
|
+
|
15
|
+
: Generates a new project directory `NAME`.
|
16
|
+
|
17
|
+
## `wayfarer job`
|
18
|
+
|
19
|
+
### `wayfarer job perform JOB URL`
|
20
|
+
|
21
|
+
: Performs `JOB` with `URL`. The job does not reach any Active Job backend.
|
22
|
+
Staged jobs will not be processed.
|
23
|
+
|
24
|
+
##### Options
|
25
|
+
|
26
|
+
* `--mock-redis`: Use an in-memory implementation of Redis instead of
|
27
|
+
talking to an actual server.
|
28
|
+
* `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
|
29
|
+
|
30
|
+
### `wayfarer job enqueue JOB URL`
|
31
|
+
|
32
|
+
: Enqueues `JOB` with `URL` to the configured Active Job backend.
|
33
|
+
|
34
|
+
##### Options
|
35
|
+
|
36
|
+
* `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
|
37
|
+
|
38
|
+
### `wayfarer job execute JOB URL`
|
39
|
+
|
40
|
+
: Execute `JOB` with `URL` by using the
|
41
|
+
[Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html).
|
42
|
+
The job does not reach any other Active Job backend. Blocks until the batch
|
43
|
+
completes.
|
44
|
+
|
45
|
+
##### Options
|
46
|
+
|
47
|
+
* `--mock-redis`: Use an in-memory implementation of Redis instead of
|
48
|
+
talking to an actual server.
|
49
|
+
* `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
|
50
|
+
* `--min-threads`: Minimum number of threads to use. Default: 1
|
51
|
+
* `--max-threads`: Maximum number of threads to use. Default: 1
|
52
|
+
|
53
|
+
## `wayfarer route`
|
54
|
+
|
55
|
+
### `wayfarer route result JOB URL`
|
56
|
+
|
57
|
+
: Prints the result of invoking `JOB`'s router with `URL`.
|
58
|
+
|
59
|
+
### `wayfarer route tree JOB URL`
|
60
|
+
|
61
|
+
: Visualises the routing tree result of invoking `JOB`'s router with `URL`.
|
@@ -0,0 +1,42 @@
|
|
1
|
+
---
|
2
|
+
hide:
|
3
|
+
- toc
|
4
|
+
---
|
5
|
+
|
6
|
+
# Configuration Keys
|
7
|
+
|
8
|
+
## `Wayfarer.config.network`
|
9
|
+
|
10
|
+
| Runtime config key | Environment variable | Description | Default | Supported values |
|
11
|
+
| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
|
12
|
+
| `network.agent` | `WAYFARER_NETWORK_AGENT` | The user agent to use. | `:http` | `:http`, `:ferrum`, `:selenium` |
|
13
|
+
| `network.pool_size` | `WAYFARER_NETWORK_POOL_SIZE` | How many user agents to spawn. | 1 | Integers |
|
14
|
+
| `network.pool_timeout` | `WAYFARER_NETWORK_POOL_TIMEOUT` | How long jobs may use an agent in seconds. | 10 | Integers |
|
15
|
+
| `network.http_headers` | `WAYFARER_NETWORK_HTTP_HEADERS` | HTTP headers to append to requests. | `{}` | Hashes |
|
16
|
+
|
17
|
+
## `Wayfarer.config.ferrum`
|
18
|
+
|
19
|
+
| Runtime config key | Environment variable | Description | Default | Supported values |
|
20
|
+
| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
|
21
|
+
| `ferrum.options` | `WAYFARER_FERRUM_OPTIONS` | Ferrum options. | `{}` | Hashes |
|
22
|
+
|
23
|
+
## `Wayfarer.config.selenium`
|
24
|
+
|
25
|
+
| Runtime config key | Environment variable | Description | Default | Supported values |
|
26
|
+
| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
|
27
|
+
| `selenium.driver` | `WAYFARER_SELENIUM_DRIVER` | Selenium driver to use. | `:chrome` | Symbols |
|
28
|
+
| `selenium.options` | `WAYFARER_SELENIUM_OPTIONS` | Selenium options. | `{}` | Hashes |
|
29
|
+
| `selenium.client_timeout` | `WAYFARER_SELENIUM_CLIENT_TIMEOUT` | Selenium client timeout in seconds. | 60 | Integers |
|
30
|
+
|
31
|
+
## `Wayfarer.config.redis`
|
32
|
+
|
33
|
+
| Runtime config key | Environment variable | Description | Default | Supported values |
|
34
|
+
| ---------------------- | ------------------------------------ | ------------------------------------------- | ------------------------------------------ | ----------------------------------- |
|
35
|
+
| `redis.url` | `WAYFARER_REDIS_URL` | Redis URL to connect to. | http://localhost:6379 | Strings |
|
36
|
+
| `redis.factory` | n/a | Redis factory lambda. | ` ->(redis) { ::Redis.new(url: redis.url)` | Lambdas |
|
37
|
+
|
38
|
+
## `Wayfarer.config.capybara`
|
39
|
+
|
40
|
+
| Runtime config key | Environment variable | Description | Default | Supported values |
|
41
|
+
| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
|
42
|
+
| `capybara.driver` | `WAYFARER_CAPYBARA_DRIVER` | The Capybara driver to use. | n/a | Symbols |
|