RubyGems - wayfarer - Versions diffs - 0.4.6 → 0.4.8 - Mend

wayfarer 0.4.6 → 0.4.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (259) hide show

checksums.yaml +4 -4
data/.env +17 -0
data/.github/workflows/lint.yaml +27 -0
data/.github/workflows/release.yaml +30 -0
data/.github/workflows/tests.yaml +21 -0
data/.gitignore +5 -1
data/.rubocop.yml +36 -0
data/.vale.ini +8 -0
data/.yardopts +1 -3
data/Dockerfile +6 -4
data/Gemfile +24 -0
data/Gemfile.lock +274 -164
data/Rakefile +7 -51
data/bin/wayfarer +1 -1
data/docker-compose.yml +23 -13
data/docs/cookbook/consent_screen.md +2 -2
data/docs/cookbook/executing_javascript.md +3 -3
data/docs/cookbook/navigation.md +12 -12
data/docs/cookbook/querying_html.md +3 -3
data/docs/cookbook/screenshots.md +2 -2
data/docs/guides/callbacks.md +25 -125
data/docs/guides/cli.md +71 -0
data/docs/guides/configuration.md +10 -35
data/docs/guides/development.md +67 -0
data/docs/guides/handlers.md +60 -0
data/docs/guides/index.md +1 -0
data/docs/guides/jobs.md +142 -31
data/docs/guides/navigation.md +1 -1
data/docs/guides/networking/capybara.md +13 -22
data/docs/guides/networking/custom_adapters.md +103 -41
data/docs/guides/networking/ferrum.md +4 -4
data/docs/guides/networking/http.md +9 -13
data/docs/guides/networking/selenium.md +10 -11
data/docs/guides/pages.md +78 -10
data/docs/guides/redis.md +10 -0
data/docs/guides/routing.md +156 -0
data/docs/guides/tasks.md +53 -9
data/docs/guides/tutorial.md +66 -0
data/docs/guides/user_agents.md +115 -0
data/docs/index.md +17 -40
data/lib/wayfarer/base.rb +125 -46
data/lib/wayfarer/batch_completion.rb +60 -0
data/lib/wayfarer/callbacks.rb +22 -48
data/lib/wayfarer/cli/route_printer.rb +85 -89
data/lib/wayfarer/cli.rb +103 -0
data/lib/wayfarer/gc.rb +18 -6
data/lib/wayfarer/handler.rb +15 -7
data/lib/wayfarer/kv.rb +28 -0
data/lib/wayfarer/logging.rb +38 -0
data/lib/wayfarer/middleware/base.rb +2 -0
data/lib/wayfarer/middleware/batch_completion.rb +19 -0
data/lib/wayfarer/middleware/chain.rb +7 -1
data/lib/wayfarer/middleware/content_type.rb +59 -0
data/lib/wayfarer/middleware/controller.rb +19 -15
data/lib/wayfarer/middleware/dedup.rb +22 -13
data/lib/wayfarer/middleware/dispatch.rb +17 -4
data/lib/wayfarer/middleware/normalize.rb +7 -14
data/lib/wayfarer/middleware/redis.rb +15 -0
data/lib/wayfarer/middleware/router.rb +33 -35
data/lib/wayfarer/middleware/stage.rb +5 -5
data/lib/wayfarer/middleware/uri_parser.rb +31 -0
data/lib/wayfarer/middleware/user_agent.rb +49 -0
data/lib/wayfarer/networking/capybara.rb +1 -1
data/lib/wayfarer/networking/context.rb +14 -3
data/lib/wayfarer/networking/ferrum.rb +1 -4
data/lib/wayfarer/networking/follow.rb +14 -7
data/lib/wayfarer/networking/http.rb +1 -1
data/lib/wayfarer/networking/pool.rb +23 -13
data/lib/wayfarer/networking/selenium.rb +15 -7
data/lib/wayfarer/networking/strategy.rb +2 -2
data/lib/wayfarer/page.rb +34 -14
data/lib/wayfarer/parsing/xml.rb +6 -6
data/lib/wayfarer/parsing.rb +21 -0
data/lib/wayfarer/redis/barrier.rb +26 -21
data/lib/wayfarer/redis/counter.rb +18 -9
data/lib/wayfarer/redis/pool.rb +1 -1
data/lib/wayfarer/redis/resettable.rb +19 -0
data/lib/wayfarer/routing/dsl.rb +166 -30
data/lib/wayfarer/routing/hash_stack.rb +33 -0
data/lib/wayfarer/routing/matchers/custom.rb +8 -5
data/lib/wayfarer/routing/matchers/{suffix.rb → empty_params.rb} +2 -6
data/lib/wayfarer/routing/matchers/host.rb +15 -9
data/lib/wayfarer/routing/matchers/path.rb +11 -31
data/lib/wayfarer/routing/matchers/query.rb +41 -17
data/lib/wayfarer/routing/matchers/result.rb +12 -0
data/lib/wayfarer/routing/matchers/scheme.rb +13 -5
data/lib/wayfarer/routing/matchers/url.rb +13 -5
data/lib/wayfarer/routing/path_consumer.rb +130 -0
data/lib/wayfarer/routing/path_finder.rb +151 -23
data/lib/wayfarer/routing/result.rb +1 -1
data/lib/wayfarer/routing/root_route.rb +17 -1
data/lib/wayfarer/routing/route.rb +66 -19
data/lib/wayfarer/routing/serializable.rb +28 -0
data/lib/wayfarer/routing/sub_route.rb +53 -0
data/lib/wayfarer/routing/target_route.rb +17 -1
data/lib/wayfarer/stringify.rb +21 -30
data/lib/wayfarer/task.rb +9 -17
data/lib/wayfarer/uri/normalization.rb +120 -0
data/lib/wayfarer.rb +72 -5
data/mise.toml +2 -0
data/mkdocs.yml +44 -8
data/rake/docs.rake +26 -0
data/rake/lint.rake +9 -0
data/rake/release.rake +23 -0
data/rake/tests.rake +32 -0
data/requirements.txt +1 -1
data/spec/factories/job.rb +8 -0
data/spec/factories/middleware.rb +2 -2
data/spec/factories/path_finder.rb +11 -0
data/spec/factories/redis.rb +19 -0
data/spec/factories/task.rb +46 -2
data/spec/spec_helpers.rb +55 -51
data/spec/support/active_job_helpers.rb +8 -0
data/spec/support/integration_helpers.rb +21 -0
data/spec/support/redis_helpers.rb +9 -0
data/spec/support/test_app.rb +66 -37
data/spec/wayfarer/base_spec.rb +200 -0
data/spec/wayfarer/batch_completion_spec.rb +142 -0
data/spec/wayfarer/cli/job_spec.rb +88 -0
data/spec/wayfarer/cli/routing_spec.rb +322 -0
data/spec/{cli → wayfarer/cli}/version_spec.rb +1 -1
data/spec/wayfarer/gc_spec.rb +29 -0
data/spec/wayfarer/handler_spec.rb +9 -0
data/spec/wayfarer/integration/callbacks_spec.rb +200 -0
data/spec/wayfarer/integration/content_type_spec.rb +37 -0
data/spec/wayfarer/integration/custom_routing_spec.rb +51 -0
data/spec/wayfarer/integration/gc_spec.rb +40 -0
data/spec/wayfarer/integration/handler_spec.rb +65 -0
data/spec/wayfarer/integration/page_spec.rb +79 -0
data/spec/wayfarer/integration/params_spec.rb +64 -0
data/spec/wayfarer/integration/parsing_spec.rb +99 -0
data/spec/wayfarer/integration/retry_spec.rb +112 -0
data/spec/wayfarer/integration/stage_spec.rb +58 -0
data/spec/wayfarer/middleware/batch_completion_spec.rb +33 -0
data/spec/{middleware → wayfarer/middleware}/chain_spec.rb +24 -19
data/spec/wayfarer/middleware/content_type_spec.rb +83 -0
data/spec/{middleware → wayfarer/middleware}/controller_spec.rb +24 -22
data/spec/wayfarer/middleware/dedup_spec.rb +66 -0
data/spec/wayfarer/middleware/normalize_spec.rb +32 -0
data/spec/wayfarer/middleware/router_spec.rb +102 -0
data/spec/wayfarer/middleware/stage_spec.rb +63 -0
data/spec/wayfarer/middleware/uri_parser_spec.rb +63 -0
data/spec/wayfarer/middleware/user_agent_spec.rb +158 -0
data/spec/wayfarer/networking/capybara_spec.rb +13 -0
data/spec/{networking → wayfarer/networking}/context_spec.rb +46 -38
data/spec/wayfarer/networking/ferrum_spec.rb +13 -0
data/spec/{networking → wayfarer/networking}/follow_spec.rb +11 -6
data/spec/wayfarer/networking/http_spec.rb +12 -0
data/spec/{networking → wayfarer/networking}/pool_spec.rb +16 -14
data/spec/wayfarer/networking/selenium_spec.rb +12 -0
data/spec/{networking → wayfarer/networking}/strategy.rb +33 -54
data/spec/wayfarer/page_spec.rb +69 -0
data/spec/{parsing → wayfarer/parsing}/json_spec.rb +1 -1
data/spec/wayfarer/parsing/xml_parse_spec.rb +25 -0
data/spec/wayfarer/redis/barrier_spec.rb +39 -0
data/spec/wayfarer/redis/counter_spec.rb +34 -0
data/spec/{redis → wayfarer/redis}/pool_spec.rb +4 -3
data/spec/{routing → wayfarer/routing}/dsl_spec.rb +12 -22
data/spec/wayfarer/routing/hash_stack_spec.rb +63 -0
data/spec/wayfarer/routing/integration_spec.rb +101 -0
data/spec/wayfarer/routing/matchers/custom_spec.rb +39 -0
data/spec/wayfarer/routing/matchers/host_spec.rb +56 -0
data/spec/wayfarer/routing/matchers/matcher.rb +17 -0
data/spec/wayfarer/routing/matchers/path_spec.rb +43 -0
data/spec/wayfarer/routing/matchers/query_spec.rb +123 -0
data/spec/wayfarer/routing/matchers/scheme_spec.rb +45 -0
data/spec/wayfarer/routing/matchers/url_spec.rb +33 -0
data/spec/wayfarer/routing/path_consumer_spec.rb +123 -0
data/spec/wayfarer/routing/path_finder_spec.rb +409 -0
data/spec/wayfarer/routing/root_route_spec.rb +51 -0
data/spec/wayfarer/routing/route_spec.rb +74 -0
data/spec/wayfarer/routing/sub_route_spec.rb +103 -0
data/spec/wayfarer/task_spec.rb +13 -0
data/spec/wayfarer/uri/normalization_spec.rb +98 -0
data/spec/wayfarer_spec.rb +2 -2
data/wayfarer.gemspec +18 -28
metadata +797 -265
data/.github/workflows/ci.yaml +0 -32
data/.rbenv-gemsets +0 -1
data/.ruby-version +0 -1
data/RELEASING.md +0 -17
data/docs/cookbook/user_agent.md +0 -7
data/docs/guides/error_handling.md +0 -53
data/docs/guides/networking.md +0 -94
data/docs/guides/performance.md +0 -130
data/docs/guides/reliability.md +0 -41
data/docs/guides/routing/steering.md +0 -30
data/docs/reference/api/base.md +0 -48
data/docs/reference/cli.md +0 -61
data/docs/reference/configuration_keys.md +0 -43
data/docs/reference/environment_variables.md +0 -83
data/lib/wayfarer/cli/base.rb +0 -45
data/lib/wayfarer/cli/generate.rb +0 -17
data/lib/wayfarer/cli/job.rb +0 -56
data/lib/wayfarer/cli/route.rb +0 -29
data/lib/wayfarer/cli/runner.rb +0 -34
data/lib/wayfarer/cli/templates/Gemfile.tt +0 -5
data/lib/wayfarer/cli/templates/job.rb.tt +0 -10
data/lib/wayfarer/config/capybara.rb +0 -10
data/lib/wayfarer/config/ferrum.rb +0 -11
data/lib/wayfarer/config/networking.rb +0 -29
data/lib/wayfarer/config/redis.rb +0 -14
data/lib/wayfarer/config/root.rb +0 -11
data/lib/wayfarer/config/selenium.rb +0 -21
data/lib/wayfarer/config/strconv.rb +0 -45
data/lib/wayfarer/config/struct.rb +0 -72
data/lib/wayfarer/middleware/fetch.rb +0 -56
data/lib/wayfarer/redis/connection.rb +0 -13
data/lib/wayfarer/redis/version.rb +0 -19
data/lib/wayfarer/routing/router.rb +0 -28
data/spec/base_spec.rb +0 -224
data/spec/callbacks_spec.rb +0 -102
data/spec/cli/generate_spec.rb +0 -39
data/spec/cli/job_spec.rb +0 -78
data/spec/config/capybara_spec.rb +0 -18
data/spec/config/ferrum_spec.rb +0 -24
data/spec/config/networking_spec.rb +0 -73
data/spec/config/redis_spec.rb +0 -32
data/spec/config/root_spec.rb +0 -31
data/spec/config/selenium_spec.rb +0 -56
data/spec/config/strconv_spec.rb +0 -58
data/spec/config/struct_spec.rb +0 -66
data/spec/fixtures/dummy_job.rb +0 -7
data/spec/gc_spec.rb +0 -59
data/spec/handler_spec.rb +0 -11
data/spec/integration/callbacks_spec.rb +0 -85
data/spec/integration/page_spec.rb +0 -62
data/spec/integration/params_spec.rb +0 -56
data/spec/integration/stage_spec.rb +0 -51
data/spec/integration/steering_spec.rb +0 -57
data/spec/middleware/dedup_spec.rb +0 -88
data/spec/middleware/dispatch_spec.rb +0 -43
data/spec/middleware/fetch_spec.rb +0 -155
data/spec/middleware/normalize_spec.rb +0 -29
data/spec/middleware/router_spec.rb +0 -105
data/spec/middleware/stage_spec.rb +0 -62
data/spec/networking/capybara_spec.rb +0 -12
data/spec/networking/ferrum_spec.rb +0 -12
data/spec/networking/http_spec.rb +0 -12
data/spec/networking/selenium_spec.rb +0 -12
data/spec/page_spec.rb +0 -47
data/spec/parsing/xml_spec.rb +0 -25
data/spec/redis/barrier_spec.rb +0 -78
data/spec/redis/counter_spec.rb +0 -32
data/spec/redis/version_spec.rb +0 -13
data/spec/routing/integration_spec.rb +0 -110
data/spec/routing/matchers/custom_spec.rb +0 -31
data/spec/routing/matchers/host_spec.rb +0 -49
data/spec/routing/matchers/path_spec.rb +0 -43
data/spec/routing/matchers/query_spec.rb +0 -137
data/spec/routing/matchers/scheme_spec.rb +0 -25
data/spec/routing/matchers/suffix_spec.rb +0 -41
data/spec/routing/matchers/uri_spec.rb +0 -27
data/spec/routing/path_finder_spec.rb +0 -33
data/spec/routing/root_route_spec.rb +0 -29
data/spec/routing/route_spec.rb +0 -43
data/spec/routing/router_spec.rb +0 -24
data/spec/task_spec.rb +0 -34
data/spec/{stringify_spec.rb → wayfarer/stringify_spec.rb} +2 -2

data/docs/guides/pages.md CHANGED Viewed

@@ -1,11 +1,14 @@
 # Pages
-Retrieved pages take the shape of `Wayfarer::Page` objects and are available
-to jobs:
+A page is the immutable state of the contents behind a URL at a point in time,
+retrieved by a [user agent](user-agents.md). In other words, a page is an HTTP
+response, or the state of a remotely controlled browser.
 ```ruby
-class DummyJob < Wayfarer::Worker
-  route { to :index }
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+  route.to :index
   def index
     page # => #<Wayfarer::Page ...>
@@ -13,10 +16,14 @@ class DummyJob < Wayfarer::Worker
     page.url         # => "https://example.com"
     page.body        # => "<html>..."
     page.status_code # => 200
-    page.headers     # => { "Content-Type" => ... }
+    page.headers     # => { "content-type" => ... }
+    page.mime_type   # => #<MIME::Type: text/html>
+    # The lazily parsed response body or `nil`, depending on the Content-Type
+    page.doc # => #<Nokogiri::HTML::Document ...>
-    # A MetaInspector object for accessing page meta data.
     # See: https://github.com/metainspector/metainspector
+    page.meta # => #<MetaInspector::Document ...>
     # Examples:
     page.meta.links.internal
     page.meta.images.favicon
@@ -26,20 +33,63 @@ class DummyJob < Wayfarer::Worker
 end
 ```
+!!! info "HTTP headers are downcased and case-sensitive"
+    HTTP headers are downcased, so you would access
+    `page.headers["content-type"]` instead of `page.headers["Content-Type"]`.
+## Response body parsing
+Wayfarer parses the bodies of HTML, XML and JSON responses according to their
+MIME types:
+* `application/html` to [`#!ruby Nokogiri::HTML::Document`](https://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/HTML/Document)
+* `text/xml` or `application/xml` to [`#!ruby Nokogiri::XML::Document`](https://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Document)
+* `application/json` to `Hash`
+### Implementing a custom response body parser
+You can register an object that implements a `#parse` method for any MIME type:
+```ruby
+class MyJPEGParser
+  def parse(body)
+    # Read EXIF metadata here.
+    # Return value is accessible as `page.doc`
+  end
+end
+Wayfarer::Parsing.registry["image/jpeg"] = MyJPEGParser.new
+```
+!!! warning "`#parse` must be thread-safe!"
+!!! info "Handling responses without a Content-Type"
+    If a response has no `Content-Type` header, Wayfarer falls back to
+    `application/octet-stream`. A parser registered for
+    `application/octet-stream` will hence also handle all responses without
+    a Content-Type.
 ## Live pages
-When automating browsers, it is possible the page changes significantly at
-runtime, for example due to JavaScript altering the DOM or URL.
+`#!ruby page` initially returns a snapshot of the browser state
+immediately after the user agent navigated to the URL. The browser state may
+change significantly after the page was retrieved, for example due to your own
+interaction, or client-side JavaScript altering the DOM or URL.
-To access a page reflecting the current browser state, pass the `live` keyword:
+To get a page that reflects the current browser state, set the `#!ruby :live`
+keyword:
 ```ruby
 class DummyJob < Wayfarer::Worker
-  route { to :index }
+  route.to :index
   def index
     page # => #<Wayfarer::Page ...>
+    # Fill in forms, click buttons, etc.
     # Replaces the current Page object with a newer one,
     # taking into account the DOM as currently rendered by the browser.
     # Effectful only when automating browsers, no-op when using plain
@@ -50,3 +100,21 @@ class DummyJob < Wayfarer::Worker
   end
 end
 ```
+!!! attention "Stateless user agents ignore `#!ruby :live`"
+    The `#!ruby :live` option is ignored by stateless user agents, such as the
+    default `#!ruby :http` user agent. Instead, stateless user agents always
+    return the same page object.
+## Accessing page metadata with MetaInspector
+You have access to a [MetaInspector](https://github.com/jaimeiniesta/metainspector)
+document for accessing metadata of HTML pages. For example, to stage all links
+internal to the current hostname:
+```ruby
+def index
+  stage page.meta.links.internal
+end
+```

data/docs/guides/redis.md ADDED Viewed

@@ -0,0 +1,10 @@
+# Redis
+Wayfarer uses Redis to keep track of:
+* URLs that were already processed within a batch
+* the number of jobs left in a batch
+## Garbage collection
+Wayfarer cleans up batch-related data

data/docs/guides/routing.md ADDED Viewed

@@ -0,0 +1,156 @@
+# Routing
+Wayfarer equips jobs with a declarative routing DSL that maps URLs to actions.
+Actions are instance methods denoted by symbols, or [handlers](/guides/handlers).
+[Pages](/guides/pages) are only retrieved from URLs which map to an action.
+!!! info "Routed URLs are normalized"
+    By default, Wayfarer [applies some transformations to each URL](../tasks/#url-normalization) to bring it
+    into a canonical form. Routing happens based on this canonical form.
+    You can always access a task's raw string as it was enqueued with `task.batch`.
+A job's route declarations equate to a predicate tree.
+When a URL is routed, the predicate tree is searched depth-first. If a
+matching leaf predicate is found, the found path's action is dispatched.
+You can extract data from URL path segments and query parameters and
+access it through `params` in jobs or handlers.
+The following routes:
+```ruby
+route.host "example.com", scheme: :https do
+  path "contact", to: :contact
+  path "users/:id" do
+    to [UserHandler, :show]
+    path "gallery", to: [UserHandler, :photos]
+  end
+end
+```
+Equate to the following predicate tree:
+```mermaid
+flowchart LR
+  Root-->Host["Host <code>example.com</code>"]
+  Host-->Scheme["Scheme <code>:https</code>"]
+  %% first-level paths
+  Scheme-->PathContact["Path <code>contact</code>"]
+  Scheme-->PathUsersId["Path <code>users/:id</code>"]
+  %% their targets
+  PathContact-->TargetRouteContact["Target <code>:contact</code>"]
+  PathUsersId-->TargetRouteUserHandler["Target <code>[UserHandler, :show]</code>"]
+  %% nested path under /users/:id
+  PathUsersId-->PathGallery["Path <code>'gallery'</code>"]
+  PathGallery-->TargetRouteUserHandlerPhotos["Target <code>[UserHandler, :photos]</code>"]
+```
+Traversing the tree depth-first for `https://example.com/users/42` stops at the
+route with the action `[UserHandler, :show]`:
+```mermaid
+flowchart LR
+  Root:::matching-->Host["Host <code>example.com</code>"]:::matching
+  Host:::matching-->Scheme["Scheme <code>:https</code>"]:::matching
+  %% sibling paths from the scheme node
+  Scheme:::matching-->PathContact["Path <code>/contact</code>"]:::mismatching
+  Scheme:::matching-->PathUsersId["Path <code>/users/:id</code>"]:::matching
+  %% successful match for /users/:id
+  PathUsersId:::matching-->TargetRouteUserHandler["Target <code>[UserHandler, :show]</code>"]:::matching
+  %% gallery branch is never visited for /users/42
+  PathContact-->TargetRouteContact["Target <code>:contact</code>"]:::unvisited
+  PathUsersId:::matching-->PathGallery["Path <code>/gallery</code>"]:::unvisited
+  PathGallery:::unvisited-->TargetRouteUserHandlerPhotos["Target <code>[UserHandler, :photos]</code>"]:::unvisited
+  classDef matching     fill:#7CB342,stroke:#7CB342,color:#fff
+  classDef mismatching  fill:#FFCDD2,stroke:#F44336,color:#B71C1C
+  classDef unvisited    fill:#BDBDBD,stroke:#BDBDBD,color:#616161
+```
+??? note "You can also visualise a job's routing tree with with the [`route` CLI subcommand](/guides/cli)"
+    ```sh
+    wayfarer route DummyJob -r dummy_job.rb http://localhost:9000/users/42/gallery
+    ```
+    ```yaml
+    ---
+    routed: true
+    params:
+      id: '42'
+    action:
+      handler: Class
+      action: :photos
+    root_route:
+      match: true
+      params: {}
+      children:
+      - route:
+          host:
+            name: example.com
+          match: true
+          params: {}
+          children:
+          - route:
+              scheme:
+                scheme: :https
+              match: true
+              params: {}
+              children:
+              - route:
+                  path:
+                    pattern: "/contact"
+                  match: false
+                  params: {}
+                  children:
+                  - target_route:
+                      action:
+                      children: []
+              - route:
+                  path:
+                    pattern: "/users/:id"
+                  match: true
+                  params:
+                    id: '42'
+                  children:
+                  - target_route:
+                      action:
+                        handler: Class
+                        action: :show
+                      children: []
+                  - route:
+                      path:
+                        pattern: "/gallery"
+                      match: true
+                      params:
+                        id: '42'
+                      children:
+                      - target_route:
+                          action:
+                            handler: Class
+                            action: :photos
+                          children: []
+    ```
+As you can see, `Target` nodes always match. This means that we could have also defined
+our routes as:
+```ruby
+route.host "example.com", scheme: :https do
+  to :contact do
+    path "/contact"
+  end
+  to [UserHandler, :show] do
+    path "/users/:id"
+  end
+end
+```

data/docs/guides/tasks.md CHANGED Viewed

@@ -1,14 +1,58 @@
 # Tasks
-Tasks are the immutable units of work processed by [jobs](/guides/jobs). A task
-consists of:
+Tasks are the immutable units of work read from a message queue and processed by
+[jobs](/guides/jobs). A task consists of two strings:
-1. The __URL__ to process
-    * Within a batch, every URL gets processed at most once.
+* The __URL__ to process
+* The __batch__ the task belongs to
-2. The __batch__ the task belongs to
-    * Like URLs, batches are strings.
+A job processing a task commonly appends more tasks to the queue in turn.
-Tasks get appended to the end of a message queue, and consumed from the
-beginning. Because jobs can enqueue other tasks, jobs are both consumers
-and producers of tasks.
+!!! info "Task URLs are not normalized"
+    The URL returned by `task.url` is not normalized but verbatim
+    as it was staged or enqueued.
+## Task deduplication
+Wayfarer ensures that no URL gets processed twice within a batch. It achieves
+this by maintaining a [Redis hash](https://redis.io/docs/data-types/hashes)
+keyed by normalized URLs.
+Wayfarer computes a canonical URL representation that it uses for cache lookups.
+### URL normalization
+Wayfarer parses URLs with [Addressable](https://github.com/sporkmonger/addressable)
+and applies further normalizations. By default, all normalizations are applied
+and can be individually disabled.
+URL normalization is used only for deduplication, and does not affect the immutable
+`task.url`, which always returns the verbatim URL as enqueued.
+This allows you to follow the URLs exactly as parsed from response bodies.
+You can configure the global normalization behaviour by setting the following
+values on `Wayfarer.config.normalization` do which all default to `true`:
+ * `remove_www`: Remove `www.` prefix from hostnames?
+ * `remove_trailing_slash`: Remove a trailing path slash?
+ * `remove_fragment`: Remove the URL fragment?
+ * `order_query_parameters:` Order query parameters alphabetically?
+ * `remove_tracking_parameters`: Remove tracking parameters from the URL?
+When a job gets deduplicated, it succeeds and causes no retries.
+### Setting a custom key function
+You can customize how deduplication keys are computed. As a derived example,
+to process only one job per hostname:
+```ruby
+Wayfarer.config[:deduplication][:key] = ->(task) { task[:uri].hostname }
+```
+## Invalid URLs
+Tasks with invalid URLs are discarded (for example`ht%0atp://localhost/` which has a
+newline in its protocol), since there is no corrective action possible.
+No exception is raised, and the job is considered successfully processed without retries.

data/docs/guides/tutorial.md ADDED Viewed

@@ -0,0 +1,66 @@
+# Tutorial
+Wayfarer is a web crawling framework written in Ruby.
+It works with plain HTTP and by automating web browsers interchangeably
+and is deployed with Redis and a message queue.
+During development it can execute fully in memory, without Redis.
+## Getting started
+In an empty directory, generate a new `Gemfile` and install Wayfarer:
+```sh
+bundle init
+bundle add activejob wayfarer
+bundle install
+```
+## Jobs, tasks and batches
+Wayfarer builds on Active Job, the message queue abstraction of Rails.
+You can use Wayfarer without Rails of course, as we do here.
+A message queue supports two operations: appending messages to the end and consuming
+messages from the front. This is how Wayfarer processes tasks, a string pair
+of URL and batch. Wayfarer enforces that URLs are not processed more than
+once within their batch (excluding retries).
+When a task is consumed, it is processed by a job, a Ruby class.
+Let's give ourselves a `dummy_job.rb` that routes all URLs to its
+`index` instance method, where we print the current `task`:
+```ruby title="dummy_job.rb"
+require "activejob"
+require "wayfarer"
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+  route.to :index
+  def index
+    puts task
+  end
+end
+```
+We can perform our job from the command line with the `wayfarer perform`
+subcommand. In between ActiveJob's log output, we see that Wayfarer
+has generated a UUID for the batch since we did not pass it:
+```sh
+bundle exec wayfarer perform -r dummy_job.rb DummyJob https://example.com
+```
+```hl_lines="2"
+[ActiveJob] [DummyJob] [68853491-...] Performing DummyJob (Job ID: 68853491-...) from Async(default) with arguments: #<Wayfarer::Task url="https://example.com", batch="63d14035-...">
+#<Wayfarer::Task url="https://example.com", batch="63d14035-...">
+[ActiveJob] [DummyJob] [68853491-...] Performed DummyJob (Job ID: 68853491-) from Async(default) in 507.65ms
+```
+If you don't provide a batch, Wayfarer uses a generated UUID instead.
+We could have also used `DummyJob.crawl

data/docs/guides/user_agents.md ADDED Viewed

@@ -0,0 +1,115 @@
+# User agents
+User agents are used by [jobs](../jobs) to retrieve the contents behind a URL into a
+[page](../pages), for example a remotely controlled Firefox process or a Ruby HTTP client.
+User agents are kept in a connection pool and all user agents in the pool
+share the same type and configuration. You can add your own custom user agents by implementing
+the [user agent API](custom_user_agents.md).
+Wayfarer comes with the following built-in user agents:
+* [`:http`](http.md) (default)
+* [`:ferrum`](ferrum.md) to automate Google Chrome
+* [`:selenium`](selenium.md) to automate a variety of browsers
+* [`:capybara`](capybara.md) to use Capybara sessions
+Configure the user agent with the global configuration option:
+```ruby
+Wayfarer.config[:network][:agent] = :ferrum # or :selenium, :capybara, ...
+```
+You can access the user agent that was checked out from the pool with
+`#user_agent` in action methods:
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+  route.to :index
+  def index
+    user_agent # => #<Ferrum::Browser ...>
+  end
+end
+```
+You can also implement [custom user agents](custom_user_agents.md) to support
+your own HTTP client or browser automation service/protocol.
+### Ad-hoc HTTP requests
+Regardless the configured user agent, you can always make ad-hoc HTTP GET requests
+that return pages with `#fetch(url)`:
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+  route.to :index
+  def index
+    page = fetch("https://example.com") # => #<Wayfarer::Page ...>
+  end
+end
+```
+!!! info "`#fetch` respects `Wayfarer.config.network.http_headers` for all provided user agents."
+## HTTP request headers
+You can set HTTP request headers for all built-in user agents:
+```ruby
+Wayfarer.config[:network][:http_headers] = { "User-Agent" => "MyCrawler" }
+```
+!!! attention "Selenium does not support configuring HTTP request headers."
+## Connection pooling
+Since user agents are expensive to create, especially in the case of browser
+processes,  Wayfarer keeps user agents within a connection pool. When a job
+performs and needs to retrieve the [page](../pages) for its task URL, an agent
+is checked out from the pool, and checked back in when the routed action method
+returns.
+The pool size is constant and it should equal the number of threads the
+underlying message queue operates with. For example, if you use Sidekiq,
+you should set the pool size to the number of Sidekiq threads:
+```ruby
+Wayfarer.config[:network][:pool][:size] = Sidekiq.options[:concurrency]
+```
+!!! attention "The connection pool size is 1 by default"
+    Since there is no reliable way to detect the number of threads that
+    the underlying message queue operates with, Wayfarer defaults to a pool
+    size of 1, which creates a bottleneck in a concurrent environment.
+!!! attention "Browser sessions are shared across jobs"
+    The same browser session is used across jobs. This means that the browser
+    is not closed between jobs, and that the browser's state carries over from
+    job to job. You may account for this by resetting the browser's state
+    according to your needs, for which you can use [callbacks](../callbacks).
+### `UserAgentTimeoutError`: avoiding pool contention
+If you encounter `UserAgentTimeoutError` exceptions, a job has waited for a
+user agent to become available for too long. By default, this timeout is 10
+seconds. This is a sign that the pool size is too small for the message queue's
+concurrency.
+```
+#<Wayfarer::UserAgentTimeoutError: Waited 10 sec, 0/1 available>
+```
+You can configure the timeout, although you will likely want to increase the
+pool size instead:
+```ruby
+Wayfarer.config[:network][:pool][:timeout] = 10 # seconds
+```

data/docs/index.md CHANGED Viewed

@@ -1,56 +1,33 @@
 ---
 hide:
   - navigation
+  - toc
 ---
 # Wayfarer
-![CI status](https://github.com/actions/starter-workflows/workflows/CI/badge.svg)
-[![RubyGem](https://badge.fury.io/rb/wayfarer.svg)](https://rubygems.org/gems/wayfarer)
+## Ruby web crawling framework built on [ActiveJob]() and [Redis]()
-## Versatile web crawling with Ruby
+<small>
+  [Read the tutorial](/guides/tutorial){ .md-button .md-button--primary }
+</small>
-* Web scraping
-* Data extraction
-* Browser automation
+=== "Command line"
-!!! attention "Unstable software"
+    ```sh
+    gem install wayfarer
+    ```
-    Wayfarer is under development and releases should be considered unstable.
+=== "Gemfile"
-    Wayfarer complies to
-    [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html) in
-    which v0.x means that there could be backward-incompatible changes for every
-    release:
+    ```ruby
+    gem "wayfarer"
+    ```
-    >Major version zero (0.y.z) is for initial development. Anything MAY change
-    at any time. The public API SHOULD NOT be considered stable.
-### Installation
-Install the RubyGem:
-```
-gem install wayfarer
-```
-Or add it to Bundler's Gemfile:
-```ruby
-gem "wayfarer"
-```
-### Features
-* Breadth-first, acyclic, multi-threaded graph traversal
-* Executes atop a variety of message queues thanks to [ActiveJob](https://edgeguides.rubyonrails.org/active_job_basics.html)
-* Browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
+* Breadth-first, acyclic page traversal
+* Plain HTTP and browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
 (<abbr title="Chrome DevTools Protocol">CDP</abbr>),
-[Selenium](https://www.selenium.dev) or plain HTTP via `net/http`
+[Selenium](https://www.selenium.dev) and custom user agents
 * Declarative routing DSL
 * URI normalization and deduplication
-* XML, HTML, JSON parsing
-* HTTP redirect handling
-* Storage-agnostic
-* Small footprint: <500 LoC
-* Open Source (MIT)
+* HTML, XML, JSON and custom Content-Type body parsing