wayfarer 0.4.6 → 0.4.8

data/.github/workflows/ci.yaml

@@ -1,32 +0,0 @@
-name: ci
-
-on:
-  push:
-    branches:
-      - '*'
-env:
-  CI: true
-
-jobs:
-  ci:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - name: Start services
-        run: docker-compose up -d
-
-      - name: Run isolated tests
-        run: docker-compose run --rm --name test --service-ports wayfarer bundle exec rake test:isolated
-
-      - name: Run Ferrum tests
-        run: docker-compose run --rm --name test --service-ports wayfarer bundle exec rake test:ferrum
-
-      - name: Run Selenium tests
-        run: docker-compose run --rm --name test --service-ports wayfarer bundle exec rake test:selenium
-
-      - name: Run CLI tests
-        run: docker-compose run --rm --name test --service-ports wayfarer bundle exec rake test:cli
-
-      - name: Run RuboCop
-        run: docker-compose run --rm --name test --service-ports wayfarer bundle exec rake rubocop

data/.rbenv-gemsets

@@ -1 +0,0 @@
-wayfarer

data/.ruby-version

@@ -1 +0,0 @@
-2.7.4

data/RELEASING.md

@@ -1,17 +0,0 @@
-# Release Procedure
-
-1. Ensure `Wayfarer::VERSION` was bumped appropriately.
-2. Ensure the version in wayfarer.gemspec matches.
-3. Open a release Pull Request develop -> master branch
-4. Merge the Pull Request
-5. Publish RubyGem and git tag as follows:
-
-```
-git checkout master
-git pull origin master --rebase
-bundle exec rake build
-gem push build/wayfarer-*.gem
-bundle exec rake clean
-git tag <VERSION>
-git push origin <VERSION>
-```

data/docs/cookbook/user_agent.md

@@ -1,7 +0,0 @@
-# User Agent
-
-See: [Guides: Networking: HTTP request headers](/guides/networking#http-request-headers)
-
-```ruby
-Wayfarer.config.network.http_headers = { "User-Agent" => "MyCrawler ..." }
-```

data/docs/guides/error_handling.md

@@ -1,53 +0,0 @@
-# Error handling
-
-## Wayfarer never swallows exceptions
-
-* Wayfarer never swallows exceptions.
-* Jobs with unhandled exceptions are not retried.
-
-## Retrying or discarding failing jobs
-
-Wayfarer relies on [Active Job's two error handling facilities](https://guides.rubyonrails.org/active_job_basics.html#exceptions).
-
-* `retry_on` to retry jobs a number of times on certain errors:
-
-    ```ruby
-    class DummyJob < Wayfarer::Base
-      retry_on MyError, attempts: 3 do |job, error|
-        # This block runs once all 3 attempts have failed
-        # (1 initial attempt + 2 retries)
-
-        raise error
-      end
-    end
-    ```
-
-* `discard_on` to throw away jobs on certain errors:
-
-    ```ruby
-    class DummyJob < Wayfarer::Base
-      discard_on MyError do |job, error|
-        # This block runs once and buries the job
-
-        raise error
-      end
-    end
-    ```
-
-!!! attention "Always re-raise errors"
-
-    You should always re-raise errors from `retry_on` and `discard_on` blocks,
-    otherwise jobs will not get retried!
-
-## Renewing agents on certain errors
-
-```ruby
-Wayfarer.config.network.renew_on = [MyError]
-```
-
-For example, if you use the Capybara
-[Cuprite](https://github.com/rubycdp/cuprite) driver:
-
-```ruby
-Wayfarer.config.network.renew_on = [Ferrum::DeadBrowserError]
-```

data/docs/guides/networking.md

@@ -1,94 +0,0 @@
-# Networking
-
-Wayfarer navigates the web in two ways:
-
-1. Via plain HTTP requests
-2. By automating browsers
-
-Both options are mutually exclusive per Ruby process.
-
-## User agents
-
-A user agent is an entity that knows how to retrieve the contents behind a URL.
-
-The user agent can be configured via the global configuration:
-
-```ruby
-Wayfarer.config.network.agent = :http # or :ferrum, :selenium
-```
-
-## Connection pooling
-
-Wayfarer keeps user agents within a connection pool. When a job executes
-and needs to retrieve the contents behind a URL, an agent is checked out from
-the pool.
-
-The pool has a constant size and it should equal the number of threads the
-underlying message queue operates with. The size can be configured via the
-global configuration:
-
-```ruby
-Wayfarer.config.network.pool_size = 8
-```
-
-### Timeouts
-
-user agents may stay checked out from the pool by jobs for a limited time
-only. Once this time limit is exceeded, a `ConnectionPool::TimeoutError`
-exception is raised. This places a hard time limit on every job.
-
-The timeout can be configured via the global configuration:
-
-```ruby
-Wayfarer.config.network.pool_timeout = 20 # seconds
-```
-
-Because jobs with unhandled exceptions fail, explicit error handling is required
-if retries are desired:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  retry_on ConnectionPool::TimeoutError, attempts: 3
-end
-```
-
-## Agent-specific client timeouts
-
-The time in seconds it may take to communicate with remote browser processes can
-be configured globally per agent:
-
-```ruby
-Wayfarer.config.ferrum.options = { timeout: 5 }
-Wayfarer.config.selenium.client_timeout = 60
-```
-
-### Shared state
-
-As user agents get checked in and out continously between jobs, their state
-carries over from job to job, too.
-
-For browser automation, this means:
-
-* A job finds the browser at the last URL the previous job has left off.
-* The browser's cookies might have been set, or other client-side state might
-  exist that significantly affects a page's behaviour.
-
-## HTTP redirect handling
-
-Browsers follow redirects transparently when they are navigated to a URL.
-
-When using plain HTTP, redirect URLs are enqueued transparently within the same
-batch. URLs that result in 3xx responses will not be retrieved again within
-their batch.
-
-## HTTP request headers
-
-Request headers can be configured via the global configuration:
-
-```ruby
-Wayfarer.config.network.http_headers = { "Field" => "Value" }
-```
-
-!!! attention "Partial support"
-
-    Selenium does not support configuring HTTP request headers.

data/docs/guides/performance.md

@@ -1,130 +0,0 @@
-# Performance
-
-How to write performant crawlers with Wayfarer.
-
-## Use a sufficiently sized user agent pool
-
-Automated browser processes or HTTP clients are kept in a [connection pool]() of
-static size. This avoids having to re-establish browser processes and enables
-their reuse.
-
-If the size of the pool is too small, the pool is a
-bottleneck. For example, if your message queue adapter uses 8 threads, but the
-pool only contains 1 user agent, the remaining 7 threads block until the agent
-is checked back in to the pool for use by one of the blocked threads.
-
-There is no reliable way to detect the number of threads of the underlying
-message queue adapter. The pool size should equal the number of threads;
-
-```ruby
-Wayfarer.config.network.pool_size = 8 # defaults to 1
-```
-
-### Job shedding
-
-There is a maximum number of seconds that jobs wait when checking out a user
-agent from the pool. Once this time is exceeded,
-a `Wayfarer::UserAgentTimeoutError` is raised. By default, the timeout is 10
-seconds.
-
-This hints there are more threads in use than user agents in the pool.
-
-## Stage less URLs
-
-Staging less URLs saves space and time:
-
-* Less tasks written to the message queue
-* Less time spent consuming tasks
-* Less time spent filtering URLs with Redis
-
-Wayfarer maintains a set of processed URLs for a batch in Redis. Every staged
-URL is checked for inclusion in this set before it gets appended as a task to
-the message queue.
-
-A common pattern is to stage all links of a page, and rely on routing to fetch
-only the relevant ones:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  route { to: index, host: "example.com" }
-
-  def index
-    stage page.meta.links.all
-  end
-end
-```
-
-Pages commonly contain a large number of URLs.
-
-Every staged URL is:
-
-1. Normalized to a canonical form, for example by sorting query parameters
-   alphabetically.
-2. Checked for inclusion in the batch Redis set or discarded.
-3. Written to the message queue.
-4. Consumed from the queue and matched against the router.
-5. Fetched, if a route matches.
-
-Narrowing down the links in the document to follow speeds up the process.
-For example using Nokogiri, interesting links can be identified with a CSS
-selector:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  route { to: index, host: "example.com" }
-
-  def index
-    stage interesting_links
-  end
-
-  private
-
-  def interesting_links
-    page.doc.css("a.interesting").map { |elem| elem["href"] }
-  end
-end
-```
-
-Because the router only accepts the single hostname `example.com`, the job can
-also ensure it stages only internal URLs by intersecting them with the
-interesting ones:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  route { to: index, host: "example.com" }
-
-  def index
-    stage interesting_internal_links
-  end
-
-  private
-
-  def interesting_internal_links
-    page.meta.links.internal & interesting_links
-  end
-
-  def interesting_links
-    page.doc.css("a.interesting").map { |elem| elem["href"] }
-  end
-end
-```
-
-
-## Use Redis >= 6.2.0
-
-Redis 6.2.0 introduced the
-[`SMISMEMBER`](https://redis.io/commands/smismember) command which enables
-Wayfarer to check whether multiple URLs have been processed in a batch with a
-single command. With earlier versions, one command per URL is required.
-
-Wayfarer detects the Redis server version and uses `SMISMEMBER` without user
-configuration when supported.
-
-## Use Oj for JSON parsing
-
-Wayfarer uses [Oj](https://github.com/ohler55/oj) for JSON parsing if the gem
-has been required at runtime:
-
-```ruby
-require "oj"
-```

data/docs/guides/reliability.md

@@ -1,41 +0,0 @@
-# Reliablity
-
-## Durability
-
-Wayfarer executes atop reliable messages queues such as Sidekiq, Resque,
-RabbitMQ, etc. Its configuration is independent of the underlying queue
-infrastructure it reads from and writes to.
-
-## Self-healing user agents
-
-Wayfarer handles the scenario where a remote browser process has crashed and
-must be replaced by a fresh browser process.
-
-This can be tested locally by automating a browser with headless mode turned
-off, and then closing the opened browser window: The current job fails, but the
-next job has access to a newly established browser session again.
-
-For example Ferrum might raise `Ferrum::DeadBrowserError`. Wayfarer's
-user agents are self-healing and react to these kinds of errors internally. When
-a browser window is closed, the Ferrum user agent attempts to establish a new
-browser process as a replacement, for the next job to use.
-
-[Wayfarer never swallows exceptions](/guides/error_handling). This means
-that even though the user agent might heal itself, jobs still need to explicitly
-retry browser errors:
-
-```ruby
-class Foobar < Wayfarer::Base
-  route { to: :index }
-
-  retry_on Ferrum::DeadBrowserError, attempts: 3, wait: :exponentially_longer
-
-  # ...
-end
-```
-
-This leads to log entries like:
-
-```
-Retrying DummyJob in 3 seconds, due to a Ferrum::DeadBrowserError.
-```

data/docs/guides/routing/steering.md

@@ -1,30 +0,0 @@
-# Steering
-
-A job's router can receive arguments computed dynamically by `::steer`.
-Steering enables [batch routing](/cookbook/batch_routing).
-
-For example, the following router has hostname and path hard-coded:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  route do
-    host "example.com", path: "/contact", to: :index
-  end
-end
-```
-
-Instead, hostname and path could be provided by `::steer`, too:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  route do |hostname, path|
-    host hostname, path: path, to: :index
-  end
-
-  steer do |_task|
-    ["example.com", "/contact"]
-  end
-end
-```
-
-Note that `steer` yields the current [task](/guides/tasks).

data/docs/reference/api/base.md

@@ -1,48 +0,0 @@
----
-title: Wayfarer::Base
----
-
-# `Wayfarer::Base`
-
-Wayfarer's complete job API.
-
----
-
-### `::route`
-:   Draw routes to instance methods.
-
----
-
-### `::steer { (Wayfarer::Task) -> [any] }`
-:   Provide router arguments.
-
----
-
-### `#task -> Wayfarer::Task`
-:   The currently processing task.
-
----
-
-### `#params -> Hash`
-:   URL parameters collected from the matching route.
-
----
-
-### `#stage(String | [String]) -> void`
-:   Add URLs to a processing set. URLs already processed within the
-    current batch get discarded are not enqueued. Every staged URL gets
-    normalized.
-
----
-
-### `#browser -> Object`
-:   The user agent that retrieved the current page.
-
----
-
-### `#page(live: true | false) -> Page`
-:   The page representing the response retrieved from the currently
-    processing URL.
-
-    With `live: true` called, a fresh `Page` is returned that reflects the
-    current browser DOM. Calls to `#page` return the most recent page.

data/docs/reference/cli.md

@@ -1,61 +0,0 @@
-# Command Line Interface
-
-## Usage
-
-```
-wayfarer [OPTIONS] [generate|job|route|version]
-```
-
-All [environment variables](../environment_variables) are respected.
-
-## `wayfarer generate`
-
-### `wayfarer generate project NAME`
-
-:   Generates a new project directory `NAME`.
-
-## `wayfarer job`
-
-### `wayfarer job perform JOB URL`
-
-:   Performs `JOB` with `URL`. The job does not reach any Active Job backend.
-    Staged jobs will not be processed.
-
-    ##### Options
-
-    * `--mock-redis`: Use an in-memory implementation of Redis instead of
-      talking to an actual server.
-    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
-
-### `wayfarer job enqueue JOB URL`
-
-:   Enqueues `JOB` with `URL` to the configured Active Job backend.
-
-    ##### Options
-
-    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
-
-### `wayfarer job execute JOB URL`
-
-:   Execute `JOB` with `URL` by using the
-    [Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html).
-    The job does not reach any other Active Job backend. Blocks until the batch
-    completes.
-
-    ##### Options
-
-    * `--mock-redis`: Use an in-memory implementation of Redis instead of
-      talking to an actual server.
-    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
-    * `--min-threads`: Minimum number of threads to use. Default: 1
-    * `--max-threads`: Maximum number of threads to use. Default: 1
-
-## `wayfarer route`
-
-### `wayfarer route result JOB URL`
-
-:   Prints the result of invoking `JOB`'s router with `URL`.
-
-### `wayfarer route tree JOB URL`
-
-:   Visualises the routing tree result of invoking `JOB`'s router with `URL`.

data/docs/reference/configuration_keys.md

@@ -1,43 +0,0 @@
----
-hide:
-  - toc
----
-
-# Configuration Keys
-
-## `Wayfarer.config.network`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `network.agent`        | `WAYFARER_NETWORK_AGENT`             | The user agent to use.                      | `:http`                          | `:http`, `:ferrum`, `:selenium`     |
-| `network.pool_size`    | `WAYFARER_NETWORK_POOL_SIZE`         | How many user agents to spawn.              | 1                                | Integers                            |
-| `network.pool_timeout` | `WAYFARER_NETWORK_POOL_TIMEOUT`      | How long jobs may use an agent in seconds.  | 10                               | Integers                            |
-| `network.http_headers` | `WAYFARER_NETWORK_HTTP_HEADERS`      | HTTP headers to append to requests.         | `{}`                             | Hashes                              |
-| `network.renew_on`     |                                      | Exception classes to renew agents on.       | `[]`                             | Classes                             |
-
-## `Wayfarer.config.ferrum`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `ferrum.options`       | `WAYFARER_FERRUM_OPTIONS`            | Ferrum options.                             | `{}`                             | Hashes                              |
-
-## `Wayfarer.config.selenium`
-
-| Runtime config key        | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ----------------------    | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `selenium.driver`         | `WAYFARER_SELENIUM_DRIVER`           | Selenium driver to use.                     | `:chrome`                        | Symbols                             |
-| `selenium.options`        | `WAYFARER_SELENIUM_OPTIONS`          | Selenium options.                           | `{}`                             | Hashes                              |
-| `selenium.client_timeout` | `WAYFARER_SELENIUM_CLIENT_TIMEOUT`   | Selenium client timeout in seconds.         | 60                               | Integers                            |
-
-## `Wayfarer.config.redis`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                                    | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | ------------------------------------------ | ----------------------------------- |
-| `redis.url`            | `WAYFARER_REDIS_URL`                 | Redis URL to connect to.                    | http://localhost:6379                      | Strings                             |
-| `redis.factory`        | n/a                                  | Redis factory lambda.                       | ` ->(redis) { ::Redis.new(url: redis.url)` | Lambdas                             |
-
-## `Wayfarer.config.capybara`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `capybara.driver`      | `WAYFARER_CAPYBARA_DRIVER`           | The Capybara driver to use.                 | n/a                              | Symbols                             |

data/docs/reference/environment_variables.md

@@ -1,83 +0,0 @@
-# Environment Variables
-
-## String formats
-
-Environment variable values can be parsed to Hash or Array at runtime
-with the following syntaxes:
-
-* Hash: Variable string `a:1,b:2,c:3` parses to `{a:1, b:2, c:3}` at runtime
-* Array: Variable string `a,b,c` parses to `[:a, :b, :c]` at runtime
-
-## Variables
-
-### `WAYFARER_AGENT`
-:   Either `ferrum`, `selenium` or `http`.
-
-    * Type: String
-    * Key: `config.agent`
-    * Default value: `:http`
-
-### `WAYFARER_POOL_SIZE`
-:   Number of user agents to maintain.
-
-    * Type: Integer
-    * Key: `config.pool_size`
-    * Default value: `1`
-
-### `WAYFARER_POOL_TIMEOUT`
-:   How long a user agent may remain checked out until the owning job
-    fails.
-
-    * Type: Integer
-    * Key: `config.agent_pool_timeout`
-    * Default value: `1`
-
----
-
-### `WAYFARER_FERRUM_OPTIONS`
-:   Key/value options passed to `Ferrum::Browser.new`.
-
-    * Type: Hash
-    * Key: `config.ferrum_options`
-    * Default value: `{}`
-
----
-
-### `WAYFARER_SELENIUM_DRIVER`
-:   Driver passed to `Selenium::WebDriver.for`.
-
-    * Type: Symbol
-    * Key: `config.selenium_driver`
-    * Default value: `:chrome`
-
----
-
-### `WAYFARER_SELENIUM_OPTIONS`
-:   Options passed to `Selenium::WebDriver.for`.
-
-    * Type: Hash
-    * Key: `config.selenium_options`
-    * Default value: `{}`
-    
----
-
-### `WAYFARER_SELENIUM_CLIENT_TIMEOUT`
-:   Selenium HTTP client timeout (seconds).
-
-    * Type: Integer
-    * Key: `config.selenium_client_timeout`
-    * Default value: `60`
-
----
-
-### `WAYFARER_HTTP_HEADERS`
-:   HTTP request headers used when retrieving pages.
-
-    * Type: Hash
-    * Key: `config.http_headers`
-    * Default value: `{}`
-    
-    !!! attention "Partial support"
-
-        Selenium does not support configuring HTTP request headers.
-