wayfarer 0.0.3 → 0.4.0

data/docs/guides/tasks.md

@@ -0,0 +1,38 @@
+# Tasks
+
+Tasks are the units of work processed by jobs. A task consists of:
+
+1. The URL to process
+2. The batch the task belongs to
+
+Like URLs, batches are strings. Within a batch, every URL gets processed at most
+once.
+
+Tasks get appended to the end of a message queue, and consumed gfrom their
+beginning by jobs.
+
+When jobs process tasks, they search their routing tree for a matching route.
+URLs that match no route are not retrieved, and their task considered
+successfully processed without further action.
+
+## Task Metadata
+
+At runtime, tasks take the shape of a `Wayfarer::Task` object. While only URL
+and batch are persisted to message queues, tasks carry an arbitrarily assignable
+`metadata` object:
+
+```ruby
+task          # => #<Task url="https://example.com" batch="547b761-d0ad-...">
+task.metadata # => #<OpenStruct>
+task.metadata.my_piece_of_information = "hello"
+```
+
+`task.metadata` is ephemeral and only accessible at runtime.
+
+Once a job consumes a task, the job instance becomes accessible on it:
+
+```
+task.job # => #<DummyJob ...>
+```
+
+

data/docs/index.md

@@ -1,82 +1,48 @@
 ---
-layout: default
-title: Versatile web crawling with (J)Ruby
+hide:
+  - navigation
 ---
 
-# Versatile web crawling with (J)Ruby
+# Wayfarer
 
-Wayfarer is the swiss army knife for web crawling.
+![CI status](https://github.com/actions/starter-workflows/workflows/CI/badge.svg)
+[![RubyGem](https://badge.fury.io/rb/wayfarer.svg)](https://rubygems.org/gems/wayfarer)
 
-MRI:
-```
-% [sudo] gem install wayfarer
-```
-
-JRuby:
-```
-% [sudo] gem install wayfarer-jruby
-```
-
-If you …
-
-* __Need to crawl page graphs breadth-first__
-* __Need to extract whatever data__
-* Do it multi-threaded
-* Integrate with Rails seamlessly
-* Want to automate a web browser
-* Need to execute arbitrary JavaScript
-* Need URI normalization
-* Need to take screenshots
-* Want to use a job queue and make work happen later
-* Want in-memory and Redis-backed frontiers, tries and Bloom filters
-
-… then you might like Wayfarer!
+## Versatile web crawling with Ruby
 
-## What it looks like
+* Web scraping
+* Data extraction
+* Browser automation
 
-Say you want to …
+!!! attention "Experimental software"
 
-* Automate Google Chrome
-* Start off with a random Wikipedia article
-* Follow all links until you find a page with the word "Foobar"
-* Take a screenshot of the page containing "Foobar"
-* Extract keywords from every page you encounter
-* Use 4 threads and Chrome processes to do so
+    Wayfarer is under development and releases should be considered unstable.
 
-This amounts to 16 lines of code with Wayfarer:
+### Installation
 
-{% highlight ruby %}
-require "wayfarer"
+Install the RubyGem:
 
-class FindFoobarOnWikipedia < Wayfarer::Job
-  config.http_adapter = :selenium
-  config.selenium_argv = [:chrome]
-  config.connection_count = 4
-
-  let(:keywords) { [] }
-
-  route.host "en.wikipedia.org", to: :article
-
-  def article
-    if page.body =~ /Foobar/
-      driver.save_screenshot("/tmp/foobar.png")
-      return halt
-    end
-
-    keywords << page.keywords
-    stage page.links
-  end
-end
-
-FindFoobarOnWikipedia.perform_now("https://en.wikipedia.org/wiki/Special:Random")
-{% endhighlight %}
+```
+gem install wayfarer
+```
 
-Wayfarer integrates with [ActiveJob]() and supports your favorite job queue out of the box. Your job is ready to be enqueued:
-{% highlight ruby %}
-FindFoobarOnWikipedia.perform_later("https://en.wikipedia.org/wiki/Special:Random")
-{% endhighlight %}
+Or add it to Bundler's Gemfile:
 
-### Where to go from here
+```ruby
+gem "wayfarer"
+```
 
-* [The tutorial]() shows how to collect all open issues from a GitHub repository
-* Read the [API documentation]()
+### Features
+
+* Breadth-first, acyclic, multi-threaded graph traversal
+* Executes atop a variety of message queues thanks to [ActiveJob](https://edgeguides.rubyonrails.org/active_job_basics.html)
+* Browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
+(<abbr title="Chrome DevTools Protocol">CDP</abbr>),
+[Selenium](https://www.selenium.dev) or plain HTTP via `net/http`
+* Declarative routing DSL
+* URI normalization and deduplication
+* XML, HTML, JSON parsing
+* HTTP redirect handling
+* Storage-agnostic
+* Small footprint: <500 LoC
+* Open Source (MIT)

data/docs/reference/api/base.md

@@ -0,0 +1,162 @@
+---
+title: Base
+---
+
+# `Wayfarer::Base`
+
+Base functionality every job is equipped with:
+
+* Router for connecting URLs with instance methods and collecting
+  data.
+* Access to a parsed document representation.
+* Access to the browser or HTTP connection that retrieved the document.
+* Ability to stage URLs for future processing.
+* Ability to inject middleware before or after the worker.
+
+---
+
+### `::route -> Wayfarer::Routing::Route`
+:   The class route that ties URLs to action methods via URL rules.
+
+    ##### Example
+
+    !!! example "Defining routes"
+
+        ```ruby
+        class DummyJob < Wayfarer::Base
+          route.to :index
+
+          def index
+          end
+        end
+        ```
+
+---
+
+### `#task -> Wayfarer::Task`
+:   The currently processing task.
+
+    !!! example "Inspecting the current task"
+
+        ```ruby
+        class DummyJob < Wayfarer::Base
+          route.to :index
+
+          def index
+            task       # => #<Wayfarer::Task ...>
+            task.url   # => "https://example.com"
+            task.batch # => "2287ae65-359e-4dc0-..."
+          end
+        end
+        ```
+
+---
+
+### `#params -> Hash`
+:   URL parameters collected from the matching route.
+
+    !!! example "Accessing URL parameters"
+
+        ```ruby
+        class DummyJob < Wayfarer::Base
+          route.path "/users/:user_id/images/:id", to: :index
+
+          def index
+            params # => { "user_id" => ..., "id" => ... }
+          end
+        end
+        ```
+
+---
+
+### `#stage(*urls) -> void`
+:   Add URLs to a processing set. URLs already processed within the
+    current batch get discarded are not enqueued. Every staged URL gets
+    normalized.
+
+    !!! example "Staging a URL"
+
+        ```ruby
+        class DummyJob < Wayfarer::Base
+          route.to :index
+
+          def index
+            stage "https://example.com"
+          end
+        end
+        ```
+
+    !!! example "Staging all URLs contained in the current page"
+
+        ```ruby
+        class DummyJob < Wayfarer::Base
+          route.to :index
+
+          def index
+            stage page.meta.links.all
+          end
+        end
+        ```
+
+---
+
+### `#browser -> Ferrum::Browser | Selenium::WebDriver | nil`
+:   The browser process used to retrieve the current response.
+    If the configured adapter is the default `:net_http`, `nil` is returned.
+
+    Guides:
+
+    * [Ferrum (Chrome DevTools Protocol)]()
+    * [Selenium]()
+
+    !!! example "Accessing a Google Chrome process"
+
+        ```ruby
+        Wayfarer.config.adapter = :ferrum
+
+        class DummyJob < Wayfarer::Base
+          route.to :index
+
+          def index
+            browser # => #<Ferrum::Browser ...>
+          end
+        end
+        ```
+
+    !!! example "Accessing a Selenium WebDriver"
+
+        ```ruby
+        Wayfarer.config.adapter = :selenium
+
+        class DummyJob < Wayfarer::Base
+          route.to :index
+
+          def index
+            browser # => #<Selenium::WebDriver ...>
+          end
+        end
+        ```
+
+---
+
+### `#page(live: false) -> Page`
+:   The page representing the response retrieved from the currently
+    processing URL.
+
+    With `page(live: true)` passed, the returned `Page` reflects the current
+    browser DOM. No-op when the `net/http` adapter is in use. Calls to
+    `page()` without the keyword return the most recent page.
+
+---
+
+### `#doc -> Nokogiri::HTML | Nokogiri::XML | Hash`
+:   The parsed HTTP response body depending on the Content-Type:
+    * When XML or HTML then a parsed Nokogiri document
+    * When JSON, a parsed Hash
+
+---
+
+### `#middleware -> [Middleware]`
+:   Template method that allows workers to inject middleware before or
+    after themselves.
+

data/docs/reference/api/route.md

@@ -0,0 +1,182 @@
+---
+title: Route
+---
+
+# `Wayfarer::Route`
+
+All public functionality documented here is exposed via
+`Wayfarer::Routing::DSL`.
+
+---
+
+### `#host(String | Regexp)`
+:   Match hosts against strings or regular expressions.
+
+    ##### Examples
+
+    !!! example "Exact hostname string matching"
+
+        ```ruby
+        route.host "example.com"
+        ```
+
+        - [x] https://{==example.com==}
+        - [x] https://{==example.com==}/foo/bar
+        - [ ] https://sub.example.com
+        - [ ] https://nasa.gov
+
+    !!! example "Hostname matching with regular expression"
+
+        ```ruby
+        route.host /example.com$/
+        ```
+
+        - [x] https://{==example.com==}
+        - [x] https://{==example.com==}/foo/bar
+        - [x] https://{==sub.example.com==}
+        - [ ] https://nasa.gov
+
+---
+
+### `#suffix(String)`
+:   Match URL paths against exact suffixes.
+
+    ##### Examples
+
+    !!! example "Match only URLs with paths ending with \*.png"
+
+        ```ruby
+        route.suffix ".png"
+        ```
+
+        - [x] https://example.com/image{==.png==}
+        - [ ] https://example.com/index.html
+        - [ ] https://example.com
+
+---
+
+### `#path(String)`
+:   Match URLs against their path.
+
+    ##### Examples
+
+    !!! example "Exact path string matching"
+
+        ```ruby
+        route.path "/users/123"
+        ```
+
+        - [x] https://example.com/{==users/123==}
+        - [x] https://example.com/{==users/123==}
+        - [ ] https://example.com/users
+        - [ ] https://example.com
+
+    !!! example "Root path matching"
+
+        ```ruby
+        route.path "/"
+        ```
+
+        - [x] https://example.com{==/==}
+        - [ ] https://example.com
+        - [ ] https://example.com/users
+
+    !!! example "Pattern matching"
+
+        ```ruby
+        route.path "/users/:id", to: :with_params
+        route.path "/contact", to: :without_params
+
+        def with_params
+          params # => { "id" => ... }
+        end
+
+        def without_params
+          params # => {}
+        end
+        ```
+
+        - [x] https://example.com{==/users/123==}
+        - [x] https://example.com{==/contact==}
+        - [ ] https://example.com
+        - [ ] https://example.com/users/123/photos
+        - [ ] https://example.com/contact/
+
+---
+
+### `#query(Hash)`
+:   Match URLs against query parameters.
+
+    ##### Examples
+
+    !!! example "Exact parameter string matching"
+
+        ```ruby
+        route.query arg: "foo"
+        ```
+
+        - [x] https://example.com?{==arg=foo==}
+        - [x] https://example.com?{==arg=foo==}&page=2
+        - [ ] https://example.com?arg=123
+        - [ ] https://example.com
+
+    !!! example "Parameter regular expression matching"
+
+        ```ruby
+        route.query arg: /foo/
+        ```
+
+        - [x] https://example.com?{==arg=foo==}
+        - [x] https://example.com?{==arg=foobar==}
+        - [x] https://example.com?{==arg=barfoo==}&page=2
+        - [ ] https://example.com?arg=123
+        - [ ] https://example.com
+
+    !!! example "Parameter integer matching"
+
+        ```ruby
+        route.query page: 5
+        ```
+
+        - [x] https://example.com?{==page=5==}
+        - [x] https://example.com?arg=foo&{==page=5==}
+        - [ ] https://example.com?page=3
+        - [ ] https://example.com
+
+    !!! example "Parameter range matching"
+
+        ```ruby
+        route.query page: 1..10
+        ```
+
+        - [x] https://example.com?{==page=1==}
+        - [x] https://example.com?arg=foo&{==page=10==}
+        - [ ] https://example.com?page=11
+        - [ ] https://example.com
+
+### `#scheme(Symbol)`
+:   Match URLs by their scheme.
+
+    ##### Examples
+
+    !!! example "Scheme matching"
+
+        ```ruby
+        route.scheme :https
+        ```
+
+        - [x] {==https==}://example.com
+        - [ ] http://example.com
+
+### `#uri(String)`
+:   Match URLs exactly.
+
+    ##### Examples
+
+    !!! example "URL matching"
+
+        ```ruby
+        route.uri "https://example.com/foo/bar?page=3"
+        ```
+
+        - [x] {==https://example.com/foo/bar?page=3==}

data/docs/reference/cli.md

@@ -0,0 +1,139 @@
+# Command Line Interface
+
+## Usage
+
+```
+wayfarer [OPTIONS] [generate|job|route|version]
+```
+
+All [environment variables](../environment_variables) are respected.
+
+## `wayfarer generate`
+
+### `wayfarer generate project NAME`
+
+:   Generates a new project directory `NAME`.
+
+    ##### Example
+
+    !!! example "Create a new project directory"
+
+        ```
+        wayfarer generate project foobar
+        ```
+
+## `wayfarer job`
+
+### `wayfarer job perform JOB URL`
+
+:   Performs `JOB` with `URL`. The job does not reach any Active Job backend.
+    Staged jobs will not be processed.
+
+    ##### Options
+
+    * `--mock-redis`: Use an in-memory implementation of Redis instead of
+      talking to an actual server.
+    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
+
+    ##### Examples
+
+    !!! example "Perform a job"
+
+        ```
+        wayfarer job perform DummyJob https://example.com
+        ```
+
+    !!! example "Specify a batch"
+
+        ```
+        wayfarer job perform --batch=my-batch DummyJob https://example.com
+        ```
+
+### `wayfarer job enqueue JOB URL`
+
+:   Enqueues `JOB` with `URL` to the configured Active Job backend.
+
+    ##### Options
+
+    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
+
+    ##### Examples
+
+    !!! example "Enqueue a job"
+
+        ```
+        wayfarer job enqueue DummyJob https://example.com
+        ```
+
+    !!! example "Specify a batch"
+
+        ```
+        wayfarer job enqueue --batch=my-batch DummyJob https://example.com
+        ```
+
+### `wayfarer job execute JOB URL`
+
+:   Execute `JOB` with `URL` by using the
+    [Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html).
+    The job does not reach any other Active Job backend. Blocks until the batch
+    completes.
+
+    ##### Options
+
+    * `--mock-redis`: Use an in-memory implementation of Redis instead of
+      talking to an actual server.
+    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
+    * `--min-threads`: Minimum number of threads to use. Default: 1
+    * `--max-threads`: Maximum number of threads to use. Default: 1
+
+    ##### Examples
+
+    !!! example "Enqueue a job"
+
+        ```
+        wayfarer job execute DummyJob https://example.com
+        ```
+
+    !!! example "Mock Redis"
+
+        ```
+        wayfarer job execute --mock-redis DummyJob https://example.com
+        ```
+
+    !!! example "Specify a batch"
+
+        ```
+        wayfarer job execute --batch=my-batch DummyJob https://example.com
+        ```
+
+    !!! example "Use up to 4 threads"
+
+        ```
+        wayfarer job execute --min-threads=1 --max-threads=4 DummyJob https://example.com
+        ```
+
+## `wayfarer route`
+
+### `wayfarer route result JOB URL`
+
+:   Prints the result of invoking `JOB`'s router with `URL`.
+
+    ##### Example
+
+    !!! example "Route a URL"
+
+        ```
+        wayfarer route result DummyJob https://example.com
+        ```
+
+### `wayfarer route tree JOB URL`
+
+:   Visualises the routing tree result of invoking `JOB`'s router with `URL`.
+
+    ##### Example
+
+    !!! example "Visualise the routing tree"
+
+        ```
+        wayfarer route tree DummyJob https://example.com
+        ```