ruby-link-checker 0.3.0 → 0.5.1

data/README.md

@@ -2,9 +2,8 @@ Fast Ruby Link Checker
 ======================
 
 [![Gem Version](http://img.shields.io/gem/v/ruby-link-checker.svg)](http://badge.fury.io/rb/ruby-link-checker)
-[![Build Status](https://github.com/dblock/ruby-link-checker/workflows/test/badge.svg?branch=main)](https://github.com/dblock/ruby-link-checker/actions)
-[![Code Climate](https://codeclimate.com/github/dblock/ruby-link-checker.svg)](https://codeclimate.com/github/dblock/ruby-link-checker)
-[![Test Coverage](https://api.codeclimate.com/v1/badges/164f1e23fc706b6efa63/test_coverage)](https://codeclimate.com/github/dblock/ruby-link-checker/test_coverage)
+[![Build Status](https://github.com/dblock/ruby-link-checker/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/dblock/ruby-link-checker/actions/workflows/test.yml)
+[![codecov](https://codecov.io/gh/dblock/ruby-link-checker/graph/badge.svg)](https://codecov.io/gh/dblock/ruby-link-checker)
 
 A fast Ruby link checker with support for multiple HTTP libraries. Does not parse documents, just checks links. Fast. Anecdotal benchmarking on a M1 mac and T1 Internet yields ~50 URLs per second with `LinkChecker::Typhoeus::Hydra`.
 
@@ -16,16 +15,23 @@ A fast Ruby link checker with support for multiple HTTP libraries. Does not pars
   - [Passing Options](#passing-options)
   - [Checkers](#checkers)
     - [LinkChecker::Typhoeus::Hydra](#linkcheckertyphoeushydra)
+    - [LinkChecker::Async::HTTP](#linkcheckerasynchttp)
     - [LinkChecker::Net::HTTP](#linkcheckernethttp)
   - [Options](#options)
     - [Retries](#retries)
+    - [Redirects](#redirects)
     - [Retry After](#retry-after)
+    - [Retry Policy](#retry-policy)
     - [Results](#results)
     - [Methods](#methods)
+    - [Deduplication](#deduplication)
+    - [Caching](#caching)
     - [Logger](#logger)
     - [User-Agent](#user-agent)
+    - [Headers](#headers)
   - [Global Configuration](#global-configuration)
   - [Callbacks and Events](#callbacks-and-events)
+- [Benchmarks](#benchmarks)
 - [Contributing](#contributing)
 - [Copyright and License](#copyright-and-license)
 
@@ -127,6 +133,63 @@ LinkChecker::Typhoeus::Hydra.configure do |config|
 end
 ```
 
+#### [LinkChecker::Async::HTTP](lib/ruby-link-checker/async/http/checker.rb)
+
+Fast, concurrent link checker that uses [async-http](https://github.com/socketry/async-http).
+
+```ruby
+require 'async'
+require 'async/http'
+require 'ruby-link-checker'
+
+# create a new instance of a checker
+checker = LinkChecker::Async::HTTP::Checker.new
+
+# log requests and response codes
+checker.logger.level = Logger::INFO
+
+links = [...] # array of URLs
+links.each do |url|
+  checker.check url
+end
+
+# examine failures and errors as they come
+checker.on :error, :failure do |result|
+  puts "FAIL: #{result}"
+end
+
+# runs all queued checks concurrently, will block until all requests have completed
+checker.run
+
+# examine results
+checker.results.each_pair do |bucket, results|
+  puts "#{bucket}: #{results.size}"
+end
+```
+
+You can pass `Async::HTTP` timeout options into a new instance of a checker, or configure timeouts globally.
+
+```ruby
+LinkChecker::Async::HTTP.configure do |config|
+  config.read_timeout = 5
+  config.open_timeout = 10
+end
+```
+
+You can also force a specific HTTP protocol version, `:http1` or `:http2`, per instance or globally. By default, `async-http` negotiates the protocol version with the server.
+
+```ruby
+LinkChecker::Async::HTTP.configure do |config|
+  config.protocol = :http2
+end
+```
+
+Async::HTTP reuses connection-pooling clients per scheme/host/port. Its default concurrency is unlimited; set `max_concurrency` to bound active requests without counting delayed retries against that limit.
+
+```ruby
+checker = LinkChecker::Async::HTTP::Checker.new(max_concurrency: 25)
+```
+
 #### [LinkChecker::Net::HTTP](lib/ruby-link-checker/net/http/checker.rb)
 
 Slow, sequential checker.
@@ -165,18 +228,43 @@ end
 
 #### Retries
 
-By default link checkers do not retry. You can set a number of times to retry all errors and failures with `retries`.
+By default link checkers do not retry. Set `retries` to enable retries for transient connection errors/timeouts and HTTP statuses 408, 425, 429, 500, 502, 503, and 504. Permanent HTTP failures and invalid URLs are not retried. HEAD-to-GET fallback still applies independently of the retry policy. The retry budget is shared across methods: with `retries: 2`, repeated transient failures produce `HEAD`, `HEAD`, `HEAD`, then `GET`.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(retries: 1)
+```
+
+#### Redirects
+
+Checkers follow up to 20 redirects per attempt and detect redirect loops. Relative redirects are resolved against the current response URL. Set `max_redirects` to change the limit, or use `0` to disable redirects.
 
 ```ruby
-checker = LinkChecker::Net::HTTP::Checker.new(retry: 1)
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(max_redirects: 10)
 ```
 
 #### Retry After
 
-If retry is enabled, link checkers can wait for a number of seconds before retrying.
+Set `retry_after` to a minimum delay in seconds (default `0`). Optional `retry_backoff` multiplies that delay on successive retries (default `1`), and `retry_jitter` adds a random fraction between `0` and the configured value (default `0`, maximum `1`). All delays are capped by `max_retry_delay` (default `60` seconds).
 
 ```ruby
-checker = LinkChecker::Net::HTTP::Checker.new(retry_after: 1) # wait for 1 second before retrying
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(
+  retries: 2,
+  retry_after: 1,
+  retry_backoff: 2,
+  retry_jitter: 0.1,
+  respect_retry_after: true,
+  max_retry_delay: 60
+)
+```
+
+With `respect_retry_after: true` (default `false`), HTTP `Retry-After` delta-seconds and dates provide a minimum delay, subject to the same cap. Invalid or past dates are ignored. Async delays yield without blocking other requests. Hydra queues delayed retries after the active batch drains, so actual delays can exceed the configured minimum without blocking completion callbacks. Delays apply to retries, not ordinary HEAD-to-GET fallback.
+
+#### Retry Policy
+
+Customize `retry_status_codes` to choose retryable HTTP statuses. Set it to `nil` to restore the previous behavior of retrying every failure and error, including malformed URLs and programming errors.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(retries: 1, retry_status_codes: [429, 503])
 ```
 
 #### Results
@@ -211,6 +299,34 @@ The following examples disables `GET` and only makes `HEAD` requests.
 checker = LinkChecker::Net::HTTP::Checker.new(methods: %w[HEAD])
 ```
 
+#### Deduplication
+
+Set `deduplicate: true` to coalesce identical in-flight requests and reuse successful results for the lifetime of the checker. Failures and errors are rechecked after completion. Each submission still receives its own result events and metadata, including events emitted before it joined an in-flight check. Deduplication is disabled by default.
+
+```ruby
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(deduplicate: true)
+checker.check('https://www.example.org', location: 'one.html')
+checker.check('https://www.example.org', location: 'two.html')
+checker.run # one check, two sets of result events
+```
+
+Identity includes the complete URL (including query values and fragment), transport, methods, headers, user-agent, and request-related settings. Changing those settings prevents reuse. URLs containing user-info credentials bypass deduplication and caching.
+
+#### Caching
+
+Persistent success caching is disabled by default. Enable it with an explicit file path and positive TTL in seconds. Recent successful checks can then be reused across checker instances; failures and errors are never stored. Cache writes are atomic and locked, and unreadable or malformed files fall back to normal checking.
+
+```ruby
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(
+  deduplicate: true,
+  cache: { file: 'tmp/link-checker/cache.json', ttl: 3600 }
+)
+```
+
+Each cache hit emits `:result` and `:success` with the new submission's metadata and `cached? == true`. Persistent hits return `LinkChecker::CachedResult` with the HTTP method, status, original/final URIs, and metadata; native request/response objects and headers are not persisted. Redirect targets containing user-info credentials are not written. Add the cache directory to your ignore rules.
+
+Caching alone does not coalesce in-flight checks. With deduplication enabled, successful results remain reusable within that checker even after the persistent TTL expires. Cached successes do not replay historical redirect/retry events.
+
 #### Logger
 
 Pass your own logger.
@@ -227,6 +343,20 @@ Pass your own user-agent. Default is `Ruby Link Checker/x.y.z`.
 checker = LinkChecker::Net::HTTP::Checker.new(user_agent: 'Custom Agent/1.0')
 ```
 
+#### Headers
+
+Some sites reject or return errors (403, 404) to requests that look like bots. Pass additional headers to mimic a browser.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(
+  headers: {
+    'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+    'Accept-Language' => 'en-US,en;q=0.5',
+    'DNT' => '1'
+  }
+)
+```
+
 ### Global Configuration
 
 All options can also be configured globally.
@@ -234,6 +364,7 @@ All options can also be configured globally.
 ```ruby
 LinkChecker.configure do |config|
   config.user_agent = 'Custom Agent/1.0'
+  config.headers = { 'DNT' => '1' }
   config.methods = ['HEAD', 'GET']
   config.logger = ::Logger.new(STDOUT)
 end
@@ -274,9 +405,22 @@ Events are called with results, which contain the following properties.
 | :request_headers  | Request headers.                                                |
 | :redirect_to      | A redirect URL in case of redirects.                            |
 | :error            | A raised error in case of errors.                               |
+| :error_type       | `:timeout`, `:connection`, or `:other` for errors; otherwise `nil`. |
+| :response_headers | Response headers, accessible using `[]`.                        |
+| :cached?          | Whether this result was reused from a successful check.         |
 
 See [result.rb](lib/ruby-link-checker/result.rb) for more details.
 
+## Benchmarks
+
+See [benchmarks/check.rb](benchmarks/check.rb) for a script comparing the performance of the three checkers (`LinkChecker::Net::HTTP`, `LinkChecker::Typhoeus::Hydra`, and `LinkChecker::Async::HTTP`) against a fixed set of URLs.
+
+```
+bundle exec ruby -Ilib benchmarks/check.rb
+```
+
+Anecdotal results show `LinkChecker::Async::HTTP` using significantly less CPU time than `LinkChecker::Typhoeus::Hydra` (both concurrent checkers), while wall-clock time is comparable between the two and depends on timeouts and the responsiveness of the URLs being checked. `LinkChecker::Net::HTTP` checks links sequentially and is significantly slower in wall-clock time for large batches of URLs; it's commented out by default in the script.
+
 ## Contributing
 
 You're encouraged to contribute to ruby-link-checker. See [CONTRIBUTING](CONTRIBUTING.md) for details.

data/RELEASING.md

@@ -2,59 +2,36 @@
 
 There are no hard rules about when to release ruby-link-checker. Release bug fixes frequently, features not so frequently and breaking API changes rarely.
 
-### Release
+Releases use [ruby-gem-release](https://github.com/dblock/ruby-gem-release/tree/v0.3.1) and the manual [Release workflow](.github/workflows/release.yml). Merges do not publish. Every workflow dispatch performs a real release.
 
-Run tests, check that all tests succeed locally.
+## One-Time Setup
 
-```
-bundle install
-rake
-```
-
-Check that the last build succeeded.
-
-Add a date to this release in [CHANGELOG.md](CHANGELOG.md).
-
-```
-### 0.2.2 (2015/7/10)
-```
+Create a GitHub environment named `release` in this repository. Restrict deployment to `main` and require approval when available.
 
-Remove the line with "Your contribution here.", since there will be no more contributions to this release.
+On [RubyGems.org](https://rubygems.org/gems/ruby-link-checker/trusted_publishers), configure a trusted publisher with repository owner `dblock`, repository name `ruby-link-checker`, workflow filename `release.yml`, and environment `release`. Set the separate workflow repository owner to `dblock` and name to `ruby-gem-release` because publication uses its reusable workflow. No stored RubyGems API key is needed.
 
-Commit your changes.
+The workflow requests `contents: write` and `id-token: write`. Branch rules must allow the release identity to push fast-forward release and next-version commits to `main` and create tags, without allowing force pushes or deletion. Adding the workflow does not configure these permissions or the trusted publisher.
 
-```
-git add README.md CHANGELOG.md lib/ruby-link-checker/version.rb
-git commit -m "Preparing for release, 0.2.2."
-```
+## Prepare and Publish
 
-Release.
+Prepare the intended version in `lib/ruby-link-checker/version.rb` and its matching `### VERSION (Next)` section in `CHANGELOG.md` through a PR. Keep exactly one `* Your contribution here.` placeholder in that section. The workflow releases the current version, dates the changelog, and removes the placeholder; it does not select a new release version.
 
-```
-$ rake release
+Run tests and RuboCop locally and review the release changes before merging:
 
-ruby-link-checker 0.2.2 built to pkg/ruby-link-checker-0.2.2.gem.
-Tagged v0.2.2.
-Pushed git commits and tags.
-Pushed ruby-link-checker 0.2.2 to rubygems.org.
+```sh
+bundle exec rake
 ```
 
-### Prepare for the Next Version
-
-Add the next release to [CHANGELOG.md](CHANGELOG.md).
+After merging, explicit release approval, and successful Trusted Publishing setup, dispatch from `main`:
 
+```sh
+gh workflow run release.yml --ref main
 ```
-### 0.2.3 (Next)
 
-* Your contribution here.
-```
+The workflow runs `bundle exec rake`, publishes the verified gem to RubyGems, pushes the release commit and tag atomically, and creates a GitHub Release with the gem attached. It then increments the patch version and pushes a new development commit with a fresh changelog section. Actions-token pushes do not trigger ordinary push workflows, so the release workflow runs tests and RuboCop before publication.
 
-Increment the third version number in [lib/ruby-link-checker/version.rb](lib/ruby-link-checker/version.rb).
+## Verify and Recover
 
-Commit your changes.
+Verify the published RubyGems version, matching tag and GitHub Release, attached gem, and next development version on `main`.
 
-```
-git add CHANGELOG.md lib/ruby-link-checker/version.rb
-git commit -m "Preparing for next development iteration, 0.2.3."
-git push origin main
-```
+Never automatically rerun a failed or timed-out publication: RubyGems may have accepted the upload. Check registry and GitHub state first. Do not republish an existing version or move its tag. Available `release.gem`, `release.json`, and `release.bundle` recovery artifacts are retained for 30 days; follow the [upstream recovery instructions](https://github.com/dblock/ruby-gem-release/blob/v0.3.1/RELEASING.md#verify-and-recover) to complete only the missing steps.

data/Rakefile

@@ -15,4 +15,12 @@ end
 require 'rubocop/rake_task'
 RuboCop::RakeTask.new(:rubocop)
 
+require 'ruby-gem-release/workflow/ruby_gem'
+require 'ruby-gem-release/changelog'
+
+RubyGemRelease::Workflow::RubyGem.new(
+  version: RubyGemRelease::Version::RubyConstant.new(path: 'lib/ruby-link-checker/version.rb', constant: 'VERSION'),
+  changelog: RubyGemRelease::Changelog::Intridea.new(placeholder: '* Your contribution here.')
+).install_tasks
+
 task default: %i[rubocop spec]

data/benchmarks/check.rb

@@ -1,31 +1,38 @@
 require 'benchmark'
 require 'net/http'
 require 'typhoeus'
+require 'async'
+require 'async/http'
 require 'ruby-link-checker'
 
 input = File.readlines(File.join(__dir__, 'data/opensearch.org/small.txt')).map(&:strip)
 
-Benchmark.bm do |benchmark|
-  # benchmark.report('LinkChecker::Net::HTTP::Checker') do
-  #   checker = LinkChecker::Net::HTTP::Checker.new
-  #   checker.logger.level = Logger::INFO
-  #   input.each do |url|
-  #     checker.check url
-  #   end
-  #   checker.results.each_pair do |bucket, results|
-  #     puts "#{bucket}: #{results.size}"
-  #   end
-  # end
+CHECKERS = [
+  # LinkChecker::Net::HTTP::Checker,
+  LinkChecker::Typhoeus::Hydra::Checker,
+  LinkChecker::Async::HTTP::Checker
+].freeze
 
-  benchmark.report('LinkChecker::Typhoeus::Hydra::Checker') do
-    checker = LinkChecker::Typhoeus::Hydra::Checker.new
-    checker.logger.level = Logger::INFO
-    input.each do |url|
-      checker.check url
-    end
-    checker.run
-    checker.results.each_pair do |bucket, results|
-      puts "#{bucket}: #{results.size}"
+Benchmark.bm(40) do |benchmark|
+  CHECKERS.each do |checker_klass|
+    benchmark.report(checker_klass.name) do
+      checker = checker_klass.new
+      checker.logger.level = Logger::INFO
+      case checker
+      when LinkChecker::Typhoeus::Hydra::Checker
+        checker.timeout = 5 # 1/12th of the default 60s
+        checker.connecttimeout = 1 # 1/10th of the default 10s
+      else
+        checker.read_timeout = 5 # 1/12th of the default 60s
+        checker.open_timeout = 5 # 1/12th of the default 60s
+      end
+      input.each do |url|
+        checker.check url
+      end
+      checker.run if checker.respond_to?(:run)
+      checker.results.each_pair do |bucket, results|
+        puts "#{bucket}: #{results.size}"
+      end
     end
   end
 end

data/lib/ruby-link-checker/async/http/checker.rb

@@ -0,0 +1,118 @@
+module LinkChecker
+  module Async
+    module HTTP
+      class Task < ::LinkChecker::Task
+        def run!
+          checker._queue(delay: delay) { _run! }
+        end
+
+        private
+
+        def _run!
+          client = checker._client_for(uri)
+          headers = ::Protocol::HTTP::Headers[
+            [['user-agent', checker.user_agent]] + checker.headers.map { |key, value| [key, value] }
+          ]
+          request = ::Protocol::HTTP::Request[method, uri.request_uri, headers]
+          response = _call(client, request)
+          result! Result.new(uri, method, original_uri, request, response, options)
+        rescue StandardError => e
+          result! ResultError.new(uri, method, original_uri, e, options)
+        ensure
+          response&.close
+        end
+
+        def _call(client, request)
+          if checker.read_timeout
+            ::Async::Task.current.with_timeout(checker.read_timeout) { client.call(request) }
+          else
+            client.call(request)
+          end
+        end
+      end
+
+      class Checker < LinkChecker::Checker
+        extend ::LinkChecker::Async::HTTP::Config
+
+        attr_accessor(*LinkChecker::Async::HTTP::Config::ATTRIBUTES)
+
+        def initialize(options = {})
+          LinkChecker::Async::HTTP::Config::ATTRIBUTES.each do |key|
+            send("#{key}=", options[key] || LinkChecker::Async::HTTP::Config.send(key))
+          end
+          @queue = []
+          @clients = {}
+          if max_concurrency && (!max_concurrency.is_a?(Integer) || max_concurrency < 1)
+            raise ArgumentError, "Invalid concurrency limit: #{max_concurrency}"
+          end
+
+          super
+        end
+
+        # Runs all queued checks concurrently and blocks until they complete.
+        def run
+          require 'async/semaphore'
+
+          Barrier do |barrier|
+            @barrier = barrier
+            @semaphore = ::Async::Semaphore.new(max_concurrency) if max_concurrency
+            _flush!
+          end
+        ensure
+          @barrier = nil
+          @semaphore = nil
+          _close_clients!
+        end
+
+        def _queue(delay: 0, &block)
+          if @barrier
+            @barrier.async do
+              sleep(delay) if delay.positive?
+              @semaphore ? @semaphore.acquire(&block) : block.call
+            end
+          else
+            @queue << [delay, block]
+          end
+        end
+
+        # Returns a client for the given URI's host/port/scheme, reusing (and
+        # pooling connections on) one client per endpoint across all checks.
+        def _client_for(uri)
+          key = [uri.scheme, uri.host, uri.port]
+          @clients[key] ||= begin
+            endpoint = ::Async::HTTP::Endpoint.parse(uri.to_s, timeout: open_timeout, **_endpoint_options)
+            ::Async::HTTP::Client.new(endpoint)
+          end
+        end
+
+        private
+
+        # Maps the configured protocol (:http1 or :http2) to the
+        # corresponding async-http protocol implementation.
+        def _endpoint_options
+          case protocol
+          when :http1
+            { protocol: ::Async::HTTP::Protocol::HTTP1 }
+          when :http2
+            { protocol: ::Async::HTTP::Protocol::HTTP2 }
+          when nil
+            {}
+          else
+            raise ArgumentError, "Unsupported protocol: #{protocol.inspect}, expected :http1 or :http2"
+          end
+        end
+
+        def _flush!
+          queue = @queue
+          @queue = []
+          queue.each { |delay, block| _queue(delay: delay, &block) }
+        end
+
+        def _close_clients!
+          @clients.each_value(&:close)
+          @clients = {}
+        end
+      end
+    end
+  end
+end

data/lib/ruby-link-checker/async/http/config.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module LinkChecker
+  module Async
+    module HTTP
+      module Config
+        extend self
+
+        ATTRIBUTES = %i[
+          read_timeout
+          open_timeout
+          protocol
+          max_concurrency
+        ].freeze
+
+        attr_accessor(*Config::ATTRIBUTES)
+
+        def reset
+          self.read_timeout = nil
+          self.open_timeout = nil
+          self.protocol = nil
+          self.max_concurrency = nil
+        end
+      end
+
+      class << self
+        def configure
+          block_given? ? yield(Config) : Config
+        end
+
+        def config
+          Config
+        end
+      end
+    end
+  end
+end
+
+LinkChecker::Async::HTTP::Config.reset

data/lib/ruby-link-checker/async/http/result.rb

@@ -0,0 +1,85 @@
+module LinkChecker
+  module Async
+    module HTTP
+      # A thin, case-insensitive wrapper around Protocol::HTTP::Headers,
+      # since Protocol::HTTP::Headers#[] expects a lower-cased key.
+      class Headers
+        def initialize(headers)
+          @headers = headers
+        end
+
+        def [](key)
+          value = @headers[key.to_s.downcase]
+          value.is_a?(Array) ? value.join(', ') : value
+        end
+      end
+
+      # Wraps an Async::HTTP response to expose a `code` and case-insensitive
+      # `headers`, similar to the other checkers' response objects.
+      class Response
+        attr_reader :status, :headers
+
+        def initialize(response)
+          @status = response.status
+          @headers = Headers.new(response.headers)
+        end
+
+        def code
+          @status.to_s
+        end
+      end
+
+      class Result < ::LinkChecker::Result
+        attr_accessor :request, :response
+
+        def initialize(uri, method, original_uri, request, response, options)
+          @request = request
+          @response = response && Response.new(response)
+          super(uri, method, original_uri, options)
+        end
+
+        def error?
+          false
+        end
+
+        def failure?
+          !success? && !redirect?
+        end
+
+        def code
+          @code ||= begin
+            response.status
+          rescue StandardError
+            -1
+          end
+        end
+
+        def request_headers
+          Headers.new(request.headers)
+        end
+
+        def response_headers
+          response&.headers || {}
+        end
+
+        def redirect_to
+          return nil unless response
+
+          response.headers['location']
+        end
+
+        def redirect?
+          return false unless response
+
+          [301, 302, 303, 307, 308].include?(code)
+        end
+
+        def success?
+          return false unless response
+
+          code >= 200 && code <= 299
+        end
+      end
+    end
+  end
+end

data/lib/ruby-link-checker/async/http.rb

@@ -0,0 +1,3 @@
+require_relative 'http/config'
+require_relative 'http/result'
+require_relative 'http/checker'