ruby-link-checker 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3edc71cca9ca0e7f43a58cc4776886f4ae55f1ef1709901be7a1aad93bf15f28
+  data.tar.gz: e306860ec44ecc137ddcf3fe2a585e2448b4a48ac957276f4f749337b2a469da
+SHA512:
+  metadata.gz: da986eb191a3016dcedeb64d554e0da5d14310591bf9f1772d82bfa3572ead7671d3d29c8c16ce8fcdfd8c0924f720f3478fb083dd3d306a5bfb6cb8a28b1859
+  data.tar.gz: eef0ee4b258053dd33bae73c71a9569719dad39cee2860e885d67a37dc1eedab215b03566de5b289e154d6cae81c78f8786f4bb08eaf15f5bf862062db1a1dee

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+### 0.1.0 (2023-04-02)
+
+* Initial public release - [@dblock](https://github.com/dblock).

data/CONTRIBUTING.md

@@ -0,0 +1,125 @@
+# Contributing to ruby-link-checker
+
+This project is work of [many contributors](https://github.com/dblock/ruby-link-checker/graphs/contributors).
+
+You're encouraged to submit [pull requests](https://github.com/dblock/ruby-link-checker/pulls), [propose features and discuss issues](https://github.com/dblock/ruby-link-checker/issues).
+
+In the examples below, substitute your Github username for `contributor` in URLs.
+
+### Fork the Project
+
+Fork the [project on Github](https://github.com/dblock/ruby-link-checker) and check out your copy.
+
+```
+git clone https://github.com/contributor/ruby-link-checker.git
+cd ruby-link-checker
+git remote add upstream https://github.com/dblock/ruby-link-checker.git
+```
+
+### Bundle Install and Test
+
+Ensure that you can build the project and run tests.
+
+```
+bundle install
+bundle exec rake
+```
+
+## Contribute Code
+
+### Create a Topic Branch
+
+Make sure your fork is up-to-date and create a topic branch for your feature or bug fix.
+
+```
+git checkout main
+git pull upstream main
+git checkout -b my-feature-branch
+```
+
+### Write Tests
+
+Try to write a test that reproduces the problem you're trying to fix or describes a feature that you want to build. Add tests to [spec](spec).
+
+We definitely appreciate pull requests that highlight or reproduce a problem, even without a fix.
+
+### Write Code
+
+Implement your feature or bug fix.
+
+Ruby style is enforced with [Rubocop](https://github.com/bbatsov/rubocop). Run `bundle exec rubocop` and fix any style issues highlighted, auto-correct issues when possible with `bundle exec rubocop -a`. To silence generally ingored issues, including line lengths or code complexity metrics, run `bundle exec rubocop --auto-gen-config`.
+
+Make sure that `bundle exec rake` completes without errors.
+
+### Write Documentation
+
+Document any external behavior in the [README](README.md).
+
+### Update Changelog
+
+Add a line to [CHANGELOG](CHANGELOG.md) under *Next Release*. Don't remove *Your contribution here*.
+
+Make it look like every other line, including a link to the issue being fixed, your name and link to your Github account.
+
+### Commit Changes
+
+Make sure git knows your name and email address:
+
+```
+git config --global user.name "Your Name"
+git config --global user.email "contributor@example.com"
+```
+
+Writing good commit logs is important. A commit log should describe what changed and why.
+
+```
+git add ...
+git commit
+```
+
+### Push
+
+```
+git push origin my-feature-branch
+```
+
+### Make a Pull Request
+
+Go to https://github.com/contributor/ruby-link-checker and select your feature branch. Click the 'Pull Request' button and fill out the form. Pull requests are usually reviewed within a few days.
+
+### Update CHANGELOG Again
+
+Update the [CHANGELOG](CHANGELOG.md) with the pull request number. A typical entry looks as follows.
+
+```
+* [#123](https://github.com/dblock/ruby-link-checker/pull/123): Reticulated splines - [@contributor](https://github.com/contributor).
+```
+
+Amend your previous commit and force push the changes.
+
+```
+git commit --amend
+git push origin my-feature-branch -f
+```
+
+### Rebase
+
+If you've been working on a change for a while, rebase with upstream/main.
+
+```
+git fetch upstream
+git rebase upstream/main
+git push origin my-feature-branch -f
+```
+
+### Check on Your Pull Request
+
+Go back to your pull request after a few minutes and see whether it passed muster with Travis-CI. Everything should look green, otherwise fix issues and amend your commit as described above.
+
+### Be Patient
+
+It's likely that your change will not be merged and that the nitpicky maintainers will ask you to do more, or fix seemingly benign problems. Hang on there!
+
+## Thank You
+
+Please do know that we really appreciate and value your time and work. We love you, really.

data/Dangerfile

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+changelog.check!
+toc.check!

data/Gemfile

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+source 'http://rubygems.org'
+
+gemspec
+
+gem 'rake'
+
+group :development, :test do
+  gem 'danger'
+  gem 'danger-changelog', '0.6.1'
+  gem 'danger-toc', '0.2.0'
+  gem 'rspec', '~> 3.0'
+  gem 'rubocop', '~> 1.0'
+  gem 'rubocop-rake'
+  gem 'rubocop-rspec'
+  gem 'vcr'
+  gem 'webmock'
+end
+
+group :test do
+  gem 'simplecov', require: false
+end

data/Gemfile.lock

@@ -0,0 +1,162 @@
+PATH
+  remote: .
+  specs:
+    ruby-link-checker (0.1.0)
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    activesupport (7.0.4.3)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 1.6, < 2)
+      minitest (>= 5.1)
+      tzinfo (~> 2.0)
+    addressable (2.8.1)
+      public_suffix (>= 2.0.2, < 6.0)
+    ast (2.4.2)
+    claide (1.1.0)
+    claide-plugins (0.9.2)
+      cork
+      nap
+      open4 (~> 1.3)
+    colored2 (3.1.2)
+    concurrent-ruby (1.2.2)
+    cork (0.3.0)
+      colored2 (~> 3.1)
+    crack (0.4.5)
+      rexml
+    danger (9.2.0)
+      claide (~> 1.0)
+      claide-plugins (>= 0.9.2)
+      colored2 (~> 3.1)
+      cork (~> 0.1)
+      faraday (>= 0.9.0, < 3.0)
+      faraday-http-cache (~> 2.0)
+      git (~> 1.7)
+      kramdown (~> 2.3)
+      kramdown-parser-gfm (~> 1.0)
+      no_proxy_fix
+      octokit (~> 5.0)
+      terminal-table (>= 1, < 4)
+    danger-changelog (0.6.1)
+      danger-plugin-api (~> 1.0)
+    danger-plugin-api (1.0.0)
+      danger (> 2.0)
+    danger-toc (0.2.0)
+      activesupport
+      danger-plugin-api (~> 1.0)
+      kramdown
+    diff-lcs (1.5.0)
+    docile (1.4.0)
+    ethon (0.16.0)
+      ffi (>= 1.15.0)
+    faraday (2.7.4)
+      faraday-net_http (>= 2.0, < 3.1)
+      ruby2_keywords (>= 0.0.4)
+    faraday-http-cache (2.4.1)
+      faraday (>= 0.8)
+    faraday-net_http (3.0.2)
+    ffi (1.15.5)
+    git (1.18.0)
+      addressable (~> 2.8)
+      rchardet (~> 1.8)
+    hashdiff (1.0.1)
+    i18n (1.12.0)
+      concurrent-ruby (~> 1.0)
+    json (2.6.3)
+    kramdown (2.4.0)
+      rexml
+    kramdown-parser-gfm (1.1.0)
+      kramdown (~> 2.0)
+    minitest (5.18.0)
+    nap (1.1.0)
+    no_proxy_fix (0.1.2)
+    octokit (5.6.1)
+      faraday (>= 1, < 3)
+      sawyer (~> 0.9)
+    open4 (1.3.4)
+    parallel (1.22.1)
+    parser (3.2.1.1)
+      ast (~> 2.4.1)
+    public_suffix (5.0.1)
+    rainbow (3.1.1)
+    rake (13.0.6)
+    rchardet (1.8.0)
+    regexp_parser (2.7.0)
+    rexml (3.2.5)
+    rspec (3.12.0)
+      rspec-core (~> 3.12.0)
+      rspec-expectations (~> 3.12.0)
+      rspec-mocks (~> 3.12.0)
+    rspec-core (3.12.1)
+      rspec-support (~> 3.12.0)
+    rspec-expectations (3.12.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.12.0)
+    rspec-mocks (3.12.4)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.12.0)
+    rspec-support (3.12.0)
+    rubocop (1.48.1)
+      json (~> 2.3)
+      parallel (~> 1.10)
+      parser (>= 3.2.0.0)
+      rainbow (>= 2.2.2, < 4.0)
+      regexp_parser (>= 1.8, < 3.0)
+      rexml (>= 3.2.5, < 4.0)
+      rubocop-ast (>= 1.26.0, < 2.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 2.4.0, < 3.0)
+    rubocop-ast (1.28.0)
+      parser (>= 3.2.1.0)
+    rubocop-capybara (2.17.1)
+      rubocop (~> 1.41)
+    rubocop-rake (0.6.0)
+      rubocop (~> 1.0)
+    rubocop-rspec (2.19.0)
+      rubocop (~> 1.33)
+      rubocop-capybara (~> 2.17)
+    ruby-progressbar (1.13.0)
+    ruby2_keywords (0.0.5)
+    sawyer (0.9.2)
+      addressable (>= 2.3.5)
+      faraday (>= 0.17.3, < 3)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
+    terminal-table (3.0.2)
+      unicode-display_width (>= 1.1.1, < 3)
+    typhoeus (1.4.0)
+      ethon (>= 0.9.0)
+    tzinfo (2.0.6)
+      concurrent-ruby (~> 1.0)
+    unicode-display_width (2.4.2)
+    vcr (6.1.0)
+    webmock (3.18.1)
+      addressable (>= 2.8.0)
+      crack (>= 0.3.2)
+      hashdiff (>= 0.4.0, < 2.0.0)
+
+PLATFORMS
+  arm64-darwin-21
+
+DEPENDENCIES
+  danger
+  danger-changelog (= 0.6.1)
+  danger-toc (= 0.2.0)
+  rake
+  rspec (~> 3.0)
+  rubocop (~> 1.0)
+  rubocop-rake
+  rubocop-rspec
+  ruby-link-checker!
+  simplecov
+  typhoeus
+  vcr
+  webmock
+
+BUNDLED WITH
+   2.4.2

data/LICENSE.md

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) Daniel Doubrovkine and contributors.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,279 @@
+Fast Ruby Link Checker
+======================
+
+[![Gem Version](http://img.shields.io/gem/v/ruby-link-checker.svg)](http://badge.fury.io/rb/ruby-link-checker)
+[![Build Status](https://github.com/dblock/ruby-link-checker/workflows/test/badge.svg?branch=main)](https://github.com/dblock/ruby-link-checker/actions)
+[![Code Climate](https://codeclimate.com/github/dblock/ruby-link-checker.svg)](https://codeclimate.com/github/dblock/ruby-link-checker)
+[![Test Coverage](https://api.codeclimate.com/v1/badges/164f1e23fc706b6efa63/test_coverage)](https://codeclimate.com/github/dblock/ruby-link-checker/test_coverage)
+
+A fast Ruby link checker with support for multiple HTTP libraries. Does not parse documents, just checks links. Fast. Anecdotal benchmarking on a M1 mac and T1 Internet yields ~50 URLs per second with `LinkChecker::Typhoeus::Hydra`.
+
+## Table of Contents
+
+- [Usage](#usage)
+  - [Dependencies](#dependencies)
+  - [Basic Usage](#basic-usage)
+  - [Passing Options](#passing-options)
+  - [Checkers](#checkers)
+    - [LinkChecker::Typhoeus::Hydra](#linkcheckertyphoeushydra)
+    - [LinkChecker::Net::HTTP](#linkcheckernethttp)
+  - [Options](#options)
+    - [Retries](#retries)
+    - [Results](#results)
+    - [Methods](#methods)
+    - [Logger](#logger)
+    - [User-Agent](#user-agent)
+  - [Global Configuration](#global-configuration)
+  - [Callbacks and Events](#callbacks-and-events)
+- [Contributing](#contributing)
+- [Copyright and License](#copyright-and-license)
+
+## Usage
+
+### Dependencies
+
+The [`LinkChecker::Typhoeus::Hydra`](lib/ruby-link-checker/typhoeus/hydra/checker.rb) link checker is recommended. 
+
+Add `typhoeus` and `ruby-link-checker` to your `Gemfile` and run `bundle install`.
+
+```ruby
+gem 'typhoeus'
+gem 'ruby-link-checker'
+```
+
+### Basic Usage
+
+```ruby
+require 'typhoeus'
+require 'ruby-link-checker'
+
+# create a new checker instance
+checker = LinkChecker::Typhoeus::Hydra::Checker.new
+
+# queue URLs to check
+links = [...]
+links.each do |url|
+  checker.check url
+end
+
+# run the checks
+checker.run
+
+# display buckets of results
+checker.results.each_pair do |bucket, results|
+  puts "#{bucket}: #{results.size}"
+end
+```
+
+### Passing Options
+
+You can pipe custom options through `check` and retrieve them in events as follows.
+
+```ruby
+checker.check 'https://www.example.org', { location: 'page.html' }
+
+checker.on :success do |result|
+  result.options # contains { location: 'page.html' }
+end
+```
+
+### Checkers
+
+#### [LinkChecker::Typhoeus::Hydra](lib/ruby-link-checker/typhoeus/hydra/checker.rb)
+
+Fast link checker that uses [Typhoeus](https://typhoeus.github.io/). 
+
+```ruby
+require 'typhoeus'
+require 'ruby-link-checker'
+
+# create a new instance of a checker
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(
+  hydra: {
+    # lower than the Typhoeus default of 200, seems to start breaking around 50+
+    max_concurrency: 25
+  }
+)
+
+# log requests and response codes
+checker.logger.level = Logger::INFO
+
+links = [...] # array of URLs
+links.each do |url|
+  checker.check url
+end
+
+# examine failures and errors as they come
+checker.on :error, :failure do |result|
+  puts "FAIL: #{result}"
+end    
+
+# execute Hydra#run, will block until all requests have completed
+checker.run
+
+# examine results
+checker.results.each_pair do |bucket, results|
+  puts "#{bucket}: #{results.size}"
+end
+```
+
+You can pass `Typhoeus` timeout options into a new instance of a checker, or configure timeouts globally.
+
+```ruby
+LinkChecker::Typhoeus::Hydra.configure do |config|
+  config.timeout = 5
+  config.connecttimeout = 10
+end
+```
+
+#### [LinkChecker::Net::HTTP](lib/ruby-link-checker/net/http/checker.rb)
+
+Slow, sequential checker.
+
+```ruby
+require 'net/http'
+require 'ruby-link-checker'
+
+# create a new instance of a checker
+checker = LinkChecker::Net::HTTP::Checker.new
+
+# log requests and response codes
+checker.logger.level = Logger::INFO
+
+links = [...] # array of URLs
+links.each do |url|
+  checker.check url
+end
+
+# examine results
+checker.results.each_pair do |bucket, results|
+  puts "#{bucket}: #{results.size}"
+end
+```
+
+You can pass `Net::HTTP` timeout options into a new instance of a checker, or configure timeouts globally.
+
+```ruby
+LinkChecker::Net::HTTP.configure do |config|
+  config.read_timeout = 5
+  config.open_timeout = 10
+end
+```
+
+### Options
+
+#### Retries
+
+By default link checkers do not retry. You can set a number of times to retry all errors and failures with `retries`.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(retry: 1)
+```
+
+#### Results
+
+By default checkers collect results. 
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(results: false)
+...
+checker.run
+
+checker.results # => { error: [...], failure: [...], success: [...] }
+```
+
+You can disable this with `results: false`.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(results: false)
+...
+checker.run
+
+checker.results # => nil
+```
+
+#### Methods
+
+By default checkers try a `HEAD` request, followed by a `GET` if `HEAD` fails. You can change this behavior by specifying other methods.
+
+The following examples disables `GET` and only makes `HEAD` requests.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(methods: %w[HEAD])
+```
+
+#### Logger
+
+Pass your own logger.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(logger: Logger.new(STDOUT))
+```
+
+#### User-Agent
+
+Pass your own user-agent. Default is `Ruby Link Checker/x.y.z`.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(user_agent: 'Custom Agent/1.0')
+```
+
+### Global Configuration
+
+All options can also be configured globally.
+
+```ruby
+LinkChecker.configure do |config|
+  config.user_agent = 'Custom Agent/1.0'
+  config.methods = ['HEAD', 'GET']
+  config.logger = ::Logger.new(STDOUT)
+end
+```
+
+### Callbacks and Events
+
+Events enable processing of results as they become available.
+
+```ruby
+checker.on :result do |result|
+  puts result # any result
+end
+
+checker.on :error, :failure do |result|
+  puts result # error or failure
+end
+```
+
+Checkers support the following events.
+
+| Event    | Description                                                    |
+|----------|----------------------------------------------------------------|
+| :retry   | A request is being retried on failure or error.                |
+| :result  | A new result, any of success, failure, or error.               |
+| :success | A valid URL, usually a 2xx response from the server.           |
+| :failure | A failed URL, usually a 4xx or a 5xx response from the server. |
+| :error   | An error, such as an invalid URL or a network timeout.         |
+
+Events are called with results, which contain the following properties.
+
+| Property          | Description                                                     |
+|-------------------|-----------------------------------------------------------------|
+| :url              | The original URL before redirects.                              |
+| :result_url       | The last URL, different from `url` in case of redirects.        |
+| :method           | The result HTTP method.                                         |
+| :code             | HTTP error code.                                                |
+| :request_headers  | Request headers.                                                |
+| :redirect_to      | A redirect URL in case of redirects.                            |
+| :error            | A raised error in case of errors.                               |
+
+See [result.rb](lib/ruby-link-checker/result.rb) for more details.
+
+## Contributing
+
+You're encouraged to contribute to ruby-link-checker. See [CONTRIBUTING](CONTRIBUTING.md) for details.
+
+## Copyright and License
+
+Copyright (c) Daniel Doubrovkine and [Contributors](CHANGELOG.md).
+
+This project is licensed under the [MIT License](LICENSE.md).

data/RELEASING.md

@@ -0,0 +1,60 @@
+# Releasing ruby-link-checker
+
+There are no hard rules about when to release ruby-link-checker. Release bug fixes frequently, features not so frequently and breaking API changes rarely.
+
+### Release
+
+Run tests, check that all tests succeed locally.
+
+```
+bundle install
+rake
+```
+
+Check that the last build succeeded.
+
+Add a date to this release in [CHANGELOG.md](CHANGELOG.md).
+
+```
+### 0.2.2 (2015/7/10)
+```
+
+Remove the line with "Your contribution here.", since there will be no more contributions to this release.
+
+Commit your changes.
+
+```
+git add README.md CHANGELOG.md lib/ruby-link-checker/version.rb
+git commit -m "Preparing for release, 0.2.2."
+```
+
+Release.
+
+```
+$ rake release
+
+ruby-link-checker 0.2.2 built to pkg/ruby-link-checker-0.2.2.gem.
+Tagged v0.2.2.
+Pushed git commits and tags.
+Pushed ruby-link-checker 0.2.2 to rubygems.org.
+```
+
+### Prepare for the Next Version
+
+Add the next release to [CHANGELOG.md](CHANGELOG.md).
+
+```
+### 0.2.3 (Next)
+
+* Your contribution here.
+```
+
+Increment the third version number in [lib/ruby-link-checker/version.rb](lib/ruby-link-checker/version.rb).
+
+Commit your changes.
+
+```
+git add CHANGELOG.md lib/ruby-link-checker/version.rb
+git commit -m "Preparing for next development iteration, 0.2.3."
+git push origin main
+```

data/Rakefile

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require 'rubygems'
+require 'bundler/gem_tasks'
+
+Bundler.setup :default, :development
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+require 'rubocop/rake_task'
+RuboCop::RakeTask.new(:rubocop)
+
+task default: %i[rubocop spec]