final_redirect_url 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 522f5bf5f63725bfacfd9615a3b37ae003e42712a9663fd5f7af11b8ce2c2f52
-  data.tar.gz: b25c7ff42bade483244d24c1642da9097cd930326f6ebbee4f1e1aba5e854699
+  metadata.gz: 7c9fa9551dd7959832dfebc8b24e0cf5c5b326100385a2bf287dab1e50b98581
+  data.tar.gz: 2ebc2f9667d598f693fbf766c9eb55991477281cc8fff327ef8d2b693570e0f6
 SHA512:
-  metadata.gz: bafed8a367867eee62492c7f101682595a7a0c37736e3d71e7e79b29f0716c962b5e00af2ef3c5338148fb7e2e533a003a8e20b42dc852dbd0fa16d866b30514
-  data.tar.gz: e3dd7b15536e3e4bdcccfed12b227042707c6134f64f32d8045f86dd9c015ed1d5dfd05e119673416a1bd97efd0d4d79a7863e929adb893685fef2e1d429982e
+  metadata.gz: 54a6f3c70af375bf6743eb59f2c4441e4e0d072db4cd654ca94001daa0994638c896dcb2940a36f969004fd42e2f0675c2a278c64535ea6225e33e9a529ac237
+  data.tar.gz: 87c4b58fd877c878b109ad2f5ade9f802710e038cab50e9cf1c47ab89ef420c18cbe49570044313a26831f293176fb228df98a7fa78052c68da534f91b666b9e

data/CHANGELOG.md

@@ -7,6 +7,55 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
 
+## [2.1.0] – 2026-11-05
+
+### Fork Contributions Integrated
+Analysed all six public forks. Two contained work worth considering:
+
+- **counterfind/final_redirect_url** (9 commits ahead): introduced Nokogiri meta-refresh parsing,
+  per-request timeouts, and HTTPS via `use_ssl`. The meta-refresh concept was adopted and
+  rewritten from scratch (removed `pry`, removed `KNOWN_SHORTENERS`, eliminated the
+  thread-unsafe module-level `@final_url`, fixed the deprecated `URI.regexp` call, replaced
+  `rescue Exception`, and switched `uri.host` → `uri.to_s` to preserve ports). Architecture
+  unchanged.
+- **HowardSnowden/final_redirect_url** (4 commits ahead): extracted `get_response` as a named
+  method and added explicit SSL cipher configuration. The pattern was incorporated into the
+  new `fetch_response` method; the hardcoded cipher list was not adopted (unnecessary for
+  modern Ruby/OpenSSL).
+- **pilot34, mbv, ymow**: trivial (limit logging, version bump, no changes) — discarded.
+
+### Added
+- **`FinalRedirectUrl.trace(url, options)`** — returns `Array<Hop>` documenting every URL and
+  HTTP status code visited in the redirect chain. Useful for debugging and link audits.
+- **`FinalRedirectUrl::Hop`** struct with `url` (String) and `status` (Integer, nil when depth
+  exhausted) attributes.
+- **Custom headers** (`options[:headers]`) — pass any HTTP headers (e.g. `User-Agent`, `Accept`,
+  `Authorization`) to bypass anti-bot blocks or authenticate requests.
+- **Configurable timeouts** — `options[:open_timeout]` and `options[:read_timeout]` default to
+  **60 seconds** (up from the previous 10 s hard-coded value).
+- **`verify_ssl` option** (`true` by default) — set `verify_ssl: false` to bypass strict SSL
+  certificate validation (e.g. for self-signed certificates in development).
+- **Stateful cookie handling** — `Set-Cookie` headers from every 3xx response are captured and
+  injected as `Cookie` headers into subsequent requests, enabling cookie-gated redirect chains
+  to be followed correctly.
+- **Meta-refresh parsing** (via Nokogiri, adapted from the `counterfind` fork) — when a 200 OK
+  response contains `<meta http-equiv="refresh">`, the gem follows the redirect URL. Handles
+  case-insensitive `HTTP-EQUIV`, quoted URLs, and malformed `content` attributes. Only fires
+  for `text/html` responses.
+- **nokogiri** added as a runtime dependency.
+
+### Changed
+- Switched from `HEAD` to `GET` requests. `GET` is required so that the response body is
+  available for meta-refresh parsing; it also improves compatibility with servers that reject
+  `HEAD` with 405.
+- `final_redirect_url` now delegates to `trace` internally, eliminating duplicated logic.
+- Default timeouts raised from 10 s to **60 s**.
+
+### No Open Issues
+All issues were resolved in v2.0.0. No new open issues exist at the time of this release.
+
+---
+
 ## [2.0.0] – 2026-11-05
 
 ### Breaking Changes
@@ -54,6 +103,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   headless browser and are outside the scope of this gem. The gem returns the last HTTP-level
   reachable URL in such cases.
 
+[2.1.0]: https://github.com/indyarocks/final_redirect_url/compare/v2.0.0...v2.1.0
 [2.0.0]: https://github.com/indyarocks/final_redirect_url/compare/v0.1.1...v2.0.0
 [i2]: https://github.com/indyarocks/final_redirect_url/issues/2
 [i3]: https://github.com/indyarocks/final_redirect_url/issues/3

data/README.md

@@ -3,69 +3,128 @@
 [![CI](https://github.com/indyarocks/final_redirect_url/actions/workflows/ci.yml/badge.svg)](https://github.com/indyarocks/final_redirect_url/actions/workflows/ci.yml)
 [![Gem Version](https://badge.fury.io/rb/final_redirect_url.svg)](https://badge.fury.io/rb/final_redirect_url)
 
-Ruby gem that follows HTTP/HTTPS redirects and returns the final destination URL.
+Ruby gem that follows HTTP/HTTPS redirects — including `<meta http-equiv="refresh">` and
+cookie-gated chains — and returns the final destination URL.
 
 ## Requirements
 
 - Ruby >= 3.0
+- [Nokogiri](https://nokogiri.org) (bundled as a runtime dependency for meta-refresh parsing)
 
 ## Installation
 
-Add this line to your application's Gemfile:
-
 ```ruby
 gem 'final_redirect_url'
 ```
 
-Then run:
-
 ```
 $ bundle install
 ```
 
-Or install directly:
+or
 
 ```
 $ gem install final_redirect_url
 ```
 
-## Usage
-
-The input URL **must** include the `http://` or `https://` scheme.
+## Basic usage
 
 ```ruby
 require 'final_redirect_url'
 
-# Simple redirect following
 FinalRedirectUrl.final_redirect_url('http://google.com')
 # => "https://www.google.com/"
 
-# Ports are preserved
-FinalRedirectUrl.final_redirect_url('http://localhost:3000/items/42')
-# => "http://localhost:3000/items/42"
-
-# Custom redirect depth (default: 10)
-FinalRedirectUrl.final_redirect_url('http://example.com', depth: 5)
-# => "http://final.example.com/"
-
 # Invalid / missing protocol → empty string
 FinalRedirectUrl.final_redirect_url('www.example.com')
 # => ""
 ```
 
+## Options
+
+All options are passed as a second argument hash:
+
+| Option         | Type    | Default | Description |
+|----------------|---------|---------|-------------|
+| `:depth`        | Integer | `10`    | Maximum redirect hops to follow |
+| `:open_timeout` | Integer | `60`    | TCP connect timeout in seconds |
+| `:read_timeout` | Integer | `60`    | Read timeout in seconds |
+| `:verify_ssl`   | Boolean | `true`  | Enforce SSL certificate validation |
+| `:headers`      | Hash    | `{}`    | Extra HTTP headers sent with every request |
+
+```ruby
+# Limit redirect depth
+FinalRedirectUrl.final_redirect_url('http://example.com', depth: 5)
+
+# Custom timeouts (seconds)
+FinalRedirectUrl.final_redirect_url('http://slow.example.com', open_timeout: 5, read_timeout: 30)
+
+# Bypass SSL certificate verification (e.g. self-signed certs in development)
+FinalRedirectUrl.final_redirect_url('https://self-signed.local', verify_ssl: false)
+
+# Custom headers — override User-Agent or add Authorization / Accept
+FinalRedirectUrl.final_redirect_url(
+  'http://bot-protected.example.com',
+  headers: {
+    'User-Agent' => 'Mozilla/5.0 (compatible; MyApp/1.0)',
+    'Accept'     => 'text/html'
+  }
+)
+```
+
+## Redirect trace
+
+`FinalRedirectUrl.trace` returns every hop in the chain as an array of `Hop` structs.
+Each `Hop` has a `url` (String) and a `status` (Integer, or `nil` when the depth limit is
+reached before a request is made).
+
+```ruby
+hops = FinalRedirectUrl.trace('http://hop1.example.com')
+
+hops.each { |hop| puts "#{hop.status}  #{hop.url}" }
+# 301  http://hop1.example.com/
+# 302  http://hop2.example.com/
+# 200  https://final.example.com/
+
+hops.last.url    # => "https://final.example.com/"
+hops.last.status # => 200
+```
+
+`trace` accepts the same options as `final_redirect_url`.
+
+## Meta-refresh support
+
+When the gem reaches a `200 OK` response whose body contains a
+`<meta http-equiv="refresh" content="N; url=...">` tag, it follows that URL as if it were an
+HTTP redirect. Case-insensitive attribute matching and quoted URL values are both handled.
+
+```ruby
+# Page at http://meta.example.com/ contains:
+#   <meta http-equiv="refresh" content="0; url=http://final.example.com/">
+FinalRedirectUrl.final_redirect_url('http://meta.example.com/')
+# => "http://final.example.com/"
+```
+
+## Cookie-gated redirects
+
+`Set-Cookie` headers from every `3xx` response are captured and re-sent as `Cookie` headers on
+all subsequent requests in the same chain, allowing the gem to follow redirects that require an
+active session cookie.
+
 ## Limitations
 
-- **JavaScript / AMP redirects** (e.g. some Google News AMP URLs) are not followed. These require a
-  headless browser. The gem returns the last HTTP-reachable URL in such cases.
-- Redirects are followed using HTTP `HEAD` requests. Servers that reject `HEAD` with `405 Method Not Allowed`
-  may not resolve correctly.
+- **JavaScript / AMP redirects** (e.g. some Google News AMP URLs) require a headless browser
+  and are outside the scope of this gem. The gem returns the last HTTP-reachable URL in such
+  cases.
+- Servers that reject `HEAD` requests with `405 Method Not Allowed` are handled correctly
+  because the gem uses `GET` for all requests.
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at <https://github.com/indyarocks/final_redirect_url>.
-This project is intended to be a safe, welcoming space for collaboration, and contributors are expected
-to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at
+<https://github.com/indyarocks/final_redirect_url>. Contributors are expected to adhere to the
+[Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+Available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/final_redirect_url.gemspec

@@ -9,8 +9,9 @@ Gem::Specification.new do |spec|
   spec.email         = ['chandan.jhun@gmail.com']
 
   spec.summary       = 'Get the final redirect URL for any given URL.'
-  spec.description   = 'Follows HTTP redirects (including HTTPS, relative paths, and ' \
-                       'non-standard ports) to resolve the final destination URL.'
+  spec.description   = 'Follows HTTP/HTTPS redirects (including meta-refresh, cookie-gated ' \
+                       'redirects, relative paths, and non-standard ports) to resolve the ' \
+                       'final destination URL. Provides a trace method for full hop inspection.'
   spec.homepage      = 'https://github.com/indyarocks/final_redirect_url'
   spec.license       = 'MIT'
 
@@ -27,6 +28,8 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
+  spec.add_dependency 'nokogiri', '~> 1.0'
+
   spec.add_development_dependency 'bundler', '~> 2.0'
   spec.add_development_dependency 'rake',    '~> 13.0'
   spec.add_development_dependency 'rspec',   '~> 3.0'

data/lib/final_redirect_url/version.rb

@@ -1,3 +1,3 @@
 module FinalRedirectUrl
-  VERSION = '2.0.0'
+  VERSION = '2.1.0'
 end

data/lib/final_redirect_url.rb

@@ -3,88 +3,164 @@ require 'net/http'
 require 'net/https'
 require 'uri'
 require 'logger'
+require 'nokogiri'
 
 module FinalRedirectUrl
-  DEFAULT_DEPTH = 10
+  DEFAULT_DEPTH        = 10
+  DEFAULT_OPEN_TIMEOUT = 60
+  DEFAULT_READ_TIMEOUT = 60
   USER_AGENT = "FinalRedirectUrl/#{VERSION} (+https://github.com/indyarocks/final_redirect_url)".freeze
 
+  # Value object representing one step in a redirect chain.
+  #   hop.url    # => "http://example.com/"  (String)
+  #   hop.status # => 301                    (Integer, or nil when depth limit reached)
+  Hop = Struct.new(:url, :status, keyword_init: true)
+
   class << self
     # Resolves all HTTP/HTTPS redirects for the given URL and returns the final
-    # destination as a string. Returns an empty string for invalid input or on
-    # unrecoverable network errors.
+    # destination as a String. Returns "" for invalid input or unrecoverable errors.
     #
-    # @param url [String] the starting URL (must include http:// or https://)
+    # @param url     [String]  starting URL (must include http:// or https://)
     # @param options [Hash]
-    # @option options [Integer] :depth maximum redirect hops (default: 10)
+    # @option options [Integer] :depth        max redirect hops          (default: 10)
+    # @option options [Integer] :open_timeout TCP connect timeout in sec  (default: 60)
+    # @option options [Integer] :read_timeout read timeout in sec         (default: 60)
+    # @option options [Boolean] :verify_ssl   enforce SSL cert validation (default: true)
+    # @option options [Hash]    :headers      extra HTTP request headers
     # @return [String]
     def final_redirect_url(url, options = {})
-      return '' unless valid_url?(url)
+      hops = trace(url, options)
+      hops.empty? ? '' : hops.last.url
+    end
+
+    # Follows all redirects and returns every hop as an Array of Hop structs.
+    # Useful for debugging, link audits, or building redirect-aware tooling.
+    #
+    # @param url     [String]
+    # @param options [Hash]  same keys as #final_redirect_url
+    # @return [Array<Hop>]
+    def trace(url, options = {})
+      return [] unless valid_url?(url)
 
-      limit = options.fetch(:depth, DEFAULT_DEPTH).to_i
-      limit = DEFAULT_DEPTH unless limit.positive?
+      limit   = positive_int(options[:depth], DEFAULT_DEPTH)
+      hops    = []
+      cookies = {}
 
-      resolve_redirects(url, limit).to_s
+      resolve(url, limit, options, hops, cookies)
+      hops
     rescue StandardError => ex
       Logger.new($stdout).error(ex.message)
-      ''
+      hops
     end
 
     private
 
-    # Returns true only for syntactically valid HTTP/HTTPS URLs.
-    # Fixes issue #4: private via class << self so the method is genuinely
-    # inaccessible from outside the module.
-    def valid_url?(url)
-      uri = URI.parse(url.to_s)
-      uri.is_a?(URI::HTTP)
-    rescue URI::InvalidURIError
-      false
-    end
-
-    # Recursively follows redirects up to +limit+ hops.
-    # Fixes issue #2: when the hop limit is reached, returns the current URI
-    # (a proper URI object, not a bare string) so url_string_from_uri never
-    # receives a plain String.
-    # Fixes issue #6: uses URI.join to build absolute redirect targets, which
-    # preserves non-standard ports from the base URI (e.g. localhost:3000).
-    def resolve_redirects(url, limit)
+    def resolve(url, limit, options, hops, cookies)
       uri = URI.parse(url)
-      return uri if limit <= 0
 
-      response = fetch_response(uri)
+      if limit <= 0
+        # Depth exhausted — record the pending URL without making a request.
+        hops << Hop.new(url: uri.to_s, status: nil)
+        return
+      end
+
+      response = fetch_response(uri, options, cookies)
+      collect_cookies(response, cookies)
 
       case response
       when Net::HTTPOK
-        uri
+        if (meta_url = meta_refresh_url(response, uri))
+          hops << Hop.new(url: uri.to_s, status: response.code.to_i)
+          warn "meta-refresh to #{meta_url}"
+          resolve(meta_url, limit - 1, options, hops, cookies)
+        else
+          hops << Hop.new(url: uri.to_s, status: response.code.to_i)
+        end
       when Net::HTTPRedirection
-        location = response['location']
+        location = absolute_url(uri, response['location'])
+        hops << Hop.new(url: uri.to_s, status: response.code.to_i)
         warn "redirected to #{location}"
-        resolve_redirects(absolute_url(uri, location), limit - 1)
+        resolve(location, limit - 1, options, hops, cookies)
       else
-        # Non-redirect, non-200 response (e.g. 404, 403, 500): return the
-        # current URI rather than trying to follow a missing Location header.
-        uri
+        # Non-200, non-redirect (403, 404, 500 …): record and stop.
+        hops << Hop.new(url: uri.to_s, status: response.code.to_i)
       end
     end
 
-    # Opens an HTTP/HTTPS connection and fires a HEAD request.
-    # HEAD is sufficient for redirect resolution and avoids downloading bodies.
-    def fetch_response(uri)
-      http = Net::HTTP.new(uri.host, uri.port)
+    def fetch_response(uri, options, cookies)
+      http             = Net::HTTP.new(uri.host, uri.port)
       http.use_ssl     = (uri.scheme == 'https')
-      http.open_timeout = 10
-      http.read_timeout = 10
+      http.verify_mode = options.fetch(:verify_ssl, true) ? OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE
+      http.open_timeout = positive_int(options[:open_timeout], DEFAULT_OPEN_TIMEOUT)
+      http.read_timeout = positive_int(options[:read_timeout], DEFAULT_READ_TIMEOUT)
 
-      request = Net::HTTP::Head.new(uri.request_uri)
+      # GET is required so we can read the body for meta-refresh detection.
+      request = Net::HTTP::Get.new(uri.request_uri)
       request['User-Agent'] = USER_AGENT
 
+      # Caller-supplied headers override defaults (including User-Agent).
+      (options[:headers] || {}).each { |k, v| request[k.to_s] = v }
+
+      # Inject accumulated cookies for cookie-gated redirect chains.
+      request['Cookie'] = cookies.map { |k, v| "#{k}=#{v}" }.join('; ') unless cookies.empty?
+
       http.request(request)
     end
 
-    # Converts a potentially relative +location+ into an absolute URL,
-    # keeping the scheme, host, and port of +base_uri+.
+    # Harvests Set-Cookie values from a response into the shared cookie jar.
+    def collect_cookies(response, cookies)
+      Array(response.get_fields('set-cookie')).each do |cookie_str|
+        name, value = cookie_str.split(';', 2).first.to_s.split('=', 2)
+        cookies[name.to_s.strip] = value.to_s.strip
+      end
+    end
+
+    # Parses <meta http-equiv="refresh"> from an HTML body and returns the
+    # resolved absolute URL, or nil when no meta-refresh is present.
+    # Inspired by the counterfind fork; rewritten for correctness and safety.
+    def meta_refresh_url(response, base_uri)
+      content_type = response['content-type'].to_s
+      return nil unless content_type.empty? || content_type.include?('html')
+      return nil if response.body.to_s.empty?
+
+      doc  = Nokogiri::HTML.parse(response.body)
+      meta = doc.at_xpath(
+        "//meta[translate(@http-equiv," \
+        "'ABCDEFGHIJKLMNOPQRSTUVWXYZ','abcdefghijklmnopqrstuvwxyz')='refresh']/@content"
+      )
+      return nil unless meta
+
+      # Content examples:
+      #   "0; url=https://final.example.com/"
+      #   "5; URL='https://final.example.com/'"
+      #   "0; https://final.example.com/"  (some malformed pages omit "url=")
+      raw      = meta.value.to_s
+      url_part = raw.split(/;\s*/i, 2).last.to_s
+                    .sub(/\Aurl\s*=\s*/i, '')
+                    .gsub(/['"]/, '')
+                    .strip
+      return nil if url_part.empty?
+
+      absolute_url(base_uri, url_part)
+    rescue StandardError
+      nil
+    end
+
+    def valid_url?(url)
+      uri = URI.parse(url.to_s)
+      uri.is_a?(URI::HTTP)
+    rescue URI::InvalidURIError
+      false
+    end
+
     def absolute_url(base_uri, location)
       URI.join(base_uri.to_s, location).to_s
     end
+
+    def positive_int(value, default)
+      return default if value.nil?
+      v = value.to_i
+      v.positive? ? v : default
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: final_redirect_url
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.0
 platform: ruby
 authors:
 - Chandan Jhunjhunwal
@@ -10,6 +10,20 @@ bindir: exe
 cert_chain: []
 date: 2026-11-05 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -66,8 +80,9 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.0'
-description: Follows HTTP redirects (including HTTPS, relative paths, and non-standard
-  ports) to resolve the final destination URL.
+description: Follows HTTP/HTTPS redirects (including meta-refresh, cookie-gated redirects,
+  relative paths, and non-standard ports) to resolve the final destination URL. Provides
+  a trace method for full hop inspection.
 email:
 - chandan.jhun@gmail.com
 executables: []