final_redirect_url 0.1.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.github/workflows/ci.yml +32 -0
- data/.gitignore +1 -0
- data/.ruby-version +1 -0
- data/CHANGELOG.md +111 -0
- data/README.md +107 -16
- data/final_redirect_url.gemspec +24 -24
- data/lib/final_redirect_url/version.rb +1 -1
- data/lib/final_redirect_url.rb +150 -34
- metadata +50 -18
- data/.travis.yml +0 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 7c9fa9551dd7959832dfebc8b24e0cf5c5b326100385a2bf287dab1e50b98581
|
|
4
|
+
data.tar.gz: 2ebc2f9667d598f693fbf766c9eb55991477281cc8fff327ef8d2b693570e0f6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 54a6f3c70af375bf6743eb59f2c4441e4e0d072db4cd654ca94001daa0994638c896dcb2940a36f969004fd42e2f0675c2a278c64535ea6225e33e9a529ac237
|
|
7
|
+
data.tar.gz: 87c4b58fd877c878b109ad2f5ade9f802710e038cab50e9cf1c47ab89ef420c18cbe49570044313a26831f293176fb228df98a7fa78052c68da534f91b666b9e
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [master, main, "feature/**"]
|
|
6
|
+
pull_request:
|
|
7
|
+
branches: [master, main]
|
|
8
|
+
|
|
9
|
+
jobs:
|
|
10
|
+
test:
|
|
11
|
+
name: Ruby ${{ matrix.ruby }}
|
|
12
|
+
runs-on: ubuntu-latest
|
|
13
|
+
strategy:
|
|
14
|
+
fail-fast: false
|
|
15
|
+
matrix:
|
|
16
|
+
ruby:
|
|
17
|
+
- "3.0"
|
|
18
|
+
- "3.1"
|
|
19
|
+
- "3.2"
|
|
20
|
+
- "3.3"
|
|
21
|
+
|
|
22
|
+
steps:
|
|
23
|
+
- uses: actions/checkout@v4
|
|
24
|
+
|
|
25
|
+
- name: Set up Ruby ${{ matrix.ruby }}
|
|
26
|
+
uses: ruby/setup-ruby@v1
|
|
27
|
+
with:
|
|
28
|
+
ruby-version: ${{ matrix.ruby }}
|
|
29
|
+
bundler-cache: true
|
|
30
|
+
|
|
31
|
+
- name: Run tests
|
|
32
|
+
run: bundle exec rspec
|
data/.gitignore
CHANGED
data/.ruby-version
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.3.1
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project
|
|
6
|
+
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## [2.1.0] – 2026-11-05
|
|
11
|
+
|
|
12
|
+
### Fork Contributions Integrated
|
|
13
|
+
Analysed all six public forks. Two contained work worth considering:
|
|
14
|
+
|
|
15
|
+
- **counterfind/final_redirect_url** (9 commits ahead): introduced Nokogiri meta-refresh parsing,
|
|
16
|
+
per-request timeouts, and HTTPS via `use_ssl`. The meta-refresh concept was adopted and
|
|
17
|
+
rewritten from scratch (removed `pry`, removed `KNOWN_SHORTENERS`, eliminated the
|
|
18
|
+
thread-unsafe module-level `@final_url`, fixed the deprecated `URI.regexp` call, replaced
|
|
19
|
+
`rescue Exception`, and switched `uri.host` → `uri.to_s` to preserve ports). Architecture
|
|
20
|
+
unchanged.
|
|
21
|
+
- **HowardSnowden/final_redirect_url** (4 commits ahead): extracted `get_response` as a named
|
|
22
|
+
method and added explicit SSL cipher configuration. The pattern was incorporated into the
|
|
23
|
+
new `fetch_response` method; the hardcoded cipher list was not adopted (unnecessary for
|
|
24
|
+
modern Ruby/OpenSSL).
|
|
25
|
+
- **pilot34, mbv, ymow**: trivial (limit logging, version bump, no changes) — discarded.
|
|
26
|
+
|
|
27
|
+
### Added
|
|
28
|
+
- **`FinalRedirectUrl.trace(url, options)`** — returns `Array<Hop>` documenting every URL and
|
|
29
|
+
HTTP status code visited in the redirect chain. Useful for debugging and link audits.
|
|
30
|
+
- **`FinalRedirectUrl::Hop`** struct with `url` (String) and `status` (Integer, nil when depth
|
|
31
|
+
exhausted) attributes.
|
|
32
|
+
- **Custom headers** (`options[:headers]`) — pass any HTTP headers (e.g. `User-Agent`, `Accept`,
|
|
33
|
+
`Authorization`) to bypass anti-bot blocks or authenticate requests.
|
|
34
|
+
- **Configurable timeouts** — `options[:open_timeout]` and `options[:read_timeout]` default to
|
|
35
|
+
**60 seconds** (up from the previous 10 s hard-coded value).
|
|
36
|
+
- **`verify_ssl` option** (`true` by default) — set `verify_ssl: false` to bypass strict SSL
|
|
37
|
+
certificate validation (e.g. for self-signed certificates in development).
|
|
38
|
+
- **Stateful cookie handling** — `Set-Cookie` headers from every 3xx response are captured and
|
|
39
|
+
injected as `Cookie` headers into subsequent requests, enabling cookie-gated redirect chains
|
|
40
|
+
to be followed correctly.
|
|
41
|
+
- **Meta-refresh parsing** (via Nokogiri, adapted from the `counterfind` fork) — when a 200 OK
|
|
42
|
+
response contains `<meta http-equiv="refresh">`, the gem follows the redirect URL. Handles
|
|
43
|
+
case-insensitive `HTTP-EQUIV`, quoted URLs, and malformed `content` attributes. Only fires
|
|
44
|
+
for `text/html` responses.
|
|
45
|
+
- **nokogiri** added as a runtime dependency.
|
|
46
|
+
|
|
47
|
+
### Changed
|
|
48
|
+
- Switched from `HEAD` to `GET` requests. `GET` is required so that the response body is
|
|
49
|
+
available for meta-refresh parsing; it also improves compatibility with servers that reject
|
|
50
|
+
`HEAD` with 405.
|
|
51
|
+
- `final_redirect_url` now delegates to `trace` internally, eliminating duplicated logic.
|
|
52
|
+
- Default timeouts raised from 10 s to **60 s**.
|
|
53
|
+
|
|
54
|
+
### No Open Issues
|
|
55
|
+
All issues were resolved in v2.0.0. No new open issues exist at the time of this release.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## [2.0.0] – 2026-11-05
|
|
60
|
+
|
|
61
|
+
### Breaking Changes
|
|
62
|
+
- **Minimum Ruby version raised to 3.0.** Ruby 2.x is no longer supported.
|
|
63
|
+
- Private helper methods (`valid_url?`, `resolve_redirects`, `fetch_response`, `absolute_url`) are now
|
|
64
|
+
truly private (defined inside `class << self` with `private`). Calling them from outside the module
|
|
65
|
+
now raises `NoMethodError`. This was always the intended behaviour; previous versions accidentally
|
|
66
|
+
exposed them as public singleton methods (fixes [#4][i4]).
|
|
67
|
+
|
|
68
|
+
### Added
|
|
69
|
+
- **GitHub Actions CI** replacing the removed Travis CI configuration. The matrix tests Ruby 3.0,
|
|
70
|
+
3.1, 3.2, and 3.3 on every push and pull request.
|
|
71
|
+
- **WebMock** test dependency — the full test suite now runs without any real network connections.
|
|
72
|
+
- `CHANGELOG.md` (this file).
|
|
73
|
+
- Gem metadata fields (`homepage_uri`, `source_code_uri`, `changelog_uri`).
|
|
74
|
+
|
|
75
|
+
### Fixed
|
|
76
|
+
- **Port numbers are now preserved** in final URLs. For example,
|
|
77
|
+
`final_redirect_url("http://localhost:3000/path")` now correctly returns
|
|
78
|
+
`"http://localhost:3000/path"` instead of `"http://localhost/path"` (fixes [#6][i6]).
|
|
79
|
+
- **Redirect depth limit now works correctly.** When the hop limit was reached the gem previously
|
|
80
|
+
returned a bare `String` to `url_string_from_uri`, causing a `NoMethodError`. The limit guard now
|
|
81
|
+
returns the current `URI` object (fixes [#2][i2]).
|
|
82
|
+
- **Relative-path redirects now carry forward the original host's port.** The gem uses `URI.join`
|
|
83
|
+
instead of manual string concatenation, so a redirect `Location: /new-path` from
|
|
84
|
+
`http://localhost:3000/old-path` resolves to `http://localhost:3000/new-path`.
|
|
85
|
+
- **Non-redirect, non-200 responses** (403, 404, 500 …) are now handled gracefully: the gem returns
|
|
86
|
+
the last reachable URL rather than crashing with a `TypeError` on a nil `Location` header.
|
|
87
|
+
- **HTTPS support** — connections now set `use_ssl = true` when the scheme is `https`. Previously
|
|
88
|
+
only plain-HTTP URLs worked.
|
|
89
|
+
- **`URI::regexp` removed** — the old `is_valid_url?` used the deprecated (and Ruby-3.x-removed)
|
|
90
|
+
`URI::regexp`. Replaced with `URI.parse` + `URI::HTTP` class check.
|
|
91
|
+
- **`rescue Exception`** replaced with `rescue StandardError` to avoid masking signals and other
|
|
92
|
+
non-recoverable errors.
|
|
93
|
+
- **`$stdout`** used instead of bare `STDOUT` constant (style/frozen-string consistency).
|
|
94
|
+
|
|
95
|
+
### Changed
|
|
96
|
+
- Switched from `Net::HTTP.get_response` (GET) to explicit `Net::HTTP::Head` requests. HEAD is
|
|
97
|
+
sufficient for redirect resolution and avoids downloading response bodies.
|
|
98
|
+
- Added `open_timeout` and `read_timeout` (10 s each) to prevent indefinite hangs.
|
|
99
|
+
- Added a `User-Agent` header so servers that require it don't return unexpected responses.
|
|
100
|
+
|
|
101
|
+
### Known Limitations
|
|
102
|
+
- JavaScript- and AMP-based redirects (e.g. certain Google News AMP URLs — [#3][i3]) require a
|
|
103
|
+
headless browser and are outside the scope of this gem. The gem returns the last HTTP-level
|
|
104
|
+
reachable URL in such cases.
|
|
105
|
+
|
|
106
|
+
[2.1.0]: https://github.com/indyarocks/final_redirect_url/compare/v2.0.0...v2.1.0
|
|
107
|
+
[2.0.0]: https://github.com/indyarocks/final_redirect_url/compare/v0.1.1...v2.0.0
|
|
108
|
+
[i2]: https://github.com/indyarocks/final_redirect_url/issues/2
|
|
109
|
+
[i3]: https://github.com/indyarocks/final_redirect_url/issues/3
|
|
110
|
+
[i4]: https://github.com/indyarocks/final_redirect_url/issues/4
|
|
111
|
+
[i6]: https://github.com/indyarocks/final_redirect_url/issues/6
|
data/README.md
CHANGED
|
@@ -1,39 +1,130 @@
|
|
|
1
1
|
# FinalRedirectUrl
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://github.com/indyarocks/final_redirect_url/actions/workflows/ci.yml)
|
|
4
|
+
[](https://badge.fury.io/rb/final_redirect_url)
|
|
4
5
|
|
|
5
|
-
|
|
6
|
+
Ruby gem that follows HTTP/HTTPS redirects — including `<meta http-equiv="refresh">` and
|
|
7
|
+
cookie-gated chains — and returns the final destination URL.
|
|
8
|
+
|
|
9
|
+
## Requirements
|
|
6
10
|
|
|
7
|
-
|
|
11
|
+
- Ruby >= 3.0
|
|
12
|
+
- [Nokogiri](https://nokogiri.org) (bundled as a runtime dependency for meta-refresh parsing)
|
|
13
|
+
|
|
14
|
+
## Installation
|
|
8
15
|
|
|
9
16
|
```ruby
|
|
10
17
|
gem 'final_redirect_url'
|
|
11
18
|
```
|
|
12
19
|
|
|
13
|
-
|
|
20
|
+
```
|
|
21
|
+
$ bundle install
|
|
22
|
+
```
|
|
14
23
|
|
|
15
|
-
|
|
24
|
+
or
|
|
16
25
|
|
|
17
|
-
|
|
26
|
+
```
|
|
27
|
+
$ gem install final_redirect_url
|
|
28
|
+
```
|
|
18
29
|
|
|
19
|
-
|
|
30
|
+
## Basic usage
|
|
20
31
|
|
|
21
|
-
|
|
32
|
+
```ruby
|
|
33
|
+
require 'final_redirect_url'
|
|
22
34
|
|
|
23
|
-
|
|
35
|
+
FinalRedirectUrl.final_redirect_url('http://google.com')
|
|
36
|
+
# => "https://www.google.com/"
|
|
24
37
|
|
|
25
|
-
|
|
38
|
+
# Invalid / missing protocol → empty string
|
|
39
|
+
FinalRedirectUrl.final_redirect_url('www.example.com')
|
|
40
|
+
# => ""
|
|
41
|
+
```
|
|
26
42
|
|
|
27
|
-
|
|
43
|
+
## Options
|
|
28
44
|
|
|
29
|
-
|
|
45
|
+
All options are passed as a second argument hash:
|
|
30
46
|
|
|
31
|
-
|
|
47
|
+
| Option | Type | Default | Description |
|
|
48
|
+
|----------------|---------|---------|-------------|
|
|
49
|
+
| `:depth` | Integer | `10` | Maximum redirect hops to follow |
|
|
50
|
+
| `:open_timeout` | Integer | `60` | TCP connect timeout in seconds |
|
|
51
|
+
| `:read_timeout` | Integer | `60` | Read timeout in seconds |
|
|
52
|
+
| `:verify_ssl` | Boolean | `true` | Enforce SSL certificate validation |
|
|
53
|
+
| `:headers` | Hash | `{}` | Extra HTTP headers sent with every request |
|
|
54
|
+
|
|
55
|
+
```ruby
|
|
56
|
+
# Limit redirect depth
|
|
57
|
+
FinalRedirectUrl.final_redirect_url('http://example.com', depth: 5)
|
|
58
|
+
|
|
59
|
+
# Custom timeouts (seconds)
|
|
60
|
+
FinalRedirectUrl.final_redirect_url('http://slow.example.com', open_timeout: 5, read_timeout: 30)
|
|
61
|
+
|
|
62
|
+
# Bypass SSL certificate verification (e.g. self-signed certs in development)
|
|
63
|
+
FinalRedirectUrl.final_redirect_url('https://self-signed.local', verify_ssl: false)
|
|
64
|
+
|
|
65
|
+
# Custom headers — override User-Agent or add Authorization / Accept
|
|
66
|
+
FinalRedirectUrl.final_redirect_url(
|
|
67
|
+
'http://bot-protected.example.com',
|
|
68
|
+
headers: {
|
|
69
|
+
'User-Agent' => 'Mozilla/5.0 (compatible; MyApp/1.0)',
|
|
70
|
+
'Accept' => 'text/html'
|
|
71
|
+
}
|
|
72
|
+
)
|
|
73
|
+
```
|
|
32
74
|
|
|
33
|
-
|
|
75
|
+
## Redirect trace
|
|
34
76
|
|
|
77
|
+
`FinalRedirectUrl.trace` returns every hop in the chain as an array of `Hop` structs.
|
|
78
|
+
Each `Hop` has a `url` (String) and a `status` (Integer, or `nil` when the depth limit is
|
|
79
|
+
reached before a request is made).
|
|
35
80
|
|
|
36
|
-
|
|
81
|
+
```ruby
|
|
82
|
+
hops = FinalRedirectUrl.trace('http://hop1.example.com')
|
|
83
|
+
|
|
84
|
+
hops.each { |hop| puts "#{hop.status} #{hop.url}" }
|
|
85
|
+
# 301 http://hop1.example.com/
|
|
86
|
+
# 302 http://hop2.example.com/
|
|
87
|
+
# 200 https://final.example.com/
|
|
37
88
|
|
|
38
|
-
|
|
89
|
+
hops.last.url # => "https://final.example.com/"
|
|
90
|
+
hops.last.status # => 200
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
`trace` accepts the same options as `final_redirect_url`.
|
|
94
|
+
|
|
95
|
+
## Meta-refresh support
|
|
96
|
+
|
|
97
|
+
When the gem reaches a `200 OK` response whose body contains a
|
|
98
|
+
`<meta http-equiv="refresh" content="N; url=...">` tag, it follows that URL as if it were an
|
|
99
|
+
HTTP redirect. Case-insensitive attribute matching and quoted URL values are both handled.
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
# Page at http://meta.example.com/ contains:
|
|
103
|
+
# <meta http-equiv="refresh" content="0; url=http://final.example.com/">
|
|
104
|
+
FinalRedirectUrl.final_redirect_url('http://meta.example.com/')
|
|
105
|
+
# => "http://final.example.com/"
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
## Cookie-gated redirects
|
|
109
|
+
|
|
110
|
+
`Set-Cookie` headers from every `3xx` response are captured and re-sent as `Cookie` headers on
|
|
111
|
+
all subsequent requests in the same chain, allowing the gem to follow redirects that require an
|
|
112
|
+
active session cookie.
|
|
113
|
+
|
|
114
|
+
## Limitations
|
|
115
|
+
|
|
116
|
+
- **JavaScript / AMP redirects** (e.g. some Google News AMP URLs) require a headless browser
|
|
117
|
+
and are outside the scope of this gem. The gem returns the last HTTP-reachable URL in such
|
|
118
|
+
cases.
|
|
119
|
+
- Servers that reject `HEAD` requests with `405 Method Not Allowed` are handled correctly
|
|
120
|
+
because the gem uses `GET` for all requests.
|
|
121
|
+
|
|
122
|
+
## Contributing
|
|
123
|
+
|
|
124
|
+
Bug reports and pull requests are welcome on GitHub at
|
|
125
|
+
<https://github.com/indyarocks/final_redirect_url>. Contributors are expected to adhere to the
|
|
126
|
+
[Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
|
127
|
+
|
|
128
|
+
## License
|
|
39
129
|
|
|
130
|
+
Available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|
data/final_redirect_url.gemspec
CHANGED
|
@@ -1,37 +1,37 @@
|
|
|
1
|
-
|
|
2
|
-
lib = File.expand_path('../lib', __FILE__)
|
|
1
|
+
lib = File.expand_path('lib', __dir__)
|
|
3
2
|
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
3
|
require 'final_redirect_url/version'
|
|
5
4
|
|
|
6
5
|
Gem::Specification.new do |spec|
|
|
7
|
-
spec.name =
|
|
6
|
+
spec.name = 'final_redirect_url'
|
|
8
7
|
spec.version = FinalRedirectUrl::VERSION
|
|
9
|
-
spec.authors = [
|
|
10
|
-
spec.email = [
|
|
8
|
+
spec.authors = ['Chandan Jhunjhunwal']
|
|
9
|
+
spec.email = ['chandan.jhun@gmail.com']
|
|
11
10
|
|
|
12
|
-
spec.summary =
|
|
13
|
-
spec.description =
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
spec.
|
|
11
|
+
spec.summary = 'Get the final redirect URL for any given URL.'
|
|
12
|
+
spec.description = 'Follows HTTP/HTTPS redirects (including meta-refresh, cookie-gated ' \
|
|
13
|
+
'redirects, relative paths, and non-standard ports) to resolve the ' \
|
|
14
|
+
'final destination URL. Provides a trace method for full hop inspection.'
|
|
15
|
+
spec.homepage = 'https://github.com/indyarocks/final_redirect_url'
|
|
16
|
+
spec.license = 'MIT'
|
|
17
17
|
|
|
18
|
-
|
|
19
|
-
# to allow pushing to a single host or delete this section to allow pushing to any host.
|
|
20
|
-
# if spec.respond_to?(:metadata)
|
|
21
|
-
# spec.metadata['allowed_push_host'] = "TODO: Set to 'http://mygemserver.com'"
|
|
22
|
-
# else
|
|
23
|
-
# raise "RubyGems 2.0 or newer is required to protect against " \
|
|
24
|
-
# "public gem pushes."
|
|
25
|
-
# end
|
|
18
|
+
spec.required_ruby_version = '>= 3.0'
|
|
26
19
|
|
|
27
|
-
spec.
|
|
20
|
+
spec.metadata['homepage_uri'] = spec.homepage
|
|
21
|
+
spec.metadata['source_code_uri'] = spec.homepage
|
|
22
|
+
spec.metadata['changelog_uri'] = "#{spec.homepage}/blob/master/CHANGELOG.md"
|
|
23
|
+
|
|
24
|
+
spec.files = `git ls-files -z`.split("\x0").reject do |f|
|
|
28
25
|
f.match(%r{^(test|spec|features)/})
|
|
29
26
|
end
|
|
30
|
-
spec.bindir =
|
|
27
|
+
spec.bindir = 'exe'
|
|
31
28
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
|
32
|
-
spec.require_paths = [
|
|
29
|
+
spec.require_paths = ['lib']
|
|
30
|
+
|
|
31
|
+
spec.add_dependency 'nokogiri', '~> 1.0'
|
|
33
32
|
|
|
34
|
-
spec.add_development_dependency
|
|
35
|
-
spec.add_development_dependency
|
|
36
|
-
spec.add_development_dependency
|
|
33
|
+
spec.add_development_dependency 'bundler', '~> 2.0'
|
|
34
|
+
spec.add_development_dependency 'rake', '~> 13.0'
|
|
35
|
+
spec.add_development_dependency 'rspec', '~> 3.0'
|
|
36
|
+
spec.add_development_dependency 'webmock', '~> 3.0'
|
|
37
37
|
end
|
data/lib/final_redirect_url.rb
CHANGED
|
@@ -1,50 +1,166 @@
|
|
|
1
1
|
require 'final_redirect_url/version'
|
|
2
2
|
require 'net/http'
|
|
3
|
+
require 'net/https'
|
|
4
|
+
require 'uri'
|
|
3
5
|
require 'logger'
|
|
6
|
+
require 'nokogiri'
|
|
4
7
|
|
|
5
8
|
module FinalRedirectUrl
|
|
9
|
+
DEFAULT_DEPTH = 10
|
|
10
|
+
DEFAULT_OPEN_TIMEOUT = 60
|
|
11
|
+
DEFAULT_READ_TIMEOUT = 60
|
|
12
|
+
USER_AGENT = "FinalRedirectUrl/#{VERSION} (+https://github.com/indyarocks/final_redirect_url)".freeze
|
|
6
13
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
14
|
+
# Value object representing one step in a redirect chain.
|
|
15
|
+
# hop.url # => "http://example.com/" (String)
|
|
16
|
+
# hop.status # => 301 (Integer, or nil when depth limit reached)
|
|
17
|
+
Hop = Struct.new(:url, :status, keyword_init: true)
|
|
18
|
+
|
|
19
|
+
class << self
|
|
20
|
+
# Resolves all HTTP/HTTPS redirects for the given URL and returns the final
|
|
21
|
+
# destination as a String. Returns "" for invalid input or unrecoverable errors.
|
|
22
|
+
#
|
|
23
|
+
# @param url [String] starting URL (must include http:// or https://)
|
|
24
|
+
# @param options [Hash]
|
|
25
|
+
# @option options [Integer] :depth max redirect hops (default: 10)
|
|
26
|
+
# @option options [Integer] :open_timeout TCP connect timeout in sec (default: 60)
|
|
27
|
+
# @option options [Integer] :read_timeout read timeout in sec (default: 60)
|
|
28
|
+
# @option options [Boolean] :verify_ssl enforce SSL cert validation (default: true)
|
|
29
|
+
# @option options [Hash] :headers extra HTTP request headers
|
|
30
|
+
# @return [String]
|
|
31
|
+
def final_redirect_url(url, options = {})
|
|
32
|
+
hops = trace(url, options)
|
|
33
|
+
hops.empty? ? '' : hops.last.url
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# Follows all redirects and returns every hop as an Array of Hop structs.
|
|
37
|
+
# Useful for debugging, link audits, or building redirect-aware tooling.
|
|
38
|
+
#
|
|
39
|
+
# @param url [String]
|
|
40
|
+
# @param options [Hash] same keys as #final_redirect_url
|
|
41
|
+
# @return [Array<Hop>]
|
|
42
|
+
def trace(url, options = {})
|
|
43
|
+
return [] unless valid_url?(url)
|
|
44
|
+
|
|
45
|
+
limit = positive_int(options[:depth], DEFAULT_DEPTH)
|
|
46
|
+
hops = []
|
|
47
|
+
cookies = {}
|
|
48
|
+
|
|
49
|
+
resolve(url, limit, options, hops, cookies)
|
|
50
|
+
hops
|
|
51
|
+
rescue StandardError => ex
|
|
52
|
+
Logger.new($stdout).error(ex.message)
|
|
53
|
+
hops
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
private
|
|
57
|
+
|
|
58
|
+
def resolve(url, limit, options, hops, cookies)
|
|
59
|
+
uri = URI.parse(url)
|
|
60
|
+
|
|
61
|
+
if limit <= 0
|
|
62
|
+
# Depth exhausted — record the pending URL without making a request.
|
|
63
|
+
hops << Hop.new(url: uri.to_s, status: nil)
|
|
64
|
+
return
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
response = fetch_response(uri, options, cookies)
|
|
68
|
+
collect_cookies(response, cookies)
|
|
69
|
+
|
|
70
|
+
case response
|
|
71
|
+
when Net::HTTPOK
|
|
72
|
+
if (meta_url = meta_refresh_url(response, uri))
|
|
73
|
+
hops << Hop.new(url: uri.to_s, status: response.code.to_i)
|
|
74
|
+
warn "meta-refresh to #{meta_url}"
|
|
75
|
+
resolve(meta_url, limit - 1, options, hops, cookies)
|
|
76
|
+
else
|
|
77
|
+
hops << Hop.new(url: uri.to_s, status: response.code.to_i)
|
|
78
|
+
end
|
|
79
|
+
when Net::HTTPRedirection
|
|
80
|
+
location = absolute_url(uri, response['location'])
|
|
81
|
+
hops << Hop.new(url: uri.to_s, status: response.code.to_i)
|
|
82
|
+
warn "redirected to #{location}"
|
|
83
|
+
resolve(location, limit - 1, options, hops, cookies)
|
|
84
|
+
else
|
|
85
|
+
# Non-200, non-redirect (403, 404, 500 …): record and stop.
|
|
86
|
+
hops << Hop.new(url: uri.to_s, status: response.code.to_i)
|
|
17
87
|
end
|
|
18
88
|
end
|
|
19
|
-
final_url
|
|
20
|
-
end
|
|
21
89
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
90
|
+
def fetch_response(uri, options, cookies)
|
|
91
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
92
|
+
http.use_ssl = (uri.scheme == 'https')
|
|
93
|
+
http.verify_mode = options.fetch(:verify_ssl, true) ? OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE
|
|
94
|
+
http.open_timeout = positive_int(options[:open_timeout], DEFAULT_OPEN_TIMEOUT)
|
|
95
|
+
http.read_timeout = positive_int(options[:read_timeout], DEFAULT_READ_TIMEOUT)
|
|
96
|
+
|
|
97
|
+
# GET is required so we can read the body for meta-refresh detection.
|
|
98
|
+
request = Net::HTTP::Get.new(uri.request_uri)
|
|
99
|
+
request['User-Agent'] = USER_AGENT
|
|
26
100
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
101
|
+
# Caller-supplied headers override defaults (including User-Agent).
|
|
102
|
+
(options[:headers] || {}).each { |k, v| request[k.to_s] = v }
|
|
103
|
+
|
|
104
|
+
# Inject accumulated cookies for cookie-gated redirect chains.
|
|
105
|
+
request['Cookie'] = cookies.map { |k, v| "#{k}=#{v}" }.join('; ') unless cookies.empty?
|
|
106
|
+
|
|
107
|
+
http.request(request)
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
# Harvests Set-Cookie values from a response into the shared cookie jar.
|
|
111
|
+
def collect_cookies(response, cookies)
|
|
112
|
+
Array(response.get_fields('set-cookie')).each do |cookie_str|
|
|
113
|
+
name, value = cookie_str.split(';', 2).first.to_s.split('=', 2)
|
|
114
|
+
cookies[name.to_s.strip] = value.to_s.strip
|
|
37
115
|
end
|
|
38
|
-
warn "redirected to #{redirect_location}"
|
|
39
|
-
get_final_redirect_url(redirect_location, limit - 1)
|
|
40
116
|
end
|
|
41
|
-
end
|
|
42
117
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
118
|
+
# Parses <meta http-equiv="refresh"> from an HTML body and returns the
|
|
119
|
+
# resolved absolute URL, or nil when no meta-refresh is present.
|
|
120
|
+
# Inspired by the counterfind fork; rewritten for correctness and safety.
|
|
121
|
+
def meta_refresh_url(response, base_uri)
|
|
122
|
+
content_type = response['content-type'].to_s
|
|
123
|
+
return nil unless content_type.empty? || content_type.include?('html')
|
|
124
|
+
return nil if response.body.to_s.empty?
|
|
125
|
+
|
|
126
|
+
doc = Nokogiri::HTML.parse(response.body)
|
|
127
|
+
meta = doc.at_xpath(
|
|
128
|
+
"//meta[translate(@http-equiv," \
|
|
129
|
+
"'ABCDEFGHIJKLMNOPQRSTUVWXYZ','abcdefghijklmnopqrstuvwxyz')='refresh']/@content"
|
|
130
|
+
)
|
|
131
|
+
return nil unless meta
|
|
132
|
+
|
|
133
|
+
# Content examples:
|
|
134
|
+
# "0; url=https://final.example.com/"
|
|
135
|
+
# "5; URL='https://final.example.com/'"
|
|
136
|
+
# "0; https://final.example.com/" (some malformed pages omit "url=")
|
|
137
|
+
raw = meta.value.to_s
|
|
138
|
+
url_part = raw.split(/;\s*/i, 2).last.to_s
|
|
139
|
+
.sub(/\Aurl\s*=\s*/i, '')
|
|
140
|
+
.gsub(/['"]/, '')
|
|
141
|
+
.strip
|
|
142
|
+
return nil if url_part.empty?
|
|
143
|
+
|
|
144
|
+
absolute_url(base_uri, url_part)
|
|
145
|
+
rescue StandardError
|
|
146
|
+
nil
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
def valid_url?(url)
|
|
150
|
+
uri = URI.parse(url.to_s)
|
|
151
|
+
uri.is_a?(URI::HTTP)
|
|
152
|
+
rescue URI::InvalidURIError
|
|
153
|
+
false
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
def absolute_url(base_uri, location)
|
|
157
|
+
URI.join(base_uri.to_s, location).to_s
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def positive_int(value, default)
|
|
161
|
+
return default if value.nil?
|
|
162
|
+
v = value.to_i
|
|
163
|
+
v.positive? ? v : default
|
|
47
164
|
end
|
|
48
|
-
url_str
|
|
49
165
|
end
|
|
50
166
|
end
|
metadata
CHANGED
|
@@ -1,43 +1,57 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: final_redirect_url
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version:
|
|
4
|
+
version: 2.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Chandan Jhunjhunwal
|
|
8
|
-
autorequire:
|
|
8
|
+
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2026-11-05 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
|
+
- !ruby/object:Gem::Dependency
|
|
14
|
+
name: nokogiri
|
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
|
16
|
+
requirements:
|
|
17
|
+
- - "~>"
|
|
18
|
+
- !ruby/object:Gem::Version
|
|
19
|
+
version: '1.0'
|
|
20
|
+
type: :runtime
|
|
21
|
+
prerelease: false
|
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
23
|
+
requirements:
|
|
24
|
+
- - "~>"
|
|
25
|
+
- !ruby/object:Gem::Version
|
|
26
|
+
version: '1.0'
|
|
13
27
|
- !ruby/object:Gem::Dependency
|
|
14
28
|
name: bundler
|
|
15
29
|
requirement: !ruby/object:Gem::Requirement
|
|
16
30
|
requirements:
|
|
17
31
|
- - "~>"
|
|
18
32
|
- !ruby/object:Gem::Version
|
|
19
|
-
version: '
|
|
33
|
+
version: '2.0'
|
|
20
34
|
type: :development
|
|
21
35
|
prerelease: false
|
|
22
36
|
version_requirements: !ruby/object:Gem::Requirement
|
|
23
37
|
requirements:
|
|
24
38
|
- - "~>"
|
|
25
39
|
- !ruby/object:Gem::Version
|
|
26
|
-
version: '
|
|
40
|
+
version: '2.0'
|
|
27
41
|
- !ruby/object:Gem::Dependency
|
|
28
42
|
name: rake
|
|
29
43
|
requirement: !ruby/object:Gem::Requirement
|
|
30
44
|
requirements:
|
|
31
45
|
- - "~>"
|
|
32
46
|
- !ruby/object:Gem::Version
|
|
33
|
-
version: '
|
|
47
|
+
version: '13.0'
|
|
34
48
|
type: :development
|
|
35
49
|
prerelease: false
|
|
36
50
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
51
|
requirements:
|
|
38
52
|
- - "~>"
|
|
39
53
|
- !ruby/object:Gem::Version
|
|
40
|
-
version: '
|
|
54
|
+
version: '13.0'
|
|
41
55
|
- !ruby/object:Gem::Dependency
|
|
42
56
|
name: rspec
|
|
43
57
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -52,18 +66,34 @@ dependencies:
|
|
|
52
66
|
- - "~>"
|
|
53
67
|
- !ruby/object:Gem::Version
|
|
54
68
|
version: '3.0'
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
69
|
+
- !ruby/object:Gem::Dependency
|
|
70
|
+
name: webmock
|
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
|
72
|
+
requirements:
|
|
73
|
+
- - "~>"
|
|
74
|
+
- !ruby/object:Gem::Version
|
|
75
|
+
version: '3.0'
|
|
76
|
+
type: :development
|
|
77
|
+
prerelease: false
|
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
79
|
+
requirements:
|
|
80
|
+
- - "~>"
|
|
81
|
+
- !ruby/object:Gem::Version
|
|
82
|
+
version: '3.0'
|
|
83
|
+
description: Follows HTTP/HTTPS redirects (including meta-refresh, cookie-gated redirects,
|
|
84
|
+
relative paths, and non-standard ports) to resolve the final destination URL. Provides
|
|
85
|
+
a trace method for full hop inspection.
|
|
58
86
|
email:
|
|
59
87
|
- chandan.jhun@gmail.com
|
|
60
88
|
executables: []
|
|
61
89
|
extensions: []
|
|
62
90
|
extra_rdoc_files: []
|
|
63
91
|
files:
|
|
92
|
+
- ".github/workflows/ci.yml"
|
|
64
93
|
- ".gitignore"
|
|
65
94
|
- ".rspec"
|
|
66
|
-
- ".
|
|
95
|
+
- ".ruby-version"
|
|
96
|
+
- CHANGELOG.md
|
|
67
97
|
- CODE_OF_CONDUCT.md
|
|
68
98
|
- Gemfile
|
|
69
99
|
- LICENSE.txt
|
|
@@ -77,8 +107,11 @@ files:
|
|
|
77
107
|
homepage: https://github.com/indyarocks/final_redirect_url
|
|
78
108
|
licenses:
|
|
79
109
|
- MIT
|
|
80
|
-
metadata:
|
|
81
|
-
|
|
110
|
+
metadata:
|
|
111
|
+
homepage_uri: https://github.com/indyarocks/final_redirect_url
|
|
112
|
+
source_code_uri: https://github.com/indyarocks/final_redirect_url
|
|
113
|
+
changelog_uri: https://github.com/indyarocks/final_redirect_url/blob/master/CHANGELOG.md
|
|
114
|
+
post_install_message:
|
|
82
115
|
rdoc_options: []
|
|
83
116
|
require_paths:
|
|
84
117
|
- lib
|
|
@@ -86,16 +119,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
86
119
|
requirements:
|
|
87
120
|
- - ">="
|
|
88
121
|
- !ruby/object:Gem::Version
|
|
89
|
-
version: '0'
|
|
122
|
+
version: '3.0'
|
|
90
123
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
91
124
|
requirements:
|
|
92
125
|
- - ">="
|
|
93
126
|
- !ruby/object:Gem::Version
|
|
94
127
|
version: '0'
|
|
95
128
|
requirements: []
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
signing_key:
|
|
129
|
+
rubygems_version: 3.5.9
|
|
130
|
+
signing_key:
|
|
99
131
|
specification_version: 4
|
|
100
|
-
summary: Get
|
|
132
|
+
summary: Get the final redirect URL for any given URL.
|
|
101
133
|
test_files: []
|