url_canonicalize 1.0.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +148 -113
- data/README.md +234 -9
- data/SECURITY.md +25 -0
- data/lib/url_canonicalize/bounded_body.rb +23 -0
- data/lib/url_canonicalize/client.rb +21 -0
- data/lib/{monkey_patches/addressable/uri.rb → url_canonicalize/core_ext/addressable.rb} +4 -2
- data/lib/{monkey_patches → url_canonicalize/core_ext}/string.rb +2 -2
- data/lib/{monkey_patches → url_canonicalize/core_ext}/uri.rb +4 -2
- data/lib/url_canonicalize/core_ext.rb +6 -0
- data/lib/url_canonicalize/destination.rb +120 -0
- data/lib/url_canonicalize/exception.rb +3 -0
- data/lib/url_canonicalize/http.rb +86 -63
- data/lib/url_canonicalize/link_header.rb +34 -0
- data/lib/url_canonicalize/media_type.rb +23 -0
- data/lib/url_canonicalize/options.rb +112 -0
- data/lib/url_canonicalize/request.rb +86 -77
- data/lib/url_canonicalize/response.rb +7 -4
- data/lib/url_canonicalize/result.rb +28 -0
- data/lib/url_canonicalize/transport.rb +47 -0
- data/lib/url_canonicalize/uri.rb +26 -13
- data/lib/url_canonicalize/version.rb +1 -1
- data/lib/url_canonicalize.rb +13 -9
- metadata +22 -25
- data/.codeclimate.yml +0 -25
- data/.github/workflows/codeql-analysis.yml +0 -70
- data/.gitignore +0 -53
- data/.hound.yml +0 -4
- data/.rspec +0 -5
- data/.rubocop.yml +0 -61
- data/.ruby-gemset +0 -1
- data/.travis.yml +0 -8
- data/Gemfile +0 -22
- data/Gemfile.lock +0 -110
- data/Guardfile +0 -18
- data/Rakefile +0 -3
- data/circle.yml +0 -29
- data/renovate.json +0 -6
- data/url_canonicalize.gemspec +0 -29
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 503c645a9625985b4391f2743587f148d750d09d43d473c7a9923be2a2ec8c7f
|
|
4
|
+
data.tar.gz: 3983c156ec6754bac592880a968a07af984a5c11109ab31d096376c0774584a2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b85dc8c4af5866b0bd9d79c92181672d3401419587f0d6868fdce28cfc2c1e973c86e208c633d3062b09dcad9e9e4794badff12b157c5e6cf0a3a95a0426f041
|
|
7
|
+
data.tar.gz: 498beaf7ca0e0a56d0f2f9b3280c42b7417ff8a14f38453ae757a480cd78388e5391ed7c3896a596648796226be8e3ee745446a412217a51f36b3025bf2a6295
|
data/CHANGELOG.md
CHANGED
|
@@ -1,117 +1,152 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
All notable changes to this project are documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to
|
|
7
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
8
|
+
|
|
9
|
+
## [Unreleased]
|
|
10
|
+
|
|
11
|
+
## [2.0.0] - 2026-07-25
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- `URLCanonicalize::Client`, a reusable, thread-safe client whose options are
|
|
16
|
+
validated and frozen at construction; request headers, a logger and the
|
|
17
|
+
transport are configurable, so tests can inject a fake transport or
|
|
18
|
+
resolver
|
|
19
|
+
([#186](https://github.com/dominicsayers/url_canonicalize/issues/186))
|
|
20
|
+
- Documented, table-tested URL normalization per RFC 3986: lowercase scheme
|
|
21
|
+
and host, punycode hosts, normalized percent-encoding, resolved dot
|
|
22
|
+
segments, removed default ports and stripped fragments, applied to the
|
|
23
|
+
input, every hop and the returned URL
|
|
24
|
+
([#187](https://github.com/dominicsayers/url_canonicalize/issues/187))
|
|
25
|
+
- SonarQube quality gate badge in the README
|
|
26
|
+
([#196](https://github.com/dominicsayers/url_canonicalize/issues/196))
|
|
27
|
+
|
|
28
|
+
### Changed
|
|
29
|
+
|
|
30
|
+
- **Breaking:** the core-class extensions (`String#canonicalize` and friends)
|
|
31
|
+
are opt-in via `require 'url_canonicalize/core_ext'`; requiring the gem
|
|
32
|
+
alone no longer patches anything
|
|
33
|
+
([#186](https://github.com/dominicsayers/url_canonicalize/issues/186))
|
|
34
|
+
- **Breaking:** `URLCanonicalize.fetch` returns an immutable
|
|
35
|
+
`URLCanonicalize::Result` exposing the canonical URL, the confirming
|
|
36
|
+
response, the parsed HTML, the request chain and the discovery source
|
|
37
|
+
([#186](https://github.com/dominicsayers/url_canonicalize/issues/186))
|
|
38
|
+
- **Breaking:** returned URLs are normalized (an empty path becomes `/`) and
|
|
39
|
+
a scheme is recognized only as a leading `scheme:` token, so a colon later
|
|
40
|
+
in a bare host's path no longer causes a parse error
|
|
41
|
+
([#187](https://github.com/dominicsayers/url_canonicalize/issues/187))
|
|
42
|
+
- **Breaking:** Ruby 3.3 or later is required; CI tests Ruby 3.3, 3.4 and
|
|
43
|
+
4.0 on GitHub Actions and the Travis and CircleCI configurations are gone
|
|
44
|
+
([#188](https://github.com/dominicsayers/url_canonicalize/issues/188))
|
|
45
|
+
- Debug output moved from the `DEBUG` environment variable to the `logger`
|
|
46
|
+
option
|
|
47
|
+
([#186](https://github.com/dominicsayers/url_canonicalize/issues/186))
|
|
48
|
+
- Update nokogiri to 1.19.4, fixing all eleven open Dependabot alerts
|
|
49
|
+
([#210](https://github.com/dominicsayers/url_canonicalize/issues/210))
|
|
50
|
+
- Replace the `bin/release` scripts with the
|
|
51
|
+
[`release_ceremony`](https://github.com/dominicsayers/release_ceremony) gem
|
|
52
|
+
they were extracted into; the release commands are now
|
|
53
|
+
`bundle exec release_ceremony prepare|publish VERSION`
|
|
54
|
+
|
|
55
|
+
### Removed
|
|
56
|
+
|
|
57
|
+
- Hound and CodeClimate configuration; RuboCop in CI and SonarQube Cloud
|
|
58
|
+
cover static analysis
|
|
59
|
+
([#211](https://github.com/dominicsayers/url_canonicalize/issues/211),
|
|
60
|
+
[#195](https://github.com/dominicsayers/url_canonicalize/issues/195))
|
|
61
|
+
- The RuboCop to-do file; its offences are fixed and the deliberate spec
|
|
62
|
+
limits are explicit policy in `.rubocop.yml`
|
|
63
|
+
([#198](https://github.com/dominicsayers/url_canonicalize/issues/198))
|
|
64
|
+
|
|
65
|
+
## [1.0.2] - 2026-07-24
|
|
66
|
+
|
|
67
|
+
### Fixed
|
|
68
|
+
|
|
69
|
+
- Add `rake` to the development bundle so the `rake build` release check can
|
|
70
|
+
run ([#204](https://github.com/dominicsayers/url_canonicalize/pull/204))
|
|
71
|
+
|
|
72
|
+
### Changed
|
|
73
|
+
|
|
74
|
+
- Announce each step of `bin/release/prepare` as it starts so the command
|
|
75
|
+
shows progress instead of appearing to hang
|
|
76
|
+
([#206](https://github.com/dominicsayers/url_canonicalize/pull/206))
|
|
77
|
+
- Update the `ruby/setup-ruby` CI action to v1.321.0
|
|
78
|
+
([#205](https://github.com/dominicsayers/url_canonicalize/pull/205))
|
|
79
|
+
|
|
80
|
+
## [1.0.1] - 2026-07-22
|
|
81
|
+
|
|
82
|
+
### Security
|
|
83
|
+
|
|
84
|
+
- Verify TLS certificates and hostnames, block unsafe destinations at every
|
|
85
|
+
hop, pin DNS results, restrict ports, bound response bodies and enforce an
|
|
86
|
+
overall deadline
|
|
87
|
+
([#184](https://github.com/dominicsayers/url_canonicalize/issues/184))
|
|
88
|
+
|
|
89
|
+
### Fixed
|
|
90
|
+
|
|
91
|
+
- Correct the redirect and canonical-link HTTP state machine: follow all
|
|
92
|
+
redirect responses consistently, bound redirect and canonical-link chains
|
|
93
|
+
with one visited-URL set and one hop budget, parse `Link` headers properly,
|
|
94
|
+
resolve relative references per RFC 3986, match HTML canonical links
|
|
95
|
+
case-insensitively against the document base URL, replace host-specific HEAD
|
|
96
|
+
behavior with a generic HEAD-to-GET fallback and preserve underlying error
|
|
97
|
+
causes ([#185](https://github.com/dominicsayers/url_canonicalize/issues/185))
|
|
98
|
+
|
|
99
|
+
### Changed
|
|
100
|
+
|
|
101
|
+
- Refresh the test and coverage strategy: replace `coveralls-ruby` with a
|
|
102
|
+
current SimpleCov release, enforce 100% line and branch coverage on CI,
|
|
103
|
+
test redirect and canonical-link behavior deterministically at the HTTP
|
|
104
|
+
boundary and isolate environment changes in every example
|
|
105
|
+
([#189](https://github.com/dominicsayers/url_canonicalize/issues/189))
|
|
106
|
+
- Refresh gem packaging and release automation: package a deterministic
|
|
107
|
+
whitelist of runtime files, verify the built gem in CI, add full RubyGems
|
|
108
|
+
metadata, publish through RubyGems Trusted Publishing from a tag-triggered
|
|
109
|
+
workflow, and document security reporting and release procedures
|
|
110
|
+
([#190](https://github.com/dominicsayers/url_canonicalize/issues/190))
|
|
111
|
+
|
|
112
|
+
## [1.0.0] - 2024-01-27
|
|
113
|
+
|
|
114
|
+
### Changed
|
|
115
|
+
|
|
116
|
+
- Require Ruby 3.1 or later
|
|
117
|
+
- Adapt to the behavior of current `uri` gem releases
|
|
118
|
+
- Update development dependencies and CI configuration
|
|
119
|
+
|
|
120
|
+
## [0.2.1] - 2022-02-01
|
|
121
|
+
|
|
122
|
+
### Changed
|
|
123
|
+
|
|
124
|
+
- Require a recent Nokogiri version
|
|
125
|
+
- Update development dependencies, RuboCop configuration and CI configuration
|
|
126
|
+
|
|
127
|
+
## [0.2.0] - 2018-06-30
|
|
128
|
+
|
|
129
|
+
### Added
|
|
130
|
+
|
|
131
|
+
- Return a Nokogiri XML document from a successful response when requested
|
|
132
|
+
([#5](https://github.com/dominicsayers/url_canonicalize/issues/5))
|
|
133
|
+
|
|
134
|
+
### Changed
|
|
135
|
+
|
|
136
|
+
- Require a secure Nokogiri version
|
|
137
|
+
|
|
138
|
+
## 0.1.15 and earlier
|
|
4
139
|
|
|
5
|
-
|
|
140
|
+
Releases from 0.0.1 (2016-10-20) to 0.1.15 (2017-03-14) predate this
|
|
141
|
+
changelog format. Their history is available from the
|
|
142
|
+
[v0.1.15 tag](https://github.com/dominicsayers/url_canonicalize/releases/tag/v0.1.15)
|
|
143
|
+
and the
|
|
144
|
+
[commit log](https://github.com/dominicsayers/url_canonicalize/commits/v0.1.15).
|
|
6
145
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
## [v0.1.15](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.15) (2017-03-14)
|
|
16
|
-
|
|
17
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.14...v0.1.15)
|
|
18
|
-
|
|
19
|
-
## [v0.1.14](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.14) (2017-03-13)
|
|
20
|
-
|
|
21
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.13...v0.1.14)
|
|
22
|
-
|
|
23
|
-
## [v0.1.13](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.13) (2017-03-13)
|
|
24
|
-
|
|
25
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.12...v0.1.13)
|
|
26
|
-
|
|
27
|
-
## [v0.1.12](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.12) (2017-03-13)
|
|
28
|
-
|
|
29
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.11...v0.1.12)
|
|
30
|
-
|
|
31
|
-
## [v0.1.11](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.11) (2017-03-13)
|
|
32
|
-
|
|
33
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.10...v0.1.11)
|
|
34
|
-
|
|
35
|
-
## [v0.1.10](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.10) (2017-03-13)
|
|
36
|
-
|
|
37
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.9...v0.1.10)
|
|
38
|
-
|
|
39
|
-
**Closed issues:**
|
|
40
|
-
|
|
41
|
-
- URLCanonicalize doesn't accept host names [\#4](https://github.com/dominicsayers/url_canonicalize/issues/4)
|
|
42
|
-
|
|
43
|
-
## [v0.1.9](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.9) (2017-03-06)
|
|
44
|
-
|
|
45
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.8...v0.1.9)
|
|
46
|
-
|
|
47
|
-
## [v0.1.8](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.8) (2017-03-06)
|
|
48
|
-
|
|
49
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.7...v0.1.8)
|
|
50
|
-
|
|
51
|
-
**Closed issues:**
|
|
52
|
-
|
|
53
|
-
- Use a later version of Ruby on CircleCI [\#3](https://github.com/dominicsayers/url_canonicalize/issues/3)
|
|
54
|
-
|
|
55
|
-
## [v0.1.7](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.7) (2017-03-06)
|
|
56
|
-
|
|
57
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.6...v0.1.7)
|
|
58
|
-
|
|
59
|
-
**Closed issues:**
|
|
60
|
-
|
|
61
|
-
- Insecure version of Nokogiri [\#2](https://github.com/dominicsayers/url_canonicalize/issues/2)
|
|
62
|
-
|
|
63
|
-
## [v0.1.6](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.6) (2017-03-06)
|
|
64
|
-
|
|
65
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.5...v0.1.6)
|
|
66
|
-
|
|
67
|
-
## [v0.1.5](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.5) (2016-10-26)
|
|
68
|
-
|
|
69
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.4...v0.1.5)
|
|
70
|
-
|
|
71
|
-
## [v0.1.4](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.4) (2016-10-26)
|
|
72
|
-
|
|
73
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.2...v0.1.4)
|
|
74
|
-
|
|
75
|
-
## [v0.1.2](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.2) (2016-10-26)
|
|
76
|
-
|
|
77
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.1...v0.1.2)
|
|
78
|
-
|
|
79
|
-
## [v0.1.1](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.1) (2016-10-26)
|
|
80
|
-
|
|
81
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.1.0...v0.1.1)
|
|
82
|
-
|
|
83
|
-
## [v0.1.0](https://github.com/dominicsayers/url_canonicalize/tree/v0.1.0) (2016-10-20)
|
|
84
|
-
|
|
85
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.7...v0.1.0)
|
|
86
|
-
|
|
87
|
-
## [v0.0.7](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.7) (2016-10-20)
|
|
88
|
-
|
|
89
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.6...v0.0.7)
|
|
90
|
-
|
|
91
|
-
## [v0.0.6](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.6) (2016-10-20)
|
|
92
|
-
|
|
93
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.5...v0.0.6)
|
|
94
|
-
|
|
95
|
-
## [v0.0.5](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.5) (2016-10-20)
|
|
96
|
-
|
|
97
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.4...v0.0.5)
|
|
98
|
-
|
|
99
|
-
## [v0.0.4](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.4) (2016-10-20)
|
|
100
|
-
|
|
101
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.3...v0.0.4)
|
|
102
|
-
|
|
103
|
-
## [v0.0.3](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.3) (2016-10-20)
|
|
104
|
-
|
|
105
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.2...v0.0.3)
|
|
106
|
-
|
|
107
|
-
## [v0.0.2](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.2) (2016-10-20)
|
|
108
|
-
|
|
109
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/v0.0.1...v0.0.2)
|
|
110
|
-
|
|
111
|
-
## [v0.0.1](https://github.com/dominicsayers/url_canonicalize/tree/v0.0.1) (2016-10-20)
|
|
112
|
-
|
|
113
|
-
[Full Changelog](https://github.com/dominicsayers/url_canonicalize/compare/fec855bec42813304f93c8ef87035f184b054344...v0.0.1)
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
|
|
146
|
+
[Unreleased]: https://github.com/dominicsayers/url_canonicalize/compare/v2.0.0...HEAD
|
|
147
|
+
[2.0.0]: https://github.com/dominicsayers/url_canonicalize/compare/v1.0.2...v2.0.0
|
|
148
|
+
[1.0.2]: https://github.com/dominicsayers/url_canonicalize/compare/v1.0.1...v1.0.2
|
|
149
|
+
[1.0.1]: https://github.com/dominicsayers/url_canonicalize/compare/v1.0.0...v1.0.1
|
|
150
|
+
[1.0.0]: https://github.com/dominicsayers/url_canonicalize/compare/v0.2.1...v1.0.0
|
|
151
|
+
[0.2.1]: https://github.com/dominicsayers/url_canonicalize/compare/v0.2.0...v0.2.1
|
|
152
|
+
[0.2.0]: https://github.com/dominicsayers/url_canonicalize/compare/v0.1.15...v0.2.0
|
data/README.md
CHANGED
|
@@ -1,16 +1,15 @@
|
|
|
1
1
|
# URLCanonicalize
|
|
2
|
-
[](https://rubygems.org/gems/url_canonicalize)
|
|
3
3
|
[](https://rubygems.org/gems/url_canonicalize)
|
|
4
|
-
[](https://dependencyci.com/github/dominicsayers/url_canonicalize)
|
|
8
|
-
[](https://hakiri.io/github/dominicsayers/url_canonicalize/master)
|
|
9
|
-
[](https://www.codacy.com/app/dominicsayers/url_canonicalize)
|
|
4
|
+
[](https://github.com/dominicsayers/url_canonicalize/actions/workflows/test.yml)
|
|
5
|
+
[](https://github.com/dominicsayers/url_canonicalize/actions/workflows/codeql-analysis.yml)
|
|
6
|
+
[](https://sonarcloud.io/summary/new_code?id=dominicsayers_url_canonicalize)
|
|
10
7
|
|
|
11
8
|
URLCanonicalize is a Ruby gem that finds the canonical version of a URL. It
|
|
12
|
-
provides
|
|
13
|
-
Addressable::URI classes.
|
|
9
|
+
provides a configurable client API, and optional `canonicalize` extension
|
|
10
|
+
methods for the String, URI::HTTP, URI::HTTPS and Addressable::URI classes.
|
|
11
|
+
|
|
12
|
+
Ruby 3.3 or later is required.
|
|
14
13
|
|
|
15
14
|
## Installation
|
|
16
15
|
|
|
@@ -22,14 +21,240 @@ gem 'url_canonicalize'
|
|
|
22
21
|
|
|
23
22
|
## Usage
|
|
24
23
|
|
|
24
|
+
The primary API is a reusable client. Its options are validated and frozen at
|
|
25
|
+
construction, so one client can be shared safely between threads:
|
|
26
|
+
|
|
27
|
+
```ruby
|
|
28
|
+
client = URLCanonicalize::Client.new(
|
|
29
|
+
total_timeout: 10,
|
|
30
|
+
max_redirects: 8,
|
|
31
|
+
max_body_bytes: 512_000
|
|
32
|
+
)
|
|
33
|
+
|
|
34
|
+
client.canonicalize('http://www.twitter.com') # => 'https://twitter.com/'
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
For one-off calls the module methods accept the same options and build a
|
|
38
|
+
client internally:
|
|
39
|
+
|
|
40
|
+
```ruby
|
|
41
|
+
URLCanonicalize.canonicalize('http://www.twitter.com') # => 'https://twitter.com/'
|
|
42
|
+
URLCanonicalize.fetch('http://www.twitter.com') # => #<URLCanonicalize::Result ...>
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
`canonicalize` returns the canonical URL as a string. `fetch` returns an
|
|
46
|
+
immutable `URLCanonicalize::Result` with:
|
|
47
|
+
|
|
48
|
+
- `url` — the canonical URL;
|
|
49
|
+
- `response` — the `Net::HTTPResponse` that confirmed it;
|
|
50
|
+
- `html` — the parsed Nokogiri document when the response was HTML;
|
|
51
|
+
- `chain` — the request chain: each hop's URL and how it was discovered
|
|
52
|
+
(`:initial`, `:redirect` or `:canonical_link`);
|
|
53
|
+
- `source` — how the final URL was discovered.
|
|
54
|
+
|
|
55
|
+
### Configuration
|
|
56
|
+
|
|
57
|
+
Beyond the security limits described below, the client accepts:
|
|
58
|
+
|
|
59
|
+
- `headers` — the request headers to send (replaces the default browser-like
|
|
60
|
+
`User-Agent` and `Accept-Language`);
|
|
61
|
+
- `logger` — any object responding to `debug`; each request and discovered
|
|
62
|
+
canonical URL is logged. Nothing is logged by default;
|
|
63
|
+
- `transport` — anything responding to `call(uri, options)` and returning a
|
|
64
|
+
`Net::HTTP`-compatible connection. The default transport resolves and
|
|
65
|
+
validates the destination address (see below) and configures TLS and
|
|
66
|
+
timeouts. Tests can inject a fake transport to avoid the network entirely;
|
|
67
|
+
a custom resolver can be injected with
|
|
68
|
+
`URLCanonicalize::Transport.new(resolver: ...)`.
|
|
69
|
+
|
|
70
|
+
### Core-class extensions
|
|
71
|
+
|
|
72
|
+
The core-class extensions are opt-in. Requiring `url_canonicalize/core_ext`
|
|
73
|
+
extends `String`, `URI::HTTP`, `URI::HTTPS` and `Addressable::URI` with a
|
|
74
|
+
`canonicalize` method:
|
|
75
|
+
|
|
25
76
|
```ruby
|
|
77
|
+
require 'url_canonicalize/core_ext'
|
|
78
|
+
|
|
26
79
|
'http://www.twitter.com'.canonicalize # => 'https://twitter.com/'
|
|
27
80
|
|
|
28
81
|
URI('http://www.twitter.com').canonicalize # => #<URI::HTTP:0x00000008767908 URL:https://twitter.com/>
|
|
29
82
|
|
|
30
83
|
Addressable::URI.canonicalize('http://www.twitter.com') # => #<Addressable::URI:0x43c9 URI:https://twitter.com/>
|
|
84
|
+
|
|
85
|
+
'https://example.com/article'.canonicalize(total_timeout: 10)
|
|
31
86
|
```
|
|
32
87
|
|
|
88
|
+
Requiring `url_canonicalize` alone no longer patches any core class: if you
|
|
89
|
+
relied on the extensions, add the extra require.
|
|
90
|
+
|
|
91
|
+
## Normalization and canonicalization
|
|
92
|
+
|
|
93
|
+
URLCanonicalize distinguishes two operations:
|
|
94
|
+
|
|
95
|
+
- **Normalization** is syntactic: rewriting a URL into its RFC 3986 normal
|
|
96
|
+
form without any network access. Parsing and reference resolution use
|
|
97
|
+
Ruby's `URI` (strict); normalization uses `Addressable::URI#normalize`.
|
|
98
|
+
- **Canonicalization** is discovery: fetching the URL, following redirects
|
|
99
|
+
and server-declared canonical links, and returning the normalized final
|
|
100
|
+
URL.
|
|
101
|
+
|
|
102
|
+
Every URL is processed in this order:
|
|
103
|
+
|
|
104
|
+
1. The input is stripped of surrounding whitespace and parsed. A leading
|
|
105
|
+
`scheme:` token is honored; an input without one is treated as a host
|
|
106
|
+
name and given `http://`. A colon elsewhere in the URL does not suppress
|
|
107
|
+
the default scheme. Non-HTTP(S) schemes raise
|
|
108
|
+
`URLCanonicalize::Exception::URI`.
|
|
109
|
+
2. The URL is normalized (see below) and fetched with a `HEAD` request,
|
|
110
|
+
retried as `GET` where the server rejects `HEAD`.
|
|
111
|
+
3. Redirects (any `3xx` with a usable `Location`, temporary or permanent)
|
|
112
|
+
are followed. The `Location` value is resolved against the request URL
|
|
113
|
+
per RFC 3986, so absolute, protocol-relative, root-relative,
|
|
114
|
+
path-relative and query-only forms all work.
|
|
115
|
+
4. On a successful response, a canonical link is looked for in the `Link`
|
|
116
|
+
response header first, then in the HTML `<link rel="canonical">` element;
|
|
117
|
+
**the header wins when they conflict**. HTML links resolve against the
|
|
118
|
+
document's `<base href>` when one is declared, otherwise against the
|
|
119
|
+
request URL.
|
|
120
|
+
5. A declared canonical URL is not trusted blindly: it is fetched and
|
|
121
|
+
validated itself. If fetching it fails, the declared canonical URL is
|
|
122
|
+
still returned, confirmed by the response that declared it.
|
|
123
|
+
6. Redirects and followed canonical links share one hop budget and one
|
|
124
|
+
visited-URL set, so cycles and over-long chains terminate
|
|
125
|
+
deterministically. The returned URL is always normalized.
|
|
126
|
+
|
|
127
|
+
Normalization makes these changes and no others:
|
|
128
|
+
|
|
129
|
+
| Component | Rule |
|
|
130
|
+
| --- | --- |
|
|
131
|
+
| Scheme and host | Lowercased |
|
|
132
|
+
| Internationalized host | Converted to punycode (`xn--…`) |
|
|
133
|
+
| Default port (`:80`/`:443`) | Removed; other ports preserved |
|
|
134
|
+
| Dot segments (`/./`, `/../`) | Resolved |
|
|
135
|
+
| Percent-encoding | Uppercase hex; unreserved characters decoded; invalid characters encoded |
|
|
136
|
+
| Empty path | Replaced by `/` |
|
|
137
|
+
| Trailing slash | Preserved (`/a/` and `/a` stay distinct) |
|
|
138
|
+
| Query string | Preserved verbatim — parameters are never reordered or dropped |
|
|
139
|
+
| Fragment (`#…`) | Stripped — fragments are client-side state |
|
|
140
|
+
| Userinfo (`user:pass@`) | Rejected as a security error at fetch time |
|
|
141
|
+
|
|
142
|
+
## Security and limits
|
|
143
|
+
|
|
144
|
+
URLCanonicalize treats every fetched URL as untrusted, including redirect and
|
|
145
|
+
`rel="canonical"` destinations. Before connecting, it resolves the host,
|
|
146
|
+
rejects the destination if any DNS answer is loopback, private, link-local,
|
|
147
|
+
multicast, unspecified, reserved or otherwise not publicly routable, and pins
|
|
148
|
+
the connection to the validated address. URLs containing user information are
|
|
149
|
+
rejected. HTTPS always uses peer certificate and hostname verification; there
|
|
150
|
+
is no option to disable TLS verification.
|
|
151
|
+
|
|
152
|
+
IPv6 validation is fail-closed: only prefixes currently allocated in IANA's
|
|
153
|
+
global-unicast registry are eligible, with special-purpose ranges excluded.
|
|
154
|
+
|
|
155
|
+
Environment HTTP proxies are deliberately disabled because proxy-side DNS
|
|
156
|
+
resolution would bypass destination validation. Ports `80` and `443` are the
|
|
157
|
+
only ports allowed by default.
|
|
158
|
+
|
|
159
|
+
The default resource limits are:
|
|
160
|
+
|
|
161
|
+
- 30 seconds for the complete canonicalization operation, across every hop;
|
|
162
|
+
- 8 seconds to open a connection, 15 seconds to read and 8 seconds to write;
|
|
163
|
+
- 1 MiB for a buffered response body;
|
|
164
|
+
- 10 followed hops, shared between redirects and followed canonical links.
|
|
165
|
+
|
|
166
|
+
Only successful GET responses with `text/html`, `application/xhtml+xml`,
|
|
167
|
+
`application/xml` or `text/xml` media types are buffered. Media-type casing and
|
|
168
|
+
parameters such as `charset` are accepted. Other response bodies are drained
|
|
169
|
+
without being retained, and only HTML/XHTML responses are parsed for canonical
|
|
170
|
+
link elements.
|
|
171
|
+
|
|
172
|
+
All limits can be made stricter, and additional ports can be explicitly
|
|
173
|
+
allowed:
|
|
174
|
+
|
|
175
|
+
```ruby
|
|
176
|
+
URLCanonicalize.fetch(
|
|
177
|
+
'https://example.com:8443/article',
|
|
178
|
+
allowed_ports: [443, 8443],
|
|
179
|
+
max_body_bytes: 256_000,
|
|
180
|
+
total_timeout: 5,
|
|
181
|
+
open_timeout: 2,
|
|
182
|
+
read_timeout: 3,
|
|
183
|
+
write_timeout: 2,
|
|
184
|
+
max_redirects: 5
|
|
185
|
+
)
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Private-network fetching is available only as an explicit escape hatch for
|
|
189
|
+
trusted URLs:
|
|
190
|
+
|
|
191
|
+
```ruby
|
|
192
|
+
URLCanonicalize.fetch(
|
|
193
|
+
'http://intranet.example.test',
|
|
194
|
+
allow_private_networks: true
|
|
195
|
+
)
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Enabling `allow_private_networks` relaxes the SSRF protection. It still pins the
|
|
199
|
+
resolved address and enforces the user-information and port policies.
|
|
200
|
+
|
|
201
|
+
Policy violations raise `URLCanonicalize::Exception::Security`, oversized
|
|
202
|
+
responses raise `URLCanonicalize::Exception::ResponseTooLarge`, and the overall
|
|
203
|
+
deadline raises `URLCanonicalize::Exception::Timeout`. Other socket and TLS
|
|
204
|
+
failures continue to surface as `URLCanonicalize::Exception::Failure`.
|
|
205
|
+
|
|
206
|
+
## How URLs are followed
|
|
207
|
+
|
|
208
|
+
Every hop starts with a `HEAD` request. If the response carries no canonical
|
|
209
|
+
hint in its headers, the request is retried as a `GET` so the HTML can be
|
|
210
|
+
inspected. A `HEAD` request rejected with `403 Forbidden`,
|
|
211
|
+
`405 Method Not Allowed` or `501 Not Implemented` is also retried once as a
|
|
212
|
+
`GET`, because some servers refuse `HEAD` requests outright; an unsuccessful
|
|
213
|
+
`GET` is not retried.
|
|
214
|
+
|
|
215
|
+
All redirect responses (`301`, `302`, `303`, `307` and `308`) are followed
|
|
216
|
+
through their `Location` header. A redirection without a usable `Location` is
|
|
217
|
+
treated as a failure.
|
|
218
|
+
|
|
219
|
+
Canonical URLs are discovered from `Link` response headers whose `rel` tokens
|
|
220
|
+
include `canonical` (RFC 8288), and from `<link rel="canonical">` elements in
|
|
221
|
+
the HTML `<head>`, matched case-insensitively. Header targets resolve against
|
|
222
|
+
the request URL; HTML targets resolve against the document base URL, honouring
|
|
223
|
+
any `<base href>` element. Absolute, protocol-relative, root-relative,
|
|
224
|
+
path-relative and query-only references are all resolved per RFC 3986, and
|
|
225
|
+
only `http` or `https` results are followed.
|
|
226
|
+
|
|
227
|
+
Redirects and followed canonical links share a single visited-URL set and a
|
|
228
|
+
single hop budget (`max_redirects`), so cycles and over-long chains always
|
|
229
|
+
terminate. A cycle or exhausted budget returns the last successfully fetched
|
|
230
|
+
response if there is one, and raises `URLCanonicalize::Exception::Redirect`
|
|
231
|
+
otherwise. When a failure is caused by an underlying exception, that exception
|
|
232
|
+
is preserved as the `cause` of the raised
|
|
233
|
+
`URLCanonicalize::Exception::Failure`.
|
|
234
|
+
|
|
235
|
+
## Errors
|
|
236
|
+
|
|
237
|
+
All errors raised by the gem are subclasses of `URLCanonicalize::Exception`,
|
|
238
|
+
so `rescue URLCanonicalize::Exception` catches everything below:
|
|
239
|
+
|
|
240
|
+
| Exception | Raised when |
|
|
241
|
+
| --------- | ----------- |
|
|
242
|
+
| `Exception::URI` | the URL cannot be parsed, or is not `http`/`https` |
|
|
243
|
+
| `Exception::Security` | a destination violates the SSRF, userinfo or port policy |
|
|
244
|
+
| `Exception::Redirect` | a redirect or canonical-link chain loops or exceeds `max_redirects` |
|
|
245
|
+
| `Exception::Timeout` | the operation exceeds `total_timeout` |
|
|
246
|
+
| `Exception::ResponseTooLarge` | a response body exceeds `max_body_bytes` |
|
|
247
|
+
| `Exception::Failure` | the request fails (network error, TLS failure or unsuccessful HTTP status) |
|
|
248
|
+
|
|
249
|
+
When a failure wraps an underlying exception, the original is available as the
|
|
250
|
+
raised error's `cause`.
|
|
251
|
+
|
|
252
|
+
## Contributing and security
|
|
253
|
+
|
|
254
|
+
Development and release instructions are in
|
|
255
|
+
[CONTRIBUTING.md](CONTRIBUTING.md). Please report vulnerabilities privately as
|
|
256
|
+
described in [SECURITY.md](SECURITY.md), not through public issues.
|
|
257
|
+
|
|
33
258
|
## More Information
|
|
34
259
|
|
|
35
260
|
URLCanonical follows HTTP redirects and also looks for `rel="canonical"` hints
|
data/SECURITY.md
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Security Policy
|
|
2
|
+
|
|
3
|
+
## Supported versions
|
|
4
|
+
|
|
5
|
+
| Version | Supported |
|
|
6
|
+
| ------- | ------------------ |
|
|
7
|
+
| 1.x | :white_check_mark: |
|
|
8
|
+
| < 1.0 | :x: |
|
|
9
|
+
|
|
10
|
+
## Reporting a vulnerability
|
|
11
|
+
|
|
12
|
+
Please do not open a public issue for security problems. Instead, use GitHub's
|
|
13
|
+
private vulnerability reporting: go to the repository's **Security** tab and
|
|
14
|
+
choose **Report a vulnerability**, or visit
|
|
15
|
+
<https://github.com/dominicsayers/url_canonicalize/security/advisories/new>.
|
|
16
|
+
|
|
17
|
+
You should receive a response within a week. Please include a proof of concept
|
|
18
|
+
or reproduction steps where possible.
|
|
19
|
+
|
|
20
|
+
## Scope notes
|
|
21
|
+
|
|
22
|
+
URLCanonicalize fetches caller-supplied URLs, so requests forged through
|
|
23
|
+
redirects or canonical links (SSRF), TLS verification, resource exhaustion and
|
|
24
|
+
destination policy bypasses are all in scope. The README's "Security and
|
|
25
|
+
limits" section describes the protections the gem provides by default.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module URLCanonicalize
|
|
4
|
+
# String-compatible response buffer that enforces a maximum byte size
|
|
5
|
+
class BoundedBody < String
|
|
6
|
+
def initialize(max_bytes)
|
|
7
|
+
@max_bytes = max_bytes
|
|
8
|
+
super()
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def <<(chunk)
|
|
12
|
+
raise URLCanonicalize::Exception::ResponseTooLarge, error_message if bytesize + chunk.bytesize > @max_bytes
|
|
13
|
+
|
|
14
|
+
super
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
private
|
|
18
|
+
|
|
19
|
+
def error_message
|
|
20
|
+
"Response body exceeds #{@max_bytes} bytes"
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module URLCanonicalize
|
|
4
|
+
# A reusable canonicalization client. The options are validated and frozen
|
|
5
|
+
# at construction, so one client can be shared safely between threads
|
|
6
|
+
class Client
|
|
7
|
+
attr_reader :options
|
|
8
|
+
|
|
9
|
+
def initialize(**values)
|
|
10
|
+
@options = URLCanonicalize::Options.new(**values)
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def canonicalize(url)
|
|
14
|
+
fetch(url).url
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def fetch(url)
|
|
18
|
+
URLCanonicalize::HTTP.new(url, options: options).fetch
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require 'addressable/uri'
|
|
4
|
+
|
|
3
5
|
module Addressable
|
|
4
6
|
# Patch for Addressable's URI class
|
|
5
7
|
class URI
|
|
6
|
-
def self.canonicalize(uri)
|
|
8
|
+
def self.canonicalize(uri, **)
|
|
7
9
|
url = parse(uri).to_s # uri can be anything Addressable::URI can handle
|
|
8
|
-
canonical_url = URLCanonicalize.canonicalize(url)
|
|
10
|
+
canonical_url = URLCanonicalize.canonicalize(url, **)
|
|
9
11
|
parse(canonical_url)
|
|
10
12
|
end
|
|
11
13
|
end
|
|
@@ -1,10 +1,12 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require 'uri'
|
|
4
|
+
|
|
3
5
|
module URI
|
|
4
6
|
# URI having the HTTP protocol
|
|
5
7
|
class HTTP
|
|
6
|
-
def canonicalize
|
|
7
|
-
new_url = URLCanonicalize.canonicalize(to_s)
|
|
8
|
+
def canonicalize(**)
|
|
9
|
+
new_url = URLCanonicalize.canonicalize(to_s, **)
|
|
8
10
|
::URI.parse(new_url)
|
|
9
11
|
end
|
|
10
12
|
end
|