url-normalize 3.1.0__tar.gz → 4.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- url_normalize-4.0.0/PKG-INFO +591 -0
- url_normalize-4.0.0/README.md +559 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/pyproject.toml +13 -7
- url_normalize-4.0.0/tests/test_cli.py +650 -0
- url_normalize-4.0.0/tests/test_component_presence.py +211 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_deconstruct_url.py +45 -2
- url_normalize-3.1.0/tests/test_tools.py → url_normalize-4.0.0/tests/test_encoding.py +2 -2
- url_normalize-4.0.0/tests/test_fragment_policy.py +238 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_normalize_fragment.py +13 -2
- url_normalize-4.0.0/tests/test_normalize_host.py +142 -0
- url_normalize-4.0.0/tests/test_normalize_path.py +167 -0
- url_normalize-4.0.0/tests/test_normalize_port.py +76 -0
- url_normalize-4.0.0/tests/test_normalize_query.py +118 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_normalize_query_filters.py +153 -4
- url_normalize-4.0.0/tests/test_param_collections.py +73 -0
- url_normalize-4.0.0/tests/test_path_policy.py +215 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_provide_url_scheme.py +45 -1
- url_normalize-4.0.0/tests/test_query_denylist.py +134 -0
- url_normalize-4.0.0/tests/test_query_sorting.py +148 -0
- url_normalize-4.0.0/tests/test_readme.py +47 -0
- url_normalize-4.0.0/tests/test_reconstruct_url.py +77 -0
- url_normalize-4.0.0/tests/test_relative_references.py +305 -0
- url_normalize-4.0.0/tests/test_tracking_params.py +185 -0
- url_normalize-4.0.0/tests/test_url_humanize.py +337 -0
- url_normalize-4.0.0/tests/test_url_normalize.py +632 -0
- url_normalize-4.0.0/tests/test_url_properties.py +214 -0
- url_normalize-4.0.0/tests/test_userinfo_display.py +58 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize/__init__.py +1 -1
- url_normalize-4.0.0/url_normalize/_encoding.py +79 -0
- url_normalize-4.0.0/url_normalize/_schemes.py +33 -0
- url_normalize-4.0.0/url_normalize/_tracking.py +48 -0
- url_normalize-4.0.0/url_normalize/_url.py +121 -0
- url_normalize-4.0.0/url_normalize/cli.py +153 -0
- url_normalize-4.0.0/url_normalize/normalize_fragment.py +37 -0
- url_normalize-4.0.0/url_normalize/normalize_host.py +119 -0
- url_normalize-4.0.0/url_normalize/normalize_path.py +101 -0
- url_normalize-4.0.0/url_normalize/normalize_port.py +33 -0
- url_normalize-4.0.0/url_normalize/normalize_query.py +93 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize/normalize_userinfo.py +2 -7
- url_normalize-4.0.0/url_normalize/param_allowlist.py +35 -0
- url_normalize-4.0.0/url_normalize/param_filter.py +52 -0
- url_normalize-4.0.0/url_normalize/provide_url_scheme.py +53 -0
- url_normalize-4.0.0/url_normalize/url_humanize.py +249 -0
- url_normalize-4.0.0/url_normalize/url_normalize.py +175 -0
- url_normalize-4.0.0/url_normalize.egg-info/PKG-INFO +591 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize.egg-info/SOURCES.txt +17 -4
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize.egg-info/requires.txt +2 -1
- url_normalize-3.1.0/PKG-INFO +0 -219
- url_normalize-3.1.0/README.md +0 -188
- url_normalize-3.1.0/tests/test_cli.py +0 -360
- url_normalize-3.1.0/tests/test_generic_url_cleanup.py +0 -72
- url_normalize-3.1.0/tests/test_normalize_host.py +0 -86
- url_normalize-3.1.0/tests/test_normalize_path.py +0 -58
- url_normalize-3.1.0/tests/test_normalize_port.py +0 -26
- url_normalize-3.1.0/tests/test_normalize_query.py +0 -57
- url_normalize-3.1.0/tests/test_reconstruct_url.py +0 -42
- url_normalize-3.1.0/tests/test_url_humanize.py +0 -167
- url_normalize-3.1.0/tests/test_url_normalize.py +0 -231
- url_normalize-3.1.0/url_normalize/cli.py +0 -82
- url_normalize-3.1.0/url_normalize/generic_url_cleanup.py +0 -27
- url_normalize-3.1.0/url_normalize/normalize_fragment.py +0 -27
- url_normalize-3.1.0/url_normalize/normalize_host.py +0 -39
- url_normalize-3.1.0/url_normalize/normalize_path.py +0 -48
- url_normalize-3.1.0/url_normalize/normalize_port.py +0 -38
- url_normalize-3.1.0/url_normalize/normalize_query.py +0 -70
- url_normalize-3.1.0/url_normalize/param_allowlist.py +0 -65
- url_normalize-3.1.0/url_normalize/provide_url_scheme.py +0 -43
- url_normalize-3.1.0/url_normalize/tools.py +0 -134
- url_normalize-3.1.0/url_normalize/url_humanize.py +0 -136
- url_normalize-3.1.0/url_normalize/url_normalize.py +0 -80
- url_normalize-3.1.0/url_normalize.egg-info/PKG-INFO +0 -219
- {url_normalize-3.1.0 → url_normalize-4.0.0}/LICENSE +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/setup.cfg +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_normalize_scheme.py +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_normalize_userinfo.py +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/tests/test_provide_url_domain.py +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize/normalize_scheme.py +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize/provide_url_domain.py +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize/py.typed +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize.egg-info/dependency_links.txt +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize.egg-info/entry_points.txt +0 -0
- {url_normalize-3.1.0 → url_normalize-4.0.0}/url_normalize.egg-info/top_level.txt +0 -0
|
@@ -0,0 +1,591 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: url-normalize
|
|
3
|
+
Version: 4.0.0
|
|
4
|
+
Summary: URL normalization for Python
|
|
5
|
+
Author-email: Nikolay Panov <github@npanov.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/niksite/url-normalize
|
|
8
|
+
Project-URL: Repository, https://github.com/niksite/url-normalize
|
|
9
|
+
Project-URL: Issues, https://github.com/niksite/url-normalize/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/niksite/url-normalize/blob/master/CHANGELOG.md
|
|
11
|
+
Keywords: url,normalization,normalize,normalizer
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.15
|
|
19
|
+
Requires-Python: >=3.11
|
|
20
|
+
Description-Content-Type: text/markdown
|
|
21
|
+
License-File: LICENSE
|
|
22
|
+
Requires-Dist: idna>=3.20
|
|
23
|
+
Provides-Extra: dev
|
|
24
|
+
Requires-Dist: hypothesis; extra == "dev"
|
|
25
|
+
Requires-Dist: pre-commit; extra == "dev"
|
|
26
|
+
Requires-Dist: pytest-cov; extra == "dev"
|
|
27
|
+
Requires-Dist: pytest-socket; extra == "dev"
|
|
28
|
+
Requires-Dist: pytest; extra == "dev"
|
|
29
|
+
Requires-Dist: ruff; extra == "dev"
|
|
30
|
+
Requires-Dist: ty; extra == "dev"
|
|
31
|
+
Dynamic: license-file
|
|
32
|
+
|
|
33
|
+
# url-normalize
|
|
34
|
+
|
|
35
|
+
[](https://github.com/niksite/url-normalize/actions/workflows/ci.yml)
|
|
36
|
+
[](https://coveralls.io/r/niksite/url-normalize)
|
|
37
|
+
[](https://pypi.org/project/url-normalize/)
|
|
38
|
+
[](https://pypi.org/project/url-normalize/)
|
|
39
|
+
[](https://github.com/niksite/url-normalize/blob/master/LICENSE)
|
|
40
|
+
[](https://github.com/astral-sh/ruff)
|
|
41
|
+
|
|
42
|
+
Normalize URL spelling consistently for storage, comparison, and display. Use `url_normalize()` for normalized URLs and `url_humanize()` for readable display URLs.
|
|
43
|
+
|
|
44
|
+
Requires **Python 3.11 or newer**. Normalization operates on URL text; it does not fetch pages or follow redirects.
|
|
45
|
+
|
|
46
|
+
## Installation
|
|
47
|
+
|
|
48
|
+
These examples use url-normalize 4.0.0. If you store normalized URLs or use custom parameter filters, read the [4.0.0 upgrade notes](CHANGELOG.md#400---2026-11-18) before upgrading. The [3.1.0 usage guide](https://github.com/niksite/url-normalize/blob/v3.1.0/README.md) covers the previous release.
|
|
49
|
+
|
|
50
|
+
Install the library and CLI:
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
python -m pip install "url-normalize>=4.0.0"
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Or install the CLI in its own environment with [uv](https://docs.astral.sh/uv/):
|
|
57
|
+
|
|
58
|
+
```sh
|
|
59
|
+
uv tool install "url-normalize>=4.0.0"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Quick start
|
|
63
|
+
|
|
64
|
+
```python
|
|
65
|
+
from url_normalize import url_normalize
|
|
66
|
+
|
|
67
|
+
print(url_normalize("HTTP://Example.COM:80/a/../b"))
|
|
68
|
+
# Output: http://example.com/b
|
|
69
|
+
|
|
70
|
+
print(url_normalize("bücher.de/résumé"))
|
|
71
|
+
# Output: https://xn--bcher-kva.de/r%C3%A9sum%C3%A9
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
From the terminal:
|
|
75
|
+
|
|
76
|
+
```sh
|
|
77
|
+
url-normalize "HTTP://Example.COM:80/a/../b"
|
|
78
|
+
# Output: http://example.com/b
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Normalization behavior
|
|
82
|
+
|
|
83
|
+
| Behavior | Input | Output |
|
|
84
|
+
| --- | --- | --- |
|
|
85
|
+
| Add a missing scheme | `example.com` | `https://example.com/` |
|
|
86
|
+
| Lowercase scheme and host; remove a default port | `HTTP://Example.COM:80/` | `http://example.com/` |
|
|
87
|
+
| Resolve dot segments | `https://example.com/a/../b` | `https://example.com/b` |
|
|
88
|
+
| Collapse repeated path slashes | `https://example.com/a//b///` | `https://example.com/a/b/` |
|
|
89
|
+
| Encode Unicode hosts and paths | `https://bücher.de/résumé` | `https://xn--bcher-kva.de/r%C3%A9sum%C3%A9` |
|
|
90
|
+
| Remove one trailing host dot, including Unicode equivalents | `https://example.com。/` | `https://example.com/` |
|
|
91
|
+
| Preserve encoded path separators; uppercase escape hex | `https://example.com/a%2fb` | `https://example.com/a%2Fb` |
|
|
92
|
+
| Preserve query order and repeated keys | `https://example.com/?b=2&a=1&a=3` | `https://example.com/?b=2&a=1&a=3` |
|
|
93
|
+
| Preserve explicit empty query values | `https://example.com/?flag=&flag` | `https://example.com/?flag=&flag` |
|
|
94
|
+
| Remove empty query and fragment components | `https://example.com/?#` | `https://example.com/` |
|
|
95
|
+
| Preserve an explicit relative reference | `../images/a.png` | `../images/a.png` |
|
|
96
|
+
| Preserve fragment routes | `https://example.com/#!/users/42` | `https://example.com/#!/users/42` |
|
|
97
|
+
| Preserve text-highlight syntax | `https://example.com/#:~:text=hello%20world` | `https://example.com/#:~:text=hello%20world` |
|
|
98
|
+
|
|
99
|
+
In query parameters, `+`, `%2B`, and `%20` remain distinct. Parameter filtering is disabled by default. Fragments are preserved and normalized by default, including hashbang routes (`#!/route`) and text directives (`#:~:text=hello`). Use `strip_fragment=True` to remove the entire fragment explicitly.
|
|
100
|
+
|
|
101
|
+
Fragments keep literal RFC 3986 fragment punctuation and existing ASCII escapes, with uppercase escape hex. This deliberately includes escaped letters, hyphens and tildes: fragment syntax can be interpreted before percent-decoding. For example, `#:%7E:text=hello` stays encoded instead of becoming an active text directive. Unicode text uses UTF-8 and NFC normalization as usual. Hashbang fragments are no longer converted into `_escaped_fragment_` query parameters.
|
|
102
|
+
|
|
103
|
+
Empty query and fragment delimiters are removed by default. Set `keep_empty_components=True` in either Python API, or use `--keep-empty-components` in the CLI, to retain them when your application needs to distinguish an empty component from an absent one. For example, `https://example.com/?#` stays unchanged with this option.
|
|
104
|
+
|
|
105
|
+
The option applies to query and fragment delimiters. Empty parameter values such as `flag=` are preserved with either setting. Empty query fields are still removed: `?a&&b&` becomes `?a&b`, while `?&&` disappears by default or becomes `?` with `keep_empty_components=True`.
|
|
106
|
+
|
|
107
|
+
Normalization applies consistent spelling rules; it does not establish whether two URLs serve the same content. Applications using normalized URLs as persistent cache keys or identifiers should review the [changelog](CHANGELOG.md) when upgrading.
|
|
108
|
+
|
|
109
|
+
## Python API
|
|
110
|
+
|
|
111
|
+
Both functions accept a URL as `str` or `None` and return `str` or `None`. An empty string and `None` are returned unchanged. Malformed input can raise `ValueError`, including `UnicodeError` for invalid hostnames.
|
|
112
|
+
|
|
113
|
+
Return types follow the input type: a `str` input gives a `str` result, `None` gives `None`, and a value typed as `str | None` gives `str | None`. Type checkers can therefore use the string result directly:
|
|
114
|
+
|
|
115
|
+
```python
|
|
116
|
+
from url_normalize import url_normalize
|
|
117
|
+
|
|
118
|
+
result: str = url_normalize("HTTP://Example.COM:80/a/../b")
|
|
119
|
+
print(result.startswith("http://"))
|
|
120
|
+
# Output: True
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
The same contract applies to `url_humanize()` and every keyword option. An empty string result is still a string, including when `strip_fragment=True` removes a fragment-only reference.
|
|
124
|
+
|
|
125
|
+
The following options are keyword-only and apply to both functions:
|
|
126
|
+
|
|
127
|
+
| Option | Default | Purpose |
|
|
128
|
+
| --- | --- | --- |
|
|
129
|
+
| `default_scheme` | `"https"` | Scheme to add to bare hosts and network-path references such as `//example.com/`. Accepts uppercase, surrounding whitespace and one optional trailing colon. |
|
|
130
|
+
| `default_domain` | `None` | Domain to add to an absolute path such as `/images/logo.png`. |
|
|
131
|
+
| `filter_params` | `False` | Enable built-in domain filtering when no custom allowlist is supplied. |
|
|
132
|
+
| `param_allowlist` | `None` | Keep one name, a sequence or set of names, or names from a domain mapping. Supplying a rule enables filtering, including an empty collection. |
|
|
133
|
+
| `param_denylist` | `None` | Remove names using the same rule forms as the allowlist, without an extra flag. Denied names override allowed names; an empty collection removes nothing. |
|
|
134
|
+
| `remove_tracking_params` | `False` | Remove a built-in set of exact, case-sensitive tracking parameter names in addition to any custom filtering. |
|
|
135
|
+
| `sort_query` | `False` | Sort by normalized, percent-encoded query key after filtering. Preserve the order of repeated values and each field's value separator. |
|
|
136
|
+
| `collapse_slashes` | `True` | Collapse repeated literal path slashes after resolving dot segments, for the schemes described below. Set `False` to preserve repeated slashes. |
|
|
137
|
+
| `remove_trailing_slash` | `False` | Remove the entire trailing run of literal path slashes, keeping `/` for the root. Uses the same scheme scope as `collapse_slashes`. |
|
|
138
|
+
| `keep_empty_components` | `False` | Keep query (`?`) and fragment (`#`) delimiters when their content is empty, including a query emptied by parameter filtering. |
|
|
139
|
+
| `strip_fragment` | `False` | Remove the entire fragment and its `#` delimiter, overriding `keep_empty_components` for the fragment only. |
|
|
140
|
+
|
|
141
|
+
Unicode URL characters always use UTF-8 percent encoding.
|
|
142
|
+
|
|
143
|
+
### Hostname boundaries
|
|
144
|
+
|
|
145
|
+
Leading dots, empty interior labels, repeated trailing dots, and dot-only hosts are preserved in both APIs. A single trailing dot is removed when other host characters precede it. ASCII dots and their IDNA equivalents (`。`, `.`, `。`) follow the same rule:
|
|
146
|
+
|
|
147
|
+
| Input | Result |
|
|
148
|
+
| --- | --- |
|
|
149
|
+
| `https://example.com./` | `https://example.com/` |
|
|
150
|
+
| `https://.example.com/` | `https://.example.com/` |
|
|
151
|
+
| `https://a..example.com/` | `https://a..example.com/` |
|
|
152
|
+
| `https://example.com../` | `https://example.com../` |
|
|
153
|
+
| `https://./` | `https://./` |
|
|
154
|
+
| `https://../` | `https://../` |
|
|
155
|
+
| `https://a。。example.com/` | `https://a..example.com/` |
|
|
156
|
+
|
|
157
|
+
This hostname policy also applies to `default_domain` and custom `param_allowlist` domain keys. Entries for `.example.com` and `example.com` remain distinct; an unusual host can have its own explicit allowlist entry. The CLI uses the same normalization policy.
|
|
158
|
+
|
|
159
|
+
Preservation does not establish DNS validity. The generic registered-name syntax in [RFC 3986 section 3.2.2](https://www.rfc-editor.org/rfc/rfc3986.html#section-3.2.2) is broader than DNS hostname syntax. Applications that require DNS-valid hostnames should validate them separately. Empty hosts in local file URLs and path references remain supported.
|
|
160
|
+
|
|
161
|
+
Literal control characters, bidi controls and Unicode line/paragraph separators remaining in a parsed host, including an IPv6 zone identifier, raise `UnicodeError` (a `ValueError` subclass). The CLI exits with status 1 and a diagnostic that does not repeat the hostname. Existing input cleanup still removes tabs, carriage returns and newlines before parsing.
|
|
162
|
+
|
|
163
|
+
### IPv6 addresses
|
|
164
|
+
|
|
165
|
+
Equivalent bracketed IPv6 addresses use a consistent spelling: lowercase hexadecimal, no leading zeroes in groups, and `::` for the longest run of at least two zero groups (the first run wins a tie). These rules follow [RFC 5952 section 4](https://www.rfc-editor.org/rfc/rfc5952.html#section-4).
|
|
166
|
+
|
|
167
|
+
| Input | Result |
|
|
168
|
+
| --- | --- |
|
|
169
|
+
| `https://[2001:0DB8:0:0:0:0:0:1]/` | `https://[2001:db8::1]/` |
|
|
170
|
+
| `https://[FE80:0:0:0:0:0:0:ABCD%25EthA]/` | `https://[fe80::abcd%25EthA]/` |
|
|
171
|
+
| `https://[::FFFF:C000:280]/` | `https://[::ffff:192.0.2.128]/` |
|
|
172
|
+
|
|
173
|
+
Zone identifiers, including their case and percent-encoding spelling, are preserved. IPv4-mapped IPv6 addresses use `::ffff:` followed by dotted decimal; other IPv6 addresses use hexadecimal groups. IPvFuture literals retain their existing case normalization. These rules apply to both APIs, the CLI, `default_domain`, and allowlist keys. Expanded and compressed allowlist keys now match the same address; when entries conflict, the canonical key takes precedence.
|
|
174
|
+
|
|
175
|
+
### Port normalization
|
|
176
|
+
|
|
177
|
+
ASCII decimal ports lose leading zeroes, and scheme defaults are removed: `http://example.com:0080/` becomes `http://example.com/`, while `https://example.com:0080/` becomes `https://example.com:80/`. An empty port and its colon are removed too. Numeric normalization uses string operations, including for heavily padded numbers.
|
|
178
|
+
|
|
179
|
+
Nonempty parsed ports must contain only ASCII digits (`0–9`), following [RFC 3986 section 3.2.3](https://www.rfc-editor.org/rfc/rfc3986.html#section-3.2.3). Other port text raises `ValueError: Port must contain only ASCII digits`. For example, `http://example.com:abc/`, `http://example.com:80/`, `http://example.com:²/`, and `http://example.com:-1/` are rejected. Percent-encoded digits are rejected too; Unicode digits are not converted to ASCII port numbers.
|
|
180
|
+
|
|
181
|
+
The check applies to ports in URL authorities, including custom schemes and `default_domain`. Colons elsewhere, such as `mailto:user:abc@example.com` and `./example.com:abc/path`, retain their existing behavior. Both Python APIs use the same rule. The CLI exits with status 1, writes a diagnostic to stderr and prints no URL when port syntax is invalid.
|
|
182
|
+
|
|
183
|
+
Normalization does not check the transport's port range: `https://example.com:065536/` becomes `https://example.com:65536/`. Range validation remains the caller's responsibility.
|
|
184
|
+
|
|
185
|
+
### Path slashes and dot segments
|
|
186
|
+
|
|
187
|
+
Repeated path slashes collapse by default, including trailing runs. Use `collapse_slashes=False` to preserve empty path segments when they matter to your application:
|
|
188
|
+
|
|
189
|
+
```python
|
|
190
|
+
from url_normalize import url_normalize
|
|
191
|
+
|
|
192
|
+
url = "https://example.com/a//b///"
|
|
193
|
+
print(url_normalize(url))
|
|
194
|
+
# Output: https://example.com/a/b/
|
|
195
|
+
|
|
196
|
+
print(url_normalize(url, collapse_slashes=False))
|
|
197
|
+
# Output: https://example.com/a//b///
|
|
198
|
+
|
|
199
|
+
print(url_normalize(url, remove_trailing_slash=True))
|
|
200
|
+
# Output: https://example.com/a/b
|
|
201
|
+
|
|
202
|
+
print(url_normalize(url, collapse_slashes=False, remove_trailing_slash=True))
|
|
203
|
+
# Output: https://example.com/a//b
|
|
204
|
+
|
|
205
|
+
print(url_normalize("https://example.com/a//../b"))
|
|
206
|
+
# Output: https://example.com/a/b
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Path operations run in order: resolve dot segments, collapse slash runs unless disabled, then optionally remove trailing slashes. In `/a//../b`, `..` removes the empty segment, leaving `/a/b` with either collapse setting. Encoded separators such as `%2F` remain encoded. Query and fragment handling follows the usual normalization rules.
|
|
210
|
+
|
|
211
|
+
Collapsing retains a single trailing slash. Enable `remove_trailing_slash` to remove it as well; `/a//b///` then becomes `/a/b`. With `collapse_slashes=False`, trailing-slash removal preserves internal slash runs, producing `/a//b`. The root remains `/`, including when the input path consists entirely of slashes.
|
|
212
|
+
|
|
213
|
+
These path operations apply to HTTP, HTTPS, WS, WSS, FTP, file URLs and path references. Document-relative paths resolve internal dots while retaining leading `../` segments that need a base URL. Other schemes keep their existing path handling. When collapsing is disabled, a root-relative path that would start with `//` retains a `/.` prefix to avoid being interpreted as a hostname; for example, `/a/..//b` becomes `/.//b`.
|
|
214
|
+
|
|
215
|
+
### Fragment removal
|
|
216
|
+
|
|
217
|
+
Set `strip_fragment=True` in either Python API, or use `--strip-fragment` in the CLI, to build URLs without document positions. For example, a crawler can use the same fetch key for links to different sections of one article. Fragments are handled separately from retrieval under [RFC 3986 section 3.5](https://www.rfc-editor.org/rfc/rfc3986.html#section-3.5).
|
|
218
|
+
|
|
219
|
+
```python
|
|
220
|
+
from url_normalize import url_normalize
|
|
221
|
+
|
|
222
|
+
url = "https://example.com/article?id=1#intro"
|
|
223
|
+
print(url_normalize(url))
|
|
224
|
+
# Output: https://example.com/article?id=1#intro
|
|
225
|
+
|
|
226
|
+
print(url_normalize(url, strip_fragment=True))
|
|
227
|
+
# Output: https://example.com/article?id=1
|
|
228
|
+
|
|
229
|
+
print(url_normalize("https://example.com/article?id=1#details", strip_fragment=True))
|
|
230
|
+
# Output: https://example.com/article?id=1
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Removal is disabled by default. Enabling it removes anchors, fragment routes such as `#!/cart`, and text highlights such as `#:~:text=hello`. Keep the default when those destinations matter to your application. The option removes the whole fragment; it does not move fragment content into the query.
|
|
234
|
+
|
|
235
|
+
Percent-encoded hashes remain part of their original components: `https://example.com/a%23b?x=%23keep#section` becomes `https://example.com/a%23b?x=%23keep`. Existing query parameters, including `_escaped_fragment_`, follow the ordinary query rules. Path cleanup, filtering and sorting remain independently configurable.
|
|
236
|
+
|
|
237
|
+
`strip_fragment=True` removes even an empty `#` when `keep_empty_components=True`. Empty query preservation still applies: with both options, `https://example.com/page?#section` becomes `https://example.com/page?`. A fragment-only reference such as `#section` becomes an empty reference (`""`); `?#section` becomes `"?"` when empty components are kept, or `""` otherwise. Relative paths keep their existing relative form.
|
|
238
|
+
|
|
239
|
+
### Query escapes
|
|
240
|
+
|
|
241
|
+
Percent-encoded reserved characters stay encoded in query keys and values, with uppercase hexadecimal digits. A literal `/` or `;` stays distinct from its escaped form because applications can assign it a delimiter role ([RFC 3986 section 2.2](https://www.rfc-editor.org/rfc/rfc3986.html#section-2.2)). The first `=` separates a key from its value. Additional equals signs in the value normalize to `%3D`, so `?q=a=b` and `?q=a%3Db` have the same spelling. Unreserved escapes such as `%7E` become `~`.
|
|
242
|
+
|
|
243
|
+
| Input | `url_normalize()` | `url_humanize()` |
|
|
244
|
+
| --- | --- | --- |
|
|
245
|
+
| `https://example.com/?q=a%2fb` | `https://example.com/?q=a%2Fb` | `https://example.com/?q=a%2Fb` |
|
|
246
|
+
| `https://example.com/?q=a/b` | `https://example.com/?q=a/b` | `https://example.com/?q=a/b` |
|
|
247
|
+
| `https://example.com/?q=a%3Bb` | `https://example.com/?q=a%3Bb` | `https://example.com/?q=a%3Bb` |
|
|
248
|
+
| `https://example.com/?q=%C3%A9%2F%C3%B1` | `https://example.com/?q=%C3%A9%2F%C3%B1` | `https://example.com/?q=é%2Fñ` |
|
|
249
|
+
| `https://example.com/?q=%7Ehello` | `https://example.com/?q=~hello` | `https://example.com/?q=~hello` |
|
|
250
|
+
|
|
251
|
+
Parameter order is preserved by default. Duplicates and the distinction between `flag` and `flag=` are preserved with either query-sorting setting. Literal `+`, encoded `%2B`, and `%20` remain distinct in normalized output. Unicode still uses NFC normalization and UTF-8 encoding; these guarantees do not imply byte-for-byte preservation for signed URLs. The CLI uses the same rules in both modes.
|
|
252
|
+
|
|
253
|
+
### Query parameter filtering
|
|
254
|
+
|
|
255
|
+
Supplying `param_allowlist` keeps only the allowed parameters; no extra flag is needed. Use `filter_params=True` without a custom allowlist to select the built-in domain rules, which cover Google, Bing, Baidu, and YouTube; see the [domain rules](url_normalize/param_allowlist.py) for their parameter names.
|
|
256
|
+
|
|
257
|
+
`None` means no custom rule. An empty list or dictionary enables filtering and removes all query parameters. An explicit rule takes precedence even if `filter_params=False`; to turn filtering off, omit the rule or pass `param_allowlist=None` with `filter_params=False`.
|
|
258
|
+
|
|
259
|
+
Filtering that removes every parameter also removes the query delimiter by default. With `keep_empty_components=True`, the delimiter remains even when filtering empties the query. For example, filtering `https://example.com/?drop=1#` with no matching allowed name produces `https://example.com/` by default or `https://example.com/?#` with this option.
|
|
260
|
+
|
|
261
|
+
**Unknown domains have an empty default allowlist**, so filtering removes all their query parameters unless you supply a custom rule. A custom allowlist replaces the built-in rules.
|
|
262
|
+
|
|
263
|
+
```python
|
|
264
|
+
from url_normalize import url_normalize
|
|
265
|
+
|
|
266
|
+
# Use the built-in Google rule.
|
|
267
|
+
result = url_normalize(
|
|
268
|
+
"www.google.com/search?q=test&utm_source=test", filter_params=True
|
|
269
|
+
)
|
|
270
|
+
print(result)
|
|
271
|
+
# Output: https://www.google.com/search?q=test
|
|
272
|
+
|
|
273
|
+
# Keep one parameter name on any domain.
|
|
274
|
+
result = url_normalize(
|
|
275
|
+
"example.com/?id=123&i=456&d=789",
|
|
276
|
+
param_allowlist="id",
|
|
277
|
+
)
|
|
278
|
+
print(result)
|
|
279
|
+
# Output: https://example.com/?id=123
|
|
280
|
+
|
|
281
|
+
# Apply a list of allowed names to any domain.
|
|
282
|
+
result = url_normalize(
|
|
283
|
+
"example.com/?page=1&id=123&ref=test",
|
|
284
|
+
param_allowlist=["page", "id"],
|
|
285
|
+
)
|
|
286
|
+
print(result)
|
|
287
|
+
# Output: https://example.com/?page=1&id=123
|
|
288
|
+
|
|
289
|
+
# Use domain-specific rules, including Unicode parameter names.
|
|
290
|
+
result = url_normalize(
|
|
291
|
+
"https://example.com/?r%C3%A9sum%C3%A9=1&drop=2",
|
|
292
|
+
param_allowlist={"example.com": "résumé"},
|
|
293
|
+
)
|
|
294
|
+
print(result)
|
|
295
|
+
# Output: https://example.com/?r%C3%A9sum%C3%A9=1
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
A string is one complete name: `"id"` means the same as `["id"]`, and `{"example.com": "id"}` means the same as `{"example.com": ["id"]}`. Strings are not split on commas; use `["page", "id"]` for two names, while `"page,id"` selects the literal name `page,id`. An empty string selects the empty parameter name (`?=value`); an empty list allows no names. Domain mappings may mix strings, sequences and sets. The CLI retains its comma-separated `--param-allowlist` format, enables filtering without `-f`, and treats `-p ""` as selecting the empty parameter name.
|
|
299
|
+
|
|
300
|
+
Both filters accept lists, tuples, sets and frozensets of strings, or mappings from domain strings to those forms (including read-only mappings). Rules are reusable and are not mutated. Their order does not change query order. Generators and other one-shot iterators are not supported; materialize them first, for example with `tuple(names)`.
|
|
301
|
+
|
|
302
|
+
```python
|
|
303
|
+
from url_normalize import url_normalize
|
|
304
|
+
|
|
305
|
+
allowed = ("id", "page")
|
|
306
|
+
blocked = frozenset({"ref"})
|
|
307
|
+
url = "https://example.com/?page=2&id=123&ref=email"
|
|
308
|
+
|
|
309
|
+
print(url_normalize(url, param_allowlist=allowed))
|
|
310
|
+
# Output: https://example.com/?page=2&id=123
|
|
311
|
+
|
|
312
|
+
print(url_normalize(url, param_denylist=blocked))
|
|
313
|
+
# Output: https://example.com/?page=2&id=123
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
Allowed names use the same normalization as query keys, including Unicode and spaces. Reserved escapes remain distinct: `param_allowlist=["a/b"]` matches `a/b=1`, while `param_allowlist=["a%2fb"]` matches `a%2Fb=1`. Domain rules match normalized hostnames and ignore a leading `www.` and the port. Rules do not automatically apply to other subdomains.
|
|
317
|
+
|
|
318
|
+
To remove selected names while keeping all other parameters, supply `param_denylist`. It accepts the same single-name, sequence, set and domain-map forms and uses the same case-sensitive name matching. It does not enable the built-in allowlist. An empty denylist or a domain map without a matching rule removes nothing. Names are literal: `"utm_*"` is not a wildcard.
|
|
319
|
+
|
|
320
|
+
```python
|
|
321
|
+
from url_normalize import url_normalize
|
|
322
|
+
|
|
323
|
+
result = url_normalize(
|
|
324
|
+
"https://example.com/products?id=123&utm_source=newsletter&variant=blue",
|
|
325
|
+
param_denylist="utm_source",
|
|
326
|
+
)
|
|
327
|
+
print(result)
|
|
328
|
+
# Output: https://example.com/products?id=123&variant=blue
|
|
329
|
+
|
|
330
|
+
# Narrow the built-in Google rule without copying its allowed names.
|
|
331
|
+
result = url_normalize(
|
|
332
|
+
"https://google.com/search?q=hello&ie=UTF-8&utm_source=newsletter",
|
|
333
|
+
filter_params=True,
|
|
334
|
+
param_denylist="ie",
|
|
335
|
+
)
|
|
336
|
+
print(result)
|
|
337
|
+
# Output: https://google.com/search?q=hello
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
When both rules apply, a parameter must be allowed and must not be denied. For example, `param_allowlist=["id", "tag"]` with `param_denylist="tag"` retains only `id`. Removal preserves the order, duplicates and value separators of the remaining parameters; ordinary URL normalization and any requested query sorting still apply. Removing the last parameter follows `keep_empty_components`, just like allowlist filtering. These rules also apply to `url_humanize()`.
|
|
341
|
+
|
|
342
|
+
### Tracking parameter removal
|
|
343
|
+
|
|
344
|
+
Set `remove_tracking_params=True` to remove known attribution parameters on any domain while retaining other query names. This works in both Python APIs and does not enable the built-in domain allowlist. It is disabled by default because removing attribution changes campaign measurement.
|
|
345
|
+
|
|
346
|
+
```python
|
|
347
|
+
from url_normalize import url_normalize
|
|
348
|
+
|
|
349
|
+
url = "https://example.com/product?id=123&ttclid=abc&srsltid=xyz"
|
|
350
|
+
print(url_normalize(url))
|
|
351
|
+
# Output: https://example.com/product?id=123&ttclid=abc&srsltid=xyz
|
|
352
|
+
|
|
353
|
+
print(url_normalize(url, remove_tracking_params=True))
|
|
354
|
+
# Output: https://example.com/product?id=123
|
|
355
|
+
|
|
356
|
+
print(
|
|
357
|
+
url_normalize(
|
|
358
|
+
"https://example.com/search?ref=main&source=books&utm_source=email",
|
|
359
|
+
remove_tracking_params=True,
|
|
360
|
+
)
|
|
361
|
+
)
|
|
362
|
+
# Output: https://example.com/search?ref=main&source=books
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
The preset removes these exact names, based on provider documentation:
|
|
366
|
+
|
|
367
|
+
| Family | Parameter names | Sources |
|
|
368
|
+
| --- | --- | --- |
|
|
369
|
+
| UTM | `utm_id`, `utm_source`, `utm_medium`, `utm_campaign`, `utm_source_platform`, `utm_term`, `utm_content`, `utm_creative_format`, `utm_marketing_tactic`, `utm_referrer` | [Google Analytics](https://support.google.com/analytics/answer/10917952), [Analytics schema](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema), [Yandex](https://yandex.com/support/metrica/en/general/source-tags) |
|
|
370
|
+
| Google attribution | `gclid`, `dclid`, `gbraid`, `wbraid`, `gad_source`, `gad_campaignid`, `srsltid` | [Analytics identifiers](https://support.google.com/analytics/answer/16479993?hl=en), [Google Ads](https://support.google.com/google-ads/answer/10417364?hl=en), [Merchant Center](https://support.google.com/merchants/answer/15191080) |
|
|
371
|
+
| Other click identifiers | `fbclid`, `msclkid`, `ttclid`, `twclid`, `yclid` | [Meta](https://github.com/facebook/capi-param-builder/blob/main/client_js/README.md), [Microsoft](https://learn.microsoft.com/en-us/advertising/guides/uet-conversion-api-integration), [TikTok](https://ads.tiktok.com/help/article/tiktok-click-id?lang=en), [X](https://business.x.com/en/help/campaign-measurement-and-analytics/conversion-tracking-for-websites), [Yandex](https://yandex.com/support/metrica/en/general/source-tags) |
|
|
372
|
+
| Mailchimp | `mc_cid`, `mc_eid` | [Mailchimp](https://mailchimp.com/developer/marketing/concepts/e-commerce/) |
|
|
373
|
+
| Matomo / Piwik | `mtm_campaign`, `mtm_keyword`, `mtm_source`, `mtm_medium`, `mtm_content`, `mtm_cid`, `mtm_group`, `mtm_placement`, `mtm_kwd`, `matomo_campaign`, `pk_campaign`, `piwik_campaign`, `pk_kwd`, `piwik_kwd`, `pk_keyword`, `matomo_kwd` | [Campaign parameters](https://matomo.org/faq/reports/advanced-how-to-manually-build-campaign-tracking-urls/), [Default parameters](https://matomo.org/faq/how-to/faq_120/), [Aliases](https://plugins.matomo.org/MarketingCampaignsReporting) |
|
|
374
|
+
|
|
375
|
+
Matching uses normalized query keys: `%75tm_source` matches `utm_source`. Names remain case-sensitive, so `UTM_SOURCE` is retained. There are no prefix or wildcard rules: `utm_custom` and `gclid_extra` remain. Generic names such as `ref`, `source`, `keyword`, and `session_id` are retained because they can select content or identify a session. The preset only examines query keys; it does not remove matching words from paths, parameter values or fragment routes.
|
|
376
|
+
|
|
377
|
+
The preset adds removals to `param_denylist` and takes precedence over `param_allowlist`. An empty custom denylist does not disable it. To retain a preset name, leave this option off and supply your own `param_denylist`. Retained parameters keep their order unless `sort_query=True` is requested. Duplicates and value separators survive either setting, and an emptied query follows `keep_empty_components`.
|
|
378
|
+
|
|
379
|
+
### Query sorting
|
|
380
|
+
|
|
381
|
+
Use `sort_query=True` in either Python API, or `--sort-query` in the CLI, when your application treats the order of different query keys as irrelevant. This can produce common cache or deduplication keys. Sorting is disabled by default because some applications depend on query order.
|
|
382
|
+
|
|
383
|
+
```python
|
|
384
|
+
from url_normalize import url_normalize
|
|
385
|
+
|
|
386
|
+
url = "https://example.com/search?tag=red&page=2&tag=blue"
|
|
387
|
+
print(url_normalize(url))
|
|
388
|
+
# Output: https://example.com/search?tag=red&page=2&tag=blue
|
|
389
|
+
|
|
390
|
+
print(url_normalize(url, sort_query=True))
|
|
391
|
+
# Output: https://example.com/search?page=2&tag=red&tag=blue
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
Sorting runs after normalization and filtering. It compares the exact normalized, percent-encoded keys in ascending, case-sensitive order. An empty key sorts first; percent escapes sort before letters; uppercase ASCII letters sort before lowercase ones. For example, `https://example.com/?z=1&é=2&a=3` becomes `https://example.com/?%C3%A9=2&a=3&z=1`. `url_humanize()` displays the same order as `https://example.com/?é=2&a=3&z=1`.
|
|
395
|
+
|
|
396
|
+
Equal normalized keys retain their original order, including when normalization makes `%61` and `a` equal. Values are never used as a sorting key. Thus `tag=red&tag=blue` remains distinct from `tag=blue&tag=red`. Sorting preserves duplicates, each occurrence of `flag` versus `flag=`, and the usual escape distinctions such as `+`, `%2B`, and `%20`. It applies only to query fields; paths and fragment routes follow their existing policies. Allowlist, denylist and tracker removal can all be combined with sorting, and empty queries still follow `keep_empty_components`.
|
|
397
|
+
|
|
398
|
+
### Default domain and scheme
|
|
399
|
+
|
|
400
|
+
Use `default_domain` for paths starting with a single `/`. URLs starting with `//` already specify their host. This option does not resolve relative paths such as `../images/logo.png` against a page URL.
|
|
401
|
+
|
|
402
|
+
```python
|
|
403
|
+
from url_normalize import url_normalize
|
|
404
|
+
|
|
405
|
+
result = url_normalize(
|
|
406
|
+
"/images/logo.png", default_domain="example.com", default_scheme="http"
|
|
407
|
+
)
|
|
408
|
+
print(result)
|
|
409
|
+
# Output: http://example.com/images/logo.png
|
|
410
|
+
```
|
|
411
|
+
|
|
412
|
+
`default_scheme` accepts a scheme name such as `https`, `HTTP`, or `git+ssh`, optionally followed by one colon. Surrounding whitespace is removed and the name is lowercased. Names follow [RFC 3986 section 3.1](https://www.rfc-editor.org/rfc/rfc3986.html#section-3.1).
|
|
413
|
+
|
|
414
|
+
```python
|
|
415
|
+
from url_normalize import url_normalize
|
|
416
|
+
|
|
417
|
+
print(url_normalize("example.com/path", default_scheme=" HTTPS: "))
|
|
418
|
+
# Output: https://example.com/path
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
Malformed values such as `https://`, `https::`, or an empty scheme raise `ValueError`; non-string values raise `TypeError`. Validation applies to every URL that remains nonempty after input cleanup, including URLs that already have a scheme and relative references. A whitespace-only URL returns `""` and `None` returns `None` before scheme validation. The CLI applies the same rules and reports configuration errors with exit status 1.
|
|
422
|
+
|
|
423
|
+
### Relative references
|
|
424
|
+
|
|
425
|
+
Explicit references starting with `./`, `../`, `?`, or `#` stay relative in both APIs and the CLI. The standalone paths `.` and `..`, including query or fragment suffixes, also stay relative. `default_domain` and `default_scheme` do not turn these references into absolute URLs.
|
|
426
|
+
|
|
427
|
+
```python
|
|
428
|
+
from url_normalize import url_normalize
|
|
429
|
+
|
|
430
|
+
print(url_normalize("../images/a.png"))
|
|
431
|
+
# Output: ../images/a.png
|
|
432
|
+
|
|
433
|
+
print(url_normalize("./a//../b"))
|
|
434
|
+
# Output: ./a/b
|
|
435
|
+
|
|
436
|
+
print(url_normalize("../a//../b///"))
|
|
437
|
+
# Output: ../a/b/
|
|
438
|
+
|
|
439
|
+
print(url_normalize("../a//../b///", collapse_slashes=False))
|
|
440
|
+
# Output: ../a/b///
|
|
441
|
+
|
|
442
|
+
print(url_normalize("../a//../b///", remove_trailing_slash=True))
|
|
443
|
+
# Output: ../a/b
|
|
444
|
+
|
|
445
|
+
print(url_normalize("?q=é#section"))
|
|
446
|
+
# Output: ?q=%C3%A9#section
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
Document-relative paths use the same operation order: resolve internal dots, collapse slashes by default, then optionally remove trailing slashes. Unresolved parents stay relative: `../a/../../b` becomes `../../b`. The `./` prefix keeps an ordinary path from being interpreted as a hostname or scheme; `./a/../b` becomes `./b`, and `./a/../this:that` becomes `./this:that`. With collapsing disabled, `./a/..//b` becomes `.//b`, preserving its empty segment without changing it into a root-relative path.
|
|
450
|
+
|
|
451
|
+
Paths that reduce to the current directory retain a nonempty reference: `./a/..` becomes `./`, or `.` with `remove_trailing_slash=True`. Similarly, `../a/..` becomes `../`, or `..` with trailing-slash removal. Percent encoding, Unicode handling, query filtering, and empty-component policy still apply. Use `keep_empty_components=True` when empty `?` or `#` delimiters matter to later resolution.
|
|
452
|
+
|
|
453
|
+
To obtain an absolute URL, [resolve the reference against a base URL](https://www.rfc-editor.org/rfc/rfc3986.html#section-5.2.2). For example:
|
|
454
|
+
|
|
455
|
+
```python
|
|
456
|
+
from urllib.parse import urljoin
|
|
457
|
+
|
|
458
|
+
from url_normalize import url_normalize
|
|
459
|
+
|
|
460
|
+
absolute = urljoin("https://example.com/docs/page.html", "../images/a.png")
|
|
461
|
+
print(url_normalize(absolute))
|
|
462
|
+
# Output: https://example.com/images/a.png
|
|
463
|
+
```
|
|
464
|
+
|
|
465
|
+
Bare inputs retain host inference: `site/page` becomes `https://site/page`. Use `./site/page` to explicitly identify a document-relative path. Network-path references such as `//example.com/page` receive the selected default scheme.
|
|
466
|
+
|
|
467
|
+
### Human-readable URLs
|
|
468
|
+
|
|
469
|
+
`url_humanize()` normalizes its input, then decodes components only when the display URL normalizes back to the same result. Some escapes therefore remain encoded. In fragments, ASCII escapes remain encoded even when a round trip would allow decoding; safe Unicode can still become readable. For example, `#:~:text=r%C3%A9sum%C3%A9%2Dnotes` becomes `#:~:text=résumé%2Dnotes`, preserving the encoded hyphen as text rather than directive syntax. Fragment spaces remain `%20`.
|
|
470
|
+
|
|
471
|
+
```python
|
|
472
|
+
from url_normalize import url_humanize, url_normalize
|
|
473
|
+
|
|
474
|
+
normalized = "https://xn--bcher-kva.de/r%C3%A9sum%C3%A9"
|
|
475
|
+
readable = url_humanize(normalized)
|
|
476
|
+
print(readable)
|
|
477
|
+
# Output: https://bücher.de/résumé
|
|
478
|
+
|
|
479
|
+
assert url_normalize(readable) == normalized
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
Credentials retain reserved escapes, including `%40` for an `@` inside a username
|
|
483
|
+
or password. This leaves a single literal `@` separating credentials from the
|
|
484
|
+
host. Safe Unicode around those escapes can still become readable:
|
|
485
|
+
|
|
486
|
+
```python
|
|
487
|
+
from url_normalize import url_humanize
|
|
488
|
+
|
|
489
|
+
print(url_humanize("https://alice%40example.net:secret@example.com/resource"))
|
|
490
|
+
# Output: https://alice%40example.net:secret@example.com/resource
|
|
491
|
+
|
|
492
|
+
print(url_humanize("https://us%C3%A9r%40example.net:p%C3%A4ss@example.com/"))
|
|
493
|
+
# Output: https://usér%40example.net:päss@example.com/
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
Existing encoded colons, slashes and brackets in credentials remain encoded too.
|
|
497
|
+
The CLI's `--humanize` mode follows the same policy.
|
|
498
|
+
|
|
499
|
+
## Command-line usage
|
|
500
|
+
|
|
501
|
+
The CLI accepts one URL per invocation and prints the result to standard output. Quote URLs containing shell characters such as `&`.
|
|
502
|
+
|
|
503
|
+
```sh
|
|
504
|
+
# Keep selected query parameters.
|
|
505
|
+
url-normalize -p page,id "example.com/?page=1&id=123&ref=test"
|
|
506
|
+
# Output: https://example.com/?page=1&id=123
|
|
507
|
+
|
|
508
|
+
# Remove selected parameters while retaining all other names.
|
|
509
|
+
url-normalize --param-denylist utm_source,ref "example.com/?id=123&utm_source=newsletter&ref=email"
|
|
510
|
+
# Output: https://example.com/?id=123
|
|
511
|
+
|
|
512
|
+
# Remove the built-in tracker names while retaining other parameters.
|
|
513
|
+
url-normalize --remove-tracking-params "https://example.com/product?id=123&ttclid=abc&srsltid=xyz"
|
|
514
|
+
# Output: https://example.com/product?id=123
|
|
515
|
+
|
|
516
|
+
# Sort query keys while preserving the order of repeated values.
|
|
517
|
+
url-normalize --sort-query "https://example.com/search?tag=red&page=2&tag=blue"
|
|
518
|
+
# Output: https://example.com/search?page=2&tag=red&tag=blue
|
|
519
|
+
|
|
520
|
+
# Supply a domain and scheme for an absolute path.
|
|
521
|
+
url-normalize -d example.com -s http "/images/logo.png"
|
|
522
|
+
# Output: http://example.com/images/logo.png
|
|
523
|
+
|
|
524
|
+
# Collapse repeated slashes in the path (the default).
|
|
525
|
+
url-normalize "https://example.com/a//b///"
|
|
526
|
+
# Output: https://example.com/a/b/
|
|
527
|
+
|
|
528
|
+
# Preserve repeated slashes in the path.
|
|
529
|
+
url-normalize --no-collapse-slashes "https://example.com/a//b///"
|
|
530
|
+
# Output: https://example.com/a//b///
|
|
531
|
+
|
|
532
|
+
# Remove trailing slashes while preserving internal empty segments.
|
|
533
|
+
url-normalize --no-collapse-slashes --remove-trailing-slash "https://example.com/a//b///"
|
|
534
|
+
# Output: https://example.com/a//b
|
|
535
|
+
|
|
536
|
+
# Retain explicitly empty query and fragment components.
|
|
537
|
+
url-normalize --keep-empty-components "https://example.com/?#"
|
|
538
|
+
# Output: https://example.com/?#
|
|
539
|
+
|
|
540
|
+
# Remove an anchor, route or text highlight from the URL.
|
|
541
|
+
url-normalize --strip-fragment "https://example.com/article?id=1#intro"
|
|
542
|
+
# Output: https://example.com/article?id=1
|
|
543
|
+
|
|
544
|
+
# Display a readable URL.
|
|
545
|
+
url-normalize -H "https://xn--bcher-kva.de/r%C3%A9sum%C3%A9"
|
|
546
|
+
# Output: https://bücher.de/résumé
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
Use `url-normalize --help` for all options and `url-normalize --version` for the installed version. Normalization errors are written to standard error with exit status 1; invalid command-line arguments use exit status 2.
|
|
550
|
+
|
|
551
|
+
To run without a persistent tool installation:
|
|
552
|
+
|
|
553
|
+
```sh
|
|
554
|
+
uvx url-normalize "example.com"
|
|
555
|
+
# Output: https://example.com/
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
## Contributing
|
|
559
|
+
|
|
560
|
+
After cloning the repository, install the development dependencies and run the checks with [uv](https://docs.astral.sh/uv/):
|
|
561
|
+
|
|
562
|
+
```sh
|
|
563
|
+
uv sync --locked --extra dev
|
|
564
|
+
uv run --locked --extra dev python -m pytest
|
|
565
|
+
uv run --locked --extra dev pre-commit run --all-files
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
The test suite requires 100% statement and branch coverage. Branch coverage checks
|
|
569
|
+
that tests take both outcomes of measured decisions, such as enabling and
|
|
570
|
+
disabling query sorting. It does not establish that every input or combination
|
|
571
|
+
works, so please include exact-output regression tests with bug fixes. CI tests
|
|
572
|
+
Python 3.11 through 3.15 on Linux, macOS, and Windows.
|
|
573
|
+
|
|
574
|
+
The twice-monthly scheduled runs also test the newest compatible runtime
|
|
575
|
+
dependencies on Linux with Python 3.15. This checks dependency releases beyond
|
|
576
|
+
`uv.lock`, while regular matrix tests use the locked versions.
|
|
577
|
+
|
|
578
|
+
Pytest also runs every `python` code block in this README independently and checks
|
|
579
|
+
its printed output against the `# Output: ` comments. Include imports in each
|
|
580
|
+
block and keep those comments accurate. Shell commands and prose tables are not
|
|
581
|
+
executed by this check.
|
|
582
|
+
|
|
583
|
+
Type checking uses ty for the package and the public API consumer examples in `tests/typing`, targeting the minimum Python version from `requires-python` and all platforms. Those examples use `typing.assert_type` to check string, `None`, optional and narrowed inputs through the package exports; they are static checks, separate from pytest. The pre-commit hook uses the version and dependencies in `uv.lock`. To run type checking alone, use `uv run --locked --extra dev ty check`.
|
|
584
|
+
|
|
585
|
+
Alongside exact-output tests, Hypothesis generates 200 HTTP(S) URL and option combinations per property to check normalization idempotence, humanization round trips, and empty-component policy. Another 200 explicit relative references check stable output, slash options, and the absence of invented schemes or authorities. Generation is deterministic for a given Python version, Hypothesis version, and test definition. Add any minimized failures as explicit regression tests.
|
|
586
|
+
|
|
587
|
+
## Changelog and license
|
|
588
|
+
|
|
589
|
+
See [CHANGELOG.md](CHANGELOG.md) for release history and upgrade notes. Distributed under the [MIT License](LICENSE).
|
|
590
|
+
|
|
591
|
+
Inspired by Sam Ruby's [urlnorm.py](http://intertwingly.net/blog/2004/08/04/Urlnorm).
|