urlpurify 1.4.2 → 1.6.0

package/README.md

@@ -6,7 +6,7 @@
 
 Unwrap redirect, affiliate, and tracking wrapper URLs and strip tracking parameters. Turn noisy links into clean, direct URLs.
 
-URLs collected from feeds, emails, and social platforms rarely point straight at their destination: they pass through search-engine redirects, email security gateways, affiliate networks, and AMP caches, and carry analytics parameters that bloat the link and leak data. urlpurify ships 70+ unwrappers for known wrapper services and a list of 150+ known tracking parameters, plus a composite `cleanUrl()` that applies both. It has zero dependencies and runs in any modern JavaScript runtime, including browsers.
+Links from feeds, emails and social platforms rarely point straight at their destination. They pass through search redirects, link shims, email gateways and affiliate networks, and pick up tracking parameters on the way, the link decoration that follows a reader from site to site. urlpurify unwraps 380+ known wrappers and strips 170+ known tracking parameters plus families like `utm_*`. It has no dependencies and runs in any modern JavaScript runtime, browsers included.
 
 ## Installation
 
@@ -14,111 +14,95 @@ URLs collected from feeds, emails, and social platforms rarely point straight at
 npm install urlpurify
 ```
 
-## Quick Start
+## Quick start
 
 ```typescript
 import { cleanUrl } from 'urlpurify'
 
-cleanUrl('https://www.google.com/url?q=https%3A%2F%2Fexample.com%2Fpost%3Futm_source%3Dnewsletter')
-// => 'https://example.com/post'
+const url = 'https://www.google.com/url?q=https%3A%2F%2Fexample.com%2Fpost%3Futm_source%3Dnewsletter'
+
+cleanUrl(url) // => 'https://example.com/post'
 ```
 
 ## API
 
 ### `cleanUrl(url, options?)`
 
-Unwraps redirect wrappers (repeatedly, since wrappers can nest), then strips tracking parameters. When the input cannot be parsed as a URL or nothing applies, the input string is returned unchanged, so the result is always safe to display.
+Unwraps wrappers, repeatedly since they nest, then strips tracking parameters. The result is serialized per the URL Standard, and input that can't be parsed comes back unchanged, so the result is always safe to display.
 
 ```typescript
-import { cleanUrl, defaultUnwrappers, unwrapWebArchive } from 'urlpurify'
+import { cleanUrl, defaultTrackingParams, defaultUnwrappers, unwrapWebArchive } from 'urlpurify'
 
-const cleaned = cleanUrl(url, {
-  // Unwrappers to apply, first match wins per pass (omit to use defaults).
+cleanUrl(url, {
+  // First match wins per pass.
   unwrappers: [...defaultUnwrappers, unwrapWebArchive],
-  // Param names (matched case-insensitively) or anchored regexes tested against the
-  // lowercased name (omit to use defaults).
-  trackingParams: ['fbclid', /^utm_[a-z0-9_-]+$/],
-  // Maximum number of unwrap passes for nested wrappers. Defaults to 3.
-  maxUnwrapDepth: 3,
+  // Names, or regexes tested on the lowercased name.
+  trackingParams: [...defaultTrackingParams, 'newsletter_id', /^cmp_[a-z0-9_]+$/],
+  // The default.
+  maxUnwrapDepth: 6,
 })
 ```
 
+An unwrapped target that isn't a usable URL is ignored, and the wrapper is kept.
+
 ### `unwrapUrl(url, unwrappers?)`
 
-Applies the unwrappers in order (one pass) and returns the first extracted target, or `undefined` when none match or the input cannot be parsed.
+Runs one pass of the unwrappers and returns the first target, or `undefined`.
 
 ### `stripTrackingParams(url, trackingParams?)`
 
-Removes matching query parameters and returns the cleaned URL. The input is returned unchanged when nothing matches or it cannot be parsed.
+Removes tracking parameters and returns the serialized URL. Both this and `cleanUrl` apply three rules on top of the list:
 
-Both `cleanUrl` and `stripTrackingParams` also drop a `ref` parameter when its value is the URL's own host (Ghost's self-referral `?ref=example.com` on `example.com`), regardless of the tracking list passed. With any other value `ref` is left alone, since it is often a real referral target.
+- `ref` is dropped when it names the URL's own host, as in Ghost's `?ref=example.com`, and kept otherwise.
+- A query with a signature (`sig`, `signature`, `X-Amz-Signature` or `X-Goog-Signature`) stays whole, since dropping anything breaks the signature.
+- `ts` stays on Alibaba DirectMail click URLs, which fail without it.
+
+A tracking list is compiled on first use and cached per array. To change it, pass a new array.
 
 ### `createParamExtractor(config)`
 
-Builds an unwrapper for the common case where the target URL sits in a query parameter:
+Builds an unwrapper for a target that sits in a query parameter:
 
 ```typescript
 import { cleanUrl, createParamExtractor, defaultUnwrappers } from 'urlpurify'
 
 const unwrapExample = createParamExtractor({
-  hosts: 'go.example.com', // Also accepts an array of hosts or a regex.
+  // The domain and every subdomain. Use `hosts` for exact hosts or a regex.
+  domains: 'example.com',
+  path: '/out',
   params: ['target'],
 })
 
 cleanUrl(url, { unwrappers: [...defaultUnwrappers, unwrapExample] })
 ```
 
-For wrappers that encode the target (base64 path segments, custom escaping), write a plain function of type `UrlUnwrapper`: it receives a `URL` and returns the target string or `undefined`.
+A value still encoded after one decode, starting with `http%3A` or `https%3A`, is decoded once more. For any other shape, write a plain `UrlUnwrapper`: a function that takes a `URL` and returns the target or `undefined`.
+
+### Tracking parameters
 
-### Defaults
+`defaultTrackingParams` holds literal names plus regexes for vendor families like `utm_*`. The two halves are also exported as `trackingParamsLiterals` and `trackingParamsPatterns`.
 
-`defaultTrackingParams` is the combined default list for `cleanUrl` and `stripTrackingParams`: literal names plus family regexes like `/^utm_[a-z0-9_-]+$/` that cover vendor namespaces where new variants keep appearing. Its parts are exported separately as `trackingParamsLiterals` (strings only, for consumers that need a plain string list) and `trackingParamsPatterns`. `defaultUnwrappers` is exported alongside. `defaultUnwrappers` enables a conservative subset of the catalog; everything else is exported individually for explicit opt-in.
+## Unwrappers
 
-**On by default:** redirects that only track the click — search-engine redirects and platform shims. Unwrapping them costs the destination nothing.
+There are more than 380 unwrappers, each in its own file in [src/unwraps](src/unwraps), with a comment naming the service and the URL shape it handles. Each is exported on its own and belongs to one category. The categories split by what unwrapping takes away from someone, which is what decides whether you want it:
 
-**Off by default:** affiliate and referral wrappers (`unwrapSkimlinks`, `unwrapAwin`, `unwrapShareasale`, `unwrapAmazonAffiliate`, `unwrapViglink`, and the rest). The reader lands on the same page either way — unwrapping only removes the writer's commission, which for a small blog is the money paying for the work. Opt in explicitly if you want them.
+| Category | Export | Unwrapping removes | Default |
+| --- | --- | --- | :---: |
+| Search clicks | `searchClickUnwrappers` | The search engine's or aggregator's click logging | ☑️ |
+| Link shims | `linkShimUnwrappers` | Nothing, or a platform's click count or a site click counter | ☑️ |
+| Press releases | `pressReleaseUnwrappers` | The wire's click stats | ☑️ |
+| Sign-in shims | `signInShimUnwrappers` | Nothing, but the shim's page needs a sign-in | |
+| Security gateways | `securityGatewayUnwrappers` | The gateway's click-time check of the target | |
+| Email tracking | `emailTrackingUnwrappers` | The sender's click stats | |
+| Affiliate links | `affiliateUnwrappers` | The publisher's commission | |
+| Advertising | `advertisingUnwrappers` | Ad click and app install attribution | |
+| Download measurement | `downloadMeasurementUnwrappers` | A podcast measurement prefix's download counts, or a site's download or click counts | |
+| Archives and proxies | `archiveProxyUnwrappers` | The archived or translated copy: you get the live page | |
 
-Prevalence is not the test: affiliate wrappers are more common than tracking shims.
+`defaultUnwrappers` combines the first three. The rest cost somebody something or change the page: an affiliate link may pay for a small blog, and an archive link points at a copy on purpose. To turn more on, spread them next to the defaults:
 
-## Unwrappers
+```typescript
+import { affiliateUnwrappers, cleanUrl, defaultUnwrappers } from 'urlpurify'
 
-Enabled by default:
-
-| Unwrapper | Description |
-| --- | --- |
-| `unwrapBing` | Bing search-result redirect (www.bing.com/ck/a?u=a1\<base64url\>) |
-| `unwrapAnonymTo` | anonym.to referrer anonymizer (anonym.to/?\<target\>) |
-| `unwrapBlueskyRedirect` | Bluesky outbound link redirect (go.bsky.app/redirect?u=\<target\>) |
-| `unwrapCsdn` | CSDN external link redirect (link.csdn.net/?target=\<target\>) |
-| `unwrapDeviantartOutgoing` | DeviantArt outbound link shim (deviantart.com/\<user\>/outgoing?\<target\>) |
-| `unwrapDisqus` | Disqus outbound link redirect (disq.us/?url=\<target\>) |
-| `unwrapDouban` | Douban external link redirect (www.douban.com/link2/?url=\<target\>) |
-| `unwrapFacebookShim` | Meta link shim (l.facebook.com / l.messenger.com /l.php?u=\<target\>) |
-| `unwrapFlipboard` | Flipboard outbound redirect (flipboard.com/redirect?url=\<target\>) |
-| `unwrapGitee` | Gitee external link redirect (gitee.com/link?target=\<target\>) |
-| `unwrapGoogle` | Google redirect (google.\<TLD\>/url?url=\<target\> or ?q=\<target\>) |
-| `unwrapGoogleAmpViewer` | Google AMP viewer (www.google.\<TLD\>/amp/s/\<host\>/\<path\>) |
-| `unwrapGoogleNews` | Google News legacy redirect (news.google.\<TLD\>/news/url?url=\<target\>) |
-| `unwrapGoogleNewsModern` | Google News modern article URLs (news.google.com/articles/\<base64\>) |
-| `unwrapGoogleScholar` | Google Scholar search-result redirect (scholar.google.\<TLD\>/scholar_url?url=\<target\>) |
-| `unwrapHashnode` | Hashnode outbound redirect (hashnode.com/util/redirect?url=\<target\>) |
-| `unwrapHrefLi` | href.li referrer stripper (href.li/?\<target\>), used by Tumblr |
-| `unwrapInstagramShim` | Instagram outbound link shim (l.instagram.com with ?u=\<target\>) |
-| `unwrapJianshuGo` | Jianshu external link redirect (links.jianshu.com/go?to=\<target\>) |
-| `unwrapJuejin` | Juejin external link redirect (link.juejin.cn/?target=\<target\>) |
-| `unwrapMedium` | Medium outbound link redirect (medium.com/r/?url=\<target\>) |
-| `unwrapMozillaOutgoing` | Mozilla outgoing-link redirector (outgoing.prod.mozaws.net/v1/\<hash\>/\<target\>) |
-| `unwrapPocket` | Pocket redirect (getpocket.com/redirect?url=\<target\>) |
-| `unwrapNaverOutgoing` | Naver outbound link redirect (cc.loginfra.com/...?u=\<target\>) |
-| `unwrapRedditOut` | Reddit outbound click tracker (out.reddit.com/?url=\<target\>) |
-| `unwrapSegmentfault` | Segmentfault external link redirect (link.segmentfault.com/?enc=\<base64\>) |
-| `unwrapSspai` | Sspai external link redirect (sspai.com/link?target=\<target\>) |
-| `unwrapSteamLinkfilter` | Steam outbound link filter (steamcommunity.com/linkfilter/?url=\<target\>) |
-| `unwrapThreadsShim` | Threads outbound link shim (l.threads.com / l.threads.net with ?u=\<target\>) |
-| `unwrapTumblr` | Tumblr outbound redirect (t.umblr.com/redirect?z=\<target\>) |
-| `unwrapVkAway` | VK away redirect (vk.com/away.php?to=\<target\>) |
-| `unwrapYahooSearch` | Yahoo Search redirect (r.search.yahoo.com/.../RU=\<target\>/RK=...) |
-| `unwrapYouTube` | YouTube external redirect (www.youtube.com/redirect?q=\<target\>) |
-| `unwrapZhihu` | Zhihu external redirect (link.zhihu.com/?target=\<target\>) |
-
-Many more are available for explicit opt-in: email security gateways (Outlook SafeLinks, Proofpoint, Mimecast), affiliate networks (Awin, Skimlinks, Commission Junction), CJK platforms, AMP caches, and others. See [src/unwraps](src/unwraps) for the full catalog, each documented in its source file.
+cleanUrl(url, { unwrappers: [...defaultUnwrappers, ...affiliateUnwrappers] })
+```

package/dist/clean.mjs

@@ -1,6 +1,14 @@
-import { parseUrl, stripWww } from "./node_modules/trousse/dist/urls.mjs";
+import { fixMalformedProtocol, isHttpUrl, isIpAddress, parseUrl, stripWww } from "./node_modules/trousse/dist/urls.mjs";
 import { defaultTrackingParams, defaultUnwrappers } from "./defaults.mjs";
 //#region src/clean.ts
+const replacementCharacter = "�";
+const refDelimitersRegex = /[\s/?#\\:@]/;
+const signatureParams = [
+	"sig",
+	"signature",
+	"x-amz-signature",
+	"x-goog-signature"
+];
 const trackingMatcherCache = /* @__PURE__ */ new WeakMap();
 const getTrackingMatcher = (params) => {
 	let cached = trackingMatcherCache.get(params);
@@ -18,52 +26,77 @@ const getTrackingMatcher = (params) => {
 	return cached;
 };
 const deleteTrackingParams = (url, trackingParams) => {
-	if (!url.search) return false;
+	if (!url.search) return;
+	const pairs = url.search.slice(1).split("&");
+	const entries = pairs.map((pair) => new URLSearchParams(pair).entries().next().value);
+	for (const entry of entries) if (entry && signatureParams.includes(entry[0].toLowerCase())) return;
 	const matcher = getTrackingMatcher(trackingParams);
-	const keysToDelete = [];
-	for (const key of url.searchParams.keys()) {
+	const host = stripWww(url.hostname);
+	const keepsTs = url.hostname === "dm-cn.aliyuncs.com" && url.pathname === "/trace/v1/report";
+	const keptPairs = pairs.filter((_pair, index) => {
+		const entry = entries[index];
+		if (!entry) return true;
+		const [key, value] = entry;
 		const name = key.toLowerCase();
-		if (matcher.literals.has(name) || matcher.patterns.some((pattern) => pattern.test(name))) keysToDelete.push(key);
-	}
-	for (const key of keysToDelete) url.searchParams.delete(key);
-	return keysToDelete.length > 0;
+		if (keepsTs && name === "ts") return true;
+		if (matcher.literals.has(name) || matcher.patterns.some((pattern) => name.search(pattern) !== -1)) return false;
+		if (name !== "ref" || refDelimitersRegex.test(value)) return true;
+		try {
+			return stripWww(new URL(`http://${value}`).hostname) !== host;
+		} catch {}
+		return true;
+	});
+	if (keptPairs.length === pairs.length) return;
+	url.search = keptPairs.join("&");
 };
-const stripHostPrefix = (host) => {
-	return stripWww(host).toLowerCase();
+const controlCharactersRegex = /[\p{Cc}\u2028\u2029]/u;
+const dottedHostRegex = /[^.]\.[^.]/;
+const fileExtensionHostRegex = /\.(?:asp|aspx|cgi|htm|html|jsp|jspa|php)$/;
+const cleanTarget = (target) => {
+	if (!target) return;
+	const repaired = fixMalformedProtocol(target.trim());
+	if (controlCharactersRegex.test(repaired)) return;
+	const hostname = parseUrl(repaired)?.hostname ?? "";
+	if (!dottedHostRegex.test(hostname) && !isIpAddress(hostname)) return;
+	if (fileExtensionHostRegex.test(hostname)) return;
+	return repaired;
 };
-const deleteSelfReferentialParams = (url) => {
-	const value = url.searchParams.get("ref");
-	if (!value || stripHostPrefix(value) !== stripHostPrefix(url.hostname)) return false;
-	url.searchParams.delete("ref");
-	return true;
+const protocolRelativeRegex = /^\/\/[^/]/;
+const addMissingScheme = (target) => {
+	if (!target || !protocolRelativeRegex.test(target)) return target;
+	return `https:${target}`;
 };
 const applyUnwrappers = (url, unwrappers) => {
 	for (const unwrap of unwrappers) {
-		const target = unwrap(url);
+		const raw = addMissingScheme(unwrap(url));
+		if (!raw || !isHttpUrl(raw)) continue;
+		const target = cleanTarget(raw);
 		if (target) return target;
 	}
 };
 const unwrapUrl = (url, unwrappers = defaultUnwrappers) => {
 	const parsed = parseUrl(url);
 	if (!parsed) return;
-	return applyUnwrappers(parsed, unwrappers);
+	const target = applyUnwrappers(parsed, unwrappers);
+	if (!target || target.includes(replacementCharacter)) return;
+	return parseUrl(target)?.href;
 };
 const stripTrackingParams = (url, trackingParams = defaultTrackingParams) => {
 	const parsed = parseUrl(url);
 	if (!parsed) return url;
-	const trackingRemoved = deleteTrackingParams(parsed, trackingParams);
-	const selfReferentialRemoved = deleteSelfReferentialParams(parsed);
-	if (trackingRemoved || selfReferentialRemoved) return parsed.toString();
-	return url;
+	deleteTrackingParams(parsed, trackingParams);
+	return parsed.href;
 };
 const cleanUrl = (url, options) => {
 	const unwrappers = options?.unwrappers ?? defaultUnwrappers;
 	const trackingParams = options?.trackingParams ?? defaultTrackingParams;
-	const maxUnwrapDepth = options?.maxUnwrapDepth ?? 3;
+	const maxUnwrapDepth = options?.maxUnwrapDepth ?? 6;
 	const parsed = parseUrl(url);
 	if (!parsed) return url;
 	let currentUrl = url;
 	let currentParsed = parsed;
+	let intactUrl = url;
+	let intactParsed = parsed;
 	for (let depth = 0; depth < maxUnwrapDepth; depth += 1) {
 		const target = applyUnwrappers(currentParsed, unwrappers);
 		if (!target) break;
@@ -71,11 +104,17 @@ const cleanUrl = (url, options) => {
 		if (!targetParsed) break;
 		currentUrl = target;
 		currentParsed = targetParsed;
+		if (!target.includes(replacementCharacter)) {
+			intactUrl = target;
+			intactParsed = targetParsed;
+		}
+	}
+	if (currentUrl.includes(replacementCharacter)) {
+		currentUrl = intactUrl;
+		currentParsed = intactParsed;
 	}
-	const trackingRemoved = deleteTrackingParams(currentParsed, trackingParams);
-	const selfReferentialRemoved = deleteSelfReferentialParams(currentParsed);
-	if (trackingRemoved || selfReferentialRemoved) return currentParsed.toString();
-	return currentUrl;
+	deleteTrackingParams(currentParsed, trackingParams);
+	return currentParsed.href;
 };
 //#endregion
 export { cleanUrl, stripTrackingParams, unwrapUrl };

package/dist/defaults.d.mts

@@ -3,6 +3,16 @@ import { trackingParamsLiterals } from "./tracking/literals.mjs";
 import { trackingParamsPatterns } from "./tracking/patterns.mjs";
 //#region src/defaults.d.ts
 export declare const defaultTrackingParams: Array<TrackingParam>;
+export declare const searchClickUnwrappers: Array<UrlUnwrapper>;
+export declare const linkShimUnwrappers: Array<UrlUnwrapper>;
+export declare const pressReleaseUnwrappers: Array<UrlUnwrapper>;
+export declare const signInShimUnwrappers: Array<UrlUnwrapper>;
+export declare const securityGatewayUnwrappers: Array<UrlUnwrapper>;
+export declare const emailTrackingUnwrappers: Array<UrlUnwrapper>;
+export declare const affiliateUnwrappers: Array<UrlUnwrapper>;
+export declare const advertisingUnwrappers: Array<UrlUnwrapper>;
+export declare const downloadMeasurementUnwrappers: Array<UrlUnwrapper>;
+export declare const archiveProxyUnwrappers: Array<UrlUnwrapper>;
 export declare const defaultUnwrappers: Array<UrlUnwrapper>;
 //#endregion
 export { trackingParamsLiterals, trackingParamsPatterns };