urlpurify 2.2.1 → 2.3.0

package/README.md

@@ -6,15 +6,25 @@
 
 Unwrap redirect, affiliate, and tracking wrapper URLs and strip tracking parameters. Turn noisy links into clean, direct URLs.
 
-Links from feeds, emails and social platforms rarely point straight at their destination. They pass through search redirects, link shims, email gateways and affiliate networks, and pick up tracking parameters on the way, the link decoration that follows a reader from site to site. urlpurify unwraps 380+ known wrappers and strips 100+ known tracking parameters plus vendor families. Campaign tags like `utm_*` and affiliate ids stay by default: they serve the site, not a tracker. It has no dependencies and runs in any modern JavaScript runtime, browsers included.
+Links from feeds, emails and social platforms rarely point straight at their destination. They pass through search redirects, link shims, email gateways and affiliate networks, and pick up tracking parameters on the way, the link decoration that follows a reader from site to site. urlpurify unwraps 400+ known wrappers and strips 100+ known tracking parameters plus vendor families. Campaign tags like `utm_*` and affiliate ids stay by default: they serve the site, not a tracker. It has no dependencies and runs in any modern JavaScript runtime, browsers included.
 
-## Installation
+[Quick start](#quick-start)
+  ·  
+[Unwrappers](#unwrappers)
+  ·  
+[Tracking parameters](#tracking-parameters)
+  ·  
+[API](#api)
+
+---
+
+## Quick start
 
 ```bash
 npm install urlpurify
 ```
 
-## Quick start
+Then:
 
 ```typescript
 import { cleanUrl } from 'urlpurify'
@@ -24,6 +34,55 @@ const url = 'https://www.google.com/url?q=https%3A%2F%2Fexample.com%2Fpost%3Ffbc
 cleanUrl(url) // => 'https://example.com/post'
 ```
 
+## Unwrappers
+
+There are more than 400 unwrappers, each in its own file in [src/unwraps](src/unwraps), with a comment naming the service and the URL shape it handles. Each is exported on its own and belongs to one category. The categories split by what unwrapping takes away from someone, which is what decides whether you want it:
+
+| Category | Export | Unwrapping removes | Default |
+| --- | --- | --- | :---: |
+| Search clicks | `searchClickUnwrappers` | The search engine's or aggregator's click logging | ☑️ |
+| Link shims | `linkShimUnwrappers` | Nothing, or a platform's click count | ☑️ |
+| Press releases | `pressReleaseUnwrappers` | The issuer's click stats from the wire | |
+| Sign-in shims | `signInShimUnwrappers` | Nothing, but the shim's page needs a sign-in | |
+| Security gateways | `securityGatewayUnwrappers` | The gateway's click-time check of the target | |
+| Email tracking | `emailTrackingUnwrappers` | The sender's click stats | |
+| Affiliate links | `affiliateUnwrappers` | The publisher's commission | |
+| Advertising | `advertisingUnwrappers` | Ad click and app install attribution | |
+| Click counters | `clickCounterUnwrappers` | A site's or feed's count of which links get clicked | |
+| Download measurement | `downloadMeasurementUnwrappers` | A podcast measurement prefix's download counts, or a site's download counts | |
+| Archives and proxies | `archiveProxyUnwrappers` | The archived or translated copy: you get the live page | |
+| Image proxies | `imageProxyUnwrappers` | The resized or optimized copy: you get the original image | |
+| Embed proxies | `embedProxyUnwrappers` | The site's frame around another provider's embed: you get the provider's page | |
+| Share intents | `shareIntentUnwrappers` | The share or bookmark form: you get the page being shared | |
+| Subscribe intents | `subscribeIntentUnwrappers` | The reader's or app's subscribe page, or a player carrying a feed: you get the feed | |
+| Feed tools | `feedToolUnwrappers` | The validator's or proxy's view: you get the feed it was given | |
+| Generic carriers | `genericCarrierUnwrappers` | The page of any site carrying a url in a `url`, `feedurl`, `moduleurl`, `feed.url`, `.url`, `link`, `redirect`, `to`, `go`, `goto` or `out` param, whether it redirects there or takes the url as an argument | |
+
+`defaultUnwrappers` combines the first two. Most of the others cost somebody something or change the page: a feed click counter tells a blog which posts get read, an affiliate link may pay for a small blog, and an archive link points at a copy on purpose.
+
+The intent categories, `shareIntentUnwrappers`, `subscribeIntentUnwrappers` and `feedToolUnwrappers`, hold links that never redirect to what they carry, so unwrapping one changes what a reader would open. They suit a caller collecting feeds or shared pages, not one cleaning links for a reader.
+
+`genericCarrierUnwrappers` reads those params on any host, so it also unwraps a feed generator or a feed that takes a url argument, such as `example.com/rss.php?url=<page>`. Exempt those links before calling it. Spread it last: the first unwrapper that matches wins, and a service's own unwrapper reads its real carrier better.
+
+To turn more on, see [Choosing unwrappers](#choosing-unwrappers).
+
+## Tracking parameters
+
+Tracking parameters are grouped like the unwrappers, by what stripping them takes away from someone:
+
+| Category | Export | Stripping removes | Default |
+| --- | --- | --- | :---: |
+| Click attribution | `clickAttributionParams` | Ad, search and platform click ids | ☑️ |
+| Visitor ids | `visitorIdParams` | Ids of one visitor, session, browser or email recipient | ☑️ |
+| Session ids | `sessionParams` | A server session id, which never changes what the page shows | ☑️ |
+| Housekeeping | `housekeepingParams` | Nothing: cache busters, previews, redirect leftovers and a `ref` naming the URL's own host | ☑️ |
+| Campaign tags | `campaignParams` | The site's own traffic stats, from tags like `utm_*` | |
+| Affiliate ids | `affiliateParams` | The publisher's commission | |
+
+`defaultTrackingParams` combines the first four. A campaign tag carries the same value for every reader, so it tells the site where a visit came from and nothing about who made it.
+
+To strip more, see [Choosing tracking parameters](#choosing-tracking-parameters).
+
 ## API
 
 ### `cleanUrl(url, options?)`
@@ -36,8 +95,16 @@ import { cleanUrl, defaultTrackingParams, defaultUnwrappers, unwrapWebArchive }
 cleanUrl(url, {
   // First match wins per pass.
   unwrappers: [...defaultUnwrappers, unwrapWebArchive],
-  // Names, or regexes tested on the lowercased name.
-  trackingParams: [...defaultTrackingParams, 'newsletter_id', /^cmp_[a-z0-9_]+$/],
+  // Names, regexes tested on the lowercased name, or entries.
+  trackingParams: [
+    ...defaultTrackingParams,
+    'newsletter_id',
+    /^cmp_[a-z0-9_]+$/,
+    // Only with a value of 32 hex digits.
+    { name: 'token', value: /^[0-9a-f]{32}$/ },
+    // Only on these hosts, and `/^/` matches every name.
+    { name: /^/, hosts: ['share.example.com', 'go.example.com'] },
+  ],
   // The default.
   maxUnwrapDepth: 12,
 })
@@ -51,14 +118,26 @@ Runs one pass of the unwrappers and returns the first target, or `undefined`.
 
 ### `stripTrackingParams(url, trackingParams?)`
 
-Removes tracking parameters and returns the serialized URL. Both this and `cleanUrl` apply three rules on top of the list:
+Removes tracking parameters and returns the serialized URL. Both this and `cleanUrl` apply two rules on top of the list:
 
-- `ref` is dropped when it names the URL's own host, as in Ghost's `?ref=example.com`, and kept otherwise.
 - A query with a signature (`sig`, `signature`, `X-Amz-Signature` or `X-Goog-Signature`) stays whole, since dropping anything breaks the signature.
 - `ts` stays on Alibaba DirectMail click URLs, which fail without it.
 
+A literal name the list holds is also dropped as a path parameter, as in `/page;jsessionid=<id>`. An entry's `value` is a regex or a function `(value, host) => boolean` tested on the decoded value, and its `hosts` are exact hostnames or a regex, matched without `www.`.
+
 A tracking list is compiled on first use and cached per array. To change it, pass a new array.
 
+### `isTrackingParam(name, value, hostname, trackingParams?)`
+
+Answers whether one query pair is tracking, for a caller filtering pairs itself. The list defaults to `defaultTrackingParams`. It sees one query pair and never the path, so the two rules above don't apply and a `;name=value` path param needs `cleanUrl` or `stripTrackingParams`.
+
+```typescript
+import { isTrackingParam, knownTrackingParams } from 'urlpurify'
+
+isTrackingParam('sid', '46551', 'example.com') // false
+isTrackingParam('utm_source', 'feed', url.hostname, knownTrackingParams) // true
+```
+
 ### `createParamExtractor(config)`
 
 Builds an unwrapper for a target that sits in a query parameter:
@@ -78,52 +157,24 @@ cleanUrl(url, { unwrappers: [...defaultUnwrappers, unwrapExample] })
 
 A value still encoded after one decode, starting with `http%3A` or `https%3A`, is decoded once more. A malformed escape, or one in a legacy charset such as `%E9` for é in Latin-1, stays as written. For any other shape, write a plain `UrlUnwrapper`: a function that takes a `URL` and returns the target or `undefined`.
 
-### Tracking parameters
+### Choosing unwrappers
 
-Tracking parameters are grouped like the unwrappers, by what stripping them takes away from someone:
-
-| Category | Export | Stripping removes | Default |
-| --- | --- | --- | :---: |
-| Click attribution | `clickAttributionParams` | Ad, search and platform click ids | ☑️ |
-| Visitor ids | `visitorIdParams` | Ids of one visitor, session, browser or email recipient | ☑️ |
-| Housekeeping | `housekeepingParams` | Nothing: cache busters, previews and redirect leftovers | ☑️ |
-| Campaign tags | `campaignParams` | The site's own traffic stats, from tags like `utm_*` | |
-| Affiliate ids | `affiliateParams` | The publisher's commission | |
-
-`defaultTrackingParams` combines the first three. A campaign tag carries the same value for every reader, so it tells the site where a visit came from and nothing about who made it. To strip more, spread them next to the defaults:
+Spread the categories you want next to the defaults:
 
 ```typescript
-import { campaignParams, defaultTrackingParams, stripTrackingParams } from 'urlpurify'
+import { affiliateUnwrappers, cleanUrl, defaultUnwrappers } from 'urlpurify'
 
-stripTrackingParams(url, [...defaultTrackingParams, ...campaignParams])
+cleanUrl(url, { unwrappers: [...defaultUnwrappers, ...affiliateUnwrappers] })
 ```
 
-`trackingParamsLiterals` and `trackingParamsPatterns` hold every known parameter, opt-in categories included, split into names and regexes.
-
-## Unwrappers
+### Choosing tracking parameters
 
-There are more than 380 unwrappers, each in its own file in [src/unwraps](src/unwraps), with a comment naming the service and the URL shape it handles. Each is exported on its own and belongs to one category. The categories split by what unwrapping takes away from someone, which is what decides whether you want it:
-
-| Category | Export | Unwrapping removes | Default |
-| --- | --- | --- | :---: |
-| Search clicks | `searchClickUnwrappers` | The search engine's or aggregator's click logging | ☑️ |
-| Link shims | `linkShimUnwrappers` | Nothing, or a platform's click count | ☑️ |
-| Press releases | `pressReleaseUnwrappers` | The issuer's click stats from the wire | |
-| Sign-in shims | `signInShimUnwrappers` | Nothing, but the shim's page needs a sign-in | |
-| Security gateways | `securityGatewayUnwrappers` | The gateway's click-time check of the target | |
-| Email tracking | `emailTrackingUnwrappers` | The sender's click stats | |
-| Affiliate links | `affiliateUnwrappers` | The publisher's commission | |
-| Advertising | `advertisingUnwrappers` | Ad click and app install attribution | |
-| Click counters | `clickCounterUnwrappers` | A site's or feed's count of which links get clicked | |
-| Download measurement | `downloadMeasurementUnwrappers` | A podcast measurement prefix's download counts, or a site's download counts | |
-| Archives and proxies | `archiveProxyUnwrappers` | The archived or translated copy: you get the live page | |
-| Image proxies | `imageProxyUnwrappers` | The resized or optimized copy: you get the original image | |
-| Embed proxies | `embedProxyUnwrappers` | The site's frame around another provider's embed: you get the provider's page | |
-
-`defaultUnwrappers` combines the first two. The rest cost somebody something or change the page: a feed click counter tells a blog which posts get read, an affiliate link may pay for a small blog, and an archive link points at a copy on purpose. To turn more on, spread them next to the defaults:
+Spread the categories you want next to the defaults:
 
 ```typescript
-import { affiliateUnwrappers, cleanUrl, defaultUnwrappers } from 'urlpurify'
+import { campaignParams, defaultTrackingParams, stripTrackingParams } from 'urlpurify'
 
-cleanUrl(url, { unwrappers: [...defaultUnwrappers, ...affiliateUnwrappers] })
+stripTrackingParams(url, [...defaultTrackingParams, ...campaignParams])
 ```
+
+`knownTrackingParams` holds every known parameter, opt-in categories included. `trackingParamsLiterals` and `trackingParamsPatterns` split its names and regexes, and leave out entries, whose name alone would strip `sid` or `ref` everywhere.

package/dist/clean.d.mts

@@ -1,4 +1,5 @@
 import { CleanUrlOptions, TrackingParam, UrlUnwrapper } from "./types.mjs";
 export declare const unwrapUrl: (url: string, unwrappers?: Array<UrlUnwrapper>) => string | undefined;
 export declare const stripTrackingParams: (url: string, trackingParams?: Array<TrackingParam>) => string;
+export declare const isTrackingParam: (name: string, value: string | undefined, hostname: string, trackingParams?: Array<TrackingParam>) => boolean;
 export declare const cleanUrl: (url: string, options?: CleanUrlOptions) => string;

package/dist/clean.mjs

@@ -1,49 +1,81 @@
 import { isAnyOf } from "./node_modules/trousse/dist/matching.mjs";
-import { fixMalformedScheme, isHostOf, isHttpUrl, isIpAddress, normalizeUrl, parseUrl, stripWww } from "./node_modules/trousse/dist/urls.mjs";
+import { isHostOf, isHostname, isIpAddress, normalizeUrl, parseUrl, resolveUrl, stripWww } from "./node_modules/trousse/dist/urls.mjs";
+import { isString } from "./node_modules/trousse/dist/is.mjs";
 import { defaultTrackingParams, defaultUnwrappers } from "./defaults.mjs";
 // src/clean.ts
-const refDelimitersRegex = /[\s/?#\\:@]/;
 const signatureParams = [
 	"sig",
 	"signature",
 	"x-amz-signature",
 	"x-goog-signature"
 ];
+const pathParamRegex = /;([^;/=]+)=[^;/]*/g;
 const trackingMatcherCache = /* @__PURE__ */ new WeakMap();
 const getTrackingMatcher = (params) => {
 	let cached = trackingMatcherCache.get(params);
 	if (!cached) {
 		const literals = /* @__PURE__ */ new Set();
 		const patterns = [];
-		for (const param of params) if (param instanceof RegExp) patterns.push(param);
-		else literals.add(param.toLowerCase());
+		const entries = [];
+		for (const param of params) {
+			if (isString(param)) {
+				literals.add(param.toLowerCase());
+				continue;
+			}
+			if (param instanceof RegExp) {
+				patterns.push(param);
+				continue;
+			}
+			entries.push(param);
+		}
 		cached = {
 			literals,
-			patterns
+			patterns,
+			entries
 		};
 		trackingMatcherCache.set(params, cached);
 	}
 	return cached;
 };
+const matchesPattern = (text, pattern) => {
+	return text.search(pattern) !== -1;
+};
+const matchesEntry = (entry, name, value, host) => {
+	if (!(isString(entry.name) ? entry.name.toLowerCase() === name : matchesPattern(name, entry.name))) return false;
+	if (entry.hosts instanceof RegExp && !matchesPattern(host, entry.hosts)) return false;
+	if (Array.isArray(entry.hosts) && !entry.hosts.includes(host)) return false;
+	if (entry.value instanceof RegExp) return matchesPattern(value, entry.value);
+	if (entry.value) return entry.value(value, host);
+	return true;
+};
+const isTrackedPair = (matcher, name, value, host) => {
+	if (matcher.literals.has(name)) return true;
+	if (matcher.patterns.some((pattern) => matchesPattern(name, pattern))) return true;
+	return matcher.entries.some((entry) => matchesEntry(entry, name, value, host));
+};
+const deletePathParams = (url, matcher) => {
+	const pathname = url.pathname.replace(pathParamRegex, (param, name) => {
+		if (matcher.literals.has(name.toLowerCase())) return "";
+		return param;
+	});
+	if (pathname !== url.pathname) url.pathname = pathname;
+};
 const deleteTrackingParams = (url, trackingParams) => {
-	if (!url.search) return;
 	const pairs = url.search.slice(1).split("&");
 	const entries = pairs.map((pair) => new URLSearchParams(pair).entries().next().value);
 	for (const entry of entries) if (entry && isAnyOf(entry[0], signatureParams)) return;
 	const matcher = getTrackingMatcher(trackingParams);
+	deletePathParams(url, matcher);
+	if (!url.search) return;
 	const host = stripWww(url.hostname);
-	const keepsTs = isHostOf(url, "dm-cn.aliyuncs.com") && url.pathname === "/trace/v1/report";
+	const keepsTs = isHostOf(url, "dm-cn.aliyuncs.com") && isAnyOf(url.pathname, "/trace/v1/report");
 	const keptPairs = pairs.filter((_pair, index) => {
 		const entry = entries[index];
 		if (!entry) return true;
 		const [key, value] = entry;
 		const name = key.toLowerCase();
 		if (keepsTs && name === "ts") return true;
-		if (matcher.literals.has(name) || matcher.patterns.some((pattern) => name.search(pattern) !== -1)) return false;
-		if (name !== "ref" || refDelimitersRegex.test(value)) return true;
-		const refUrl = parseUrl(`http://${value}`);
-		if (!refUrl) return true;
-		return stripWww(refUrl.hostname) !== host;
+		return !isTrackedPair(matcher, name, value, host);
 	});
 	if (keptPairs.length === pairs.length) return;
 	url.search = keptPairs.join("&");
@@ -53,10 +85,13 @@ const dottedHostRegex = /[^.]\.[^.]/;
 const fileExtensionHostRegex = /\.(?:asp|aspx|cgi|htm|html|jsp|jspa|php)$/;
 const cleanTarget = (target) => {
 	if (!target) return;
-	const repaired = fixMalformedScheme(target.trim());
-	if (controlCharactersRegex.test(repaired)) return;
+	const trimmed = target.trim();
+	if (controlCharactersRegex.test(trimmed)) return;
+	const repaired = resolveUrl(trimmed);
+	if (!repaired) return;
 	const hostname = parseUrl(repaired)?.hostname ?? "";
 	if (!dottedHostRegex.test(hostname) && !isIpAddress(hostname)) return;
+	if (!isHostname(hostname)) return;
 	if (fileExtensionHostRegex.test(hostname)) return;
 	return repaired;
 };
@@ -70,10 +105,7 @@ const addMissingScheme = (target) => {
 };
 const applyUnwrappers = (url, unwrappers) => {
 	for (const unwrap of unwrappers) {
-		const unwrapped = addMissingScheme(unwrap(url)?.trim());
-		const raw = unwrapped && !isHttpUrl(unwrapped) ? fixMalformedScheme(unwrapped) : unwrapped;
-		if (!raw || !isHttpUrl(raw)) continue;
-		const target = cleanTarget(raw);
+		const target = cleanTarget(addMissingScheme(unwrap(url)?.trim()));
 		if (target) return target;
 	}
 	const collapsed = parseUrl(normalizeUrl(url.href, { collapseSlashes: true }));
@@ -93,6 +125,11 @@ const stripTrackingParams = (url, trackingParams = defaultTrackingParams) => {
 	deleteTrackingParams(parsed, trackingParams);
 	return parsed.href;
 };
+const isTrackingParam = (name, value, hostname, trackingParams = defaultTrackingParams) => {
+	const matcher = getTrackingMatcher(trackingParams);
+	const host = stripWww(hostname.toLowerCase());
+	return isTrackedPair(matcher, name.toLowerCase(), value ?? "", host);
+};
 const cleanUrl = (url, options) => {
 	const unwrappers = options?.unwrappers ?? defaultUnwrappers;
 	const trackingParams = options?.trackingParams ?? defaultTrackingParams;
@@ -110,4 +147,4 @@ const cleanUrl = (url, options) => {
 	deleteTrackingParams(currentParsed, trackingParams);
 	return currentParsed.href;
 };
-export { cleanUrl, stripTrackingParams, unwrapUrl };
+export { cleanUrl, isTrackingParam, stripTrackingParams, unwrapUrl };

package/dist/defaults.d.mts

@@ -3,8 +3,10 @@ import { affiliateParams } from "./tracking/affiliate.mjs";
 import { campaignParams } from "./tracking/campaign.mjs";
 import { clickAttributionParams } from "./tracking/clickAttribution.mjs";
 import { housekeepingParams } from "./tracking/housekeeping.mjs";
+import { sessionParams } from "./tracking/session.mjs";
 import { visitorIdParams } from "./tracking/visitorId.mjs";
 export declare const defaultTrackingParams: Array<TrackingParam>;
+export declare const knownTrackingParams: Array<TrackingParam>;
 export declare const trackingParamsLiterals: Array<string>;
 export declare const trackingParamsPatterns: Array<RegExp>;
 export declare const searchClickUnwrappers: Array<UrlUnwrapper>;
@@ -20,5 +22,9 @@ export declare const downloadMeasurementUnwrappers: Array<UrlUnwrapper>;
 export declare const archiveProxyUnwrappers: Array<UrlUnwrapper>;
 export declare const imageProxyUnwrappers: Array<UrlUnwrapper>;
 export declare const embedProxyUnwrappers: Array<UrlUnwrapper>;
+export declare const shareIntentUnwrappers: Array<UrlUnwrapper>;
+export declare const subscribeIntentUnwrappers: Array<UrlUnwrapper>;
+export declare const feedToolUnwrappers: Array<UrlUnwrapper>;
+export declare const genericCarrierUnwrappers: Array<UrlUnwrapper>;
 export declare const defaultUnwrappers: Array<UrlUnwrapper>;
-export { affiliateParams, campaignParams, clickAttributionParams, housekeepingParams, visitorIdParams };
+export { affiliateParams, campaignParams, clickAttributionParams, housekeepingParams, sessionParams, visitorIdParams };