feedcanon 1.0.0-beta.2 → 1.0.0

package/README.md

@@ -4,22 +4,59 @@
 [![npm version](https://img.shields.io/npm/v/feedcanon.svg)](https://www.npmjs.com/package/feedcanon)
 [![license](https://img.shields.io/npm/l/feedcanon.svg)](https://github.com/macieklamberski/feedcanon/blob/main/LICENSE)
 
-Find the canonical URL for any web feed by comparing actual content. Turn messy feed URLs into their cleanest, most reliable form.
+Find the canonical URL for any web feed by comparing actual content. Turn messy feed URLs into their cleanest form.
 
-Many URLs can point to the same feed—varying by protocol, www prefixes, trailing slashes, order of params, or domain aliases. Feedcanon compares actual feed content, considers the feed's declared self URL, and tests simpler URL alternatives to find the cleanest working one. Perfect for feed readers that need consistent, deduplicated subscriptions.
+Many URLs can point to the same feed, varying by protocol, www prefixes, trailing slashes, order of params, or domain aliases. Feedcanon compares actual feed content, respects the feed's declared self URL, and tests simpler URL alternatives to find the cleanest working one.
+
+Perfect for feed readers to deduplicate subscriptions when users add the same feed via different URLs.
+
+**[Read full docs ↗](https://feedcanon.dev)**
+  ·  
+[Quick Start](#quick-start)
 
 ---
 
-## How It Works
+## Example
+
+The 9 URLs below all work and return identical content. None redirect to each other, normally making each appear unique. Feedcanon compares content, normalizes URLs and resolves them to a single URL.
+
+```dockerfile
+'http://feeds.kottke.org/main' ──────────┐
+'http://feeds.kottke.org/main/' ─────────┤
+'https://feeds.kottke.org/main' ─────────┤
+'https://feeds.kottke.org/main/' ────────┤
+'https://feeds.kottke.org///main/' ──────┼──→ 'https://feeds.kottke.org/main'
+'http://feeds.feedburner.com/kottke' ────┤
+'http://feeds.feedburner.com/kottke/' ───┤
+'https://feeds.feedburner.com/kottke' ───┤
+'https://feeds.feedburner.com/kottke/' ──┘
+```
+
+## Overview
+
+### How It Works
 
-- Read the feed's declared self URL (`atom:link rel="self"`) and validate it serves identical content.
-- Generate URL variants from cleanest to least clean, testing each until one works.
-- Verify URLs serve the same feed using exact body match, then signature-based matching.
-- Attempt to upgrade HTTP URLs to HTTPS when both serve identical content.
-- Normalize platform-specific domains (e.g., FeedBurner aliases like `feedproxy.google.com` → `feeds.feedburner.com`).
+1. Fetch the input URL and parse the feed to establish reference content.
+2. Extract the feed's declared self URL and validate it serves identical content.
+3. Generate URL variants ordered from cleanest to least clean.
+4. Test variants in order—the first one serving identical content wins.
+5. Upgrade HTTP to HTTPS if both serve identical content.
+
+### Customization
+
+Feedcanon is designed to be flexible. Every major component can be replaced or extended.
+
+- **Progress callbacks** — monitor the process with `onFetch`, `onMatch`, and `onExists` callbacks.
+- **Database lookup** — use `existsFn` to check if a URL already exists in your database.
+- **Custom fetch** — use your own HTTP client (Axios, Got, Ky, etc.)
+- **Custom parser** — bring your own parser (Feedsmith by default).
+- **Custom tiers** — define your own URL normalization variants.
+- **Custom platforms** — add handlers to normalize domain aliases (like FeedBurner).
 
 ## Quick Start
 
+Basic installation and common usage patterns. For a full overview, visit the [documentation website](https://feedcanon.dev).
+
 ### Installation
 
 ```bash
@@ -28,61 +65,37 @@ npm install feedcanon
 
 ### Basic Usage
 
+When you just need to clean up a feed URL and get its canonical form.
+
 ```typescript
 import { findCanonical } from 'feedcanon'
 
-const url = await findCanonical('https://www.example.com/feed/?utm_source=twitter')
+const url = await findCanonical('http://www.example.com/feed/?utm_source=twitter')
 
 // 'https://example.com/feed'
 ```
 
-### With Callbacks
+Returns `undefined` if the feed is invalid or unreachable.
 
-```typescript
-import { findCanonical } from 'feedcanon'
+### Using Callbacks
 
-const url = await findCanonical('https://example.com/feed', {
-  onFetch: ({ url, response }) => {
-    console.log('Fetched:', url, response.status)
-  },
-  onMatch: ({ url, feed }) => {
-    console.log('Found matching URL:', url)
-  },
-})
-```
-
-### Custom Fetch
+When you want to log the canonicalization process for debugging. Or store all URL aliases that resolve to the same feed.
 
 ```typescript
 import { findCanonical } from 'feedcanon'
-import axios from 'axios'
-
-const url = await findCanonical('https://example.com/feed', {
-  fetchFn: async (url) => {
-    const response = await axios.get(url)
-  
-    return {
-      status: response.status,
-      url: response.request.res.responseUrl,
-      body: response.data,
-      headers: new Headers(response.headers),
-    }
-  },
-})
-```
-
-### Database Integration
 
-```typescript
-import { findCanonical } from 'feedcanon'
+const aliases = []
 
-const url = await findCanonical('https://example.com/feed', {
-  existsFn: async (url) => {
-    // Return data if URL exists in your database, undefined otherwise.
-    return await db.feeds.findByUrl(url)
-  },
-  onExists: ({ url, data }) => {
-    console.log('URL already exists:', url)
+const url = await findCanonical('http://www.example.com/feed/', {
+  onMatch: ({ url }) => {
+    aliases.push(url)
   },
 })
+
+// url: 'https://example.com/feed'
+// aliases: [
+//   'http://www.example.com/feed/',
+//   'https://www.example.com/feed/',
+//   'https://example.com/feed',
+// ]
 ```

package/dist/defaults.cjs

@@ -1,4 +1,5 @@
 const require_feedburner = require('./platforms/feedburner.cjs');
+let feedsmith = require("feedsmith");
 
 //#region src/defaults.ts
 const defaultPlatforms = [require_feedburner.feedburnerHandler];
@@ -31,6 +32,7 @@ const defaultStrippedParams = [
 	"_ke",
 	"mc_cid",
 	"mc_eid",
+	"mc_tc",
 	"mkt_tok",
 	"msclkid",
 	"twclid",
@@ -63,15 +65,20 @@ const defaultStrippedParams = [
 	"hsa_tgt",
 	"hsa_ver",
 	"hsCtaTracking",
+	"_hsenc",
+	"_hsmi",
 	"__hstc",
 	"__hsfp",
 	"__hssc",
 	"cid",
 	"s_kwcid",
+	"sc_cid",
 	"ef_id",
 	"obOrigUrl",
 	"dicbo",
 	"yclid",
+	"ysclid",
+	"_openstat",
 	"awinaffid",
 	"awinmid",
 	"clickref",
@@ -107,14 +114,12 @@ const defaultStrippedParams = [
 	"nocache",
 	"rand",
 	"random",
+	"sbdcrw",
 	"forceByPassCache",
 	"sucurianticache",
 	"cleancache",
 	"rebuildcache",
 	"kontrol_health_check_timestamp",
-	"_x_tr_sl",
-	"_x_tr_tl",
-	"_x_tr_hl",
 	"action_object_map",
 	"action_ref_map",
 	"action_type_map",
@@ -127,6 +132,10 @@ const defaultStrippedParams = [
 	"at_custom4",
 	"at_medium",
 	"at_preview_index",
+	"_bhlid",
+	"_branch_match_id",
+	"_branch_referrer",
+	"__readwiseLocation",
 	"campaign_id",
 	"click_sum",
 	"fref",
@@ -158,12 +167,42 @@ const defaultNormalizeOptions = {
 	stripHash: true,
 	sortQueryParams: true,
 	stripQueryParams: defaultStrippedParams,
+	stripQuery: false,
 	stripEmptyQuery: true,
 	normalizeEncoding: true,
-	lowercaseHostname: true,
 	normalizeUnicode: true,
 	convertToPunycode: true
 };
+const defaultFetch = async (url, options) => {
+	const response = await fetch(url, {
+		method: options?.method ?? "GET",
+		headers: options?.headers
+	});
+	return {
+		headers: response.headers,
+		body: await response.text(),
+		url: response.url,
+		status: response.status
+	};
+};
+const defaultParser = {
+	parse: (body) => {
+		try {
+			return (0, feedsmith.parseFeed)(body);
+		} catch {}
+	},
+	getSelfUrl: (parsed) => {
+		switch (parsed.format) {
+			case "atom": return parsed.feed.links?.find((link) => link.rel === "self")?.href;
+			case "rss":
+			case "rdf": return parsed.feed.atom?.links?.find((link) => link.rel === "self")?.href;
+			case "json": return parsed.feed.feed_url;
+		}
+	},
+	getSignature: (parsed) => {
+		return parsed.feed;
+	}
+};
 const defaultTiers = [
 	{
 		stripProtocol: false,
@@ -174,9 +213,9 @@ const defaultTiers = [
 		collapseSlashes: true,
 		stripHash: true,
 		sortQueryParams: true,
+		stripQuery: false,
 		stripEmptyQuery: true,
 		normalizeEncoding: true,
-		lowercaseHostname: true,
 		normalizeUnicode: true,
 		convertToPunycode: true
 	},
@@ -189,9 +228,9 @@ const defaultTiers = [
 		collapseSlashes: true,
 		stripHash: true,
 		sortQueryParams: true,
+		stripQuery: false,
 		stripEmptyQuery: true,
 		normalizeEncoding: true,
-		lowercaseHostname: true,
 		normalizeUnicode: true,
 		convertToPunycode: true
 	},
@@ -204,16 +243,18 @@ const defaultTiers = [
 		collapseSlashes: true,
 		stripHash: true,
 		sortQueryParams: true,
+		stripQuery: false,
 		stripEmptyQuery: true,
 		normalizeEncoding: true,
-		lowercaseHostname: true,
 		normalizeUnicode: true,
 		convertToPunycode: true
 	}
 ];
 
 //#endregion
+exports.defaultFetch = defaultFetch;
 exports.defaultNormalizeOptions = defaultNormalizeOptions;
+exports.defaultParser = defaultParser;
 exports.defaultPlatforms = defaultPlatforms;
 exports.defaultStrippedParams = defaultStrippedParams;
 exports.defaultTiers = defaultTiers;

package/dist/defaults.d.cts

@@ -1,9 +1,11 @@
-import { NormalizeOptions, PlatformHandler, Tier } from "./types.cjs";
+import { FeedsmithFeed, FetchFn, NormalizeOptions, ParserAdapter, PlatformHandler, Tier } from "./types.cjs";
 
 //#region src/defaults.d.ts
 declare const defaultPlatforms: Array<PlatformHandler>;
 declare const defaultStrippedParams: string[];
 declare const defaultNormalizeOptions: NormalizeOptions;
+declare const defaultFetch: FetchFn;
+declare const defaultParser: ParserAdapter<FeedsmithFeed>;
 declare const defaultTiers: Array<Tier>;
 //#endregion
-export { defaultNormalizeOptions, defaultPlatforms, defaultStrippedParams, defaultTiers };
+export { defaultFetch, defaultNormalizeOptions, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers };

package/dist/defaults.d.ts

@@ -1,9 +1,11 @@
-import { NormalizeOptions, PlatformHandler, Tier } from "./types.js";
+import { FeedsmithFeed, FetchFn, NormalizeOptions, ParserAdapter, PlatformHandler, Tier } from "./types.js";
 
 //#region src/defaults.d.ts
 declare const defaultPlatforms: Array<PlatformHandler>;
 declare const defaultStrippedParams: string[];
 declare const defaultNormalizeOptions: NormalizeOptions;
+declare const defaultFetch: FetchFn;
+declare const defaultParser: ParserAdapter<FeedsmithFeed>;
 declare const defaultTiers: Array<Tier>;
 //#endregion
-export { defaultNormalizeOptions, defaultPlatforms, defaultStrippedParams, defaultTiers };
+export { defaultFetch, defaultNormalizeOptions, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers };

package/dist/defaults.js

@@ -1,4 +1,5 @@
 import { feedburnerHandler } from "./platforms/feedburner.js";
+import { parseFeed } from "feedsmith";
 
 //#region src/defaults.ts
 const defaultPlatforms = [feedburnerHandler];
@@ -31,6 +32,7 @@ const defaultStrippedParams = [
 	"_ke",
 	"mc_cid",
 	"mc_eid",
+	"mc_tc",
 	"mkt_tok",
 	"msclkid",
 	"twclid",
@@ -63,15 +65,20 @@ const defaultStrippedParams = [
 	"hsa_tgt",
 	"hsa_ver",
 	"hsCtaTracking",
+	"_hsenc",
+	"_hsmi",
 	"__hstc",
 	"__hsfp",
 	"__hssc",
 	"cid",
 	"s_kwcid",
+	"sc_cid",
 	"ef_id",
 	"obOrigUrl",
 	"dicbo",
 	"yclid",
+	"ysclid",
+	"_openstat",
 	"awinaffid",
 	"awinmid",
 	"clickref",
@@ -107,14 +114,12 @@ const defaultStrippedParams = [
 	"nocache",
 	"rand",
 	"random",
+	"sbdcrw",
 	"forceByPassCache",
 	"sucurianticache",
 	"cleancache",
 	"rebuildcache",
 	"kontrol_health_check_timestamp",
-	"_x_tr_sl",
-	"_x_tr_tl",
-	"_x_tr_hl",
 	"action_object_map",
 	"action_ref_map",
 	"action_type_map",
@@ -127,6 +132,10 @@ const defaultStrippedParams = [
 	"at_custom4",
 	"at_medium",
 	"at_preview_index",
+	"_bhlid",
+	"_branch_match_id",
+	"_branch_referrer",
+	"__readwiseLocation",
 	"campaign_id",
 	"click_sum",
 	"fref",
@@ -158,12 +167,42 @@ const defaultNormalizeOptions = {
 	stripHash: true,
 	sortQueryParams: true,
 	stripQueryParams: defaultStrippedParams,
+	stripQuery: false,
 	stripEmptyQuery: true,
 	normalizeEncoding: true,
-	lowercaseHostname: true,
 	normalizeUnicode: true,
 	convertToPunycode: true
 };
+const defaultFetch = async (url, options) => {
+	const response = await fetch(url, {
+		method: options?.method ?? "GET",
+		headers: options?.headers
+	});
+	return {
+		headers: response.headers,
+		body: await response.text(),
+		url: response.url,
+		status: response.status
+	};
+};
+const defaultParser = {
+	parse: (body) => {
+		try {
+			return parseFeed(body);
+		} catch {}
+	},
+	getSelfUrl: (parsed) => {
+		switch (parsed.format) {
+			case "atom": return parsed.feed.links?.find((link) => link.rel === "self")?.href;
+			case "rss":
+			case "rdf": return parsed.feed.atom?.links?.find((link) => link.rel === "self")?.href;
+			case "json": return parsed.feed.feed_url;
+		}
+	},
+	getSignature: (parsed) => {
+		return parsed.feed;
+	}
+};
 const defaultTiers = [
 	{
 		stripProtocol: false,
@@ -174,9 +213,9 @@ const defaultTiers = [
 		collapseSlashes: true,
 		stripHash: true,
 		sortQueryParams: true,
+		stripQuery: false,
 		stripEmptyQuery: true,
 		normalizeEncoding: true,
-		lowercaseHostname: true,
 		normalizeUnicode: true,
 		convertToPunycode: true
 	},
@@ -189,9 +228,9 @@ const defaultTiers = [
 		collapseSlashes: true,
 		stripHash: true,
 		sortQueryParams: true,
+		stripQuery: false,
 		stripEmptyQuery: true,
 		normalizeEncoding: true,
-		lowercaseHostname: true,
 		normalizeUnicode: true,
 		convertToPunycode: true
 	},
@@ -204,13 +243,13 @@ const defaultTiers = [
 		collapseSlashes: true,
 		stripHash: true,
 		sortQueryParams: true,
+		stripQuery: false,
 		stripEmptyQuery: true,
 		normalizeEncoding: true,
-		lowercaseHostname: true,
 		normalizeUnicode: true,
 		convertToPunycode: true
 	}
 ];
 
 //#endregion
-export { defaultNormalizeOptions, defaultPlatforms, defaultStrippedParams, defaultTiers };
+export { defaultFetch, defaultNormalizeOptions, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers };

package/dist/exports.cjs

@@ -4,11 +4,12 @@ const require_utils = require('./utils.cjs');
 const require_index = require('./index.cjs');
 
 exports.addMissingProtocol = require_utils.addMissingProtocol;
+exports.defaultFetch = require_defaults.defaultFetch;
+exports.defaultParser = require_defaults.defaultParser;
 exports.defaultPlatforms = require_defaults.defaultPlatforms;
 exports.defaultStrippedParams = require_defaults.defaultStrippedParams;
 exports.defaultTiers = require_defaults.defaultTiers;
 exports.feedburnerHandler = require_feedburner.feedburnerHandler;
-exports.feedsmithParser = require_utils.feedsmithParser;
 exports.findCanonical = require_index.findCanonical;
 exports.normalizeUrl = require_utils.normalizeUrl;
 exports.resolveFeedProtocol = require_utils.resolveFeedProtocol;

package/dist/exports.d.cts

@@ -1,6 +1,6 @@
 import { ExistsFn, FeedsmithFeed, FetchFn, FetchFnOptions, FetchFnResponse, FindCanonicalOptions, NormalizeOptions, OnExistsFn, OnFetchFn, OnMatchFn, ParserAdapter, PlatformHandler } from "./types.cjs";
-import { defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.cjs";
+import { defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.cjs";
 import { findCanonical } from "./index.cjs";
 import { feedburnerHandler } from "./platforms/feedburner.cjs";
-import { addMissingProtocol, feedsmithParser, normalizeUrl, resolveFeedProtocol, resolveUrl } from "./utils.cjs";
-export { type ExistsFn, type FeedsmithFeed, type FetchFn, type FetchFnOptions, type FetchFnResponse, type FindCanonicalOptions, type NormalizeOptions, type OnExistsFn, type OnFetchFn, type OnMatchFn, type ParserAdapter, type PlatformHandler, addMissingProtocol, defaultPlatforms, defaultStrippedParams, defaultTiers, feedburnerHandler, feedsmithParser, findCanonical, normalizeUrl, resolveFeedProtocol, resolveUrl };
+import { addMissingProtocol, normalizeUrl, resolveFeedProtocol, resolveUrl } from "./utils.cjs";
+export { type ExistsFn, type FeedsmithFeed, type FetchFn, type FetchFnOptions, type FetchFnResponse, type FindCanonicalOptions, type NormalizeOptions, type OnExistsFn, type OnFetchFn, type OnMatchFn, type ParserAdapter, type PlatformHandler, addMissingProtocol, defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers, feedburnerHandler, findCanonical, normalizeUrl, resolveFeedProtocol, resolveUrl };

package/dist/exports.d.ts

@@ -1,6 +1,6 @@
 import { ExistsFn, FeedsmithFeed, FetchFn, FetchFnOptions, FetchFnResponse, FindCanonicalOptions, NormalizeOptions, OnExistsFn, OnFetchFn, OnMatchFn, ParserAdapter, PlatformHandler } from "./types.js";
-import { defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.js";
+import { defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.js";
 import { findCanonical } from "./index.js";
 import { feedburnerHandler } from "./platforms/feedburner.js";
-import { addMissingProtocol, feedsmithParser, normalizeUrl, resolveFeedProtocol, resolveUrl } from "./utils.js";
-export { type ExistsFn, type FeedsmithFeed, type FetchFn, type FetchFnOptions, type FetchFnResponse, type FindCanonicalOptions, type NormalizeOptions, type OnExistsFn, type OnFetchFn, type OnMatchFn, type ParserAdapter, type PlatformHandler, addMissingProtocol, defaultPlatforms, defaultStrippedParams, defaultTiers, feedburnerHandler, feedsmithParser, findCanonical, normalizeUrl, resolveFeedProtocol, resolveUrl };
+import { addMissingProtocol, normalizeUrl, resolveFeedProtocol, resolveUrl } from "./utils.js";
+export { type ExistsFn, type FeedsmithFeed, type FetchFn, type FetchFnOptions, type FetchFnResponse, type FindCanonicalOptions, type NormalizeOptions, type OnExistsFn, type OnFetchFn, type OnMatchFn, type ParserAdapter, type PlatformHandler, addMissingProtocol, defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers, feedburnerHandler, findCanonical, normalizeUrl, resolveFeedProtocol, resolveUrl };

package/dist/exports.js

@@ -1,6 +1,6 @@
 import { feedburnerHandler } from "./platforms/feedburner.js";
-import { defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.js";
-import { addMissingProtocol, feedsmithParser, normalizeUrl, resolveFeedProtocol, resolveUrl } from "./utils.js";
+import { defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.js";
+import { addMissingProtocol, normalizeUrl, resolveFeedProtocol, resolveUrl } from "./utils.js";
 import { findCanonical } from "./index.js";
 
-export { addMissingProtocol, defaultPlatforms, defaultStrippedParams, defaultTiers, feedburnerHandler, feedsmithParser, findCanonical, normalizeUrl, resolveFeedProtocol, resolveUrl };
+export { addMissingProtocol, defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers, feedburnerHandler, findCanonical, normalizeUrl, resolveFeedProtocol, resolveUrl };

package/dist/index.cjs

@@ -3,7 +3,7 @@ const require_utils = require('./utils.cjs');
 
 //#region src/index.ts
 async function findCanonical(inputUrl, options) {
-	const { parser = require_utils.feedsmithParser, fetchFn = require_utils.nativeFetch, existsFn, tiers = require_defaults.defaultTiers, platforms = require_defaults.defaultPlatforms, stripQueryParams = require_defaults.defaultStrippedParams, onFetch, onMatch, onExists } = options ?? {};
+	const { parser = require_defaults.defaultParser, fetchFn = require_defaults.defaultFetch, existsFn, tiers = require_defaults.defaultTiers, platforms = require_defaults.defaultPlatforms, stripQueryParams = require_defaults.defaultStrippedParams, onFetch, onMatch, onExists } = options ?? {};
 	const stripParams = (url) => {
 		return stripQueryParams?.length ? require_utils.normalizeUrl(url, {
 			stripQueryParams,
@@ -35,7 +35,7 @@ async function findCanonical(inputUrl, options) {
 	if (!initialResponseBody) return;
 	let initialResponseSignature;
 	let selfRequestUrl;
-	const initialResponseFeed = parser.parse(initialResponseBody);
+	const initialResponseFeed = await parser.parse(initialResponseBody);
 	if (!initialResponseFeed) return;
 	onMatch?.({
 		url: initialRequestUrl,
@@ -47,13 +47,13 @@ async function findCanonical(inputUrl, options) {
 		selfRequestUrl = resolveAndApplyPlatformHandlers(selfRequestUrlRaw, initialResponseUrl);
 		selfRequestUrl = selfRequestUrl ? stripParams(selfRequestUrl) : void 0;
 	}
-	const compareWithInitialResponse = (comparedResponseBody) => {
+	const compareWithInitialResponse = async (comparedResponseBody) => {
 		if (!comparedResponseBody) return false;
 		if (initialResponseBody === comparedResponseBody) return true;
-		const comparedResponseFeed = parser.parse(comparedResponseBody);
+		const comparedResponseFeed = await parser.parse(comparedResponseBody);
 		if (comparedResponseFeed) {
-			initialResponseSignature ||= parser.getSignature(initialResponseFeed, parser.getSelfUrl(initialResponseFeed));
-			const comparedResponseSignature = parser.getSignature(comparedResponseFeed, parser.getSelfUrl(comparedResponseFeed));
+			initialResponseSignature ||= JSON.stringify(parser.getSignature(initialResponseFeed));
+			const comparedResponseSignature = JSON.stringify(parser.getSignature(comparedResponseFeed));
 			return initialResponseSignature === comparedResponseSignature;
 		}
 		return false;
@@ -70,7 +70,7 @@ async function findCanonical(inputUrl, options) {
 			response
 		});
 		if (response.status < 200 || response.status >= 300) return;
-		if (!compareWithInitialResponse(response.body)) return;
+		if (!await compareWithInitialResponse(response.body)) return;
 		return response;
 	};
 	let variantSourceUrl = initialResponseUrl;

package/dist/index.js

@@ -1,9 +1,9 @@
-import { defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.js";
-import { applyPlatformHandlers, feedsmithParser, nativeFetch, normalizeUrl, resolveUrl } from "./utils.js";
+import { defaultFetch, defaultParser, defaultPlatforms, defaultStrippedParams, defaultTiers } from "./defaults.js";
+import { applyPlatformHandlers, normalizeUrl, resolveUrl } from "./utils.js";
 
 //#region src/index.ts
 async function findCanonical(inputUrl, options) {
-	const { parser = feedsmithParser, fetchFn = nativeFetch, existsFn, tiers = defaultTiers, platforms = defaultPlatforms, stripQueryParams = defaultStrippedParams, onFetch, onMatch, onExists } = options ?? {};
+	const { parser = defaultParser, fetchFn = defaultFetch, existsFn, tiers = defaultTiers, platforms = defaultPlatforms, stripQueryParams = defaultStrippedParams, onFetch, onMatch, onExists } = options ?? {};
 	const stripParams = (url) => {
 		return stripQueryParams?.length ? normalizeUrl(url, {
 			stripQueryParams,
@@ -35,7 +35,7 @@ async function findCanonical(inputUrl, options) {
 	if (!initialResponseBody) return;
 	let initialResponseSignature;
 	let selfRequestUrl;
-	const initialResponseFeed = parser.parse(initialResponseBody);
+	const initialResponseFeed = await parser.parse(initialResponseBody);
 	if (!initialResponseFeed) return;
 	onMatch?.({
 		url: initialRequestUrl,
@@ -47,13 +47,13 @@ async function findCanonical(inputUrl, options) {
 		selfRequestUrl = resolveAndApplyPlatformHandlers(selfRequestUrlRaw, initialResponseUrl);
 		selfRequestUrl = selfRequestUrl ? stripParams(selfRequestUrl) : void 0;
 	}
-	const compareWithInitialResponse = (comparedResponseBody) => {
+	const compareWithInitialResponse = async (comparedResponseBody) => {
 		if (!comparedResponseBody) return false;
 		if (initialResponseBody === comparedResponseBody) return true;
-		const comparedResponseFeed = parser.parse(comparedResponseBody);
+		const comparedResponseFeed = await parser.parse(comparedResponseBody);
 		if (comparedResponseFeed) {
-			initialResponseSignature ||= parser.getSignature(initialResponseFeed, parser.getSelfUrl(initialResponseFeed));
-			const comparedResponseSignature = parser.getSignature(comparedResponseFeed, parser.getSelfUrl(comparedResponseFeed));
+			initialResponseSignature ||= JSON.stringify(parser.getSignature(initialResponseFeed));
+			const comparedResponseSignature = JSON.stringify(parser.getSignature(comparedResponseFeed));
 			return initialResponseSignature === comparedResponseSignature;
 		}
 		return false;
@@ -70,7 +70,7 @@ async function findCanonical(inputUrl, options) {
 			response
 		});
 		if (response.status < 200 || response.status >= 300) return;
-		if (!compareWithInitialResponse(response.body)) return;
+		if (!await compareWithInitialResponse(response.body)) return;
 		return response;
 	};
 	let variantSourceUrl = initialResponseUrl;

package/dist/platforms/feedburner.cjs

@@ -1,13 +1,13 @@
 
 //#region src/platforms/feedburner.ts
-const hosts = new Set([
+const hosts = [
 	"feeds.feedburner.com",
 	"feeds2.feedburner.com",
 	"feedproxy.google.com"
-]);
+];
 const feedburnerHandler = {
 	match: (url) => {
-		return hosts.has(url.hostname);
+		return hosts.includes(url.hostname);
 	},
 	normalize: (url) => {
 		const normalized = new URL(url);

package/dist/platforms/feedburner.js

@@ -1,12 +1,12 @@
 //#region src/platforms/feedburner.ts
-const hosts = new Set([
+const hosts = [
 	"feeds.feedburner.com",
 	"feeds2.feedburner.com",
 	"feedproxy.google.com"
-]);
+];
 const feedburnerHandler = {
 	match: (url) => {
-		return hosts.has(url.hostname);
+		return hosts.includes(url.hostname);
 	},
 	normalize: (url) => {
 		const normalized = new URL(url);

package/dist/types.d.cts

@@ -3,9 +3,9 @@ import * as feedsmith0 from "feedsmith";
 //#region src/types.d.ts
 type FeedsmithFeed = ReturnType<typeof feedsmith0.parseFeed>;
 type ParserAdapter<T> = {
-  parse: (body: string) => T | undefined;
+  parse: (body: string) => Promise<T | undefined> | T | undefined;
   getSelfUrl: (parsed: T) => string | undefined;
-  getSignature: (parsed: T, selfUrl?: string) => string;
+  getSignature: (parsed: T) => object;
 };
 type PlatformHandler = {
   match: (url: URL) => boolean;
@@ -21,9 +21,9 @@ type NormalizeOptions = {
   stripHash?: boolean;
   sortQueryParams?: boolean;
   stripQueryParams?: Array<string>;
+  stripQuery?: boolean;
   stripEmptyQuery?: boolean;
   normalizeEncoding?: boolean;
-  lowercaseHostname?: boolean;
   normalizeUnicode?: boolean;
   convertToPunycode?: boolean;
 };

package/dist/types.d.ts

@@ -3,9 +3,9 @@ import * as feedsmith0 from "feedsmith";
 //#region src/types.d.ts
 type FeedsmithFeed = ReturnType<typeof feedsmith0.parseFeed>;
 type ParserAdapter<T> = {
-  parse: (body: string) => T | undefined;
+  parse: (body: string) => Promise<T | undefined> | T | undefined;
   getSelfUrl: (parsed: T) => string | undefined;
-  getSignature: (parsed: T, selfUrl?: string) => string;
+  getSignature: (parsed: T) => object;
 };
 type PlatformHandler = {
   match: (url: URL) => boolean;
@@ -21,9 +21,9 @@ type NormalizeOptions = {
   stripHash?: boolean;
   sortQueryParams?: boolean;
   stripQueryParams?: Array<string>;
+  stripQuery?: boolean;
   stripEmptyQuery?: boolean;
   normalizeEncoding?: boolean;
-  lowercaseHostname?: boolean;
   normalizeUnicode?: boolean;
   convertToPunycode?: boolean;
 };

package/dist/utils.cjs

@@ -1,9 +1,17 @@
 const require_defaults = require('./defaults.cjs');
 let node_url = require("node:url");
 let entities = require("entities");
-let feedsmith = require("feedsmith");
 
 //#region src/utils.ts
+const strippedParamsCache = /* @__PURE__ */ new WeakMap();
+const getStrippedParamsSet = (params) => {
+	let cached = strippedParamsCache.get(params);
+	if (!cached) {
+		cached = new Set(params.map((param) => param.toLowerCase()));
+		strippedParamsCache.set(params, cached);
+	}
+	return cached;
+};
 const ipv4Pattern = /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/;
 const ipv6Pattern = /^([0-9a-f]{0,4}:){2,7}[0-9a-f]{0,4}$/i;
 const safePathChars = /[a-zA-Z0-9._~!$&'()*+,;=:@-]/;
@@ -84,7 +92,6 @@ const normalizeUrl = (url, options = require_defaults.defaultNormalizeOptions) =
 			const ascii = (0, node_url.domainToASCII)(parsed.hostname);
 			if (ascii) parsed.hostname = ascii;
 		}
-		if (options.lowercaseHostname) parsed.hostname = parsed.hostname.toLowerCase();
 		if (options.stripAuthentication) {
 			parsed.username = "";
 			parsed.password = "";
@@ -97,9 +104,15 @@ const normalizeUrl = (url, options = require_defaults.defaultNormalizeOptions) =
 		if (options.stripTrailingSlash && pathname.length > 1 && pathname.endsWith("/")) pathname = pathname.slice(0, -1);
 		if (options.stripRootSlash && pathname === "/") pathname = "";
 		parsed.pathname = pathname;
-		if (options.stripQueryParams && parsed.search) for (const param of options.stripQueryParams) parsed.searchParams.delete(param);
+		if (options.stripQuery) parsed.search = "";
+		if (options.stripQueryParams && parsed.search) {
+			const strippedSet = getStrippedParamsSet(options.stripQueryParams);
+			const paramsToDelete = [];
+			for (const [key] of parsed.searchParams) if (strippedSet.has(key.toLowerCase())) paramsToDelete.push(key);
+			for (const param of paramsToDelete) parsed.searchParams.delete(param);
+		}
 		if (options.sortQueryParams) parsed.searchParams.sort();
-		if (options.stripEmptyQuery && parsed.search === "?") parsed.search = "";
+		if (options.stripEmptyQuery && parsed.href.endsWith("?")) parsed.search = "";
 		let result = parsed.href;
 		if (options.stripProtocol) result = result.replace(/^https?:\/\//, "");
 		return result;
@@ -107,18 +120,6 @@ const normalizeUrl = (url, options = require_defaults.defaultNormalizeOptions) =
 		return url;
 	}
 };
-const nativeFetch = async (url, options) => {
-	const response = await fetch(url, {
-		method: options?.method ?? "GET",
-		headers: options?.headers
-	});
-	return {
-		headers: response.headers,
-		body: await response.text(),
-		url: response.url,
-		status: response.status
-	};
-};
 const applyPlatformHandlers = (url, platforms) => {
 	try {
 		let parsed = new URL(url);
@@ -131,31 +132,10 @@ const applyPlatformHandlers = (url, platforms) => {
 		return url;
 	}
 };
-const feedsmithParser = {
-	parse: (body) => {
-		try {
-			return (0, feedsmith.parseFeed)(body);
-		} catch {}
-	},
-	getSelfUrl: (parsed) => {
-		switch (parsed.format) {
-			case "atom": return parsed.feed.links?.find((link) => link.rel === "self")?.href;
-			case "rss":
-			case "rdf": return parsed.feed.atom?.links?.find((link) => link.rel === "self")?.href;
-			case "json": return parsed.feed.feed_url;
-		}
-	},
-	getSignature: (parsed, selfUrl) => {
-		const signature = JSON.stringify(parsed.feed);
-		return selfUrl ? signature.replaceAll(`"${selfUrl}"`, "\"__SELF_URL__\"") : signature;
-	}
-};
 
 //#endregion
 exports.addMissingProtocol = addMissingProtocol;
 exports.applyPlatformHandlers = applyPlatformHandlers;
-exports.feedsmithParser = feedsmithParser;
-exports.nativeFetch = nativeFetch;
 exports.normalizeUrl = normalizeUrl;
 exports.resolveFeedProtocol = resolveFeedProtocol;
 exports.resolveUrl = resolveUrl;

package/dist/utils.d.cts

@@ -1,10 +1,9 @@
-import { FeedsmithFeed, NormalizeOptions, ParserAdapter } from "./types.cjs";
+import { NormalizeOptions } from "./types.cjs";
 
 //#region src/utils.d.ts
 declare const resolveFeedProtocol: (url: string, protocol?: "http" | "https") => string;
 declare const addMissingProtocol: (url: string, protocol?: "http" | "https") => string;
 declare const resolveUrl: (url: string, base?: string) => string | undefined;
 declare const normalizeUrl: (url: string, options?: NormalizeOptions) => string;
-declare const feedsmithParser: ParserAdapter<FeedsmithFeed>;
 //#endregion
-export { addMissingProtocol, feedsmithParser, normalizeUrl, resolveFeedProtocol, resolveUrl };
+export { addMissingProtocol, normalizeUrl, resolveFeedProtocol, resolveUrl };

package/dist/utils.d.ts

@@ -1,10 +1,9 @@
-import { FeedsmithFeed, NormalizeOptions, ParserAdapter } from "./types.js";
+import { NormalizeOptions } from "./types.js";
 
 //#region src/utils.d.ts
 declare const resolveFeedProtocol: (url: string, protocol?: "http" | "https") => string;
 declare const addMissingProtocol: (url: string, protocol?: "http" | "https") => string;
 declare const resolveUrl: (url: string, base?: string) => string | undefined;
 declare const normalizeUrl: (url: string, options?: NormalizeOptions) => string;
-declare const feedsmithParser: ParserAdapter<FeedsmithFeed>;
 //#endregion
-export { addMissingProtocol, feedsmithParser, normalizeUrl, resolveFeedProtocol, resolveUrl };
+export { addMissingProtocol, normalizeUrl, resolveFeedProtocol, resolveUrl };

package/dist/utils.js

@@ -1,9 +1,17 @@
 import { defaultNormalizeOptions } from "./defaults.js";
 import { domainToASCII } from "node:url";
 import { decodeHTML } from "entities";
-import { parseFeed } from "feedsmith";
 
 //#region src/utils.ts
+const strippedParamsCache = /* @__PURE__ */ new WeakMap();
+const getStrippedParamsSet = (params) => {
+	let cached = strippedParamsCache.get(params);
+	if (!cached) {
+		cached = new Set(params.map((param) => param.toLowerCase()));
+		strippedParamsCache.set(params, cached);
+	}
+	return cached;
+};
 const ipv4Pattern = /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/;
 const ipv6Pattern = /^([0-9a-f]{0,4}:){2,7}[0-9a-f]{0,4}$/i;
 const safePathChars = /[a-zA-Z0-9._~!$&'()*+,;=:@-]/;
@@ -84,7 +92,6 @@ const normalizeUrl = (url, options = defaultNormalizeOptions) => {
 			const ascii = domainToASCII(parsed.hostname);
 			if (ascii) parsed.hostname = ascii;
 		}
-		if (options.lowercaseHostname) parsed.hostname = parsed.hostname.toLowerCase();
 		if (options.stripAuthentication) {
 			parsed.username = "";
 			parsed.password = "";
@@ -97,9 +104,15 @@ const normalizeUrl = (url, options = defaultNormalizeOptions) => {
 		if (options.stripTrailingSlash && pathname.length > 1 && pathname.endsWith("/")) pathname = pathname.slice(0, -1);
 		if (options.stripRootSlash && pathname === "/") pathname = "";
 		parsed.pathname = pathname;
-		if (options.stripQueryParams && parsed.search) for (const param of options.stripQueryParams) parsed.searchParams.delete(param);
+		if (options.stripQuery) parsed.search = "";
+		if (options.stripQueryParams && parsed.search) {
+			const strippedSet = getStrippedParamsSet(options.stripQueryParams);
+			const paramsToDelete = [];
+			for (const [key] of parsed.searchParams) if (strippedSet.has(key.toLowerCase())) paramsToDelete.push(key);
+			for (const param of paramsToDelete) parsed.searchParams.delete(param);
+		}
 		if (options.sortQueryParams) parsed.searchParams.sort();
-		if (options.stripEmptyQuery && parsed.search === "?") parsed.search = "";
+		if (options.stripEmptyQuery && parsed.href.endsWith("?")) parsed.search = "";
 		let result = parsed.href;
 		if (options.stripProtocol) result = result.replace(/^https?:\/\//, "");
 		return result;
@@ -107,18 +120,6 @@ const normalizeUrl = (url, options = defaultNormalizeOptions) => {
 		return url;
 	}
 };
-const nativeFetch = async (url, options) => {
-	const response = await fetch(url, {
-		method: options?.method ?? "GET",
-		headers: options?.headers
-	});
-	return {
-		headers: response.headers,
-		body: await response.text(),
-		url: response.url,
-		status: response.status
-	};
-};
 const applyPlatformHandlers = (url, platforms) => {
 	try {
 		let parsed = new URL(url);
@@ -131,25 +132,6 @@ const applyPlatformHandlers = (url, platforms) => {
 		return url;
 	}
 };
-const feedsmithParser = {
-	parse: (body) => {
-		try {
-			return parseFeed(body);
-		} catch {}
-	},
-	getSelfUrl: (parsed) => {
-		switch (parsed.format) {
-			case "atom": return parsed.feed.links?.find((link) => link.rel === "self")?.href;
-			case "rss":
-			case "rdf": return parsed.feed.atom?.links?.find((link) => link.rel === "self")?.href;
-			case "json": return parsed.feed.feed_url;
-		}
-	},
-	getSignature: (parsed, selfUrl) => {
-		const signature = JSON.stringify(parsed.feed);
-		return selfUrl ? signature.replaceAll(`"${selfUrl}"`, "\"__SELF_URL__\"") : signature;
-	}
-};
 
 //#endregion
-export { addMissingProtocol, applyPlatformHandlers, feedsmithParser, nativeFetch, normalizeUrl, resolveFeedProtocol, resolveUrl };
+export { addMissingProtocol, applyPlatformHandlers, normalizeUrl, resolveFeedProtocol, resolveUrl };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "feedcanon",
-  "description": "Find the canonical URL for any web feed by comparing actual content. Turn messy feed URLs into their cleanest, most reliable form.",
+  "description": "Find the canonical URL for any web feed by comparing actual content. Turn messy feed URLs into their cleanest form.",
   "repository": {
     "type": "git",
     "url": "https://github.com/macieklamberski/feedcanon.git"
@@ -49,7 +49,9 @@
   ],
   "scripts": {
     "prepare": "lefthook install",
-    "build": "tsdown src/exports.ts src/defaults.ts --format cjs,esm --dts --clean --unbundle --no-fixed-extension"
+    "build": "tsdown src/exports.ts src/defaults.ts --format cjs,esm --dts --clean --unbundle --no-fixed-extension",
+    "docs:dev": "vitepress dev docs",
+    "docs:build": "vitepress build docs"
   },
   "dependencies": {
     "entities": "^7.0.0",
@@ -58,7 +60,8 @@
   "devDependencies": {
     "@types/bun": "^1.3.5",
     "kvalita": "1.9.0",
-    "tsdown": "^0.18.2"
+    "tsdown": "^0.18.3",
+    "vitepress": "^1.6.4"
   },
-  "version": "1.0.0-beta.2"
+  "version": "1.0.0"
 }