feedsweep 1.1.0 → 2.0.0

package/README.md

@@ -11,15 +11,19 @@ Feedsweep takes raw feed item HTML and runs it through a pipeline that genuinely
 ## Installation
 
 ```bash
-npm install feedsweep
+npm install feedsweep linkedom
 ```
 
+`linkedom` is an optional peer dependency. You only need it if you use the bundled `parseHtml` helper — see [DOM library](#dom-library) for jsdom / happy-dom / browser-native alternatives.
+
 ## Quick Start
 
 ```typescript
 import { transformContent } from 'feedsweep'
+import { parseHtml } from 'feedsweep/linkedom'
 
 const result = await transformContent('<p>Check <img data-src="photo.jpg"> and visit /about</p>', {
+  parseHtmlFn: parseHtml,
   baseUrl: 'https://example.com/post/1',
 })
 ```
@@ -37,13 +41,17 @@ Inventory of every transform exported from the package. Most are enabled by defa
 | `stripInterBlockBreaks` | Remove `<br>` tags between block elements |
 | `stripParagraphBoundaryBreaks` | Remove `<br>` tags adjacent to paragraph boundaries |
 | `stripDuplicateTitleHeading` | Remove first `<h1>`–`<h6>` matching article title |
+| `demoteHeadings` | Shift every heading down by one level (`<h1>`→`<h2>`, …, `<h5>`→`<h6>`) when the body contains an `<h1>`, so it sits below the reader's own page title |
 | `unwrapRedirectUrls` | Remove Google/Bing/Facebook/etc. redirect wrappers |
 | `stripDeadAnchors` | Unwrap `<a>` with empty, `#`, or `javascript:` href |
+| `stripInertElements` | Remove platform chrome and dead placeholders — subscribe widgets, share buttons, related-posts widgets, ad slots (AdSense / AdThrive), author bio blocks, email preheaders, Substack image controls, and Drupal `<drupal-render-placeholder>` tags. Pass `inertSelectors` to extend or replace |
 | `removeTrackingPixels` | Strip 1×1 tracking pixel images |
+| `unwrapEmojiImages` | Replace WordPress/Facebook/Twitter/GitHub emoji `<img>` tags with their alt-text glyph |
 | `stripTrackingParams` | Remove UTM and other tracking parameters |
 | `convertBreaksToParagraphs` | Convert `<br><br>` runs into semantic `<p>` blocks |
 | `injectEnclosures` | Inject feed enclosures into content as native `<audio>`/`<video>` or iframe placeholders |
 | `replaceEmbedsWithPlaceholders` | Convert `<iframe>` to embed placeholders |
+| `convertBookmarkCards` | Convert link-preview cards into `data-bookmark-*` placeholders via a registry of per-provider `BookmarkResolver`s (`defaultBookmarkResolvers`: Ghost `kg-bookmark-card`, Substack `embedded-publication-wrap`). Extend via `bookmarkResolvers` |
 | `enrichEmbedPlaceholders` | Populate placeholder metadata (`title`, `description`, `duration`, etc.) via a caller-supplied async fn. Opt-in; not in defaults |
 | `proxyAssetUrls` | Rewrite image, video, and audio URLs through a caller-supplied proxy |
 | `resolveRelativeUrls` | Convert relative URLs to absolute using base URL |
@@ -51,19 +59,24 @@ Inventory of every transform exported from the package. Most are enabled by defa
 | `unwrapDoublyNestedLists` | Unwrap `<ul>`/`<ol>` that wrap a single `<li>` containing a same-type list |
 | `mergeFragmentedLists` | Merge consecutive sibling `<ul>` / `<ol>` lists with matching attributes |
 | `paragraphizePlainText` | Wrap plain text in `<p>` tags |
+| `stripOversizedBase64Sources` | Drop base64 `src`/`srcset`/`poster` payloads larger than 50 KB before parsing |
 | `linkifyUrls` | Wrap bare URLs in `<a>` tags |
 | `trimPreWhitespace` | Remove common leading indentation from `<pre>` |
 | `highlightCode` | Syntax-highlight `<code>` blocks with highlight.js |
 | `stripEmptyTags` | Remove empty `<p>`, `<div>`, `<span>` and other tags |
 | `stripComments` | Remove HTML `<!-- comments -->` |
 | `unwrapCdataComments` | Strip malformed `<!--[CDATA[ … ]]-->` wrappers before parsing so the wrapped article reaches the DOM as real HTML |
+| `stripControlChars` | Strip rendering-hostile control characters (NUL, BEL, ESC, DEL, C1 range) before parsing. Preserves tab / LF / CR |
 
 ## Options
 
 ```typescript
 import { fixLazyImages, resolveRelativeUrls, transformContent } from 'feedsweep'
+import { parseHtml } from 'feedsweep/linkedom'
 
 const result = transformContent(html, {
+  // Required: function that turns an HTML string into a `Document`. See "DOM library".
+  parseHtmlFn: parseHtml,
   // Base URL for resolving relative URLs.
   baseUrl: 'https://example.com/post/1',
   // Feed item enclosures (audio/video).
@@ -79,4 +92,40 @@ const result = transformContent(html, {
 })
 ```
 
-The `stringTransforms`, `domTransforms`, and `finalStringTransforms` options each fully replace the corresponding default phase when provided. Every transform is also exported individually from `feedsweep`, so you can compose any pipeline — list them explicitly to build from scratch, or spread `defaultDomTransforms` (etc.) from `feedsweep/defaults` to extend or filter the defaults.
+The `stringTransforms` and `domTransforms` options each fully replace the corresponding default phase when provided. Every transform is also exported individually from `feedsweep`, so you can compose any pipeline — list them explicitly to build from scratch, or spread `defaultDomTransforms` (etc.) from `feedsweep/defaults` to extend or filter the defaults.
+
+## DOM library
+
+Feedsweep is parser-agnostic. You provide `parseHtmlFn` — a function that turns an HTML string into a `Document`. Use any DOM library that produces a standards-compliant `Document`.
+
+```typescript
+// linkedom (recommended default)
+import { transformContent } from 'feedsweep'
+import { parseHtml } from 'feedsweep/linkedom'
+
+await transformContent(html, { parseHtmlFn: parseHtml, baseUrl })
+
+// jsdom
+import { transformContent } from 'feedsweep'
+import { JSDOM } from 'jsdom'
+
+await transformContent(html, {
+  parseHtmlFn: (raw) => new JSDOM(`<!doctype html><body>${raw}</body>`).window.document,
+  baseUrl,
+})
+
+// happy-dom
+import { transformContent } from 'feedsweep'
+import { Window } from 'happy-dom'
+
+await transformContent(html, {
+  parseHtmlFn: (raw) => {
+    const window = new Window()
+    window.document.body.innerHTML = raw
+    return window.document
+  },
+  baseUrl,
+})
+```
+
+The bundled `feedsweep/linkedom` parser bakes in two workarounds for linkedom-specific spec violations (attribute case-folding and SVG XML mode). jsdom and happy-dom do not need them.

package/dist/bookmarks/ghost.d.ts

@@ -0,0 +1,6 @@
+import { BookmarkResolver } from "../types.js";
+
+//#region src/bookmarks/ghost.d.ts
+declare const ghostBookmarkResolver: BookmarkResolver;
+//#endregion
+export { ghostBookmarkResolver };

package/dist/bookmarks/ghost.js

@@ -0,0 +1,21 @@
+//#region src/bookmarks/ghost.ts
+const ghostBookmarkResolver = {
+	selector: ".kg-bookmark-card",
+	extract: (element) => {
+		const url = element.querySelector("a.kg-bookmark-container")?.getAttribute("href") ?? void 0;
+		const title = element.querySelector(".kg-bookmark-title")?.textContent?.trim();
+		if (!url || !title) return;
+		return {
+			provider: "ghost",
+			url,
+			title,
+			description: element.querySelector(".kg-bookmark-description")?.textContent?.trim(),
+			author: element.querySelector(".kg-bookmark-author")?.textContent?.trim(),
+			publisher: element.querySelector(".kg-bookmark-publisher")?.textContent?.trim(),
+			icon: element.querySelector("img.kg-bookmark-icon")?.getAttribute("src") ?? void 0,
+			thumbnail: element.querySelector(".kg-bookmark-thumbnail img")?.getAttribute("src") ?? void 0
+		};
+	}
+};
+//#endregion
+export { ghostBookmarkResolver };

package/dist/bookmarks/substack.d.ts

@@ -0,0 +1,6 @@
+import { BookmarkResolver } from "../types.js";
+
+//#region src/bookmarks/substack.d.ts
+declare const substackBookmarkResolver: BookmarkResolver;
+//#endregion
+export { substackBookmarkResolver };

package/dist/bookmarks/substack.js

@@ -0,0 +1,26 @@
+//#region src/bookmarks/substack.ts
+const parsePublicationAttrs = (raw) => {
+	if (!raw) return;
+	try {
+		return JSON.parse(raw);
+	} catch {}
+};
+const substackBookmarkResolver = {
+	selector: ".embedded-publication-wrap",
+	extract: (element) => {
+		const attrs = parsePublicationAttrs(element.getAttribute("data-attrs"));
+		const url = attrs?.base_url;
+		const title = attrs?.name?.trim();
+		if (!url || !title) return;
+		return {
+			provider: "substack",
+			url,
+			title,
+			description: attrs.hero_text?.trim(),
+			author: attrs.author_name?.trim(),
+			icon: attrs.logo_url
+		};
+	}
+};
+//#endregion
+export { substackBookmarkResolver };

package/dist/common.d.ts

@@ -1,15 +1,13 @@
-import { EmbedResolverResult, MaybePromise } from "./types.js";
+import { BookmarkResolverResult, EmbedResolverResult, MaybePromise } from "./types.js";
 
 //#region src/common.d.ts
-declare const stripOversizedBase64Sources: (html: string, maxSize: number) => string;
-declare const expandSvgSelfClose: (html: string) => string;
-declare const parseFragment: (html: string) => Document;
-declare const transformHtml: (html: string, transform: (document: Document) => MaybePromise<void>) => Promise<string>;
-declare const applyDomTransforms: (html: string, transforms: Array<(document: Document) => MaybePromise<void>>) => Promise<string>;
+declare const isSafeThumbnailUrl: (url: string) => boolean;
+declare const applyDomTransforms: (document: Document, transforms: Array<(document: Document) => MaybePromise<void>>) => Promise<string>;
 declare const applyStringTransforms: (html: string, transforms: Array<(html: string) => MaybePromise<string>>) => Promise<string>;
-declare const applyEmbedMetadata: (element: HTMLElement, metadata: Partial<EmbedResolverResult>, options?: {
-  setIfMissing?: boolean;
-}) => void;
+declare const createPlaceholder: <Type extends object>(document: Document, type: string, fields: Type) => HTMLElement;
+declare const normalizeEmbedFields: (metadata: Partial<EmbedResolverResult>) => Record<string, string | undefined>;
+declare const updateEmbedPlaceholder: (element: HTMLElement, metadata: Partial<EmbedResolverResult>) => void;
 declare const createEmbedPlaceholder: (document: Document, src: string, metadata?: Partial<EmbedResolverResult>) => HTMLElement;
+declare const createBookmarkPlaceholder: (document: Document, result: BookmarkResolverResult) => HTMLElement;
 //#endregion
-export { applyDomTransforms, applyEmbedMetadata, applyStringTransforms, createEmbedPlaceholder, expandSvgSelfClose, parseFragment, stripOversizedBase64Sources, transformHtml };
+export { applyDomTransforms, applyStringTransforms, createBookmarkPlaceholder, createEmbedPlaceholder, createPlaceholder, isSafeThumbnailUrl, normalizeEmbedFields, updateEmbedPlaceholder };

package/dist/common.js

@@ -1,6 +1,5 @@
 import { coerceNumber } from "./utils.js";
-import { resolveUrl } from "feedcanon";
-import { parseHTML } from "linkedom";
+import { resolveUrl, upgradeProtocol } from "feedcanon";
 //#region src/common.ts
 const Node = {
 	ELEMENT_NODE: 1,
@@ -12,58 +11,11 @@ const NodeFilter = {
 	SHOW_TEXT: 4,
 	SHOW_COMMENT: 128
 };
-const base64SrcRegex = /((?:src|srcset|poster)=["'])data:[^"']*;base64,[^"']*(["'])/g;
 const safeThumbnailDataUrlRegex = /^data:image\/(png|jpe?g|gif|webp|avif);/i;
 const isSafeThumbnailUrl = (url) => {
 	return resolveUrl(url) !== void 0 || safeThumbnailDataUrlRegex.test(url);
 };
-const stripOversizedBase64Sources = (html, maxSize) => {
-	return html.replace(base64SrcRegex, (match, prefix, suffix) => {
-		if (match.length < maxSize) return match;
-		return `${prefix}${suffix}`;
-	});
-};
-const normalizeAttributeCase = (document) => {
-	for (const element of document.querySelectorAll("*")) {
-		const original = Array.from(element.attributes).map((attribute) => ({
-			name: attribute.name,
-			value: attribute.value
-		}));
-		const final = /* @__PURE__ */ new Map();
-		let needsRewrite = false;
-		for (const { name, value } of original) {
-			const lower = name.toLowerCase();
-			if (lower !== name) needsRewrite = true;
-			if (final.has(lower)) {
-				needsRewrite = true;
-				continue;
-			}
-			final.set(lower, value);
-		}
-		if (!needsRewrite) continue;
-		for (const { name } of original) element.removeAttribute(name);
-		for (const [name, value] of final) element.setAttribute(name, value);
-	}
-};
-const svgRegionRegex = /<svg\b[^>]*>[\s\S]*?<\/svg>/gi;
-const svgSelfCloseRegex = /<([a-z][a-z0-9-]*)((?:\s[^>]*)?)\s*\/>/gi;
-const expandSvgSelfClose = (html) => {
-	return html.replace(svgRegionRegex, (svgBlock) => {
-		return svgBlock.replace(svgSelfCloseRegex, "<$1$2></$1>");
-	});
-};
-const parseFragment = (html) => {
-	const { document } = parseHTML(`<!doctype html><html><head></head><body>${expandSvgSelfClose(html)}</body></html>`);
-	normalizeAttributeCase(document);
-	return document;
-};
-const transformHtml = async (html, transform) => {
-	const document = parseFragment(html);
-	await transform(document);
-	return document.body.innerHTML;
-};
-const applyDomTransforms = async (html, transforms) => {
-	const document = parseFragment(stripOversizedBase64Sources(html, 50 * 1024));
+const applyDomTransforms = async (document, transforms) => {
 	for (const transform of transforms) await transform(document);
 	return document.body.innerHTML;
 };
@@ -152,36 +104,61 @@ const getDimensions = (element) => {
 		height: height ?? fromStyle(styleHeightRegex)
 	};
 };
-const applyEmbedMetadata = (element, metadata, options) => {
-	const setIfMissing = options?.setIfMissing ?? false;
-	const set = (name, value) => {
-		if (setIfMissing && element.hasAttribute(name)) return;
-		element.setAttribute(name, value);
+const createPlaceholder = (document, type, fields) => {
+	const element = document.createElement("div");
+	for (const [key, value] of Object.entries(fields)) if (value) element.setAttribute(`data-${type}-${key}`, value);
+	return element;
+};
+const normalizeEmbedFields = (metadata) => {
+	return {
+		src: metadata.src ? upgradeProtocol(metadata.src) : void 0,
+		provider: metadata.provider,
+		id: metadata.id,
+		url: metadata.url ? upgradeProtocol(metadata.url) : void 0,
+		thumbnail: metadata.thumbnail && isSafeThumbnailUrl(metadata.thumbnail) ? metadata.thumbnail : void 0,
+		width: metadata.width ? String(metadata.width) : void 0,
+		height: metadata.height ? String(metadata.height) : void 0,
+		title: metadata.title,
+		description: metadata.description,
+		author: metadata.author,
+		avatar: metadata.avatar && isSafeThumbnailUrl(metadata.avatar) ? metadata.avatar : void 0,
+		duration: metadata.duration ? String(metadata.duration) : void 0
 	};
-	if (metadata.provider) set("data-embed-provider", metadata.provider);
-	if (metadata.id) set("data-embed-id", metadata.id);
-	if (metadata.src) set("data-embed-src", metadata.src);
-	if (metadata.url) set("data-embed-url", metadata.url);
-	if (metadata.thumbnail && isSafeThumbnailUrl(metadata.thumbnail)) set("data-embed-thumbnail", metadata.thumbnail);
-	if (metadata.width) set("data-embed-width", String(metadata.width));
-	if (metadata.height) set("data-embed-height", String(metadata.height));
-	if (metadata.title) set("data-embed-title", metadata.title);
-	if (metadata.description) set("data-embed-description", metadata.description);
-	if (metadata.author) set("data-embed-author", metadata.author);
-	if (metadata.avatar && isSafeThumbnailUrl(metadata.avatar)) set("data-embed-avatar", metadata.avatar);
-	if (metadata.duration) set("data-embed-duration", String(metadata.duration));
+};
+const updateEmbedPlaceholder = (element, metadata) => {
+	for (const [key, value] of Object.entries(normalizeEmbedFields(metadata))) {
+		const name = `data-embed-${key}`;
+		if (value && !element.hasAttribute(name)) element.setAttribute(name, value);
+	}
 };
 const createEmbedPlaceholder = (document, src, metadata) => {
-	const element = document.createElement("div");
-	element.setAttribute("data-embed", "iframe");
-	element.setAttribute("data-embed-src", metadata?.src ?? src);
-	if (metadata) applyEmbedMetadata(element, metadata);
-	const fallbackUrl = metadata?.url ?? metadata?.src ?? src;
+	const element = createPlaceholder(document, "embed", normalizeEmbedFields({
+		...metadata,
+		src: metadata?.src ?? src
+	}));
+	const fallbackUrl = upgradeProtocol(metadata?.url ?? metadata?.src ?? src);
 	const link = document.createElement("a");
 	link.setAttribute("href", fallbackUrl);
 	link.textContent = fallbackUrl;
 	element.appendChild(link);
 	return element;
 };
+const createBookmarkPlaceholder = (document, result) => {
+	const { provider, title, url, icon, thumbnail, ...rest } = result;
+	const safeUrl = upgradeProtocol(url);
+	const element = createPlaceholder(document, "bookmark", {
+		provider,
+		...rest,
+		url: safeUrl,
+		title,
+		icon: icon && isSafeThumbnailUrl(icon) ? upgradeProtocol(icon) : void 0,
+		thumbnail: thumbnail && isSafeThumbnailUrl(thumbnail) ? upgradeProtocol(thumbnail) : void 0
+	});
+	const link = document.createElement("a");
+	link.setAttribute("href", safeUrl);
+	link.textContent = title;
+	element.appendChild(link);
+	return element;
+};
 //#endregion
-export { Node, NodeFilter, applyDomTransforms, applyEmbedMetadata, applyStringTransforms, createEmbedPlaceholder, expandSvgSelfClose, getDimensions, hasAncestorWithTagName, isBlockElement, isBr, isSkippable, isWhitespaceText, normalizeAttributeCase, parseFragment, stripOversizedBase64Sources, transformHtml };
+export { Node, NodeFilter, applyDomTransforms, applyStringTransforms, createBookmarkPlaceholder, createEmbedPlaceholder, createPlaceholder, getDimensions, hasAncestorWithTagName, isBlockElement, isBr, isSafeThumbnailUrl, isSkippable, isWhitespaceText, normalizeEmbedFields, updateEmbedPlaceholder };

package/dist/defaults.d.ts

@@ -1,15 +1,18 @@
-import { DomTransform, EmbedResolver, ResolveUrlFn, StringTransform, UrlUnwrapper } from "./types.js";
+import { BookmarkResolver, DomTransform, EmbedResolver, ResolveUrlFn, StringTransform, UrlUnwrapper } from "./types.js";
 
 //#region src/defaults.d.ts
 declare const defaultStringTransforms: Array<StringTransform>;
 declare const defaultDomTransforms: Array<DomTransform>;
-declare const defaultFinalStringTransforms: Array<StringTransform>;
 declare const defaultEmbedResolvers: Array<EmbedResolver>;
+declare const defaultBookmarkResolvers: Array<BookmarkResolver>;
 declare const defaultResolveUrlFn: ResolveUrlFn;
 declare const defaultLazySrcAttributes: string[];
 declare const defaultLazySrcsetAttributes: string[];
 declare const defaultTrackingHosts: string[];
 declare const defaultTrackingPathSegments: string[];
+declare const defaultEmojiImageHosts: string[];
+declare const defaultPreservedPreClasses: string[];
+declare const defaultInertSelectors: string[];
 declare const defaultUrlUnwrappers: Array<UrlUnwrapper>;
 //#endregion
-export { defaultDomTransforms, defaultEmbedResolvers, defaultFinalStringTransforms, defaultLazySrcAttributes, defaultLazySrcsetAttributes, defaultResolveUrlFn, defaultStringTransforms, defaultTrackingHosts, defaultTrackingPathSegments, defaultUrlUnwrappers };
+export { defaultBookmarkResolvers, defaultDomTransforms, defaultEmbedResolvers, defaultEmojiImageHosts, defaultInertSelectors, defaultLazySrcAttributes, defaultLazySrcsetAttributes, defaultPreservedPreClasses, defaultResolveUrlFn, defaultStringTransforms, defaultTrackingHosts, defaultTrackingPathSegments, defaultUrlUnwrappers };

package/dist/defaults.js

@@ -1,6 +1,10 @@
+import { ghostBookmarkResolver } from "./bookmarks/ghost.js";
+import { substackBookmarkResolver } from "./bookmarks/substack.js";
 import { youtubeEmbedResolver } from "./embeds/youtube.js";
+import { convertBookmarkCards } from "./transforms/dom/convertBookmarkCards.js";
 import { convertBreaksToParagraphs } from "./transforms/dom/convertBreaksToParagraphs.js";
 import { decodeDoubleEncodedTags } from "./transforms/dom/decodeDoubleEncodedTags.js";
+import { demoteHeadings } from "./transforms/dom/demoteHeadings.js";
 import { fixLazyImages } from "./transforms/dom/fixLazyImages.js";
 import { highlightCode } from "./transforms/dom/highlightCode.js";
 import { injectEnclosures } from "./transforms/dom/injectEnclosures.js";
@@ -16,14 +20,18 @@ import { stripComments } from "./transforms/dom/stripComments.js";
 import { stripDeadAnchors } from "./transforms/dom/stripDeadAnchors.js";
 import { stripDuplicateTitleHeading } from "./transforms/dom/stripDuplicateTitleHeading.js";
 import { stripEmptyTags } from "./transforms/dom/stripEmptyTags.js";
+import { stripInertElements } from "./transforms/dom/stripInertElements.js";
 import { stripInterBlockBreaks } from "./transforms/dom/stripInterBlockBreaks.js";
 import { stripParagraphBoundaryBreaks } from "./transforms/dom/stripParagraphBoundaryBreaks.js";
 import { stripTrackingParams } from "./transforms/dom/stripTrackingParams.js";
 import { trimPreWhitespace } from "./transforms/dom/trimPreWhitespace.js";
 import { unwrapDoublyNestedLists } from "./transforms/dom/unwrapDoublyNestedLists.js";
+import { unwrapEmojiImages } from "./transforms/dom/unwrapEmojiImages.js";
 import { unwrapRedirectUrls } from "./transforms/dom/unwrapRedirectUrls.js";
 import { unwrapWrappers } from "./transforms/dom/unwrapWrappers.js";
 import { paragraphizePlainText } from "./transforms/string/paragraphizePlainText.js";
+import { stripControlChars } from "./transforms/string/stripControlChars.js";
+import { stripOversizedBase64Sources } from "./transforms/string/stripOversizedBase64Sources.js";
 import { unwrapCdataComments } from "./transforms/string/unwrapCdataComments.js";
 import { unwrapBing } from "./unwraps/bing.js";
 import { unwrapFacebookShim } from "./unwraps/facebook.js";
@@ -39,18 +47,27 @@ import { unwrapYahooSearch } from "./unwraps/yahooSearch.js";
 import { unwrapYouTube } from "./unwraps/youtube.js";
 import { resolveUrl } from "feedcanon";
 //#region src/defaults.ts
-const defaultStringTransforms = [unwrapCdataComments, paragraphizePlainText];
+const defaultStringTransforms = [
+	stripControlChars,
+	stripOversizedBase64Sources,
+	unwrapCdataComments,
+	paragraphizePlainText
+];
 const defaultDomTransforms = [
 	decodeDoubleEncodedTags,
 	stripComments,
 	unwrapDoublyNestedLists,
 	stripDuplicateTitleHeading,
+	demoteHeadings,
 	fixLazyImages,
+	stripInertElements,
 	resolveRelativeUrls,
 	unwrapRedirectUrls,
 	stripDeadAnchors,
 	stripTrackingParams,
+	convertBookmarkCards,
 	removeTrackingPixels,
+	unwrapEmojiImages,
 	convertBreaksToParagraphs,
 	stripInterBlockBreaks,
 	stripParagraphBoundaryBreaks,
@@ -66,8 +83,8 @@ const defaultDomTransforms = [
 	unwrapWrappers,
 	stripEmptyTags
 ];
-const defaultFinalStringTransforms = [];
 const defaultEmbedResolvers = [youtubeEmbedResolver];
+const defaultBookmarkResolvers = [ghostBookmarkResolver, substackBookmarkResolver];
 const defaultResolveUrlFn = (url, baseUrl) => resolveUrl(url, baseUrl);
 const defaultLazySrcAttributes = [
 	"data-src",
@@ -138,6 +155,40 @@ const defaultTrackingPathSegments = [
 	"count",
 	"impression"
 ];
+const defaultEmojiImageHosts = [
+	"s.w.org/images/core/emoji/",
+	"s0.wp.com/wp-content/mu-plugins/wpcom-smileys/",
+	"fbcdn.net/images/emoji.php/",
+	"abs.twimg.com/emoji/",
+	"githubassets.com/images/icons/emoji/"
+];
+const defaultPreservedPreClasses = ["wp-block-verse", "wp-block-preformatted"];
+const defaultInertSelectors = [
+	".image-link-expand",
+	"[data-component-name=\"SubscribeWidget\"]",
+	".subscription-widget-wrap-editor",
+	"drupal-render-placeholder",
+	".adsbygoogle",
+	".embedded-publication-wrap",
+	".yarpp-related",
+	".sharethis-inline-share-buttons",
+	".sharedaddy",
+	".wp-block-jetpack-subscriptions",
+	".wp-block-post-author",
+	".kg-signup-card",
+	".mc4wp-form",
+	".formkit-form",
+	".mcnPreviewText",
+	".saboxplugin-wrap",
+	".addtoany_share_save_container",
+	"iframe[src*=\"embeds.beehiiv.com\"]",
+	".jp-relatedposts",
+	".adthrive-ad",
+	".jetpack_subscription_widget",
+	".crp_related",
+	"form[action*=\"buttondown.email\"]",
+	".sqs-block-newsletter"
+];
 const defaultUrlUnwrappers = [
 	unwrapBing,
 	unwrapGoogle,
@@ -153,4 +204,4 @@ const defaultUrlUnwrappers = [
 	unwrapRedditOut
 ];
 //#endregion
-export { defaultDomTransforms, defaultEmbedResolvers, defaultFinalStringTransforms, defaultLazySrcAttributes, defaultLazySrcsetAttributes, defaultResolveUrlFn, defaultStringTransforms, defaultTrackingHosts, defaultTrackingPathSegments, defaultUrlUnwrappers };
+export { defaultBookmarkResolvers, defaultDomTransforms, defaultEmbedResolvers, defaultEmojiImageHosts, defaultInertSelectors, defaultLazySrcAttributes, defaultLazySrcsetAttributes, defaultPreservedPreClasses, defaultResolveUrlFn, defaultStringTransforms, defaultTrackingHosts, defaultTrackingPathSegments, defaultUrlUnwrappers };

package/dist/index.d.ts

@@ -1,9 +1,13 @@
-import { AssetProxyFn, AssetType, DomTransform, EmbedResolver, EmbedResolverResult, Enclosure, EnrichEmbedFn, MaybePromise, ResolveUrlFn, StringTransform, TransformContentOptions, TransformContext } from "./types.js";
+import { AssetProxyFn, AssetType, BookmarkResolver, BookmarkResolverResult, DomTransform, EmbedResolver, EmbedResolverResult, Enclosure, EnrichEmbedFn, MaybePromise, ParseHtmlFn, ResolveUrlFn, StringTransform, TransformContentOptions, TransformContext } from "./types.js";
 import { defaultResolveUrlFn } from "./defaults.js";
-import { applyDomTransforms, applyEmbedMetadata, applyStringTransforms, createEmbedPlaceholder, expandSvgSelfClose, parseFragment, stripOversizedBase64Sources, transformHtml } from "./common.js";
+import { ghostBookmarkResolver } from "./bookmarks/ghost.js";
+import { substackBookmarkResolver } from "./bookmarks/substack.js";
+import { applyDomTransforms, applyStringTransforms, createBookmarkPlaceholder, createEmbedPlaceholder, createPlaceholder, isSafeThumbnailUrl, normalizeEmbedFields, updateEmbedPlaceholder } from "./common.js";
 import { composeThumbnailUrl, extractVideoId, youtubeEmbedResolver, youtubeResolveEmbed } from "./embeds/youtube.js";
+import { convertBookmarkCards } from "./transforms/dom/convertBookmarkCards.js";
 import { convertBreaksToParagraphs } from "./transforms/dom/convertBreaksToParagraphs.js";
 import { decodeDoubleEncodedTags } from "./transforms/dom/decodeDoubleEncodedTags.js";
+import { demoteHeadings } from "./transforms/dom/demoteHeadings.js";
 import { enrichEmbedPlaceholders } from "./transforms/dom/enrichEmbedPlaceholders.js";
 import { fixLazyImages } from "./transforms/dom/fixLazyImages.js";
 import { detectLanguage, highlightCode } from "./transforms/dom/highlightCode.js";
@@ -20,14 +24,18 @@ import { stripComments } from "./transforms/dom/stripComments.js";
 import { stripDeadAnchors } from "./transforms/dom/stripDeadAnchors.js";
 import { stripDuplicateTitleHeading } from "./transforms/dom/stripDuplicateTitleHeading.js";
 import { stripEmptyTags } from "./transforms/dom/stripEmptyTags.js";
+import { stripInertElements } from "./transforms/dom/stripInertElements.js";
 import { stripInterBlockBreaks } from "./transforms/dom/stripInterBlockBreaks.js";
 import { stripParagraphBoundaryBreaks } from "./transforms/dom/stripParagraphBoundaryBreaks.js";
 import { stripTrackingParams } from "./transforms/dom/stripTrackingParams.js";
 import { trimPreWhitespace } from "./transforms/dom/trimPreWhitespace.js";
 import { unwrapDoublyNestedLists } from "./transforms/dom/unwrapDoublyNestedLists.js";
+import { unwrapEmojiImages } from "./transforms/dom/unwrapEmojiImages.js";
 import { extractRedirectTarget, unwrapRedirectUrls } from "./transforms/dom/unwrapRedirectUrls.js";
 import { unwrapWrappers } from "./transforms/dom/unwrapWrappers.js";
 import { paragraphizePlainText } from "./transforms/string/paragraphizePlainText.js";
+import { stripControlChars } from "./transforms/string/stripControlChars.js";
+import { stripOversizedBase64Sources } from "./transforms/string/stripOversizedBase64Sources.js";
 import { unwrapCdataComments } from "./transforms/string/unwrapCdataComments.js";
 import { unwrapAceml } from "./unwraps/aceml.js";
 import { unwrapAdjust } from "./unwraps/adjust.js";
@@ -106,6 +114,6 @@ import { unwrapZhihu } from "./unwraps/zhihu.js";
 import { ParamExtractorConfig, chooseBaseUrl, coerceNumber, createParamExtractor } from "./utils.js";
 
 //#region src/index.d.ts
-declare const transformContent: (html: string, options?: TransformContentOptions) => Promise<string>;
+declare const transformContent: (html: string, options: TransformContentOptions) => Promise<string>;
 //#endregion
-export { type AssetProxyFn, type AssetType, type DomTransform, type EmbedResolver, type EmbedResolverResult, type Enclosure, type EnrichEmbedFn, type MaybePromise, type ParamExtractorConfig, type ResolveUrlFn, type StringTransform, type TransformContentOptions, type TransformContext, applyDomTransforms, applyEmbedMetadata, applyStringTransforms, chooseBaseUrl, coerceNumber, composeThumbnailUrl, convertBreaksToParagraphs, createEmbedPlaceholder, createParamExtractor, decodeDoubleEncodedTags, defaultResolveUrlFn, detectLanguage, enrichEmbedPlaceholders, expandSvgSelfClose, extractRedirectTarget, extractVideoId, fixLazyImages, highlightCode, injectEnclosures, linkifyUrls, mergeConsecutiveOneLinerPres, mergeFragmentedLists, paragraphizePlainText, parseFragment, proxyAssetUrls, removeTrackingPixels, replaceEmbedsWithPlaceholders, replacePreLineBreaks, resolveRelativeUrls, stripComments, stripDeadAnchors, stripDuplicateTitleHeading, stripEmptyTags, stripInterBlockBreaks, stripOversizedBase64Sources, stripParagraphBoundaryBreaks, stripTrackingParams, transformContent, transformHtml, trimPreWhitespace, unwrapAceml, unwrapAdjust, unwrapAmazonAffiliate, unwrapAmpCache, unwrapAwin, unwrapBing, unwrapCdataComments, unwrapCjNetwork, unwrapDigidip, unwrapDisqus, unwrapDouban, unwrapDoublyNestedLists, unwrapDuckduckgo, unwrapEbayRover, unwrapEffiliation, unwrapEmbedly, unwrapFacebookShim, unwrapFeedsportal, unwrapFirebaseDynamicLinks, unwrapFlipboard, unwrapGateSc, unwrapGeoriot, unwrapGitee, unwrapGoogle, unwrapGoogleAmpViewer, unwrapGoogleNews, unwrapGoogleNewsModern, unwrapGoogleScholar, unwrapGoogleTranslate, unwrapHashnode, unwrapIcptrack, unwrapIdealoPartner, unwrapInstagramShim, unwrapJianshuGo, unwrapJuejin, unwrapLeverAnalytics, unwrapLinksynergy, unwrapMailchimp, unwrapMailpanion, unwrapMailpgn, unwrapMailtrack, unwrapMedium, unwrapMimecast, unwrapMozillaOutgoing, unwrapNarrativ, unwrapNicoMs, unwrapOutlookSafelinks, unwrapPartnerAds, unwrapPocket, unwrapPostmark, unwrapProofpointV1, unwrapProofpointV2, unwrapProofpointV3, unwrapPxf, unwrapRecruitics, unwrapRedditOut, unwrapRedirectUrls, unwrapRedirectingat, unwrapSegmentfault, unwrapShareasale, unwrapSjv, unwrapSkimlinks, unwrapSlack, unwrapSmartredirect, unwrapSspai, unwrapSteamLinkfilter, unwrapTelegramIv, unwrapTradedoubler, unwrapTumblr, unwrapValuecommerce, unwrapViglink, unwrapVkAway, unwrapWebArchive, unwrapWrappers, unwrapYahooSearch, unwrapYandexTurbo, unwrapYouTube, unwrapZhihu, youtubeEmbedResolver, youtubeResolveEmbed };
+export { type AssetProxyFn, type AssetType, type BookmarkResolver, type BookmarkResolverResult, type DomTransform, type EmbedResolver, type EmbedResolverResult, type Enclosure, type EnrichEmbedFn, type MaybePromise, type ParamExtractorConfig, type ParseHtmlFn, type ResolveUrlFn, type StringTransform, type TransformContentOptions, type TransformContext, applyDomTransforms, applyStringTransforms, chooseBaseUrl, coerceNumber, composeThumbnailUrl, convertBookmarkCards, convertBreaksToParagraphs, createBookmarkPlaceholder, createEmbedPlaceholder, createParamExtractor, createPlaceholder, decodeDoubleEncodedTags, defaultResolveUrlFn, demoteHeadings, detectLanguage, enrichEmbedPlaceholders, extractRedirectTarget, extractVideoId, fixLazyImages, ghostBookmarkResolver, highlightCode, injectEnclosures, isSafeThumbnailUrl, linkifyUrls, mergeConsecutiveOneLinerPres, mergeFragmentedLists, normalizeEmbedFields, paragraphizePlainText, proxyAssetUrls, removeTrackingPixels, replaceEmbedsWithPlaceholders, replacePreLineBreaks, resolveRelativeUrls, stripComments, stripControlChars, stripDeadAnchors, stripDuplicateTitleHeading, stripEmptyTags, stripInertElements, stripInterBlockBreaks, stripOversizedBase64Sources, stripParagraphBoundaryBreaks, stripTrackingParams, substackBookmarkResolver, transformContent, trimPreWhitespace, unwrapAceml, unwrapAdjust, unwrapAmazonAffiliate, unwrapAmpCache, unwrapAwin, unwrapBing, unwrapCdataComments, unwrapCjNetwork, unwrapDigidip, unwrapDisqus, unwrapDouban, unwrapDoublyNestedLists, unwrapDuckduckgo, unwrapEbayRover, unwrapEffiliation, unwrapEmbedly, unwrapEmojiImages, unwrapFacebookShim, unwrapFeedsportal, unwrapFirebaseDynamicLinks, unwrapFlipboard, unwrapGateSc, unwrapGeoriot, unwrapGitee, unwrapGoogle, unwrapGoogleAmpViewer, unwrapGoogleNews, unwrapGoogleNewsModern, unwrapGoogleScholar, unwrapGoogleTranslate, unwrapHashnode, unwrapIcptrack, unwrapIdealoPartner, unwrapInstagramShim, unwrapJianshuGo, unwrapJuejin, unwrapLeverAnalytics, unwrapLinksynergy, unwrapMailchimp, unwrapMailpanion, unwrapMailpgn, unwrapMailtrack, unwrapMedium, unwrapMimecast, unwrapMozillaOutgoing, unwrapNarrativ, unwrapNicoMs, unwrapOutlookSafelinks, unwrapPartnerAds, unwrapPocket, unwrapPostmark, unwrapProofpointV1, unwrapProofpointV2, unwrapProofpointV3, unwrapPxf, unwrapRecruitics, unwrapRedditOut, unwrapRedirectUrls, unwrapRedirectingat, unwrapSegmentfault, unwrapShareasale, unwrapSjv, unwrapSkimlinks, unwrapSlack, unwrapSmartredirect, unwrapSspai, unwrapSteamLinkfilter, unwrapTelegramIv, unwrapTradedoubler, unwrapTumblr, unwrapValuecommerce, unwrapViglink, unwrapVkAway, unwrapWebArchive, unwrapWrappers, unwrapYahooSearch, unwrapYandexTurbo, unwrapYouTube, unwrapZhihu, updateEmbedPlaceholder, youtubeEmbedResolver, youtubeResolveEmbed };

package/dist/index.js

@@ -1,8 +1,12 @@
 import { chooseBaseUrl, coerceNumber, createParamExtractor } from "./utils.js";
-import { applyDomTransforms, applyEmbedMetadata, applyStringTransforms, createEmbedPlaceholder, expandSvgSelfClose, parseFragment, stripOversizedBase64Sources, transformHtml } from "./common.js";
+import { applyDomTransforms, applyStringTransforms, createBookmarkPlaceholder, createEmbedPlaceholder, createPlaceholder, isSafeThumbnailUrl, normalizeEmbedFields, updateEmbedPlaceholder } from "./common.js";
+import { ghostBookmarkResolver } from "./bookmarks/ghost.js";
+import { substackBookmarkResolver } from "./bookmarks/substack.js";
 import { composeThumbnailUrl, extractVideoId, youtubeEmbedResolver, youtubeResolveEmbed } from "./embeds/youtube.js";
+import { convertBookmarkCards } from "./transforms/dom/convertBookmarkCards.js";
 import { convertBreaksToParagraphs } from "./transforms/dom/convertBreaksToParagraphs.js";
 import { decodeDoubleEncodedTags } from "./transforms/dom/decodeDoubleEncodedTags.js";
+import { demoteHeadings } from "./transforms/dom/demoteHeadings.js";
 import { fixLazyImages } from "./transforms/dom/fixLazyImages.js";
 import { detectLanguage, highlightCode } from "./transforms/dom/highlightCode.js";
 import { injectEnclosures } from "./transforms/dom/injectEnclosures.js";
@@ -18,14 +22,18 @@ import { stripComments } from "./transforms/dom/stripComments.js";
 import { stripDeadAnchors } from "./transforms/dom/stripDeadAnchors.js";
 import { stripDuplicateTitleHeading } from "./transforms/dom/stripDuplicateTitleHeading.js";
 import { stripEmptyTags } from "./transforms/dom/stripEmptyTags.js";
+import { stripInertElements } from "./transforms/dom/stripInertElements.js";
 import { stripInterBlockBreaks } from "./transforms/dom/stripInterBlockBreaks.js";
 import { stripParagraphBoundaryBreaks } from "./transforms/dom/stripParagraphBoundaryBreaks.js";
 import { stripTrackingParams } from "./transforms/dom/stripTrackingParams.js";
 import { trimPreWhitespace } from "./transforms/dom/trimPreWhitespace.js";
 import { unwrapDoublyNestedLists } from "./transforms/dom/unwrapDoublyNestedLists.js";
+import { unwrapEmojiImages } from "./transforms/dom/unwrapEmojiImages.js";
 import { extractRedirectTarget, unwrapRedirectUrls } from "./transforms/dom/unwrapRedirectUrls.js";
 import { unwrapWrappers } from "./transforms/dom/unwrapWrappers.js";
 import { paragraphizePlainText } from "./transforms/string/paragraphizePlainText.js";
+import { stripControlChars } from "./transforms/string/stripControlChars.js";
+import { stripOversizedBase64Sources } from "./transforms/string/stripOversizedBase64Sources.js";
 import { unwrapCdataComments } from "./transforms/string/unwrapCdataComments.js";
 import { unwrapBing } from "./unwraps/bing.js";
 import { unwrapFacebookShim } from "./unwraps/facebook.js";
@@ -39,7 +47,7 @@ import { unwrapRedditOut } from "./unwraps/redditOut.js";
 import { unwrapVkAway } from "./unwraps/vkAway.js";
 import { unwrapYahooSearch } from "./unwraps/yahooSearch.js";
 import { unwrapYouTube } from "./unwraps/youtube.js";
-import { defaultDomTransforms, defaultEmbedResolvers, defaultFinalStringTransforms, defaultLazySrcAttributes, defaultLazySrcsetAttributes, defaultResolveUrlFn, defaultStringTransforms, defaultTrackingHosts, defaultTrackingPathSegments, defaultUrlUnwrappers } from "./defaults.js";
+import { defaultBookmarkResolvers, defaultDomTransforms, defaultEmbedResolvers, defaultEmojiImageHosts, defaultInertSelectors, defaultLazySrcAttributes, defaultLazySrcsetAttributes, defaultPreservedPreClasses, defaultResolveUrlFn, defaultStringTransforms, defaultTrackingHosts, defaultTrackingPathSegments, defaultUrlUnwrappers } from "./defaults.js";
 import { enrichEmbedPlaceholders } from "./transforms/dom/enrichEmbedPlaceholders.js";
 import { unwrapAceml } from "./unwraps/aceml.js";
 import { unwrapAdjust } from "./unwraps/adjust.js";
@@ -104,15 +112,19 @@ import { unwrapWebArchive } from "./unwraps/webArchive.js";
 import { unwrapYandexTurbo } from "./unwraps/yandexTurbo.js";
 import { unwrapZhihu } from "./unwraps/zhihu.js";
 //#region src/index.ts
-const transformContent = async (html, options = {}) => {
+const transformContent = async (html, options) => {
 	const context = {
 		baseUrl: options.baseUrl,
 		enclosures: options.enclosures,
 		embedResolvers: options.embedResolvers ?? defaultEmbedResolvers,
+		bookmarkResolvers: options.bookmarkResolvers ?? defaultBookmarkResolvers,
 		lazySrcAttributes: options.lazySrcAttributes ?? defaultLazySrcAttributes,
 		lazySrcsetAttributes: options.lazySrcsetAttributes ?? defaultLazySrcsetAttributes,
 		trackingHosts: options.trackingHosts ?? defaultTrackingHosts,
 		trackingPathSegments: options.trackingPathSegments ?? defaultTrackingPathSegments,
+		emojiImageHosts: options.emojiImageHosts ?? defaultEmojiImageHosts,
+		inertSelectors: options.inertSelectors ?? defaultInertSelectors,
+		preservedPreClasses: options.preservedPreClasses ?? defaultPreservedPreClasses,
 		urlUnwrappers: options.urlUnwrappers ?? defaultUrlUnwrappers,
 		resolveUrlFn: options.resolveUrlFn ?? defaultResolveUrlFn,
 		assetProxyFn: options.assetProxyFn,
@@ -121,8 +133,8 @@ const transformContent = async (html, options = {}) => {
 	};
 	const stringFns = options.stringTransforms ?? defaultStringTransforms;
 	const domFns = options.domTransforms ?? defaultDomTransforms;
-	const finalFns = options.finalStringTransforms ?? defaultFinalStringTransforms;
-	return await applyStringTransforms(await applyDomTransforms(await applyStringTransforms(html, stringFns.map((transform) => transform(context))), domFns.map((transform) => transform(context))), finalFns.map((transform) => transform(context)));
+	const afterString = await applyStringTransforms(html, stringFns.map((transform) => transform(context)));
+	return await applyDomTransforms(await options.parseHtmlFn(afterString), domFns.map((transform) => transform(context)));
 };
 //#endregion
-export { applyDomTransforms, applyEmbedMetadata, applyStringTransforms, chooseBaseUrl, coerceNumber, composeThumbnailUrl, convertBreaksToParagraphs, createEmbedPlaceholder, createParamExtractor, decodeDoubleEncodedTags, defaultResolveUrlFn, detectLanguage, enrichEmbedPlaceholders, expandSvgSelfClose, extractRedirectTarget, extractVideoId, fixLazyImages, highlightCode, injectEnclosures, linkifyUrls, mergeConsecutiveOneLinerPres, mergeFragmentedLists, paragraphizePlainText, parseFragment, proxyAssetUrls, removeTrackingPixels, replaceEmbedsWithPlaceholders, replacePreLineBreaks, resolveRelativeUrls, stripComments, stripDeadAnchors, stripDuplicateTitleHeading, stripEmptyTags, stripInterBlockBreaks, stripOversizedBase64Sources, stripParagraphBoundaryBreaks, stripTrackingParams, transformContent, transformHtml, trimPreWhitespace, unwrapAceml, unwrapAdjust, unwrapAmazonAffiliate, unwrapAmpCache, unwrapAwin, unwrapBing, unwrapCdataComments, unwrapCjNetwork, unwrapDigidip, unwrapDisqus, unwrapDouban, unwrapDoublyNestedLists, unwrapDuckduckgo, unwrapEbayRover, unwrapEffiliation, unwrapEmbedly, unwrapFacebookShim, unwrapFeedsportal, unwrapFirebaseDynamicLinks, unwrapFlipboard, unwrapGateSc, unwrapGeoriot, unwrapGitee, unwrapGoogle, unwrapGoogleAmpViewer, unwrapGoogleNews, unwrapGoogleNewsModern, unwrapGoogleScholar, unwrapGoogleTranslate, unwrapHashnode, unwrapIcptrack, unwrapIdealoPartner, unwrapInstagramShim, unwrapJianshuGo, unwrapJuejin, unwrapLeverAnalytics, unwrapLinksynergy, unwrapMailchimp, unwrapMailpanion, unwrapMailpgn, unwrapMailtrack, unwrapMedium, unwrapMimecast, unwrapMozillaOutgoing, unwrapNarrativ, unwrapNicoMs, unwrapOutlookSafelinks, unwrapPartnerAds, unwrapPocket, unwrapPostmark, unwrapProofpointV1, unwrapProofpointV2, unwrapProofpointV3, unwrapPxf, unwrapRecruitics, unwrapRedditOut, unwrapRedirectUrls, unwrapRedirectingat, unwrapSegmentfault, unwrapShareasale, unwrapSjv, unwrapSkimlinks, unwrapSlack, unwrapSmartredirect, unwrapSspai, unwrapSteamLinkfilter, unwrapTelegramIv, unwrapTradedoubler, unwrapTumblr, unwrapValuecommerce, unwrapViglink, unwrapVkAway, unwrapWebArchive, unwrapWrappers, unwrapYahooSearch, unwrapYandexTurbo, unwrapYouTube, unwrapZhihu, youtubeEmbedResolver, youtubeResolveEmbed };
+export { applyDomTransforms, applyStringTransforms, chooseBaseUrl, coerceNumber, composeThumbnailUrl, convertBookmarkCards, convertBreaksToParagraphs, createBookmarkPlaceholder, createEmbedPlaceholder, createParamExtractor, createPlaceholder, decodeDoubleEncodedTags, defaultResolveUrlFn, demoteHeadings, detectLanguage, enrichEmbedPlaceholders, extractRedirectTarget, extractVideoId, fixLazyImages, ghostBookmarkResolver, highlightCode, injectEnclosures, isSafeThumbnailUrl, linkifyUrls, mergeConsecutiveOneLinerPres, mergeFragmentedLists, normalizeEmbedFields, paragraphizePlainText, proxyAssetUrls, removeTrackingPixels, replaceEmbedsWithPlaceholders, replacePreLineBreaks, resolveRelativeUrls, stripComments, stripControlChars, stripDeadAnchors, stripDuplicateTitleHeading, stripEmptyTags, stripInertElements, stripInterBlockBreaks, stripOversizedBase64Sources, stripParagraphBoundaryBreaks, stripTrackingParams, substackBookmarkResolver, transformContent, trimPreWhitespace, unwrapAceml, unwrapAdjust, unwrapAmazonAffiliate, unwrapAmpCache, unwrapAwin, unwrapBing, unwrapCdataComments, unwrapCjNetwork, unwrapDigidip, unwrapDisqus, unwrapDouban, unwrapDoublyNestedLists, unwrapDuckduckgo, unwrapEbayRover, unwrapEffiliation, unwrapEmbedly, unwrapEmojiImages, unwrapFacebookShim, unwrapFeedsportal, unwrapFirebaseDynamicLinks, unwrapFlipboard, unwrapGateSc, unwrapGeoriot, unwrapGitee, unwrapGoogle, unwrapGoogleAmpViewer, unwrapGoogleNews, unwrapGoogleNewsModern, unwrapGoogleScholar, unwrapGoogleTranslate, unwrapHashnode, unwrapIcptrack, unwrapIdealoPartner, unwrapInstagramShim, unwrapJianshuGo, unwrapJuejin, unwrapLeverAnalytics, unwrapLinksynergy, unwrapMailchimp, unwrapMailpanion, unwrapMailpgn, unwrapMailtrack, unwrapMedium, unwrapMimecast, unwrapMozillaOutgoing, unwrapNarrativ, unwrapNicoMs, unwrapOutlookSafelinks, unwrapPartnerAds, unwrapPocket, unwrapPostmark, unwrapProofpointV1, unwrapProofpointV2, unwrapProofpointV3, unwrapPxf, unwrapRecruitics, unwrapRedditOut, unwrapRedirectUrls, unwrapRedirectingat, unwrapSegmentfault, unwrapShareasale, unwrapSjv, unwrapSkimlinks, unwrapSlack, unwrapSmartredirect, unwrapSspai, unwrapSteamLinkfilter, unwrapTelegramIv, unwrapTradedoubler, unwrapTumblr, unwrapValuecommerce, unwrapViglink, unwrapVkAway, unwrapWebArchive, unwrapWrappers, unwrapYahooSearch, unwrapYandexTurbo, unwrapYouTube, unwrapZhihu, updateEmbedPlaceholder, youtubeEmbedResolver, youtubeResolveEmbed };

package/dist/parsers/linkedom.d.ts

@@ -0,0 +1,4 @@
+//#region src/parsers/linkedom.d.ts
+declare const parseHtml: (html: string) => Document;
+//#endregion
+export { parseHtml };

package/dist/parsers/linkedom.js

@@ -0,0 +1,38 @@
+import { parseHTML } from "linkedom";
+//#region src/parsers/linkedom.ts
+const normalizeAttributeCase = (document) => {
+	for (const element of document.querySelectorAll("*")) {
+		const original = Array.from(element.attributes).map((attribute) => ({
+			name: attribute.name,
+			value: attribute.value
+		}));
+		const final = /* @__PURE__ */ new Map();
+		let needsRewrite = false;
+		for (const { name, value } of original) {
+			const lower = name.toLowerCase();
+			if (lower !== name) needsRewrite = true;
+			if (final.has(lower)) {
+				needsRewrite = true;
+				continue;
+			}
+			final.set(lower, value);
+		}
+		if (!needsRewrite) continue;
+		for (const { name } of original) element.removeAttribute(name);
+		for (const [name, value] of final) element.setAttribute(name, value);
+	}
+};
+const svgRegionRegex = /<svg\b[^>]*>[\s\S]*?<\/svg>/gi;
+const svgSelfCloseRegex = /<([a-z][a-z0-9-]*)((?:\s[^>]*)?)\s*\/>/gi;
+const expandSvgSelfClose = (html) => {
+	return html.replace(svgRegionRegex, (svgBlock) => {
+		return svgBlock.replace(svgSelfCloseRegex, "<$1$2></$1>");
+	});
+};
+const parseHtml = (html) => {
+	const { document } = parseHTML(`<!doctype html><html><head></head><body>${expandSvgSelfClose(html)}</body></html>`);
+	normalizeAttributeCase(document);
+	return document;
+};
+//#endregion
+export { parseHtml };

package/dist/transforms/dom/convertBookmarkCards.d.ts

@@ -0,0 +1,6 @@
+import { DomTransform } from "../../types.js";
+
+//#region src/transforms/dom/convertBookmarkCards.d.ts
+declare const convertBookmarkCards: DomTransform;
+//#endregion
+export { convertBookmarkCards };

package/dist/transforms/dom/convertBookmarkCards.js

@@ -0,0 +1,14 @@
+import { createBookmarkPlaceholder } from "../../common.js";
+//#region src/transforms/dom/convertBookmarkCards.ts
+const convertBookmarkCards = (context) => {
+	const { bookmarkResolvers } = context;
+	return async (document) => {
+		for (const resolver of bookmarkResolvers) for (const element of document.querySelectorAll(resolver.selector)) {
+			const result = await resolver.extract(element);
+			if (!result) continue;
+			element.replaceWith(createBookmarkPlaceholder(document, result));
+		}
+	};
+};
+//#endregion
+export { convertBookmarkCards };

package/dist/transforms/dom/demoteHeadings.d.ts

@@ -0,0 +1,6 @@
+import { DomTransform } from "../../types.js";
+
+//#region src/transforms/dom/demoteHeadings.d.ts
+declare const demoteHeadings: DomTransform;
+//#endregion
+export { demoteHeadings };

package/dist/transforms/dom/demoteHeadings.js

@@ -0,0 +1,20 @@
+//#region src/transforms/dom/demoteHeadings.ts
+const headingSelector = "h1, h2, h3, h4, h5";
+const demoteHeadings = () => {
+	return (document) => {
+		if (!document.querySelector("h1")) return;
+		const headings = document.querySelectorAll(headingSelector);
+		for (const heading of headings) {
+			const nextTagName = `h${Number(heading.tagName.slice(1)) + 1}`;
+			const replacement = document.createElement(nextTagName);
+			for (const name of heading.getAttributeNames().reverse()) {
+				const value = heading.getAttribute(name);
+				if (value !== null) replacement.setAttribute(name, value);
+			}
+			while (heading.firstChild) replacement.appendChild(heading.firstChild);
+			heading.replaceWith(replacement);
+		}
+	};
+};
+//#endregion
+export { demoteHeadings };

package/dist/transforms/dom/enrichEmbedPlaceholders.js

@@ -1,4 +1,4 @@
-import { applyEmbedMetadata } from "../../common.js";
+import { updateEmbedPlaceholder } from "../../common.js";
 //#region src/transforms/dom/enrichEmbedPlaceholders.ts
 const enrichEmbedPlaceholders = (context) => {
 	const enrichEmbedFn = context.enrichEmbedFn;
@@ -24,7 +24,7 @@ const enrichEmbedPlaceholders = (context) => {
 		for (let i = 0; i < count; i++) {
 			const embed = embeds[i];
 			const data = enriched.get(`${embed.provider}:${embed.id}`);
-			if (data) applyEmbedMetadata(placeholders[i], data, { setIfMissing: true });
+			if (data) updateEmbedPlaceholder(placeholders[i], data);
 		}
 	};
 };

package/dist/transforms/dom/fixLazyImages.js

@@ -1,4 +1,3 @@
-import { normalizeAttributeCase } from "../../common.js";
 //#region src/transforms/dom/fixLazyImages.ts
 const imgPattern = /<img\s/i;
 const urlShapeRegex = /[:/.]/;
@@ -45,7 +44,6 @@ const fixLazyImages = (context) => {
 			}
 		}
 		const noscripts = document.querySelectorAll("noscript");
-		let replacedNoscript = false;
 		for (const noscript of noscripts) {
 			const sibling = noscript.previousElementSibling;
 			if (sibling?.localName !== "img") continue;
@@ -53,9 +51,7 @@ const fixLazyImages = (context) => {
 			if (!imgPattern.test(inner)) continue;
 			sibling.remove();
 			noscript.outerHTML = inner;
-			replacedNoscript = true;
 		}
-		if (replacedNoscript) normalizeAttributeCase(document);
 	};
 };
 //#endregion

package/dist/transforms/dom/mergeConsecutiveOneLinerPres.js

@@ -1,7 +1,15 @@
 //#region src/transforms/dom/mergeConsecutiveOneLinerPres.ts
 const trailingBrRegex = /<br\s*\/?>\s*$/i;
 const surroundingNewlinesRegex = /^\n+|\n+$/g;
-const mergeConsecutiveOneLinerPres = () => {
+const classTokenSeparator = /\s+/;
+const mergeConsecutiveOneLinerPres = ({ preservedPreClasses }) => {
+	const preservedSet = new Set(preservedPreClasses);
+	const isPreserved = (element) => {
+		const classAttribute = element.getAttribute("class");
+		if (!classAttribute) return false;
+		for (const token of classAttribute.split(classTokenSeparator)) if (preservedSet.has(token)) return true;
+		return false;
+	};
 	return (document) => {
 		const pres = document.querySelectorAll("pre");
 		for (const pre of pres) {
@@ -20,6 +28,7 @@ const mergeConsecutiveOneLinerPres = () => {
 				sibling = sibling.nextSibling;
 			}
 			if (run.length < 2) continue;
+			if (run.some(isPreserved)) continue;
 			const isSingleLine = (element) => {
 				return !element.innerHTML.replace(surroundingNewlinesRegex, "").includes("\n");
 			};

package/dist/transforms/dom/proxyAssetUrls.js

@@ -1,5 +1,17 @@
 import { parseSrcset, stringifySrcset } from "srcset";
 //#region src/transforms/dom/proxyAssetUrls.ts
+const proxyableSelectors = [
+	"img",
+	"video",
+	"audio",
+	"source",
+	"track",
+	"image",
+	"[data-embed-thumbnail]",
+	"[data-embed-avatar]",
+	"[data-bookmark-icon]",
+	"[data-bookmark-thumbnail]"
+];
 const sourceTypeFromParent = (element) => {
 	const parent = element.parentElement?.localName;
 	if (parent === "video") return "video";
@@ -30,7 +42,7 @@ const proxySrcset = (element, type, assetProxyFn) => {
 const proxyAssetUrls = ({ assetProxyFn }) => {
 	if (!assetProxyFn) return () => {};
 	return (document) => {
-		const elements = document.querySelectorAll("img, video, audio, source, track, image, [data-embed-thumbnail], [data-embed-avatar]");
+		const elements = document.querySelectorAll(proxyableSelectors.join(", "));
 		for (const element of elements) {
 			switch (element.localName) {
 				case "img":
@@ -57,6 +69,8 @@ const proxyAssetUrls = ({ assetProxyFn }) => {
 			}
 			if (element.hasAttribute("data-embed-thumbnail")) proxyAttribute(element, "data-embed-thumbnail", "image", assetProxyFn);
 			if (element.hasAttribute("data-embed-avatar")) proxyAttribute(element, "data-embed-avatar", "image", assetProxyFn);
+			if (element.hasAttribute("data-bookmark-icon")) proxyAttribute(element, "data-bookmark-icon", "image", assetProxyFn);
+			if (element.hasAttribute("data-bookmark-thumbnail")) proxyAttribute(element, "data-bookmark-thumbnail", "image", assetProxyFn);
 		}
 	};
 };

package/dist/transforms/dom/stripInertElements.d.ts

@@ -0,0 +1,6 @@
+import { DomTransform } from "../../types.js";
+
+//#region src/transforms/dom/stripInertElements.d.ts
+declare const stripInertElements: DomTransform;
+//#endregion
+export { stripInertElements };

package/dist/transforms/dom/stripInertElements.js

@@ -0,0 +1,11 @@
+//#region src/transforms/dom/stripInertElements.ts
+const stripInertElements = ({ inertSelectors }) => {
+	const selector = inertSelectors.join(",");
+	return (document) => {
+		if (!selector) return;
+		const elements = document.querySelectorAll(selector);
+		for (const element of elements) element.remove();
+	};
+};
+//#endregion
+export { stripInertElements };

package/dist/transforms/dom/unwrapEmojiImages.d.ts

@@ -0,0 +1,6 @@
+import { DomTransform } from "../../types.js";
+
+//#region src/transforms/dom/unwrapEmojiImages.d.ts
+declare const unwrapEmojiImages: DomTransform;
+//#endregion
+export { unwrapEmojiImages };

package/dist/transforms/dom/unwrapEmojiImages.js

@@ -0,0 +1,21 @@
+//#region src/transforms/dom/unwrapEmojiImages.ts
+const nonAsciiRegex = /[€-￿]/;
+const asciiLetterRegex = /[a-zA-Z]/;
+const isEmojiShapedAlt = (alt) => {
+	return nonAsciiRegex.test(alt) && !asciiLetterRegex.test(alt);
+};
+const unwrapEmojiImages = (context) => {
+	const selector = [
+		"img.wp-smiley[alt]",
+		"img.emoji[alt]",
+		...context.emojiImageHosts.map((host) => `img[alt][src*="${host}"]`)
+	].join(", ");
+	return (document) => {
+		for (const image of document.querySelectorAll(selector)) {
+			const alt = image.getAttribute("alt");
+			if (alt && isEmojiShapedAlt(alt)) image.replaceWith(document.createTextNode(alt));
+		}
+	};
+};
+//#endregion
+export { unwrapEmojiImages };

package/dist/transforms/dom/unwrapWrappers.js

@@ -7,9 +7,13 @@ const wrapperTags = new Set([
 	"header",
 	"footer"
 ]);
-const hasEmbedAttribute = (element) => {
+const preservedPrefixes = ["data-embed", "data-bookmark"];
+const hasPreservedAttribute = (element) => {
 	const attributes = element.attributes;
-	for (let i = 0, n = attributes.length; i < n; i++) if (attributes[i].name.startsWith("data-embed")) return true;
+	for (let i = 0, n = attributes.length; i < n; i++) {
+		const name = attributes[i].name;
+		for (const prefix of preservedPrefixes) if (name.startsWith(prefix)) return true;
+	}
 	return false;
 };
 const unwrapWrappers = () => {
@@ -20,7 +24,7 @@ const unwrapWrappers = () => {
 			if (!wrapperTags.has(element.localName)) continue;
 			const parent = element.parentNode;
 			if (!parent) continue;
-			if (hasEmbedAttribute(element)) continue;
+			if (hasPreservedAttribute(element)) continue;
 			while (element.firstChild) parent.insertBefore(element.firstChild, element);
 			element.remove();
 		}

package/dist/transforms/string/stripControlChars.d.ts

@@ -0,0 +1,6 @@
+import { StringTransform } from "../../types.js";
+
+//#region src/transforms/string/stripControlChars.d.ts
+declare const stripControlChars: StringTransform;
+//#endregion
+export { stripControlChars };

package/dist/transforms/string/stripControlChars.js

@@ -0,0 +1,21 @@
+//#region src/transforms/string/stripControlChars.ts
+const ranges = [
+	"\\x00-\\x08",
+	"\\x0B\\x0C",
+	"\\x0E-\\x1F",
+	"\\x7F-\\x9F",
+	"\\uFDD0-\\uFDEF",
+	"\\uFFFE\\uFFFF",
+	...Array.from({ length: 16 }, (_, index) => {
+		const plane = (index + 1).toString(16).toUpperCase();
+		return `\\u{${plane}FFFE}\\u{${plane}FFFF}`;
+	})
+];
+const controlCharRegex = new RegExp(`[${ranges.join("")}]`, "gu");
+const stripControlChars = () => {
+	return (html) => {
+		return html.replace(controlCharRegex, "");
+	};
+};
+//#endregion
+export { stripControlChars };

package/dist/transforms/string/stripOversizedBase64Sources.d.ts

@@ -0,0 +1,6 @@
+import { StringTransform } from "../../types.js";
+
+//#region src/transforms/string/stripOversizedBase64Sources.d.ts
+declare const stripOversizedBase64Sources: StringTransform;
+//#endregion
+export { stripOversizedBase64Sources };

package/dist/transforms/string/stripOversizedBase64Sources.js

@@ -0,0 +1,13 @@
+//#region src/transforms/string/stripOversizedBase64Sources.ts
+const base64SrcRegex = /((?:src|srcset|poster)=["'])data:[^"']*;base64,[^"']*(["'])/g;
+const maxBase64Size = 50 * 1024;
+const stripOversizedBase64Sources = () => {
+	return (html) => {
+		return html.replace(base64SrcRegex, (match, prefix, suffix) => {
+			if (match.length < maxBase64Size) return match;
+			return `${prefix}${suffix}`;
+		});
+	};
+};
+//#endregion
+export { stripOversizedBase64Sources };

package/dist/types.d.ts

@@ -41,6 +41,20 @@ type EmbedResolver = {
   selector: string;
   extract: (element: Element) => MaybePromise<EmbedResolverResult | undefined>;
 };
+type BookmarkResolverResult = {
+  provider: string;
+  url: string;
+  title: string;
+  description?: string;
+  author?: string;
+  publisher?: string;
+  icon?: string;
+  thumbnail?: string;
+};
+type BookmarkResolver = {
+  selector: string;
+  extract: (element: Element) => MaybePromise<BookmarkResolverResult | undefined>;
+};
 type UrlUnwrapper = (url: URL) => string | undefined;
 type AssetType = 'image' | 'video' | 'audio';
 type AssetProxyFn = (url: string, type: AssetType) => string | undefined;
@@ -48,10 +62,14 @@ type TransformContext = {
   baseUrl?: string;
   enclosures?: Array<Enclosure>;
   embedResolvers: Array<EmbedResolver>;
+  bookmarkResolvers: Array<BookmarkResolver>;
   lazySrcAttributes: Array<string>;
   lazySrcsetAttributes: Array<string>;
   trackingHosts: Array<string>;
   trackingPathSegments: Array<string>;
+  emojiImageHosts: Array<string>;
+  inertSelectors: Array<string>;
+  preservedPreClasses: Array<string>;
   urlUnwrappers: Array<UrlUnwrapper>;
   resolveUrlFn: ResolveUrlFn;
   assetProxyFn?: AssetProxyFn;
@@ -60,14 +78,20 @@ type TransformContext = {
 };
 type DomTransform = (context: TransformContext) => (document: Document) => MaybePromise<void>;
 type StringTransform = (context: TransformContext) => (html: string) => MaybePromise<string>;
+type ParseHtmlFn = (html: string) => MaybePromise<Document>;
 type TransformContentOptions = {
+  parseHtmlFn: ParseHtmlFn;
   baseUrl?: string;
   enclosures?: Array<Enclosure>;
   embedResolvers?: Array<EmbedResolver>;
+  bookmarkResolvers?: Array<BookmarkResolver>;
   lazySrcAttributes?: Array<string>;
   lazySrcsetAttributes?: Array<string>;
   trackingHosts?: Array<string>;
   trackingPathSegments?: Array<string>;
+  emojiImageHosts?: Array<string>;
+  inertSelectors?: Array<string>;
+  preservedPreClasses?: Array<string>;
   urlUnwrappers?: Array<UrlUnwrapper>;
   resolveUrlFn?: ResolveUrlFn;
   assetProxyFn?: AssetProxyFn;
@@ -75,7 +99,6 @@ type TransformContentOptions = {
   articleTitle?: string;
   stringTransforms?: Array<StringTransform>;
   domTransforms?: Array<DomTransform>;
-  finalStringTransforms?: Array<StringTransform>;
 };
 //#endregion
-export { AssetProxyFn, AssetType, DomTransform, EmbedResolver, EmbedResolverResult, Enclosure, EnrichEmbedFn, MaybePromise, ResolveUrlFn, StringTransform, TransformContentOptions, TransformContext, UrlUnwrapper };
+export { AssetProxyFn, AssetType, BookmarkResolver, BookmarkResolverResult, DomTransform, EmbedResolver, EmbedResolverResult, Enclosure, EnrichEmbedFn, MaybePromise, ParseHtmlFn, ResolveUrlFn, StringTransform, TransformContentOptions, TransformContext, UrlUnwrapper };

package/package.json

@@ -29,6 +29,10 @@
     "./defaults": {
       "types": "./dist/defaults.d.ts",
       "default": "./dist/defaults.js"
+    },
+    "./linkedom": {
+      "types": "./dist/parsers/linkedom.d.ts",
+      "default": "./dist/parsers/linkedom.js"
     }
   },
   "files": [
@@ -36,23 +40,29 @@
   ],
   "scripts": {
     "prepare": "lefthook install",
-    "build": "tsdown src/index.ts src/defaults.ts --format esm --dts --clean --unbundle --no-fixed-extension"
+    "build": "tsdown src/index.ts src/defaults.ts src/parsers/linkedom.ts --format esm --dts --clean --unbundle --no-fixed-extension"
   },
   "dependencies": {
     "@wordpress/autop": "^4.46.0",
     "highlight.js": "^11.11.1",
-    "linkedom": "^0.18.12",
     "linkifyjs": "^4.3.2",
     "srcset": "^5.0.3"
   },
   "peerDependencies": {
-    "feedcanon": "^2.0.0-next.3",
-    "feedscout": "^2.0.0-next.2"
+    "feedcanon": "^2.0.0-next.4",
+    "feedscout": "^2.0.0-next.2",
+    "linkedom": "^0.18.12"
+  },
+  "peerDependenciesMeta": {
+    "linkedom": {
+      "optional": true
+    }
   },
   "devDependencies": {
     "@types/bun": "^1.3.13",
     "kvalita": "^1.13.0",
+    "linkedom": "^0.18.12",
     "tsdown": "^0.22.0"
   },
-  "version": "1.1.0"
+  "version": "2.0.0"
 }