marked-cjk-friendly 0.1.0

package/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2024 Tatsunori Uchino, and authors and contributors of original packages
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,91 @@
+# marked-cjk-friendly
+
+A [Marked.js](https://github.com/markedjs/marked) extension to make Markdown emphasis (`**`) in CommonMark compatible with Chinese, Japanese, and Korean (CJK).
+
+<span lang="ja">CommonMarkの強調記号(`**`)を日本語・中国語・韓国語にきちんと対応させるための[Marked.js](https://github.com/markedjs/marked)拡張</span>
+
+<span lang="zh-Hans-CN">一个 [Marked.js](https://github.com/markedjs/marked) 扩展，用于使 CommonMark 的强调标记(`**`)能够正确支持中文、日语和韩语文本。</span>
+
+<span lang="ko">CommonMark의 강조 표시(`**`) 를 한국어, 일본어, 중국어와 호환되도록 만드는 [Marked.js](https://github.com/markedjs/marked) 확장</span>
+
+## Problem / <span lang="ja">問題</span> / <span lang="zh-Hans-CN">问题</span> / <span lang="ko">문제</span>
+
+CommonMark has a problem that the following emphasis marks `**` are not recognized as emphasis marks in Japanese, Chinese, and Korean.
+
+<span lang="ja">CommonMarkには、日本語・中国語・韓国語内の次のような強調記号(`**`)が強調記号として認識されない問題があります。</span>
+
+<span lang="zh-Hans-CN">CommonMark存在以下问题：在中文、日语和韩语文本中，强调标记`**`不会被识别为强调标记。</span>
+
+<span lang="ko">CommonMark는 한국어, 일본어, 중국어에서 다음과 같은 강조 표시 `**`가 강조 표시로 인식되지 않는 문제가 있습니다.</span>
+
+```md
+**このアスタリスクは強調記号として認識されず、そのまま表示されます。**この文のせいで。
+
+**该星号不会被识别，而是直接显示。**这是因为它没有被识别为强调符号。
+
+**이 별표는 강조 표시로 인식되지 않고 그대로 표시됩니다(이 괄호 때문에)**이 문장 때문에.
+```
+
+CommonMark issue: https://github.com/commonmark/commonmark-spec/issues/650
+
+## Runtime Requirements / <span lang="ja">実行環境の要件</span> / <span lang="zh-Hans-CN">运行环境要求</span> / <span lang="ko">런타임 요구 사항</span>
+
+This package is ESM-only. It requires Node.js 18 or later.
+
+<span lang="ja">本パッケージはESM専用です。Node.js 18以上が必要です。</span>
+
+<span lang="zh-Hans-CN">此包仅支持ESM。需要Node.js 18或更高版本。</span>
+
+<span lang="ko">본 패키지는 ESM 전용입니다. Node.js 18 이상이 필요합니다.</span>
+
+## Installation / <span lang="ja">インストール</span> / <span lang="zh-Hans-CN">安装</span> / <span lang="ko">설치</span>
+
+Install `marked-cjk-friendly` via [npm](https://www.npmjs.com/):
+
+```bash
+npm install marked-cjk-friendly
+```
+
+If you use another package manager, please replace `npm install` with the command of the package manager you use (e.g. `pnpm add` or `yarn add`).
+
+## Usage / <span lang="ja">使い方</span> / <span lang="zh-Hans-CN">用法</span> / <span lang="ko">사용법</span>
+
+Import `marked` and `marked-cjk-friendly`, and use the extension as follows:
+
+```js
+import { Marked } from "marked";
+import markedCjkFriendly from "marked-cjk-friendly";
+
+const marked = new Marked(markedCjkFriendly());
+const html = marked.parse("**この文は太字になりますか？**即便加入这句也可以吗？");
+```
+
+Or using the `use` method:
+
+```js
+import { marked } from "marked";
+import markedCjkFriendly from "marked-cjk-friendly";
+
+marked.use(markedCjkFriendly());
+const html = marked("**この文は太字になりますか？**即便加入这句也可以吗？");
+```
+
+## Compatibility with the other languages / <span lang="ja">他言語との互換性</span> / <span lang="zh-Hans-CN">与其他语言的兼容性</span> / <span lang="ko">다른 언어와의 호환성</span>
+
+This modification of the specification does not affect the other languages than Chinese, Japanese, and Korean. Even if your application or document has translations or content in other languages, it will not be affected, so please feel free to use this package.
+
+<span lang="ja">この仕様変更提案は、日本語・中国語・韓国語以外の言語には影響しません。アプリケーションやドキュメントに他言語の翻訳やコンテンツが含まれていても影響はありませんので、安心して本パッケージをご利用ください。</span>
+
+<span lang="zh-Hans-CN">除中文、日文和韩文外，建议的规范变更不会影响其他语言。请放心使用此软件包。</span>
+
+<span lang="ko">이 사양 변경 제안은 한국어, 일본어, 중국어 이외의 언어에는 영향을 미치지 않습니다. 애플리케이션이나 문서에 다른 언어의 번역이나 콘텐츠가 포함되어 있어도 영향을 받지 않으므로 안심하고 본 패키지를 사용하시기 바랍니다.</span>
+
+## Specification / <span lang="ja">規格書</span> / <span lang="zh-Hans-CN">规范</span> / <span lang="ko">설명서</span>
+
+https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md (English)
+
+## Related packages / <span lang="ja">関連パッケージ</span> / <span lang="zh-Hans-CN">相关包</span> / <span lang="ko">관련 패키지</span>
+
+- [markdown-it-cjk-friendly](https://npmjs.com/package/markdown-it-cjk-friendly) — markdown-it plugin
+- [remark-cjk-friendly](https://npmjs.com/package/remark-cjk-friendly) — remark plugin
+  - [micromark-extension-cjk-friendly](https://npmjs.com/package/micromark-extension-cjk-friendly) — micromark extension

package/dist/index.d.ts

@@ -0,0 +1,18 @@
+import { MarkedExtension } from "marked";
+
+//#region src/index.d.ts
+
+/**
+ * Creates a marked extension that makes emphasis markers CJK-friendly.
+ *
+ * This extension modifies how `*` and `_` emphasis delimiters are detected
+ * so that they work correctly adjacent to CJK (Chinese, Japanese, Korean) characters.
+ *
+ * In standard CommonMark, emphasis delimiters like `**` may not be recognized
+ * when adjacent to CJK punctuation, because CJK characters are not classified
+ * as Unicode punctuation or whitespace. This extension treats CJK characters
+ * as equivalent to punctuation for flanking delimiter detection.
+ */
+declare function markedCjkFriendly(): MarkedExtension;
+//#endregion
+export { markedCjkFriendly as default };

package/dist/index.js

@@ -0,0 +1,184 @@
+//#region src/index.ts
+const CJK = "\\u1100-\\u11ff\\u20a9\\u2329-\\u232a\\u2630-\\u2637\\u268a-\\u268f\\u2e80-\\u2e99\\u2e9b-\\u2ef3\\u2f00-\\u2fd5\\u2ff0-\\u303e\\u3041-\\u3096\\u3099-\\u30ff\\u3105-\\u312f\\u3131-\\u318e\\u3190-\\u31e5\\u31ef-\\u321e\\u3220-\\u3247\\u3250-\\ua48c\\ua490-\\ua4c6\\ua960-\\ua97c\\uac00-\\ud7a3\\ud7b0-\\ud7c6\\ud7cb-\\ud7fb\\uf900-\\ufaff\\ufe10-\\ufe19\\ufe30-\\ufe52\\ufe54-\\ufe66\\ufe68-\\ufe6b\\uff01-\\uffbe\\uffc2-\\uffc7\\uffca-\\uffcf\\uffd2-\\uffd7\\uffda-\\uffdc\\uffe0-\\uffe6\\uffe8-\\uffee\\u{16fe0}-\\u{16fe4}\\u{16ff0}-\\u{16ff6}\\u{17000}-\\u{18cd5}\\u{18cff}-\\u{18d1e}\\u{18d80}-\\u{18df2}\\u{1aff0}-\\u{1aff3}\\u{1aff5}-\\u{1affb}\\u{1affd}-\\u{1affe}\\u{1b000}-\\u{1b122}\\u{1b132}\\u{1b150}-\\u{1b152}\\u{1b155}\\u{1b164}-\\u{1b167}\\u{1b170}-\\u{1b2fb}\\u{1d300}-\\u{1d356}\\u{1d360}-\\u{1d376}\\u{1f200}\\u{1f202}\\u{1f210}-\\u{1f219}\\u{1f21b}-\\u{1f22e}\\u{1f230}-\\u{1f231}\\u{1f237}\\u{1f23b}\\u{1f240}-\\u{1f248}\\u{1f260}-\\u{1f265}\\u{20000}-\\u{3fffd}";
+const cjkTest = new RegExp(`[${CJK}]`, "u");
+const punctuationCjk = new RegExp(`^((?![*_])[\\s\\p{P}\\p{S}${CJK}])`, "u");
+function buildRDelimAst(punct, punctSpace, notPunctSpace) {
+	return new RegExp(`^[^_*]*?__[^_*]*?\\*[^_*]*?(?=__)|[^*]+(?=[^*])|(?!\\*)${punct}(\\*+)(?=[\\s]|$)|${notPunctSpace}(\\*+)(?!\\*)(?=${punctSpace}|$)|(?!\\*)${punctSpace}(\\*+)(?=${notPunctSpace})|[\\s](\\*+)(?!\\*)(?=${punct})|(?!\\*)${punct}(\\*+)(?!\\*)(?=${punct})|${notPunctSpace}(\\*+)(?=${notPunctSpace})`, "gu");
+}
+function buildRDelimUnd(punct, punctSpace, notPunctSpace) {
+	return new RegExp(`^[^_*]*?\\*\\*[^_*]*?_[^_*]*?(?=\\*\\*)|[^_]+(?=[^_])|(?!_)${punct}(_+)(?=[\\s]|$)|${notPunctSpace}(_+)(?!_)(?=${punctSpace}|$)|(?!_)${punctSpace}(_+)(?=${notPunctSpace})|[\\s](_+)(?!_)(?=${punct})|(?!_)${punct}(_+)(?!_)(?=${punct})`, "gu");
+}
+function buildRDelimDel(punct, punctSpace, notPunctSpace) {
+	return new RegExp(`^[^~]+(?=[^~])|(?!~)${punct}(~~?)(?=[\\s]|$)|${notPunctSpace}(~~?)(?!~)(?=${punctSpace}|$)|(?!~)${punctSpace}(~~?)(?=${notPunctSpace})|[\\s](~~?)(?!~)(?=${punct})|(?!~)${punct}(~~?)(?!~)(?=${punct})|${notPunctSpace}(~~?)(?=${notPunctSpace})`, "gu");
+}
+const cjkPunct = `[\\p{P}\\p{S}${CJK}]`;
+const cjkPunctSpace = `[\\s\\p{P}\\p{S}${CJK}]`;
+const cjkNotPunctSpace = `[^\\s\\p{P}\\p{S}${CJK}]`;
+const cjkDelPunct = `(?![*_])[\\p{P}\\p{S}${CJK}]`;
+const cjkDelPunctSpace = `(?![*_])[\\s\\p{P}\\p{S}${CJK}]`;
+const cjkDelNotPunctSpace = `(?:[^\\s\\p{P}\\p{S}${CJK}]|[*_])`;
+const cjkPunctGfm = `(?!~)[\\p{P}\\p{S}${CJK}]`;
+const cjkPunctSpaceGfm = `(?!~)[\\s\\p{P}\\p{S}${CJK}]`;
+const cjkNotPunctSpaceGfm = `(?:[^\\s\\p{P}\\p{S}${CJK}]|~)`;
+const emStrongRDelimAstCjk = buildRDelimAst(cjkPunct, cjkPunctSpace, cjkNotPunctSpace);
+const emStrongRDelimAstCjkGfm = buildRDelimAst(cjkPunctGfm, cjkPunctSpaceGfm, cjkNotPunctSpaceGfm);
+const emStrongRDelimUndCjk = buildRDelimUnd(cjkPunct, cjkPunctSpace, cjkNotPunctSpace);
+const delRDelimCjk = buildRDelimDel(cjkDelPunct, cjkDelPunctSpace, cjkDelNotPunctSpace);
+/**
+* Checks whether the character immediately before `src` should be treated as CJK.
+*
+* Marked passes `prevChar` as a single UTF-16 code unit, so supplementary-plane
+* characters may arrive as a lone surrogate and variation sequences may point at
+* the variation selector instead of the base character. Recover from `maskedSrc`
+* when needed so CJK-aware flanking checks see the actual preceding code point.
+*/
+function isPrevCharCjk(prevChar, maskedSrc, src) {
+	let prevIsCjk = punctuationCjk.test(prevChar);
+	if (!prevIsCjk && prevChar) {
+		const prevIdx = maskedSrc.length - src.length;
+		if (prevIdx >= 1) {
+			let idx = prevIdx - 1;
+			let code = maskedSrc.charCodeAt(idx);
+			if (code >= 65024 && code <= 65038 && idx >= 1) {
+				idx--;
+				code = maskedSrc.charCodeAt(idx);
+			}
+			if ((code & 64512) === 56320 && idx >= 1) {
+				const cp = maskedSrc.codePointAt(idx - 1);
+				if (cp !== void 0 && cp > 65535) if (cp >= 917760 && cp <= 917999) prevIsCjk = true;
+				else prevIsCjk = cjkTest.test(String.fromCodePoint(cp));
+			} else prevIsCjk = cjkTest.test(String.fromCharCode(code));
+		}
+	}
+	return prevIsCjk;
+}
+/**
+* Checks whether a matched right delimiter is immediately adjacent to CJK text.
+*
+* Marked classifies these matches as left-only, right-only, or both using regex
+* groups. When CJK sits next to the delimiter, we treat the delimiter as "both"
+* so GFM emphasis/strikethrough can close across CJK punctuation boundaries.
+*/
+function isCjkAdjacentToDelimiter(rightMatch, clippedMaskedSrc) {
+	if (!(rightMatch[1] || rightMatch[2] || rightMatch[3] || rightMatch[4])) return false;
+	const charBefore = String.fromCodePoint(rightMatch[0].codePointAt(0));
+	const afterPos = rightMatch.index + rightMatch[0].length;
+	const charAfter = afterPos < clippedMaskedSrc.length ? String.fromCodePoint(clippedMaskedSrc.codePointAt(afterPos)) : "";
+	return cjkTest.test(charBefore) || cjkTest.test(charAfter);
+}
+/**
+* Creates a marked extension that makes emphasis markers CJK-friendly.
+*
+* This extension modifies how `*` and `_` emphasis delimiters are detected
+* so that they work correctly adjacent to CJK (Chinese, Japanese, Korean) characters.
+*
+* In standard CommonMark, emphasis delimiters like `**` may not be recognized
+* when adjacent to CJK punctuation, because CJK characters are not classified
+* as Unicode punctuation or whitespace. This extension treats CJK characters
+* as equivalent to punctuation for flanking delimiter detection.
+*/
+function markedCjkFriendly() {
+	return { tokenizer: {
+		emStrong(src, maskedSrc, prevChar = "") {
+			const { rules } = this;
+			const match = rules.inline.emStrongLDelim.exec(src);
+			if (!match) return false;
+			if (match[3] && prevChar.match(rules.other.unicodeAlphaNumeric)) return false;
+			const nextChar = match[1] || match[2] || "";
+			const prevIsCjk = isPrevCharCjk(prevChar, maskedSrc, src);
+			if (!nextChar || !prevChar || rules.inline.punctuation.exec(prevChar) || prevIsCjk || cjkTest.test(nextChar)) {
+				const lLength = [...match[0]].length - 1;
+				let rDelim;
+				let rLength;
+				let delimTotal = lLength;
+				let midDelimTotal = 0;
+				const isGfm = rules.inline.emStrongRDelimAst.source.includes("(?!~)");
+				let endReg;
+				if (match[0][0] === "*") endReg = isGfm ? emStrongRDelimAstCjkGfm : emStrongRDelimAstCjk;
+				else endReg = emStrongRDelimUndCjk;
+				endReg.lastIndex = 0;
+				const clippedMaskedSrc = maskedSrc.slice(-1 * src.length + lLength);
+				let rMatch;
+				while ((rMatch = endReg.exec(clippedMaskedSrc)) != null) {
+					rDelim = rMatch[1] || rMatch[2] || rMatch[3] || rMatch[4] || rMatch[5] || rMatch[6];
+					if (!rDelim) continue;
+					rLength = [...rDelim].length;
+					const isCjkAdjacent = isCjkAdjacentToDelimiter(rMatch, clippedMaskedSrc);
+					if ((rMatch[3] || rMatch[4]) && !isCjkAdjacent) {
+						delimTotal += rLength;
+						continue;
+					}
+					if (rMatch[5] || rMatch[6] || isCjkAdjacent) {
+						if (lLength % 3 && !((lLength + rLength) % 3)) {
+							midDelimTotal += rLength;
+							continue;
+						}
+					}
+					delimTotal -= rLength;
+					if (delimTotal > 0) continue;
+					rLength = Math.min(rLength, rLength + delimTotal + midDelimTotal);
+					const lastCharLength = rMatch[0].codePointAt(0) > 65535 ? 2 : 1;
+					const raw = src.slice(0, lLength + rMatch.index + lastCharLength + rLength);
+					if (Math.min(lLength, rLength) % 2) {
+						const text$1 = raw.slice(1, -1);
+						return {
+							type: "em",
+							raw,
+							text: text$1,
+							tokens: this.lexer.inlineTokens(text$1)
+						};
+					}
+					const text = raw.slice(2, -2);
+					return {
+						type: "strong",
+						raw,
+						text,
+						tokens: this.lexer.inlineTokens(text)
+					};
+				}
+			}
+		},
+		del(src, maskedSrc, prevChar = "") {
+			const { rules } = this;
+			const match = rules.inline.delLDelim.exec(src);
+			if (!match) return false;
+			const nextChar = match[1] || "";
+			const prevIsCjk = isPrevCharCjk(prevChar, maskedSrc, src);
+			if (!nextChar || !prevChar || rules.inline.punctuation.exec(prevChar) || prevIsCjk || cjkTest.test(nextChar)) {
+				const lLength = [...match[0]].length - 1;
+				let rDelim;
+				let rLength;
+				let delimTotal = lLength;
+				delRDelimCjk.lastIndex = 0;
+				const clippedMaskedSrc = maskedSrc.slice(-1 * src.length + lLength);
+				let rMatch;
+				while ((rMatch = delRDelimCjk.exec(clippedMaskedSrc)) != null) {
+					rDelim = rMatch[1] || rMatch[2] || rMatch[3] || rMatch[4] || rMatch[5] || rMatch[6];
+					if (!rDelim) continue;
+					rLength = [...rDelim].length;
+					if (rLength !== lLength) continue;
+					const isCjkAdjacent = isCjkAdjacentToDelimiter(rMatch, clippedMaskedSrc);
+					if ((rMatch[3] || rMatch[4]) && !isCjkAdjacent) {
+						delimTotal += rLength;
+						continue;
+					}
+					delimTotal -= rLength;
+					if (delimTotal > 0) continue;
+					rLength = Math.min(rLength, rLength + delimTotal);
+					const lastCharLength = rMatch[0].codePointAt(0) > 65535 ? 2 : 1;
+					const raw = src.slice(0, lLength + rMatch.index + lastCharLength + rLength);
+					const text = raw.slice(lLength, -lLength);
+					return {
+						type: "del",
+						raw,
+						text,
+						tokens: this.lexer.inlineTokens(text)
+					};
+				}
+			}
+		}
+	} };
+}
+
+//#endregion
+export { markedCjkFriendly as default };

package/package.json

@@ -0,0 +1,59 @@
+{
+  "name": "marked-cjk-friendly",
+  "version": "0.1.0",
+  "type": "module",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "default": "./dist/index.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "files": [
+    "dist",
+    "LICENSE",
+    "README.md"
+  ],
+  "repository": {
+    "url": "https://github.com/tats-u/markdown-cjk-friendly"
+  },
+  "license": "MIT",
+  "author": "Tatsunori Uchino <tats.u@live.jp> (https://github.com/tats-u)",
+  "bugs": "https://github.com/tats-u/markdown-cjk-friendly/issues",
+  "keywords": [
+    "marked",
+    "marked-extension",
+    "markdown",
+    "japanese",
+    "chinese",
+    "korean",
+    "cjk"
+  ],
+  "description": "Marked.js extension to make Markdown emphasis (`**`) in CommonMark more friendly with Chinese, Japanese, and Korean (CJK)",
+  "sideEffects": false,
+  "devDependencies": {
+    "marked": "^17.0.0"
+  },
+  "peerDependencies": {
+    "marked": ">=15.0.0"
+  },
+  "engines": {
+    "node": ">=18"
+  },
+  "scripts": {
+    "build": "tsdown",
+    "build:lib": "tsdown",
+    "dev": "tsdown --watch --sourcemaps",
+    "dev:lib": "tsdown --watch --sourcemaps",
+    "test": "vitest run",
+    "test:lib": "vitest run",
+    "test:up": "vitest -u",
+    "test:watch": "vitest watch",
+    "test:lib:watch": "vitest watch",
+    "bench": "vitest bench --run",
+    "bench:watch": "vitest bench",
+    "lint:type": "tsc --noEmit"
+  }
+}