marktip 0.7.2__tar.gz → 0.7.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. {marktip-0.7.2 → marktip-0.7.3}/CMakeLists.txt +1 -1
  2. marktip-0.7.2/README.md → marktip-0.7.3/PKG-INFO +37 -2
  3. marktip-0.7.2/PKG-INFO → marktip-0.7.3/README.md +8 -30
  4. {marktip-0.7.2 → marktip-0.7.3}/pyproject.toml +5 -3
  5. marktip-0.7.3/scripts/build_cjk_map.py +100 -0
  6. {marktip-0.7.2 → marktip-0.7.3}/src/marktip.cpp +2 -2
  7. marktip-0.7.3/tests/test_cjk_friendly.py +94 -0
  8. marktip-0.7.3/third_party/md4c/src/cjk.h +84 -0
  9. {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/md4c.c +116 -115
  10. {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/upstream_metadata/README.md +24 -6
  11. marktip-0.7.3/third_party/md4c/upstream_metadata/Unicode-LICENSE.txt +39 -0
  12. {marktip-0.7.2 → marktip-0.7.3}/.github/workflows/build-distributions.yml +0 -0
  13. {marktip-0.7.2 → marktip-0.7.3}/.gitignore +0 -0
  14. {marktip-0.7.2 → marktip-0.7.3}/LICENSE +0 -0
  15. {marktip-0.7.2 → marktip-0.7.3}/python/marktip/__init__.py +0 -0
  16. {marktip-0.7.2 → marktip-0.7.3}/scripts/benchmark.py +0 -0
  17. {marktip-0.7.2 → marktip-0.7.3}/src/ast.cpp +0 -0
  18. {marktip-0.7.2 → marktip-0.7.3}/src/ast.h +0 -0
  19. {marktip-0.7.2 → marktip-0.7.3}/src/errors.h +0 -0
  20. {marktip-0.7.2 → marktip-0.7.3}/src/parser.cpp +0 -0
  21. {marktip-0.7.2 → marktip-0.7.3}/src/parser.h +0 -0
  22. {marktip-0.7.2 → marktip-0.7.3}/src/serializer.cpp +0 -0
  23. {marktip-0.7.2 → marktip-0.7.3}/src/serializer.h +0 -0
  24. {marktip-0.7.2 → marktip-0.7.3}/tests/test_content_model.py +0 -0
  25. {marktip-0.7.2 → marktip-0.7.3}/tests/test_document.py +0 -0
  26. {marktip-0.7.2 → marktip-0.7.3}/tests/test_errors.py +0 -0
  27. {marktip-0.7.2 → marktip-0.7.3}/tests/test_node_type_policy.py +0 -0
  28. {marktip-0.7.2 → marktip-0.7.3}/tests/test_nul_normalization.py +0 -0
  29. {marktip-0.7.2 → marktip-0.7.3}/tests/test_parse.py +0 -0
  30. {marktip-0.7.2 → marktip-0.7.3}/tests/test_roundtrip.py +0 -0
  31. {marktip-0.7.2 → marktip-0.7.3}/tests/test_serialize.py +0 -0
  32. {marktip-0.7.2 → marktip-0.7.3}/tests/test_strict.py +0 -0
  33. {marktip-0.7.2 → marktip-0.7.3}/tests/test_uri_policy.py +0 -0
  34. {marktip-0.7.2 → marktip-0.7.3}/tests/test_validate.py +0 -0
  35. {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/LICENSE.md +0 -0
  36. {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/entity.c +0 -0
  37. {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/entity.h +0 -0
  38. {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/md4c.h +0 -0
@@ -7,7 +7,7 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
7
7
  set(CMAKE_CXX_EXTENSIONS OFF)
8
8
 
9
9
  if(NOT DEFINED SKBUILD_PROJECT_VERSION)
10
- set(SKBUILD_PROJECT_VERSION "0.7.2")
10
+ set(SKBUILD_PROJECT_VERSION "0.7.3")
11
11
  endif()
12
12
 
13
13
  set(PYBIND11_FINDPYTHON ON)
@@ -1,3 +1,32 @@
1
+ Metadata-Version: 2.4
2
+ Name: marktip
3
+ Version: 0.7.3
4
+ Summary: Fast C++/MD4C Markdown parser and canonical Markdown serializer for Tiptap-style JSON
5
+ Keywords: markdown,tiptap,md4c,parser
6
+ Author: Saneaven (Minjae Kyung)
7
+ License-Expression: MIT AND Unicode-3.0
8
+ License-File: LICENSE
9
+ License-File: third_party/md4c/LICENSE.md
10
+ License-File: third_party/md4c/upstream_metadata/Unicode-LICENSE.txt
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Operating System :: MacOS
14
+ Classifier: Operating System :: Microsoft :: Windows
15
+ Classifier: Operating System :: POSIX :: Linux
16
+ Classifier: Programming Language :: C++
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.9
19
+ Classifier: Programming Language :: Python :: 3.10
20
+ Classifier: Programming Language :: Python :: 3.11
21
+ Classifier: Programming Language :: Python :: 3.12
22
+ Classifier: Programming Language :: Python :: 3.13
23
+ Classifier: Programming Language :: Python :: 3.14
24
+ Classifier: Topic :: Text Processing :: Markup :: Markdown
25
+ Requires-Python: >=3.9
26
+ Provides-Extra: test
27
+ Requires-Dist: pytest>=8; extra == "test"
28
+ Description-Content-Type: text/markdown
29
+
1
30
  # marktip
2
31
 
3
32
  Fast C++/MD4C Markdown conversion for Tiptap-style JSON.
@@ -24,14 +53,20 @@ doc = tm.from_dict(ast)
24
53
  ```
25
54
 
26
55
  `from_markdown` follows GFM/CommonMark by default.
27
- Pass `cjk_friendly=True` to relax the emphasis and strikethrough rules so delimiters next to CJK text still open and close (e.g. `**볼드**은` parses as bold), a non-standard extension that is off by default:
56
+ Pass `cjk_friendly=True` to follow the [CommonMark CJK-friendly Amendments Specification](https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md) for emphasis (`*`, `_`) and strikethrough (`~`). This opt-in extension recognizes CJK punctuation as well as letters, so `**《제목》**a`, `a**《제목》**`, and `**「約束」**a` retain their bold marks:
28
57
 
29
58
  ```python
30
59
  doc = tm.from_markdown("**볼드**은 강조", cjk_friendly=True)
31
60
  ```
32
61
 
33
62
  Strikethrough follows the GFM reference implementations (cmark-gfm/micromark): intra-word `~~` strikes (`a~~b~~c`), including next to CJK letters (`~~삭제~~은`).
34
- `cjk_friendly` still matters for `*`/`_`, and for `~` in punctuation-adjacent cases such as `~~"인용"~~라고`.
63
+ `cjk_friendly` still matters for punctuation-adjacent cases such as `~~"인용"~~라고`.
64
+
65
+ The CJK classifier uses Unicode 17.0.0 East Asian Width `W`, `F`, or `H` (excluding default emoji presentation characters), plus the Hangul script. This includes fullwidth digits such as `1`. Before a delimiter, CJK characters followed by a non-emoji general-use variation selector (`U+FE00`–`U+FE0E`) are classified as one sequence; ideographic variation selectors (`U+E0100`–`U+E01EF`) and standardized fullwidth quotation-mark sequences are also recognized. `U+FE0F` is an emoji presentation selector and is excluded from that sequence rule.
66
+
67
+ Underscores retain CommonMark's additional restriction on emphasis inside words. The CJK-friendly specification changes delimiter flanking without removing that protection for names such as `snake_case`: `_《제목》_을` and `__《제목》__을` stay literal, even with `cjk_friendly=True`. Use `*` or `**` when emphasis must adjoin a word.
68
+
69
+ > **Changed in 0.7.3** — CJK-friendly character classification, variation-selector sequences, flanking, and underscore conditions follow the specification. The same flanking rules apply to `~~`. An editor document containing bold `《제목》` followed by `a` now survives serialization and reparsing with `cjk_friendly=True`.
35
70
 
36
71
  Pass `html=False` to parse raw HTML as literal text instead of `htmlBlock`/`htmlInline` nodes.
37
72
  Content is preserved (the serializer escapes it), and `<br>` inside table cells still maps to `hardBreak` so marktip's own table output round-trips:
@@ -1,31 +1,3 @@
1
- Metadata-Version: 2.4
2
- Name: marktip
3
- Version: 0.7.2
4
- Summary: Fast C++/MD4C Markdown parser and canonical Markdown serializer for Tiptap-style JSON
5
- Keywords: markdown,tiptap,md4c,parser
6
- Author: Saneaven (Minjae Kyung)
7
- License-Expression: MIT
8
- License-File: LICENSE
9
- License-File: third_party/md4c/LICENSE.md
10
- Classifier: Development Status :: 3 - Alpha
11
- Classifier: Intended Audience :: Developers
12
- Classifier: Operating System :: MacOS
13
- Classifier: Operating System :: Microsoft :: Windows
14
- Classifier: Operating System :: POSIX :: Linux
15
- Classifier: Programming Language :: C++
16
- Classifier: Programming Language :: Python :: 3
17
- Classifier: Programming Language :: Python :: 3.9
18
- Classifier: Programming Language :: Python :: 3.10
19
- Classifier: Programming Language :: Python :: 3.11
20
- Classifier: Programming Language :: Python :: 3.12
21
- Classifier: Programming Language :: Python :: 3.13
22
- Classifier: Programming Language :: Python :: 3.14
23
- Classifier: Topic :: Text Processing :: Markup :: Markdown
24
- Requires-Python: >=3.9
25
- Provides-Extra: test
26
- Requires-Dist: pytest>=8; extra == "test"
27
- Description-Content-Type: text/markdown
28
-
29
1
  # marktip
30
2
 
31
3
  Fast C++/MD4C Markdown conversion for Tiptap-style JSON.
@@ -52,14 +24,20 @@ doc = tm.from_dict(ast)
52
24
  ```
53
25
 
54
26
  `from_markdown` follows GFM/CommonMark by default.
55
- Pass `cjk_friendly=True` to relax the emphasis and strikethrough rules so delimiters next to CJK text still open and close (e.g. `**볼드**은` parses as bold), a non-standard extension that is off by default:
27
+ Pass `cjk_friendly=True` to follow the [CommonMark CJK-friendly Amendments Specification](https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md) for emphasis (`*`, `_`) and strikethrough (`~`). This opt-in extension recognizes CJK punctuation as well as letters, so `**《제목》**a`, `a**《제목》**`, and `**「約束」**a` retain their bold marks:
56
28
 
57
29
  ```python
58
30
  doc = tm.from_markdown("**볼드**은 강조", cjk_friendly=True)
59
31
  ```
60
32
 
61
33
  Strikethrough follows the GFM reference implementations (cmark-gfm/micromark): intra-word `~~` strikes (`a~~b~~c`), including next to CJK letters (`~~삭제~~은`).
62
- `cjk_friendly` still matters for `*`/`_`, and for `~` in punctuation-adjacent cases such as `~~"인용"~~라고`.
34
+ `cjk_friendly` still matters for punctuation-adjacent cases such as `~~"인용"~~라고`.
35
+
36
+ The CJK classifier uses Unicode 17.0.0 East Asian Width `W`, `F`, or `H` (excluding default emoji presentation characters), plus the Hangul script. This includes fullwidth digits such as `1`. Before a delimiter, CJK characters followed by a non-emoji general-use variation selector (`U+FE00`–`U+FE0E`) are classified as one sequence; ideographic variation selectors (`U+E0100`–`U+E01EF`) and standardized fullwidth quotation-mark sequences are also recognized. `U+FE0F` is an emoji presentation selector and is excluded from that sequence rule.
37
+
38
+ Underscores retain CommonMark's additional restriction on emphasis inside words. The CJK-friendly specification changes delimiter flanking without removing that protection for names such as `snake_case`: `_《제목》_을` and `__《제목》__을` stay literal, even with `cjk_friendly=True`. Use `*` or `**` when emphasis must adjoin a word.
39
+
40
+ > **Changed in 0.7.3** — CJK-friendly character classification, variation-selector sequences, flanking, and underscore conditions follow the specification. The same flanking rules apply to `~~`. An editor document containing bold `《제목》` followed by `a` now survives serialization and reparsing with `cjk_friendly=True`.
63
41
 
64
42
  Pass `html=False` to parse raw HTML as literal text instead of `htmlBlock`/`htmlInline` nodes.
65
43
  Content is preserved (the serializer escapes it), and `<br>` inside table cells still maps to `hardBreak` so marktip's own table output round-trips:
@@ -4,12 +4,12 @@ build-backend = "scikit_build_core.build"
4
4
 
5
5
  [project]
6
6
  name = "marktip"
7
- version = "0.7.2"
7
+ version = "0.7.3"
8
8
  description = "Fast C++/MD4C Markdown parser and canonical Markdown serializer for Tiptap-style JSON"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.9"
11
- license = "MIT"
12
- license-files = ["LICENSE", "third_party/md4c/LICENSE.md"]
11
+ license = "MIT AND Unicode-3.0"
12
+ license-files = ["LICENSE", "third_party/md4c/LICENSE.md", "third_party/md4c/upstream_metadata/Unicode-LICENSE.txt"]
13
13
  authors = [{ name = "Saneaven (Minjae Kyung)" }]
14
14
  keywords = ["markdown", "tiptap", "md4c", "parser"]
15
15
  classifiers = [
@@ -38,9 +38,11 @@ wheel.packages = ["python/marktip"]
38
38
  sdist.include = [
39
39
  "third_party/md4c/src/md4c.c",
40
40
  "third_party/md4c/src/md4c.h",
41
+ "third_party/md4c/src/cjk.h",
41
42
  "third_party/md4c/src/entity.c",
42
43
  "third_party/md4c/src/entity.h",
43
44
  "third_party/md4c/LICENSE.md",
45
+ "third_party/md4c/upstream_metadata/Unicode-LICENSE.txt",
44
46
  ]
45
47
 
46
48
  [tool.pytest.ini_options]
@@ -0,0 +1,100 @@
1
+ """Generate the CJK-friendly classifier from Unicode 17.0.0 data files."""
2
+
3
+ import argparse
4
+ import hashlib
5
+ from pathlib import Path
6
+ import re
7
+
8
+
9
+ VERSION = "17.0.0"
10
+ SOURCES = {
11
+ "EastAsianWidth.txt": "EastAsianWidth.txt",
12
+ "Scripts.txt": "Scripts.txt",
13
+ "emoji-data.txt": "emoji/emoji-data.txt",
14
+ "StandardizedVariants.txt": "StandardizedVariants.txt",
15
+ }
16
+
17
+
18
+ def records(path):
19
+ for line in path.read_text(encoding="utf-8").splitlines():
20
+ body = line.split("#", 1)[0].strip()
21
+ if body:
22
+ yield [field.strip() for field in body.split(";")]
23
+
24
+
25
+ def codepoints(value):
26
+ parts = value.split("..")
27
+ return range(int(parts[0], 16), int(parts[-1], 16) + 1)
28
+
29
+
30
+ def intervals(values):
31
+ start = previous = None
32
+ for value in sorted(values):
33
+ if previous is not None and value != previous + 1:
34
+ yield start, previous
35
+ start = None
36
+ if start is None:
37
+ start = value
38
+ previous = value
39
+ if previous is not None:
40
+ yield start, previous
41
+
42
+
43
+ def generate(directory):
44
+ for filename in SOURCES:
45
+ header = (directory / filename).read_text(encoding="utf-8")[:1000]
46
+ if not re.search(r"17\.0(?:\.0)?\b", header):
47
+ raise ValueError(f"{filename} must contain Unicode {VERSION} data")
48
+ widths = {}
49
+ for bounds, width in records(directory / "EastAsianWidth.txt"):
50
+ widths.update(dict.fromkeys(codepoints(bounds), width))
51
+ hangul = set()
52
+ for bounds, script in records(directory / "Scripts.txt"):
53
+ if script == "Hangul":
54
+ hangul.update(codepoints(bounds))
55
+ emoji = set()
56
+ for bounds, property_name in records(directory / "emoji-data.txt"):
57
+ if property_name == "Emoji_Presentation":
58
+ emoji.update(codepoints(bounds))
59
+ cjk = {cp for cp, width in widths.items() if width in {"W", "F", "H"}} - emoji
60
+ cjk.update(hangul)
61
+ ambiguous = []
62
+ for sequence, description, *_ in records(directory / "StandardizedVariants.txt"):
63
+ base, selector = (int(cp, 16) for cp in sequence.split())
64
+ if (
65
+ widths.get(base) == "A"
66
+ and re.search(r"\bfullwidth form\b", description)
67
+ and "non-fullwidth" not in description
68
+ ):
69
+ ambiguous.append((base, selector))
70
+
71
+ lines = [
72
+ "/* Generated by scripts/build_cjk_map.py; Unicode " + VERSION + ".",
73
+ " * Data license: upstream_metadata/Unicode-LICENSE.txt.",
74
+ ]
75
+ for filename, source in SOURCES.items():
76
+ digest = hashlib.sha256((directory / filename).read_bytes()).hexdigest()
77
+ lines.append(" * https://www.unicode.org/Public/" + VERSION + "/ucd/" + source)
78
+ lines.append(" * SHA-256: " + digest)
79
+ lines.extend([
80
+ " */",
81
+ "#define R(cp_min, cp_max) ((cp_min) | 0x40000000), ((cp_max) | 0x80000000)",
82
+ "#define S(cp) (cp)",
83
+ "static const unsigned MD_CJK_MAP[] = {",
84
+ ])
85
+ for start, end in intervals(cjk):
86
+ value = f"S(0x{start:04x})" if start == end else f"R(0x{start:04x},0x{end:04x})"
87
+ lines.append(" " + value + ",")
88
+ lines.extend(["};", "#undef R", "#undef S", "static const unsigned MD_CJK_AMBIGUOUS_PUNCT[][2] = {"])
89
+ for base, selector in sorted(ambiguous):
90
+ lines.append(f" {{ 0x{base:04x}, 0x{selector:04x} }},")
91
+ lines.extend(["};", ""])
92
+ return "\n".join(lines)
93
+
94
+
95
+ if __name__ == "__main__":
96
+ parser = argparse.ArgumentParser(description=__doc__)
97
+ parser.add_argument("ucd_directory", type=Path)
98
+ parser.add_argument("--output", type=Path, default=Path("third_party/md4c/src/cjk.h"))
99
+ args = parser.parse_args()
100
+ args.output.write_text(generate(args.ucd_directory), encoding="utf-8")
@@ -11,7 +11,7 @@
11
11
  namespace py = pybind11;
12
12
 
13
13
  #ifndef MARKTIP_VERSION
14
- #define MARKTIP_VERSION "0.7.2"
14
+ #define MARKTIP_VERSION "0.7.3"
15
15
  #endif
16
16
 
17
17
  namespace {
@@ -125,7 +125,7 @@ PYBIND11_MODULE(_core, module) {
125
125
  // pybind11 takes the doc as a bare const char* and only copies it while def() runs.
126
126
  const std::string from_markdown_doc =
127
127
  "Parse Markdown into a Document."
128
- " Set cjk_friendly=True to relax the emphasis rules around CJK text"
128
+ " Set cjk_friendly=True to use CJK-friendly emphasis and strikethrough rules"
129
129
  " (non-standard extension; the default follows GFM/CommonMark exactly)."
130
130
  " Set html=False to parse raw HTML as literal text instead of htmlBlock/htmlInline nodes;"
131
131
  " <br> inside table cells still maps to hardBreak." +
@@ -0,0 +1,94 @@
1
+ import pytest
2
+
3
+ import marktip as tm
4
+
5
+
6
+ def marked_texts(markdown, mark_type, *, cjk_friendly=True):
7
+ content = tm.from_markdown(markdown, cjk_friendly=cjk_friendly).to_dict()["content"][0]["content"]
8
+ return [node["text"] for node in content if {"type": mark_type} in node.get("marks", [])]
9
+
10
+
11
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
12
+ @pytest.mark.parametrize("prefix,suffix", [("", "a"), ("a", ""), ("a", "a"), ("", "을")])
13
+ @pytest.mark.parametrize("value", ["《제목》", "「約束」"])
14
+ def test_cjk_punctuation_adjoins_latin_or_hangul(delimiter, mark_type, prefix, suffix, value):
15
+ markdown = prefix + delimiter + value + delimiter + suffix
16
+ assert marked_texts(markdown, mark_type) == [value]
17
+ if prefix == "a" or suffix == "a":
18
+ assert marked_texts(markdown, mark_type, cjk_friendly=False) == []
19
+
20
+
21
+ @pytest.mark.parametrize("value", ["1", "₩", "カ", "ᅡ", "〰", "〽", "㊗", "㊙", "🈂", "🈷", "𠀀"])
22
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
23
+ def test_unicode_width_and_hangul_script_define_cjk(value, delimiter, mark_type):
24
+ assert marked_texts("a" + delimiter + value + delimiter + "a", mark_type) == [value]
25
+ assert marked_texts(value + delimiter + '"제목"' + delimiter + value, mark_type) == ['"제목"']
26
+
27
+
28
+ @pytest.mark.parametrize("value", ["😀", "⌚", "🈁", "©", "☀"])
29
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
30
+ def test_default_emoji_and_non_cjk_symbols_do_not_relax_flanking(value, delimiter, mark_type):
31
+ assert marked_texts("a" + delimiter + value + delimiter + "a", mark_type) == []
32
+
33
+
34
+ @pytest.mark.parametrize("selector", ["\ufe00", "\ufe01", "\ufe0e", "\U000e0100", "\U000e01ef"])
35
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
36
+ def test_cjk_variation_sequence_before_opener(selector, delimiter, mark_type):
37
+ markdown = "漢" + selector + delimiter + '"제목"' + delimiter + "1"
38
+ assert marked_texts(markdown, mark_type) == ['"제목"']
39
+ assert marked_texts(markdown, mark_type, cjk_friendly=False) == []
40
+
41
+
42
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
43
+ def test_emoji_selector_does_not_form_a_cjk_sequence(delimiter, mark_type):
44
+ assert marked_texts("漢\ufe0f" + delimiter + '"제목"' + delimiter + "1", mark_type) == []
45
+
46
+
47
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
48
+ def test_non_cjk_punctuation_sequence_still_blocks_closing(delimiter, mark_type):
49
+ assert marked_texts("a" + delimiter + "x.\ufe00" + delimiter + "b", mark_type) == []
50
+ assert marked_texts("a" + delimiter + "x.\ufe0f" + delimiter + "b", mark_type) == ["x.\ufe0f"]
51
+
52
+
53
+ @pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
54
+ @pytest.mark.parametrize("value", ["x.\ufe00", "x”\ufe00"])
55
+ def test_punctuation_sequence_can_close_before_cjk(delimiter, mark_type, value):
56
+ assert marked_texts("a" + delimiter + value + delimiter + "字", mark_type) == [value]
57
+
58
+
59
+ @pytest.mark.parametrize("quote", ["‘", "’", "“", "”"])
60
+ def test_fullwidth_ambiguous_punctuation_sequence(quote):
61
+ value = "제목" + quote + "\ufe01"
62
+ assert marked_texts("a**" + value + "**a", "bold") == [value]
63
+ assert marked_texts("a**제목" + quote + "\ufe00**a", "bold") == []
64
+
65
+
66
+ @pytest.mark.parametrize("delimiter", ["_", "__"])
67
+ @pytest.mark.parametrize("prefix,suffix", [("", "을"), ("a", ""), ("a", "a")])
68
+ def test_underscores_retain_commonmark_intraword_restrictions(delimiter, prefix, suffix):
69
+ markdown = prefix + delimiter + "《제목》" + delimiter + suffix
70
+ assert tm.from_markdown(markdown, cjk_friendly=True).to_dict()["content"][0]["content"] == [
71
+ {"type": "text", "text": markdown}
72
+ ]
73
+
74
+
75
+ @pytest.mark.parametrize("delimiter,mark_type", [("_", "italic"), ("__", "bold")])
76
+ def test_underscores_accept_preceding_punctuation_sequence(delimiter, mark_type):
77
+ assert marked_texts("a.\ufe00" + delimiter + "제목" + delimiter + "!", mark_type) == ["제목"]
78
+ assert marked_texts("(" + delimiter + "《제목》" + delimiter + ")", mark_type) == ["《제목》"]
79
+
80
+
81
+ @pytest.mark.parametrize("mark_type", ["bold", "italic", "strike"])
82
+ def test_editor_mark_on_cjk_punctuation_roundtrips_before_latin(mark_type):
83
+ ast = {
84
+ "type": "doc",
85
+ "content": [{
86
+ "type": "paragraph",
87
+ "content": [
88
+ {"type": "text", "text": "《제목》", "marks": [{"type": mark_type}]},
89
+ {"type": "text", "text": "a"},
90
+ ],
91
+ }],
92
+ }
93
+ markdown = tm.from_dict(ast).to_markdown()
94
+ assert tm.from_markdown(markdown, cjk_friendly=True).to_dict() == ast
@@ -0,0 +1,84 @@
1
+ /* Generated by scripts/build_cjk_map.py; Unicode 17.0.0.
2
+ * Data license: upstream_metadata/Unicode-LICENSE.txt.
3
+ * https://www.unicode.org/Public/17.0.0/ucd/EastAsianWidth.txt
4
+ * SHA-256: ea7ce50f3444a050333448dffef1cadd9325af55cbb764b4a2280faf52170a33
5
+ * https://www.unicode.org/Public/17.0.0/ucd/Scripts.txt
6
+ * SHA-256: 9f5e50d3abaee7d6ce09480f325c706f485ae3240912527e651954d2d6b035bf
7
+ * https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt
8
+ * SHA-256: 2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b
9
+ * https://www.unicode.org/Public/17.0.0/ucd/StandardizedVariants.txt
10
+ * SHA-256: f55100b2fb11d3d75a37b8c1ab752192dbd1c4b12328c5ec6b38e3807c0ca597
11
+ */
12
+ #define R(cp_min, cp_max) ((cp_min) | 0x40000000), ((cp_max) | 0x80000000)
13
+ #define S(cp) (cp)
14
+ static const unsigned MD_CJK_MAP[] = {
15
+ R(0x1100,0x11ff),
16
+ S(0x20a9),
17
+ R(0x2329,0x232a),
18
+ R(0x2630,0x2637),
19
+ R(0x268a,0x268f),
20
+ R(0x2e80,0x2e99),
21
+ R(0x2e9b,0x2ef3),
22
+ R(0x2f00,0x2fd5),
23
+ R(0x2ff0,0x303e),
24
+ R(0x3041,0x3096),
25
+ R(0x3099,0x30ff),
26
+ R(0x3105,0x312f),
27
+ R(0x3131,0x318e),
28
+ R(0x3190,0x31e5),
29
+ R(0x31ef,0x321e),
30
+ R(0x3220,0x3247),
31
+ R(0x3250,0xa48c),
32
+ R(0xa490,0xa4c6),
33
+ R(0xa960,0xa97c),
34
+ R(0xac00,0xd7a3),
35
+ R(0xd7b0,0xd7c6),
36
+ R(0xd7cb,0xd7fb),
37
+ R(0xf900,0xfaff),
38
+ R(0xfe10,0xfe19),
39
+ R(0xfe30,0xfe52),
40
+ R(0xfe54,0xfe66),
41
+ R(0xfe68,0xfe6b),
42
+ R(0xff01,0xffbe),
43
+ R(0xffc2,0xffc7),
44
+ R(0xffca,0xffcf),
45
+ R(0xffd2,0xffd7),
46
+ R(0xffda,0xffdc),
47
+ R(0xffe0,0xffe6),
48
+ R(0xffe8,0xffee),
49
+ R(0x16fe0,0x16fe4),
50
+ R(0x16ff0,0x16ff6),
51
+ R(0x17000,0x18cd5),
52
+ R(0x18cff,0x18d1e),
53
+ R(0x18d80,0x18df2),
54
+ R(0x1aff0,0x1aff3),
55
+ R(0x1aff5,0x1affb),
56
+ R(0x1affd,0x1affe),
57
+ R(0x1b000,0x1b122),
58
+ S(0x1b132),
59
+ R(0x1b150,0x1b152),
60
+ S(0x1b155),
61
+ R(0x1b164,0x1b167),
62
+ R(0x1b170,0x1b2fb),
63
+ R(0x1d300,0x1d356),
64
+ R(0x1d360,0x1d376),
65
+ S(0x1f200),
66
+ S(0x1f202),
67
+ R(0x1f210,0x1f219),
68
+ R(0x1f21b,0x1f22e),
69
+ R(0x1f230,0x1f231),
70
+ S(0x1f237),
71
+ S(0x1f23b),
72
+ R(0x1f240,0x1f248),
73
+ R(0x1f260,0x1f265),
74
+ R(0x20000,0x2fffd),
75
+ R(0x30000,0x3fffd),
76
+ };
77
+ #undef R
78
+ #undef S
79
+ static const unsigned MD_CJK_AMBIGUOUS_PUNCT[][2] = {
80
+ { 0x2018, 0xfe01 },
81
+ { 0x2019, 0xfe01 },
82
+ { 0x201c, 0xfe01 },
83
+ { 0x201d, 0xfe01 },
84
+ };
@@ -678,40 +678,27 @@ struct MD_UNICODE_FOLD_INFO_tag {
678
678
  return (md_unicode_bsearch__(codepoint, PUNCT_MAP, SIZEOF_ARRAY(PUNCT_MAP)) >= 0);
679
679
  }
680
680
 
681
- /* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS):
682
- * CJK letters (Hangul, kana, Han ideographs, fullwidth forms). Used to
683
- * relax the emphasis flanking rules around CJK text. */
681
+ /* MARKTIP LOCAL PATCH: CJK-friendly uses Unicode width, script and emoji data. */
682
+ #include "cjk.h"
683
+
684
684
  static int
685
685
  md_is_cjk_char__(unsigned codepoint)
686
686
  {
687
- #define R(cp_min, cp_max) ((cp_min) | 0x40000000), ((cp_max) | 0x80000000)
688
- #define S(cp) (cp)
689
- static const unsigned CJK_MAP[] = {
690
- R(0x1100,0x11ff), /* Hangul jamo */
691
- S(0x3005), S(0x3007), S(0x303b),
692
- R(0x3040,0x30fa), /* hiragana, katakana (except middle dot) */
693
- R(0x30fc,0x30ff),
694
- R(0x3105,0x312f), /* bopomofo */
695
- R(0x3130,0x318f), /* Hangul compatibility jamo */
696
- R(0x31f0,0x31ff), /* katakana phonetic extensions */
697
- R(0x3400,0x4dbf), /* CJK ideographs extension A */
698
- R(0x4e00,0x9fff), /* CJK unified ideographs */
699
- R(0xa960,0xa97f), /* Hangul jamo extended-A */
700
- R(0xac00,0xd7a3), /* Hangul syllables */
701
- R(0xd7b0,0xd7ff), /* Hangul jamo extended-B */
702
- R(0xf900,0xfaff), /* CJK compatibility ideographs */
703
- R(0xff21,0xff3a), /* fullwidth Latin capitals */
704
- R(0xff41,0xff5a), /* fullwidth Latin small letters */
705
- R(0xff66,0xff9f), /* halfwidth katakana */
706
- R(0x20000,0x3ffff) /* CJK ideographs extensions B..H */
707
- };
708
- #undef R
709
- #undef S
710
-
711
687
  if(codepoint < 0x1100)
712
688
  return FALSE;
713
689
 
714
- return (md_unicode_bsearch__(codepoint, CJK_MAP, SIZEOF_ARRAY(CJK_MAP)) >= 0);
690
+ return (md_unicode_bsearch__(codepoint, MD_CJK_MAP, SIZEOF_ARRAY(MD_CJK_MAP)) >= 0);
691
+ }
692
+
693
+ static int
694
+ md_is_cjk_ambiguous_punct__(unsigned codepoint, unsigned selector)
695
+ {
696
+ unsigned i;
697
+ for(i = 0; i < SIZEOF_ARRAY(MD_CJK_AMBIGUOUS_PUNCT); i++) {
698
+ if(MD_CJK_AMBIGUOUS_PUNCT[i][0] == codepoint && MD_CJK_AMBIGUOUS_PUNCT[i][1] == selector)
699
+ return TRUE;
700
+ }
701
+ return FALSE;
715
702
  }
716
703
 
717
704
  static void
@@ -907,10 +894,6 @@ struct MD_UNICODE_FOLD_INFO_tag {
907
894
  #define ISUNICODEPUNCT(off) md_is_unicode_punct__(md_decode_utf16le__(STR(off), ctx->size - (off), NULL))
908
895
  #define ISUNICODEPUNCTBEFORE(off) md_is_unicode_punct__(md_decode_utf16le_before__(ctx, off))
909
896
 
910
- /* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS) */
911
- #define ISCJK(off) md_is_cjk_char__(md_decode_utf16le__(STR(off), ctx->size - (off), NULL))
912
- #define ISCJKBEFORE(off) md_is_cjk_char__(md_decode_utf16le_before__(ctx, off))
913
-
914
897
  static inline int
915
898
  md_decode_unicode(const CHAR* str, OFF off, SZ str_size, SZ* p_char_size)
916
899
  {
@@ -992,10 +975,6 @@ struct MD_UNICODE_FOLD_INFO_tag {
992
975
  #define ISUNICODEPUNCT(off) md_is_unicode_punct__(md_decode_utf8__(STR(off), ctx->size - (off), NULL))
993
976
  #define ISUNICODEPUNCTBEFORE(off) md_is_unicode_punct__(md_decode_utf8_before__(ctx, off))
994
977
 
995
- /* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS) */
996
- #define ISCJK(off) md_is_cjk_char__(md_decode_utf8__(STR(off), ctx->size - (off), NULL))
997
- #define ISCJKBEFORE(off) md_is_cjk_char__(md_decode_utf8_before__(ctx, off))
998
-
999
978
  static inline unsigned
1000
979
  md_decode_unicode(const CHAR* str, OFF off, SZ str_size, SZ* p_char_size)
1001
980
  {
@@ -1009,10 +988,6 @@ struct MD_UNICODE_FOLD_INFO_tag {
1009
988
  #define ISUNICODEPUNCT(off) ISPUNCT(off)
1010
989
  #define ISUNICODEPUNCTBEFORE(off) ISPUNCT((off)-1)
1011
990
 
1012
- /* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS): no CJK in ASCII mode. */
1013
- #define ISCJK(off) FALSE
1014
- #define ISCJKBEFORE(off) FALSE
1015
-
1016
991
  static inline void
1017
992
  md_get_unicode_fold_info(unsigned codepoint, MD_UNICODE_FOLD_INFO* info)
1018
993
  {
@@ -3039,6 +3014,101 @@ md_is_autolink(MD_CTX* ctx, OFF beg, OFF max_end, OFF* p_end, int* p_missing_mai
3039
3014
  return FALSE;
3040
3015
  }
3041
3016
 
3017
+ /* MARKTIP LOCAL PATCH: CJK-friendly flanking also classifies the preceding sequence. */
3018
+ #define MD_ATTN_SPACE 0x01
3019
+ #define MD_ATTN_PUNCT 0x02
3020
+ #define MD_ATTN_CJK 0x04
3021
+ #define MD_ATTN_IVS 0x08
3022
+ #define MD_ATTN_VS 0x10
3023
+
3024
+ static unsigned
3025
+ md_decode_attention_before(MD_CTX* ctx, OFF off, OFF* p_start)
3026
+ {
3027
+ OFF start = off - 1;
3028
+ SZ size;
3029
+ #if defined MD4C_USE_UTF16
3030
+ if(start > 0 && IS_UTF16_SURROGATE_LO(CH(start)) && IS_UTF16_SURROGATE_HI(CH(start-1)))
3031
+ start--;
3032
+ #elif defined MD4C_USE_UTF8
3033
+ while(start > 0 && off - start < 4 && IS_UTF8_TAIL(CH(start)))
3034
+ start--;
3035
+ #endif
3036
+ *p_start = start;
3037
+ return md_decode_unicode(STR(0), start, ctx->size, &size);
3038
+ }
3039
+
3040
+ static unsigned
3041
+ md_attention_category(unsigned codepoint, int cjk_friendly)
3042
+ {
3043
+ unsigned category = 0;
3044
+ if(ISUNICODEWHITESPACE_(codepoint))
3045
+ return MD_ATTN_SPACE;
3046
+ #if defined MD4C_USE_UTF16 || defined MD4C_USE_UTF8
3047
+ if(md_is_unicode_punct__(codepoint))
3048
+ category |= MD_ATTN_PUNCT;
3049
+ if(cjk_friendly) {
3050
+ if(0xfe00 <= codepoint && codepoint <= 0xfe0e)
3051
+ return MD_ATTN_VS;
3052
+ if(0xe0100 <= codepoint && codepoint <= 0xe01ef)
3053
+ return MD_ATTN_IVS;
3054
+ if(md_is_cjk_char__(codepoint))
3055
+ category |= MD_ATTN_CJK;
3056
+ }
3057
+ #else
3058
+ MD_UNUSED(cjk_friendly);
3059
+ if(ISPUNCT_(codepoint))
3060
+ category |= MD_ATTN_PUNCT;
3061
+ #endif
3062
+ return category;
3063
+ }
3064
+
3065
+ static unsigned
3066
+ md_attention_flags(MD_CTX* ctx, OFF line_beg, OFF line_end, OFF beg, OFF end, CHAR ch)
3067
+ {
3068
+ int cjk_friendly = (ctx->parser.flags & MD_FLAG_CJKFRIENDLYEMPHASIS) != 0;
3069
+ unsigned before = MD_ATTN_SPACE;
3070
+ unsigned after = MD_ATTN_SPACE;
3071
+ unsigned flags = 0;
3072
+ OFF previous = beg;
3073
+ unsigned previous_codepoint = 0;
3074
+ int left_flanking, right_flanking;
3075
+ SZ size;
3076
+
3077
+ if(beg > line_beg) {
3078
+ previous_codepoint = md_decode_attention_before(ctx, beg, &previous);
3079
+ before = md_attention_category(previous_codepoint, cjk_friendly);
3080
+ }
3081
+ if(end < line_end)
3082
+ after = md_attention_category(md_decode_unicode(STR(0), end, ctx->size, &size), cjk_friendly);
3083
+
3084
+ #if defined MD4C_USE_UTF16 || defined MD4C_USE_UTF8
3085
+ if(before == MD_ATTN_VS && previous > line_beg) {
3086
+ OFF base_start;
3087
+ unsigned base = md_decode_attention_before(ctx, previous, &base_start);
3088
+ unsigned category = md_attention_category(base, cjk_friendly);
3089
+ if(!(category & MD_ATTN_SPACE)) {
3090
+ before = category & ~MD_ATTN_IVS;
3091
+ if(md_is_cjk_ambiguous_punct__(base, previous_codepoint))
3092
+ before = MD_ATTN_CJK | MD_ATTN_PUNCT;
3093
+ }
3094
+ }
3095
+ #endif
3096
+
3097
+ left_flanking = !(after & MD_ATTN_SPACE) &&
3098
+ (!(after & MD_ATTN_PUNCT) || (before & (MD_ATTN_SPACE | MD_ATTN_PUNCT)) ||
3099
+ (after & MD_ATTN_CJK) || (before & (MD_ATTN_CJK | MD_ATTN_IVS)));
3100
+ right_flanking = !(before & MD_ATTN_SPACE) &&
3101
+ (!(before & MD_ATTN_PUNCT) || (after & (MD_ATTN_SPACE | MD_ATTN_PUNCT)) ||
3102
+ (before & MD_ATTN_CJK) || (after & MD_ATTN_CJK));
3103
+
3104
+ /* Underscores retain CommonMark's additional intraword restrictions. */
3105
+ if(left_flanking && (ch != _T('_') || !right_flanking || (before & MD_ATTN_PUNCT)))
3106
+ flags |= MD_MARK_POTENTIAL_OPENER;
3107
+ if(right_flanking && (ch != _T('_') || !left_flanking || (after & MD_ATTN_PUNCT)))
3108
+ flags |= MD_MARK_POTENTIAL_CLOSER;
3109
+ return flags;
3110
+ }
3111
+
3042
3112
  static int
3043
3113
  md_collect_marks(MD_CTX* ctx, const MD_LINE* lines, MD_SIZE n_lines, int table_mode)
3044
3114
  {
@@ -3090,51 +3160,14 @@ md_collect_marks(MD_CTX* ctx, const MD_LINE* lines, MD_SIZE n_lines, int table_m
3090
3160
  /* A potential (string) emphasis start/end. */
3091
3161
  if(ch == _T('*') || ch == _T('_')) {
3092
3162
  OFF tmp = off+1;
3093
- int left_level; /* What precedes: 0 = whitespace; 1 = punctuation; 2 = other char. */
3094
- int right_level; /* What follows: 0 = whitespace; 1 = punctuation; 2 = other char. */
3095
- int left_is_cjk = FALSE;
3096
- int right_is_cjk = FALSE;
3163
+ unsigned flags;
3097
3164
 
3098
3165
  while(tmp < line->end && CH(tmp) == ch)
3099
3166
  tmp++;
3100
3167
 
3101
- if(off == line->beg || ISUNICODEWHITESPACEBEFORE(off))
3102
- left_level = 0;
3103
- else if(ISUNICODEPUNCTBEFORE(off))
3104
- left_level = 1;
3105
- else
3106
- left_level = 2;
3107
-
3108
- if(tmp == line->end || ISUNICODEWHITESPACE(tmp))
3109
- right_level = 0;
3110
- else if(ISUNICODEPUNCT(tmp))
3111
- right_level = 1;
3112
- else
3113
- right_level = 2;
3114
-
3115
- /* MARKTIP LOCAL PATCH: a CJK character next to the delimiter
3116
- * does not block it from being an opener/closer on the other
3117
- * side (cf. markdown-it-cjk-friendly). */
3118
- if(ctx->parser.flags & MD_FLAG_CJKFRIENDLYEMPHASIS) {
3119
- if(off > line->beg)
3120
- left_is_cjk = ISCJKBEFORE(off);
3121
- if(tmp < line->end)
3122
- right_is_cjk = ISCJK(tmp);
3123
- }
3124
-
3125
- /* Intra-word underscore doesn't have special meaning. */
3126
- if(ch == _T('_') && left_level == 2 && right_level == 2) {
3127
- left_level = 0;
3128
- right_level = 0;
3129
- }
3130
-
3131
- if(left_level != 0 || right_level != 0) {
3132
- unsigned flags = 0;
3133
-
3134
- if(left_level > 0 && (left_level >= right_level || right_is_cjk))
3135
- flags |= MD_MARK_POTENTIAL_CLOSER;
3136
- if(right_level > 0 && (right_level >= left_level || left_is_cjk))
3137
- flags |= MD_MARK_POTENTIAL_OPENER;
3168
+ /* MARKTIP LOCAL PATCH: '*' and '_' share the CJK-friendly classifier. */
3169
+ flags = md_attention_flags(ctx, line->beg, line->end, off, tmp, ch);
3170
+ if(flags != 0) {
3138
3171
  if(flags == (MD_MARK_POTENTIAL_OPENER | MD_MARK_POTENTIAL_CLOSER))
3139
3172
  flags |= MD_MARK_EMPH_OC;
3140
3173
 
@@ -3357,40 +3390,8 @@ md_collect_marks(MD_CTX* ctx, const MD_LINE* lines, MD_SIZE n_lines, int table_m
3357
3390
  unsigned flags = 0;
3358
3391
 
3359
3392
  if(ch == _T('~')) {
3360
- /* MARKTIP LOCAL PATCH: strikethrough uses the same
3361
- * graded flanking rules as '*' (cf. cmark-gfm, which
3362
- * treats '~' like '*'), so intra-word "~~" opens and
3363
- * closes (e.g. "a~~b~~c"). The CJK-friendly relaxation
3364
- * mirrors the emphasis patch above. */
3365
- int left_level, right_level;
3366
- int left_is_cjk = FALSE;
3367
- int right_is_cjk = FALSE;
3368
-
3369
- if(off == line->beg || ISUNICODEWHITESPACEBEFORE(off))
3370
- left_level = 0;
3371
- else if(ISUNICODEPUNCTBEFORE(off))
3372
- left_level = 1;
3373
- else
3374
- left_level = 2;
3375
-
3376
- if(tmp == line->end || ISUNICODEWHITESPACE(tmp))
3377
- right_level = 0;
3378
- else if(ISUNICODEPUNCT(tmp))
3379
- right_level = 1;
3380
- else
3381
- right_level = 2;
3382
-
3383
- if(ctx->parser.flags & MD_FLAG_CJKFRIENDLYEMPHASIS) {
3384
- if(off > line->beg)
3385
- left_is_cjk = ISCJKBEFORE(off);
3386
- if(tmp < line->end)
3387
- right_is_cjk = ISCJK(tmp);
3388
- }
3389
-
3390
- if(left_level > 0 && (left_level >= right_level || right_is_cjk))
3391
- flags |= MD_MARK_POTENTIAL_CLOSER;
3392
- if(right_level > 0 && (right_level >= left_level || left_is_cjk))
3393
- flags |= MD_MARK_POTENTIAL_OPENER;
3393
+ /* MARKTIP LOCAL PATCH: '~' uses '*' flanking, including intraword runs. */
3394
+ flags = md_attention_flags(ctx, line->beg, line->end, off, tmp, ch);
3394
3395
  } else {
3395
3396
  /* '$' keeps the original strict flanking behavior. */
3396
3397
  flags = MD_MARK_POTENTIAL_OPENER | MD_MARK_POTENTIAL_CLOSER;
@@ -21,12 +21,15 @@ An upstream bump therefore has to re-check that `const ENTITY* entity_lookup(con
21
21
 
22
22
  ## Local patches (deviations from upstream)
23
23
 
24
- - `MD_FLAG_CJKFRIENDLYEMPHASIS` (md4c.h, md4c.c): opt-in flag that relaxes the
25
- emphasis (`*`, `_`) and strikethrough (`~`) flanking rules so that a CJK
26
- character next to a delimiter does not prevent it from opening/closing
27
- (cf. markdown-it-cjk-friendly). Adds `md_is_cjk_char__()` and the
28
- `ISCJK`/`ISCJKBEFORE` macros. All patched hunks are marked with
29
- `MARKTIP LOCAL PATCH` comments. Behavior is unchanged unless the flag is set.
24
+ - `MD_FLAG_CJKFRIENDLYEMPHASIS` (md4c.h, md4c.c, cjk.h): opt-in flag implementing the
25
+ [CommonMark CJK-friendly Amendments Specification](https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md)
26
+ for emphasis (`*`, `_`) and strikethrough (`~`). CJK characters include East
27
+ Asian Width `W`, `F`, or `H` except default emoji presentation, plus Hangul.
28
+ Flanking distinguishes CJK punctuation and preceding variation-selector
29
+ sequences, including standardized fullwidth quotation marks. Underscores
30
+ retain CommonMark's additional intraword conditions, so `_《제목》_을` stays
31
+ literal. `md_attention_flags()` shares flanking between emphasis and
32
+ strikethrough. Patched hunks use `MARKTIP LOCAL PATCH` comments.
30
33
  - Intra-word strikethrough (md4c.c, `~` branch in `md_collect_marks`):
31
34
  `~`/`~~` delimiter runs use the same graded flanking rules as `*` instead of
32
35
  upstream's strict require-whitespace/punctuation rule, matching cmark-gfm
@@ -36,3 +39,18 @@ An upstream bump therefore has to re-check that `const ENTITY* entity_lookup(con
36
39
  from upstream md4c 0.5.3 (upstream rejects intra-word tildes; the GFM
37
40
  reference implementations accept them). Marked with a `MARKTIP LOCAL PATCH`
38
41
  comment.
42
+
43
+ ## CJK classifier data
44
+
45
+ `src/cjk.h` is generated from Unicode 17.0.0 `EastAsianWidth.txt`, `Scripts.txt`,
46
+ `emoji/emoji-data.txt`, and `StandardizedVariants.txt`. Source URLs and SHA-256
47
+ digests are recorded in the header. The data license is `Unicode-LICENSE.txt`.
48
+ Ordinary whitespace and punctuation classification use MD4C's Unicode maps.
49
+
50
+ To regenerate, download the four files from
51
+ `https://www.unicode.org/Public/17.0.0/ucd/` into one directory (put
52
+ `emoji-data.txt` at its root), then run from the repository root:
53
+
54
+ ```bash
55
+ python scripts/build_cjk_map.py /path/to/unicode-17.0.0
56
+ ```
@@ -0,0 +1,39 @@
1
+ UNICODE LICENSE V3
2
+
3
+ COPYRIGHT AND PERMISSION NOTICE
4
+
5
+ Copyright © 1991-2026 Unicode, Inc.
6
+
7
+ NOTICE TO USER: Carefully read the following legal agreement. BY
8
+ DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
9
+ SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
10
+ TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
11
+ DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
12
+
13
+ Permission is hereby granted, free of charge, to any person obtaining a
14
+ copy of data files and any associated documentation (the "Data Files") or
15
+ software and any associated documentation (the "Software") to deal in the
16
+ Data Files or Software without restriction, including without limitation
17
+ the rights to use, copy, modify, merge, publish, distribute, and/or sell
18
+ copies of the Data Files or Software, and to permit persons to whom the
19
+ Data Files or Software are furnished to do so, provided that either (a)
20
+ this copyright and permission notice appear with all copies of the Data
21
+ Files or Software, or (b) this copyright and permission notice appear in
22
+ associated Documentation.
23
+
24
+ THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
25
+ KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
26
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
27
+ THIRD PARTY RIGHTS.
28
+
29
+ IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
30
+ BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
31
+ OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
32
+ WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
33
+ ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
34
+ FILES OR SOFTWARE.
35
+
36
+ Except as contained in this notice, the name of a copyright holder shall
37
+ not be used in advertising or otherwise to promote the sale, use or other
38
+ dealings in these Data Files or Software without prior written
39
+ authorization of the copyright holder.
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes