marktip 0.7.2__tar.gz → 0.7.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {marktip-0.7.2 → marktip-0.7.3}/CMakeLists.txt +1 -1
- marktip-0.7.2/README.md → marktip-0.7.3/PKG-INFO +37 -2
- marktip-0.7.2/PKG-INFO → marktip-0.7.3/README.md +8 -30
- {marktip-0.7.2 → marktip-0.7.3}/pyproject.toml +5 -3
- marktip-0.7.3/scripts/build_cjk_map.py +100 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/marktip.cpp +2 -2
- marktip-0.7.3/tests/test_cjk_friendly.py +94 -0
- marktip-0.7.3/third_party/md4c/src/cjk.h +84 -0
- {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/md4c.c +116 -115
- {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/upstream_metadata/README.md +24 -6
- marktip-0.7.3/third_party/md4c/upstream_metadata/Unicode-LICENSE.txt +39 -0
- {marktip-0.7.2 → marktip-0.7.3}/.github/workflows/build-distributions.yml +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/.gitignore +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/LICENSE +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/python/marktip/__init__.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/scripts/benchmark.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/ast.cpp +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/ast.h +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/errors.h +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/parser.cpp +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/parser.h +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/serializer.cpp +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/src/serializer.h +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_content_model.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_document.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_errors.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_node_type_policy.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_nul_normalization.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_parse.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_roundtrip.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_serialize.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_strict.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_uri_policy.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/tests/test_validate.py +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/LICENSE.md +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/entity.c +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/entity.h +0 -0
- {marktip-0.7.2 → marktip-0.7.3}/third_party/md4c/src/md4c.h +0 -0
|
@@ -1,3 +1,32 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: marktip
|
|
3
|
+
Version: 0.7.3
|
|
4
|
+
Summary: Fast C++/MD4C Markdown parser and canonical Markdown serializer for Tiptap-style JSON
|
|
5
|
+
Keywords: markdown,tiptap,md4c,parser
|
|
6
|
+
Author: Saneaven (Minjae Kyung)
|
|
7
|
+
License-Expression: MIT AND Unicode-3.0
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
License-File: third_party/md4c/LICENSE.md
|
|
10
|
+
License-File: third_party/md4c/upstream_metadata/Unicode-LICENSE.txt
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Operating System :: MacOS
|
|
14
|
+
Classifier: Operating System :: Microsoft :: Windows
|
|
15
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
16
|
+
Classifier: Programming Language :: C++
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
24
|
+
Classifier: Topic :: Text Processing :: Markup :: Markdown
|
|
25
|
+
Requires-Python: >=3.9
|
|
26
|
+
Provides-Extra: test
|
|
27
|
+
Requires-Dist: pytest>=8; extra == "test"
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
1
30
|
# marktip
|
|
2
31
|
|
|
3
32
|
Fast C++/MD4C Markdown conversion for Tiptap-style JSON.
|
|
@@ -24,14 +53,20 @@ doc = tm.from_dict(ast)
|
|
|
24
53
|
```
|
|
25
54
|
|
|
26
55
|
`from_markdown` follows GFM/CommonMark by default.
|
|
27
|
-
Pass `cjk_friendly=True` to
|
|
56
|
+
Pass `cjk_friendly=True` to follow the [CommonMark CJK-friendly Amendments Specification](https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md) for emphasis (`*`, `_`) and strikethrough (`~`). This opt-in extension recognizes CJK punctuation as well as letters, so `**《제목》**a`, `a**《제목》**`, and `**「約束」**a` retain their bold marks:
|
|
28
57
|
|
|
29
58
|
```python
|
|
30
59
|
doc = tm.from_markdown("**볼드**은 강조", cjk_friendly=True)
|
|
31
60
|
```
|
|
32
61
|
|
|
33
62
|
Strikethrough follows the GFM reference implementations (cmark-gfm/micromark): intra-word `~~` strikes (`a~~b~~c`), including next to CJK letters (`~~삭제~~은`).
|
|
34
|
-
`cjk_friendly` still matters for
|
|
63
|
+
`cjk_friendly` still matters for punctuation-adjacent cases such as `~~"인용"~~라고`.
|
|
64
|
+
|
|
65
|
+
The CJK classifier uses Unicode 17.0.0 East Asian Width `W`, `F`, or `H` (excluding default emoji presentation characters), plus the Hangul script. This includes fullwidth digits such as `1`. Before a delimiter, CJK characters followed by a non-emoji general-use variation selector (`U+FE00`–`U+FE0E`) are classified as one sequence; ideographic variation selectors (`U+E0100`–`U+E01EF`) and standardized fullwidth quotation-mark sequences are also recognized. `U+FE0F` is an emoji presentation selector and is excluded from that sequence rule.
|
|
66
|
+
|
|
67
|
+
Underscores retain CommonMark's additional restriction on emphasis inside words. The CJK-friendly specification changes delimiter flanking without removing that protection for names such as `snake_case`: `_《제목》_을` and `__《제목》__을` stay literal, even with `cjk_friendly=True`. Use `*` or `**` when emphasis must adjoin a word.
|
|
68
|
+
|
|
69
|
+
> **Changed in 0.7.3** — CJK-friendly character classification, variation-selector sequences, flanking, and underscore conditions follow the specification. The same flanking rules apply to `~~`. An editor document containing bold `《제목》` followed by `a` now survives serialization and reparsing with `cjk_friendly=True`.
|
|
35
70
|
|
|
36
71
|
Pass `html=False` to parse raw HTML as literal text instead of `htmlBlock`/`htmlInline` nodes.
|
|
37
72
|
Content is preserved (the serializer escapes it), and `<br>` inside table cells still maps to `hardBreak` so marktip's own table output round-trips:
|
|
@@ -1,31 +1,3 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: marktip
|
|
3
|
-
Version: 0.7.2
|
|
4
|
-
Summary: Fast C++/MD4C Markdown parser and canonical Markdown serializer for Tiptap-style JSON
|
|
5
|
-
Keywords: markdown,tiptap,md4c,parser
|
|
6
|
-
Author: Saneaven (Minjae Kyung)
|
|
7
|
-
License-Expression: MIT
|
|
8
|
-
License-File: LICENSE
|
|
9
|
-
License-File: third_party/md4c/LICENSE.md
|
|
10
|
-
Classifier: Development Status :: 3 - Alpha
|
|
11
|
-
Classifier: Intended Audience :: Developers
|
|
12
|
-
Classifier: Operating System :: MacOS
|
|
13
|
-
Classifier: Operating System :: Microsoft :: Windows
|
|
14
|
-
Classifier: Operating System :: POSIX :: Linux
|
|
15
|
-
Classifier: Programming Language :: C++
|
|
16
|
-
Classifier: Programming Language :: Python :: 3
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.9
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
22
|
-
Classifier: Programming Language :: Python :: 3.14
|
|
23
|
-
Classifier: Topic :: Text Processing :: Markup :: Markdown
|
|
24
|
-
Requires-Python: >=3.9
|
|
25
|
-
Provides-Extra: test
|
|
26
|
-
Requires-Dist: pytest>=8; extra == "test"
|
|
27
|
-
Description-Content-Type: text/markdown
|
|
28
|
-
|
|
29
1
|
# marktip
|
|
30
2
|
|
|
31
3
|
Fast C++/MD4C Markdown conversion for Tiptap-style JSON.
|
|
@@ -52,14 +24,20 @@ doc = tm.from_dict(ast)
|
|
|
52
24
|
```
|
|
53
25
|
|
|
54
26
|
`from_markdown` follows GFM/CommonMark by default.
|
|
55
|
-
Pass `cjk_friendly=True` to
|
|
27
|
+
Pass `cjk_friendly=True` to follow the [CommonMark CJK-friendly Amendments Specification](https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md) for emphasis (`*`, `_`) and strikethrough (`~`). This opt-in extension recognizes CJK punctuation as well as letters, so `**《제목》**a`, `a**《제목》**`, and `**「約束」**a` retain their bold marks:
|
|
56
28
|
|
|
57
29
|
```python
|
|
58
30
|
doc = tm.from_markdown("**볼드**은 강조", cjk_friendly=True)
|
|
59
31
|
```
|
|
60
32
|
|
|
61
33
|
Strikethrough follows the GFM reference implementations (cmark-gfm/micromark): intra-word `~~` strikes (`a~~b~~c`), including next to CJK letters (`~~삭제~~은`).
|
|
62
|
-
`cjk_friendly` still matters for
|
|
34
|
+
`cjk_friendly` still matters for punctuation-adjacent cases such as `~~"인용"~~라고`.
|
|
35
|
+
|
|
36
|
+
The CJK classifier uses Unicode 17.0.0 East Asian Width `W`, `F`, or `H` (excluding default emoji presentation characters), plus the Hangul script. This includes fullwidth digits such as `1`. Before a delimiter, CJK characters followed by a non-emoji general-use variation selector (`U+FE00`–`U+FE0E`) are classified as one sequence; ideographic variation selectors (`U+E0100`–`U+E01EF`) and standardized fullwidth quotation-mark sequences are also recognized. `U+FE0F` is an emoji presentation selector and is excluded from that sequence rule.
|
|
37
|
+
|
|
38
|
+
Underscores retain CommonMark's additional restriction on emphasis inside words. The CJK-friendly specification changes delimiter flanking without removing that protection for names such as `snake_case`: `_《제목》_을` and `__《제목》__을` stay literal, even with `cjk_friendly=True`. Use `*` or `**` when emphasis must adjoin a word.
|
|
39
|
+
|
|
40
|
+
> **Changed in 0.7.3** — CJK-friendly character classification, variation-selector sequences, flanking, and underscore conditions follow the specification. The same flanking rules apply to `~~`. An editor document containing bold `《제목》` followed by `a` now survives serialization and reparsing with `cjk_friendly=True`.
|
|
63
41
|
|
|
64
42
|
Pass `html=False` to parse raw HTML as literal text instead of `htmlBlock`/`htmlInline` nodes.
|
|
65
43
|
Content is preserved (the serializer escapes it), and `<br>` inside table cells still maps to `hardBreak` so marktip's own table output round-trips:
|
|
@@ -4,12 +4,12 @@ build-backend = "scikit_build_core.build"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "marktip"
|
|
7
|
-
version = "0.7.
|
|
7
|
+
version = "0.7.3"
|
|
8
8
|
description = "Fast C++/MD4C Markdown parser and canonical Markdown serializer for Tiptap-style JSON"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.9"
|
|
11
|
-
license = "MIT"
|
|
12
|
-
license-files = ["LICENSE", "third_party/md4c/LICENSE.md"]
|
|
11
|
+
license = "MIT AND Unicode-3.0"
|
|
12
|
+
license-files = ["LICENSE", "third_party/md4c/LICENSE.md", "third_party/md4c/upstream_metadata/Unicode-LICENSE.txt"]
|
|
13
13
|
authors = [{ name = "Saneaven (Minjae Kyung)" }]
|
|
14
14
|
keywords = ["markdown", "tiptap", "md4c", "parser"]
|
|
15
15
|
classifiers = [
|
|
@@ -38,9 +38,11 @@ wheel.packages = ["python/marktip"]
|
|
|
38
38
|
sdist.include = [
|
|
39
39
|
"third_party/md4c/src/md4c.c",
|
|
40
40
|
"third_party/md4c/src/md4c.h",
|
|
41
|
+
"third_party/md4c/src/cjk.h",
|
|
41
42
|
"third_party/md4c/src/entity.c",
|
|
42
43
|
"third_party/md4c/src/entity.h",
|
|
43
44
|
"third_party/md4c/LICENSE.md",
|
|
45
|
+
"third_party/md4c/upstream_metadata/Unicode-LICENSE.txt",
|
|
44
46
|
]
|
|
45
47
|
|
|
46
48
|
[tool.pytest.ini_options]
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
"""Generate the CJK-friendly classifier from Unicode 17.0.0 data files."""
|
|
2
|
+
|
|
3
|
+
import argparse
|
|
4
|
+
import hashlib
|
|
5
|
+
from pathlib import Path
|
|
6
|
+
import re
|
|
7
|
+
|
|
8
|
+
|
|
9
|
+
VERSION = "17.0.0"
|
|
10
|
+
SOURCES = {
|
|
11
|
+
"EastAsianWidth.txt": "EastAsianWidth.txt",
|
|
12
|
+
"Scripts.txt": "Scripts.txt",
|
|
13
|
+
"emoji-data.txt": "emoji/emoji-data.txt",
|
|
14
|
+
"StandardizedVariants.txt": "StandardizedVariants.txt",
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
def records(path):
|
|
19
|
+
for line in path.read_text(encoding="utf-8").splitlines():
|
|
20
|
+
body = line.split("#", 1)[0].strip()
|
|
21
|
+
if body:
|
|
22
|
+
yield [field.strip() for field in body.split(";")]
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
def codepoints(value):
|
|
26
|
+
parts = value.split("..")
|
|
27
|
+
return range(int(parts[0], 16), int(parts[-1], 16) + 1)
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
def intervals(values):
|
|
31
|
+
start = previous = None
|
|
32
|
+
for value in sorted(values):
|
|
33
|
+
if previous is not None and value != previous + 1:
|
|
34
|
+
yield start, previous
|
|
35
|
+
start = None
|
|
36
|
+
if start is None:
|
|
37
|
+
start = value
|
|
38
|
+
previous = value
|
|
39
|
+
if previous is not None:
|
|
40
|
+
yield start, previous
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def generate(directory):
|
|
44
|
+
for filename in SOURCES:
|
|
45
|
+
header = (directory / filename).read_text(encoding="utf-8")[:1000]
|
|
46
|
+
if not re.search(r"17\.0(?:\.0)?\b", header):
|
|
47
|
+
raise ValueError(f"{filename} must contain Unicode {VERSION} data")
|
|
48
|
+
widths = {}
|
|
49
|
+
for bounds, width in records(directory / "EastAsianWidth.txt"):
|
|
50
|
+
widths.update(dict.fromkeys(codepoints(bounds), width))
|
|
51
|
+
hangul = set()
|
|
52
|
+
for bounds, script in records(directory / "Scripts.txt"):
|
|
53
|
+
if script == "Hangul":
|
|
54
|
+
hangul.update(codepoints(bounds))
|
|
55
|
+
emoji = set()
|
|
56
|
+
for bounds, property_name in records(directory / "emoji-data.txt"):
|
|
57
|
+
if property_name == "Emoji_Presentation":
|
|
58
|
+
emoji.update(codepoints(bounds))
|
|
59
|
+
cjk = {cp for cp, width in widths.items() if width in {"W", "F", "H"}} - emoji
|
|
60
|
+
cjk.update(hangul)
|
|
61
|
+
ambiguous = []
|
|
62
|
+
for sequence, description, *_ in records(directory / "StandardizedVariants.txt"):
|
|
63
|
+
base, selector = (int(cp, 16) for cp in sequence.split())
|
|
64
|
+
if (
|
|
65
|
+
widths.get(base) == "A"
|
|
66
|
+
and re.search(r"\bfullwidth form\b", description)
|
|
67
|
+
and "non-fullwidth" not in description
|
|
68
|
+
):
|
|
69
|
+
ambiguous.append((base, selector))
|
|
70
|
+
|
|
71
|
+
lines = [
|
|
72
|
+
"/* Generated by scripts/build_cjk_map.py; Unicode " + VERSION + ".",
|
|
73
|
+
" * Data license: upstream_metadata/Unicode-LICENSE.txt.",
|
|
74
|
+
]
|
|
75
|
+
for filename, source in SOURCES.items():
|
|
76
|
+
digest = hashlib.sha256((directory / filename).read_bytes()).hexdigest()
|
|
77
|
+
lines.append(" * https://www.unicode.org/Public/" + VERSION + "/ucd/" + source)
|
|
78
|
+
lines.append(" * SHA-256: " + digest)
|
|
79
|
+
lines.extend([
|
|
80
|
+
" */",
|
|
81
|
+
"#define R(cp_min, cp_max) ((cp_min) | 0x40000000), ((cp_max) | 0x80000000)",
|
|
82
|
+
"#define S(cp) (cp)",
|
|
83
|
+
"static const unsigned MD_CJK_MAP[] = {",
|
|
84
|
+
])
|
|
85
|
+
for start, end in intervals(cjk):
|
|
86
|
+
value = f"S(0x{start:04x})" if start == end else f"R(0x{start:04x},0x{end:04x})"
|
|
87
|
+
lines.append(" " + value + ",")
|
|
88
|
+
lines.extend(["};", "#undef R", "#undef S", "static const unsigned MD_CJK_AMBIGUOUS_PUNCT[][2] = {"])
|
|
89
|
+
for base, selector in sorted(ambiguous):
|
|
90
|
+
lines.append(f" {{ 0x{base:04x}, 0x{selector:04x} }},")
|
|
91
|
+
lines.extend(["};", ""])
|
|
92
|
+
return "\n".join(lines)
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
if __name__ == "__main__":
|
|
96
|
+
parser = argparse.ArgumentParser(description=__doc__)
|
|
97
|
+
parser.add_argument("ucd_directory", type=Path)
|
|
98
|
+
parser.add_argument("--output", type=Path, default=Path("third_party/md4c/src/cjk.h"))
|
|
99
|
+
args = parser.parse_args()
|
|
100
|
+
args.output.write_text(generate(args.ucd_directory), encoding="utf-8")
|
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
namespace py = pybind11;
|
|
12
12
|
|
|
13
13
|
#ifndef MARKTIP_VERSION
|
|
14
|
-
#define MARKTIP_VERSION "0.7.
|
|
14
|
+
#define MARKTIP_VERSION "0.7.3"
|
|
15
15
|
#endif
|
|
16
16
|
|
|
17
17
|
namespace {
|
|
@@ -125,7 +125,7 @@ PYBIND11_MODULE(_core, module) {
|
|
|
125
125
|
// pybind11 takes the doc as a bare const char* and only copies it while def() runs.
|
|
126
126
|
const std::string from_markdown_doc =
|
|
127
127
|
"Parse Markdown into a Document."
|
|
128
|
-
" Set cjk_friendly=True to
|
|
128
|
+
" Set cjk_friendly=True to use CJK-friendly emphasis and strikethrough rules"
|
|
129
129
|
" (non-standard extension; the default follows GFM/CommonMark exactly)."
|
|
130
130
|
" Set html=False to parse raw HTML as literal text instead of htmlBlock/htmlInline nodes;"
|
|
131
131
|
" <br> inside table cells still maps to hardBreak." +
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
import pytest
|
|
2
|
+
|
|
3
|
+
import marktip as tm
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
def marked_texts(markdown, mark_type, *, cjk_friendly=True):
|
|
7
|
+
content = tm.from_markdown(markdown, cjk_friendly=cjk_friendly).to_dict()["content"][0]["content"]
|
|
8
|
+
return [node["text"] for node in content if {"type": mark_type} in node.get("marks", [])]
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
12
|
+
@pytest.mark.parametrize("prefix,suffix", [("", "a"), ("a", ""), ("a", "a"), ("", "을")])
|
|
13
|
+
@pytest.mark.parametrize("value", ["《제목》", "「約束」"])
|
|
14
|
+
def test_cjk_punctuation_adjoins_latin_or_hangul(delimiter, mark_type, prefix, suffix, value):
|
|
15
|
+
markdown = prefix + delimiter + value + delimiter + suffix
|
|
16
|
+
assert marked_texts(markdown, mark_type) == [value]
|
|
17
|
+
if prefix == "a" or suffix == "a":
|
|
18
|
+
assert marked_texts(markdown, mark_type, cjk_friendly=False) == []
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
@pytest.mark.parametrize("value", ["1", "₩", "カ", "ᅡ", "〰", "〽", "㊗", "㊙", "🈂", "🈷", "𠀀"])
|
|
22
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
23
|
+
def test_unicode_width_and_hangul_script_define_cjk(value, delimiter, mark_type):
|
|
24
|
+
assert marked_texts("a" + delimiter + value + delimiter + "a", mark_type) == [value]
|
|
25
|
+
assert marked_texts(value + delimiter + '"제목"' + delimiter + value, mark_type) == ['"제목"']
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
@pytest.mark.parametrize("value", ["😀", "⌚", "🈁", "©", "☀"])
|
|
29
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
30
|
+
def test_default_emoji_and_non_cjk_symbols_do_not_relax_flanking(value, delimiter, mark_type):
|
|
31
|
+
assert marked_texts("a" + delimiter + value + delimiter + "a", mark_type) == []
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
@pytest.mark.parametrize("selector", ["\ufe00", "\ufe01", "\ufe0e", "\U000e0100", "\U000e01ef"])
|
|
35
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
36
|
+
def test_cjk_variation_sequence_before_opener(selector, delimiter, mark_type):
|
|
37
|
+
markdown = "漢" + selector + delimiter + '"제목"' + delimiter + "1"
|
|
38
|
+
assert marked_texts(markdown, mark_type) == ['"제목"']
|
|
39
|
+
assert marked_texts(markdown, mark_type, cjk_friendly=False) == []
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
43
|
+
def test_emoji_selector_does_not_form_a_cjk_sequence(delimiter, mark_type):
|
|
44
|
+
assert marked_texts("漢\ufe0f" + delimiter + '"제목"' + delimiter + "1", mark_type) == []
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
48
|
+
def test_non_cjk_punctuation_sequence_still_blocks_closing(delimiter, mark_type):
|
|
49
|
+
assert marked_texts("a" + delimiter + "x.\ufe00" + delimiter + "b", mark_type) == []
|
|
50
|
+
assert marked_texts("a" + delimiter + "x.\ufe0f" + delimiter + "b", mark_type) == ["x.\ufe0f"]
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("**", "bold"), ("*", "italic"), ("~~", "strike")])
|
|
54
|
+
@pytest.mark.parametrize("value", ["x.\ufe00", "x”\ufe00"])
|
|
55
|
+
def test_punctuation_sequence_can_close_before_cjk(delimiter, mark_type, value):
|
|
56
|
+
assert marked_texts("a" + delimiter + value + delimiter + "字", mark_type) == [value]
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
@pytest.mark.parametrize("quote", ["‘", "’", "“", "”"])
|
|
60
|
+
def test_fullwidth_ambiguous_punctuation_sequence(quote):
|
|
61
|
+
value = "제목" + quote + "\ufe01"
|
|
62
|
+
assert marked_texts("a**" + value + "**a", "bold") == [value]
|
|
63
|
+
assert marked_texts("a**제목" + quote + "\ufe00**a", "bold") == []
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
@pytest.mark.parametrize("delimiter", ["_", "__"])
|
|
67
|
+
@pytest.mark.parametrize("prefix,suffix", [("", "을"), ("a", ""), ("a", "a")])
|
|
68
|
+
def test_underscores_retain_commonmark_intraword_restrictions(delimiter, prefix, suffix):
|
|
69
|
+
markdown = prefix + delimiter + "《제목》" + delimiter + suffix
|
|
70
|
+
assert tm.from_markdown(markdown, cjk_friendly=True).to_dict()["content"][0]["content"] == [
|
|
71
|
+
{"type": "text", "text": markdown}
|
|
72
|
+
]
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
@pytest.mark.parametrize("delimiter,mark_type", [("_", "italic"), ("__", "bold")])
|
|
76
|
+
def test_underscores_accept_preceding_punctuation_sequence(delimiter, mark_type):
|
|
77
|
+
assert marked_texts("a.\ufe00" + delimiter + "제목" + delimiter + "!", mark_type) == ["제목"]
|
|
78
|
+
assert marked_texts("(" + delimiter + "《제목》" + delimiter + ")", mark_type) == ["《제목》"]
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
@pytest.mark.parametrize("mark_type", ["bold", "italic", "strike"])
|
|
82
|
+
def test_editor_mark_on_cjk_punctuation_roundtrips_before_latin(mark_type):
|
|
83
|
+
ast = {
|
|
84
|
+
"type": "doc",
|
|
85
|
+
"content": [{
|
|
86
|
+
"type": "paragraph",
|
|
87
|
+
"content": [
|
|
88
|
+
{"type": "text", "text": "《제목》", "marks": [{"type": mark_type}]},
|
|
89
|
+
{"type": "text", "text": "a"},
|
|
90
|
+
],
|
|
91
|
+
}],
|
|
92
|
+
}
|
|
93
|
+
markdown = tm.from_dict(ast).to_markdown()
|
|
94
|
+
assert tm.from_markdown(markdown, cjk_friendly=True).to_dict() == ast
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
/* Generated by scripts/build_cjk_map.py; Unicode 17.0.0.
|
|
2
|
+
* Data license: upstream_metadata/Unicode-LICENSE.txt.
|
|
3
|
+
* https://www.unicode.org/Public/17.0.0/ucd/EastAsianWidth.txt
|
|
4
|
+
* SHA-256: ea7ce50f3444a050333448dffef1cadd9325af55cbb764b4a2280faf52170a33
|
|
5
|
+
* https://www.unicode.org/Public/17.0.0/ucd/Scripts.txt
|
|
6
|
+
* SHA-256: 9f5e50d3abaee7d6ce09480f325c706f485ae3240912527e651954d2d6b035bf
|
|
7
|
+
* https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt
|
|
8
|
+
* SHA-256: 2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b
|
|
9
|
+
* https://www.unicode.org/Public/17.0.0/ucd/StandardizedVariants.txt
|
|
10
|
+
* SHA-256: f55100b2fb11d3d75a37b8c1ab752192dbd1c4b12328c5ec6b38e3807c0ca597
|
|
11
|
+
*/
|
|
12
|
+
#define R(cp_min, cp_max) ((cp_min) | 0x40000000), ((cp_max) | 0x80000000)
|
|
13
|
+
#define S(cp) (cp)
|
|
14
|
+
static const unsigned MD_CJK_MAP[] = {
|
|
15
|
+
R(0x1100,0x11ff),
|
|
16
|
+
S(0x20a9),
|
|
17
|
+
R(0x2329,0x232a),
|
|
18
|
+
R(0x2630,0x2637),
|
|
19
|
+
R(0x268a,0x268f),
|
|
20
|
+
R(0x2e80,0x2e99),
|
|
21
|
+
R(0x2e9b,0x2ef3),
|
|
22
|
+
R(0x2f00,0x2fd5),
|
|
23
|
+
R(0x2ff0,0x303e),
|
|
24
|
+
R(0x3041,0x3096),
|
|
25
|
+
R(0x3099,0x30ff),
|
|
26
|
+
R(0x3105,0x312f),
|
|
27
|
+
R(0x3131,0x318e),
|
|
28
|
+
R(0x3190,0x31e5),
|
|
29
|
+
R(0x31ef,0x321e),
|
|
30
|
+
R(0x3220,0x3247),
|
|
31
|
+
R(0x3250,0xa48c),
|
|
32
|
+
R(0xa490,0xa4c6),
|
|
33
|
+
R(0xa960,0xa97c),
|
|
34
|
+
R(0xac00,0xd7a3),
|
|
35
|
+
R(0xd7b0,0xd7c6),
|
|
36
|
+
R(0xd7cb,0xd7fb),
|
|
37
|
+
R(0xf900,0xfaff),
|
|
38
|
+
R(0xfe10,0xfe19),
|
|
39
|
+
R(0xfe30,0xfe52),
|
|
40
|
+
R(0xfe54,0xfe66),
|
|
41
|
+
R(0xfe68,0xfe6b),
|
|
42
|
+
R(0xff01,0xffbe),
|
|
43
|
+
R(0xffc2,0xffc7),
|
|
44
|
+
R(0xffca,0xffcf),
|
|
45
|
+
R(0xffd2,0xffd7),
|
|
46
|
+
R(0xffda,0xffdc),
|
|
47
|
+
R(0xffe0,0xffe6),
|
|
48
|
+
R(0xffe8,0xffee),
|
|
49
|
+
R(0x16fe0,0x16fe4),
|
|
50
|
+
R(0x16ff0,0x16ff6),
|
|
51
|
+
R(0x17000,0x18cd5),
|
|
52
|
+
R(0x18cff,0x18d1e),
|
|
53
|
+
R(0x18d80,0x18df2),
|
|
54
|
+
R(0x1aff0,0x1aff3),
|
|
55
|
+
R(0x1aff5,0x1affb),
|
|
56
|
+
R(0x1affd,0x1affe),
|
|
57
|
+
R(0x1b000,0x1b122),
|
|
58
|
+
S(0x1b132),
|
|
59
|
+
R(0x1b150,0x1b152),
|
|
60
|
+
S(0x1b155),
|
|
61
|
+
R(0x1b164,0x1b167),
|
|
62
|
+
R(0x1b170,0x1b2fb),
|
|
63
|
+
R(0x1d300,0x1d356),
|
|
64
|
+
R(0x1d360,0x1d376),
|
|
65
|
+
S(0x1f200),
|
|
66
|
+
S(0x1f202),
|
|
67
|
+
R(0x1f210,0x1f219),
|
|
68
|
+
R(0x1f21b,0x1f22e),
|
|
69
|
+
R(0x1f230,0x1f231),
|
|
70
|
+
S(0x1f237),
|
|
71
|
+
S(0x1f23b),
|
|
72
|
+
R(0x1f240,0x1f248),
|
|
73
|
+
R(0x1f260,0x1f265),
|
|
74
|
+
R(0x20000,0x2fffd),
|
|
75
|
+
R(0x30000,0x3fffd),
|
|
76
|
+
};
|
|
77
|
+
#undef R
|
|
78
|
+
#undef S
|
|
79
|
+
static const unsigned MD_CJK_AMBIGUOUS_PUNCT[][2] = {
|
|
80
|
+
{ 0x2018, 0xfe01 },
|
|
81
|
+
{ 0x2019, 0xfe01 },
|
|
82
|
+
{ 0x201c, 0xfe01 },
|
|
83
|
+
{ 0x201d, 0xfe01 },
|
|
84
|
+
};
|
|
@@ -678,40 +678,27 @@ struct MD_UNICODE_FOLD_INFO_tag {
|
|
|
678
678
|
return (md_unicode_bsearch__(codepoint, PUNCT_MAP, SIZEOF_ARRAY(PUNCT_MAP)) >= 0);
|
|
679
679
|
}
|
|
680
680
|
|
|
681
|
-
/* MARKTIP LOCAL PATCH
|
|
682
|
-
|
|
683
|
-
|
|
681
|
+
/* MARKTIP LOCAL PATCH: CJK-friendly uses Unicode width, script and emoji data. */
|
|
682
|
+
#include "cjk.h"
|
|
683
|
+
|
|
684
684
|
static int
|
|
685
685
|
md_is_cjk_char__(unsigned codepoint)
|
|
686
686
|
{
|
|
687
|
-
#define R(cp_min, cp_max) ((cp_min) | 0x40000000), ((cp_max) | 0x80000000)
|
|
688
|
-
#define S(cp) (cp)
|
|
689
|
-
static const unsigned CJK_MAP[] = {
|
|
690
|
-
R(0x1100,0x11ff), /* Hangul jamo */
|
|
691
|
-
S(0x3005), S(0x3007), S(0x303b),
|
|
692
|
-
R(0x3040,0x30fa), /* hiragana, katakana (except middle dot) */
|
|
693
|
-
R(0x30fc,0x30ff),
|
|
694
|
-
R(0x3105,0x312f), /* bopomofo */
|
|
695
|
-
R(0x3130,0x318f), /* Hangul compatibility jamo */
|
|
696
|
-
R(0x31f0,0x31ff), /* katakana phonetic extensions */
|
|
697
|
-
R(0x3400,0x4dbf), /* CJK ideographs extension A */
|
|
698
|
-
R(0x4e00,0x9fff), /* CJK unified ideographs */
|
|
699
|
-
R(0xa960,0xa97f), /* Hangul jamo extended-A */
|
|
700
|
-
R(0xac00,0xd7a3), /* Hangul syllables */
|
|
701
|
-
R(0xd7b0,0xd7ff), /* Hangul jamo extended-B */
|
|
702
|
-
R(0xf900,0xfaff), /* CJK compatibility ideographs */
|
|
703
|
-
R(0xff21,0xff3a), /* fullwidth Latin capitals */
|
|
704
|
-
R(0xff41,0xff5a), /* fullwidth Latin small letters */
|
|
705
|
-
R(0xff66,0xff9f), /* halfwidth katakana */
|
|
706
|
-
R(0x20000,0x3ffff) /* CJK ideographs extensions B..H */
|
|
707
|
-
};
|
|
708
|
-
#undef R
|
|
709
|
-
#undef S
|
|
710
|
-
|
|
711
687
|
if(codepoint < 0x1100)
|
|
712
688
|
return FALSE;
|
|
713
689
|
|
|
714
|
-
return (md_unicode_bsearch__(codepoint,
|
|
690
|
+
return (md_unicode_bsearch__(codepoint, MD_CJK_MAP, SIZEOF_ARRAY(MD_CJK_MAP)) >= 0);
|
|
691
|
+
}
|
|
692
|
+
|
|
693
|
+
static int
|
|
694
|
+
md_is_cjk_ambiguous_punct__(unsigned codepoint, unsigned selector)
|
|
695
|
+
{
|
|
696
|
+
unsigned i;
|
|
697
|
+
for(i = 0; i < SIZEOF_ARRAY(MD_CJK_AMBIGUOUS_PUNCT); i++) {
|
|
698
|
+
if(MD_CJK_AMBIGUOUS_PUNCT[i][0] == codepoint && MD_CJK_AMBIGUOUS_PUNCT[i][1] == selector)
|
|
699
|
+
return TRUE;
|
|
700
|
+
}
|
|
701
|
+
return FALSE;
|
|
715
702
|
}
|
|
716
703
|
|
|
717
704
|
static void
|
|
@@ -907,10 +894,6 @@ struct MD_UNICODE_FOLD_INFO_tag {
|
|
|
907
894
|
#define ISUNICODEPUNCT(off) md_is_unicode_punct__(md_decode_utf16le__(STR(off), ctx->size - (off), NULL))
|
|
908
895
|
#define ISUNICODEPUNCTBEFORE(off) md_is_unicode_punct__(md_decode_utf16le_before__(ctx, off))
|
|
909
896
|
|
|
910
|
-
/* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS) */
|
|
911
|
-
#define ISCJK(off) md_is_cjk_char__(md_decode_utf16le__(STR(off), ctx->size - (off), NULL))
|
|
912
|
-
#define ISCJKBEFORE(off) md_is_cjk_char__(md_decode_utf16le_before__(ctx, off))
|
|
913
|
-
|
|
914
897
|
static inline int
|
|
915
898
|
md_decode_unicode(const CHAR* str, OFF off, SZ str_size, SZ* p_char_size)
|
|
916
899
|
{
|
|
@@ -992,10 +975,6 @@ struct MD_UNICODE_FOLD_INFO_tag {
|
|
|
992
975
|
#define ISUNICODEPUNCT(off) md_is_unicode_punct__(md_decode_utf8__(STR(off), ctx->size - (off), NULL))
|
|
993
976
|
#define ISUNICODEPUNCTBEFORE(off) md_is_unicode_punct__(md_decode_utf8_before__(ctx, off))
|
|
994
977
|
|
|
995
|
-
/* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS) */
|
|
996
|
-
#define ISCJK(off) md_is_cjk_char__(md_decode_utf8__(STR(off), ctx->size - (off), NULL))
|
|
997
|
-
#define ISCJKBEFORE(off) md_is_cjk_char__(md_decode_utf8_before__(ctx, off))
|
|
998
|
-
|
|
999
978
|
static inline unsigned
|
|
1000
979
|
md_decode_unicode(const CHAR* str, OFF off, SZ str_size, SZ* p_char_size)
|
|
1001
980
|
{
|
|
@@ -1009,10 +988,6 @@ struct MD_UNICODE_FOLD_INFO_tag {
|
|
|
1009
988
|
#define ISUNICODEPUNCT(off) ISPUNCT(off)
|
|
1010
989
|
#define ISUNICODEPUNCTBEFORE(off) ISPUNCT((off)-1)
|
|
1011
990
|
|
|
1012
|
-
/* MARKTIP LOCAL PATCH (MD_FLAG_CJKFRIENDLYEMPHASIS): no CJK in ASCII mode. */
|
|
1013
|
-
#define ISCJK(off) FALSE
|
|
1014
|
-
#define ISCJKBEFORE(off) FALSE
|
|
1015
|
-
|
|
1016
991
|
static inline void
|
|
1017
992
|
md_get_unicode_fold_info(unsigned codepoint, MD_UNICODE_FOLD_INFO* info)
|
|
1018
993
|
{
|
|
@@ -3039,6 +3014,101 @@ md_is_autolink(MD_CTX* ctx, OFF beg, OFF max_end, OFF* p_end, int* p_missing_mai
|
|
|
3039
3014
|
return FALSE;
|
|
3040
3015
|
}
|
|
3041
3016
|
|
|
3017
|
+
/* MARKTIP LOCAL PATCH: CJK-friendly flanking also classifies the preceding sequence. */
|
|
3018
|
+
#define MD_ATTN_SPACE 0x01
|
|
3019
|
+
#define MD_ATTN_PUNCT 0x02
|
|
3020
|
+
#define MD_ATTN_CJK 0x04
|
|
3021
|
+
#define MD_ATTN_IVS 0x08
|
|
3022
|
+
#define MD_ATTN_VS 0x10
|
|
3023
|
+
|
|
3024
|
+
static unsigned
|
|
3025
|
+
md_decode_attention_before(MD_CTX* ctx, OFF off, OFF* p_start)
|
|
3026
|
+
{
|
|
3027
|
+
OFF start = off - 1;
|
|
3028
|
+
SZ size;
|
|
3029
|
+
#if defined MD4C_USE_UTF16
|
|
3030
|
+
if(start > 0 && IS_UTF16_SURROGATE_LO(CH(start)) && IS_UTF16_SURROGATE_HI(CH(start-1)))
|
|
3031
|
+
start--;
|
|
3032
|
+
#elif defined MD4C_USE_UTF8
|
|
3033
|
+
while(start > 0 && off - start < 4 && IS_UTF8_TAIL(CH(start)))
|
|
3034
|
+
start--;
|
|
3035
|
+
#endif
|
|
3036
|
+
*p_start = start;
|
|
3037
|
+
return md_decode_unicode(STR(0), start, ctx->size, &size);
|
|
3038
|
+
}
|
|
3039
|
+
|
|
3040
|
+
static unsigned
|
|
3041
|
+
md_attention_category(unsigned codepoint, int cjk_friendly)
|
|
3042
|
+
{
|
|
3043
|
+
unsigned category = 0;
|
|
3044
|
+
if(ISUNICODEWHITESPACE_(codepoint))
|
|
3045
|
+
return MD_ATTN_SPACE;
|
|
3046
|
+
#if defined MD4C_USE_UTF16 || defined MD4C_USE_UTF8
|
|
3047
|
+
if(md_is_unicode_punct__(codepoint))
|
|
3048
|
+
category |= MD_ATTN_PUNCT;
|
|
3049
|
+
if(cjk_friendly) {
|
|
3050
|
+
if(0xfe00 <= codepoint && codepoint <= 0xfe0e)
|
|
3051
|
+
return MD_ATTN_VS;
|
|
3052
|
+
if(0xe0100 <= codepoint && codepoint <= 0xe01ef)
|
|
3053
|
+
return MD_ATTN_IVS;
|
|
3054
|
+
if(md_is_cjk_char__(codepoint))
|
|
3055
|
+
category |= MD_ATTN_CJK;
|
|
3056
|
+
}
|
|
3057
|
+
#else
|
|
3058
|
+
MD_UNUSED(cjk_friendly);
|
|
3059
|
+
if(ISPUNCT_(codepoint))
|
|
3060
|
+
category |= MD_ATTN_PUNCT;
|
|
3061
|
+
#endif
|
|
3062
|
+
return category;
|
|
3063
|
+
}
|
|
3064
|
+
|
|
3065
|
+
static unsigned
|
|
3066
|
+
md_attention_flags(MD_CTX* ctx, OFF line_beg, OFF line_end, OFF beg, OFF end, CHAR ch)
|
|
3067
|
+
{
|
|
3068
|
+
int cjk_friendly = (ctx->parser.flags & MD_FLAG_CJKFRIENDLYEMPHASIS) != 0;
|
|
3069
|
+
unsigned before = MD_ATTN_SPACE;
|
|
3070
|
+
unsigned after = MD_ATTN_SPACE;
|
|
3071
|
+
unsigned flags = 0;
|
|
3072
|
+
OFF previous = beg;
|
|
3073
|
+
unsigned previous_codepoint = 0;
|
|
3074
|
+
int left_flanking, right_flanking;
|
|
3075
|
+
SZ size;
|
|
3076
|
+
|
|
3077
|
+
if(beg > line_beg) {
|
|
3078
|
+
previous_codepoint = md_decode_attention_before(ctx, beg, &previous);
|
|
3079
|
+
before = md_attention_category(previous_codepoint, cjk_friendly);
|
|
3080
|
+
}
|
|
3081
|
+
if(end < line_end)
|
|
3082
|
+
after = md_attention_category(md_decode_unicode(STR(0), end, ctx->size, &size), cjk_friendly);
|
|
3083
|
+
|
|
3084
|
+
#if defined MD4C_USE_UTF16 || defined MD4C_USE_UTF8
|
|
3085
|
+
if(before == MD_ATTN_VS && previous > line_beg) {
|
|
3086
|
+
OFF base_start;
|
|
3087
|
+
unsigned base = md_decode_attention_before(ctx, previous, &base_start);
|
|
3088
|
+
unsigned category = md_attention_category(base, cjk_friendly);
|
|
3089
|
+
if(!(category & MD_ATTN_SPACE)) {
|
|
3090
|
+
before = category & ~MD_ATTN_IVS;
|
|
3091
|
+
if(md_is_cjk_ambiguous_punct__(base, previous_codepoint))
|
|
3092
|
+
before = MD_ATTN_CJK | MD_ATTN_PUNCT;
|
|
3093
|
+
}
|
|
3094
|
+
}
|
|
3095
|
+
#endif
|
|
3096
|
+
|
|
3097
|
+
left_flanking = !(after & MD_ATTN_SPACE) &&
|
|
3098
|
+
(!(after & MD_ATTN_PUNCT) || (before & (MD_ATTN_SPACE | MD_ATTN_PUNCT)) ||
|
|
3099
|
+
(after & MD_ATTN_CJK) || (before & (MD_ATTN_CJK | MD_ATTN_IVS)));
|
|
3100
|
+
right_flanking = !(before & MD_ATTN_SPACE) &&
|
|
3101
|
+
(!(before & MD_ATTN_PUNCT) || (after & (MD_ATTN_SPACE | MD_ATTN_PUNCT)) ||
|
|
3102
|
+
(before & MD_ATTN_CJK) || (after & MD_ATTN_CJK));
|
|
3103
|
+
|
|
3104
|
+
/* Underscores retain CommonMark's additional intraword restrictions. */
|
|
3105
|
+
if(left_flanking && (ch != _T('_') || !right_flanking || (before & MD_ATTN_PUNCT)))
|
|
3106
|
+
flags |= MD_MARK_POTENTIAL_OPENER;
|
|
3107
|
+
if(right_flanking && (ch != _T('_') || !left_flanking || (after & MD_ATTN_PUNCT)))
|
|
3108
|
+
flags |= MD_MARK_POTENTIAL_CLOSER;
|
|
3109
|
+
return flags;
|
|
3110
|
+
}
|
|
3111
|
+
|
|
3042
3112
|
static int
|
|
3043
3113
|
md_collect_marks(MD_CTX* ctx, const MD_LINE* lines, MD_SIZE n_lines, int table_mode)
|
|
3044
3114
|
{
|
|
@@ -3090,51 +3160,14 @@ md_collect_marks(MD_CTX* ctx, const MD_LINE* lines, MD_SIZE n_lines, int table_m
|
|
|
3090
3160
|
/* A potential (string) emphasis start/end. */
|
|
3091
3161
|
if(ch == _T('*') || ch == _T('_')) {
|
|
3092
3162
|
OFF tmp = off+1;
|
|
3093
|
-
|
|
3094
|
-
int right_level; /* What follows: 0 = whitespace; 1 = punctuation; 2 = other char. */
|
|
3095
|
-
int left_is_cjk = FALSE;
|
|
3096
|
-
int right_is_cjk = FALSE;
|
|
3163
|
+
unsigned flags;
|
|
3097
3164
|
|
|
3098
3165
|
while(tmp < line->end && CH(tmp) == ch)
|
|
3099
3166
|
tmp++;
|
|
3100
3167
|
|
|
3101
|
-
|
|
3102
|
-
|
|
3103
|
-
|
|
3104
|
-
left_level = 1;
|
|
3105
|
-
else
|
|
3106
|
-
left_level = 2;
|
|
3107
|
-
|
|
3108
|
-
if(tmp == line->end || ISUNICODEWHITESPACE(tmp))
|
|
3109
|
-
right_level = 0;
|
|
3110
|
-
else if(ISUNICODEPUNCT(tmp))
|
|
3111
|
-
right_level = 1;
|
|
3112
|
-
else
|
|
3113
|
-
right_level = 2;
|
|
3114
|
-
|
|
3115
|
-
/* MARKTIP LOCAL PATCH: a CJK character next to the delimiter
|
|
3116
|
-
* does not block it from being an opener/closer on the other
|
|
3117
|
-
* side (cf. markdown-it-cjk-friendly). */
|
|
3118
|
-
if(ctx->parser.flags & MD_FLAG_CJKFRIENDLYEMPHASIS) {
|
|
3119
|
-
if(off > line->beg)
|
|
3120
|
-
left_is_cjk = ISCJKBEFORE(off);
|
|
3121
|
-
if(tmp < line->end)
|
|
3122
|
-
right_is_cjk = ISCJK(tmp);
|
|
3123
|
-
}
|
|
3124
|
-
|
|
3125
|
-
/* Intra-word underscore doesn't have special meaning. */
|
|
3126
|
-
if(ch == _T('_') && left_level == 2 && right_level == 2) {
|
|
3127
|
-
left_level = 0;
|
|
3128
|
-
right_level = 0;
|
|
3129
|
-
}
|
|
3130
|
-
|
|
3131
|
-
if(left_level != 0 || right_level != 0) {
|
|
3132
|
-
unsigned flags = 0;
|
|
3133
|
-
|
|
3134
|
-
if(left_level > 0 && (left_level >= right_level || right_is_cjk))
|
|
3135
|
-
flags |= MD_MARK_POTENTIAL_CLOSER;
|
|
3136
|
-
if(right_level > 0 && (right_level >= left_level || left_is_cjk))
|
|
3137
|
-
flags |= MD_MARK_POTENTIAL_OPENER;
|
|
3168
|
+
/* MARKTIP LOCAL PATCH: '*' and '_' share the CJK-friendly classifier. */
|
|
3169
|
+
flags = md_attention_flags(ctx, line->beg, line->end, off, tmp, ch);
|
|
3170
|
+
if(flags != 0) {
|
|
3138
3171
|
if(flags == (MD_MARK_POTENTIAL_OPENER | MD_MARK_POTENTIAL_CLOSER))
|
|
3139
3172
|
flags |= MD_MARK_EMPH_OC;
|
|
3140
3173
|
|
|
@@ -3357,40 +3390,8 @@ md_collect_marks(MD_CTX* ctx, const MD_LINE* lines, MD_SIZE n_lines, int table_m
|
|
|
3357
3390
|
unsigned flags = 0;
|
|
3358
3391
|
|
|
3359
3392
|
if(ch == _T('~')) {
|
|
3360
|
-
/* MARKTIP LOCAL PATCH:
|
|
3361
|
-
|
|
3362
|
-
* treats '~' like '*'), so intra-word "~~" opens and
|
|
3363
|
-
* closes (e.g. "a~~b~~c"). The CJK-friendly relaxation
|
|
3364
|
-
* mirrors the emphasis patch above. */
|
|
3365
|
-
int left_level, right_level;
|
|
3366
|
-
int left_is_cjk = FALSE;
|
|
3367
|
-
int right_is_cjk = FALSE;
|
|
3368
|
-
|
|
3369
|
-
if(off == line->beg || ISUNICODEWHITESPACEBEFORE(off))
|
|
3370
|
-
left_level = 0;
|
|
3371
|
-
else if(ISUNICODEPUNCTBEFORE(off))
|
|
3372
|
-
left_level = 1;
|
|
3373
|
-
else
|
|
3374
|
-
left_level = 2;
|
|
3375
|
-
|
|
3376
|
-
if(tmp == line->end || ISUNICODEWHITESPACE(tmp))
|
|
3377
|
-
right_level = 0;
|
|
3378
|
-
else if(ISUNICODEPUNCT(tmp))
|
|
3379
|
-
right_level = 1;
|
|
3380
|
-
else
|
|
3381
|
-
right_level = 2;
|
|
3382
|
-
|
|
3383
|
-
if(ctx->parser.flags & MD_FLAG_CJKFRIENDLYEMPHASIS) {
|
|
3384
|
-
if(off > line->beg)
|
|
3385
|
-
left_is_cjk = ISCJKBEFORE(off);
|
|
3386
|
-
if(tmp < line->end)
|
|
3387
|
-
right_is_cjk = ISCJK(tmp);
|
|
3388
|
-
}
|
|
3389
|
-
|
|
3390
|
-
if(left_level > 0 && (left_level >= right_level || right_is_cjk))
|
|
3391
|
-
flags |= MD_MARK_POTENTIAL_CLOSER;
|
|
3392
|
-
if(right_level > 0 && (right_level >= left_level || left_is_cjk))
|
|
3393
|
-
flags |= MD_MARK_POTENTIAL_OPENER;
|
|
3393
|
+
/* MARKTIP LOCAL PATCH: '~' uses '*' flanking, including intraword runs. */
|
|
3394
|
+
flags = md_attention_flags(ctx, line->beg, line->end, off, tmp, ch);
|
|
3394
3395
|
} else {
|
|
3395
3396
|
/* '$' keeps the original strict flanking behavior. */
|
|
3396
3397
|
flags = MD_MARK_POTENTIAL_OPENER | MD_MARK_POTENTIAL_CLOSER;
|
|
@@ -21,12 +21,15 @@ An upstream bump therefore has to re-check that `const ENTITY* entity_lookup(con
|
|
|
21
21
|
|
|
22
22
|
## Local patches (deviations from upstream)
|
|
23
23
|
|
|
24
|
-
- `MD_FLAG_CJKFRIENDLYEMPHASIS` (md4c.h, md4c.c): opt-in flag
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
24
|
+
- `MD_FLAG_CJKFRIENDLYEMPHASIS` (md4c.h, md4c.c, cjk.h): opt-in flag implementing the
|
|
25
|
+
[CommonMark CJK-friendly Amendments Specification](https://github.com/tats-u/markdown-cjk-friendly/blob/main/specification.md)
|
|
26
|
+
for emphasis (`*`, `_`) and strikethrough (`~`). CJK characters include East
|
|
27
|
+
Asian Width `W`, `F`, or `H` except default emoji presentation, plus Hangul.
|
|
28
|
+
Flanking distinguishes CJK punctuation and preceding variation-selector
|
|
29
|
+
sequences, including standardized fullwidth quotation marks. Underscores
|
|
30
|
+
retain CommonMark's additional intraword conditions, so `_《제목》_을` stays
|
|
31
|
+
literal. `md_attention_flags()` shares flanking between emphasis and
|
|
32
|
+
strikethrough. Patched hunks use `MARKTIP LOCAL PATCH` comments.
|
|
30
33
|
- Intra-word strikethrough (md4c.c, `~` branch in `md_collect_marks`):
|
|
31
34
|
`~`/`~~` delimiter runs use the same graded flanking rules as `*` instead of
|
|
32
35
|
upstream's strict require-whitespace/punctuation rule, matching cmark-gfm
|
|
@@ -36,3 +39,18 @@ An upstream bump therefore has to re-check that `const ENTITY* entity_lookup(con
|
|
|
36
39
|
from upstream md4c 0.5.3 (upstream rejects intra-word tildes; the GFM
|
|
37
40
|
reference implementations accept them). Marked with a `MARKTIP LOCAL PATCH`
|
|
38
41
|
comment.
|
|
42
|
+
|
|
43
|
+
## CJK classifier data
|
|
44
|
+
|
|
45
|
+
`src/cjk.h` is generated from Unicode 17.0.0 `EastAsianWidth.txt`, `Scripts.txt`,
|
|
46
|
+
`emoji/emoji-data.txt`, and `StandardizedVariants.txt`. Source URLs and SHA-256
|
|
47
|
+
digests are recorded in the header. The data license is `Unicode-LICENSE.txt`.
|
|
48
|
+
Ordinary whitespace and punctuation classification use MD4C's Unicode maps.
|
|
49
|
+
|
|
50
|
+
To regenerate, download the four files from
|
|
51
|
+
`https://www.unicode.org/Public/17.0.0/ucd/` into one directory (put
|
|
52
|
+
`emoji-data.txt` at its root), then run from the repository root:
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
python scripts/build_cjk_map.py /path/to/unicode-17.0.0
|
|
56
|
+
```
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
UNICODE LICENSE V3
|
|
2
|
+
|
|
3
|
+
COPYRIGHT AND PERMISSION NOTICE
|
|
4
|
+
|
|
5
|
+
Copyright © 1991-2026 Unicode, Inc.
|
|
6
|
+
|
|
7
|
+
NOTICE TO USER: Carefully read the following legal agreement. BY
|
|
8
|
+
DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
|
|
9
|
+
SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
|
|
10
|
+
TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
|
|
11
|
+
DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
|
|
12
|
+
|
|
13
|
+
Permission is hereby granted, free of charge, to any person obtaining a
|
|
14
|
+
copy of data files and any associated documentation (the "Data Files") or
|
|
15
|
+
software and any associated documentation (the "Software") to deal in the
|
|
16
|
+
Data Files or Software without restriction, including without limitation
|
|
17
|
+
the rights to use, copy, modify, merge, publish, distribute, and/or sell
|
|
18
|
+
copies of the Data Files or Software, and to permit persons to whom the
|
|
19
|
+
Data Files or Software are furnished to do so, provided that either (a)
|
|
20
|
+
this copyright and permission notice appear with all copies of the Data
|
|
21
|
+
Files or Software, or (b) this copyright and permission notice appear in
|
|
22
|
+
associated Documentation.
|
|
23
|
+
|
|
24
|
+
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
|
|
25
|
+
KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
26
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
|
|
27
|
+
THIRD PARTY RIGHTS.
|
|
28
|
+
|
|
29
|
+
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
|
|
30
|
+
BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
|
|
31
|
+
OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
|
|
32
|
+
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
|
|
33
|
+
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
|
|
34
|
+
FILES OR SOFTWARE.
|
|
35
|
+
|
|
36
|
+
Except as contained in this notice, the name of a copyright holder shall
|
|
37
|
+
not be used in advertising or otherwise to promote the sale, use or other
|
|
38
|
+
dealings in these Data Files or Software without prior written
|
|
39
|
+
authorization of the copyright holder.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|