ltem-core 0.1.0__tar.gz

ltem_core-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+.pytest_cache/
+.venv/
+node_modules/
+target/
+dist/
+.DS_Store

ltem_core-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 EurekaAI Co., Ltd.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ltem_core-0.1.0/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.5
+Name: ltem-core
+Version: 0.1.0
+Summary: LTEM (LLM-friendly Table-Enhanced Markdown): Markdown tables with merged cells and multi-row headers
+Author: Sangkeun Jung
+License-Expression: MIT
+License-File: LICENSE
+Keywords: colspan,llm,ltem,markdown,rowspan,table
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.10
+Provides-Extra: markdown
+Requires-Dist: markdown-it-py>=3; extra == 'markdown'
+Description-Content-Type: text/markdown
+
+# ltem (Python)
+
+Reference Python implementation of [LTEM](../../../README.md): Markdown pipe tables with merged cells and multi-row headers. It follows the LTEM specification, version 1.0.
+
+Status: version 0.1.0, the first release, for LTEM 1.0. The API may change before version 1.0.0.
+
+## Install
+
+```
+pip install ltem-core              # no dependencies
+pip install "ltem-core[markdown]"  # also renders inline Markdown in cells (uses markdown-it-py)
+```
+
+## Use
+
+```python
+import ltem
+
+result = ltem.parse_table("""\
+| Region | Revenue | <<  |
+| ^^     | Q1      | Q2  |
+|--------|---------|-----|
+| Asia   | 100     | 120 |""")
+
+table = result.table            # the table model (see ast/ in the LTEM repository)
+table.to_dict()                 # JSON-ready dict
+ltem.render_html(table)         # <table> with colspan and rowspan
+ltem.write(table)               # canonical LTEM
+result.problems                 # validation problems, if any
+
+for item in ltem.from_html(html):   # HTML tables -> table models
+    print(ltem.write(item.table), item.warnings)
+
+text, sources = ltem.write_with_sources(table)   # canonical LTEM and where each cell is
+```
+
+### Where cells are
+
+`ParseResult.sources` and `write_with_sources` give, for each cell of the table and in the same order, a `CellSource(row, col, rowspan, colspan, line, start, end)`:
+
+- `line` counts lines from 0: lines of the parsed text for `parse_table` and `find_tables`, lines of the table for `write_with_sources`.
+- `start` and `end` index that line in Python string units (code points). They cover the cell's source without the spaces and tabs around it, including any span attribute and escapes.
+- For an empty cell, `start` equals `end`, at the end of its space between the pipes. A cell added because its row was short (rule V14) has `start == end ==` the length of its line.
+
+Without `markdown-it-py`, cell text is HTML-escaped and backslash escapes are applied, but emphasis, links and code are shown as written.
+
+`ltem.find_tables(markdown)` finds every LTEM table in a Markdown document. It is a light scan that skips code blocks. It is not a full Markdown parser.
+
+## Command line
+
+```
+ltem parse    [FILE]          # table models as JSON
+ltem render   [FILE]          # HTML of each table
+ltem validate [FILE] [--json] # validation problems; exit status 1 if any
+ltem fmt      [FILE]          # rewrite every table in canonical form
+ltem from-html [FILE] [--header-rows N] [--plain]  # convert HTML tables to LTEM
+```
+
+`from_html(html, header_rows=None, inline="markdown")` options:
+
+- `header_rows`: the number of header rows. By default it is the `<thead>` rows; without `<thead>`, the leading rows made only of `<th>` cells; and if there are none, the first row (with a warning). With `0`, the table has no header: it gets one header row of empty cells and every HTML row becomes a body row (spec Section 11).
+- `inline`: `"markdown"` keeps emphasis, code, links and images as Markdown; `"plain"` keeps text only. Either way, text is escaped so that it reads literally.
+
+`<th>` cells below the header rows become ordinary cells, since LTEM 1.0 has no row headers; the converter reports this once per table in `warnings`.
+
+`from_html` forms the grid as browsers do: cells take the next free position, `rowspan="0"` reaches the end of its row group, and spans stop at the edge of `<thead>`, `<tbody>` or `<tfoot>`. The header rows are the `<thead>` rows; without `<thead>`, the leading rows made only of `<th>` cells; and if there are none, the first row. Cell content becomes inline Markdown: `<br>` and block elements become line breaks, and `<strong>`, `<em>`, `<code>`, `<a>`, `<img>`, lists, `<sup>` and `<sub>` are kept. A table inside a cell is flattened to text with a warning, since LTEM 1.0 has no nested tables.
+
+## Development
+
+From `ltem-lib/python`:
+
+```
+uv sync --all-packages
+uv run --directory ltem pytest
+```
+
+The tests run the conformance cases in [`conformance/`](../../../conformance/).

ltem_core-0.1.0/README.md

@@ -0,0 +1,77 @@
+# ltem (Python)
+
+Reference Python implementation of [LTEM](../../../README.md): Markdown pipe tables with merged cells and multi-row headers. It follows the LTEM specification, version 1.0.
+
+Status: version 0.1.0, the first release, for LTEM 1.0. The API may change before version 1.0.0.
+
+## Install
+
+```
+pip install ltem-core              # no dependencies
+pip install "ltem-core[markdown]"  # also renders inline Markdown in cells (uses markdown-it-py)
+```
+
+## Use
+
+```python
+import ltem
+
+result = ltem.parse_table("""\
+| Region | Revenue | <<  |
+| ^^     | Q1      | Q2  |
+|--------|---------|-----|
+| Asia   | 100     | 120 |""")
+
+table = result.table            # the table model (see ast/ in the LTEM repository)
+table.to_dict()                 # JSON-ready dict
+ltem.render_html(table)         # <table> with colspan and rowspan
+ltem.write(table)               # canonical LTEM
+result.problems                 # validation problems, if any
+
+for item in ltem.from_html(html):   # HTML tables -> table models
+    print(ltem.write(item.table), item.warnings)
+
+text, sources = ltem.write_with_sources(table)   # canonical LTEM and where each cell is
+```
+
+### Where cells are
+
+`ParseResult.sources` and `write_with_sources` give, for each cell of the table and in the same order, a `CellSource(row, col, rowspan, colspan, line, start, end)`:
+
+- `line` counts lines from 0: lines of the parsed text for `parse_table` and `find_tables`, lines of the table for `write_with_sources`.
+- `start` and `end` index that line in Python string units (code points). They cover the cell's source without the spaces and tabs around it, including any span attribute and escapes.
+- For an empty cell, `start` equals `end`, at the end of its space between the pipes. A cell added because its row was short (rule V14) has `start == end ==` the length of its line.
+
+Without `markdown-it-py`, cell text is HTML-escaped and backslash escapes are applied, but emphasis, links and code are shown as written.
+
+`ltem.find_tables(markdown)` finds every LTEM table in a Markdown document. It is a light scan that skips code blocks. It is not a full Markdown parser.
+
+## Command line
+
+```
+ltem parse    [FILE]          # table models as JSON
+ltem render   [FILE]          # HTML of each table
+ltem validate [FILE] [--json] # validation problems; exit status 1 if any
+ltem fmt      [FILE]          # rewrite every table in canonical form
+ltem from-html [FILE] [--header-rows N] [--plain]  # convert HTML tables to LTEM
+```
+
+`from_html(html, header_rows=None, inline="markdown")` options:
+
+- `header_rows`: the number of header rows. By default it is the `<thead>` rows; without `<thead>`, the leading rows made only of `<th>` cells; and if there are none, the first row (with a warning). With `0`, the table has no header: it gets one header row of empty cells and every HTML row becomes a body row (spec Section 11).
+- `inline`: `"markdown"` keeps emphasis, code, links and images as Markdown; `"plain"` keeps text only. Either way, text is escaped so that it reads literally.
+
+`<th>` cells below the header rows become ordinary cells, since LTEM 1.0 has no row headers; the converter reports this once per table in `warnings`.
+
+`from_html` forms the grid as browsers do: cells take the next free position, `rowspan="0"` reaches the end of its row group, and spans stop at the edge of `<thead>`, `<tbody>` or `<tfoot>`. The header rows are the `<thead>` rows; without `<thead>`, the leading rows made only of `<th>` cells; and if there are none, the first row. Cell content becomes inline Markdown: `<br>` and block elements become line breaks, and `<strong>`, `<em>`, `<code>`, `<a>`, `<img>`, lists, `<sup>` and `<sub>` are kept. A table inside a cell is flattened to text with a warning, since LTEM 1.0 has no nested tables.
+
+## Development
+
+From `ltem-lib/python`:
+
+```
+uv sync --all-packages
+uv run --directory ltem pytest
+```
+
+The tests run the conformance cases in [`conformance/`](../../../conformance/).

ltem_core-0.1.0/pyproject.toml

@@ -0,0 +1,34 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+name = "ltem-core"
+version = "0.1.0"
+description = "LTEM (LLM-friendly Table-Enhanced Markdown): Markdown tables with merged cells and multi-row headers"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Sangkeun Jung" }]
+keywords = ["markdown", "table", "colspan", "rowspan", "llm", "ltem"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Topic :: Text Processing :: Markup :: Markdown",
+]
+dependencies = []
+
+[project.optional-dependencies]
+markdown = ["markdown-it-py>=3"]
+
+[project.scripts]
+ltem = "ltem.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/ltem"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

ltem_core-0.1.0/src/ltem/__init__.py

@@ -0,0 +1,51 @@
+"""LTEM (LLM-friendly Table-Enhanced Markdown).
+
+Reference Python implementation of the LTEM specification, version 1.0:
+Markdown pipe tables with merged cells and multi-row headers.
+
+    >>> import ltem
+    >>> result = ltem.parse_table('''
+    ... | Region | Revenue | <<  |
+    ... | ^^     | Q1      | Q2  |
+    ... |--------|---------|-----|
+    ... | Asia   | 100     | 120 |
+    ... '''.strip())
+    >>> [(c.text, c.rowspan, c.colspan) for c in result.table.cells][:2]
+    [('Region', 2, 1), ('Revenue', 1, 2)]
+"""
+
+from ._syntax import (ends_table_body, indentation, is_delimiter_row, is_upper_header_row, split_row,
+                      split_row_spans, starts_block)
+from .fromhtml import HtmlTable, from_html, html_to_ltem
+from .html import plain_inline, render_html
+from .model import Cell, CellSource, Problem, Table
+from .parser import ParseResult, find_tables, parse_table
+from .writer import write, write_with_sources
+
+__version__ = "0.1.0"
+SPEC_VERSION = "1.0"
+
+__all__ = [
+    "Cell",
+    "CellSource",
+    "HtmlTable",
+    "ParseResult",
+    "Problem",
+    "SPEC_VERSION",
+    "Table",
+    "find_tables",
+    "from_html",
+    "ends_table_body",
+    "html_to_ltem",
+    "indentation",
+    "is_delimiter_row",
+    "is_upper_header_row",
+    "parse_table",
+    "plain_inline",
+    "render_html",
+    "split_row",
+    "split_row_spans",
+    "starts_block",
+    "write",
+    "write_with_sources",
+]

ltem_core-0.1.0/src/ltem/__main__.py

@@ -0,0 +1,4 @@
+from .cli import main
+import sys
+
+sys.exit(main())

ltem_core-0.1.0/src/ltem/_grid.py

@@ -0,0 +1,151 @@
+"""Span resolution (spec Section 3): grid rows, compact rows, markers and recovery."""
+
+from __future__ import annotations
+
+import copy
+from dataclasses import dataclass
+
+from ._syntax import COLSPAN_MARKER, ROWSPAN_MARKER, split_attribute
+from .model import Problem
+
+
+@dataclass
+class _Origin:
+    row: int
+    col: int
+    rowspan: int
+    colspan: int
+    text: str
+    fixed_cols: bool  # colspan came from an attribute: markers may not extend it
+    fixed_rows: bool  # rowspan came from an attribute
+    source: int | None = None  # index of the cell in its source row; None if added by padding
+
+
+class GridBuilder:
+    """Places the cells of a table row by row, as spec Section 3 describes."""
+
+    def __init__(self, cols: int, header_rows: int, rows: int) -> None:
+        self.cols, self.header_rows, self.rows = cols, header_rows, rows
+        self.owner: list[list[_Origin | None]] = [[None] * cols for _ in range(rows)]
+        self.origins: list[_Origin] = []
+        self.problems: list[Problem] = []
+
+    # -- helpers ---------------------------------------------------------------
+
+    def _report(self, rule: str, row: int, col: int | None, message: str) -> None:
+        self.problems.append(Problem(rule, row, col, message))
+
+    def _free(self, r0: int, r1: int, c0: int, c1: int) -> bool:
+        return all(self.owner[r][c] is None
+                   for r in range(r0, min(r1, self.rows)) for c in range(c0, min(c1, self.cols)))
+
+    def _cover(self, origin: _Origin, r0: int, r1: int, c0: int, c1: int) -> None:
+        for r in range(r0, min(r1, self.rows)):
+            for c in range(c0, min(c1, self.cols)):
+                self.owner[r][c] = origin
+
+    # -- placing (spec 3.3, 3.4) -------------------------------------------------
+
+    def _extend_left(self, r: int, c: int) -> bool:
+        left = self.owner[r][c - 1] if c > 0 else None
+        if left is None or left.row != r or left.col + left.colspan != c or left.fixed_cols:
+            self._report("V13", r, c, "'<<' has no cell to extend")
+            return False
+        if not self._free(r + 1, left.row + left.rowspan, c, c + 1):
+            self._report("V4", r, c, "'<<' would make two cells overlap")
+            return False
+        left.colspan += 1
+        self._cover(left, left.row, left.row + left.rowspan, c, c + 1)
+        return True
+
+    def _extend_up(self, r: int, c: int) -> bool:
+        up = self.owner[r - 1][c] if r > 0 else None
+        if up is None or up.col != c or up.row + up.rowspan != r or up.fixed_rows:
+            self._report("V13", r, c, "'^^' has no cell to extend")
+            return False
+        if up.row < self.header_rows <= r:
+            self._report("V7", r, c, "'^^' would extend a header cell into the body")
+            return False
+        if not self._free(r, r + 1, c + 1, up.col + up.colspan):
+            self._report("V4", r, c, "'^^' would make two cells overlap")
+            return False
+        up.rowspan += 1
+        self._cover(up, r, r + 1, up.col, up.col + up.colspan)
+        return True
+
+    def _place(self, r: int, c: int, text: str, source: int | None = None) -> None:
+        """Place a cell at the position (r, c), which is not covered."""
+        if text == COLSPAN_MARKER and self._extend_left(r, c):
+            return
+        if text == ROWSPAN_MARKER and self._extend_up(r, c):
+            return
+        colspan = rowspan = 1
+        fixed_cols = fixed_rows = False
+        if text not in (COLSPAN_MARKER, ROWSPAN_MARKER):  # an unusable marker stays literal text
+            text, attr = split_attribute(text)
+            if attr:
+                for message in attr.problems:
+                    self._report("V1", r, c, message)
+                colspan, fixed_cols = attr.colspan or 1, attr.colspan is not None
+                rowspan, fixed_rows = attr.rowspan or 1, attr.rowspan is not None
+        if c + colspan > self.cols:
+            self._report("V2", r, c, f"colspan {colspan} reaches past the last column")
+            colspan = self.cols - c
+        if r + rowspan > self.rows:
+            self._report("V3", r, c, f"rowspan {rowspan} reaches past the last row")
+            rowspan = self.rows - r
+        if r < self.header_rows < r + rowspan:
+            self._report("V7", r, c, "a header cell spans into the body")
+            rowspan = self.header_rows - r
+        # V4: cut to the largest rectangle that overlaps nothing, columns first, then rows.
+        k = 1
+        while k < colspan and self.owner[r][c + k] is None:
+            k += 1
+        j = 1
+        while j < rowspan and self._free(r + j, r + j + 1, c, c + k):
+            j += 1
+        if (k, j) != (colspan, rowspan):
+            self._report("V4", r, c, f"span cut from {colspan}x{rowspan} to {k}x{j} to avoid an overlap")
+            colspan, rowspan = k, j
+        origin = _Origin(r, c, rowspan, colspan, text, fixed_cols, fixed_rows, source)
+        self.origins.append(origin)
+        self._cover(origin, r, r + rowspan, c, c + colspan)
+
+    # -- rows (spec 3.2) ---------------------------------------------------------
+
+    def _compact_fits(self, r: int, texts: list[str]) -> bool:
+        trial = copy.deepcopy(self)
+        used = 0
+        for c in range(self.cols):
+            if trial.owner[r][c] is None:
+                if used == len(texts):
+                    return False
+                trial._place(r, c, texts[used])
+                used += 1
+        return all(t == "" for t in texts[used:])
+
+    def add_row(self, r: int, texts: list[str]) -> str:
+        """Place row r. Returns "grid", "compact" or "padded" (the row did not fit: V14)."""
+        n = len(texts)
+        if n != self.cols and self._compact_fits(r, texts):
+            remaining = iter(enumerate(texts))
+            for c in range(self.cols):
+                if self.owner[r][c] is None:
+                    index, text = next(remaining)
+                    self._place(r, c, text, index)
+            return "compact"
+        form = "grid"
+        if n != self.cols:
+            self._report("V14", r, None, f"row has {n} cells; it fits neither as a grid row "
+                                         f"({self.cols} cells) nor as a compact row")
+            texts = (texts + [""] * self.cols)[: self.cols]
+            form = "padded"
+        for c, text in enumerate(texts):
+            covering = self.owner[r][c]
+            if covering is not None:
+                if text not in ("", COLSPAN_MARKER, ROWSPAN_MARKER):
+                    rule = "V5" if covering.row < r else "V6"
+                    self._report(rule, r, c, f"a covered position holds text {text!r}; it is dropped")
+                continue
+            self._place(r, c, text, c if c < n else None)
+        return form

ltem_core-0.1.0/src/ltem/_syntax.py

@@ -0,0 +1,204 @@
+"""Line-level syntax: rows, delimiter rows, cell kinds and span attributes (spec Section 2)."""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+
+DELIMITER_CELL = re.compile(r":?-+:?")
+# A span attribute ends the cell and is the whole cell or follows a space or tab (spec 2.6).
+SPAN_ATTRIBUTE = re.compile(r"(?:^|(?<=[ \t]))\{([cr]:[0-9]+(?:,[cr]:[0-9]+)*)\}$")
+# Reserved table-identifier line (spec Section 9).
+IDENTIFIER_ROW = re.compile(r"^ {0,3}\|[ \t]*#[ \t]+[A-Za-z_][A-Za-z0-9_]*[ \t]*\|[ \t]*$")
+PIPE_LED = re.compile(r"^ {0,3}\|")
+LINE_BREAK = re.compile(r"\\\\")
+
+COLSPAN_MARKER = "<<"
+ROWSPAN_MARKER = "^^"
+
+
+def split_row_spans(line: str) -> list[tuple[str, int, int]]:
+    """Split a row into cells, as GFM does (spec 2.1), keeping where each cell is.
+
+    Each cell is (text, start, end). `text` is trimmed of spaces and tabs, and a pipe
+    preceded by a backslash becomes a literal pipe. `start` and `end` index `line` and
+    cover the cell's source without the spaces and tabs around it; for an empty cell
+    both are the end of its space between the pipes.
+    """
+    b, e = 0, len(line)
+    while b < e and line[b] in " \t":
+        b += 1
+    while e > b and line[e - 1] in " \t":
+        e -= 1
+    if b < e and line[b] == "|":
+        b += 1
+    if e > b and line[e - 1] == "|" and not (e - 2 >= b and line[e - 2] == "\\"):
+        e -= 1
+    cells: list[tuple[str, int, int]] = []
+    cur: list[str] = []
+    cell_start = i = b
+    while i < e:
+        ch = line[i]
+        if ch == "\\" and i + 1 < e and line[i + 1] == "|":
+            cur.append("|")
+            i += 2
+            continue
+        if ch == "|":
+            cells.append(_cell_span(line, cell_start, i, cur))
+            cur = []
+            cell_start = i + 1
+        else:
+            cur.append(ch)
+        i += 1
+    cells.append(_cell_span(line, cell_start, e, cur))
+    return cells
+
+
+def _cell_span(line: str, start: int, end: int, chars: list[str]) -> tuple[str, int, int]:
+    while start < end and line[start] in " \t":
+        start += 1
+    while end > start and line[end - 1] in " \t":
+        end -= 1
+    return "".join(chars).strip(" \t"), start, end
+
+
+def split_row(line: str) -> list[str]:
+    """Split a row into trimmed cell texts, as GFM does (spec 2.1).
+
+    A pipe preceded by a backslash does not split cells and becomes a literal pipe.
+    """
+    return [text for text, _, _ in split_row_spans(line)]
+
+
+def indentation(line: str) -> int:
+    """Width of the leading spaces and tabs, with tab stops every 4 columns."""
+    width = 0
+    for ch in line:
+        if ch == " ":
+            width += 1
+        elif ch == "\t":
+            width += 4 - width % 4
+        else:
+            break
+    return width
+
+
+_LIST_DASH = re.compile(r"^ {0,3}-[ \t]")
+
+
+def is_delimiter_row(line: str) -> bool:
+    """A GFM delimiter row (spec 2.2).
+
+    It has at least one pipe, is indented less than 4 spaces, and does not begin a list
+    item: a line such as "- | -" is a list item, as in GFM.
+    """
+    if "|" not in line or indentation(line) >= 4 or _LIST_DASH.match(line):
+        return False
+    return all(DELIMITER_CELL.fullmatch(c) for c in split_row(line))
+
+
+def is_upper_header_row(line: str) -> bool:
+    """Whether a line can be a header row above the last one (spec 2.3, 9).
+
+    It begins with a pipe, is not a reserved identifier line and does not look like a
+    delimiter row.
+    """
+    return bool(PIPE_LED.match(line)) and not IDENTIFIER_ROW.match(line) and not is_delimiter_row(line)
+
+
+# Lines that start another block and so end a table body (spec 2.4).
+_BODY_END = re.compile(
+    r"^ {0,3}(?:"
+    r">"                                  # block quote
+    r"|#{1,6}(?:[ \t]|$)"                # ATX heading
+    r"|`{3,}|~{3,}"                       # fenced code
+    r"|([-*_])(?:[ \t]*\1){2,}[ \t]*$"     # thematic break
+    r"|(?:[-+*]|[0-9]{1,9}[.)])(?:[ \t]|$)"  # list item
+    r")"
+)
+
+
+_SETEXT_UNDERLINE = re.compile(r"^ {0,3}(?:=+|-+)[ \t]*$")
+
+
+def is_setext_underline(line: str) -> bool:
+    """A line of "=" or of "-" characters, which underlines a paragraph line as a heading."""
+    return bool(_SETEXT_UNDERLINE.match(line))
+
+
+def starts_block(line: str) -> bool:
+    """Whether a line begins a block quote, ATX heading, fenced code, thematic break or list item."""
+    return bool(_BODY_END.match(line))
+
+
+def ends_table_body(line: str) -> bool:
+    """Whether a table ends before this line (spec 2.4): blank, indented code, or another block."""
+    return not line.strip(" \t") or indentation(line) >= 4 or starts_block(line)
+
+
+def alignment(cell: str) -> str | None:
+    left, right = cell.startswith(":"), cell.endswith(":")
+    if left and right:
+        return "center"
+    if right:
+        return "right"
+    if left:
+        return "left"
+    return None
+
+
+@dataclass
+class Attribute:
+    colspan: int | None
+    rowspan: int | None
+    problems: list[str]
+
+
+def split_attribute(text: str) -> tuple[str, Attribute | None]:
+    """Remove a trailing span attribute from a content cell (spec 2.6).
+
+    Returns the remaining text and the attribute, or (text, None) if there is none.
+    """
+    m = SPAN_ATTRIBUTE.search(text)
+    if not m:
+        return text, None
+    colspan = rowspan = None
+    problems = []
+    for spec in m.group(1).split(","):
+        key, value = spec.split(":")
+        n = int(value)
+        if n < 1:
+            problems.append(f"span value must be a positive integer, got {value}")
+            n = 1
+        if key == "c":
+            if colspan is not None:
+                problems.append("'c' appears more than once")
+                continue
+            colspan = n
+        else:
+            if rowspan is not None:
+                problems.append("'r' appears more than once")
+                continue
+            rowspan = n
+    return text[: m.start()].rstrip(" \t"), Attribute(colspan, rowspan, problems)
+
+
+# The LTEM escapes of spec 4.3, as written in cell source.
+_ESCAPED_MARKER = re.compile(r"^\\(<<|\^\^)$")
+_ESCAPED_ATTRIBUTE = re.compile(r"(?:^|(?<=[ \t]))\\(\{[cr]:[0-9]+(?:,[cr]:[0-9]+)*\})$")
+
+
+def unescape_ltem(line: str) -> str:
+    """Remove the backslash of an LTEM escape (spec 4.3), as GFM does for \\|."""
+    m = _ESCAPED_MARKER.match(line)
+    if m:
+        return m.group(1)
+    return _ESCAPED_ATTRIBUTE.sub(r"\1", line)
+
+
+def cell_lines(text: str) -> str:
+    """Turn cell source into model text (spec Section 7).
+
+    Split at each \\\\, trim each line and remove LTEM escapes.
+    """
+    return "\n".join(unescape_ltem(part.strip(" \t")) for part in LINE_BREAK.split(text))