lattuce-latex2md 0.2.0__tar.gz

lattuce_latex2md-0.2.0/PKG-INFO

@@ -0,0 +1,142 @@
+Metadata-Version: 2.4
+Name: lattuce-latex2md
+Version: 0.2.0
+Summary: Markdown <-> LaTeX conversion built on pandoc, with math, mhchem and citation clean-up.
+Project-URL: Source, https://github.com/lattuce-ai/latex2md
+Project-URL: Issues, https://github.com/lattuce-ai/latex2md/issues
+Keywords: markdown,latex,pandoc,converter,arxiv
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing :: Markup :: LaTeX
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+
+# latex2md
+
+Convert **Markdown → LaTeX** and **LaTeX → Markdown**, as a Python library or a CLI.
+
+It runs [pandoc](https://pandoc.org) and cleans up the output on both sides, so you get
+plain, readable Markdown and LaTeX fragments you can paste into an existing document.
+
+## Install
+
+You need **pandoc** 3.0 or later (CI tests 3.1 and 3.6) on your `PATH`:
+
+```bash
+sudo apt-get install pandoc
+```
+
+```bash
+pip install lattuce-latex2md
+```
+
+The distribution is called `lattuce-latex2md` (the name `latex2md` is taken on PyPI
+by an unrelated project); the import name is `latex2md`. From a checkout:
+`pip install -e .`.
+
+If pandoc is somewhere else, set `LATEX2MD_PANDOC=/path/to/pandoc`.
+
+## CLI
+
+```bash
+latex2md to-latex notes.md -o notes.tex
+```
+
+```bash
+latex2md to-md paper.tex -o paper.md
+```
+
+```bash
+latex2md auto paper.tex
+```
+
+- With no input file, or `-`, it reads stdin. With no `-o`, it writes to stdout.
+- `auto` picks the direction from the extension (`.md/.markdown` or `.tex/.latex/.ltx`).
+- `--standalone` (for `to-latex` and `auto`) writes a complete document with a preamble.
+- `--base-dir DIR` (for `to-md` and `auto`) is the folder `\input`/`\include` may read from.
+  It defaults to the input file's folder (or the current folder for stdin).
+
+## Python
+
+```python
+from latex2md import md_to_latex, latex_to_md, ConversionError
+
+tex = md_to_latex("# Title\n\nEnergy is $E = mc^2$ [@einstein1905].")
+md = latex_to_md(r"\section{Title} Energy is $E = mc^2$ \cite{einstein1905}.")
+full = md_to_latex(markdown_text, standalone=True)
+
+# Multi-file projects (e.g. an unpacked arXiv source archive):
+md = latex_file_to_md("paper/main.tex")                      # YAML front matter
+md = latex_file_to_md("paper/main.tex", front_matter=False)  # abstract as a section
+md = latex_to_md(tex_text, base_dir="paper/")                # text + project folder
+```
+
+All errors (pandoc missing, failing or timing out) raise `ConversionError`; its
+`.stderr` holds pandoc's message.
+
+## What it handles
+
+| | Markdown → LaTeX | LaTeX → Markdown |
+|---|---|---|
+| Headings | `\section{}` + `\label{}` | `#`, `##`, … (labels dropped) |
+| Math | `\( \)` and `\[ \]` | `$…$` and `$$…$$` |
+| Chemistry (`\ce{}`) | kept; `--standalone` loads `mhchem` | kept inside math |
+| Citations | `[@key]` → `\cite{key}` | `\cite{key}` → `[@key]` |
+| Lists | `itemize` / `enumerate` | tight `-` / `1.` lists |
+| Tables | `longtable` (needs `longtable` + `booktabs`) | pipe tables; first row becomes header |
+| Code | `verbatim` | fenced blocks |
+| Comments | — | `%` comments removed; `\%`, `\verb` and verbatim kept |
+| Multi-file projects | — | `\input`, `\include`, `\subfile`, `\import`, `\lstinputlisting` inlined from the project folder |
+| Title, authors, abstract | — | YAML front matter, or an "Abstract" section with `front_matter=False` |
+| Macros (`\newcommand`) | — | expanded |
+| Floats | — | `figure` → image with caption as alt text; `table` → "**Table:** caption" + pipe table |
+| Cross-references | — | `\ref` → its number as text (no dead `#anchor` links); `\eqref` → `(label)` |
+| Bibliography | — | `thebibliography` → "References" list; a sibling `.bbl` fills `\bibliography{}` |
+
+Fragment output (the default) avoids macros that only pandoc's template defines
+(`\tightlist`, `Shaded`, `\hypertarget`), so it compiles in an ordinary preamble.
+
+## Safety
+
+LaTeX can name any file on disk (`\input{/etc/passwd}`), so latex2md never lets
+pandoc read files itself: pandoc always runs with `--sandbox`, and latex2md
+inlines includes only when they resolve inside the base folder. Absolute paths,
+`..` escapes, symlinks pointing outside, cycles, nesting deeper than 16 levels and
+more than 20 MB of included files are skipped. Without a base folder, include
+commands are removed. This makes it safe to run on untrusted uploads.
+
+## Limitations
+
+- Custom macros and unusual packages in LaTeX are only as good as pandoc's LaTeX reader;
+  unknown commands are dropped.
+- Equations are not numbered in Markdown, so `\eqref` shows the label, e.g. `(eq:loss)`.
+- Bibliographies from `.bib` files need the compiled `.bbl`; citations stay as `[@key]`.
+- Figures with `\includegraphics` keep the path; image files are not copied.
+
+## Settings
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `LATEX2MD_PANDOC` | `pandoc` on `PATH` | pandoc binary to run |
+| `LATEX2MD_TIMEOUT` | `60` | seconds before a conversion is stopped |
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+```
+
+```bash
+pytest
+```
+
+## Releasing
+
+Bump `version` in `pyproject.toml` and `__version__` in `src/latex2md/__init__.py`,
+then push a matching tag (`v0.2.0`). The `release` workflow tests, builds and
+publishes to PyPI with trusted publishing (setup steps are in
+`.github/workflows/release.yml`).

lattuce_latex2md-0.2.0/README.md

@@ -0,0 +1,125 @@
+# latex2md
+
+Convert **Markdown → LaTeX** and **LaTeX → Markdown**, as a Python library or a CLI.
+
+It runs [pandoc](https://pandoc.org) and cleans up the output on both sides, so you get
+plain, readable Markdown and LaTeX fragments you can paste into an existing document.
+
+## Install
+
+You need **pandoc** 3.0 or later (CI tests 3.1 and 3.6) on your `PATH`:
+
+```bash
+sudo apt-get install pandoc
+```
+
+```bash
+pip install lattuce-latex2md
+```
+
+The distribution is called `lattuce-latex2md` (the name `latex2md` is taken on PyPI
+by an unrelated project); the import name is `latex2md`. From a checkout:
+`pip install -e .`.
+
+If pandoc is somewhere else, set `LATEX2MD_PANDOC=/path/to/pandoc`.
+
+## CLI
+
+```bash
+latex2md to-latex notes.md -o notes.tex
+```
+
+```bash
+latex2md to-md paper.tex -o paper.md
+```
+
+```bash
+latex2md auto paper.tex
+```
+
+- With no input file, or `-`, it reads stdin. With no `-o`, it writes to stdout.
+- `auto` picks the direction from the extension (`.md/.markdown` or `.tex/.latex/.ltx`).
+- `--standalone` (for `to-latex` and `auto`) writes a complete document with a preamble.
+- `--base-dir DIR` (for `to-md` and `auto`) is the folder `\input`/`\include` may read from.
+  It defaults to the input file's folder (or the current folder for stdin).
+
+## Python
+
+```python
+from latex2md import md_to_latex, latex_to_md, ConversionError
+
+tex = md_to_latex("# Title\n\nEnergy is $E = mc^2$ [@einstein1905].")
+md = latex_to_md(r"\section{Title} Energy is $E = mc^2$ \cite{einstein1905}.")
+full = md_to_latex(markdown_text, standalone=True)
+
+# Multi-file projects (e.g. an unpacked arXiv source archive):
+md = latex_file_to_md("paper/main.tex")                      # YAML front matter
+md = latex_file_to_md("paper/main.tex", front_matter=False)  # abstract as a section
+md = latex_to_md(tex_text, base_dir="paper/")                # text + project folder
+```
+
+All errors (pandoc missing, failing or timing out) raise `ConversionError`; its
+`.stderr` holds pandoc's message.
+
+## What it handles
+
+| | Markdown → LaTeX | LaTeX → Markdown |
+|---|---|---|
+| Headings | `\section{}` + `\label{}` | `#`, `##`, … (labels dropped) |
+| Math | `\( \)` and `\[ \]` | `$…$` and `$$…$$` |
+| Chemistry (`\ce{}`) | kept; `--standalone` loads `mhchem` | kept inside math |
+| Citations | `[@key]` → `\cite{key}` | `\cite{key}` → `[@key]` |
+| Lists | `itemize` / `enumerate` | tight `-` / `1.` lists |
+| Tables | `longtable` (needs `longtable` + `booktabs`) | pipe tables; first row becomes header |
+| Code | `verbatim` | fenced blocks |
+| Comments | — | `%` comments removed; `\%`, `\verb` and verbatim kept |
+| Multi-file projects | — | `\input`, `\include`, `\subfile`, `\import`, `\lstinputlisting` inlined from the project folder |
+| Title, authors, abstract | — | YAML front matter, or an "Abstract" section with `front_matter=False` |
+| Macros (`\newcommand`) | — | expanded |
+| Floats | — | `figure` → image with caption as alt text; `table` → "**Table:** caption" + pipe table |
+| Cross-references | — | `\ref` → its number as text (no dead `#anchor` links); `\eqref` → `(label)` |
+| Bibliography | — | `thebibliography` → "References" list; a sibling `.bbl` fills `\bibliography{}` |
+
+Fragment output (the default) avoids macros that only pandoc's template defines
+(`\tightlist`, `Shaded`, `\hypertarget`), so it compiles in an ordinary preamble.
+
+## Safety
+
+LaTeX can name any file on disk (`\input{/etc/passwd}`), so latex2md never lets
+pandoc read files itself: pandoc always runs with `--sandbox`, and latex2md
+inlines includes only when they resolve inside the base folder. Absolute paths,
+`..` escapes, symlinks pointing outside, cycles, nesting deeper than 16 levels and
+more than 20 MB of included files are skipped. Without a base folder, include
+commands are removed. This makes it safe to run on untrusted uploads.
+
+## Limitations
+
+- Custom macros and unusual packages in LaTeX are only as good as pandoc's LaTeX reader;
+  unknown commands are dropped.
+- Equations are not numbered in Markdown, so `\eqref` shows the label, e.g. `(eq:loss)`.
+- Bibliographies from `.bib` files need the compiled `.bbl`; citations stay as `[@key]`.
+- Figures with `\includegraphics` keep the path; image files are not copied.
+
+## Settings
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `LATEX2MD_PANDOC` | `pandoc` on `PATH` | pandoc binary to run |
+| `LATEX2MD_TIMEOUT` | `60` | seconds before a conversion is stopped |
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+```
+
+```bash
+pytest
+```
+
+## Releasing
+
+Bump `version` in `pyproject.toml` and `__version__` in `src/latex2md/__init__.py`,
+then push a matching tag (`v0.2.0`). The `release` workflow tests, builds and
+publishes to PyPI with trusted publishing (setup steps are in
+`.github/workflows/release.yml`).

lattuce_latex2md-0.2.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "lattuce-latex2md"
+version = "0.2.0"
+description = "Markdown <-> LaTeX conversion built on pandoc, with math, mhchem and citation clean-up."
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = []
+keywords = ["markdown", "latex", "pandoc", "converter", "arxiv"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Topic :: Text Processing :: Markup :: LaTeX",
+    "Topic :: Text Processing :: Markup :: Markdown",
+]
+
+[project.urls]
+Source = "https://github.com/lattuce-ai/latex2md"
+Issues = "https://github.com/lattuce-ai/latex2md/issues"
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "build>=1.2"]
+
+[project.scripts]
+latex2md = "latex2md.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+latex2md = ["*.lua"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

lattuce_latex2md-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

lattuce_latex2md-0.2.0/src/latex2md/__init__.py

@@ -0,0 +1,7 @@
+"""Markdown <-> LaTeX conversion built on pandoc."""
+from latex2md.latex_to_md import latex_file_to_md, latex_to_md
+from latex2md.md_to_latex import md_to_latex
+from latex2md.pandoc import ConversionError, pandoc_available
+
+__all__ = ["ConversionError", "latex_file_to_md", "latex_to_md", "md_to_latex", "pandoc_available"]
+__version__ = "0.2.0"

lattuce_latex2md-0.2.0/src/latex2md/__main__.py

@@ -0,0 +1,3 @@
+from latex2md.cli import main
+
+raise SystemExit(main())

lattuce_latex2md-0.2.0/src/latex2md/cli.py

@@ -0,0 +1,79 @@
+"""Command-line interface: ``latex2md to-latex|to-md|auto``."""
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from latex2md.latex_to_md import latex_file_to_md, latex_to_md
+from latex2md.md_to_latex import md_to_latex
+from latex2md.pandoc import ConversionError
+
+MARKDOWN_SUFFIXES = {".md", ".markdown", ".mdown"}
+LATEX_SUFFIXES = {".tex", ".latex", ".ltx"}
+
+
+def _read(path: str) -> str:
+    if path == "-":
+        return sys.stdin.read()
+    return Path(path).read_text(encoding="utf-8")
+
+
+def _write(path: str | None, text: str) -> None:
+    if not path or path == "-":
+        sys.stdout.write(text)
+    else:
+        Path(path).write_text(text, encoding="utf-8")
+
+
+def _direction(command: str, path: str) -> str:
+    if command != "auto":
+        return command
+    suffix = Path(path).suffix.lower()
+    if suffix in MARKDOWN_SUFFIXES:
+        return "to-latex"
+    if suffix in LATEX_SUFFIXES:
+        return "to-md"
+    raise ConversionError(f"cannot tell the input format from {path!r}; use to-latex or to-md")
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="latex2md", description="Convert between Markdown and LaTeX.")
+    sub = parser.add_subparsers(dest="command", required=True)
+    for name, help_text in (
+        ("to-latex", "Markdown -> LaTeX"),
+        ("to-md", "LaTeX -> Markdown"),
+        ("auto", "pick the direction from the input file extension"),
+    ):
+        cmd = sub.add_parser(name, help=help_text)
+        cmd.add_argument("input", nargs="?", default="-", help="input file, or - for stdin (default)")
+        cmd.add_argument("-o", "--output", help="output file (default: stdout)")
+        if name != "to-latex":
+            cmd.add_argument(
+                "--base-dir",
+                help="folder that \\input/\\include may read from (default: the input file's folder, or cwd for stdin)",
+            )
+        if name != "to-md":
+            cmd.add_argument("--standalone", action="store_true", help="emit a complete LaTeX document")
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    try:
+        direction = _direction(args.command, args.input)
+        if direction == "to-latex":
+            result = md_to_latex(_read(args.input), standalone=getattr(args, "standalone", False))
+        elif args.input == "-":
+            result = latex_to_md(_read(args.input), base_dir=args.base_dir or Path.cwd())
+        else:
+            result = latex_file_to_md(args.input, base_dir=args.base_dir)
+        _write(args.output, result)
+    except (ConversionError, OSError) as exc:
+        print(f"latex2md: {exc}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

lattuce_latex2md-0.2.0/src/latex2md/comments.py

@@ -0,0 +1,25 @@
+"""LaTeX comment stripping that leaves verbatim content alone."""
+from __future__ import annotations
+
+import re
+
+# A % starts a comment unless it is escaped (\%).
+_COMMENT_RE = re.compile(r"(?<!\\)%.*$", re.M)
+_COMMENT_ONLY_LINE_RE = re.compile(r"^[ \t]*(?<!\\)%.*\n", re.M)
+_VERBATIM_RE = re.compile(r"\\begin\{(verbatim\*?|lstlisting|minted|comment)\}.*?\\end\{\1\}", re.S)
+_VERB_INLINE_RE = re.compile(r"\\verb\*?([^a-zA-Z\s*]).*?\1")
+
+
+def strip_comments(latex: str) -> str:
+    """Remove % comments, leaving verbatim environments and \\verb untouched."""
+    protected: list[str] = []
+
+    def protect(match: re.Match[str]) -> str:
+        protected.append(match.group(0))
+        return f"\x00{len(protected) - 1}\x00"
+
+    text = _VERBATIM_RE.sub(protect, latex)
+    text = _VERB_INLINE_RE.sub(protect, text)
+    text = _COMMENT_ONLY_LINE_RE.sub("", text)
+    text = _COMMENT_RE.sub("", text)
+    return re.sub(r"\x00(\d+)\x00", lambda m: protected[int(m.group(1))], text)

lattuce_latex2md-0.2.0/src/latex2md/includes.py

@@ -0,0 +1,113 @@
+"""Inline \\input / \\include files before pandoc sees the document.
+
+pandoc always runs with ``--sandbox`` (see :mod:`latex2md.pandoc`), so it cannot
+read files on its own. This module resolves include commands itself, but only
+for files inside ``base_dir``: absolute paths, ``..`` escapes and symlinks that
+point outside are dropped. That makes it safe to convert untrusted LaTeX, such
+as arXiv source archives.
+"""
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+from latex2md.comments import strip_comments
+
+MAX_INCLUDE_DEPTH = 16
+MAX_INCLUDED_BYTES = 20 * 1024 * 1024
+
+_GROUP = r"\{\s*([^{}]+?)\s*\}"
+# \input{file}, \include{file}, \subfile{file}, \input file
+_INPUT_RE = re.compile(r"\\(input|include|subfile)\s*(?:" + _GROUP + r"|\s+([^\s{}\\]+))")
+# \import{dir}{file}, \subimport{dir}{file}, starred variants
+_IMPORT_RE = re.compile(r"\\(?:sub)?(?:import|inputfrom|includefrom)\*?\s*" + _GROUP + r"\s*" + _GROUP)
+# \lstinputlisting[opts]{file}, \verbatiminput{file}
+_LISTING_RE = re.compile(r"\\(lstinputlisting|verbatiminput)\s*(?:\[[^\]]*\])?\s*" + _GROUP)
+
+
+class _Budget:
+    def __init__(self) -> None:
+        self.remaining = MAX_INCLUDED_BYTES
+
+
+def _resolve(base_dir: Path, current_dir: Path, name: str, *, add_tex: bool) -> Path | None:
+    name = name.strip().strip('"')
+    if not name or Path(name).is_absolute():
+        return None
+    candidates = [current_dir / name, base_dir / name]
+    if add_tex and not name.endswith(".tex"):
+        candidates = [path for c in candidates for path in (c.with_name(c.name + ".tex"), c)]
+    root = base_dir.resolve()
+    for candidate in candidates:
+        try:
+            resolved = candidate.resolve()
+        except OSError:
+            continue
+        if resolved.is_relative_to(root) and resolved.is_file():
+            return resolved
+    return None
+
+
+def _read(path: Path, budget: _Budget) -> str | None:
+    size = path.stat().st_size
+    if size > budget.remaining:
+        return None
+    budget.remaining -= size
+    return path.read_text(encoding="utf-8", errors="replace")
+
+
+def _expand(text: str, *, base_dir: Path, current_dir: Path, stack: tuple[Path, ...], budget: _Budget) -> str:
+    def include(path: Path | None) -> str:
+        if path is None or path in stack or len(stack) >= MAX_INCLUDE_DEPTH:
+            return ""
+        content = _read(path, budget)
+        if content is None:
+            return ""
+        body = _subfile_body(strip_comments(content))
+        expanded = _expand(body, base_dir=base_dir, current_dir=path.parent, stack=(*stack, path), budget=budget)
+        return f"\n{expanded}\n"
+
+    def replace_input(match: re.Match[str]) -> str:
+        name = match.group(2) or match.group(3) or ""
+        return include(_resolve(base_dir, current_dir, name, add_tex=True))
+
+    def replace_import(match: re.Match[str]) -> str:
+        directory, name = match.group(1), match.group(2)
+        return include(_resolve(base_dir, current_dir, str(Path(directory) / name), add_tex=True))
+
+    def replace_listing(match: re.Match[str]) -> str:
+        path = _resolve(base_dir, current_dir, match.group(2), add_tex=False)
+        content = _read(path, budget) if path else None
+        if content is None:
+            return ""
+        return "\n\\begin{verbatim}\n" + content.rstrip("\n").replace("\\end{verbatim}", "") + "\n\\end{verbatim}\n"
+
+    text = _LISTING_RE.sub(replace_listing, text)
+    text = _IMPORT_RE.sub(replace_import, text)
+    return _INPUT_RE.sub(replace_input, text)
+
+
+_DOCUMENT_BODY_RE = re.compile(r"\\begin\{document\}(.*)\\end\{document\}", re.S)
+
+
+def _subfile_body(text: str) -> str:
+    """A \\subfile is a full document; keep only its body."""
+    if "\\documentclass" in text:
+        match = _DOCUMENT_BODY_RE.search(text)
+        if match:
+            return match.group(1)
+    return text
+
+
+def expand_includes(latex: str, base_dir: str | Path | None) -> str:
+    """Inline include commands resolved under ``base_dir``; drop the rest.
+
+    ``latex`` must already have its comments stripped. With no ``base_dir``
+    every include command is removed.
+    """
+    if base_dir is None:
+        text = _LISTING_RE.sub("", latex)
+        text = _IMPORT_RE.sub("", text)
+        return _INPUT_RE.sub("", text)
+    root = Path(base_dir).resolve()
+    return _expand(latex, base_dir=root, current_dir=root, stack=(), budget=_Budget())