chatgpt-md-converter 0.3.7__tar.gz → 0.3.8__tar.gz

{chatgpt_md_converter-0.3.7 → chatgpt_md_converter-0.3.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: chatgpt_md_converter
-Version: 0.3.7
+Version: 0.3.8
 Summary: A package for converting markdown to HTML for chat Telegram bots
 Home-page: https://github.com/botfather-dev/formatter-chatgpt-telegram
 Author: Kostiantyn Kriuchkov

chatgpt_md_converter-0.3.8/README.md

@@ -0,0 +1,147 @@
+# ChatGPT Markdown to Telegram HTML Parser
+
+## Overview
+
+This project provides a solution for converting Telegram-style Markdown formatted text into HTML markup supported by the Telegram Bot API, specifically tailored for use in ChatGPT bots developed with the OpenAI API. It includes features for handling various Markdown elements and ensures proper tag closure, making it suitable for streaming mode applications.
+
+## Features
+
+- Converts Telegram-style Markdown syntax to Telegram-compatible HTML
+- Supports text styling:
+  - Bold: `**text**` → `<b>text</b>`
+  - Italic: `*text*` or `_text_` → `<i>text</i>`
+  - Underline: `__text__` → `<u>text</u>`
+  - Strikethrough: `~~text~~` → `<s>text</s>`
+  - Spoiler: `||text||` → `<span class="tg-spoiler">text</span>`
+  - Inline code: `` `code` `` → `<code>code</code>`
+- Handles nested text styling
+- Converts links: `[text](URL)` → `<a href="URL">text</a>`
+- Processes code blocks with language specification
+- Supports blockquotes:
+  - Regular blockquotes: `> text` → `<blockquote>text</blockquote>`
+  - Expandable blockquotes: `**> text` → `<blockquote expandable>text</blockquote>`
+- Automatically appends missing closing delimiters for code blocks
+- Escapes HTML special characters to prevent unwanted HTML rendering
+
+## Usage
+
+To use the Markdown to Telegram HTML Parser in your ChatGPT bot, integrate the provided Python functions into your bot's processing pipeline. Here is a brief overview of how to incorporate the parser:
+
+1. **Ensure Closing Delimiters**: Automatically appends missing closing delimiters for backticks to ensure proper parsing.
+
+2. **Extract and Convert Code Blocks**: Extracts Markdown code blocks, converts them to HTML `<pre><code>` format, and replaces them with placeholders to prevent formatting within code blocks.
+
+3. **Markdown to HTML Conversion**: Applies various regex substitutions and custom logic to convert supported Markdown formatting to Telegram-compatible HTML tags.
+
+4. **Reinsert Code Blocks**: Reinserts the previously extracted and converted code blocks back into the main text, replacing placeholders with the appropriate HTML content.
+
+Simply call the `telegram_format(text: str) -> str` function with your Markdown-formatted text as input to receive the converted HTML output ready for use with the Telegram Bot API.
+
+## Installation
+
+```sh
+pip install chatgpt-md-converter
+```
+
+## Example
+
+```python
+from chatgpt_md_converter import telegram_format
+
+# Basic formatting example
+text = """
+Here is some **bold**, __underline__, and `inline code`.
+This is a ||spoiler text|| and *italic*.
+
+Code example:
+print('Hello, world!')
+"""
+
+# Blockquotes example
+blockquote_text = """
+> Regular blockquote
+> Multiple lines
+
+**> Expandable blockquote
+> Hidden by default
+> Multiple lines
+"""
+
+formatted_text = telegram_format(text)
+formatted_blockquote = telegram_format(blockquote_text)
+
+print(formatted_text)
+print(formatted_blockquote)
+```
+
+### Output:
+
+```
+Here is some <b>bold</b>, <u>underline</u>, and <code>inline code</code>.
+This is a <span class="tg-spoiler">spoiler text</span> and <i>italic</i>.
+
+Code example:
+print('Hello, world!')
+
+<blockquote>Regular blockquote
+Multiple lines</blockquote>
+
+<blockquote expandable>Expandable blockquote
+Hidden by default
+Multiple lines</blockquote>
+```
+
+## Requirements
+
+- Python 3.x
+- No external libraries required (uses built-in `re` module for regex operations)
+
+## Contribution
+
+Feel free to contribute to this project by submitting pull requests or opening issues for bugs, feature requests, or improvements.
+
+## Prompting LLMs for Telegram-Specific Formatting
+
+> **Note**:
+> Since standard Markdown doesn't include Telegram-specific features like spoilers (`||text||`) and expandable blockquotes (`**> text`), you'll need to explicitly instruct LLMs to use these formats. Here's a suggested prompt addition to include in your system message or initial instructions:
+
+````
+When formatting your responses for Telegram, please use these special formatting conventions:
+
+1. For content that should be hidden as a spoiler (revealed only when users click):
+   Use: ||spoiler content here||
+   Example: This is visible, but ||this is hidden until clicked||.
+
+2. For lengthy explanations or optional content that should be collapsed:
+   Use: **> Expandable section title
+
+   > Content line 1
+   > Content line 2
+   > (Each line of the expandable blockquote should start with ">")
+
+3. Continue using standard markdown for other formatting:
+   - **bold text**
+   - *italic text*
+   - __underlined text__
+   - ~~strikethrough~~
+   - `inline code`
+   - ```code blocks```
+   - [link text](URL)
+
+Apply spoilers for:
+
+- Solution reveals
+- Potential plot spoilers
+- Sensitive information
+- Surprising facts
+
+Use expandable blockquotes for:
+
+- Detailed explanations
+- Long examples
+- Optional reading
+- Technical details
+- Additional context not needed by all users
+````
+
+You can add this prompt to your system message when initializing your ChatGPT interactions to ensure the model properly formats content for optimal display in Telegram.

chatgpt_md_converter-0.3.8/chatgpt_md_converter/__init__.py

@@ -0,0 +1,4 @@
+from .telegram_formatter import telegram_format
+from .html_splitter import split_html_for_telegram
+
+__all__ = ["telegram_format", "split_html_for_telegram"]

{chatgpt_md_converter-0.3.7 → chatgpt_md_converter-0.3.8}/chatgpt_md_converter/extractors.py

@@ -2,31 +2,32 @@ import re
 
 
 def ensure_closing_delimiters(text: str) -> str:
-    """Append missing closing backtick delimiters."""
+    # Append missing closing backtick delimiters.
 
     code_block_re = re.compile(
         r"(?P<fence>`{3,})(?P<lang>\w+)?\n?[\s\S]*?(?<=\n)?(?P=fence)",
         flags=re.DOTALL,
     )
 
-    # Remove complete code blocks from consideration so inner backticks
-    # don't affect delimiter balancing.
-    cleaned = code_block_re.sub("", text)
-
-    # Detect unclosed fences by tracking opening fence lengths.
-    stack = []
-    for line in cleaned.splitlines():
-        m = re.match(r"^(?P<fence>`{3,})(?P<lang>\w+)?$", line.strip())
-        if not m:
-            continue
-        fence = m.group("fence")
-        if stack and fence == stack[-1]:
-            stack.pop()
+    # Track an open fence.  Once a fence is opened, everything until the same
+    # fence is encountered again is treated as plain text.  This mimics how
+    # Markdown handles fences and allows fence-like strings inside code blocks.
+    open_fence = None
+    for line in text.splitlines():
+        stripped = line.strip()
+        if open_fence is None:
+            m = re.match(r"^(?P<fence>`{3,})(?P<lang>\w+)?$", stripped)
+            if m:
+                open_fence = m.group("fence")
         else:
-            stack.append(fence)
+            if stripped.endswith(open_fence):
+                open_fence = None
 
-    if stack:
-        text += "\n" + stack[-1]
+    # If a fence was left open, append a matching closing fence.
+    if open_fence is not None:
+        if not text.endswith("\n"):
+            text += "\n"
+        text += open_fence
 
     cleaned_inline = code_block_re.sub("", text)
 
@@ -91,4 +92,4 @@ def reinsert_code_blocks(text: str, code_blocks: dict) -> str:
     """
     for placeholder, html_code_block in code_blocks.items():
         text = text.replace(placeholder, html_code_block, 1)
-    return text
+    return text

chatgpt_md_converter-0.3.8/chatgpt_md_converter/html_splitter.py

@@ -0,0 +1,239 @@
+import re
+from html.parser import HTMLParser
+
+MAX_LENGTH = 4096
+MIN_LENGTH = 500
+
+
+class HTMLTagTracker(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.open_tags = []
+
+    def handle_starttag(self, tag, attrs):
+        # saving tags
+        if tag in (
+                  "b", "i", "u", "s", "code", "pre", "a", "span", "blockquote",
+                  "strong", "em", "ins", "strike", "del", "tg-spoiler", "tg-emoji"
+                  ):
+            self.open_tags.append((tag, attrs))
+
+    def handle_endtag(self, tag):
+        for i in range(len(self.open_tags) - 1, -1, -1):
+            if self.open_tags[i][0] == tag:
+                del self.open_tags[i]
+                break
+
+    def get_open_tags_html(self):
+        parts = []
+        for tag, attrs in self.open_tags:
+            attr_str = ""
+            if attrs:
+                attr_str = " " + " ".join(f'{k}="{v}"' for k, v in attrs)
+            parts.append(f"<{tag}{attr_str}>")
+        return "".join(parts)
+
+    def get_closing_tags_html(self):
+        return "".join(f"</{tag}>" for tag, _ in reversed(self.open_tags))
+
+
+def split_pre_block(pre_block: str, max_length) -> list[str]:
+    """
+    Splits long HTML-formatted text into chunks suitable for sending via Telegram,
+    preserving valid HTML tag nesting and handling <pre>/<code> blocks separately.
+
+    Args:
+        text (str): The input HTML-formatted string.
+        trim_leading_newlines (bool): If True, removes leading newline characters (`\\n`)
+            from each resulting chunk before sending. This is useful to avoid
+            unnecessary blank space at the beginning of messages in Telegram.
+
+    Returns:
+        list[str]: A list of HTML-formatted message chunks, each within Telegram's length limit.
+    """
+
+    # language-aware: <pre><code class="language-python">...</code></pre>
+    match = re.match(r"<pre><code(.*?)>(.*)</code></pre>", pre_block, re.DOTALL)
+    if match:
+        attr, content = match.groups()
+        lines = content.splitlines(keepends=True)
+        chunks, buf = [], ""
+        overhead = len(f"<pre><code{attr}></code></pre>")
+        for line in lines:
+            if len(buf) + len(line) + overhead > max_length:
+                chunks.append(f"<pre><code{attr}>{buf}</code></pre>")
+                buf = ""
+            buf += line
+        if buf:
+            chunks.append(f"<pre><code{attr}>{buf}</code></pre>")
+        return chunks
+    else:
+        # regular <pre>...</pre>
+        inner = pre_block[5:-6]
+        lines = inner.splitlines(keepends=True)
+        chunks, buf = [], ""
+        overhead = len('<pre></pre>')
+        for line in lines:
+            if len(buf) + len(line) + overhead > max_length:
+                chunks.append(f"<pre>{buf}</pre>")
+                buf = ""
+            buf += line
+        if buf:
+            chunks.append(f"<pre>{buf}</pre>")
+        return chunks
+
+
+def _is_only_tags(block: str) -> bool:
+    return bool(re.fullmatch(r'(?:\s*<[^>]+>\s*)+', block))
+
+
+def _effective_length(content: str) -> int:
+    tracker = HTMLTagTracker()
+    tracker.feed(content)
+    return len(tracker.get_open_tags_html()) + len(content) + len(tracker.get_closing_tags_html())
+
+
+def split_html_for_telegram(text: str, trim_empty_leading_lines: bool = False, max_length: int = MAX_LENGTH) -> list[str]:
+    """Split long HTML-formatted text into Telegram-compatible chunks.
+
+    Parameters
+    ----------
+    text: str
+        Input HTML text.
+    trim_empty_leading_lines: bool, optional
+        If True, removes `\n` sybmols from start of chunks.
+    max_length: int, optional
+        Maximum allowed length for a single chunk (must be >= ``MIN_LENGTH = 500``).
+        Default = 4096 (symbols)
+
+    Returns
+    -------
+    list[str]
+        List of HTML chunks.
+    """
+
+    if max_length < MIN_LENGTH:
+        raise ValueError("max_length should be at least %d" % MIN_LENGTH)
+
+    pattern = re.compile(r"(<pre>.*?</pre>|<pre><code.*?</code></pre>)", re.DOTALL)
+    parts = pattern.split(text)
+
+    chunks: list[str] = []
+    prefix = ""
+    current = ""
+    whitespace_re = re.compile(r"(\\s+)")
+    tag_re = re.compile(r"(<[^>]+>)")
+
+    def finalize():
+        nonlocal current, prefix
+        tracker = HTMLTagTracker()
+        tracker.feed(prefix + current)
+        chunk = prefix + current + tracker.get_closing_tags_html()
+        chunks.append(chunk)
+        prefix = tracker.get_open_tags_html()
+        current = ""
+
+    def append_piece(piece: str):
+        nonlocal current, prefix
+
+        def split_on_whitespace(chunk: str) -> list[str] | None:
+            parts = [part for part in whitespace_re.split(chunk) if part]
+            if len(parts) <= 1:
+                return None
+            return parts
+
+        def split_on_tags(chunk: str) -> list[str] | None:
+            parts = [part for part in tag_re.split(chunk) if part]
+            if len(parts) <= 1:
+                return None
+            return parts
+
+        def fittable_prefix_length(chunk: str) -> int:
+            low, high = 1, len(chunk)
+            best = 0
+            while low <= high:
+                mid = (low + high) // 2
+                candidate = chunk[:mid]
+                if _effective_length(prefix + current + candidate) <= max_length:
+                    best = mid
+                    low = mid + 1
+                else:
+                    high = mid - 1
+            return best
+
+        while piece:
+            if _effective_length(prefix + current + piece) <= max_length:
+                current += piece
+                return
+
+            if len(piece) > max_length:
+                if _is_only_tags(piece):
+                    raise ValueError("block contains only html tags")
+                splitted = split_on_whitespace(piece)
+                if splitted:
+                    for part in splitted:
+                        append_piece(part)
+                    return
+                tag_split = split_on_tags(piece)
+                if tag_split:
+                    for part in tag_split:
+                        append_piece(part)
+                    return
+            elif current:
+                finalize()
+                continue
+            else:
+                splitted = split_on_whitespace(piece)
+                if splitted:
+                    for part in splitted:
+                        append_piece(part)
+                    return
+                tag_split = split_on_tags(piece)
+                if tag_split:
+                    for part in tag_split:
+                        append_piece(part)
+                    return
+
+            fitted = fittable_prefix_length(piece)
+            if fitted == 0:
+                if current:
+                    finalize()
+                    continue
+                raise ValueError("unable to split content within max_length")
+
+            current += piece[:fitted]
+            piece = piece[fitted:]
+
+            if piece:
+                finalize()
+
+
+    for part in parts:
+        if not part:
+            continue
+        if part.startswith("<pre>") or part.startswith("<pre><code"):
+            pre_chunks = split_pre_block(part, max_length=max_length)
+            for pc in pre_chunks:
+                append_piece(pc)
+            continue
+        blocks = re.split(r"(\n\s*\n|<br\s*/?>|\n)", part)
+        for block in blocks:
+            if block:
+                append_piece(block)
+
+    if current:
+        finalize()
+
+    merged: list[str] = []
+    buf = ""
+    for chunk in chunks:
+        if len(buf) + len(chunk) <= max_length:
+            buf += chunk
+        else:
+            if buf:
+                merged.append(buf)
+            buf = chunk.lstrip("\n") if trim_empty_leading_lines and merged else chunk
+    if buf:
+        merged.append(buf.lstrip("\n") if trim_empty_leading_lines and merged else buf)
+
+    return merged

{chatgpt_md_converter-0.3.7 → chatgpt_md_converter-0.3.8}/chatgpt_md_converter.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: chatgpt_md_converter
-Version: 0.3.7
+Version: 0.3.8
 Summary: A package for converting markdown to HTML for chat Telegram bots
 Home-page: https://github.com/botfather-dev/formatter-chatgpt-telegram
 Author: Kostiantyn Kriuchkov

{chatgpt_md_converter-0.3.7 → chatgpt_md_converter-0.3.8}/chatgpt_md_converter.egg-info/SOURCES.txt

@@ -1,4 +1,5 @@
 LICENSE
+README.md
 setup.py
 chatgpt_md_converter/__init__.py
 chatgpt_md_converter/converters.py

{chatgpt_md_converter-0.3.7 → chatgpt_md_converter-0.3.8}/setup.py

@@ -2,11 +2,11 @@ from setuptools import setup
 
 setup(
     name="chatgpt_md_converter",
-    version="0.3.7",
+    version="0.3.8",
     author="Kostiantyn Kriuchkov",
     author_email="latand666@gmail.com",
     description="A package for converting markdown to HTML for chat Telegram bots",
-    long_description=open("README.MD").read(),
+    long_description=open("README.md").read(),
     long_description_content_type="text/markdown",
     url="https://github.com/botfather-dev/formatter-chatgpt-telegram",
     classifiers=[

{chatgpt_md_converter-0.3.7 → chatgpt_md_converter-0.3.8}/tests/test_parser.py

@@ -915,3 +915,23 @@ print("hello world ```"')
       print(f"Expected was: \n\n{expected_output}\n\n")
       print(f"output was: \n\n{output}")
     assert output == expected_output, show_output()
+
+def test_some_new():
+    input_text = """
+``````markdown
+`````
+````python
+print("hello world ```")
+```
+""" # Markdown code wasn't closed
+
+    expected_output = """<pre><code class=\"language-markdown\">`````
+````python
+print("hello world ```")
+```
+</code></pre>""" # But after closed correctly
+    output = telegram_format(input_text)
+    def show_output():
+      print(f"Expected was: \n\n{expected_output}\n\n")
+      print(f"output was: \n\n{output}")
+    assert output == expected_output, show_output()

chatgpt_md_converter-0.3.8/tests/test_splitter.py

@@ -0,0 +1,103 @@
+import re
+
+import pytest
+
+from chatgpt_md_converter.html_splitter import (MIN_LENGTH,
+                                                split_html_for_telegram)
+
+from . import html_examples
+
+
+def test_html_splitter():
+    chunks = split_html_for_telegram(html_examples.input_text)
+    valid_chunks = [
+        html_examples.valid_chunk_1,
+        html_examples.valid_chunk_2,
+        html_examples.valid_chunk_3,
+    ]
+    for index, chunk in enumerate(chunks):
+        assert chunk == valid_chunks[index], (
+            f"expected: \n\n{valid_chunks[index]} \n\n got: \n\n{chunk}"
+        )
+
+def test_html_splitter__remove_leading_brakes():
+    chunks = split_html_for_telegram(html_examples.input_text, trim_empty_leading_lines=True)
+    valid_chunks = [
+        html_examples.valid_chunk_1,
+        html_examples.valid_chunk_2,
+        html_examples.valid_chunk_3_remove_leading_brakes,
+    ]
+    for index, chunk in enumerate(chunks):
+        assert chunk == valid_chunks[index], (
+            f"expected: \n\n{valid_chunks[index]} \n\n got: \n\n{chunk}"
+        )
+
+def test_html_splitter_max_length_550():
+    chunks = split_html_for_telegram(
+        html_examples.long_code_input, max_length=550, trim_empty_leading_lines=True
+    )
+
+    def load_expected_chunks_550():
+        raw = re.split(r"END\n?", html_examples.expected_550)
+        chunks = []
+        for part in raw:
+            if not part.strip():
+                continue
+            lines = part.splitlines()
+            chunks.append("\n".join(lines[1:]))
+        return chunks
+
+    valid_chunks = load_expected_chunks_550()
+    for index, chunk in enumerate(chunks):
+        assert chunk == valid_chunks[index], (
+            f"expected: \n\n{valid_chunks[index]} \n\n got: \n\n{chunk}"
+        )
+        assert len(chunk) <= 550
+
+def test_split_html_respects_max_length_by_words():
+    text = "<b>" + "<i>word</i> " * 100 + "</b>"
+    chunks = split_html_for_telegram(text, max_length=550)
+    assert len(chunks) > 1
+    for chunk in chunks:
+        assert len(chunk) <= 550
+        assert chunk.startswith("<b>")
+        assert chunk.endswith("</b>")
+        assert chunk.count("<i>") == chunk.count("</i>")
+
+
+def test_split_html_only_tags_raises():
+    text = "<b></b>" * 200
+    with pytest.raises(ValueError):
+        split_html_for_telegram(text, max_length=600)
+
+
+def test_split_html_min_length_enforced():
+    with pytest.raises(ValueError):
+        split_html_for_telegram("hello", max_length=MIN_LENGTH - 1)
+
+
+def test_split_html_long_word_exceeds_limit():
+    text = "a" * 600
+    chunks = split_html_for_telegram(text, max_length=550)
+    assert chunks == ["a" * 550, "a" * 50]
+
+
+LONG_TEXT = "<b><i>" + "word " * 96 + "word!" + "</i></b>"
+
+SHORT_TEXT = "<u>" + "another " * 9 + "another" + "</u>"
+
+
+def test_split_html_keeps_newline_without_trim():
+    text = LONG_TEXT + "\n\n" + SHORT_TEXT
+    chunks = split_html_for_telegram(text, max_length=500, trim_empty_leading_lines=False)
+    assert chunks[0] == LONG_TEXT
+    assert chunks[1].startswith("\n")
+    assert chunks[1].endswith(SHORT_TEXT)
+    assert chunks[1].lstrip("\n").startswith("<u>")
+    assert chunks[1].lstrip("\n").endswith("</u>")
+
+
+def test_split_html_trims_leading_newline_on_new_chunk():
+    text = LONG_TEXT + "\n\n" + SHORT_TEXT
+    chunks = split_html_for_telegram(text, max_length=500, trim_empty_leading_lines=True)
+    assert chunks == [LONG_TEXT, SHORT_TEXT]

chatgpt_md_converter-0.3.7/chatgpt_md_converter/__init__.py

@@ -1,3 +0,0 @@
-from .telegram_formatter import telegram_format
-
-__all__ = ["telegram_format"]

chatgpt_md_converter-0.3.7/chatgpt_md_converter/html_splitter.py

@@ -1,114 +0,0 @@
-import re
-from html.parser import HTMLParser
-
-MAX_LENGTH = 4096
-
-
-class HTMLTagTracker(HTMLParser):
-    def __init__(self):
-        super().__init__()
-        self.open_tags = []
-
-    def handle_starttag(self, tag, attrs):
-        # saving tags
-        if tag in ("b", "i", "u", "s", "code", "pre", "a", "span", "blockquote"):
-            self.open_tags.append((tag, attrs))
-
-    def handle_endtag(self, tag):
-        for i in range(len(self.open_tags) - 1, -1, -1):
-            if self.open_tags[i][0] == tag:
-                del self.open_tags[i]
-                break
-
-    def get_open_tags_html(self):
-        parts = []
-        for tag, attrs in self.open_tags:
-            attr_str = ""
-            if attrs:
-                attr_str = " " + " ".join(f'{k}="{v}"' for k, v in attrs)
-            parts.append(f"<{tag}{attr_str}>")
-        return "".join(parts)
-
-    def get_closing_tags_html(self):
-        return "".join(f"</{tag}>" for tag, _ in reversed(self.open_tags))
-
-
-def split_pre_block(pre_block: str) -> list[str]:
-    # language-aware: <pre><code class="language-python">...</code></pre>
-    match = re.match(r"<pre><code(.*?)>(.*)</code></pre>", pre_block, re.DOTALL)
-    if match:
-        attr, content = match.groups()
-        lines = content.splitlines(keepends=True)
-        chunks, buf = [], ""
-        for line in lines:
-            if len(buf) + len(line) + len('<pre><code></code></pre>') > MAX_LENGTH:
-                chunks.append(f"<pre><code{attr}>{buf}</code></pre>")
-                buf = ""
-            buf += line
-        if buf:
-            chunks.append(f"<pre><code{attr}>{buf}</code></pre>")
-        return chunks
-    else:
-        # regular <pre>...</pre>
-        inner = pre_block[5:-6]
-        lines = inner.splitlines(keepends=True)
-        chunks, buf = [], ""
-        for line in lines:
-            if len(buf) + len(line) + len('<pre></pre>') > MAX_LENGTH:
-                chunks.append(f"<pre>{buf}</pre>")
-                buf = ""
-            buf += line
-        if buf:
-            chunks.append(f"<pre>{buf}</pre>")
-        return chunks
-
-
-def split_html_for_telegram(text: str) -> list[str]:
-    chunks = []
-    pattern = re.compile(r"(<pre>.*?</pre>|<pre><code.*?</code></pre>)", re.DOTALL)
-    parts = pattern.split(text)
-
-    for part in parts:
-        if not part:
-            continue
-        if part.startswith("<pre>") or part.startswith("<pre><code"):
-            pre_chunks = split_pre_block(part)
-            chunks.extend(pre_chunks)
-        else:
-            # breaking down regular HTML
-            tracker = HTMLTagTracker()
-            current = ""
-            blocks = re.split(r"(\n\s*\n|<br\s*/?>|\n)", part)
-            for block in blocks:
-                prospective = current + block
-                if len(prospective) > MAX_LENGTH:
-                    tracker.feed(current)
-                    open_tags = tracker.get_open_tags_html()
-                    close_tags = tracker.get_closing_tags_html()
-                    chunks.append(open_tags + current + close_tags)
-                    current = block
-                    tracker = HTMLTagTracker()
-                else:
-                    current = prospective
-            if current.strip():
-                tracker.feed(current)
-                open_tags = tracker.get_open_tags_html()
-                close_tags = tracker.get_closing_tags_html()
-                chunks.append(open_tags + current + close_tags)
-
-    # post-unification: combine chunks if they don't exceed the limit in total
-    merged_chunks = []
-    buf = ""
-    for chunk in chunks:
-        # chunk = chunk.lstrip("\n")  # removing leading line breaks
-
-        if len(buf) + len(chunk) <= MAX_LENGTH:
-            buf += chunk
-        else:
-            if buf:
-                merged_chunks.append(buf)
-            buf = chunk
-    if buf:
-        merged_chunks.append(buf)
-
-    return merged_chunks

chatgpt_md_converter-0.3.7/tests/test_splitter.py

@@ -1,436 +0,0 @@
-from chatgpt_md_converter.html_splitter import split_html_for_telegram
-
-input_text ="""
-Absolutely! Here’s a Markdown-formatted message exceeding 5,000 characters, exploring <b>The History and Impact of Computer Programming</b>. (You can verify the character count using any online tool.)
-
----
-
-<b>The History and Impact of Computer Programming</b>
-
-<i>“The computer was born to solve problems that did not exist before.”</i>  
-— Bill Gates
-
----
-
-<b>Table of Contents</b>
-
-1. <a href="#introduction">Introduction</a>  
-2. <a href="#ancient-beginnings-from-algorithms-to-machines">Ancient Beginnings: From Algorithms to Machines</a>  
-    • <a href="#al-khwarizmi-and-the-algorithm">Al-Khwarizmi and the Algorithm</a>  
-    • <a href="#the-analytical-engine">The Analytical Engine</a>  
-    • <a href="#punch-cards-and-the-jacquard-loom">Punch Cards and the Jacquard Loom</a>  
-3. <a href="#20th-century-the-birth-of-modern-programming">20th Century: The Birth of Modern Programming</a>  
-    • <a href="#eniac-and-early-programmers">ENIAC and Early Programmers</a>  
-    • <a href="#assembly-language-and-early-high-level-languages">Assembly Language and Early High-level Languages</a>  
-    • <a href="#cobol-fortran-and-the-expansion">COBOL, FORTRAN, and the Expansion</a>  
-4. <a href="#modern-era-languages-paradigms-and-the-internet">Modern Era: Languages, Paradigms, and the Internet</a>  
-    • <a href="#object-oriented-programming">Object-Oriented Programming</a>  
-    • <a href="#internet-and-open-source">Internet and Open Source</a>  
-    • <a href="#mobile-and-cloud-computing">Mobile and Cloud Computing</a>    
-5. <a href="#programmings-societal-impact">Programming’s Societal Impact</a>  
-6. <a href="#ethics-challenges-and-the-future">Ethics, Challenges, and the Future</a>  
-7. <a href="#conclusion">Conclusion</a>  
-8. <a href="#useful-resources">Useful Resources</a>  
-
----
-
-<b>Introduction</b>
-
-Computer programming is the science and art of giving computers instructions to perform specific tasks. Today, it's impossible to imagine a world without software: from banking systems and mobile applications to traffic lights and airplanes, programming is everywhere.
-
-But how did programming begin, and what has it become today? This document explores the journey of programming, from ancient mathematical roots to the future of artificial intelligence.
-
----
-
-<b>Ancient Beginnings: From Algorithms to Machines</b>
-
-<b>Al-Khwarizmi and the Algorithm</b>
-
-The term "<b>algorithm</b>" (the foundation of programming) comes from Abu Abdullah Muhammad ibn Musa Al-Khwarizmi, a 9th-century Persian mathematician. His works on systematic procedures laid the groundwork for computational thinking.
-
-<b>The Analytical Engine</b>
-
-In the 19th century, <b>Charles Babbage</b> designed the Analytical Engine, a mechanical general-purpose computer. Though never built in his lifetime, it could—in theory—read instructions from punched cards.
-
-<b>Ada Lovelace</b>, Babbage's collaborator, is often called the first computer programmer. She wrote notes describing algorithms (in essence, programs) for the Analytical Engine to compute Bernoulli numbers.
-
-<blockquote>"That brain of mine is something more than merely mortal; as time will show."
-– Ada Lovelace</blockquote>
-
-<b>Punch Cards and the Jacquard Loom</b>
-
-The concept of programming a machine with punched cards predates computers. <b>Joseph Marie Jacquard</b> invented a loom in 1804 that used punched cards to control patterns in woven fabric—an early example of machine automation.
-
----
-
-<b>20th Century: The Birth of Modern Programming</b>
-
-<b>ENIAC and Early Programmers</b>
-
-ENIAC (Electronic Numerical Integrator and Computer), completed in 1945, is often cited as the first electronic general-purpose computer.
-
-Early programming was entirely manual and physically laborious—think patch cables and switches!
-
-Notably, many of the earliest programmers were women, such as <b>Kathleen McNulty</b>, <b>Jean Jennings</b>, and <b>Grace Hopper</b>.
-
-<b>Assembly Language and Early High-level Languages</b>
-
-The problem of complexity led to <b>assembly languages</b>, where mnemonics like <code>MOV</code> and <code>ADD</code> replaced binary codes. Programming became more accessible, but code was still hardware-specific.
-
-The 1950s saw the creation of:
-
-• <b>FORTRAN</b> (FORmula TRANslation) for scientific computation
-• <b>COBOL</b> (COmmon Business-Oriented Language) for business applications
-
-<b>Code Example: Hello World in COBOL</b>
-<pre><code class="language-cobol">IDENTIFICATION DIVISION.
-PROGRAM-ID. HELLO-WORLD.
-PROCEDURE DIVISION.
-    DISPLAY "Hello, World!".
-STOP RUN.
-</code></pre>
-
-<b>COBOL, FORTRAN, and the Expansion</b>
-
-With the advent of high-level languages, programming became less about circuitry and more about solving problems. Standardized languages allowed code to run on multiple machines.
-
-Other languages soon emerged:
-
-• <b>LISP</b> (for AI research)
-• <b>ALGOL</b> (basis for many future languages)
-• <b>BASIC</b> (for beginners and education)
-
----
-
-<b>Modern Era: Languages, Paradigms, and the Internet</b>
-
-<b>Object-Oriented Programming</b>
-
-The 1970s and 1980s introduced <b>object-oriented programming</b> (OOP), where data and behavior are bundled together. The most influential languages here include:
-
-• <b>Smalltalk</b>: pioneered OOP concepts
-• <b>C++</b>: combined OOP with the efficiency of C
-• <b>Java</b>: “Write Once, Run Anywhere” with the Java Virtual Machine
-
-<b>Code Example: Simple Class in Java</b>
-<pre><code class="language-java">public class HelloWorld {
-    public static void main(String[] args) {
-        System.out.println("Hello, World!");
-    }
-}
-</code></pre>
-
-<b>Internet and Open Source</b>
-
-The rise of the World Wide Web transformed programming. JavaScript, PHP, and Python became staples for Internet-connected software.
-
-<b>Open source</b> projects like Linux, Apache, and MySQL changed collaboration forever—developers worldwide could contribute to shared codebases.
-
-| Year | Technology  | Impact                                  |
-|------|-------------|-----------------------------------------|
-| 1991 | Linux       | Free, open-source operating systems     |
-| 1995 | JavaScript  | Interactive web applications            |
-| 2001 | Wikipedia   | Collaborative knowledge base            |
-
-<b>Mobile and Cloud Computing</b>
-
-Smartphones spawned new languages and frameworks (Swift, Kotlin, React Native).
-
-<b>Cloud computing</b> and <b>APIs</b> mean programs can collaborate on a global scale, in real-time.
-
----
-
-<b>Programming’s Societal Impact</b>
-
-Programming is reshaping society in profound ways:
-
-• <b>Healthcare</b>: Medical imaging, diagnostics, record management
-• <b>Finance</b>: Online banking, stock trading algorithms
-• <b>Entertainment</b>: Gaming, music streaming, social networks
-• <b>Education</b>: E-learning, interactive simulations, content platforms
-• <b>Transportation</b>: Navigation, ride-sharing apps, autonomous vehicles
-• <b>Science</b>: Processing large datasets, running complex simulations
-
-<blockquote>“Software is eating the world.”
-– Marc Andreessen</blockquote>
-
-<b>Programming Jobs In Demand</b>
-
-<pre><code class="language-mermaid">pie
-    title Programming Job Market (2024)
-    "Web Development" : 31
-    "Data Science" : 22
-    "Mobile Development" : 12
-    "Embedded Systems" : 8
-    "Cybersecurity" : 9
-    "Other": 18
-</code></pre>
-
-<b>Note:</b> Mermaid diagrams require compatible renderers (e.g., GitHub, Obsidian).
-
----
-
-<b>Ethics, Challenges, and the Future</b>
-
-With great power comes responsibility. Programmers face new challenges:
-
-• <b>Bias in Algorithms:</b> Unintentional biases in data can lead to unfair outcomes (e.g., in hiring software or criminal justice prediction).
-• <b>Privacy:</b> Handling personal data securely is more critical than ever.
-• <b>Safety:</b> In fields like self-driving cars or medical devices, software bugs can have real-world consequences.
-• <b>Sustainability:</b> Software should be efficient, minimizing environmental impact in data centers.
-
-<b>Emerging Trends:</b>
-
-• <b>Artificial Intelligence:</b> Programs that learn, adapt, and sometimes surprise their creators.
-• <b>Quantum Computing:</b> New paradigms for solving currently intractable problems.
-• <b>No-Code/Low-Code:</b> Empowering more people to harness computational power.
-
----
-
-<b>Conclusion</b>
-
-From mechanical looms to neural networks, programming continues to redefine what’s possible. It’s not just for professional engineers: millions of people use programming as a tool for art, science, business, and personal growth.
-
-<b>Everyone can learn to code.</b> It might change your life—or even the world.
-
-<blockquote>"Any sufficiently advanced technology is indistinguishable from magic."
-— Arthur C. Clarke</blockquote>
-
----
-
-<b>Useful Resources</b>
-
-• <a href="https://www-cs-faculty.stanford.edu/~knuth/taocp.html">The Art of Computer Programming (Donald Knuth)</a>
-• <a href="https://www.khanacademy.org/computing/computer-programming">Khan Academy Computer Programming</a>
-• <a href="https://www.w3schools.com/">W3Schools Online Tutorials</a>
-• <a href="https://www.freecodecamp.org/">freeCodeCamp</a>
-• <a href="https://stackoverflow.com/">Stack Overflow</a>
-• <a href="https://guides.github.com/">GitHub Guides</a>
-
----
-
-<i>Thank you for reading! If you’re inspired to begin your coding journey, there has never been a better time to start.</i>
-
----"""
-
-valid_chunk_1 = """
-Absolutely! Here’s a Markdown-formatted message exceeding 5,000 characters, exploring <b>The History and Impact of Computer Programming</b>. (You can verify the character count using any online tool.)
-
----
-
-<b>The History and Impact of Computer Programming</b>
-
-<i>“The computer was born to solve problems that did not exist before.”</i>  
-— Bill Gates
-
----
-
-<b>Table of Contents</b>
-
-1. <a href="#introduction">Introduction</a>  
-2. <a href="#ancient-beginnings-from-algorithms-to-machines">Ancient Beginnings: From Algorithms to Machines</a>  
-    • <a href="#al-khwarizmi-and-the-algorithm">Al-Khwarizmi and the Algorithm</a>  
-    • <a href="#the-analytical-engine">The Analytical Engine</a>  
-    • <a href="#punch-cards-and-the-jacquard-loom">Punch Cards and the Jacquard Loom</a>  
-3. <a href="#20th-century-the-birth-of-modern-programming">20th Century: The Birth of Modern Programming</a>  
-    • <a href="#eniac-and-early-programmers">ENIAC and Early Programmers</a>  
-    • <a href="#assembly-language-and-early-high-level-languages">Assembly Language and Early High-level Languages</a>  
-    • <a href="#cobol-fortran-and-the-expansion">COBOL, FORTRAN, and the Expansion</a>  
-4. <a href="#modern-era-languages-paradigms-and-the-internet">Modern Era: Languages, Paradigms, and the Internet</a>  
-    • <a href="#object-oriented-programming">Object-Oriented Programming</a>  
-    • <a href="#internet-and-open-source">Internet and Open Source</a>  
-    • <a href="#mobile-and-cloud-computing">Mobile and Cloud Computing</a>    
-5. <a href="#programmings-societal-impact">Programming’s Societal Impact</a>  
-6. <a href="#ethics-challenges-and-the-future">Ethics, Challenges, and the Future</a>  
-7. <a href="#conclusion">Conclusion</a>  
-8. <a href="#useful-resources">Useful Resources</a>  
-
----
-
-<b>Introduction</b>
-
-Computer programming is the science and art of giving computers instructions to perform specific tasks. Today, it's impossible to imagine a world without software: from banking systems and mobile applications to traffic lights and airplanes, programming is everywhere.
-
-But how did programming begin, and what has it become today? This document explores the journey of programming, from ancient mathematical roots to the future of artificial intelligence.
-
----
-
-<b>Ancient Beginnings: From Algorithms to Machines</b>
-
-<b>Al-Khwarizmi and the Algorithm</b>
-
-The term "<b>algorithm</b>" (the foundation of programming) comes from Abu Abdullah Muhammad ibn Musa Al-Khwarizmi, a 9th-century Persian mathematician. His works on systematic procedures laid the groundwork for computational thinking.
-
-<b>The Analytical Engine</b>
-
-In the 19th century, <b>Charles Babbage</b> designed the Analytical Engine, a mechanical general-purpose computer. Though never built in his lifetime, it could—in theory—read instructions from punched cards.
-
-<b>Ada Lovelace</b>, Babbage's collaborator, is often called the first computer programmer. She wrote notes describing algorithms (in essence, programs) for the Analytical Engine to compute Bernoulli numbers.
-
-<blockquote>"That brain of mine is something more than merely mortal; as time will show."
-– Ada Lovelace</blockquote>
-
-<b>Punch Cards and the Jacquard Loom</b>
-
-The concept of programming a machine with punched cards predates computers. <b>Joseph Marie Jacquard</b> invented a loom in 1804 that used punched cards to control patterns in woven fabric—an early example of machine automation.
-
----
-
-<b>20th Century: The Birth of Modern Programming</b>
-
-<b>ENIAC and Early Programmers</b>
-
-ENIAC (Electronic Numerical Integrator and Computer), completed in 1945, is often cited as the first electronic general-purpose computer.
-
-Early programming was entirely manual and physically laborious—think patch cables and switches!
-
-Notably, many of the earliest programmers were women, such as <b>Kathleen McNulty</b>, <b>Jean Jennings</b>, and <b>Grace Hopper</b>.
-
-<b>Assembly Language and Early High-level Languages</b>
-
-"""
-
-valid_chunk_2 = """The problem of complexity led to <b>assembly languages</b>, where mnemonics like <code>MOV</code> and <code>ADD</code> replaced binary codes. Programming became more accessible, but code was still hardware-specific.
-
-The 1950s saw the creation of:
-
-• <b>FORTRAN</b> (FORmula TRANslation) for scientific computation
-• <b>COBOL</b> (COmmon Business-Oriented Language) for business applications
-
-<b>Code Example: Hello World in COBOL</b>
-<pre><code class="language-cobol">IDENTIFICATION DIVISION.
-PROGRAM-ID. HELLO-WORLD.
-PROCEDURE DIVISION.
-    DISPLAY "Hello, World!".
-STOP RUN.
-</code></pre>
-
-<b>COBOL, FORTRAN, and the Expansion</b>
-
-With the advent of high-level languages, programming became less about circuitry and more about solving problems. Standardized languages allowed code to run on multiple machines.
-
-Other languages soon emerged:
-
-• <b>LISP</b> (for AI research)
-• <b>ALGOL</b> (basis for many future languages)
-• <b>BASIC</b> (for beginners and education)
-
----
-
-<b>Modern Era: Languages, Paradigms, and the Internet</b>
-
-<b>Object-Oriented Programming</b>
-
-The 1970s and 1980s introduced <b>object-oriented programming</b> (OOP), where data and behavior are bundled together. The most influential languages here include:
-
-• <b>Smalltalk</b>: pioneered OOP concepts
-• <b>C++</b>: combined OOP with the efficiency of C
-• <b>Java</b>: “Write Once, Run Anywhere” with the Java Virtual Machine
-
-<b>Code Example: Simple Class in Java</b>
-<pre><code class="language-java">public class HelloWorld {
-    public static void main(String[] args) {
-        System.out.println("Hello, World!");
-    }
-}
-</code></pre>
-
-<b>Internet and Open Source</b>
-
-The rise of the World Wide Web transformed programming. JavaScript, PHP, and Python became staples for Internet-connected software.
-
-<b>Open source</b> projects like Linux, Apache, and MySQL changed collaboration forever—developers worldwide could contribute to shared codebases.
-
-| Year | Technology  | Impact                                  |
-|------|-------------|-----------------------------------------|
-| 1991 | Linux       | Free, open-source operating systems     |
-| 1995 | JavaScript  | Interactive web applications            |
-| 2001 | Wikipedia   | Collaborative knowledge base            |
-
-<b>Mobile and Cloud Computing</b>
-
-Smartphones spawned new languages and frameworks (Swift, Kotlin, React Native).
-
-<b>Cloud computing</b> and <b>APIs</b> mean programs can collaborate on a global scale, in real-time.
-
----
-
-<b>Programming’s Societal Impact</b>
-
-Programming is reshaping society in profound ways:
-
-• <b>Healthcare</b>: Medical imaging, diagnostics, record management
-• <b>Finance</b>: Online banking, stock trading algorithms
-• <b>Entertainment</b>: Gaming, music streaming, social networks
-• <b>Education</b>: E-learning, interactive simulations, content platforms
-• <b>Transportation</b>: Navigation, ride-sharing apps, autonomous vehicles
-• <b>Science</b>: Processing large datasets, running complex simulations
-
-<blockquote>“Software is eating the world.”
-– Marc Andreessen</blockquote>
-
-<b>Programming Jobs In Demand</b>
-
-<pre><code class="language-mermaid">pie
-    title Programming Job Market (2024)
-    "Web Development" : 31
-    "Data Science" : 22
-    "Mobile Development" : 12
-    "Embedded Systems" : 8
-    "Cybersecurity" : 9
-    "Other": 18
-</code></pre>"""
-
-valid_chunk_3 = """
-
-<b>Note:</b> Mermaid diagrams require compatible renderers (e.g., GitHub, Obsidian).
-
----
-
-<b>Ethics, Challenges, and the Future</b>
-
-With great power comes responsibility. Programmers face new challenges:
-
-• <b>Bias in Algorithms:</b> Unintentional biases in data can lead to unfair outcomes (e.g., in hiring software or criminal justice prediction).
-• <b>Privacy:</b> Handling personal data securely is more critical than ever.
-• <b>Safety:</b> In fields like self-driving cars or medical devices, software bugs can have real-world consequences.
-• <b>Sustainability:</b> Software should be efficient, minimizing environmental impact in data centers.
-
-<b>Emerging Trends:</b>
-
-• <b>Artificial Intelligence:</b> Programs that learn, adapt, and sometimes surprise their creators.
-• <b>Quantum Computing:</b> New paradigms for solving currently intractable problems.
-• <b>No-Code/Low-Code:</b> Empowering more people to harness computational power.
-
----
-
-<b>Conclusion</b>
-
-From mechanical looms to neural networks, programming continues to redefine what’s possible. It’s not just for professional engineers: millions of people use programming as a tool for art, science, business, and personal growth.
-
-<b>Everyone can learn to code.</b> It might change your life—or even the world.
-
-<blockquote>"Any sufficiently advanced technology is indistinguishable from magic."
-— Arthur C. Clarke</blockquote>
-
----
-
-<b>Useful Resources</b>
-
-• <a href="https://www-cs-faculty.stanford.edu/~knuth/taocp.html">The Art of Computer Programming (Donald Knuth)</a>
-• <a href="https://www.khanacademy.org/computing/computer-programming">Khan Academy Computer Programming</a>
-• <a href="https://www.w3schools.com/">W3Schools Online Tutorials</a>
-• <a href="https://www.freecodecamp.org/">freeCodeCamp</a>
-• <a href="https://stackoverflow.com/">Stack Overflow</a>
-• <a href="https://guides.github.com/">GitHub Guides</a>
-
----
-
-<i>Thank you for reading! If you’re inspired to begin your coding journey, there has never been a better time to start.</i>
-
----"""
-
-def test_splitter_test():
-  chunks = split_html_for_telegram(input_text)
-  valid_chunks = [valid_chunk_1, valid_chunk_2, valid_chunk_3]
-  assert chunks == valid_chunks