streamdown 0.26.0__tar.gz → 0.28.0__tar.gz

streamdown-0.26.0/tools/deploy.sh → streamdown-0.28.0/2q

@@ -10,7 +10,7 @@ pipy() {
         pip install --upgrade $i
     done
     python3 -m build .
-    twine upload dists/*$version*
+    twine upload dists/*${version}*
 }
 tag_update
 pipy

{streamdown-0.26.0 → streamdown-0.28.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: streamdown
-Version: 0.26.0
+Version: 0.28.0
 Summary: A streaming markdown renderer for modern terminals with syntax highlighting
 Project-URL: Homepage, https://github.com/kristopolous/Streamdown
 Project-URL: Bug Tracker, https://github.com/kristopolous/Streamdown/issues
@@ -25,6 +25,7 @@ Requires-Dist: pygments
 Requires-Dist: pylatexenc
 Requires-Dist: term-image
 Requires-Dist: toml
+Requires-Dist: wcwidth
 Description-Content-Type: text/markdown
 
 <p align="center">
@@ -38,9 +39,7 @@ Description-Content-Type: text/markdown
 
 Streamdown works with any streaming markdown such as [simonw's llm](https://github.com/simonw/llm) or even something basic like curl. 
 
-It is designed for AI and can be used to do parser based sophisticated pipelines and routing, cracking open various monolithic AI solutions to permit them to integrate. Think of it as output level routing at the semantic level.
-
-You can also just use it like a normal person.
+It just works. 
 
 It supports standard piping and files as arguments like any normal pager but can also run as a wrapper so you retain full keyboard interactivity. Arrow keys, control, alt, all still work.
 ```bash
@@ -48,16 +47,24 @@ $ pip install streamdown
 ```
 ![Streamdown is Amazing](https://github.com/user-attachments/assets/268cb340-78cc-4df0-a773-c5ac95eceeeb)
 
+## Fast and Realtime.
+Watch Streamdown run over a FIFO pipe through `tee` in tmux on an M4 using BitNet.  This is run straight. No clever unbuffering tricks. You can see the unstructured content on the right and the realtime Streamdown render on the left.
+
+[bitnet.webm](https://github.com/user-attachments/assets/62eb625e-82c4-462d-9991-ed681d6fbcd0)
+
+
 ### Provides clean copyable code for long code lines
 Other renderers inject line breaks when copying code that wraps around. Streamdown's better and now you are too!
+
+Set `PrettyBroken` and `PrettyPad` to False to make Streamdown ensure code is always cleanly mouse Copyable
 ![Handle That Mandle](https://github.com/user-attachments/assets/a27aa70c-f691-4796-84f0-c2eb18c7de23)
-**Tip**: You can make things prettier if you don't mind if this guarantee is broken. See the `PrettyBroken` flag below! (There's still 2 other convenient ways of getting code blocks out.)
+
 
 ### Supports images
 Here's kitty and alacritty. 
 ![doggie](https://github.com/user-attachments/assets/81c43983-68cd-40c1-b1d5-aa3a52004504)
 
-### Supports hyperlinks (OSC 8) and clipboard (OSC 52)
+### hyperlinks (OSC 8) and clipboard (OSC 52)
 The optional `Clipboard` feature puts the final codeblock into your clipboard. See below for details.
 
 [links.webm](https://github.com/user-attachments/assets/a5f71791-7c58-4183-ad3b-309f470c08a3)
@@ -70,7 +77,7 @@ This allows you to interactively debug  in a way that the agent doesn't just wan
 It takes about 2 minutes to set up and about 0.2s to use. Fast, fluid and free.
 ![screenquery](https://github.com/user-attachments/assets/517be4fe-6962-4e4c-b2f2-563471bc48d0)
 
-### ...It even supports CJK 
+### ...even CJK 
 Compare how streamdown wraps and spaces this tabular Chinese description of programming languages to other leading markdown renderers.
 
 Only one generates the text without truncation. 很美！
@@ -84,6 +91,10 @@ For instance, here is the [latex plugin](https://github.com/kristopolous/Streamd
 ![calc](https://github.com/user-attachments/assets/0b0027ca-8ef0-4b4a-b4ae-e36ff623a683)
 
 
+
+It is designed for AI and can be used to do parser based sophisticated pipelines and routing, cracking open various monolithic AI solutions to permit them to integrate. Think of it as output level routing at the semantic level.
+
+You can also just use it like a normal person.
 ## Configuration 
 
 It's located at `~/.config/streamdown/config.toml` (following the XDG Base Directory Specification). If this file does not exist upon first run, it will be created with default values. 
@@ -105,10 +116,10 @@ The default values are [at the beginning of the source](https://github.com/krist
 *   `Bright`: Multipliers for level 2 headers. 
 *   `Margin` (integer, default: `2`): The left and right indent for the output. 
 *   `Width` (integer, default: `0`): Along with the `Margin`, `Width` specifies the base width of the content, which when set to 0, means use the terminal width. See [#6](https://github.com/kristopolous/Streamdown/issues/6) for more details
-*   `PrettyPad` (boolean, default: `false`): Uses a unicode vertical pad trick to add a half height background to code blocks. This makes copy/paste have artifacts. See [#2](https://github.com/kristopolous/Streamdown/issues/2). I like it on. But that's just me
-*   `PrettyBroken` (boolean, default: `false`): This will break the copy/paste assurance above. The output is much prettier, but it's also broken. So it's pretty broken. Works nicely with PrettyPad.
+*   `PrettyPad` (boolean, default: `true`): Uses a unicode vertical pad trick to add a half height background to code blocks. This makes copy/paste have artifacts. See [#2](https://github.com/kristopolous/Streamdown/issues/2). I like it on. But that's just me
+*   `PrettyBroken` (boolean, default: `true`): This will break the copy/paste assurance above. The output is much prettier, but it's also broken. So it's pretty broken. Works nicely with PrettyPad.
 *   `ListIndent` (integer, default: `2`): This is the recursive indent for the list styles.
-*   `Syntax` (string, default `monokai`): This is the syntax [highlighting theme which come via pygments](https://pygments.org/styles/).
+*   `Syntax` (string, default `native`): This is the syntax [highlighting theme which come via pygments](https://pygments.org/styles/).
 
 Example:
 ```toml
@@ -168,7 +179,7 @@ optional arguments:
                         Set the logging level
   -b BASE, --base BASE  Set the hsv base: h,s,v
   -c CONFIG, --config CONFIG
-                        Use a custom config
+                        Use a custom config override
   -w WIDTH, --width WIDTH
                         Set the width WIDTH
   -e EXEC, --exec EXEC  Wrap a program EXEC for more 'proper' i/o handling

{streamdown-0.26.0 → streamdown-0.28.0}/README.md

@@ -9,9 +9,7 @@
 
 Streamdown works with any streaming markdown such as [simonw's llm](https://github.com/simonw/llm) or even something basic like curl. 
 
-It is designed for AI and can be used to do parser based sophisticated pipelines and routing, cracking open various monolithic AI solutions to permit them to integrate. Think of it as output level routing at the semantic level.
-
-You can also just use it like a normal person.
+It just works. 
 
 It supports standard piping and files as arguments like any normal pager but can also run as a wrapper so you retain full keyboard interactivity. Arrow keys, control, alt, all still work.
 ```bash
@@ -19,16 +17,24 @@ $ pip install streamdown
 ```
 ![Streamdown is Amazing](https://github.com/user-attachments/assets/268cb340-78cc-4df0-a773-c5ac95eceeeb)
 
+## Fast and Realtime.
+Watch Streamdown run over a FIFO pipe through `tee` in tmux on an M4 using BitNet.  This is run straight. No clever unbuffering tricks. You can see the unstructured content on the right and the realtime Streamdown render on the left.
+
+[bitnet.webm](https://github.com/user-attachments/assets/62eb625e-82c4-462d-9991-ed681d6fbcd0)
+
+
 ### Provides clean copyable code for long code lines
 Other renderers inject line breaks when copying code that wraps around. Streamdown's better and now you are too!
+
+Set `PrettyBroken` and `PrettyPad` to False to make Streamdown ensure code is always cleanly mouse Copyable
 ![Handle That Mandle](https://github.com/user-attachments/assets/a27aa70c-f691-4796-84f0-c2eb18c7de23)
-**Tip**: You can make things prettier if you don't mind if this guarantee is broken. See the `PrettyBroken` flag below! (There's still 2 other convenient ways of getting code blocks out.)
+
 
 ### Supports images
 Here's kitty and alacritty. 
 ![doggie](https://github.com/user-attachments/assets/81c43983-68cd-40c1-b1d5-aa3a52004504)
 
-### Supports hyperlinks (OSC 8) and clipboard (OSC 52)
+### hyperlinks (OSC 8) and clipboard (OSC 52)
 The optional `Clipboard` feature puts the final codeblock into your clipboard. See below for details.
 
 [links.webm](https://github.com/user-attachments/assets/a5f71791-7c58-4183-ad3b-309f470c08a3)
@@ -41,7 +47,7 @@ This allows you to interactively debug  in a way that the agent doesn't just wan
 It takes about 2 minutes to set up and about 0.2s to use. Fast, fluid and free.
 ![screenquery](https://github.com/user-attachments/assets/517be4fe-6962-4e4c-b2f2-563471bc48d0)
 
-### ...It even supports CJK 
+### ...even CJK 
 Compare how streamdown wraps and spaces this tabular Chinese description of programming languages to other leading markdown renderers.
 
 Only one generates the text without truncation. 很美！
@@ -55,6 +61,10 @@ For instance, here is the [latex plugin](https://github.com/kristopolous/Streamd
 ![calc](https://github.com/user-attachments/assets/0b0027ca-8ef0-4b4a-b4ae-e36ff623a683)
 
 
+
+It is designed for AI and can be used to do parser based sophisticated pipelines and routing, cracking open various monolithic AI solutions to permit them to integrate. Think of it as output level routing at the semantic level.
+
+You can also just use it like a normal person.
 ## Configuration 
 
 It's located at `~/.config/streamdown/config.toml` (following the XDG Base Directory Specification). If this file does not exist upon first run, it will be created with default values. 
@@ -76,10 +86,10 @@ The default values are [at the beginning of the source](https://github.com/krist
 *   `Bright`: Multipliers for level 2 headers. 
 *   `Margin` (integer, default: `2`): The left and right indent for the output. 
 *   `Width` (integer, default: `0`): Along with the `Margin`, `Width` specifies the base width of the content, which when set to 0, means use the terminal width. See [#6](https://github.com/kristopolous/Streamdown/issues/6) for more details
-*   `PrettyPad` (boolean, default: `false`): Uses a unicode vertical pad trick to add a half height background to code blocks. This makes copy/paste have artifacts. See [#2](https://github.com/kristopolous/Streamdown/issues/2). I like it on. But that's just me
-*   `PrettyBroken` (boolean, default: `false`): This will break the copy/paste assurance above. The output is much prettier, but it's also broken. So it's pretty broken. Works nicely with PrettyPad.
+*   `PrettyPad` (boolean, default: `true`): Uses a unicode vertical pad trick to add a half height background to code blocks. This makes copy/paste have artifacts. See [#2](https://github.com/kristopolous/Streamdown/issues/2). I like it on. But that's just me
+*   `PrettyBroken` (boolean, default: `true`): This will break the copy/paste assurance above. The output is much prettier, but it's also broken. So it's pretty broken. Works nicely with PrettyPad.
 *   `ListIndent` (integer, default: `2`): This is the recursive indent for the list styles.
-*   `Syntax` (string, default `monokai`): This is the syntax [highlighting theme which come via pygments](https://pygments.org/styles/).
+*   `Syntax` (string, default `native`): This is the syntax [highlighting theme which come via pygments](https://pygments.org/styles/).
 
 Example:
 ```toml
@@ -139,7 +149,7 @@ optional arguments:
                         Set the logging level
   -b BASE, --base BASE  Set the hsv base: h,s,v
   -c CONFIG, --config CONFIG
-                        Use a custom config
+                        Use a custom config override
   -w WIDTH, --width WIDTH
                         Set the width WIDTH
   -e EXEC, --exec EXEC  Wrap a program EXEC for more 'proper' i/o handling

{streamdown-0.26.0 → streamdown-0.28.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "streamdown"
-version = "0.26.0"
+version = "0.28.0"
 description = "A streaming markdown renderer for modern terminals with syntax highlighting"
 readme = "README.md"
 requires-python = ">=3.8"
@@ -30,6 +30,7 @@ dependencies = [
     "pygments",
     "appdirs",
     "toml",
+    "wcwidth",
     "pylatexenc",
     'term-image'
 ]

{streamdown-0.26.0 → streamdown-0.28.0}/requirements.txt

@@ -2,4 +2,5 @@ pygments
 pylatexenc
 appdirs
 term-image
+wcwidth
 toml

{streamdown-0.26.0 → streamdown-0.28.0}/streamdown/sd.py

@@ -6,6 +6,7 @@
 #     "pylatexenc",
 #     "appdirs",
 #     "term-image",
+#     "wcwidth",
 #     "toml"
 # ]
 # ///
@@ -25,13 +26,13 @@ import termios, tty
 import math
 import re
 import shutil
-import subprocess
 import traceback
 import colorsys
 import base64
 from io import BytesIO
 from term_image.image import from_file, from_url
 import pygments.util
+from wcwidth import wcwidth
 from functools import reduce
 from argparse import ArgumentParser
 from pygments import highlight
@@ -64,27 +65,34 @@ Mid     = { H = 1.00, S = 1.00, V = 0.50 }
 Symbol  = { H = 1.00, S = 1.00, V = 1.50 }
 Head    = { H = 1.00, S = 1.00, V = 1.75 }
 Grey    = { H = 1.00, S = 0.25, V = 1.37 }
-Bright  = { H = 1.00, S = 2.00, V = 2.00 }
-Syntax  = "dracula"
+Bright  = { H = 1.00, S = 0.60, V = 2.00 }
+Syntax  = "native"
 """
 
 def ensure_config_file(config):
+    config_dir = appdirs.user_config_dir("streamdown")
+    os.makedirs(config_dir, exist_ok=True)
+    config_path = os.path.join(config_dir, "config.toml")
+    if not os.path.exists(config_path):
+        open(config_path, 'w').write(default_toml)
+
+    toml_res = toml.load(config_path)
+
     if config:
-        config_path = config
-    else:
-        config_dir = appdirs.user_config_dir("streamdown")
-        os.makedirs(config_dir, exist_ok=True)
-        config_path = os.path.join(config_dir, "config.toml")
-        if not os.path.exists(config_path):
-            open(config_path, 'w').write(default_toml)
+        if os.path.exists(config):
+            config_string = open(config).read()
+        else:
+            config_string = config
+        toml_res |= toml.loads(config_string)
 
-    return config_path, open(config_path).read()
+    return toml_res
 
 
 FG = "\033[38;2;"
 BG = "\033[48;2;"
 RESET = "\033[0m"
 FGRESET = "\033[39m"
+FORMATRESET = "\033[24;23;22m"
 BGRESET = "\033[49m"
 
 BOLD      = ["\033[1m", "\033[22m"]
@@ -99,10 +107,11 @@ ANSIESCAPE = r'\033(?:\[[0-9;?]*[a-zA-Z]|][0-9]*;;.*?\\|\\)'
 KEYCODE_RE = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
 
 visible = lambda x: re.sub(ANSIESCAPE, "", x)
-# cjk characters are double width
-visible_length = lambda x: len(visible(x)) + dbl_count(x)
+# many characters have different widths
+visible_length = lambda x: sum(wcwidth(c) for c in visible(x))
 extract_ansi_codes = lambda text: re.findall(ESCAPE, text)
 remove_ansi = lambda line, codeList: reduce(lambda line, code: line.replace(code, ''), codeList, line)
+split_up = lambda line: re.findall(r'(\x1b[^m]*m|[^\x1b]*)', line)
 
 def gettmpdir():
     tmp_dir_all = os.path.join(tempfile.gettempdir(), "sd")
@@ -206,7 +215,8 @@ class ParseState:
         return offset + (state.current_width(listwidth = True) if Style.PrettyBroken else self.WidthFull)
 
     def current_width(self, listwidth = False):
-        return self.Width - (len(visible(self.space_left(listwidth))))
+        # this will double count the left margin
+        return self.Width - (len(visible(self.space_left(listwidth)))) + Style.Margin
 
     def space_left(self, listwidth = False):
         pre = ' ' * (len(state.list_item_stack)) * Style.ListIndent if listwidth else ''
@@ -235,7 +245,7 @@ def format_table(rowList):
 
     # Calculate max width per column (integer division)
     # Subtract num_cols + 1 for the vertical borders '│'
-    available_width = state.current_width() - (num_cols + 1)
+    available_width = state.current_width() - (num_cols * 2)
 
     width_base = available_width // num_cols
     width_mod  = available_width % num_cols
@@ -249,7 +259,7 @@ def format_table(rowList):
     # you are styling, do it before here!
     for ix in range(len(rowList)):
         row = rowList[ix]
-        wrapped_cell = text_wrap(row, width=col_width_list[ix], force_truncate=True)
+        wrapped_cell = text_wrap(row, width=col_width_list[ix], force_truncate=True, preserve_format=True)
 
         # Ensure at least one line, even for empty cells
         if not wrapped_cell:
@@ -290,17 +300,17 @@ def emit_h(level, text):
     for text in lineList:
         spaces_to_center = (state.current_width() -  visible_length(text)) / 2
         if level == 1:      #
-            res.append(f"{state.space_left()}\n{state.space_left()}{BOLD[1]}{' ' * math.floor(spaces_to_center)}{text}{BOLD[1]}")
+            res.append(f"{state.space_left()}\n{state.space_left()}{BOLD[0]}{' ' * math.floor(spaces_to_center)}{text}{BOLD[1]}\n")
         elif level == 2:    ##
             res.append(f"{state.space_left()}\n{state.space_left()}{BOLD[0]}{FG}{Style.Bright}{' ' * math.floor(spaces_to_center)}{text}{' ' * math.ceil(spaces_to_center)}{BOLD[1]}{FGRESET}")
         elif level == 3:    ###
-            res.append(f"{state.space_left()}{FG}{Style.Head}{BOLD[0]}{text}{RESET}")
+            res.append(f"{state.space_left()}{FG}{Style.Head}{BOLD[0]}{text}{BOLD[1]}{FGRESET}")
         elif level == 4:    ####
-            res.append(f"{state.space_left()}{FG}{Style.Symbol}{text}{RESET}")
+            res.append(f"{state.space_left()}{FG}{Style.Symbol}{BOLD[0]}{text}{BOLD[1]}{FGRESET}")
         elif level == 5:    #####
-            res.append(f"{state.space_left()}{text}{RESET}")
+            res.append(f"{state.space_left()}{text}{FGRESET}")
         else: 
-            res.append(f"{state.space_left()}{FG}{Style.Grey}{text}{RESET}")
+            res.append(f"{state.space_left()}{FG}{Style.Grey}{text}{FGRESET}")
     return "\n".join(res)
 
 def code_wrap(text_in):
@@ -379,19 +389,18 @@ def split_text(text):
         text
     ) if x]
 
-def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix="", force_truncate=False):
+def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix="", force_truncate=False, preserve_format=False):
     if width == -1:
         width = state.Width
 
     # The empty word clears the buffer at the end.
     formatted = line_format(text)
-    #print(bytes(formatted, 'utf-8'), formatted)
     words = split_text(formatted) + [""]
-    #print([bytes(i, 'utf-8') for i in words])
 
     lines = []
     current_line = ""
     current_style = []
+    resetter = "" if preserve_format else FORMATRESET 
     
     oldword = ''
     for word in words:
@@ -411,7 +420,7 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
         else:
             # Word doesn't fit, finalize the previous line
             prefix = first_line_prefix if not lines else subsequent_line_prefix
-            line_content = prefix + current_line
+            line_content = prefix + current_line  
             # This is expensive, fix.
             while force_truncate and visible_length(line_content) >= width:
                 line_content = line_content[:len(line_content) - 2] + "…"
@@ -423,7 +432,7 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
                 # that we have closed our hyperlink OSC
                 if LINK[0] in line_content:
                     line_content += LINK[1]
-                lines.append(line_content + state.bg + ' ' * margin)
+                lines.append(line_content + resetter + state.bg + ' ' * margin)
 
             current_line = (" " * indent) + "".join(current_style) + word
 
@@ -438,23 +447,17 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
     if len(lines) < 1:
         return []
 
-    return lines
+    if len(lines) == 1:
+        lines[0] = lines[0].rstrip()
 
-def dbl_count(s):
-    dbl_re = re.compile(
-        r'[\u2e80-\u2eff\u3000-\u303f\u3400-\u4dbf'
-        r'\U00004e00-\U00009fff\U0001f300-\U0001f6ff'
-        r'\U0001f900-\U0001f9ff\U0001fa70-\U0001faff]',
-        re.UNICODE
-    )
-    return len(dbl_re.findall(visible(s)))
+    return lines
 
 def cjk_count(s):
     cjk_re = re.compile(
         r'[\u4E00-\u9FFF'      # CJK Unified Ideographs
         r'\u3400-\u4DBF'       # CJK Unified Ideographs Extension A
         r'\uF900-\uFAFF'       # CJK Compatibility Ideographs
-		r'\uFF00-\uFFEF'       # CJK Compatibility Punctuation
+        r'\uFF00-\uFFEF'       # CJK Compatibility Punctuation
         r'\u3000-\u303F'      # CJK Symbols and Punctuation
         r'\U0002F800-\U0002FA1F]' # CJK Compatibility Ideographs Supplement
     )
@@ -692,11 +695,11 @@ def parse(stream):
 
         # <code><pre>
         if not state.in_code:
-            code_match = re.match(r"^\s*```\s*([^\s]+|$)\s*$", line)
+            code_match = re.match(r"^\s*(```|<pre>)\s*([^\s]+|$)\s*$", line)
             if code_match:
                 state.in_code = Code.Backtick
                 state.code_indent = len(line) - len(line.lstrip())
-                state.code_language = code_match.group(1) or 'Bash'
+                state.code_language = code_match.group(2) or 'Bash'
 
             elif state.CodeSpaces and last_line_empty_cache and not state.in_list:
                 code_match = re.match(r"^    \s*[^\s\*]", line)
@@ -726,7 +729,7 @@ def parse(stream):
         if state.in_code:
             try:
                 # This is turning it OFF
-                if ( (                     state.in_code == Code.Backtick and     line.strip() == "```"  ) or 
+                if ( (                     state.in_code == Code.Backtick and     line.strip() in ["</pre>", "```"]  ) or 
                      (state.CodeSpaces and state.in_code == Code.Spaces   and not line.startswith('    ')) ):
                     if state.scrape:
                         ext = "sh"
@@ -773,7 +776,8 @@ def parse(stream):
                     try:
                         lexer = get_lexer_by_name(state.code_language)
                         custom_style = override_background(Style.Syntax, ansi2hex(Style.Dark))
-                    except pygments.util.ClassNotFound:
+                    except pygments.util.ClassNotFound as e:
+                        logging.debug(e)
                         lexer = get_lexer_by_name("Bash")
                         custom_style = override_background("default", ansi2hex(Style.Dark))
 
@@ -806,37 +810,56 @@ def parse(stream):
                     # then naively search back until our visible_lengths() match. This is not fast and there's certainly smarter
                     # ways of doing it but this thing is way trickery than you think
                     highlighted_code = highlight(state.code_buffer + tline, lexer, formatter)
-                    #print("(",highlighted_code,")")
+                    #print("(",bytes(highlighted_code,'utf-8'),")")
+                    parts = split_up(highlighted_code)
 
                     # Sometimes the highlighter will do things like a full reset or a background reset.
                     # This is mostly not what we want
-                    highlighted_code = re.sub(r"\033\[[34]9(;00|)m", "\033[23m", highlighted_code)
+                    parts = [ re.sub(r"\033\[[34]9(;00|)m", FORMATRESET, x) for x in parts]
     
                     # Since we are streaming we ignore the resets and newlines at the end
-                    if highlighted_code.endswith(FGRESET + "\n"):
-                        highlighted_code = highlighted_code[: -(1 + len(FGRESET))]
+                    while parts[-1] in [FGRESET, FORMATRESET]:
+                        parts.pop()
+
+                    tline_len = visible_length(tline)
+
+                    # now we find the new stuff:
+                    ttl = 0
+                    for i in range(len(parts)-1, 0, -1):
+                        idx = parts[i]
+                        if len(idx) == 0:
+                            continue
+
+                        ttl += len(idx) if idx[0] != '\x1b' else 0
 
-                    # turns out highlight will eat leading newlines on empty lines
-                    vislen = visible_length(state.code_buffer.lstrip())
+                        if ttl > 1+tline_len:
+                            break
 
-                    delta = -2 
-                    while visible_length(highlighted_code[:(state.code_gen-delta)]) > vislen:
-                        delta += 1
+                    newlen = visible_length("".join(parts[i:]))
+
+                    snipfrom = newlen - len(tline) + 2
+                    if snipfrom > 0:
+                        parts[i] = parts[i][snipfrom:]
 
                     state.code_buffer += tline
+                    this_batch = "".join(parts[i:])
 
-                    this_batch = highlighted_code[state.code_gen-delta :]
                     if this_batch.startswith(FGRESET):
                         this_batch = this_batch[len(FGRESET) :]
 
+                    # clean it before prepending with potential format 
+                    this_batch = this_batch.strip()
+                    while i - 1 >= 0 and parts[i-1] and parts[i-1][0] == '\x1b':
+                         this_batch = parts[i-1] + this_batch
+                         i -= 1
+
                     ## this is the crucial counter that will determine
-                    # the begninning of the next line
+                    # the beginning of the next line
                     state.code_gen = len(highlighted_code)
-
                     code_line = ' ' * indent + this_batch.strip()
 
                     margin = state.full_width( -len(pre[1]) ) - visible_length(code_line) % state.WidthFull
-                    yield f"{pre[0]}{Style.Codebg}{pre[1]}{code_line}{' ' * max(0, margin)}{BGRESET}"  
+                    yield f"{pre[0]}{Style.Codebg}{pre[1]}{code_line}{FORMATRESET}{' ' * max(0, margin)}{BGRESET}"  
                 continue
             except Goto:
                 pass
@@ -918,7 +941,7 @@ def parse(stream):
         # a weird thing
         if state.in_list:
             indent = (len(state.list_item_stack) - 1) * Style.ListIndent #+ (len(bullet) - 1)
-            wrap_width = state.current_width() - indent - (2 * Style.ListIndent) 
+            wrap_width = state.current_width(listwidth = True) - Style.ListIndent
             
             wrapped_lineList = text_wrap(content, wrap_width, Style.ListIndent,
                 first_line_prefix = f"{(' ' * indent)}{FG}{Style.Symbol}{bullet}{RESET} ",
@@ -956,7 +979,7 @@ def parse(stream):
         state.list_item_stack = []
 
         if len(line) == 0: yield ""
-        if len(line) < state.Width:
+        if visible_length(line) < state.Width:
             # we want to prevent word wrap
             yield f"{state.space_left()}{line_format(line.lstrip())}"
         else:
@@ -1016,20 +1039,19 @@ def ansi2hex(ansi_code):
 def apply_multipliers(style, name, H, S, V):
     m = style.get(name)
     r, g, b = colorsys.hsv_to_rgb(min(1.0, H * m["H"]), min(1.0, S * m["S"]), min(1.0, V * m["V"]))
-    return ';'.join([str(int(x * 256)) for x in [r, g, b]]) + "m"
+    return ';'.join([str(int(x * 255)) for x in [r, g, b]]) + "m"
 
 def width_calc():
-    if not state.WidthFull or not state.WidthArg:
-        if state.WidthArg:
-            width = state.WidthArg
-        else:
-            width = 80
-
-            try:
-                width = shutil.get_terminal_size().columns
-                state.WidthWrap = True
-            except (AttributeError, OSError):
-                pass
+    if state.WidthArg:
+       width = state.WidthArg
+    else:
+       try:
+           width = shutil.get_terminal_size().columns
+           state.WidthWrap = True
+       except (AttributeError, OSError):
+           # this means it's a pager, we can just ignore the base64 clipboard
+           width = 80
+           pass
 
 
     # This can't be done because our list item stack can change as well so
@@ -1052,7 +1074,7 @@ def main():
     parser.add_argument("filenameList", nargs="*", help="Input file to process (also takes stdin)")
     parser.add_argument("-l", "--loglevel", default="INFO", help="Set the logging level")
     parser.add_argument("-b", "--base", default=None, help="Set the hsv base: h,s,v")
-    parser.add_argument("-c", "--config", default=None, help="Use a custom config")
+    parser.add_argument("-c", "--config", default=None, help="Use a custom config override")
     parser.add_argument("-w", "--width", default="0", help="Set the width WIDTH")
     parser.add_argument("-e", "--exec", help="Wrap a program EXEC for more 'proper' i/o handling")
     parser.add_argument("-s", "--scrape", help="Scrape code snippets to a directory SCRAPE")
@@ -1060,18 +1082,21 @@ def main():
     args = parser.parse_args()
 
     if args.version:
-        import importlib.metadata
-
         try:
-            version = importlib.metadata.version("streamdown")
+            import importlib.metadata
+            print(importlib.metadata.version("streamdown"))
         except importlib.metadata.PackageNotFoundError:
-            version = "Unknown"
+            import subprocess
+            print(subprocess.run(
+                ['git', 'describe', '--always', '--dirty', '--tags'],
+                cwd=os.path.dirname(os.path.abspath(__file__)),
+                stdout=subprocess.PIPE,
+                text=True
+            ).stdout.strip())
 
-        print(f"Streamdown - {version}")
         sys.exit(0)
 
-    config_toml_path, config_toml_content = ensure_config_file(args.config)
-    config = toml.loads(config_toml_content)
+    config = ensure_config_file(args.config)
     style = toml.loads(default_toml).get('style') | config.get("style", {})
     features = toml.loads(default_toml).get('features') | config.get("features", {})
     H, S, V = style.get("HSV")
@@ -1114,7 +1139,7 @@ def main():
             # Set stdin to raw mode so we don't need to press enter
             tty.setcbreak(sys.stdin.fileno())
             sys.stdout.write("\x1b[?7h")
-            emit(sys.stdin)
+            emit(inp)
 
         elif args.filenameList:
             # Let's say we only care about logging in streams
@@ -1142,7 +1167,7 @@ def main():
         logging.warning(f"Exception thrown: {type(ex)} {ex}")
         traceback.print_exc()
 
-    if state.Clipboard and state.code_buffer_raw:
+    if os.isatty(sys.stdout.fileno()) and state.Clipboard and state.code_buffer_raw:
         code = state.code_buffer_raw
         # code needs to be a base64 encoded string before emitting
         code_bytes = code.encode('utf-8')

{streamdown-0.26.0 → streamdown-0.28.0}/tests/chunk-buffer.sh

@@ -3,6 +3,7 @@ TIMEOUT=${TIMEOUT:-0.05}
 
 while [[ $# -gt 0 ]]; do
     echo "## File: $1"
+    echo
     echo -e "----\n\n"
     cat $1 | awk -v RS='🫣' '{ printf "%s", $0; system("sleep '$TIMEOUT'") }' 
     shift

streamdown-0.28.0/tests/dimcheck.md

@@ -0,0 +1,7 @@
+
+### **3. Force a Full Restart 💥**
+Sometimes a reload isn’t enough. Stop and restart the service:
+```bash
+sudo systemctl stop postgresql
+sudo systemctl start postgresql
+```

{streamdown-0.26.0 → streamdown-0.28.0}/tests/fizzbuzz.md

@@ -42,7 +42,7 @@ def fizzbuzz(n):
         else:
             print(i)
 
-# Example usage: Print FizzBuzz up to 100 Example usage: Print FizzBuzz up to 100 Example usage: Print FizzBuzz up to 100 Example usage: Print FizzBuzz up to 100
+# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100  
 fizzbuzz(100)
 
 #  Example usage: different range:

streamdown-0.28.0/tests/list-test.md

@@ -0,0 +1,5 @@
+hi
+* one
+* two
+  * three
+  * four

streamdown-0.28.0/tests/uline.md

@@ -0,0 +1,51 @@
+🫣Here’s a DIY approach using Python to map text to Unicode’s **Mathematical Script** (or other math alphanumerics):  
+
+---
+
+### Example Python Code for **Mathematical Script Letters**:
+```python
+def to_math_script(text: str) -> str:
+    """
+    Convert uppercase and lowercase letters to Mathematical Script (Unicode U+1D49C-1D4FD).
+    Non-alphabetic characters remain unchanged.
+    """
+   🫣 res = []
+    for c in text:
+        if c.isupper():
+            base = 0x1D49C  # Math Script Capital "A"
+            res += [chr(base + (ord(c) - ord('A')))]
+        elif c.islower():
+            base = 0x1D4BA  # Math Script Small "a"
+            res += [chr(base + (ord(c) - ord('a')))]
+        else:
+            res += [c]
+    return "".join(res)
+
+# Test:
+print(to_math_script("Hello World!ΑΩ"))  
+# Output: ℍ Escorts(math script "ell o World")!ΑΩ
+```
+
+---
+
+### Other Unicode Math Scripts:
+- **Fraktur**: `U+1🫣D504`–`U+1D537` (` Francisco ` → 𝔣𝔯𝔞𝔨𝔱𝔲𝔯)  
+- **Bold Fraktur**: `U+1D56C`–`U+1D59F`  
+- **Double-struck (Blackboard)**: `U+1D538`–`U+1D55F` (category: `ℂℍℕℙℚℝ`).  
+
+You can extend the code to support these by changing the `base` values and [Unicode ranges](https://unicode.org/charts/PDF/U1D400.pdf).
+
+---
+
+### Resources:
+1. **Unicode Charts**:  
+   - [Math Alphanumeric Symbols](https://unicode.org/charts/PDF/U1D400.pdf).  
+2. **Python’s `unicodedata`**:  
+   ```python
+   import unicodedata
+   print(unicodedata.name("𝒜"))  # "MATHEMATICAL SCRIPT CAPITAL A"
+   ```  
+3. **Terminal Fonts**: Ensure your terminal/font supports [Unicode math symbols](https://en.wikipedia.org/wiki/Mathematical_Alphanumeric_Symbols).
+
+Let me know if you want to target a different script!🫣
+🫣

streamdown-0.28.0/tools/deploy.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+set -eEuo pipefail
+version=$(grep version pyproject.toml  | cut -d '"' -f 2)
+tag_update() {
+    git tag -m v$version v$version
+    git push --tags
+}
+pipy() {
+    source .venv/bin/activate
+    for i in pip hatch build; do
+        pip install --upgrade $i
+    done
+    python3 -m build .
+    twine upload dist/*${version}*
+}
+#tag_update
+pipy