streamdown 0.17.0__tar.gz → 0.19.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. {streamdown-0.17.0 → streamdown-0.19.0}/PKG-INFO +12 -6
  2. {streamdown-0.17.0 → streamdown-0.19.0}/README.md +11 -5
  3. {streamdown-0.17.0 → streamdown-0.19.0}/pyproject.toml +1 -1
  4. streamdown-0.19.0/ss-new.py +45 -0
  5. streamdown-0.19.0/ss.py +69 -0
  6. {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/sd.py +60 -23
  7. streamdown-0.19.0/streamdown/ss +1 -0
  8. streamdown-0.19.0/streamdown/ss1 +42 -0
  9. streamdown-0.19.0/tests/bold_reset_with_link.md +1 -0
  10. streamdown-0.19.0/tests/chinese.md +4 -0
  11. streamdown-0.19.0/tests/cjj.mv +4 -0
  12. streamdown-0.19.0/tests/cjk-table.md +13 -0
  13. streamdown-0.19.0/tests/cjk-wrap.md +150 -0
  14. streamdown-0.19.0/tests/jimmy_webb.md +1458 -0
  15. streamdown-0.19.0/tests/strip-chunks.sh +2 -0
  16. {streamdown-0.17.0 → streamdown-0.19.0}/tests/table_test.md +0 -3
  17. streamdown-0.17.0/streamdown/ss +0 -21
  18. streamdown-0.17.0/tests/longer-example.md +0 -18
  19. streamdown-0.17.0/tests/table.md +0 -4
  20. streamdown-0.17.0/tests/white-space-code.md +0 -19
  21. {streamdown-0.17.0 → streamdown-0.19.0}/.gitignore +0 -0
  22. {streamdown-0.17.0 → streamdown-0.19.0}/.vimrc +0 -0
  23. {streamdown-0.17.0 → streamdown-0.19.0}/24-bit-color.sh +0 -0
  24. {streamdown-0.17.0 → streamdown-0.19.0}/LICENSE.MIT +0 -0
  25. {streamdown-0.17.0 → streamdown-0.19.0}/assets/logo.png +0 -0
  26. {streamdown-0.17.0 → streamdown-0.19.0}/assets/logo.svg +0 -0
  27. {streamdown-0.17.0 → streamdown-0.19.0}/configurable.png +0 -0
  28. {streamdown-0.17.0 → streamdown-0.19.0}/copyable.png +0 -0
  29. {streamdown-0.17.0 → streamdown-0.19.0}/dunder.png +0 -0
  30. {streamdown-0.17.0 → streamdown-0.19.0}/error.txt +0 -0
  31. {streamdown-0.17.0 → streamdown-0.19.0}/newdir/file_0.py +0 -0
  32. {streamdown-0.17.0 → streamdown-0.19.0}/newdir/file_1.rb +0 -0
  33. {streamdown-0.17.0 → streamdown-0.19.0}/newdir/file_2.jl +0 -0
  34. {streamdown-0.17.0 → streamdown-0.19.0}/passthrough.py +0 -0
  35. {streamdown-0.17.0 → streamdown-0.19.0}/python-go.png +0 -0
  36. {streamdown-0.17.0 → streamdown-0.19.0}/requirements.txt +0 -0
  37. {streamdown-0.17.0 → streamdown-0.19.0}/somelog.txt +0 -0
  38. {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/__init__.py +0 -0
  39. {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/plugins/README.md +0 -0
  40. {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/plugins/latex.py +0 -0
  41. {streamdown-0.17.0 → streamdown-0.19.0}/table.png +0 -0
  42. {streamdown-0.17.0 → streamdown-0.19.0}/temp.py +0 -0
  43. {streamdown-0.17.0 → streamdown-0.19.0}/test.py +0 -0
  44. {streamdown-0.17.0 → streamdown-0.19.0}/test_input.md +0 -0
  45. {streamdown-0.17.0 → streamdown-0.19.0}/tests/README.md +0 -0
  46. {streamdown-0.17.0 → streamdown-0.19.0}/tests/backtick-with-post-spaces.md +0 -0
  47. {streamdown-0.17.0 → streamdown-0.19.0}/tests/block.md +0 -0
  48. {streamdown-0.17.0 → streamdown-0.19.0}/tests/broken-example.md +0 -0
  49. {streamdown-0.17.0 → streamdown-0.19.0}/tests/chunk-buffer.sh +0 -0
  50. {streamdown-0.17.0 → streamdown-0.19.0}/tests/code.md +0 -0
  51. {streamdown-0.17.0 → streamdown-0.19.0}/tests/escape.md +0 -0
  52. {streamdown-0.17.0 → streamdown-0.19.0}/tests/example.md +0 -0
  53. {streamdown-0.17.0 → streamdown-0.19.0}/tests/fizzbuzz.md +0 -0
  54. {streamdown-0.17.0 → streamdown-0.19.0}/tests/inline.md +0 -0
  55. {streamdown-0.17.0 → streamdown-0.19.0}/tests/line-buffer.sh +0 -0
  56. {streamdown-0.17.0 → streamdown-0.19.0}/tests/line-wrap.md +0 -0
  57. {streamdown-0.17.0 → streamdown-0.19.0}/tests/links.md +0 -0
  58. {streamdown-0.17.0 → streamdown-0.19.0}/tests/managerie.md +0 -0
  59. {streamdown-0.17.0 → streamdown-0.19.0}/tests/mandlebrot.md +0 -0
  60. {streamdown-0.17.0 → streamdown-0.19.0}/tests/markdown.md +0 -0
  61. {streamdown-0.17.0 → streamdown-0.19.0}/tests/nested-example.md +0 -0
  62. {streamdown-0.17.0 → streamdown-0.19.0}/tests/outline.md +0 -0
  63. {streamdown-0.17.0 → streamdown-0.19.0}/tests/pvgo_512.jpg +0 -0
  64. {streamdown-0.17.0 → streamdown-0.19.0}/tests/pythonvgo.md +0 -0
  65. {streamdown-0.17.0 → streamdown-0.19.0}/tests/table-break.md +0 -0
  66. {streamdown-0.17.0 → streamdown-0.19.0}/tests/test.md +0 -0
  67. {streamdown-0.17.0 → streamdown-0.19.0}/tests/test_input.md +0 -0
  68. {streamdown-0.17.0 → streamdown-0.19.0}/tests/wm.md +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: streamdown
3
- Version: 0.17.0
3
+ Version: 0.19.0
4
4
  Summary: A streaming markdown renderer for modern terminals with syntax highlighting
5
5
  Project-URL: Homepage, https://github.com/kristopolous/Streamdown
6
6
  Project-URL: Bug Tracker, https://github.com/kristopolous/Streamdown/issues
@@ -31,23 +31,25 @@ Description-Content-Type: text/markdown
31
31
  <img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
32
32
  <br/>
33
33
  <a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
34
+ <br/><strong>Terminal streaming markdown that rocks</strong>
35
+
34
36
  </p>
35
37
 
36
- **The streaming markdown renderer for the terminal that rocks**
37
38
 
38
- Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown. You even get full readline and keyboard navigation support.
39
+ Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
40
+ It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
39
41
  ```bash
40
42
  $ pip install streamdown
41
43
  ```
42
44
  ![Streamdown is Amazing](https://github.com/user-attachments/assets/268cb340-78cc-4df0-a773-c5ac95eceeeb)
43
45
 
44
46
  ### Provides clean copyable code for long code lines
45
- Some *inferior* renderers inject line breaks when copying code that wraps around. We're better and now you are too!
47
+ Other renderers inject line breaks when copying code that wraps around. We're better and now you are too!
46
48
  ![Handle That Mandle](https://github.com/user-attachments/assets/a27aa70c-f691-4796-84f0-c2eb18c7de23)
47
49
  **Tip**: You can make things prettier if you don't mind if this guarantee is broken. See the `PrettyBroken` flag below!
48
50
 
49
51
  ### Supports images
50
- Here's kitty and alacritty. Try to do that in glow...
52
+ Here's kitty and alacritty.
51
53
  ![doggie](https://github.com/user-attachments/assets/81c43983-68cd-40c1-b1d5-aa3a52004504)
52
54
 
53
55
  ### Supports hyperlinks (OSC 8) and clipboard (OSC 52)
@@ -56,9 +58,13 @@ Here's kitty and alacritty. Try to do that in glow...
56
58
  ### Supports tables
57
59
  ![table](https://github.com/user-attachments/assets/dbe3d13e-6bac-4f45-bf30-f1857ed98898)
58
60
 
59
- As well as everything else...
61
+ #### As well as everything else...
60
62
  ![dunder](https://github.com/user-attachments/assets/d41d7fec-6dec-4387-b53d-f2098f269a5e)
61
63
 
64
+ #### ...even CJK
65
+ Compare how streamdown wraps around and spaces this tabular Chinese description of programming languages to the same file using glow.
66
+ ![cjk](https://github.com/user-attachments/assets/b831a5f3-7ef0-48b8-8d17-bb7b605df16a)
67
+
62
68
  ### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
63
69
  ![configurable](https://github.com/user-attachments/assets/19ca2ec9-8ea1-4a79-87ca-8352789269fe)
64
70
 
@@ -2,23 +2,25 @@
2
2
  <img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
3
3
  <br/>
4
4
  <a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
5
+ <br/><strong>Terminal streaming markdown that rocks</strong>
6
+
5
7
  </p>
6
8
 
7
- **The streaming markdown renderer for the terminal that rocks**
8
9
 
9
- Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown. You even get full readline and keyboard navigation support.
10
+ Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
11
+ It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
10
12
  ```bash
11
13
  $ pip install streamdown
12
14
  ```
13
15
  ![Streamdown is Amazing](https://github.com/user-attachments/assets/268cb340-78cc-4df0-a773-c5ac95eceeeb)
14
16
 
15
17
  ### Provides clean copyable code for long code lines
16
- Some *inferior* renderers inject line breaks when copying code that wraps around. We're better and now you are too!
18
+ Other renderers inject line breaks when copying code that wraps around. We're better and now you are too!
17
19
  ![Handle That Mandle](https://github.com/user-attachments/assets/a27aa70c-f691-4796-84f0-c2eb18c7de23)
18
20
  **Tip**: You can make things prettier if you don't mind if this guarantee is broken. See the `PrettyBroken` flag below!
19
21
 
20
22
  ### Supports images
21
- Here's kitty and alacritty. Try to do that in glow...
23
+ Here's kitty and alacritty.
22
24
  ![doggie](https://github.com/user-attachments/assets/81c43983-68cd-40c1-b1d5-aa3a52004504)
23
25
 
24
26
  ### Supports hyperlinks (OSC 8) and clipboard (OSC 52)
@@ -27,9 +29,13 @@ Here's kitty and alacritty. Try to do that in glow...
27
29
  ### Supports tables
28
30
  ![table](https://github.com/user-attachments/assets/dbe3d13e-6bac-4f45-bf30-f1857ed98898)
29
31
 
30
- As well as everything else...
32
+ #### As well as everything else...
31
33
  ![dunder](https://github.com/user-attachments/assets/d41d7fec-6dec-4387-b53d-f2098f269a5e)
32
34
 
35
+ #### ...even CJK
36
+ Compare how streamdown wraps around and spaces this tabular Chinese description of programming languages to the same file using glow.
37
+ ![cjk](https://github.com/user-attachments/assets/b831a5f3-7ef0-48b8-8d17-bb7b605df16a)
38
+
33
39
  ### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
34
40
  ![configurable](https://github.com/user-attachments/assets/19ca2ec9-8ea1-4a79-87ca-8352789269fe)
35
41
 
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "streamdown"
7
- version = "0.17.0"
7
+ version = "0.19.0"
8
8
  description = "A streaming markdown renderer for modern terminals with syntax highlighting"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.8"
@@ -0,0 +1,45 @@
1
+ import os
2
+ import pty
3
+ import sys
4
+ import tty
5
+ import termios
6
+ import select
7
+ import re
8
+
9
+ def main():
10
+ orig_attrs = termios.tcgetattr(sys.stdin)
11
+ try:
12
+ tty.setraw(sys.stdin)
13
+ pid, fd = pty.fork()
14
+
15
+ if pid == 0:
16
+ # Child process: run any command
17
+ os.execvp("llm", ["llm", "chat"])
18
+ else:
19
+ buffer = b""
20
+
21
+ while True:
22
+ r, _, _ = select.select([fd, sys.stdin], [], [])
23
+
24
+ if sys.stdin in r:
25
+ data = os.read(sys.stdin.fileno(), 1024)
26
+ if not data:
27
+ break
28
+ os.write(fd, data)
29
+
30
+ if fd in r:
31
+ data = os.read(fd, 1024)
32
+ if not data:
33
+ break
34
+
35
+ buffer += data
36
+ # Replace "fizz" only in printable content
37
+ output = re.sub(rb'fizz', b'fizzbuzz', buffer)
38
+ os.write(sys.stdout.fileno(), output)
39
+ buffer = b""
40
+ finally:
41
+ termios.tcsetattr(sys.stdin, termios.TCSADRAIN, orig_attrs)
42
+
43
+ if __name__ == "__main__":
44
+ main()
45
+
@@ -0,0 +1,69 @@
1
+ import os
2
+ import pty
3
+ import select
4
+ import sys
5
+ import tty
6
+ import termios
7
+ import fcntl
8
+ import struct
9
+
10
+ def set_pty_size(fd, target_fd):
11
+ """Set window size of PTY to match the real terminal."""
12
+ s = fcntl.ioctl(target_fd, termios.TIOCGWINSZ, b"\x00" * 8)
13
+ fcntl.ioctl(fd, termios.TIOCSWINSZ, s)
14
+
15
+ def main():
16
+ # Save original terminal settings
17
+ orig_attrs = termios.tcgetattr(sys.stdin.fileno())
18
+ try:
19
+ tty.setraw(sys.stdin.fileno()) # raw mode to send Ctrl-C, etc.
20
+ pid, fd = pty.fork()
21
+
22
+ if pid == 0:
23
+ os.execvp("bash", ["bash"])
24
+ else:
25
+ set_pty_size(fd, sys.stdin.fileno())
26
+
27
+ while True:
28
+ r, _, _ = select.select([fd, sys.stdin], [], [])
29
+
30
+ if sys.stdin in r:
31
+ user_input = os.read(sys.stdin.fileno(), 1024)
32
+ if not user_input:
33
+ break
34
+ os.write(fd, user_input)
35
+
36
+ if fd in r:
37
+ output = os.read(fd, 1024)
38
+ if not output:
39
+ break
40
+
41
+ # Carefully handle ANSI sequences
42
+ # Split on ANSI escape sequences to avoid breaking them
43
+ chunks = []
44
+ i = 0
45
+ while i < len(output):
46
+ if output[i:i+1] == b'\x1b':
47
+ end = i + 1
48
+ while end < len(output) and not (64 <= output[end] <= 126):
49
+ end += 1
50
+ end += 1 # include final letter
51
+ chunks.append(output[i:end])
52
+ i = end
53
+ else:
54
+ j = i
55
+ while j < len(output) and output[j:j+1] != b'\x1b':
56
+ j += 1
57
+ # Perform replacement on plain text only
58
+ text = output[i:j].replace(b"fizz", b"fizzbuzz")
59
+ chunks.append(text)
60
+ i = j
61
+
62
+ os.write(sys.stdout.fileno(), b''.join(chunks))
63
+
64
+ finally:
65
+ termios.tcsetattr(sys.stdin.fileno(), termios.TCSADRAIN, orig_attrs)
66
+
67
+ if __name__ == "__main__":
68
+ main()
69
+
@@ -36,7 +36,7 @@ from functools import reduce
36
36
  from argparse import ArgumentParser
37
37
  from pygments import highlight
38
38
  from pygments.lexers import get_lexer_by_name
39
- from pygments.formatters import Terminal256Formatter
39
+ from pygments.formatters import TerminalTrueColorFormatter
40
40
  from pygments.styles import get_style_by_name
41
41
 
42
42
  if __package__ is None:
@@ -46,7 +46,7 @@ else:
46
46
 
47
47
  default_toml = """
48
48
  [features]
49
- CodeSpaces = true
49
+ CodeSpaces = false
50
50
  Clipboard = true
51
51
  Logging = false
52
52
  Timeout = 0.1
@@ -62,7 +62,7 @@ HSV = [0.8, 0.5, 0.5]
62
62
  Dark = { H = 1.00, S = 1.50, V = 0.25 }
63
63
  Mid = { H = 1.00, S = 1.00, V = 0.50 }
64
64
  Symbol = { H = 1.00, S = 1.00, V = 1.50 }
65
- Head = { H = 1.00, S = 2.00, V = 1.50 }
65
+ Head = { H = 1.00, S = 1.00, V = 1.75 }
66
66
  Grey = { H = 1.00, S = 0.25, V = 1.37 }
67
67
  Bright = { H = 1.00, S = 2.00, V = 2.00 }
68
68
  Syntax = "monokai"
@@ -99,7 +99,8 @@ ANSIESCAPE = r'\033(?:\[[0-9;?]*[a-zA-Z]|][0-9]*;;.*?\\|\\)'
99
99
  KEYCODE_RE = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
100
100
 
101
101
  visible = lambda x: re.sub(ANSIESCAPE, "", x)
102
- visible_length = lambda x: len(visible(x))
102
+ # cjk characters are double width
103
+ visible_length = lambda x: len(visible(x)) + cjk_count(x)
103
104
  extract_ansi_codes = lambda text: re.findall(ESCAPE, text)
104
105
  remove_ansi = lambda line, codeList: reduce(lambda line, code: line.replace(code, ''), codeList, line)
105
106
 
@@ -231,7 +232,7 @@ def format_table(rowList):
231
232
  # Note this is where every cell is formatted so if
232
233
  # you are styling, do it before here!
233
234
  for row in rowList:
234
- wrapped_cell = text_wrap(row, width=col_width)
235
+ wrapped_cell = text_wrap(row, width=col_width, force_truncate=True)
235
236
 
236
237
  # Ensure at least one line, even for empty cells
237
238
  if not wrapped_cell:
@@ -285,7 +286,7 @@ def code_wrap(text_in):
285
286
  # get the indentation of the first line
286
287
  indent = len(text_in) - len(text_in.lstrip())
287
288
  text = text_in.lstrip()
288
- mywidth = state.full_width() - indent
289
+ mywidth = state.full_width(-4 if Style.PrettyBroken else 0) - indent
289
290
 
290
291
  # We take special care to preserve empty lines
291
292
  if len(text) == 0:
@@ -304,7 +305,7 @@ def ansi_collapse(codelist, inp):
304
305
  # We break SGR strings into various classes concerning their applicate or removal
305
306
  nums = {
306
307
  'fg': r'3\d', 'bg': r'4\d',
307
- 'b': r'2?1', 'i': r'2?3', 'u': r'3?2',
308
+ 'b': r'2?[12]', 'i': r'2?3', 'u': r'3?2',
308
309
  'reset': '0'
309
310
  }
310
311
 
@@ -334,16 +335,24 @@ def ansi_collapse(codelist, inp):
334
335
 
335
336
  return codelist + inp
336
337
 
337
- def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix=""):
338
+
339
+ def split_text(text):
340
+ return [x for x in re.split(
341
+ r'(?<=[\u3000-\u303F\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|(?=[\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|\s+',
342
+ text
343
+ ) if x]
344
+
345
+ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix="", force_truncate=False):
338
346
  if width == -1:
339
347
  width = state.Width
340
348
 
341
349
  # The empty word clears the buffer at the end.
342
- words = line_format(text).split() + [""]
350
+ words = split_text(line_format(text)) + [""]
343
351
  lines = []
344
352
  current_line = ""
345
353
  current_style = []
346
354
 
355
+ oldword = ''
347
356
  for word in words:
348
357
  # we apply the style if we see it at the beginning of the word
349
358
  codes = extract_ansi_codes(word)
@@ -352,13 +361,25 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
352
361
  current_style.append(codes.pop(0))
353
362
 
354
363
  if len(word) and visible_length(current_line) + visible_length(word) + 1 <= width: # +1 for space
355
- current_line += (" " if current_line else "") + word
364
+ space = ""
365
+ if len(visible(word)) > 0 and current_line:
366
+ space = " "
367
+ if (":" in visible(word) or cjk_count(word)) and cjk_count(oldword):
368
+ space = ""
369
+ current_line += space + word
356
370
  else:
357
371
  # Word doesn't fit, finalize the previous line
358
372
  prefix = first_line_prefix if not lines else subsequent_line_prefix
359
373
  line_content = prefix + current_line
374
+ # This is expensive, fix.
375
+ while force_truncate and visible_length(line_content) >= width:
376
+ line_content = line_content[:len(line_content) - 2] + "…"
377
+
360
378
  margin = max(0, width - visible_length(line_content))
361
- lines.append(line_content + state.bg + ' ' * margin)
379
+
380
+ if line_content.strip() != "":
381
+ lines.append(line_content + state.bg + ' ' * margin)
382
+
362
383
  current_line = (" " * indent) + "".join(current_style) + word
363
384
 
364
385
  if len(codes):
@@ -367,13 +388,27 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
367
388
  if codes:
368
389
  current_style = ansi_collapse(current_style, codes)
369
390
 
391
+ oldword = word
392
+
370
393
  if len(lines) < 1:
371
394
  return []
372
395
 
373
396
  return lines
374
397
 
398
+ def cjk_count(s):
399
+ cjk_re = re.compile(
400
+ r'[\u4E00-\u9FFF' # CJK Unified Ideographs
401
+ r'|\u3400-\u4DBF' # CJK Unified Ideographs Extension A
402
+ r'|\uF900-\uFAFF' # CJK Compatibility Ideographs
403
+ r'|\uFF00-\uFFEF' # CJK Compatibility Punctuation
404
+ r'|\u3000-\u303F' # CJK Symbols and Punctuation
405
+ r'|\U0002F800-\U0002FA1F]' # CJK Compatibility Ideographs Supplement
406
+ )
407
+
408
+ return len(cjk_re.findall(visible(s)))
409
+
375
410
  def line_format(line):
376
- not_text = lambda token: not (token.isalnum() or token == '\\')
411
+ not_text = lambda token: not (token.isalnum() or token == '\\') or cjk_count(token)
377
412
  footnotes = lambda match: ''.join([chr(SUPER[int(i)]) for i in match.group(1)])
378
413
 
379
414
  def process_images(match):
@@ -402,7 +437,7 @@ def line_format(line):
402
437
  result = ""
403
438
 
404
439
  for match in tokenList:
405
- token = match.group(1)
440
+ token = re.sub(r'\s+',' ', match.group(1))
406
441
  next_token = line[match.end()] if match.end() < len(line) else ""
407
442
  prev_token = line[match.start()-1] if match.start() > 0 else ""
408
443
 
@@ -458,6 +493,7 @@ def parse(stream):
458
493
  last_line_empty_cache = None
459
494
  byte = None
460
495
  TimeoutIx = 0
496
+ lexer = None
461
497
  while True:
462
498
  if state.is_pty or state.is_exec:
463
499
  byte = None
@@ -548,7 +584,7 @@ def parse(stream):
548
584
  line = line[len(block_match.group(0)):]
549
585
  else:
550
586
  if state.block_depth > 0:
551
- line = FGRESET + line
587
+ yield FGRESET
552
588
  state.block_depth = 0
553
589
 
554
590
  # --- Collapse Multiple Empty Lines if not in code blocks ---
@@ -569,7 +605,7 @@ def parse(stream):
569
605
  # \n buffer
570
606
  if not state.in_list and len(state.ordered_list_numbers) > 0:
571
607
  state.ordered_list_numbers[0] = 0
572
- elif not line.startswith(' ' * state.list_indent_text):
608
+ elif (not line.startswith(' ' * state.list_indent_text)) and line.strip() != "":
573
609
  state.in_list = False
574
610
  state.list_indent_text = 0
575
611
 
@@ -578,7 +614,7 @@ def parse(stream):
578
614
  if len(line) - len(line.lstrip()) >= state.first_indent:
579
615
  line = line[state.first_indent:]
580
616
  else:
581
- logging.warning("Indentation decreased from first line.")
617
+ logging.debug("Indentation decreased from first line.")
582
618
 
583
619
 
584
620
  # Indent guaranteed
@@ -648,8 +684,8 @@ def parse(stream):
648
684
  logging.debug(f"code: {state.in_code}")
649
685
  state.emit_flush = True
650
686
  # We suppress the newline - it's not an explicit style
651
- state.has_newline = False
652
- yield RESET
687
+ #state.has_newline = False
688
+ #yield RESET
653
689
 
654
690
  if code_type == Code.Backtick:
655
691
  continue
@@ -658,7 +694,7 @@ def parse(stream):
658
694
  # nor do we want to be here.
659
695
  raise Goto()
660
696
 
661
- if state.code_first_line:
697
+ if state.code_first_line or lexer is None:
662
698
  state.code_first_line = False
663
699
  try:
664
700
  lexer = get_lexer_by_name(state.code_language)
@@ -667,7 +703,7 @@ def parse(stream):
667
703
  lexer = get_lexer_by_name("Bash")
668
704
  custom_style = get_style_by_name("default")
669
705
 
670
- formatter = Terminal256Formatter(style=custom_style)
706
+ formatter = TerminalTrueColorFormatter(style=custom_style)
671
707
  if line.startswith(' ' * state.code_indent):
672
708
  line = line[state.code_indent :]
673
709
 
@@ -687,7 +723,7 @@ def parse(stream):
687
723
  indent, line_wrap = code_wrap(line)
688
724
 
689
725
  state.where_from = "in code"
690
- pre = [state.space_left(listwidth = True), ' '] if Style.PrettyBroken else ['', '']
726
+ pre = [state.space_left(listwidth = True), ' '] if Style.PrettyBroken else ['', '']
691
727
 
692
728
  for tline in line_wrap:
693
729
  # wrap-around is a bunch of tricks. We essentially format longer and longer portions of code. The problem is
@@ -698,7 +734,7 @@ def parse(stream):
698
734
 
699
735
  # Sometimes the highlighter will do things like a full reset or a background reset.
700
736
  # This is not what we want
701
- highlighted_code = re.sub(r"\033\[39(;00|)m", '', highlighted_code)
737
+ highlighted_code = re.sub(r"\033\[49(;00|)m", '', highlighted_code)
702
738
 
703
739
  # Since we are streaming we ignore the resets and newlines at the end
704
740
  if highlighted_code.endswith(FGRESET + "\n"):
@@ -805,7 +841,7 @@ def parse(stream):
805
841
  # This is intentional ... we can get here in llama 4 using
806
842
  # a weird thing
807
843
  if state.in_list:
808
- indent = (len(state.list_item_stack) - 1) * Style.ListIndent
844
+ indent = (len(state.list_item_stack) - 1) * Style.ListIndent #+ (len(bullet) - 1)
809
845
  wrap_width = state.current_width() - indent - (2 * Style.ListIndent)
810
846
 
811
847
  wrapped_lineList = text_wrap(content, wrap_width, Style.ListIndent,
@@ -964,6 +1000,7 @@ def main():
964
1000
  os.close(state.exec_slave) # We don't need slave in parent
965
1001
  # Set stdin to raw mode so we don't need to press enter
966
1002
  tty.setcbreak(sys.stdin.fileno())
1003
+ sys.stdout.write("\x1b[?7h")
967
1004
  emit(sys.stdin)
968
1005
 
969
1006
  elif args.filenameList:
@@ -0,0 +1 @@
1
+ * **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
@@ -0,0 +1,42 @@
1
+ * `model.safetensors`: The name of the file you want to download. You'll need to know the exact filename. You can find the files in the model repository on the Hugging Face Hub website ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)). Look under the "Files and versions" tab. `safetensors` is the preferred format for model weights now. If it's a `.bin` file, you can download that instead.
2
+ * `--local-dir ./bitnet-b1.58-2B-4T`: The directory to save the file to.
3
+
4
+ * **Download using `transformers` library (recommended for most use cases):**
5
+
6
+ The `transformers` library provides a convenient way to download and cache models. This is often the easiest approach if you're using the model with `transformers`. You don't *directly* use the `huggingface-cli` for this, but it's worth knowing.
7
+
8
+ ```python
9
+ from transformers import AutoModelForCausalLM, AutoTokenizer
10
+
11
+ model_name = "microsoft/bitnet-b1.58-2B-4T"
12
+
13
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
14
+ model = AutoModelForCausalLM.from_pretrained(model_name)
15
+
16
+ # The model and tokenizer will be downloaded and cached in your
17
+ # transformers cache directory (usually ~/.cache/huggingface/transformers).
18
+ ```
19
+
20
+ This approach automatically handles downloading the necessary files and caching them for future use. It also handles the correct file formats and configurations.
21
+
22
+ **4. Checking the Download**
23
+
24
+ After the download completes, verify that the files are in the specified directory. You can use `ls` (Linux/macOS) or `dir` (Windows) to list the contents of the directory.
25
+
26
+ **Important Considerations:**
27
+
28
+ * **Disk Space:** The `bitnet-b1.58-2B-4T` model is quite large (several gigabytes). Make sure you have enough free disk space before downloading.
29
+ * **Network Connection:** A stable and fast internet connection is essential for a smooth download.
30
+ * **Caching:** The Hugging Face Hub and `transformers` library use caching to avoid re-downloading models unnecessarily. The default cache directory is usually `~/.cache/huggingface/transformers`.
31
+ * **File Formats:** Models are often stored in `safetensors` or `.bin` formats. `safetensors` is generally preferred for security and performance.
32
+ * **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
33
+ * **Gated Models:** Some models require you to accept terms of use before you can download them. The `huggingface-cli login` command will guide you through this process if necessary.
34
+
35
+ **Example Workflow (Recommended):**
36
+
37
+ 1. `huggingface-cli login` (if not already logged in)
38
+ 2. Use the `transformers` library in a Python script to download and load the model (as shown in the example above). This is the most convenient and reliable method for most use cases.
39
+
40
+ Let me know if you have any other questions or if you'd like help with a specific task related to this model!
41
+
42
+ >
@@ -0,0 +1 @@
1
+ * **Model Card:** When this wraps with this link ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) the next line used to be bold because of some stripping error. This should no longer be the case
@@ -0,0 +1,4 @@
1
+ **XIǍO LÓNG:** *(mocking the academics)*
2
+ "Ohhh, ‘**Dadaist**’? ‘**Absurd**’? 那你们怎是在说**模形式**?你们这群人连**京剧密码**都解么没看懂那个**转身**是在问**递归关系**?那个**抖袖**是在说**模形式**?你们这群人连**京剧密码**都解不开,还好意思说自己是数学家?"
3
+ *(Translation: "Ohhh, ‘Dadaist’? ‘Absurd’? Then how come none of you realized the **spin** was about **recurrence relations**? The **sleeve flick** was about **modular forms**? You can’t even crack **Peking Opera code**, and you call yourselves mathematicians?")*
4
+
@@ -0,0 +1,4 @@
1
+ | 特性 | Ruby | Python | OCaml |
2
+ |------------------|--------------------------------|--------------------------------|-------------------------------|
3
+ | **适用场景** | 快速开发、Web 应用、自动化脚本 | 数据科学、机器学习、Web 开发、自动化脚本 | 性能关键的应用、编译器开发、金融建模 |
4
+
@@ -0,0 +1,13 @@
1
+ | 特性 | Ruby | Python | OCaml |
2
+ |------------------|--------------------------------|--------------------------------|-------------------------------|
3
+ | **设计哲学** | 使编程更加自然和有趣 | 强调代码的可读性和简洁性 | 提供高效的函数式编程功能 |
4
+ | **应用领域** | Web 开发、自动化脚本、原型开发 | Web 开发、数据科学、机器学习、自动化脚本 | 编译器开发、系统编程、金融建模 |
5
+ | **特点** | 动态类型、面向对象、元编程能力强 | 动态类型、面向对象、大量第三方库支持 | 静态类型、函数式编程、模块化、高性能 |
6
+ | **语法示例** | ```ruby<br>class Greeter<br> def initialize(name)<br> @name = name<br> end<br><br> def greet<br> puts "Hello, #{@name}!"<br> end<br>end<br><br>greeter = Greeter.new("World")<br>greeter.greet``` | ```python<br>class Greeter:<br> def __init__(self, name):<br> self.name = name<br><br> def greet(self):<br> print(f"Hello, {self.name}!")<br><br>greeter = Greeter("World")<br>greeter.greet()``` | ```ocaml<br>type greeter = {<br> name: string;<br>}<br><br>let greet {name} =<br> print_endline ("Hello, " ^ name ^ "!")<br><br>let greeter = {name = "World"}<br>let () = greet greeter``` |
7
+ | **性能** | 相对较低,但通过 JRuby、TruffleRuby 可以提高性能 | 适中,通过 Cython、PyPy 可以显著提高性能 | 高,编译后的代码接近 C 语言的性能 |
8
+ | **优点** | 开发速度快,社区活跃 | 丰富的第三方库,应用广泛 | 适合性能关键的应用,编译器优化能力强 |
9
+ | **生态系统** | Ruby on Rails 是最著名的 Web 框架,其他库和工具也很丰富 | 丰富的第三方库,如 NumPy、Pandas、Django、Flask | 相对较少,但有高质量的库,如 Core 和 Async |
10
+ | **学习曲线** | 较平缓,语法灵活,适合初学者 | 平坦,语法清晰,适合初学者 | 较陡峭,需要理解函数式编程概念 |
11
+ | **文档** | 丰富,社区支持好 | 丰富,有大量的教程和资源 | 适中,但社区提供的资源质量高 |
12
+ | **适用场景** | 快速开发、Web 应用、自动化脚本 | 数据科学、机器学习、Web 开发、自动化脚本 | 性能关键的应用、编译器开发、金融建模 |
13
+