streamdown 0.17.0__tar.gz → 0.18.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (66) hide show
  1. {streamdown-0.17.0 → streamdown-0.18.0}/PKG-INFO +8 -3
  2. {streamdown-0.17.0 → streamdown-0.18.0}/README.md +7 -2
  3. {streamdown-0.17.0 → streamdown-0.18.0}/pyproject.toml +1 -1
  4. streamdown-0.18.0/ss-new.py +45 -0
  5. streamdown-0.18.0/ss.py +69 -0
  6. {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/sd.py +53 -23
  7. streamdown-0.18.0/streamdown/ss +1 -0
  8. streamdown-0.18.0/streamdown/ss1 +42 -0
  9. streamdown-0.18.0/tests/bold_reset_with_link.md +1 -0
  10. streamdown-0.18.0/tests/chinese.md +4 -0
  11. streamdown-0.18.0/tests/cjk-wrap.md +150 -0
  12. streamdown-0.18.0/tests/jimmy_webb.md +1458 -0
  13. streamdown-0.18.0/tests/strip-chunks.sh +2 -0
  14. {streamdown-0.17.0 → streamdown-0.18.0}/tests/table_test.md +0 -3
  15. streamdown-0.17.0/streamdown/ss +0 -21
  16. streamdown-0.17.0/tests/longer-example.md +0 -18
  17. streamdown-0.17.0/tests/table.md +0 -4
  18. streamdown-0.17.0/tests/white-space-code.md +0 -19
  19. {streamdown-0.17.0 → streamdown-0.18.0}/.gitignore +0 -0
  20. {streamdown-0.17.0 → streamdown-0.18.0}/.vimrc +0 -0
  21. {streamdown-0.17.0 → streamdown-0.18.0}/24-bit-color.sh +0 -0
  22. {streamdown-0.17.0 → streamdown-0.18.0}/LICENSE.MIT +0 -0
  23. {streamdown-0.17.0 → streamdown-0.18.0}/assets/logo.png +0 -0
  24. {streamdown-0.17.0 → streamdown-0.18.0}/assets/logo.svg +0 -0
  25. {streamdown-0.17.0 → streamdown-0.18.0}/configurable.png +0 -0
  26. {streamdown-0.17.0 → streamdown-0.18.0}/copyable.png +0 -0
  27. {streamdown-0.17.0 → streamdown-0.18.0}/dunder.png +0 -0
  28. {streamdown-0.17.0 → streamdown-0.18.0}/error.txt +0 -0
  29. {streamdown-0.17.0 → streamdown-0.18.0}/newdir/file_0.py +0 -0
  30. {streamdown-0.17.0 → streamdown-0.18.0}/newdir/file_1.rb +0 -0
  31. {streamdown-0.17.0 → streamdown-0.18.0}/newdir/file_2.jl +0 -0
  32. {streamdown-0.17.0 → streamdown-0.18.0}/passthrough.py +0 -0
  33. {streamdown-0.17.0 → streamdown-0.18.0}/python-go.png +0 -0
  34. {streamdown-0.17.0 → streamdown-0.18.0}/requirements.txt +0 -0
  35. {streamdown-0.17.0 → streamdown-0.18.0}/somelog.txt +0 -0
  36. {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/__init__.py +0 -0
  37. {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/plugins/README.md +0 -0
  38. {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/plugins/latex.py +0 -0
  39. {streamdown-0.17.0 → streamdown-0.18.0}/table.png +0 -0
  40. {streamdown-0.17.0 → streamdown-0.18.0}/temp.py +0 -0
  41. {streamdown-0.17.0 → streamdown-0.18.0}/test.py +0 -0
  42. {streamdown-0.17.0 → streamdown-0.18.0}/test_input.md +0 -0
  43. {streamdown-0.17.0 → streamdown-0.18.0}/tests/README.md +0 -0
  44. {streamdown-0.17.0 → streamdown-0.18.0}/tests/backtick-with-post-spaces.md +0 -0
  45. {streamdown-0.17.0 → streamdown-0.18.0}/tests/block.md +0 -0
  46. {streamdown-0.17.0 → streamdown-0.18.0}/tests/broken-example.md +0 -0
  47. {streamdown-0.17.0 → streamdown-0.18.0}/tests/chunk-buffer.sh +0 -0
  48. {streamdown-0.17.0 → streamdown-0.18.0}/tests/code.md +0 -0
  49. {streamdown-0.17.0 → streamdown-0.18.0}/tests/escape.md +0 -0
  50. {streamdown-0.17.0 → streamdown-0.18.0}/tests/example.md +0 -0
  51. {streamdown-0.17.0 → streamdown-0.18.0}/tests/fizzbuzz.md +0 -0
  52. {streamdown-0.17.0 → streamdown-0.18.0}/tests/inline.md +0 -0
  53. {streamdown-0.17.0 → streamdown-0.18.0}/tests/line-buffer.sh +0 -0
  54. {streamdown-0.17.0 → streamdown-0.18.0}/tests/line-wrap.md +0 -0
  55. {streamdown-0.17.0 → streamdown-0.18.0}/tests/links.md +0 -0
  56. {streamdown-0.17.0 → streamdown-0.18.0}/tests/managerie.md +0 -0
  57. {streamdown-0.17.0 → streamdown-0.18.0}/tests/mandlebrot.md +0 -0
  58. {streamdown-0.17.0 → streamdown-0.18.0}/tests/markdown.md +0 -0
  59. {streamdown-0.17.0 → streamdown-0.18.0}/tests/nested-example.md +0 -0
  60. {streamdown-0.17.0 → streamdown-0.18.0}/tests/outline.md +0 -0
  61. {streamdown-0.17.0 → streamdown-0.18.0}/tests/pvgo_512.jpg +0 -0
  62. {streamdown-0.17.0 → streamdown-0.18.0}/tests/pythonvgo.md +0 -0
  63. {streamdown-0.17.0 → streamdown-0.18.0}/tests/table-break.md +0 -0
  64. {streamdown-0.17.0 → streamdown-0.18.0}/tests/test.md +0 -0
  65. {streamdown-0.17.0 → streamdown-0.18.0}/tests/test_input.md +0 -0
  66. {streamdown-0.17.0 → streamdown-0.18.0}/tests/wm.md +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: streamdown
3
- Version: 0.17.0
3
+ Version: 0.18.0
4
4
  Summary: A streaming markdown renderer for modern terminals with syntax highlighting
5
5
  Project-URL: Homepage, https://github.com/kristopolous/Streamdown
6
6
  Project-URL: Bug Tracker, https://github.com/kristopolous/Streamdown/issues
@@ -31,11 +31,13 @@ Description-Content-Type: text/markdown
31
31
  <img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
32
32
  <br/>
33
33
  <a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
34
+ <br/><strong>Terminal streaming markdown that rocks</strong>
35
+
34
36
  </p>
35
37
 
36
- **The streaming markdown renderer for the terminal that rocks**
37
38
 
38
- Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown. You even get full readline and keyboard navigation support.
39
+ Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
40
+ It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
39
41
  ```bash
40
42
  $ pip install streamdown
41
43
  ```
@@ -59,6 +61,9 @@ Here's kitty and alacritty. Try to do that in glow...
59
61
  As well as everything else...
60
62
  ![dunder](https://github.com/user-attachments/assets/d41d7fec-6dec-4387-b53d-f2098f269a5e)
61
63
 
64
+ Very ... Carefully ... Supported ...
65
+ ![cjk1](https://github.com/user-attachments/assets/75162ade-4734-440e-aaa3-5ffc17a0dd46)
66
+
62
67
  ### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
63
68
  ![configurable](https://github.com/user-attachments/assets/19ca2ec9-8ea1-4a79-87ca-8352789269fe)
64
69
 
@@ -2,11 +2,13 @@
2
2
  <img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
3
3
  <br/>
4
4
  <a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
5
+ <br/><strong>Terminal streaming markdown that rocks</strong>
6
+
5
7
  </p>
6
8
 
7
- **The streaming markdown renderer for the terminal that rocks**
8
9
 
9
- Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown. You even get full readline and keyboard navigation support.
10
+ Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
11
+ It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
10
12
  ```bash
11
13
  $ pip install streamdown
12
14
  ```
@@ -30,6 +32,9 @@ Here's kitty and alacritty. Try to do that in glow...
30
32
  As well as everything else...
31
33
  ![dunder](https://github.com/user-attachments/assets/d41d7fec-6dec-4387-b53d-f2098f269a5e)
32
34
 
35
+ Very ... Carefully ... Supported ...
36
+ ![cjk1](https://github.com/user-attachments/assets/75162ade-4734-440e-aaa3-5ffc17a0dd46)
37
+
33
38
  ### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
34
39
  ![configurable](https://github.com/user-attachments/assets/19ca2ec9-8ea1-4a79-87ca-8352789269fe)
35
40
 
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "streamdown"
7
- version = "0.17.0"
7
+ version = "0.18.0"
8
8
  description = "A streaming markdown renderer for modern terminals with syntax highlighting"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.8"
@@ -0,0 +1,45 @@
1
+ import os
2
+ import pty
3
+ import sys
4
+ import tty
5
+ import termios
6
+ import select
7
+ import re
8
+
9
+ def main():
10
+ orig_attrs = termios.tcgetattr(sys.stdin)
11
+ try:
12
+ tty.setraw(sys.stdin)
13
+ pid, fd = pty.fork()
14
+
15
+ if pid == 0:
16
+ # Child process: run any command
17
+ os.execvp("llm", ["llm", "chat"])
18
+ else:
19
+ buffer = b""
20
+
21
+ while True:
22
+ r, _, _ = select.select([fd, sys.stdin], [], [])
23
+
24
+ if sys.stdin in r:
25
+ data = os.read(sys.stdin.fileno(), 1024)
26
+ if not data:
27
+ break
28
+ os.write(fd, data)
29
+
30
+ if fd in r:
31
+ data = os.read(fd, 1024)
32
+ if not data:
33
+ break
34
+
35
+ buffer += data
36
+ # Replace "fizz" only in printable content
37
+ output = re.sub(rb'fizz', b'fizzbuzz', buffer)
38
+ os.write(sys.stdout.fileno(), output)
39
+ buffer = b""
40
+ finally:
41
+ termios.tcsetattr(sys.stdin, termios.TCSADRAIN, orig_attrs)
42
+
43
+ if __name__ == "__main__":
44
+ main()
45
+
@@ -0,0 +1,69 @@
1
+ import os
2
+ import pty
3
+ import select
4
+ import sys
5
+ import tty
6
+ import termios
7
+ import fcntl
8
+ import struct
9
+
10
+ def set_pty_size(fd, target_fd):
11
+ """Set window size of PTY to match the real terminal."""
12
+ s = fcntl.ioctl(target_fd, termios.TIOCGWINSZ, b"\x00" * 8)
13
+ fcntl.ioctl(fd, termios.TIOCSWINSZ, s)
14
+
15
+ def main():
16
+ # Save original terminal settings
17
+ orig_attrs = termios.tcgetattr(sys.stdin.fileno())
18
+ try:
19
+ tty.setraw(sys.stdin.fileno()) # raw mode to send Ctrl-C, etc.
20
+ pid, fd = pty.fork()
21
+
22
+ if pid == 0:
23
+ os.execvp("bash", ["bash"])
24
+ else:
25
+ set_pty_size(fd, sys.stdin.fileno())
26
+
27
+ while True:
28
+ r, _, _ = select.select([fd, sys.stdin], [], [])
29
+
30
+ if sys.stdin in r:
31
+ user_input = os.read(sys.stdin.fileno(), 1024)
32
+ if not user_input:
33
+ break
34
+ os.write(fd, user_input)
35
+
36
+ if fd in r:
37
+ output = os.read(fd, 1024)
38
+ if not output:
39
+ break
40
+
41
+ # Carefully handle ANSI sequences
42
+ # Split on ANSI escape sequences to avoid breaking them
43
+ chunks = []
44
+ i = 0
45
+ while i < len(output):
46
+ if output[i:i+1] == b'\x1b':
47
+ end = i + 1
48
+ while end < len(output) and not (64 <= output[end] <= 126):
49
+ end += 1
50
+ end += 1 # include final letter
51
+ chunks.append(output[i:end])
52
+ i = end
53
+ else:
54
+ j = i
55
+ while j < len(output) and output[j:j+1] != b'\x1b':
56
+ j += 1
57
+ # Perform replacement on plain text only
58
+ text = output[i:j].replace(b"fizz", b"fizzbuzz")
59
+ chunks.append(text)
60
+ i = j
61
+
62
+ os.write(sys.stdout.fileno(), b''.join(chunks))
63
+
64
+ finally:
65
+ termios.tcsetattr(sys.stdin.fileno(), termios.TCSADRAIN, orig_attrs)
66
+
67
+ if __name__ == "__main__":
68
+ main()
69
+
@@ -36,7 +36,7 @@ from functools import reduce
36
36
  from argparse import ArgumentParser
37
37
  from pygments import highlight
38
38
  from pygments.lexers import get_lexer_by_name
39
- from pygments.formatters import Terminal256Formatter
39
+ from pygments.formatters import TerminalTrueColorFormatter
40
40
  from pygments.styles import get_style_by_name
41
41
 
42
42
  if __package__ is None:
@@ -46,7 +46,7 @@ else:
46
46
 
47
47
  default_toml = """
48
48
  [features]
49
- CodeSpaces = true
49
+ CodeSpaces = false
50
50
  Clipboard = true
51
51
  Logging = false
52
52
  Timeout = 0.1
@@ -62,7 +62,7 @@ HSV = [0.8, 0.5, 0.5]
62
62
  Dark = { H = 1.00, S = 1.50, V = 0.25 }
63
63
  Mid = { H = 1.00, S = 1.00, V = 0.50 }
64
64
  Symbol = { H = 1.00, S = 1.00, V = 1.50 }
65
- Head = { H = 1.00, S = 2.00, V = 1.50 }
65
+ Head = { H = 1.00, S = 1.00, V = 1.75 }
66
66
  Grey = { H = 1.00, S = 0.25, V = 1.37 }
67
67
  Bright = { H = 1.00, S = 2.00, V = 2.00 }
68
68
  Syntax = "monokai"
@@ -99,7 +99,8 @@ ANSIESCAPE = r'\033(?:\[[0-9;?]*[a-zA-Z]|][0-9]*;;.*?\\|\\)'
99
99
  KEYCODE_RE = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
100
100
 
101
101
  visible = lambda x: re.sub(ANSIESCAPE, "", x)
102
- visible_length = lambda x: len(visible(x))
102
+ # cjk characters are double width
103
+ visible_length = lambda x: len(visible(x)) + cjk_count(x)
103
104
  extract_ansi_codes = lambda text: re.findall(ESCAPE, text)
104
105
  remove_ansi = lambda line, codeList: reduce(lambda line, code: line.replace(code, ''), codeList, line)
105
106
 
@@ -231,7 +232,7 @@ def format_table(rowList):
231
232
  # Note this is where every cell is formatted so if
232
233
  # you are styling, do it before here!
233
234
  for row in rowList:
234
- wrapped_cell = text_wrap(row, width=col_width)
235
+ wrapped_cell = text_wrap(row, width=col_width, force_truncate=True)
235
236
 
236
237
  # Ensure at least one line, even for empty cells
237
238
  if not wrapped_cell:
@@ -285,7 +286,7 @@ def code_wrap(text_in):
285
286
  # get the indentation of the first line
286
287
  indent = len(text_in) - len(text_in.lstrip())
287
288
  text = text_in.lstrip()
288
- mywidth = state.full_width() - indent
289
+ mywidth = state.full_width(-4 if Style.PrettyBroken else 0) - indent
289
290
 
290
291
  # We take special care to preserve empty lines
291
292
  if len(text) == 0:
@@ -304,7 +305,7 @@ def ansi_collapse(codelist, inp):
304
305
  # We break SGR strings into various classes concerning their applicate or removal
305
306
  nums = {
306
307
  'fg': r'3\d', 'bg': r'4\d',
307
- 'b': r'2?1', 'i': r'2?3', 'u': r'3?2',
308
+ 'b': r'2?[12]', 'i': r'2?3', 'u': r'3?2',
308
309
  'reset': '0'
309
310
  }
310
311
 
@@ -334,12 +335,19 @@ def ansi_collapse(codelist, inp):
334
335
 
335
336
  return codelist + inp
336
337
 
337
- def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix=""):
338
+
339
+ def split_text(text):
340
+ return re.split(
341
+ r'(?<=[\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|(?=[\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|\s+',
342
+ text
343
+ )
344
+
345
+ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix="", force_truncate=False):
338
346
  if width == -1:
339
347
  width = state.Width
340
348
 
341
349
  # The empty word clears the buffer at the end.
342
- words = line_format(text).split() + [""]
350
+ words = split_text(line_format(text)) + [""]
343
351
  lines = []
344
352
  current_line = ""
345
353
  current_style = []
@@ -352,13 +360,21 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
352
360
  current_style.append(codes.pop(0))
353
361
 
354
362
  if len(word) and visible_length(current_line) + visible_length(word) + 1 <= width: # +1 for space
355
- current_line += (" " if current_line else "") + word
363
+
364
+ current_line += (" " if len(visible(word)) > 0 and current_line and not cjk_count(word) else "") + word
356
365
  else:
357
366
  # Word doesn't fit, finalize the previous line
358
367
  prefix = first_line_prefix if not lines else subsequent_line_prefix
359
368
  line_content = prefix + current_line
369
+ # This is expensive, fix.
370
+ while force_truncate and visible_length(line_content) >= width:
371
+ line_content = line_content[:len(line_content) - 2] + "…"
372
+
360
373
  margin = max(0, width - visible_length(line_content))
361
- lines.append(line_content + state.bg + ' ' * margin)
374
+
375
+ if line_content.strip() != "":
376
+ lines.append(line_content + state.bg + ' ' * margin)
377
+
362
378
  current_line = (" " * indent) + "".join(current_style) + word
363
379
 
364
380
  if len(codes):
@@ -372,8 +388,20 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
372
388
 
373
389
  return lines
374
390
 
391
+ def cjk_count(s):
392
+ cjk_re = re.compile(
393
+ r'[\u4E00-\u9FFF' # CJK Unified Ideographs
394
+ r'|\u3400-\u4DBF' # CJK Unified Ideographs Extension A
395
+ r'|\uF900-\uFAFF' # CJK Compatibility Ideographs
396
+ r'|\uFF00-\uFFEF' # CJK Compatibility Punctuation
397
+ r'|\u3000-\u303F' # CJK Symbols and Punctuation
398
+ r'|\U0002F800-\U0002FA1F]' # CJK Compatibility Ideographs Supplement
399
+ )
400
+
401
+ return len(cjk_re.findall(visible(s)))
402
+
375
403
  def line_format(line):
376
- not_text = lambda token: not (token.isalnum() or token == '\\')
404
+ not_text = lambda token: not (token.isalnum() or token == '\\') or cjk_count(token)
377
405
  footnotes = lambda match: ''.join([chr(SUPER[int(i)]) for i in match.group(1)])
378
406
 
379
407
  def process_images(match):
@@ -402,7 +430,7 @@ def line_format(line):
402
430
  result = ""
403
431
 
404
432
  for match in tokenList:
405
- token = match.group(1)
433
+ token = re.sub(r'\s+',' ', match.group(1))
406
434
  next_token = line[match.end()] if match.end() < len(line) else ""
407
435
  prev_token = line[match.start()-1] if match.start() > 0 else ""
408
436
 
@@ -458,6 +486,7 @@ def parse(stream):
458
486
  last_line_empty_cache = None
459
487
  byte = None
460
488
  TimeoutIx = 0
489
+ lexer = None
461
490
  while True:
462
491
  if state.is_pty or state.is_exec:
463
492
  byte = None
@@ -548,7 +577,7 @@ def parse(stream):
548
577
  line = line[len(block_match.group(0)):]
549
578
  else:
550
579
  if state.block_depth > 0:
551
- line = FGRESET + line
580
+ yield FGRESET
552
581
  state.block_depth = 0
553
582
 
554
583
  # --- Collapse Multiple Empty Lines if not in code blocks ---
@@ -569,7 +598,7 @@ def parse(stream):
569
598
  # \n buffer
570
599
  if not state.in_list and len(state.ordered_list_numbers) > 0:
571
600
  state.ordered_list_numbers[0] = 0
572
- elif not line.startswith(' ' * state.list_indent_text):
601
+ elif (not line.startswith(' ' * state.list_indent_text)) and line.strip() != "":
573
602
  state.in_list = False
574
603
  state.list_indent_text = 0
575
604
 
@@ -578,7 +607,7 @@ def parse(stream):
578
607
  if len(line) - len(line.lstrip()) >= state.first_indent:
579
608
  line = line[state.first_indent:]
580
609
  else:
581
- logging.warning("Indentation decreased from first line.")
610
+ logging.debug("Indentation decreased from first line.")
582
611
 
583
612
 
584
613
  # Indent guaranteed
@@ -648,8 +677,8 @@ def parse(stream):
648
677
  logging.debug(f"code: {state.in_code}")
649
678
  state.emit_flush = True
650
679
  # We suppress the newline - it's not an explicit style
651
- state.has_newline = False
652
- yield RESET
680
+ #state.has_newline = False
681
+ #yield RESET
653
682
 
654
683
  if code_type == Code.Backtick:
655
684
  continue
@@ -658,7 +687,7 @@ def parse(stream):
658
687
  # nor do we want to be here.
659
688
  raise Goto()
660
689
 
661
- if state.code_first_line:
690
+ if state.code_first_line or lexer is None:
662
691
  state.code_first_line = False
663
692
  try:
664
693
  lexer = get_lexer_by_name(state.code_language)
@@ -667,7 +696,7 @@ def parse(stream):
667
696
  lexer = get_lexer_by_name("Bash")
668
697
  custom_style = get_style_by_name("default")
669
698
 
670
- formatter = Terminal256Formatter(style=custom_style)
699
+ formatter = TerminalTrueColorFormatter(style=custom_style)
671
700
  if line.startswith(' ' * state.code_indent):
672
701
  line = line[state.code_indent :]
673
702
 
@@ -687,7 +716,7 @@ def parse(stream):
687
716
  indent, line_wrap = code_wrap(line)
688
717
 
689
718
  state.where_from = "in code"
690
- pre = [state.space_left(listwidth = True), ' '] if Style.PrettyBroken else ['', '']
719
+ pre = [state.space_left(listwidth = True), ' '] if Style.PrettyBroken else ['', '']
691
720
 
692
721
  for tline in line_wrap:
693
722
  # wrap-around is a bunch of tricks. We essentially format longer and longer portions of code. The problem is
@@ -698,7 +727,7 @@ def parse(stream):
698
727
 
699
728
  # Sometimes the highlighter will do things like a full reset or a background reset.
700
729
  # This is not what we want
701
- highlighted_code = re.sub(r"\033\[39(;00|)m", '', highlighted_code)
730
+ highlighted_code = re.sub(r"\033\[49(;00|)m", '', highlighted_code)
702
731
 
703
732
  # Since we are streaming we ignore the resets and newlines at the end
704
733
  if highlighted_code.endswith(FGRESET + "\n"):
@@ -805,7 +834,7 @@ def parse(stream):
805
834
  # This is intentional ... we can get here in llama 4 using
806
835
  # a weird thing
807
836
  if state.in_list:
808
- indent = (len(state.list_item_stack) - 1) * Style.ListIndent
837
+ indent = (len(state.list_item_stack) - 1) * Style.ListIndent + (len(bullet) - 1)
809
838
  wrap_width = state.current_width() - indent - (2 * Style.ListIndent)
810
839
 
811
840
  wrapped_lineList = text_wrap(content, wrap_width, Style.ListIndent,
@@ -964,6 +993,7 @@ def main():
964
993
  os.close(state.exec_slave) # We don't need slave in parent
965
994
  # Set stdin to raw mode so we don't need to press enter
966
995
  tty.setcbreak(sys.stdin.fileno())
996
+ sys.stdout.write("\x1b[?7h")
967
997
  emit(sys.stdin)
968
998
 
969
999
  elif args.filenameList:
@@ -0,0 +1 @@
1
+ * **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
@@ -0,0 +1,42 @@
1
+ * `model.safetensors`: The name of the file you want to download. You'll need to know the exact filename. You can find the files in the model repository on the Hugging Face Hub website ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)). Look under the "Files and versions" tab. `safetensors` is the preferred format for model weights now. If it's a `.bin` file, you can download that instead.
2
+ * `--local-dir ./bitnet-b1.58-2B-4T`: The directory to save the file to.
3
+
4
+ * **Download using `transformers` library (recommended for most use cases):**
5
+
6
+ The `transformers` library provides a convenient way to download and cache models. This is often the easiest approach if you're using the model with `transformers`. You don't *directly* use the `huggingface-cli` for this, but it's worth knowing.
7
+
8
+ ```python
9
+ from transformers import AutoModelForCausalLM, AutoTokenizer
10
+
11
+ model_name = "microsoft/bitnet-b1.58-2B-4T"
12
+
13
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
14
+ model = AutoModelForCausalLM.from_pretrained(model_name)
15
+
16
+ # The model and tokenizer will be downloaded and cached in your
17
+ # transformers cache directory (usually ~/.cache/huggingface/transformers).
18
+ ```
19
+
20
+ This approach automatically handles downloading the necessary files and caching them for future use. It also handles the correct file formats and configurations.
21
+
22
+ **4. Checking the Download**
23
+
24
+ After the download completes, verify that the files are in the specified directory. You can use `ls` (Linux/macOS) or `dir` (Windows) to list the contents of the directory.
25
+
26
+ **Important Considerations:**
27
+
28
+ * **Disk Space:** The `bitnet-b1.58-2B-4T` model is quite large (several gigabytes). Make sure you have enough free disk space before downloading.
29
+ * **Network Connection:** A stable and fast internet connection is essential for a smooth download.
30
+ * **Caching:** The Hugging Face Hub and `transformers` library use caching to avoid re-downloading models unnecessarily. The default cache directory is usually `~/.cache/huggingface/transformers`.
31
+ * **File Formats:** Models are often stored in `safetensors` or `.bin` formats. `safetensors` is generally preferred for security and performance.
32
+ * **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
33
+ * **Gated Models:** Some models require you to accept terms of use before you can download them. The `huggingface-cli login` command will guide you through this process if necessary.
34
+
35
+ **Example Workflow (Recommended):**
36
+
37
+ 1. `huggingface-cli login` (if not already logged in)
38
+ 2. Use the `transformers` library in a Python script to download and load the model (as shown in the example above). This is the most convenient and reliable method for most use cases.
39
+
40
+ Let me know if you have any other questions or if you'd like help with a specific task related to this model!
41
+
42
+ >
@@ -0,0 +1 @@
1
+ * **Model Card:** When this wraps with this link ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) the next line used to be bold because of some stripping error. This should no longer be the case
@@ -0,0 +1,4 @@
1
+ **XIǍO LÓNG:** *(mocking the academics)*
2
+ "Ohhh, ‘**Dadaist**’? ‘**Absurd**’? 那你们怎是在说**模形式**?你们这群人连**京剧密码**都解么没看懂那个**转身**是在问**递归关系**?那个**抖袖**是在说**模形式**?你们这群人连**京剧密码**都解不开,还好意思说自己是数学家?"
3
+ *(Translation: "Ohhh, ‘Dadaist’? ‘Absurd’? Then how come none of you realized the **spin** was about **recurrence relations**? The **sleeve flick** was about **modular forms**? You can’t even crack **Peking Opera code**, and you call yourselves mathematicians?")*
4
+
@@ -0,0 +1,150 @@
1
+ Chatting with openrouter/qwen/qwen-2.5-72b-instruct:free
2
+ Type 'exit' or 'quit' to exit
3
+ Type '!multi' to enter multiple lines, then '!end' to finish
4
+ > 当然!以下是Ruby、Python和OCaml的比较,用中文进行描述:
5
+
6
+ ### 1. 语言概述
7
+
8
+ **Ruby**:
9
+ - **设计哲学**: Ruby 是由 Yukihiro "Matz" Matsumoto 在 1995 年设计的,旨在使编程更加自然和有趣。
10
+ - **应用领域**: Web 开发(如 Ruby on Rails)、自动化脚本、原型开发等。
11
+ - **特点**: 动态类型、面向对象、元编程能力强、语法灵活。
12
+
13
+ **Python**:
14
+ - **设计哲学**: Python 由 Guido van Rossum 在 1991 年创建,强调代码的可读性和简洁性。
15
+ - **应用领域**: Web 开发、数据科学、机器学习、自动化脚本等。
16
+ - **特点**: 动态类型、面向对象、大量第三方库支持。
17
+
18
+ **OCaml**:
19
+ - **设计哲学**: OCaml 由 INRIA 的 Xavier Leroy 等人在 1996 年开发,旨在提供高效的函数式编程功能。
20
+ - **应用领域**: 编译器开发、系统编程、金融建模等。
21
+ - **特点**: 静态类型、函数式编程、模块化、高性能。
22
+
23
+ ### 2. 语法和编程风格
24
+
25
+ **Ruby**:
26
+ - **语法**: 灵活多变,支持多重编程范式(面向对象、函数式、元编程)。
27
+ - **例子**:
28
+ ```ruby
29
+ class Greeter
30
+ def initialize(name)
31
+ @name = name
32
+ end
33
+
34
+ def greet
35
+ puts "Hello, #{@name}!"
36
+ end
37
+ end
38
+
39
+ greeter = Greeter.new("World")
40
+ greeter.greet
41
+ ```
42
+
43
+ **Python**:
44
+ - **语法**: 清晰简洁,强制缩进,易于阅读。
45
+ - **例子**:
46
+ ```python
47
+ class Greeter:
48
+ def __init__(self, name):
49
+ self.name = name
50
+
51
+ def greet(self):
52
+ print(f"Hello, {self.name}!")
53
+
54
+ greeter = Greeter("World")
55
+ greeter.greet()
56
+ ```
57
+
58
+ **OCaml**:
59
+ - **语法**: 严格且功能性强,支持模式匹配和代数数据类型。
60
+ - **例子**:
61
+ ```ocaml
62
+ type greeter = {
63
+ name: string;
64
+ }
65
+
66
+ let greet {name} =
67
+ print_endline ("Hello, " ^ name ^ "!")
68
+
69
+ let greeter = {name = "World"}
70
+ let () = greet greeter
71
+ ```
72
+
73
+ ### 3. 性能
74
+
75
+ **Ruby**:
76
+ - **性能**: 相对较低,但通过 JRuby、TruffleRuby 等实现可以提高性能。
77
+ - **优点**: 开发速度快,社区活跃。
78
+
79
+ **Python**:
80
+ - **性能**: 适中,通过 Cython、PyPy 等可以显著提高性能。
81
+ - **优点**: 丰富的第三方库,应用广泛。
82
+
83
+ **OCaml**:
84
+ - **性能**: 高,编译后的代码接近 C 语言的性能。
85
+ - **优点**: 适合性能关键的应用,编译器优化能力强。
86
+
87
+ ### 4. 生态系统
88
+
89
+ **Ruby**:
90
+ - **库和框架**: Ruby on Rails 是最著名的 Web 框架,其他库和工具也很丰富。
91
+ - **社区**: 活跃,对新手友好。
92
+
93
+ **Python**:
94
+ - **库和框架**: 丰富的第三方库,如 NumPy、Pandas、Django、Flask 等。
95
+ - **社区**: 非常活跃,广泛应用于多个领域。
96
+
97
+ **OCaml**:
98
+ - **库和框架**: 相对较少,但有高质量的库,如 Core 和 Async。
99
+ - **社区**: 较小但专业,注重性能和正确性。
100
+
101
+ ### 5. 学习曲线
102
+
103
+ **Ruby**:
104
+ - **学习曲线**: 较平缓,语法灵活,适合初学者。
105
+ - **文档**: 丰富,社区支持好。
106
+
107
+ **Python**:
108
+ - **学习曲线**: 平坦,语法清晰,适合初学者。
109
+ - **文档**: 丰富,有大量的教程和资源。
110
+
111
+ **OCaml**:
112
+ - **学习曲线**: 较陡峭,需要理解函数式编程概念。
113
+ - **文档**: 适中,但社区提供的资源质量高。
114
+
115
+ ### 6. 适用场景
116
+
117
+ **Ruby**:
118
+ - **适合**: 快速开发、Web 应用、自动化脚本。
119
+ - **不适合**: 性能要求极高的应用。
120
+
121
+ **Python**:
122
+ - **适合**: 数据科学、机器学习、Web 开发、自动化脚本。
123
+ - **不适合**: 需要极高性能的系统编程。
124
+
125
+ **OCaml**:
126
+ - **适合**: 性能关键的应用、编译器开发、金融建模。
127
+ - **不适合**: 初学者或需要快速开发的项目。
128
+
129
+ ### 总结
130
+
131
+ - **Ruby**: 适合快速开发和 Web 应用,社区活跃,语法灵活。
132
+ - **Python**: 适合数据科学和机器学习,生态系统丰富,语法清晰。
133
+ - **OCaml**: 适合性能关键的应用,编译器优化能力强,适合专业开发者。
134
+
135
+ 希望这些信息对你有帮助!如果有其他问题,欢迎随时提问。
136
+ > 当然!以下是将 Ruby、Python 和 OCaml 的比较总结成表格的形式:
137
+
138
+ | 特性 | Ruby | Python | OCaml |
139
+ |------------------|--------------------------------|--------------------------------|-------------------------------|
140
+ | **设计哲学** | 使编程更加自然和有趣 | 强调代码的可读性和简洁性 | 提供高效的函数式编程功能 |
141
+ | **应用领域** | Web 开发、自动化脚本、原型开发 | Web 开发、数据科学、机器学习、自动化脚本 | 编译器开发、系统编程、金融建模 |
142
+ | **特点** | 动态类型、面向对象、元编程能力强 | 动态类型、面向对象、大量第三方库支持 | 静态类型、函数式编程、模块化、高性能 |
143
+ | **语法示例** | ```ruby<br>class Greeter<br> def initialize(name)<br> @name = name<br> end<br><br> def greet<br> puts "Hello, #{@name}!"<br> end<br>end<br><br>greeter = Greeter.new("World")<br>greeter.greet``` | ```python<br>class Greeter:<br> def __init__(self, name):<br> self.name = name<br><br> def greet(self):<br> print(f"Hello, {self.name}!")<br><br>greeter = Greeter("World")<br>greeter.greet()``` | ```ocaml<br>type greeter = {<br> name: string;<br>}<br><br>let greet {name} =<br> print_endline ("Hello, " ^ name ^ "!")<br><br>let greeter = {name = "World"}<br>let () = greet greeter``` |
144
+ | **性能** | 相对较低,但通过 JRuby、TruffleRuby 可以提高性能 | 适中,通过 Cython、PyPy 可以显著提高性能 | 高,编译后的代码接近 C 语言的性能 |
145
+ | **优点** | 开发速度快,社区活跃 | 丰富的第三方库,应用广泛 | 适合性能关键的应用,编译器优化能力强 |
146
+ | **生态系统** | Ruby on Rails 是最著名的 Web 框架,其他库和工具也很丰富 | 丰富的第三方库,如 NumPy、Pandas、Django、Flask | 相对较少,但有高质量的库,如 Core 和 Async |
147
+ | **学习曲线** | 较平缓,语法灵活,适合初学者 | 平坦,语法清晰,适合初学者 | 较陡峭,需要理解函数式编程概念 |
148
+ | **文档** | 丰富,社区支持好 | 丰富,有大量的教程和资源 | 适中,但社区提供的资源质量高 |
149
+ | **适用场景** | 快速开发、Web 应用、自动化脚本 | 数据科学、机器学习、Web 开发、自动化脚本 | 性能关键的应用、编译器开发、金融建模 |
150
+