streamdown 0.17.0__tar.gz → 0.18.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {streamdown-0.17.0 → streamdown-0.18.0}/PKG-INFO +8 -3
- {streamdown-0.17.0 → streamdown-0.18.0}/README.md +7 -2
- {streamdown-0.17.0 → streamdown-0.18.0}/pyproject.toml +1 -1
- streamdown-0.18.0/ss-new.py +45 -0
- streamdown-0.18.0/ss.py +69 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/sd.py +53 -23
- streamdown-0.18.0/streamdown/ss +1 -0
- streamdown-0.18.0/streamdown/ss1 +42 -0
- streamdown-0.18.0/tests/bold_reset_with_link.md +1 -0
- streamdown-0.18.0/tests/chinese.md +4 -0
- streamdown-0.18.0/tests/cjk-wrap.md +150 -0
- streamdown-0.18.0/tests/jimmy_webb.md +1458 -0
- streamdown-0.18.0/tests/strip-chunks.sh +2 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/table_test.md +0 -3
- streamdown-0.17.0/streamdown/ss +0 -21
- streamdown-0.17.0/tests/longer-example.md +0 -18
- streamdown-0.17.0/tests/table.md +0 -4
- streamdown-0.17.0/tests/white-space-code.md +0 -19
- {streamdown-0.17.0 → streamdown-0.18.0}/.gitignore +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/.vimrc +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/24-bit-color.sh +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/LICENSE.MIT +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/assets/logo.png +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/assets/logo.svg +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/configurable.png +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/copyable.png +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/dunder.png +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/error.txt +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/newdir/file_0.py +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/newdir/file_1.rb +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/newdir/file_2.jl +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/passthrough.py +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/python-go.png +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/requirements.txt +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/somelog.txt +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/__init__.py +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/plugins/README.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/streamdown/plugins/latex.py +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/table.png +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/temp.py +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/test.py +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/test_input.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/README.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/backtick-with-post-spaces.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/block.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/broken-example.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/chunk-buffer.sh +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/code.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/escape.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/example.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/fizzbuzz.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/inline.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/line-buffer.sh +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/line-wrap.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/links.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/managerie.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/mandlebrot.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/markdown.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/nested-example.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/outline.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/pvgo_512.jpg +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/pythonvgo.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/table-break.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/test.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/test_input.md +0 -0
- {streamdown-0.17.0 → streamdown-0.18.0}/tests/wm.md +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: streamdown
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.18.0
|
|
4
4
|
Summary: A streaming markdown renderer for modern terminals with syntax highlighting
|
|
5
5
|
Project-URL: Homepage, https://github.com/kristopolous/Streamdown
|
|
6
6
|
Project-URL: Bug Tracker, https://github.com/kristopolous/Streamdown/issues
|
|
@@ -31,11 +31,13 @@ Description-Content-Type: text/markdown
|
|
|
31
31
|
<img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
|
|
32
32
|
<br/>
|
|
33
33
|
<a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
|
|
34
|
+
<br/><strong>Terminal streaming markdown that rocks</strong>
|
|
35
|
+
|
|
34
36
|
</p>
|
|
35
37
|
|
|
36
|
-
**The streaming markdown renderer for the terminal that rocks**
|
|
37
38
|
|
|
38
|
-
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown
|
|
39
|
+
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
|
|
40
|
+
It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
|
|
39
41
|
```bash
|
|
40
42
|
$ pip install streamdown
|
|
41
43
|
```
|
|
@@ -59,6 +61,9 @@ Here's kitty and alacritty. Try to do that in glow...
|
|
|
59
61
|
As well as everything else...
|
|
60
62
|

|
|
61
63
|
|
|
64
|
+
Very ... Carefully ... Supported ...
|
|
65
|
+

|
|
66
|
+
|
|
62
67
|
### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
|
|
63
68
|

|
|
64
69
|
|
|
@@ -2,11 +2,13 @@
|
|
|
2
2
|
<img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
|
|
3
3
|
<br/>
|
|
4
4
|
<a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
|
|
5
|
+
<br/><strong>Terminal streaming markdown that rocks</strong>
|
|
6
|
+
|
|
5
7
|
</p>
|
|
6
8
|
|
|
7
|
-
**The streaming markdown renderer for the terminal that rocks**
|
|
8
9
|
|
|
9
|
-
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown
|
|
10
|
+
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
|
|
11
|
+
It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
|
|
10
12
|
```bash
|
|
11
13
|
$ pip install streamdown
|
|
12
14
|
```
|
|
@@ -30,6 +32,9 @@ Here's kitty and alacritty. Try to do that in glow...
|
|
|
30
32
|
As well as everything else...
|
|
31
33
|

|
|
32
34
|
|
|
35
|
+
Very ... Carefully ... Supported ...
|
|
36
|
+

|
|
37
|
+
|
|
33
38
|
### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
|
|
34
39
|

|
|
35
40
|
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
import os
|
|
2
|
+
import pty
|
|
3
|
+
import sys
|
|
4
|
+
import tty
|
|
5
|
+
import termios
|
|
6
|
+
import select
|
|
7
|
+
import re
|
|
8
|
+
|
|
9
|
+
def main():
|
|
10
|
+
orig_attrs = termios.tcgetattr(sys.stdin)
|
|
11
|
+
try:
|
|
12
|
+
tty.setraw(sys.stdin)
|
|
13
|
+
pid, fd = pty.fork()
|
|
14
|
+
|
|
15
|
+
if pid == 0:
|
|
16
|
+
# Child process: run any command
|
|
17
|
+
os.execvp("llm", ["llm", "chat"])
|
|
18
|
+
else:
|
|
19
|
+
buffer = b""
|
|
20
|
+
|
|
21
|
+
while True:
|
|
22
|
+
r, _, _ = select.select([fd, sys.stdin], [], [])
|
|
23
|
+
|
|
24
|
+
if sys.stdin in r:
|
|
25
|
+
data = os.read(sys.stdin.fileno(), 1024)
|
|
26
|
+
if not data:
|
|
27
|
+
break
|
|
28
|
+
os.write(fd, data)
|
|
29
|
+
|
|
30
|
+
if fd in r:
|
|
31
|
+
data = os.read(fd, 1024)
|
|
32
|
+
if not data:
|
|
33
|
+
break
|
|
34
|
+
|
|
35
|
+
buffer += data
|
|
36
|
+
# Replace "fizz" only in printable content
|
|
37
|
+
output = re.sub(rb'fizz', b'fizzbuzz', buffer)
|
|
38
|
+
os.write(sys.stdout.fileno(), output)
|
|
39
|
+
buffer = b""
|
|
40
|
+
finally:
|
|
41
|
+
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, orig_attrs)
|
|
42
|
+
|
|
43
|
+
if __name__ == "__main__":
|
|
44
|
+
main()
|
|
45
|
+
|
streamdown-0.18.0/ss.py
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
import os
|
|
2
|
+
import pty
|
|
3
|
+
import select
|
|
4
|
+
import sys
|
|
5
|
+
import tty
|
|
6
|
+
import termios
|
|
7
|
+
import fcntl
|
|
8
|
+
import struct
|
|
9
|
+
|
|
10
|
+
def set_pty_size(fd, target_fd):
|
|
11
|
+
"""Set window size of PTY to match the real terminal."""
|
|
12
|
+
s = fcntl.ioctl(target_fd, termios.TIOCGWINSZ, b"\x00" * 8)
|
|
13
|
+
fcntl.ioctl(fd, termios.TIOCSWINSZ, s)
|
|
14
|
+
|
|
15
|
+
def main():
|
|
16
|
+
# Save original terminal settings
|
|
17
|
+
orig_attrs = termios.tcgetattr(sys.stdin.fileno())
|
|
18
|
+
try:
|
|
19
|
+
tty.setraw(sys.stdin.fileno()) # raw mode to send Ctrl-C, etc.
|
|
20
|
+
pid, fd = pty.fork()
|
|
21
|
+
|
|
22
|
+
if pid == 0:
|
|
23
|
+
os.execvp("bash", ["bash"])
|
|
24
|
+
else:
|
|
25
|
+
set_pty_size(fd, sys.stdin.fileno())
|
|
26
|
+
|
|
27
|
+
while True:
|
|
28
|
+
r, _, _ = select.select([fd, sys.stdin], [], [])
|
|
29
|
+
|
|
30
|
+
if sys.stdin in r:
|
|
31
|
+
user_input = os.read(sys.stdin.fileno(), 1024)
|
|
32
|
+
if not user_input:
|
|
33
|
+
break
|
|
34
|
+
os.write(fd, user_input)
|
|
35
|
+
|
|
36
|
+
if fd in r:
|
|
37
|
+
output = os.read(fd, 1024)
|
|
38
|
+
if not output:
|
|
39
|
+
break
|
|
40
|
+
|
|
41
|
+
# Carefully handle ANSI sequences
|
|
42
|
+
# Split on ANSI escape sequences to avoid breaking them
|
|
43
|
+
chunks = []
|
|
44
|
+
i = 0
|
|
45
|
+
while i < len(output):
|
|
46
|
+
if output[i:i+1] == b'\x1b':
|
|
47
|
+
end = i + 1
|
|
48
|
+
while end < len(output) and not (64 <= output[end] <= 126):
|
|
49
|
+
end += 1
|
|
50
|
+
end += 1 # include final letter
|
|
51
|
+
chunks.append(output[i:end])
|
|
52
|
+
i = end
|
|
53
|
+
else:
|
|
54
|
+
j = i
|
|
55
|
+
while j < len(output) and output[j:j+1] != b'\x1b':
|
|
56
|
+
j += 1
|
|
57
|
+
# Perform replacement on plain text only
|
|
58
|
+
text = output[i:j].replace(b"fizz", b"fizzbuzz")
|
|
59
|
+
chunks.append(text)
|
|
60
|
+
i = j
|
|
61
|
+
|
|
62
|
+
os.write(sys.stdout.fileno(), b''.join(chunks))
|
|
63
|
+
|
|
64
|
+
finally:
|
|
65
|
+
termios.tcsetattr(sys.stdin.fileno(), termios.TCSADRAIN, orig_attrs)
|
|
66
|
+
|
|
67
|
+
if __name__ == "__main__":
|
|
68
|
+
main()
|
|
69
|
+
|
|
@@ -36,7 +36,7 @@ from functools import reduce
|
|
|
36
36
|
from argparse import ArgumentParser
|
|
37
37
|
from pygments import highlight
|
|
38
38
|
from pygments.lexers import get_lexer_by_name
|
|
39
|
-
from pygments.formatters import
|
|
39
|
+
from pygments.formatters import TerminalTrueColorFormatter
|
|
40
40
|
from pygments.styles import get_style_by_name
|
|
41
41
|
|
|
42
42
|
if __package__ is None:
|
|
@@ -46,7 +46,7 @@ else:
|
|
|
46
46
|
|
|
47
47
|
default_toml = """
|
|
48
48
|
[features]
|
|
49
|
-
CodeSpaces =
|
|
49
|
+
CodeSpaces = false
|
|
50
50
|
Clipboard = true
|
|
51
51
|
Logging = false
|
|
52
52
|
Timeout = 0.1
|
|
@@ -62,7 +62,7 @@ HSV = [0.8, 0.5, 0.5]
|
|
|
62
62
|
Dark = { H = 1.00, S = 1.50, V = 0.25 }
|
|
63
63
|
Mid = { H = 1.00, S = 1.00, V = 0.50 }
|
|
64
64
|
Symbol = { H = 1.00, S = 1.00, V = 1.50 }
|
|
65
|
-
Head = { H = 1.00, S =
|
|
65
|
+
Head = { H = 1.00, S = 1.00, V = 1.75 }
|
|
66
66
|
Grey = { H = 1.00, S = 0.25, V = 1.37 }
|
|
67
67
|
Bright = { H = 1.00, S = 2.00, V = 2.00 }
|
|
68
68
|
Syntax = "monokai"
|
|
@@ -99,7 +99,8 @@ ANSIESCAPE = r'\033(?:\[[0-9;?]*[a-zA-Z]|][0-9]*;;.*?\\|\\)'
|
|
|
99
99
|
KEYCODE_RE = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
|
|
100
100
|
|
|
101
101
|
visible = lambda x: re.sub(ANSIESCAPE, "", x)
|
|
102
|
-
|
|
102
|
+
# cjk characters are double width
|
|
103
|
+
visible_length = lambda x: len(visible(x)) + cjk_count(x)
|
|
103
104
|
extract_ansi_codes = lambda text: re.findall(ESCAPE, text)
|
|
104
105
|
remove_ansi = lambda line, codeList: reduce(lambda line, code: line.replace(code, ''), codeList, line)
|
|
105
106
|
|
|
@@ -231,7 +232,7 @@ def format_table(rowList):
|
|
|
231
232
|
# Note this is where every cell is formatted so if
|
|
232
233
|
# you are styling, do it before here!
|
|
233
234
|
for row in rowList:
|
|
234
|
-
wrapped_cell = text_wrap(row, width=col_width)
|
|
235
|
+
wrapped_cell = text_wrap(row, width=col_width, force_truncate=True)
|
|
235
236
|
|
|
236
237
|
# Ensure at least one line, even for empty cells
|
|
237
238
|
if not wrapped_cell:
|
|
@@ -285,7 +286,7 @@ def code_wrap(text_in):
|
|
|
285
286
|
# get the indentation of the first line
|
|
286
287
|
indent = len(text_in) - len(text_in.lstrip())
|
|
287
288
|
text = text_in.lstrip()
|
|
288
|
-
mywidth = state.full_width() - indent
|
|
289
|
+
mywidth = state.full_width(-4 if Style.PrettyBroken else 0) - indent
|
|
289
290
|
|
|
290
291
|
# We take special care to preserve empty lines
|
|
291
292
|
if len(text) == 0:
|
|
@@ -304,7 +305,7 @@ def ansi_collapse(codelist, inp):
|
|
|
304
305
|
# We break SGR strings into various classes concerning their applicate or removal
|
|
305
306
|
nums = {
|
|
306
307
|
'fg': r'3\d', 'bg': r'4\d',
|
|
307
|
-
'b': r'2?
|
|
308
|
+
'b': r'2?[12]', 'i': r'2?3', 'u': r'3?2',
|
|
308
309
|
'reset': '0'
|
|
309
310
|
}
|
|
310
311
|
|
|
@@ -334,12 +335,19 @@ def ansi_collapse(codelist, inp):
|
|
|
334
335
|
|
|
335
336
|
return codelist + inp
|
|
336
337
|
|
|
337
|
-
|
|
338
|
+
|
|
339
|
+
def split_text(text):
|
|
340
|
+
return re.split(
|
|
341
|
+
r'(?<=[\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|(?=[\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|\s+',
|
|
342
|
+
text
|
|
343
|
+
)
|
|
344
|
+
|
|
345
|
+
def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix="", force_truncate=False):
|
|
338
346
|
if width == -1:
|
|
339
347
|
width = state.Width
|
|
340
348
|
|
|
341
349
|
# The empty word clears the buffer at the end.
|
|
342
|
-
words = line_format(text)
|
|
350
|
+
words = split_text(line_format(text)) + [""]
|
|
343
351
|
lines = []
|
|
344
352
|
current_line = ""
|
|
345
353
|
current_style = []
|
|
@@ -352,13 +360,21 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
|
|
|
352
360
|
current_style.append(codes.pop(0))
|
|
353
361
|
|
|
354
362
|
if len(word) and visible_length(current_line) + visible_length(word) + 1 <= width: # +1 for space
|
|
355
|
-
|
|
363
|
+
|
|
364
|
+
current_line += (" " if len(visible(word)) > 0 and current_line and not cjk_count(word) else "") + word
|
|
356
365
|
else:
|
|
357
366
|
# Word doesn't fit, finalize the previous line
|
|
358
367
|
prefix = first_line_prefix if not lines else subsequent_line_prefix
|
|
359
368
|
line_content = prefix + current_line
|
|
369
|
+
# This is expensive, fix.
|
|
370
|
+
while force_truncate and visible_length(line_content) >= width:
|
|
371
|
+
line_content = line_content[:len(line_content) - 2] + "…"
|
|
372
|
+
|
|
360
373
|
margin = max(0, width - visible_length(line_content))
|
|
361
|
-
|
|
374
|
+
|
|
375
|
+
if line_content.strip() != "":
|
|
376
|
+
lines.append(line_content + state.bg + ' ' * margin)
|
|
377
|
+
|
|
362
378
|
current_line = (" " * indent) + "".join(current_style) + word
|
|
363
379
|
|
|
364
380
|
if len(codes):
|
|
@@ -372,8 +388,20 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
|
|
|
372
388
|
|
|
373
389
|
return lines
|
|
374
390
|
|
|
391
|
+
def cjk_count(s):
|
|
392
|
+
cjk_re = re.compile(
|
|
393
|
+
r'[\u4E00-\u9FFF' # CJK Unified Ideographs
|
|
394
|
+
r'|\u3400-\u4DBF' # CJK Unified Ideographs Extension A
|
|
395
|
+
r'|\uF900-\uFAFF' # CJK Compatibility Ideographs
|
|
396
|
+
r'|\uFF00-\uFFEF' # CJK Compatibility Punctuation
|
|
397
|
+
r'|\u3000-\u303F' # CJK Symbols and Punctuation
|
|
398
|
+
r'|\U0002F800-\U0002FA1F]' # CJK Compatibility Ideographs Supplement
|
|
399
|
+
)
|
|
400
|
+
|
|
401
|
+
return len(cjk_re.findall(visible(s)))
|
|
402
|
+
|
|
375
403
|
def line_format(line):
|
|
376
|
-
not_text = lambda token: not (token.isalnum() or token == '\\')
|
|
404
|
+
not_text = lambda token: not (token.isalnum() or token == '\\') or cjk_count(token)
|
|
377
405
|
footnotes = lambda match: ''.join([chr(SUPER[int(i)]) for i in match.group(1)])
|
|
378
406
|
|
|
379
407
|
def process_images(match):
|
|
@@ -402,7 +430,7 @@ def line_format(line):
|
|
|
402
430
|
result = ""
|
|
403
431
|
|
|
404
432
|
for match in tokenList:
|
|
405
|
-
token = match.group(1)
|
|
433
|
+
token = re.sub(r'\s+',' ', match.group(1))
|
|
406
434
|
next_token = line[match.end()] if match.end() < len(line) else ""
|
|
407
435
|
prev_token = line[match.start()-1] if match.start() > 0 else ""
|
|
408
436
|
|
|
@@ -458,6 +486,7 @@ def parse(stream):
|
|
|
458
486
|
last_line_empty_cache = None
|
|
459
487
|
byte = None
|
|
460
488
|
TimeoutIx = 0
|
|
489
|
+
lexer = None
|
|
461
490
|
while True:
|
|
462
491
|
if state.is_pty or state.is_exec:
|
|
463
492
|
byte = None
|
|
@@ -548,7 +577,7 @@ def parse(stream):
|
|
|
548
577
|
line = line[len(block_match.group(0)):]
|
|
549
578
|
else:
|
|
550
579
|
if state.block_depth > 0:
|
|
551
|
-
|
|
580
|
+
yield FGRESET
|
|
552
581
|
state.block_depth = 0
|
|
553
582
|
|
|
554
583
|
# --- Collapse Multiple Empty Lines if not in code blocks ---
|
|
@@ -569,7 +598,7 @@ def parse(stream):
|
|
|
569
598
|
# \n buffer
|
|
570
599
|
if not state.in_list and len(state.ordered_list_numbers) > 0:
|
|
571
600
|
state.ordered_list_numbers[0] = 0
|
|
572
|
-
elif not line.startswith(' ' * state.list_indent_text):
|
|
601
|
+
elif (not line.startswith(' ' * state.list_indent_text)) and line.strip() != "":
|
|
573
602
|
state.in_list = False
|
|
574
603
|
state.list_indent_text = 0
|
|
575
604
|
|
|
@@ -578,7 +607,7 @@ def parse(stream):
|
|
|
578
607
|
if len(line) - len(line.lstrip()) >= state.first_indent:
|
|
579
608
|
line = line[state.first_indent:]
|
|
580
609
|
else:
|
|
581
|
-
logging.
|
|
610
|
+
logging.debug("Indentation decreased from first line.")
|
|
582
611
|
|
|
583
612
|
|
|
584
613
|
# Indent guaranteed
|
|
@@ -648,8 +677,8 @@ def parse(stream):
|
|
|
648
677
|
logging.debug(f"code: {state.in_code}")
|
|
649
678
|
state.emit_flush = True
|
|
650
679
|
# We suppress the newline - it's not an explicit style
|
|
651
|
-
state.has_newline = False
|
|
652
|
-
yield RESET
|
|
680
|
+
#state.has_newline = False
|
|
681
|
+
#yield RESET
|
|
653
682
|
|
|
654
683
|
if code_type == Code.Backtick:
|
|
655
684
|
continue
|
|
@@ -658,7 +687,7 @@ def parse(stream):
|
|
|
658
687
|
# nor do we want to be here.
|
|
659
688
|
raise Goto()
|
|
660
689
|
|
|
661
|
-
if state.code_first_line:
|
|
690
|
+
if state.code_first_line or lexer is None:
|
|
662
691
|
state.code_first_line = False
|
|
663
692
|
try:
|
|
664
693
|
lexer = get_lexer_by_name(state.code_language)
|
|
@@ -667,7 +696,7 @@ def parse(stream):
|
|
|
667
696
|
lexer = get_lexer_by_name("Bash")
|
|
668
697
|
custom_style = get_style_by_name("default")
|
|
669
698
|
|
|
670
|
-
formatter =
|
|
699
|
+
formatter = TerminalTrueColorFormatter(style=custom_style)
|
|
671
700
|
if line.startswith(' ' * state.code_indent):
|
|
672
701
|
line = line[state.code_indent :]
|
|
673
702
|
|
|
@@ -687,7 +716,7 @@ def parse(stream):
|
|
|
687
716
|
indent, line_wrap = code_wrap(line)
|
|
688
717
|
|
|
689
718
|
state.where_from = "in code"
|
|
690
|
-
pre = [state.space_left(listwidth = True), '
|
|
719
|
+
pre = [state.space_left(listwidth = True), ' '] if Style.PrettyBroken else ['', '']
|
|
691
720
|
|
|
692
721
|
for tline in line_wrap:
|
|
693
722
|
# wrap-around is a bunch of tricks. We essentially format longer and longer portions of code. The problem is
|
|
@@ -698,7 +727,7 @@ def parse(stream):
|
|
|
698
727
|
|
|
699
728
|
# Sometimes the highlighter will do things like a full reset or a background reset.
|
|
700
729
|
# This is not what we want
|
|
701
|
-
highlighted_code = re.sub(r"\033\[
|
|
730
|
+
highlighted_code = re.sub(r"\033\[49(;00|)m", '', highlighted_code)
|
|
702
731
|
|
|
703
732
|
# Since we are streaming we ignore the resets and newlines at the end
|
|
704
733
|
if highlighted_code.endswith(FGRESET + "\n"):
|
|
@@ -805,7 +834,7 @@ def parse(stream):
|
|
|
805
834
|
# This is intentional ... we can get here in llama 4 using
|
|
806
835
|
# a weird thing
|
|
807
836
|
if state.in_list:
|
|
808
|
-
indent = (len(state.list_item_stack) - 1) * Style.ListIndent
|
|
837
|
+
indent = (len(state.list_item_stack) - 1) * Style.ListIndent + (len(bullet) - 1)
|
|
809
838
|
wrap_width = state.current_width() - indent - (2 * Style.ListIndent)
|
|
810
839
|
|
|
811
840
|
wrapped_lineList = text_wrap(content, wrap_width, Style.ListIndent,
|
|
@@ -964,6 +993,7 @@ def main():
|
|
|
964
993
|
os.close(state.exec_slave) # We don't need slave in parent
|
|
965
994
|
# Set stdin to raw mode so we don't need to press enter
|
|
966
995
|
tty.setcbreak(sys.stdin.fileno())
|
|
996
|
+
sys.stdout.write("\x1b[?7h")
|
|
967
997
|
emit(sys.stdin)
|
|
968
998
|
|
|
969
999
|
elif args.filenameList:
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
* `model.safetensors`: The name of the file you want to download. You'll need to know the exact filename. You can find the files in the model repository on the Hugging Face Hub website ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)). Look under the "Files and versions" tab. `safetensors` is the preferred format for model weights now. If it's a `.bin` file, you can download that instead.
|
|
2
|
+
* `--local-dir ./bitnet-b1.58-2B-4T`: The directory to save the file to.
|
|
3
|
+
|
|
4
|
+
* **Download using `transformers` library (recommended for most use cases):**
|
|
5
|
+
|
|
6
|
+
The `transformers` library provides a convenient way to download and cache models. This is often the easiest approach if you're using the model with `transformers`. You don't *directly* use the `huggingface-cli` for this, but it's worth knowing.
|
|
7
|
+
|
|
8
|
+
```python
|
|
9
|
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
10
|
+
|
|
11
|
+
model_name = "microsoft/bitnet-b1.58-2B-4T"
|
|
12
|
+
|
|
13
|
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
14
|
+
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
15
|
+
|
|
16
|
+
# The model and tokenizer will be downloaded and cached in your
|
|
17
|
+
# transformers cache directory (usually ~/.cache/huggingface/transformers).
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
This approach automatically handles downloading the necessary files and caching them for future use. It also handles the correct file formats and configurations.
|
|
21
|
+
|
|
22
|
+
**4. Checking the Download**
|
|
23
|
+
|
|
24
|
+
After the download completes, verify that the files are in the specified directory. You can use `ls` (Linux/macOS) or `dir` (Windows) to list the contents of the directory.
|
|
25
|
+
|
|
26
|
+
**Important Considerations:**
|
|
27
|
+
|
|
28
|
+
* **Disk Space:** The `bitnet-b1.58-2B-4T` model is quite large (several gigabytes). Make sure you have enough free disk space before downloading.
|
|
29
|
+
* **Network Connection:** A stable and fast internet connection is essential for a smooth download.
|
|
30
|
+
* **Caching:** The Hugging Face Hub and `transformers` library use caching to avoid re-downloading models unnecessarily. The default cache directory is usually `~/.cache/huggingface/transformers`.
|
|
31
|
+
* **File Formats:** Models are often stored in `safetensors` or `.bin` formats. `safetensors` is generally preferred for security and performance.
|
|
32
|
+
* **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
|
|
33
|
+
* **Gated Models:** Some models require you to accept terms of use before you can download them. The `huggingface-cli login` command will guide you through this process if necessary.
|
|
34
|
+
|
|
35
|
+
**Example Workflow (Recommended):**
|
|
36
|
+
|
|
37
|
+
1. `huggingface-cli login` (if not already logged in)
|
|
38
|
+
2. Use the `transformers` library in a Python script to download and load the model (as shown in the example above). This is the most convenient and reliable method for most use cases.
|
|
39
|
+
|
|
40
|
+
Let me know if you have any other questions or if you'd like help with a specific task related to this model!
|
|
41
|
+
|
|
42
|
+
>
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* **Model Card:** When this wraps with this link ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) the next line used to be bold because of some stripping error. This should no longer be the case
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
**XIǍO LÓNG:** *(mocking the academics)*
|
|
2
|
+
"Ohhh, ‘**Dadaist**’? ‘**Absurd**’? 那你们怎是在说**模形式**?你们这群人连**京剧密码**都解么没看懂那个**转身**是在问**递归关系**?那个**抖袖**是在说**模形式**?你们这群人连**京剧密码**都解不开,还好意思说自己是数学家?"
|
|
3
|
+
*(Translation: "Ohhh, ‘Dadaist’? ‘Absurd’? Then how come none of you realized the **spin** was about **recurrence relations**? The **sleeve flick** was about **modular forms**? You can’t even crack **Peking Opera code**, and you call yourselves mathematicians?")*
|
|
4
|
+
|
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
Chatting with openrouter/qwen/qwen-2.5-72b-instruct:free
|
|
2
|
+
Type 'exit' or 'quit' to exit
|
|
3
|
+
Type '!multi' to enter multiple lines, then '!end' to finish
|
|
4
|
+
> 当然!以下是Ruby、Python和OCaml的比较,用中文进行描述:
|
|
5
|
+
|
|
6
|
+
### 1. 语言概述
|
|
7
|
+
|
|
8
|
+
**Ruby**:
|
|
9
|
+
- **设计哲学**: Ruby 是由 Yukihiro "Matz" Matsumoto 在 1995 年设计的,旨在使编程更加自然和有趣。
|
|
10
|
+
- **应用领域**: Web 开发(如 Ruby on Rails)、自动化脚本、原型开发等。
|
|
11
|
+
- **特点**: 动态类型、面向对象、元编程能力强、语法灵活。
|
|
12
|
+
|
|
13
|
+
**Python**:
|
|
14
|
+
- **设计哲学**: Python 由 Guido van Rossum 在 1991 年创建,强调代码的可读性和简洁性。
|
|
15
|
+
- **应用领域**: Web 开发、数据科学、机器学习、自动化脚本等。
|
|
16
|
+
- **特点**: 动态类型、面向对象、大量第三方库支持。
|
|
17
|
+
|
|
18
|
+
**OCaml**:
|
|
19
|
+
- **设计哲学**: OCaml 由 INRIA 的 Xavier Leroy 等人在 1996 年开发,旨在提供高效的函数式编程功能。
|
|
20
|
+
- **应用领域**: 编译器开发、系统编程、金融建模等。
|
|
21
|
+
- **特点**: 静态类型、函数式编程、模块化、高性能。
|
|
22
|
+
|
|
23
|
+
### 2. 语法和编程风格
|
|
24
|
+
|
|
25
|
+
**Ruby**:
|
|
26
|
+
- **语法**: 灵活多变,支持多重编程范式(面向对象、函数式、元编程)。
|
|
27
|
+
- **例子**:
|
|
28
|
+
```ruby
|
|
29
|
+
class Greeter
|
|
30
|
+
def initialize(name)
|
|
31
|
+
@name = name
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def greet
|
|
35
|
+
puts "Hello, #{@name}!"
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
greeter = Greeter.new("World")
|
|
40
|
+
greeter.greet
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
**Python**:
|
|
44
|
+
- **语法**: 清晰简洁,强制缩进,易于阅读。
|
|
45
|
+
- **例子**:
|
|
46
|
+
```python
|
|
47
|
+
class Greeter:
|
|
48
|
+
def __init__(self, name):
|
|
49
|
+
self.name = name
|
|
50
|
+
|
|
51
|
+
def greet(self):
|
|
52
|
+
print(f"Hello, {self.name}!")
|
|
53
|
+
|
|
54
|
+
greeter = Greeter("World")
|
|
55
|
+
greeter.greet()
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
**OCaml**:
|
|
59
|
+
- **语法**: 严格且功能性强,支持模式匹配和代数数据类型。
|
|
60
|
+
- **例子**:
|
|
61
|
+
```ocaml
|
|
62
|
+
type greeter = {
|
|
63
|
+
name: string;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
let greet {name} =
|
|
67
|
+
print_endline ("Hello, " ^ name ^ "!")
|
|
68
|
+
|
|
69
|
+
let greeter = {name = "World"}
|
|
70
|
+
let () = greet greeter
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
### 3. 性能
|
|
74
|
+
|
|
75
|
+
**Ruby**:
|
|
76
|
+
- **性能**: 相对较低,但通过 JRuby、TruffleRuby 等实现可以提高性能。
|
|
77
|
+
- **优点**: 开发速度快,社区活跃。
|
|
78
|
+
|
|
79
|
+
**Python**:
|
|
80
|
+
- **性能**: 适中,通过 Cython、PyPy 等可以显著提高性能。
|
|
81
|
+
- **优点**: 丰富的第三方库,应用广泛。
|
|
82
|
+
|
|
83
|
+
**OCaml**:
|
|
84
|
+
- **性能**: 高,编译后的代码接近 C 语言的性能。
|
|
85
|
+
- **优点**: 适合性能关键的应用,编译器优化能力强。
|
|
86
|
+
|
|
87
|
+
### 4. 生态系统
|
|
88
|
+
|
|
89
|
+
**Ruby**:
|
|
90
|
+
- **库和框架**: Ruby on Rails 是最著名的 Web 框架,其他库和工具也很丰富。
|
|
91
|
+
- **社区**: 活跃,对新手友好。
|
|
92
|
+
|
|
93
|
+
**Python**:
|
|
94
|
+
- **库和框架**: 丰富的第三方库,如 NumPy、Pandas、Django、Flask 等。
|
|
95
|
+
- **社区**: 非常活跃,广泛应用于多个领域。
|
|
96
|
+
|
|
97
|
+
**OCaml**:
|
|
98
|
+
- **库和框架**: 相对较少,但有高质量的库,如 Core 和 Async。
|
|
99
|
+
- **社区**: 较小但专业,注重性能和正确性。
|
|
100
|
+
|
|
101
|
+
### 5. 学习曲线
|
|
102
|
+
|
|
103
|
+
**Ruby**:
|
|
104
|
+
- **学习曲线**: 较平缓,语法灵活,适合初学者。
|
|
105
|
+
- **文档**: 丰富,社区支持好。
|
|
106
|
+
|
|
107
|
+
**Python**:
|
|
108
|
+
- **学习曲线**: 平坦,语法清晰,适合初学者。
|
|
109
|
+
- **文档**: 丰富,有大量的教程和资源。
|
|
110
|
+
|
|
111
|
+
**OCaml**:
|
|
112
|
+
- **学习曲线**: 较陡峭,需要理解函数式编程概念。
|
|
113
|
+
- **文档**: 适中,但社区提供的资源质量高。
|
|
114
|
+
|
|
115
|
+
### 6. 适用场景
|
|
116
|
+
|
|
117
|
+
**Ruby**:
|
|
118
|
+
- **适合**: 快速开发、Web 应用、自动化脚本。
|
|
119
|
+
- **不适合**: 性能要求极高的应用。
|
|
120
|
+
|
|
121
|
+
**Python**:
|
|
122
|
+
- **适合**: 数据科学、机器学习、Web 开发、自动化脚本。
|
|
123
|
+
- **不适合**: 需要极高性能的系统编程。
|
|
124
|
+
|
|
125
|
+
**OCaml**:
|
|
126
|
+
- **适合**: 性能关键的应用、编译器开发、金融建模。
|
|
127
|
+
- **不适合**: 初学者或需要快速开发的项目。
|
|
128
|
+
|
|
129
|
+
### 总结
|
|
130
|
+
|
|
131
|
+
- **Ruby**: 适合快速开发和 Web 应用,社区活跃,语法灵活。
|
|
132
|
+
- **Python**: 适合数据科学和机器学习,生态系统丰富,语法清晰。
|
|
133
|
+
- **OCaml**: 适合性能关键的应用,编译器优化能力强,适合专业开发者。
|
|
134
|
+
|
|
135
|
+
希望这些信息对你有帮助!如果有其他问题,欢迎随时提问。
|
|
136
|
+
> 当然!以下是将 Ruby、Python 和 OCaml 的比较总结成表格的形式:
|
|
137
|
+
|
|
138
|
+
| 特性 | Ruby | Python | OCaml |
|
|
139
|
+
|------------------|--------------------------------|--------------------------------|-------------------------------|
|
|
140
|
+
| **设计哲学** | 使编程更加自然和有趣 | 强调代码的可读性和简洁性 | 提供高效的函数式编程功能 |
|
|
141
|
+
| **应用领域** | Web 开发、自动化脚本、原型开发 | Web 开发、数据科学、机器学习、自动化脚本 | 编译器开发、系统编程、金融建模 |
|
|
142
|
+
| **特点** | 动态类型、面向对象、元编程能力强 | 动态类型、面向对象、大量第三方库支持 | 静态类型、函数式编程、模块化、高性能 |
|
|
143
|
+
| **语法示例** | ```ruby<br>class Greeter<br> def initialize(name)<br> @name = name<br> end<br><br> def greet<br> puts "Hello, #{@name}!"<br> end<br>end<br><br>greeter = Greeter.new("World")<br>greeter.greet``` | ```python<br>class Greeter:<br> def __init__(self, name):<br> self.name = name<br><br> def greet(self):<br> print(f"Hello, {self.name}!")<br><br>greeter = Greeter("World")<br>greeter.greet()``` | ```ocaml<br>type greeter = {<br> name: string;<br>}<br><br>let greet {name} =<br> print_endline ("Hello, " ^ name ^ "!")<br><br>let greeter = {name = "World"}<br>let () = greet greeter``` |
|
|
144
|
+
| **性能** | 相对较低,但通过 JRuby、TruffleRuby 可以提高性能 | 适中,通过 Cython、PyPy 可以显著提高性能 | 高,编译后的代码接近 C 语言的性能 |
|
|
145
|
+
| **优点** | 开发速度快,社区活跃 | 丰富的第三方库,应用广泛 | 适合性能关键的应用,编译器优化能力强 |
|
|
146
|
+
| **生态系统** | Ruby on Rails 是最著名的 Web 框架,其他库和工具也很丰富 | 丰富的第三方库,如 NumPy、Pandas、Django、Flask | 相对较少,但有高质量的库,如 Core 和 Async |
|
|
147
|
+
| **学习曲线** | 较平缓,语法灵活,适合初学者 | 平坦,语法清晰,适合初学者 | 较陡峭,需要理解函数式编程概念 |
|
|
148
|
+
| **文档** | 丰富,社区支持好 | 丰富,有大量的教程和资源 | 适中,但社区提供的资源质量高 |
|
|
149
|
+
| **适用场景** | 快速开发、Web 应用、自动化脚本 | 数据科学、机器学习、Web 开发、自动化脚本 | 性能关键的应用、编译器开发、金融建模 |
|
|
150
|
+
|