streamdown 0.17.0__tar.gz → 0.19.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {streamdown-0.17.0 → streamdown-0.19.0}/PKG-INFO +12 -6
- {streamdown-0.17.0 → streamdown-0.19.0}/README.md +11 -5
- {streamdown-0.17.0 → streamdown-0.19.0}/pyproject.toml +1 -1
- streamdown-0.19.0/ss-new.py +45 -0
- streamdown-0.19.0/ss.py +69 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/sd.py +60 -23
- streamdown-0.19.0/streamdown/ss +1 -0
- streamdown-0.19.0/streamdown/ss1 +42 -0
- streamdown-0.19.0/tests/bold_reset_with_link.md +1 -0
- streamdown-0.19.0/tests/chinese.md +4 -0
- streamdown-0.19.0/tests/cjj.mv +4 -0
- streamdown-0.19.0/tests/cjk-table.md +13 -0
- streamdown-0.19.0/tests/cjk-wrap.md +150 -0
- streamdown-0.19.0/tests/jimmy_webb.md +1458 -0
- streamdown-0.19.0/tests/strip-chunks.sh +2 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/table_test.md +0 -3
- streamdown-0.17.0/streamdown/ss +0 -21
- streamdown-0.17.0/tests/longer-example.md +0 -18
- streamdown-0.17.0/tests/table.md +0 -4
- streamdown-0.17.0/tests/white-space-code.md +0 -19
- {streamdown-0.17.0 → streamdown-0.19.0}/.gitignore +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/.vimrc +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/24-bit-color.sh +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/LICENSE.MIT +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/assets/logo.png +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/assets/logo.svg +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/configurable.png +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/copyable.png +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/dunder.png +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/error.txt +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/newdir/file_0.py +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/newdir/file_1.rb +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/newdir/file_2.jl +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/passthrough.py +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/python-go.png +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/requirements.txt +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/somelog.txt +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/__init__.py +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/plugins/README.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/streamdown/plugins/latex.py +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/table.png +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/temp.py +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/test.py +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/test_input.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/README.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/backtick-with-post-spaces.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/block.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/broken-example.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/chunk-buffer.sh +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/code.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/escape.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/example.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/fizzbuzz.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/inline.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/line-buffer.sh +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/line-wrap.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/links.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/managerie.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/mandlebrot.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/markdown.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/nested-example.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/outline.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/pvgo_512.jpg +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/pythonvgo.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/table-break.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/test.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/test_input.md +0 -0
- {streamdown-0.17.0 → streamdown-0.19.0}/tests/wm.md +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: streamdown
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.19.0
|
|
4
4
|
Summary: A streaming markdown renderer for modern terminals with syntax highlighting
|
|
5
5
|
Project-URL: Homepage, https://github.com/kristopolous/Streamdown
|
|
6
6
|
Project-URL: Bug Tracker, https://github.com/kristopolous/Streamdown/issues
|
|
@@ -31,23 +31,25 @@ Description-Content-Type: text/markdown
|
|
|
31
31
|
<img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
|
|
32
32
|
<br/>
|
|
33
33
|
<a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
|
|
34
|
+
<br/><strong>Terminal streaming markdown that rocks</strong>
|
|
35
|
+
|
|
34
36
|
</p>
|
|
35
37
|
|
|
36
|
-
**The streaming markdown renderer for the terminal that rocks**
|
|
37
38
|
|
|
38
|
-
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown
|
|
39
|
+
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
|
|
40
|
+
It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
|
|
39
41
|
```bash
|
|
40
42
|
$ pip install streamdown
|
|
41
43
|
```
|
|
42
44
|

|
|
43
45
|
|
|
44
46
|
### Provides clean copyable code for long code lines
|
|
45
|
-
|
|
47
|
+
Other renderers inject line breaks when copying code that wraps around. We're better and now you are too!
|
|
46
48
|

|
|
47
49
|
**Tip**: You can make things prettier if you don't mind if this guarantee is broken. See the `PrettyBroken` flag below!
|
|
48
50
|
|
|
49
51
|
### Supports images
|
|
50
|
-
Here's kitty and alacritty.
|
|
52
|
+
Here's kitty and alacritty.
|
|
51
53
|

|
|
52
54
|
|
|
53
55
|
### Supports hyperlinks (OSC 8) and clipboard (OSC 52)
|
|
@@ -56,9 +58,13 @@ Here's kitty and alacritty. Try to do that in glow...
|
|
|
56
58
|
### Supports tables
|
|
57
59
|

|
|
58
60
|
|
|
59
|
-
As well as everything else...
|
|
61
|
+
#### As well as everything else...
|
|
60
62
|

|
|
61
63
|
|
|
64
|
+
#### ...even CJK
|
|
65
|
+
Compare how streamdown wraps around and spaces this tabular Chinese description of programming languages to the same file using glow.
|
|
66
|
+

|
|
67
|
+
|
|
62
68
|
### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
|
|
63
69
|

|
|
64
70
|
|
|
@@ -2,23 +2,25 @@
|
|
|
2
2
|
<img src=https://github.com/user-attachments/assets/0468eac0-2a00-4e98-82ca-09e6ac679357/>
|
|
3
3
|
<br/>
|
|
4
4
|
<a href=https://pypi.org/project/streamdown><img src=https://badge.fury.io/py/streamdown.svg/></a>
|
|
5
|
+
<br/><strong>Terminal streaming markdown that rocks</strong>
|
|
6
|
+
|
|
5
7
|
</p>
|
|
6
8
|
|
|
7
|
-
**The streaming markdown renderer for the terminal that rocks**
|
|
8
9
|
|
|
9
|
-
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown
|
|
10
|
+
Streamdown works with [simonw's llm](https://github.com/simonw/llm) along with any other streaming markdown, even something basic like curl.
|
|
11
|
+
It supports standard piping like any normal pager and a clean `execvp` option for robustly wrapping around interactive programs with readline or their own ANSI stuff to manage.
|
|
10
12
|
```bash
|
|
11
13
|
$ pip install streamdown
|
|
12
14
|
```
|
|
13
15
|

|
|
14
16
|
|
|
15
17
|
### Provides clean copyable code for long code lines
|
|
16
|
-
|
|
18
|
+
Other renderers inject line breaks when copying code that wraps around. We're better and now you are too!
|
|
17
19
|

|
|
18
20
|
**Tip**: You can make things prettier if you don't mind if this guarantee is broken. See the `PrettyBroken` flag below!
|
|
19
21
|
|
|
20
22
|
### Supports images
|
|
21
|
-
Here's kitty and alacritty.
|
|
23
|
+
Here's kitty and alacritty.
|
|
22
24
|

|
|
23
25
|
|
|
24
26
|
### Supports hyperlinks (OSC 8) and clipboard (OSC 52)
|
|
@@ -27,9 +29,13 @@ Here's kitty and alacritty. Try to do that in glow...
|
|
|
27
29
|
### Supports tables
|
|
28
30
|

|
|
29
31
|
|
|
30
|
-
As well as everything else...
|
|
32
|
+
#### As well as everything else...
|
|
31
33
|

|
|
32
34
|
|
|
35
|
+
#### ...even CJK
|
|
36
|
+
Compare how streamdown wraps around and spaces this tabular Chinese description of programming languages to the same file using glow.
|
|
37
|
+

|
|
38
|
+
|
|
33
39
|
### Colors are highly (and quickly) configurable for people who care a lot, or just a little.
|
|
34
40
|

|
|
35
41
|
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
import os
|
|
2
|
+
import pty
|
|
3
|
+
import sys
|
|
4
|
+
import tty
|
|
5
|
+
import termios
|
|
6
|
+
import select
|
|
7
|
+
import re
|
|
8
|
+
|
|
9
|
+
def main():
|
|
10
|
+
orig_attrs = termios.tcgetattr(sys.stdin)
|
|
11
|
+
try:
|
|
12
|
+
tty.setraw(sys.stdin)
|
|
13
|
+
pid, fd = pty.fork()
|
|
14
|
+
|
|
15
|
+
if pid == 0:
|
|
16
|
+
# Child process: run any command
|
|
17
|
+
os.execvp("llm", ["llm", "chat"])
|
|
18
|
+
else:
|
|
19
|
+
buffer = b""
|
|
20
|
+
|
|
21
|
+
while True:
|
|
22
|
+
r, _, _ = select.select([fd, sys.stdin], [], [])
|
|
23
|
+
|
|
24
|
+
if sys.stdin in r:
|
|
25
|
+
data = os.read(sys.stdin.fileno(), 1024)
|
|
26
|
+
if not data:
|
|
27
|
+
break
|
|
28
|
+
os.write(fd, data)
|
|
29
|
+
|
|
30
|
+
if fd in r:
|
|
31
|
+
data = os.read(fd, 1024)
|
|
32
|
+
if not data:
|
|
33
|
+
break
|
|
34
|
+
|
|
35
|
+
buffer += data
|
|
36
|
+
# Replace "fizz" only in printable content
|
|
37
|
+
output = re.sub(rb'fizz', b'fizzbuzz', buffer)
|
|
38
|
+
os.write(sys.stdout.fileno(), output)
|
|
39
|
+
buffer = b""
|
|
40
|
+
finally:
|
|
41
|
+
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, orig_attrs)
|
|
42
|
+
|
|
43
|
+
if __name__ == "__main__":
|
|
44
|
+
main()
|
|
45
|
+
|
streamdown-0.19.0/ss.py
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
import os
|
|
2
|
+
import pty
|
|
3
|
+
import select
|
|
4
|
+
import sys
|
|
5
|
+
import tty
|
|
6
|
+
import termios
|
|
7
|
+
import fcntl
|
|
8
|
+
import struct
|
|
9
|
+
|
|
10
|
+
def set_pty_size(fd, target_fd):
|
|
11
|
+
"""Set window size of PTY to match the real terminal."""
|
|
12
|
+
s = fcntl.ioctl(target_fd, termios.TIOCGWINSZ, b"\x00" * 8)
|
|
13
|
+
fcntl.ioctl(fd, termios.TIOCSWINSZ, s)
|
|
14
|
+
|
|
15
|
+
def main():
|
|
16
|
+
# Save original terminal settings
|
|
17
|
+
orig_attrs = termios.tcgetattr(sys.stdin.fileno())
|
|
18
|
+
try:
|
|
19
|
+
tty.setraw(sys.stdin.fileno()) # raw mode to send Ctrl-C, etc.
|
|
20
|
+
pid, fd = pty.fork()
|
|
21
|
+
|
|
22
|
+
if pid == 0:
|
|
23
|
+
os.execvp("bash", ["bash"])
|
|
24
|
+
else:
|
|
25
|
+
set_pty_size(fd, sys.stdin.fileno())
|
|
26
|
+
|
|
27
|
+
while True:
|
|
28
|
+
r, _, _ = select.select([fd, sys.stdin], [], [])
|
|
29
|
+
|
|
30
|
+
if sys.stdin in r:
|
|
31
|
+
user_input = os.read(sys.stdin.fileno(), 1024)
|
|
32
|
+
if not user_input:
|
|
33
|
+
break
|
|
34
|
+
os.write(fd, user_input)
|
|
35
|
+
|
|
36
|
+
if fd in r:
|
|
37
|
+
output = os.read(fd, 1024)
|
|
38
|
+
if not output:
|
|
39
|
+
break
|
|
40
|
+
|
|
41
|
+
# Carefully handle ANSI sequences
|
|
42
|
+
# Split on ANSI escape sequences to avoid breaking them
|
|
43
|
+
chunks = []
|
|
44
|
+
i = 0
|
|
45
|
+
while i < len(output):
|
|
46
|
+
if output[i:i+1] == b'\x1b':
|
|
47
|
+
end = i + 1
|
|
48
|
+
while end < len(output) and not (64 <= output[end] <= 126):
|
|
49
|
+
end += 1
|
|
50
|
+
end += 1 # include final letter
|
|
51
|
+
chunks.append(output[i:end])
|
|
52
|
+
i = end
|
|
53
|
+
else:
|
|
54
|
+
j = i
|
|
55
|
+
while j < len(output) and output[j:j+1] != b'\x1b':
|
|
56
|
+
j += 1
|
|
57
|
+
# Perform replacement on plain text only
|
|
58
|
+
text = output[i:j].replace(b"fizz", b"fizzbuzz")
|
|
59
|
+
chunks.append(text)
|
|
60
|
+
i = j
|
|
61
|
+
|
|
62
|
+
os.write(sys.stdout.fileno(), b''.join(chunks))
|
|
63
|
+
|
|
64
|
+
finally:
|
|
65
|
+
termios.tcsetattr(sys.stdin.fileno(), termios.TCSADRAIN, orig_attrs)
|
|
66
|
+
|
|
67
|
+
if __name__ == "__main__":
|
|
68
|
+
main()
|
|
69
|
+
|
|
@@ -36,7 +36,7 @@ from functools import reduce
|
|
|
36
36
|
from argparse import ArgumentParser
|
|
37
37
|
from pygments import highlight
|
|
38
38
|
from pygments.lexers import get_lexer_by_name
|
|
39
|
-
from pygments.formatters import
|
|
39
|
+
from pygments.formatters import TerminalTrueColorFormatter
|
|
40
40
|
from pygments.styles import get_style_by_name
|
|
41
41
|
|
|
42
42
|
if __package__ is None:
|
|
@@ -46,7 +46,7 @@ else:
|
|
|
46
46
|
|
|
47
47
|
default_toml = """
|
|
48
48
|
[features]
|
|
49
|
-
CodeSpaces =
|
|
49
|
+
CodeSpaces = false
|
|
50
50
|
Clipboard = true
|
|
51
51
|
Logging = false
|
|
52
52
|
Timeout = 0.1
|
|
@@ -62,7 +62,7 @@ HSV = [0.8, 0.5, 0.5]
|
|
|
62
62
|
Dark = { H = 1.00, S = 1.50, V = 0.25 }
|
|
63
63
|
Mid = { H = 1.00, S = 1.00, V = 0.50 }
|
|
64
64
|
Symbol = { H = 1.00, S = 1.00, V = 1.50 }
|
|
65
|
-
Head = { H = 1.00, S =
|
|
65
|
+
Head = { H = 1.00, S = 1.00, V = 1.75 }
|
|
66
66
|
Grey = { H = 1.00, S = 0.25, V = 1.37 }
|
|
67
67
|
Bright = { H = 1.00, S = 2.00, V = 2.00 }
|
|
68
68
|
Syntax = "monokai"
|
|
@@ -99,7 +99,8 @@ ANSIESCAPE = r'\033(?:\[[0-9;?]*[a-zA-Z]|][0-9]*;;.*?\\|\\)'
|
|
|
99
99
|
KEYCODE_RE = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
|
|
100
100
|
|
|
101
101
|
visible = lambda x: re.sub(ANSIESCAPE, "", x)
|
|
102
|
-
|
|
102
|
+
# cjk characters are double width
|
|
103
|
+
visible_length = lambda x: len(visible(x)) + cjk_count(x)
|
|
103
104
|
extract_ansi_codes = lambda text: re.findall(ESCAPE, text)
|
|
104
105
|
remove_ansi = lambda line, codeList: reduce(lambda line, code: line.replace(code, ''), codeList, line)
|
|
105
106
|
|
|
@@ -231,7 +232,7 @@ def format_table(rowList):
|
|
|
231
232
|
# Note this is where every cell is formatted so if
|
|
232
233
|
# you are styling, do it before here!
|
|
233
234
|
for row in rowList:
|
|
234
|
-
wrapped_cell = text_wrap(row, width=col_width)
|
|
235
|
+
wrapped_cell = text_wrap(row, width=col_width, force_truncate=True)
|
|
235
236
|
|
|
236
237
|
# Ensure at least one line, even for empty cells
|
|
237
238
|
if not wrapped_cell:
|
|
@@ -285,7 +286,7 @@ def code_wrap(text_in):
|
|
|
285
286
|
# get the indentation of the first line
|
|
286
287
|
indent = len(text_in) - len(text_in.lstrip())
|
|
287
288
|
text = text_in.lstrip()
|
|
288
|
-
mywidth = state.full_width() - indent
|
|
289
|
+
mywidth = state.full_width(-4 if Style.PrettyBroken else 0) - indent
|
|
289
290
|
|
|
290
291
|
# We take special care to preserve empty lines
|
|
291
292
|
if len(text) == 0:
|
|
@@ -304,7 +305,7 @@ def ansi_collapse(codelist, inp):
|
|
|
304
305
|
# We break SGR strings into various classes concerning their applicate or removal
|
|
305
306
|
nums = {
|
|
306
307
|
'fg': r'3\d', 'bg': r'4\d',
|
|
307
|
-
'b': r'2?
|
|
308
|
+
'b': r'2?[12]', 'i': r'2?3', 'u': r'3?2',
|
|
308
309
|
'reset': '0'
|
|
309
310
|
}
|
|
310
311
|
|
|
@@ -334,16 +335,24 @@ def ansi_collapse(codelist, inp):
|
|
|
334
335
|
|
|
335
336
|
return codelist + inp
|
|
336
337
|
|
|
337
|
-
|
|
338
|
+
|
|
339
|
+
def split_text(text):
|
|
340
|
+
return [x for x in re.split(
|
|
341
|
+
r'(?<=[\u3000-\u303F\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|(?=[\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF])|\s+',
|
|
342
|
+
text
|
|
343
|
+
) if x]
|
|
344
|
+
|
|
345
|
+
def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_line_prefix="", force_truncate=False):
|
|
338
346
|
if width == -1:
|
|
339
347
|
width = state.Width
|
|
340
348
|
|
|
341
349
|
# The empty word clears the buffer at the end.
|
|
342
|
-
words = line_format(text)
|
|
350
|
+
words = split_text(line_format(text)) + [""]
|
|
343
351
|
lines = []
|
|
344
352
|
current_line = ""
|
|
345
353
|
current_style = []
|
|
346
354
|
|
|
355
|
+
oldword = ''
|
|
347
356
|
for word in words:
|
|
348
357
|
# we apply the style if we see it at the beginning of the word
|
|
349
358
|
codes = extract_ansi_codes(word)
|
|
@@ -352,13 +361,25 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
|
|
|
352
361
|
current_style.append(codes.pop(0))
|
|
353
362
|
|
|
354
363
|
if len(word) and visible_length(current_line) + visible_length(word) + 1 <= width: # +1 for space
|
|
355
|
-
|
|
364
|
+
space = ""
|
|
365
|
+
if len(visible(word)) > 0 and current_line:
|
|
366
|
+
space = " "
|
|
367
|
+
if (":" in visible(word) or cjk_count(word)) and cjk_count(oldword):
|
|
368
|
+
space = ""
|
|
369
|
+
current_line += space + word
|
|
356
370
|
else:
|
|
357
371
|
# Word doesn't fit, finalize the previous line
|
|
358
372
|
prefix = first_line_prefix if not lines else subsequent_line_prefix
|
|
359
373
|
line_content = prefix + current_line
|
|
374
|
+
# This is expensive, fix.
|
|
375
|
+
while force_truncate and visible_length(line_content) >= width:
|
|
376
|
+
line_content = line_content[:len(line_content) - 2] + "…"
|
|
377
|
+
|
|
360
378
|
margin = max(0, width - visible_length(line_content))
|
|
361
|
-
|
|
379
|
+
|
|
380
|
+
if line_content.strip() != "":
|
|
381
|
+
lines.append(line_content + state.bg + ' ' * margin)
|
|
382
|
+
|
|
362
383
|
current_line = (" " * indent) + "".join(current_style) + word
|
|
363
384
|
|
|
364
385
|
if len(codes):
|
|
@@ -367,13 +388,27 @@ def text_wrap(text, width = -1, indent = 0, first_line_prefix="", subsequent_lin
|
|
|
367
388
|
if codes:
|
|
368
389
|
current_style = ansi_collapse(current_style, codes)
|
|
369
390
|
|
|
391
|
+
oldword = word
|
|
392
|
+
|
|
370
393
|
if len(lines) < 1:
|
|
371
394
|
return []
|
|
372
395
|
|
|
373
396
|
return lines
|
|
374
397
|
|
|
398
|
+
def cjk_count(s):
|
|
399
|
+
cjk_re = re.compile(
|
|
400
|
+
r'[\u4E00-\u9FFF' # CJK Unified Ideographs
|
|
401
|
+
r'|\u3400-\u4DBF' # CJK Unified Ideographs Extension A
|
|
402
|
+
r'|\uF900-\uFAFF' # CJK Compatibility Ideographs
|
|
403
|
+
r'|\uFF00-\uFFEF' # CJK Compatibility Punctuation
|
|
404
|
+
r'|\u3000-\u303F' # CJK Symbols and Punctuation
|
|
405
|
+
r'|\U0002F800-\U0002FA1F]' # CJK Compatibility Ideographs Supplement
|
|
406
|
+
)
|
|
407
|
+
|
|
408
|
+
return len(cjk_re.findall(visible(s)))
|
|
409
|
+
|
|
375
410
|
def line_format(line):
|
|
376
|
-
not_text = lambda token: not (token.isalnum() or token == '\\')
|
|
411
|
+
not_text = lambda token: not (token.isalnum() or token == '\\') or cjk_count(token)
|
|
377
412
|
footnotes = lambda match: ''.join([chr(SUPER[int(i)]) for i in match.group(1)])
|
|
378
413
|
|
|
379
414
|
def process_images(match):
|
|
@@ -402,7 +437,7 @@ def line_format(line):
|
|
|
402
437
|
result = ""
|
|
403
438
|
|
|
404
439
|
for match in tokenList:
|
|
405
|
-
token = match.group(1)
|
|
440
|
+
token = re.sub(r'\s+',' ', match.group(1))
|
|
406
441
|
next_token = line[match.end()] if match.end() < len(line) else ""
|
|
407
442
|
prev_token = line[match.start()-1] if match.start() > 0 else ""
|
|
408
443
|
|
|
@@ -458,6 +493,7 @@ def parse(stream):
|
|
|
458
493
|
last_line_empty_cache = None
|
|
459
494
|
byte = None
|
|
460
495
|
TimeoutIx = 0
|
|
496
|
+
lexer = None
|
|
461
497
|
while True:
|
|
462
498
|
if state.is_pty or state.is_exec:
|
|
463
499
|
byte = None
|
|
@@ -548,7 +584,7 @@ def parse(stream):
|
|
|
548
584
|
line = line[len(block_match.group(0)):]
|
|
549
585
|
else:
|
|
550
586
|
if state.block_depth > 0:
|
|
551
|
-
|
|
587
|
+
yield FGRESET
|
|
552
588
|
state.block_depth = 0
|
|
553
589
|
|
|
554
590
|
# --- Collapse Multiple Empty Lines if not in code blocks ---
|
|
@@ -569,7 +605,7 @@ def parse(stream):
|
|
|
569
605
|
# \n buffer
|
|
570
606
|
if not state.in_list and len(state.ordered_list_numbers) > 0:
|
|
571
607
|
state.ordered_list_numbers[0] = 0
|
|
572
|
-
elif not line.startswith(' ' * state.list_indent_text):
|
|
608
|
+
elif (not line.startswith(' ' * state.list_indent_text)) and line.strip() != "":
|
|
573
609
|
state.in_list = False
|
|
574
610
|
state.list_indent_text = 0
|
|
575
611
|
|
|
@@ -578,7 +614,7 @@ def parse(stream):
|
|
|
578
614
|
if len(line) - len(line.lstrip()) >= state.first_indent:
|
|
579
615
|
line = line[state.first_indent:]
|
|
580
616
|
else:
|
|
581
|
-
logging.
|
|
617
|
+
logging.debug("Indentation decreased from first line.")
|
|
582
618
|
|
|
583
619
|
|
|
584
620
|
# Indent guaranteed
|
|
@@ -648,8 +684,8 @@ def parse(stream):
|
|
|
648
684
|
logging.debug(f"code: {state.in_code}")
|
|
649
685
|
state.emit_flush = True
|
|
650
686
|
# We suppress the newline - it's not an explicit style
|
|
651
|
-
state.has_newline = False
|
|
652
|
-
yield RESET
|
|
687
|
+
#state.has_newline = False
|
|
688
|
+
#yield RESET
|
|
653
689
|
|
|
654
690
|
if code_type == Code.Backtick:
|
|
655
691
|
continue
|
|
@@ -658,7 +694,7 @@ def parse(stream):
|
|
|
658
694
|
# nor do we want to be here.
|
|
659
695
|
raise Goto()
|
|
660
696
|
|
|
661
|
-
if state.code_first_line:
|
|
697
|
+
if state.code_first_line or lexer is None:
|
|
662
698
|
state.code_first_line = False
|
|
663
699
|
try:
|
|
664
700
|
lexer = get_lexer_by_name(state.code_language)
|
|
@@ -667,7 +703,7 @@ def parse(stream):
|
|
|
667
703
|
lexer = get_lexer_by_name("Bash")
|
|
668
704
|
custom_style = get_style_by_name("default")
|
|
669
705
|
|
|
670
|
-
formatter =
|
|
706
|
+
formatter = TerminalTrueColorFormatter(style=custom_style)
|
|
671
707
|
if line.startswith(' ' * state.code_indent):
|
|
672
708
|
line = line[state.code_indent :]
|
|
673
709
|
|
|
@@ -687,7 +723,7 @@ def parse(stream):
|
|
|
687
723
|
indent, line_wrap = code_wrap(line)
|
|
688
724
|
|
|
689
725
|
state.where_from = "in code"
|
|
690
|
-
pre = [state.space_left(listwidth = True), '
|
|
726
|
+
pre = [state.space_left(listwidth = True), ' '] if Style.PrettyBroken else ['', '']
|
|
691
727
|
|
|
692
728
|
for tline in line_wrap:
|
|
693
729
|
# wrap-around is a bunch of tricks. We essentially format longer and longer portions of code. The problem is
|
|
@@ -698,7 +734,7 @@ def parse(stream):
|
|
|
698
734
|
|
|
699
735
|
# Sometimes the highlighter will do things like a full reset or a background reset.
|
|
700
736
|
# This is not what we want
|
|
701
|
-
highlighted_code = re.sub(r"\033\[
|
|
737
|
+
highlighted_code = re.sub(r"\033\[49(;00|)m", '', highlighted_code)
|
|
702
738
|
|
|
703
739
|
# Since we are streaming we ignore the resets and newlines at the end
|
|
704
740
|
if highlighted_code.endswith(FGRESET + "\n"):
|
|
@@ -805,7 +841,7 @@ def parse(stream):
|
|
|
805
841
|
# This is intentional ... we can get here in llama 4 using
|
|
806
842
|
# a weird thing
|
|
807
843
|
if state.in_list:
|
|
808
|
-
indent = (len(state.list_item_stack) - 1) * Style.ListIndent
|
|
844
|
+
indent = (len(state.list_item_stack) - 1) * Style.ListIndent #+ (len(bullet) - 1)
|
|
809
845
|
wrap_width = state.current_width() - indent - (2 * Style.ListIndent)
|
|
810
846
|
|
|
811
847
|
wrapped_lineList = text_wrap(content, wrap_width, Style.ListIndent,
|
|
@@ -964,6 +1000,7 @@ def main():
|
|
|
964
1000
|
os.close(state.exec_slave) # We don't need slave in parent
|
|
965
1001
|
# Set stdin to raw mode so we don't need to press enter
|
|
966
1002
|
tty.setcbreak(sys.stdin.fileno())
|
|
1003
|
+
sys.stdout.write("\x1b[?7h")
|
|
967
1004
|
emit(sys.stdin)
|
|
968
1005
|
|
|
969
1006
|
elif args.filenameList:
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
* `model.safetensors`: The name of the file you want to download. You'll need to know the exact filename. You can find the files in the model repository on the Hugging Face Hub website ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)). Look under the "Files and versions" tab. `safetensors` is the preferred format for model weights now. If it's a `.bin` file, you can download that instead.
|
|
2
|
+
* `--local-dir ./bitnet-b1.58-2B-4T`: The directory to save the file to.
|
|
3
|
+
|
|
4
|
+
* **Download using `transformers` library (recommended for most use cases):**
|
|
5
|
+
|
|
6
|
+
The `transformers` library provides a convenient way to download and cache models. This is often the easiest approach if you're using the model with `transformers`. You don't *directly* use the `huggingface-cli` for this, but it's worth knowing.
|
|
7
|
+
|
|
8
|
+
```python
|
|
9
|
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
10
|
+
|
|
11
|
+
model_name = "microsoft/bitnet-b1.58-2B-4T"
|
|
12
|
+
|
|
13
|
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
14
|
+
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
15
|
+
|
|
16
|
+
# The model and tokenizer will be downloaded and cached in your
|
|
17
|
+
# transformers cache directory (usually ~/.cache/huggingface/transformers).
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
This approach automatically handles downloading the necessary files and caching them for future use. It also handles the correct file formats and configurations.
|
|
21
|
+
|
|
22
|
+
**4. Checking the Download**
|
|
23
|
+
|
|
24
|
+
After the download completes, verify that the files are in the specified directory. You can use `ls` (Linux/macOS) or `dir` (Windows) to list the contents of the directory.
|
|
25
|
+
|
|
26
|
+
**Important Considerations:**
|
|
27
|
+
|
|
28
|
+
* **Disk Space:** The `bitnet-b1.58-2B-4T` model is quite large (several gigabytes). Make sure you have enough free disk space before downloading.
|
|
29
|
+
* **Network Connection:** A stable and fast internet connection is essential for a smooth download.
|
|
30
|
+
* **Caching:** The Hugging Face Hub and `transformers` library use caching to avoid re-downloading models unnecessarily. The default cache directory is usually `~/.cache/huggingface/transformers`.
|
|
31
|
+
* **File Formats:** Models are often stored in `safetensors` or `.bin` formats. `safetensors` is generally preferred for security and performance.
|
|
32
|
+
* **Model Card:** Always read the model card on the Hugging Face Hub ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) for important information about the model, its intended use, limitations, and potential biases.
|
|
33
|
+
* **Gated Models:** Some models require you to accept terms of use before you can download them. The `huggingface-cli login` command will guide you through this process if necessary.
|
|
34
|
+
|
|
35
|
+
**Example Workflow (Recommended):**
|
|
36
|
+
|
|
37
|
+
1. `huggingface-cli login` (if not already logged in)
|
|
38
|
+
2. Use the `transformers` library in a Python script to download and load the model (as shown in the example above). This is the most convenient and reliable method for most use cases.
|
|
39
|
+
|
|
40
|
+
Let me know if you have any other questions or if you'd like help with a specific task related to this model!
|
|
41
|
+
|
|
42
|
+
>
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* **Model Card:** When this wraps with this link ([https://huggingface.co/microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T)) the next line used to be bold because of some stripping error. This should no longer be the case
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
**XIǍO LÓNG:** *(mocking the academics)*
|
|
2
|
+
"Ohhh, ‘**Dadaist**’? ‘**Absurd**’? 那你们怎是在说**模形式**?你们这群人连**京剧密码**都解么没看懂那个**转身**是在问**递归关系**?那个**抖袖**是在说**模形式**?你们这群人连**京剧密码**都解不开,还好意思说自己是数学家?"
|
|
3
|
+
*(Translation: "Ohhh, ‘Dadaist’? ‘Absurd’? Then how come none of you realized the **spin** was about **recurrence relations**? The **sleeve flick** was about **modular forms**? You can’t even crack **Peking Opera code**, and you call yourselves mathematicians?")*
|
|
4
|
+
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
| 特性 | Ruby | Python | OCaml |
|
|
2
|
+
|------------------|--------------------------------|--------------------------------|-------------------------------|
|
|
3
|
+
| **设计哲学** | 使编程更加自然和有趣 | 强调代码的可读性和简洁性 | 提供高效的函数式编程功能 |
|
|
4
|
+
| **应用领域** | Web 开发、自动化脚本、原型开发 | Web 开发、数据科学、机器学习、自动化脚本 | 编译器开发、系统编程、金融建模 |
|
|
5
|
+
| **特点** | 动态类型、面向对象、元编程能力强 | 动态类型、面向对象、大量第三方库支持 | 静态类型、函数式编程、模块化、高性能 |
|
|
6
|
+
| **语法示例** | ```ruby<br>class Greeter<br> def initialize(name)<br> @name = name<br> end<br><br> def greet<br> puts "Hello, #{@name}!"<br> end<br>end<br><br>greeter = Greeter.new("World")<br>greeter.greet``` | ```python<br>class Greeter:<br> def __init__(self, name):<br> self.name = name<br><br> def greet(self):<br> print(f"Hello, {self.name}!")<br><br>greeter = Greeter("World")<br>greeter.greet()``` | ```ocaml<br>type greeter = {<br> name: string;<br>}<br><br>let greet {name} =<br> print_endline ("Hello, " ^ name ^ "!")<br><br>let greeter = {name = "World"}<br>let () = greet greeter``` |
|
|
7
|
+
| **性能** | 相对较低,但通过 JRuby、TruffleRuby 可以提高性能 | 适中,通过 Cython、PyPy 可以显著提高性能 | 高,编译后的代码接近 C 语言的性能 |
|
|
8
|
+
| **优点** | 开发速度快,社区活跃 | 丰富的第三方库,应用广泛 | 适合性能关键的应用,编译器优化能力强 |
|
|
9
|
+
| **生态系统** | Ruby on Rails 是最著名的 Web 框架,其他库和工具也很丰富 | 丰富的第三方库,如 NumPy、Pandas、Django、Flask | 相对较少,但有高质量的库,如 Core 和 Async |
|
|
10
|
+
| **学习曲线** | 较平缓,语法灵活,适合初学者 | 平坦,语法清晰,适合初学者 | 较陡峭,需要理解函数式编程概念 |
|
|
11
|
+
| **文档** | 丰富,社区支持好 | 丰富,有大量的教程和资源 | 适中,但社区提供的资源质量高 |
|
|
12
|
+
| **适用场景** | 快速开发、Web 应用、自动化脚本 | 数据科学、机器学习、Web 开发、自动化脚本 | 性能关键的应用、编译器开发、金融建模 |
|
|
13
|
+
|