lex-llm 0.6.11 → 0.6.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2fcdfa125fb151f6cac7d0468881476c99b8145738282c20e52d630a7b739702
|
|
4
|
+
data.tar.gz: 57c2ded197c24f6d05c0227d3bbdd9d8ea843e04df69c5fa4fa1b33b934ad0d7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: cf38e525d5124d5e17a063ff88e7a14d083830f326b90ce9212e8558d8106e934afd456bba8d37a34c3f822f66f7ad772f34300706006d909619063f36a13199
|
|
7
|
+
data.tar.gz: a98036763e921140f0a1e32e66640826de90b3c211ec7e2fc19aa5bbd2496015996eac6c29798fa2ef073ff9f276ec58bc5b581c4edf019aa8402f75e5c6273a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.6.12 - 2026-07-15
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- **Strip leaked Gemma4 special tokens from response content.** Gemma4 models emit `<turn|>`, `<|turn>`, `<channel|>` as literal text when the serving engine fails to intercept them. `<turn|>` and `<channel|>` are end-of-turn signals — content is truncated at the first occurrence (everything after it is garbage). Other leaked tokens (`<|channel>`, `<|turn|>`) are stripped entirely. Prevents users seeing raw control tokens in responses and fixes cases where the leaked token caused downstream "no response returned" errors.
|
|
7
|
+
- **Remove `<|channel>` / `<channel|>` from `THINK_TAG_PAIRS`.** The channel delimiter is NOT always a thinking block — `<|channel>` is a generic channel marker and only becomes thinking when followed by `thought\n`. Having `<channel|>` as a close tag in `THINK_TAG_PAIRS` caused the extractor to consume ALL content before an orphaned `<channel|>` stop token as thinking, producing empty content (dead stop). Now handled only as a leaked stop token to truncate at.
|
|
8
|
+
|
|
3
9
|
## 0.6.11 - 2026-07-15
|
|
4
10
|
|
|
5
11
|
### Added
|
|
@@ -10,9 +10,14 @@ module Legion
|
|
|
10
10
|
|
|
11
11
|
THINK_TAG_PAIRS = [
|
|
12
12
|
['<thinking>', '</thinking>'],
|
|
13
|
-
['<think>', '</think>']
|
|
14
|
-
['<|channel>', '<channel|>']
|
|
13
|
+
['<think>', '</think>']
|
|
15
14
|
].freeze
|
|
15
|
+
# Gemma4 special tokens that leak into content text when the serving
|
|
16
|
+
# engine fails to intercept them. <turn|> and <channel|> are stop
|
|
17
|
+
# signals — content is truncated at the first occurrence.
|
|
18
|
+
# Others are stripped entirely.
|
|
19
|
+
LEAKED_STOP_TOKENS = ['<turn|>', '<|turn>', '<channel|>'].freeze
|
|
20
|
+
LEAKED_STRIP_TOKENS = ['<|channel>', '<|turn|>'].freeze
|
|
16
21
|
UNTAGGED_PREAMBLE_MAX_LENGTH = 4_000
|
|
17
22
|
UNTAGGED_PREAMBLE_STARTS = [
|
|
18
23
|
'the user',
|
|
@@ -76,10 +81,28 @@ module Legion
|
|
|
76
81
|
clean, untagged_thinking = extract_untagged_preamble(clean.strip)
|
|
77
82
|
thinking_parts << untagged_thinking
|
|
78
83
|
|
|
84
|
+
clean = truncate_at_leaked_stop_token(clean)
|
|
85
|
+
clean = strip_leaked_tokens(clean)
|
|
86
|
+
|
|
79
87
|
[clean, compact_thinking(thinking_parts)]
|
|
80
88
|
end
|
|
81
89
|
private_class_method :extract_from_content
|
|
82
90
|
|
|
91
|
+
def truncate_at_leaked_stop_token(text)
|
|
92
|
+
earliest = nil
|
|
93
|
+
LEAKED_STOP_TOKENS.each do |token|
|
|
94
|
+
idx = text.index(token)
|
|
95
|
+
earliest = idx if idx && (earliest.nil? || idx < earliest)
|
|
96
|
+
end
|
|
97
|
+
earliest ? text[0, earliest].rstrip : text
|
|
98
|
+
end
|
|
99
|
+
private_class_method :truncate_at_leaked_stop_token
|
|
100
|
+
|
|
101
|
+
def strip_leaked_tokens(text)
|
|
102
|
+
LEAKED_STRIP_TOKENS.reduce(text) { |t, token| t.gsub(token, '') }
|
|
103
|
+
end
|
|
104
|
+
private_class_method :strip_leaked_tokens
|
|
105
|
+
|
|
83
106
|
def extract_untagged_preamble(content)
|
|
84
107
|
return [content, nil] unless content.is_a?(String)
|
|
85
108
|
|