tritobol 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tritobol-0.1.0/LICENSE +21 -0
- tritobol-0.1.0/PKG-INFO +487 -0
- tritobol-0.1.0/README.md +461 -0
- tritobol-0.1.0/pyproject.toml +38 -0
- tritobol-0.1.0/setup.cfg +4 -0
- tritobol-0.1.0/tests/test_batch.py +129 -0
- tritobol-0.1.0/tests/test_benchmarks.py +25 -0
- tritobol-0.1.0/tests/test_cli.py +163 -0
- tritobol-0.1.0/tests/test_secrets.py +48 -0
- tritobol-0.1.0/tests/test_tritobol.py +392 -0
- tritobol-0.1.0/tritobol/__init__.py +49 -0
- tritobol-0.1.0/tritobol/__main__.py +378 -0
- tritobol-0.1.0/tritobol/batch.py +481 -0
- tritobol-0.1.0/tritobol/core.py +786 -0
- tritobol-0.1.0/tritobol/data/LICENSE.deepseek-tokenizer +21 -0
- tritobol-0.1.0/tritobol/data/README.md +12 -0
- tritobol-0.1.0/tritobol/data/deepseek_v41_tokenizer.json +267359 -0
- tritobol-0.1.0/tritobol/secrets.py +97 -0
- tritobol-0.1.0/tritobol/tokenizer.py +188 -0
- tritobol-0.1.0/tritobol.egg-info/PKG-INFO +487 -0
- tritobol-0.1.0/tritobol.egg-info/SOURCES.txt +23 -0
- tritobol-0.1.0/tritobol.egg-info/dependency_links.txt +1 -0
- tritobol-0.1.0/tritobol.egg-info/entry_points.txt +2 -0
- tritobol-0.1.0/tritobol.egg-info/requires.txt +8 -0
- tritobol-0.1.0/tritobol.egg-info/top_level.txt +1 -0
tritobol-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 tritobol contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
tritobol-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,487 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: tritobol
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Trim redundant prompt text and keep the newest context within a model-specific token budget.
|
|
5
|
+
Author: tritobol contributors
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Keywords: llm,tokenizer,prompt,context,preprocessing,multi-model
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
15
|
+
Classifier: Topic :: Text Processing
|
|
16
|
+
Requires-Python: >=3.10
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
License-File: LICENSE
|
|
19
|
+
Requires-Dist: tokenizers>=0.19
|
|
20
|
+
Provides-Extra: app
|
|
21
|
+
Requires-Dist: flask>=3.0; extra == "app"
|
|
22
|
+
Provides-Extra: dev
|
|
23
|
+
Requires-Dist: pytest>=7; extra == "dev"
|
|
24
|
+
Requires-Dist: hypothesis>=6; extra == "dev"
|
|
25
|
+
Dynamic: license-file
|
|
26
|
+
|
|
27
|
+
# tritobol
|
|
28
|
+
|
|
29
|
+

|
|
30
|
+

|
|
31
|
+

|
|
32
|
+

|
|
33
|
+

|
|
34
|
+

|
|
35
|
+
|
|
36
|
+
**trim + token + bolster** — a lightweight Python token preprocessor that removes
|
|
37
|
+
redundant prompt text and keeps the most recent context when an input is too
|
|
38
|
+
long.
|
|
39
|
+
|
|
40
|
+
`tritobol` does **not** summarize or rewrite your content. It removes repeated
|
|
41
|
+
material, whitespace noise, and decorative separators. It conservatively
|
|
42
|
+
compresses supported code/configuration blocks only when local validation
|
|
43
|
+
passes, and — only if a hard token budget is still exceeded — slides the window
|
|
44
|
+
forward so the newest tokens survive.
|
|
45
|
+
|
|
46
|
+
By default, token counts use the bundled **DeepSeek V4.1 Flash** tokenizer
|
|
47
|
+
(129,280-token vocabulary). To optimize for another model, load its compatible
|
|
48
|
+
tokenizer locally; the optimizer, trimming pipeline, and token-budget logic are
|
|
49
|
+
independent of any one provider.
|
|
50
|
+
|
|
51
|
+
## Install
|
|
52
|
+
|
|
53
|
+
The repository and clone directory are named `tritobol-token_optimizer-`;
|
|
54
|
+
the Python package and CLI command are named `tritobol`. To install from a
|
|
55
|
+
source clone and start the guided workflow:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
git clone https://github.com/pizzafeastbuddies/tritobol-token_optimizer-.git
|
|
59
|
+
cd tritobol-token_optimizer-
|
|
60
|
+
python -m pip install -e .
|
|
61
|
+
tritobol
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
For a non-editable source install, run this from the repository root instead:
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
python -m pip install .
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Publishing to PyPI (maintainers)
|
|
71
|
+
|
|
72
|
+
Configure a PyPI trusted publisher for the `pizzafeastbuddies/tritobol-token_optimizer-`
|
|
73
|
+
repository, using `.github/workflows/publish.yml` and the GitHub Actions
|
|
74
|
+
environment `pypi`. Publishing is triggered by a non-prerelease GitHub release;
|
|
75
|
+
the release tag must match the package version in `pyproject.toml` (for example,
|
|
76
|
+
`v0.1.0`).
|
|
77
|
+
|
|
78
|
+
The guide asks for one UTF-8 text file, optionally asks for a maximum token
|
|
79
|
+
budget, and shows token counts plus a unified diff before saving. A budget may
|
|
80
|
+
discard the oldest text when the optimized result is still too long; a blank
|
|
81
|
+
budget keeps the normal no-hard-limit behavior. The suggested output is a
|
|
82
|
+
separate `*.optimized.*` file; Tritobol never overwrites the source and will
|
|
83
|
+
not replace an existing output. Saving requires an explicit confirmation.
|
|
84
|
+
Press Enter at the file prompt or use Ctrl+C/Ctrl+D (Ctrl+Z then Enter on
|
|
85
|
+
Windows) to cancel. The same flow can be started with `tritobol --interactive`.
|
|
86
|
+
When stdin is redirected, no-argument invocation continues to read the piped
|
|
87
|
+
text as before.
|
|
88
|
+
|
|
89
|
+
The only runtime dependency is [`tokenizers`](https://pypi.org/project/tokenizers/).
|
|
90
|
+
The tokenizer JSON is bundled with the package (about 6 MB); no model weights are
|
|
91
|
+
downloaded.
|
|
92
|
+
|
|
93
|
+
## Use another model's tokenizer
|
|
94
|
+
|
|
95
|
+
`HuggingFaceTokenizer` loads a local `tokenizer.json` supported by the
|
|
96
|
+
[`tokenizers`](https://pypi.org/project/tokenizers/) library. Supply a tokenizer
|
|
97
|
+
file that matches the model you intend to use; there is no automatic model-name
|
|
98
|
+
lookup or network download.
|
|
99
|
+
|
|
100
|
+
```python
|
|
101
|
+
from tritobol import HuggingFaceTokenizer, Tritobol
|
|
102
|
+
|
|
103
|
+
tokenizer = HuggingFaceTokenizer(
|
|
104
|
+
"path/to/model/tokenizer.json",
|
|
105
|
+
model_name="my-model", # optional label for logs and inspection
|
|
106
|
+
)
|
|
107
|
+
tool = Tritobol(tokenizer=tokenizer)
|
|
108
|
+
|
|
109
|
+
result = tool.optimize(prompt, max_tokens=8192)
|
|
110
|
+
print(tool.count_tokens(prompt)) # counted with my-model's tokenizer
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The CLI accepts the same local tokenizer file:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
tritobol prompt.txt --tokenizer path/to/tokenizer.json --model-name my-model --max-tokens 8192
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Applications can also provide a custom tokenizer object implementing
|
|
120
|
+
`count(text)` and `tail(text, max_tokens)`; `tail` should return a
|
|
121
|
+
`WindowResult`. Token counts cover the supplied text only. They do not include
|
|
122
|
+
provider-specific chat templates, message framing, tools, or API-side overhead.
|
|
123
|
+
Use a tokenizer compatible with the target model and account for those additional
|
|
124
|
+
inputs separately when enforcing a request's actual context limit.
|
|
125
|
+
|
|
126
|
+
## Code and configuration safety
|
|
127
|
+
|
|
128
|
+
The default strategy is `task_type="auto"`: code-like inputs use a more
|
|
129
|
+
conservative duplicate threshold, while long-form documents use a lower
|
|
130
|
+
threshold to remove more repeated prose. You can explicitly select
|
|
131
|
+
`"coding"` or `"document"` on `Tritobol.optimize()` or `optimize_text()`.
|
|
132
|
+
|
|
133
|
+
Fenced Python blocks longer than four non-empty lines have blank lines and
|
|
134
|
+
immediately repeated ordinary comment-only lines removed; unique explanatory
|
|
135
|
+
comments are kept. `# type:`, `# noqa`, `# pylint:`, and
|
|
136
|
+
`# pragma:` directives are retained. The candidate must pass Python's local
|
|
137
|
+
syntax compiler or that segment is restored. JSON blocks are compacted only
|
|
138
|
+
after a successful parse. YAML blocks are changed only when PyYAML is already
|
|
139
|
+
installed and both parsed values match; otherwise they remain unchanged.
|
|
140
|
+
Short snippets and unsupported languages remain protected. Repeated code blocks
|
|
141
|
+
are not deduplicated. A caller-supplied hard token limit may still truncate any
|
|
142
|
+
final output, so avoid setting one below the size needed for a complete code
|
|
143
|
+
segment.
|
|
144
|
+
|
|
145
|
+
Each `OptimizationResult` includes redaction-safe `segments` logs with status,
|
|
146
|
+
reason, and token counts. The CLI also supports a JSON report with
|
|
147
|
+
`--report report.json`.
|
|
148
|
+
|
|
149
|
+
## Important implementation sections
|
|
150
|
+
|
|
151
|
+
- `tritobol/core.py` contains the deterministic optimization pipeline, token
|
|
152
|
+
statistics, segment logs, and the optional tail-window behavior.
|
|
153
|
+
- `tritobol/tokenizer.py` provides the bundled and local-tokenizer adapters;
|
|
154
|
+
token counts are based on the configured tokenizer, not an estimate.
|
|
155
|
+
- `tritobol/batch.py` handles local files and project trees, validates
|
|
156
|
+
supported code/configuration output, and writes to a separate destination.
|
|
157
|
+
- `tritobol/__main__.py` implements the CLI, including preview diffs and JSON
|
|
158
|
+
reports.
|
|
159
|
+
- `benchmarks/run_benchmarks.py` contains fixed synthetic benchmark inputs and
|
|
160
|
+
the calculations for token savings and designated-phrase retention.
|
|
161
|
+
|
|
162
|
+
## Sensitive values
|
|
163
|
+
|
|
164
|
+
Credential-shaped values such as common API-key/token assignments, recognizable
|
|
165
|
+
key prefixes, PEM private-key blocks, and explicit `CONFIDENTIAL:` or
|
|
166
|
+
`BEGIN/END CONFIDENTIAL` markers are scanned locally by default.
|
|
167
|
+
`sensitive_data="skip"` protects detected values from text normalization and
|
|
168
|
+
duplicate removal while leaving them unchanged in the output (unless a hard
|
|
169
|
+
token window later discards that part).
|
|
170
|
+
`sensitive_data="mask"` replaces detections with `[REDACTED]`; `"off"` disables
|
|
171
|
+
the scanner. This is a conservative pattern scanner, not a general detector of
|
|
172
|
+
confidential prose, and it is not a substitute for a dedicated secret scanner.
|
|
173
|
+
Sensitive values are never included in segment log details.
|
|
174
|
+
|
|
175
|
+
## Quick start
|
|
176
|
+
|
|
177
|
+
```python
|
|
178
|
+
from tritobol import Tritobol
|
|
179
|
+
|
|
180
|
+
tool = Tritobol()
|
|
181
|
+
|
|
182
|
+
raw = """
|
|
183
|
+
The deployment checklist starts with a database backup.
|
|
184
|
+
|
|
185
|
+
|
|
186
|
+
|
|
187
|
+
|
|
188
|
+
The deployment checklist starts with a database backup.
|
|
189
|
+
|
|
190
|
+
After the backup, run the migrations and check the health endpoint.
|
|
191
|
+
"""
|
|
192
|
+
|
|
193
|
+
result = tool.optimize(raw, max_tokens=4096)
|
|
194
|
+
|
|
195
|
+
print(result.text)
|
|
196
|
+
print(result.stats.summary())
|
|
197
|
+
# tokens 36 -> 22 | saved 14 (38.9%) | chars 198 -> 124 | ...
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
`optimize()` always returns an `OptimizationResult` with:
|
|
201
|
+
|
|
202
|
+
| Field | Meaning |
|
|
203
|
+
| --- | --- |
|
|
204
|
+
| `result.text` | Cleaned and possibly windowed text |
|
|
205
|
+
| `result.stats.original_tokens` | Token count before optimization |
|
|
206
|
+
| `result.stats.optimized_tokens` | Token count after optimization |
|
|
207
|
+
| `result.stats.saved_tokens` | Absolute token reduction |
|
|
208
|
+
| `result.stats.saved_percent` | Reduction as a percentage |
|
|
209
|
+
| `result.stats.truncated` | Whether the sliding window ran |
|
|
210
|
+
| `result.warning` | The warning text, even if printing was disabled |
|
|
211
|
+
|
|
212
|
+
## How it works
|
|
213
|
+
|
|
214
|
+
tritobol normalizes text, removes redundant content, counts tokens with the
|
|
215
|
+
configured tokenizer, and keeps the newest context if the optional token budget
|
|
216
|
+
is exceeded. See the [runnable demo](examples/demo.py) for complete examples.
|
|
217
|
+
From the project root, run it with:
|
|
218
|
+
|
|
219
|
+
```bash
|
|
220
|
+
python examples/demo.py
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
## Sliding window for long texts
|
|
224
|
+
|
|
225
|
+
When the trimmed text is still longer than `max_tokens`, tritobol discards the
|
|
226
|
+
**earliest** tokens and keeps the **latest** context. It prints one clear
|
|
227
|
+
warning to stderr:
|
|
228
|
+
|
|
229
|
+
```text
|
|
230
|
+
WARNING: tritobol truncated the input because it exceeded max_tokens=80: kept the latest 80 tokens and discarded the earliest 619 tokens (input was 699 tokens).
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
```python
|
|
234
|
+
long_transcript = "\n\n".join(
|
|
235
|
+
f"Section {i}: " + "detail " * 30 for i in range(20)
|
|
236
|
+
)
|
|
237
|
+
|
|
238
|
+
result = tool.optimize(long_transcript, max_tokens=80)
|
|
239
|
+
print(result.stats.truncated) # True
|
|
240
|
+
print(result.stats.discarded_tokens) # 619
|
|
241
|
+
print("Section 19" in result.text) # True - newest context survives
|
|
242
|
+
print("Section 0" in result.text) # False - oldest context was dropped
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
The cut is made at a tokenizer character offset, so the returned text is not
|
|
246
|
+
decoded through token IDs and does not acquire replacement characters.
|
|
247
|
+
|
|
248
|
+
To suppress the print while still receiving `result.warning`, pass
|
|
249
|
+
`warn_on_truncation=False`. To capture the warning yourself, pass a text stream
|
|
250
|
+
with `warning_stream=my_stream`.
|
|
251
|
+
|
|
252
|
+
## Roadmap
|
|
253
|
+
|
|
254
|
+
The current release focuses on local text cleanup, token counting, prompt
|
|
255
|
+
optimization, preview, and JSON reporting. Planned areas for future development
|
|
256
|
+
include:
|
|
257
|
+
|
|
258
|
+
- CLI conventions and machine-readable output suitable for standalone use and
|
|
259
|
+
optional integration with the KeyShield workflow.
|
|
260
|
+
- More conservative code and security-audit handling, with tests for preserving
|
|
261
|
+
identifiers, file paths, vulnerability descriptions, and key findings.
|
|
262
|
+
- Compression-risk assessment and rollback to the original input when a result
|
|
263
|
+
fails validation or is judged too risky.
|
|
264
|
+
- Project-level caching and Markdown report export.
|
|
265
|
+
- Lower-friction installation across local machines, WSL, and CI, while being
|
|
266
|
+
explicit that accurate model-specific token counts require the matching
|
|
267
|
+
tokenizer.
|
|
268
|
+
|
|
269
|
+
These are goals, not features of the current release. Tritobol does not yet
|
|
270
|
+
provide risk-scored rollback, project caching, Markdown reports, or automatic
|
|
271
|
+
model/vocabulary selection. Review previews and test outputs before using
|
|
272
|
+
optimized text in security-sensitive workflows.
|
|
273
|
+
|
|
274
|
+
## Command line
|
|
275
|
+
|
|
276
|
+
For an interactive, single-file walkthrough, run `tritobol` with no arguments
|
|
277
|
+
in a terminal (or pass `--interactive`). It uses Tritobol's default tokenizer
|
|
278
|
+
and optimization settings, and does not set a token budget unless you enter
|
|
279
|
+
one in response to the optional prompt.
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
# Read a file, write optimized text to stdout, stats to stderr
|
|
283
|
+
tritobol prompt.txt
|
|
284
|
+
|
|
285
|
+
# Apply an 8,192-token budget and write to a file
|
|
286
|
+
tritobol prompt.txt --max-tokens 8192 -o optimized.txt
|
|
287
|
+
|
|
288
|
+
# Read from stdin
|
|
289
|
+
type prompt.txt | tritobol --max-tokens 4096
|
|
290
|
+
|
|
291
|
+
# Explicitly allow trimming inside fenced Markdown code blocks
|
|
292
|
+
tritobol prompt.txt --no-protect-code-blocks
|
|
293
|
+
|
|
294
|
+
# Process a project into a separate output directory, honoring .gitignore
|
|
295
|
+
tritobol . --output optimized --report optimization-report.json
|
|
296
|
+
|
|
297
|
+
# Explicitly mask detected credential-shaped values
|
|
298
|
+
tritobol prompt.txt --sensitive-data mask
|
|
299
|
+
|
|
300
|
+
# Preview the changes without writing an optimized file
|
|
301
|
+
tritobol prompt.txt --diff
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
If `-o` is omitted, only the optimized text goes to stdout. Warnings and token
|
|
305
|
+
statistics go to stderr, so shell pipelines remain clean.
|
|
306
|
+
`--diff` instead writes a unified diff to stdout and never writes an output
|
|
307
|
+
file; it accepts one file or stdin, and cannot be combined with `-o` or project
|
|
308
|
+
directory processing. Review the diff before explicitly running again with
|
|
309
|
+
`-o optimized.txt` to save a separate copy.
|
|
310
|
+
|
|
311
|
+
Project mode writes supported text/configuration files into a separate output
|
|
312
|
+
tree, copies unsupported files unchanged, skips symbolic links, and reports
|
|
313
|
+
per-file token savings and protected/compressed segments. Aggregate token
|
|
314
|
+
counts cover optimized supported text files; copied unsupported files are
|
|
315
|
+
listed as protected but are not token-counted. Report paths are project-relative
|
|
316
|
+
to avoid including the user's full local filesystem path. The sensitive scanner
|
|
317
|
+
applies only to files that are optimized. It recognizes common
|
|
318
|
+
`.gitignore` globs, directory patterns, negation, and nested ignore files; it
|
|
319
|
+
does not claim complete parity with every Git ignore edge case. The processing
|
|
320
|
+
is local and makes no network requests. Supported optimization file types are
|
|
321
|
+
Markdown/plain-text documents, Python, JSON, and YAML. YAML validation is
|
|
322
|
+
available only when PyYAML is already installed; Tritobol does not install or
|
|
323
|
+
download it automatically.
|
|
324
|
+
|
|
325
|
+
## What gets trimmed
|
|
326
|
+
|
|
327
|
+
The default pipeline is deterministic and conservative:
|
|
328
|
+
|
|
329
|
+
1. **Fenced Markdown code blocks** — short snippets and unsupported languages
|
|
330
|
+
are preserved; supported Python and JSON blocks are selectively compressed
|
|
331
|
+
and parsed before acceptance. A hard token budget can still truncate final
|
|
332
|
+
text; disable code handling with `Tritobol(protect_code_blocks=False)` or
|
|
333
|
+
`--no-protect-code-blocks`.
|
|
334
|
+
2. **Invisible characters** — BOM, zero-width characters, and control bytes in
|
|
335
|
+
prose.
|
|
336
|
+
3. **HTML comments** — `<!-- ... -->` blocks in prose.
|
|
337
|
+
4. **Whitespace noise** — indentation, trailing spaces, and repeated blank lines
|
|
338
|
+
in prose.
|
|
339
|
+
5. **Decorative separators** — lines made only of `=`, `-`, `*`, `_`, `#`, etc.
|
|
340
|
+
6. **Duplicate paragraphs** — exact repeats after whitespace/case normalization.
|
|
341
|
+
7. **Duplicate lines** — repeated substantive lines; the first occurrence wins.
|
|
342
|
+
8. **Duplicate sentences** — repeated sentences, including inside paragraphs.
|
|
343
|
+
9. **Sliding window** — only when `max_tokens` is supplied and still exceeded.
|
|
344
|
+
|
|
345
|
+
Short strings are protected from duplicate removal. The default threshold is
|
|
346
|
+
24 characters for coding-oriented input and 12 for document-oriented input; you
|
|
347
|
+
can explicitly adjust the base threshold:
|
|
348
|
+
|
|
349
|
+
```python
|
|
350
|
+
tool = Tritobol(min_duplicate_chars=40)
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
Because only redundant material is removed, savings depend on the input. Noisy
|
|
354
|
+
prompts may save more than already-clean prose, which may save nothing. The
|
|
355
|
+
included synthetic benchmark gives reproducible measurements for its fixed
|
|
356
|
+
examples; those results should not be generalized to arbitrary prompts or
|
|
357
|
+
treated as a semantic-quality guarantee. A hard `max_tokens` budget can
|
|
358
|
+
produce a larger reduction by discarding earlier context.
|
|
359
|
+
|
|
360
|
+
## Reproducible benchmark
|
|
361
|
+
|
|
362
|
+
Run the bundled local benchmark from the project root:
|
|
363
|
+
|
|
364
|
+
```bash
|
|
365
|
+
python -m benchmarks.run_benchmarks
|
|
366
|
+
python -m benchmarks.run_benchmarks --json
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
It uses three fixed, synthetic examples and the bundled DeepSeek V4.1 Flash
|
|
370
|
+
tokenizer, with no model/API calls and no hard token budget. For each example
|
|
371
|
+
and in aggregate it reports original and optimized token counts, saved tokens
|
|
372
|
+
and percentage, plus retention of explicitly listed phrases. Phrase retention
|
|
373
|
+
is a literal substring check, not an assessment of meaning, factual accuracy,
|
|
374
|
+
or model performance. This small fixture suite is useful for detecting
|
|
375
|
+
behavior changes against stable examples; it is not a representative corpus or
|
|
376
|
+
a claim about real-world savings or information preservation.
|
|
377
|
+
|
|
378
|
+
## API surface
|
|
379
|
+
|
|
380
|
+
```python
|
|
381
|
+
from tritobol import (
|
|
382
|
+
DeepSeekTokenizer,
|
|
383
|
+
HuggingFaceTokenizer,
|
|
384
|
+
Tritobol,
|
|
385
|
+
optimize_text,
|
|
386
|
+
)
|
|
387
|
+
|
|
388
|
+
tool = Tritobol()
|
|
389
|
+
|
|
390
|
+
tool.count_tokens(text) # exact count for the configured tokenizer
|
|
391
|
+
tool.trim(text) # redundancy removal only
|
|
392
|
+
tool.fit_window(text, max_tokens) # sliding window only
|
|
393
|
+
tool.optimize(text, max_tokens=None) # trim + optional window + stats
|
|
394
|
+
|
|
395
|
+
optimize_text(text, max_tokens=1024) # convenience wrapper
|
|
396
|
+
|
|
397
|
+
# Existing DeepSeekTokenizer remains available; HuggingFaceTokenizer accepts
|
|
398
|
+
# another compatible local tokenizer.json.
|
|
399
|
+
other_tool = Tritobol(
|
|
400
|
+
tokenizer=HuggingFaceTokenizer("path/to/tokenizer.json")
|
|
401
|
+
)
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
Process a project tree locally from Python:
|
|
405
|
+
|
|
406
|
+
```python
|
|
407
|
+
from pathlib import Path
|
|
408
|
+
from tritobol import process_path
|
|
409
|
+
|
|
410
|
+
report = process_path(
|
|
411
|
+
"my-project",
|
|
412
|
+
"optimized-project",
|
|
413
|
+
respect_gitignore=True,
|
|
414
|
+
task_type="auto",
|
|
415
|
+
sensitive_data="skip",
|
|
416
|
+
)
|
|
417
|
+
print(report.summary())
|
|
418
|
+
report.write_json(Path("optimization-report.json"))
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
## Tests
|
|
422
|
+
|
|
423
|
+
```bash
|
|
424
|
+
python -m unittest discover -s tests -v
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
The test cases print original token count, optimized token count, and saved
|
|
428
|
+
percentage for each scenario:
|
|
429
|
+
|
|
430
|
+
```text
|
|
431
|
+
[low noise / whitespace] original=45 optimized=39 saved=13.3%
|
|
432
|
+
[duplicate paragraphs] original=112 optimized=62 saved=44.6%
|
|
433
|
+
[high noise / repeated blocks] original=180 optimized=42 saved=76.7%
|
|
434
|
+
[long text / sliding window] original=700 optimized=80 saved=88.6%
|
|
435
|
+
```
|
|
436
|
+
|
|
437
|
+
A runnable demonstration is also included:
|
|
438
|
+
|
|
439
|
+
```bash
|
|
440
|
+
python examples/demo.py
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
## Bundled tokenizer provenance
|
|
444
|
+
|
|
445
|
+
The bundled file is `tokenizer.json` from the official
|
|
446
|
+
[`deepseek-ai/DeepSeek-V4.1-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
|
|
447
|
+
checkpoint:
|
|
448
|
+
|
|
449
|
+
- Vocabulary: 129,280 tokens
|
|
450
|
+
- License: MIT, Copyright (c) 2023 DeepSeek
|
|
451
|
+
|
|
452
|
+
See `tritobol/data/README.md` and `tritobol/data/LICENSE.deepseek-tokenizer` for
|
|
453
|
+
details.
|
|
454
|
+
|
|
455
|
+
## Analysis & details
|
|
456
|
+
|
|
457
|
+
Please click [project_overview](PROJECT_OVERVIEW.md) for analyses & details.
|
|
458
|
+
|
|
459
|
+
## License
|
|
460
|
+
|
|
461
|
+
The tritobol source is MIT licensed. The bundled tokenizer retains DeepSeek's
|
|
462
|
+
MIT license notice.
|
|
463
|
+
|
|
464
|
+
## Web application (local UI)
|
|
465
|
+
|
|
466
|
+
A local-first browser interface lives in `tritobol_app/`. It includes account
|
|
467
|
+
registration and sign-in; account data is stored in the installation's SQLite
|
|
468
|
+
database, while prompt history stays in the browser. To run it from a source
|
|
469
|
+
clone, install the optional app dependency and start the local server:
|
|
470
|
+
|
|
471
|
+
```bash
|
|
472
|
+
python -m pip install -e ".[app]"
|
|
473
|
+
python -m tritobol_app
|
|
474
|
+
```
|
|
475
|
+
|
|
476
|
+
On Windows, you can instead double-click `start.bat` (Python 3.10+ required).
|
|
477
|
+
The app binds to `127.0.0.1` and uses Flask's development server; it is for
|
|
478
|
+
local use only.
|
|
479
|
+
|
|
480
|
+
### Deployment status
|
|
481
|
+
|
|
482
|
+
Public deployment, including Netlify deployment, is not ready and is deferred.
|
|
483
|
+
After the hackathon winner announcements, interested team members may decide
|
|
484
|
+
whether to continue the project and plan deployment. No public hosting is
|
|
485
|
+
currently available.
|
|
486
|
+
|
|
487
|
+
See [`docs/APPLICATION.md`](docs/APPLICATION.md) for more about the local app.
|