tritobol 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
tritobol-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 tritobol contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,487 @@
1
+ Metadata-Version: 2.4
2
+ Name: tritobol
3
+ Version: 0.1.0
4
+ Summary: Trim redundant prompt text and keep the newest context within a model-specific token budget.
5
+ Author: tritobol contributors
6
+ License-Expression: MIT
7
+ Keywords: llm,tokenizer,prompt,context,preprocessing,multi-model
8
+ Classifier: Development Status :: 3 - Alpha
9
+ Classifier: Intended Audience :: Developers
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: Programming Language :: Python :: 3.10
12
+ Classifier: Programming Language :: Python :: 3.11
13
+ Classifier: Programming Language :: Python :: 3.12
14
+ Classifier: Programming Language :: Python :: 3.13
15
+ Classifier: Topic :: Text Processing
16
+ Requires-Python: >=3.10
17
+ Description-Content-Type: text/markdown
18
+ License-File: LICENSE
19
+ Requires-Dist: tokenizers>=0.19
20
+ Provides-Extra: app
21
+ Requires-Dist: flask>=3.0; extra == "app"
22
+ Provides-Extra: dev
23
+ Requires-Dist: pytest>=7; extra == "dev"
24
+ Requires-Dist: hypothesis>=6; extra == "dev"
25
+ Dynamic: license-file
26
+
27
+ # tritobol
28
+
29
+ ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python&logoColor=white)
30
+ ![Multi-model compatible](https://img.shields.io/badge/Tokenizer-Multi--model%20compatible-6C5CE7)
31
+ ![CLI app](https://img.shields.io/badge/Interface-CLI%20app-2D3436)
32
+ ![Token and prompt optimizer](https://img.shields.io/badge/Tool-Token%20%26%20prompt%20optimizer-00B894)
33
+ ![Kiroween 2026 Hackathon](https://img.shields.io/badge/Kiroween%202026%20Hackathon-F39C12)
34
+ ![Built with heart while learning](https://img.shields.io/badge/Built%20with%20heart-While%20learning-E84393)
35
+
36
+ **trim + token + bolster** — a lightweight Python token preprocessor that removes
37
+ redundant prompt text and keeps the most recent context when an input is too
38
+ long.
39
+
40
+ `tritobol` does **not** summarize or rewrite your content. It removes repeated
41
+ material, whitespace noise, and decorative separators. It conservatively
42
+ compresses supported code/configuration blocks only when local validation
43
+ passes, and — only if a hard token budget is still exceeded — slides the window
44
+ forward so the newest tokens survive.
45
+
46
+ By default, token counts use the bundled **DeepSeek V4.1 Flash** tokenizer
47
+ (129,280-token vocabulary). To optimize for another model, load its compatible
48
+ tokenizer locally; the optimizer, trimming pipeline, and token-budget logic are
49
+ independent of any one provider.
50
+
51
+ ## Install
52
+
53
+ The repository and clone directory are named `tritobol-token_optimizer-`;
54
+ the Python package and CLI command are named `tritobol`. To install from a
55
+ source clone and start the guided workflow:
56
+
57
+ ```bash
58
+ git clone https://github.com/pizzafeastbuddies/tritobol-token_optimizer-.git
59
+ cd tritobol-token_optimizer-
60
+ python -m pip install -e .
61
+ tritobol
62
+ ```
63
+
64
+ For a non-editable source install, run this from the repository root instead:
65
+
66
+ ```bash
67
+ python -m pip install .
68
+ ```
69
+
70
+ ### Publishing to PyPI (maintainers)
71
+
72
+ Configure a PyPI trusted publisher for the `pizzafeastbuddies/tritobol-token_optimizer-`
73
+ repository, using `.github/workflows/publish.yml` and the GitHub Actions
74
+ environment `pypi`. Publishing is triggered by a non-prerelease GitHub release;
75
+ the release tag must match the package version in `pyproject.toml` (for example,
76
+ `v0.1.0`).
77
+
78
+ The guide asks for one UTF-8 text file, optionally asks for a maximum token
79
+ budget, and shows token counts plus a unified diff before saving. A budget may
80
+ discard the oldest text when the optimized result is still too long; a blank
81
+ budget keeps the normal no-hard-limit behavior. The suggested output is a
82
+ separate `*.optimized.*` file; Tritobol never overwrites the source and will
83
+ not replace an existing output. Saving requires an explicit confirmation.
84
+ Press Enter at the file prompt or use Ctrl+C/Ctrl+D (Ctrl+Z then Enter on
85
+ Windows) to cancel. The same flow can be started with `tritobol --interactive`.
86
+ When stdin is redirected, no-argument invocation continues to read the piped
87
+ text as before.
88
+
89
+ The only runtime dependency is [`tokenizers`](https://pypi.org/project/tokenizers/).
90
+ The tokenizer JSON is bundled with the package (about 6 MB); no model weights are
91
+ downloaded.
92
+
93
+ ## Use another model's tokenizer
94
+
95
+ `HuggingFaceTokenizer` loads a local `tokenizer.json` supported by the
96
+ [`tokenizers`](https://pypi.org/project/tokenizers/) library. Supply a tokenizer
97
+ file that matches the model you intend to use; there is no automatic model-name
98
+ lookup or network download.
99
+
100
+ ```python
101
+ from tritobol import HuggingFaceTokenizer, Tritobol
102
+
103
+ tokenizer = HuggingFaceTokenizer(
104
+ "path/to/model/tokenizer.json",
105
+ model_name="my-model", # optional label for logs and inspection
106
+ )
107
+ tool = Tritobol(tokenizer=tokenizer)
108
+
109
+ result = tool.optimize(prompt, max_tokens=8192)
110
+ print(tool.count_tokens(prompt)) # counted with my-model's tokenizer
111
+ ```
112
+
113
+ The CLI accepts the same local tokenizer file:
114
+
115
+ ```bash
116
+ tritobol prompt.txt --tokenizer path/to/tokenizer.json --model-name my-model --max-tokens 8192
117
+ ```
118
+
119
+ Applications can also provide a custom tokenizer object implementing
120
+ `count(text)` and `tail(text, max_tokens)`; `tail` should return a
121
+ `WindowResult`. Token counts cover the supplied text only. They do not include
122
+ provider-specific chat templates, message framing, tools, or API-side overhead.
123
+ Use a tokenizer compatible with the target model and account for those additional
124
+ inputs separately when enforcing a request's actual context limit.
125
+
126
+ ## Code and configuration safety
127
+
128
+ The default strategy is `task_type="auto"`: code-like inputs use a more
129
+ conservative duplicate threshold, while long-form documents use a lower
130
+ threshold to remove more repeated prose. You can explicitly select
131
+ `"coding"` or `"document"` on `Tritobol.optimize()` or `optimize_text()`.
132
+
133
+ Fenced Python blocks longer than four non-empty lines have blank lines and
134
+ immediately repeated ordinary comment-only lines removed; unique explanatory
135
+ comments are kept. `# type:`, `# noqa`, `# pylint:`, and
136
+ `# pragma:` directives are retained. The candidate must pass Python's local
137
+ syntax compiler or that segment is restored. JSON blocks are compacted only
138
+ after a successful parse. YAML blocks are changed only when PyYAML is already
139
+ installed and both parsed values match; otherwise they remain unchanged.
140
+ Short snippets and unsupported languages remain protected. Repeated code blocks
141
+ are not deduplicated. A caller-supplied hard token limit may still truncate any
142
+ final output, so avoid setting one below the size needed for a complete code
143
+ segment.
144
+
145
+ Each `OptimizationResult` includes redaction-safe `segments` logs with status,
146
+ reason, and token counts. The CLI also supports a JSON report with
147
+ `--report report.json`.
148
+
149
+ ## Important implementation sections
150
+
151
+ - `tritobol/core.py` contains the deterministic optimization pipeline, token
152
+ statistics, segment logs, and the optional tail-window behavior.
153
+ - `tritobol/tokenizer.py` provides the bundled and local-tokenizer adapters;
154
+ token counts are based on the configured tokenizer, not an estimate.
155
+ - `tritobol/batch.py` handles local files and project trees, validates
156
+ supported code/configuration output, and writes to a separate destination.
157
+ - `tritobol/__main__.py` implements the CLI, including preview diffs and JSON
158
+ reports.
159
+ - `benchmarks/run_benchmarks.py` contains fixed synthetic benchmark inputs and
160
+ the calculations for token savings and designated-phrase retention.
161
+
162
+ ## Sensitive values
163
+
164
+ Credential-shaped values such as common API-key/token assignments, recognizable
165
+ key prefixes, PEM private-key blocks, and explicit `CONFIDENTIAL:` or
166
+ `BEGIN/END CONFIDENTIAL` markers are scanned locally by default.
167
+ `sensitive_data="skip"` protects detected values from text normalization and
168
+ duplicate removal while leaving them unchanged in the output (unless a hard
169
+ token window later discards that part).
170
+ `sensitive_data="mask"` replaces detections with `[REDACTED]`; `"off"` disables
171
+ the scanner. This is a conservative pattern scanner, not a general detector of
172
+ confidential prose, and it is not a substitute for a dedicated secret scanner.
173
+ Sensitive values are never included in segment log details.
174
+
175
+ ## Quick start
176
+
177
+ ```python
178
+ from tritobol import Tritobol
179
+
180
+ tool = Tritobol()
181
+
182
+ raw = """
183
+ The deployment checklist starts with a database backup.
184
+
185
+
186
+
187
+
188
+ The deployment checklist starts with a database backup.
189
+
190
+ After the backup, run the migrations and check the health endpoint.
191
+ """
192
+
193
+ result = tool.optimize(raw, max_tokens=4096)
194
+
195
+ print(result.text)
196
+ print(result.stats.summary())
197
+ # tokens 36 -> 22 | saved 14 (38.9%) | chars 198 -> 124 | ...
198
+ ```
199
+
200
+ `optimize()` always returns an `OptimizationResult` with:
201
+
202
+ | Field | Meaning |
203
+ | --- | --- |
204
+ | `result.text` | Cleaned and possibly windowed text |
205
+ | `result.stats.original_tokens` | Token count before optimization |
206
+ | `result.stats.optimized_tokens` | Token count after optimization |
207
+ | `result.stats.saved_tokens` | Absolute token reduction |
208
+ | `result.stats.saved_percent` | Reduction as a percentage |
209
+ | `result.stats.truncated` | Whether the sliding window ran |
210
+ | `result.warning` | The warning text, even if printing was disabled |
211
+
212
+ ## How it works
213
+
214
+ tritobol normalizes text, removes redundant content, counts tokens with the
215
+ configured tokenizer, and keeps the newest context if the optional token budget
216
+ is exceeded. See the [runnable demo](examples/demo.py) for complete examples.
217
+ From the project root, run it with:
218
+
219
+ ```bash
220
+ python examples/demo.py
221
+ ```
222
+
223
+ ## Sliding window for long texts
224
+
225
+ When the trimmed text is still longer than `max_tokens`, tritobol discards the
226
+ **earliest** tokens and keeps the **latest** context. It prints one clear
227
+ warning to stderr:
228
+
229
+ ```text
230
+ WARNING: tritobol truncated the input because it exceeded max_tokens=80: kept the latest 80 tokens and discarded the earliest 619 tokens (input was 699 tokens).
231
+ ```
232
+
233
+ ```python
234
+ long_transcript = "\n\n".join(
235
+ f"Section {i}: " + "detail " * 30 for i in range(20)
236
+ )
237
+
238
+ result = tool.optimize(long_transcript, max_tokens=80)
239
+ print(result.stats.truncated) # True
240
+ print(result.stats.discarded_tokens) # 619
241
+ print("Section 19" in result.text) # True - newest context survives
242
+ print("Section 0" in result.text) # False - oldest context was dropped
243
+ ```
244
+
245
+ The cut is made at a tokenizer character offset, so the returned text is not
246
+ decoded through token IDs and does not acquire replacement characters.
247
+
248
+ To suppress the print while still receiving `result.warning`, pass
249
+ `warn_on_truncation=False`. To capture the warning yourself, pass a text stream
250
+ with `warning_stream=my_stream`.
251
+
252
+ ## Roadmap
253
+
254
+ The current release focuses on local text cleanup, token counting, prompt
255
+ optimization, preview, and JSON reporting. Planned areas for future development
256
+ include:
257
+
258
+ - CLI conventions and machine-readable output suitable for standalone use and
259
+ optional integration with the KeyShield workflow.
260
+ - More conservative code and security-audit handling, with tests for preserving
261
+ identifiers, file paths, vulnerability descriptions, and key findings.
262
+ - Compression-risk assessment and rollback to the original input when a result
263
+ fails validation or is judged too risky.
264
+ - Project-level caching and Markdown report export.
265
+ - Lower-friction installation across local machines, WSL, and CI, while being
266
+ explicit that accurate model-specific token counts require the matching
267
+ tokenizer.
268
+
269
+ These are goals, not features of the current release. Tritobol does not yet
270
+ provide risk-scored rollback, project caching, Markdown reports, or automatic
271
+ model/vocabulary selection. Review previews and test outputs before using
272
+ optimized text in security-sensitive workflows.
273
+
274
+ ## Command line
275
+
276
+ For an interactive, single-file walkthrough, run `tritobol` with no arguments
277
+ in a terminal (or pass `--interactive`). It uses Tritobol's default tokenizer
278
+ and optimization settings, and does not set a token budget unless you enter
279
+ one in response to the optional prompt.
280
+
281
+ ```bash
282
+ # Read a file, write optimized text to stdout, stats to stderr
283
+ tritobol prompt.txt
284
+
285
+ # Apply an 8,192-token budget and write to a file
286
+ tritobol prompt.txt --max-tokens 8192 -o optimized.txt
287
+
288
+ # Read from stdin
289
+ type prompt.txt | tritobol --max-tokens 4096
290
+
291
+ # Explicitly allow trimming inside fenced Markdown code blocks
292
+ tritobol prompt.txt --no-protect-code-blocks
293
+
294
+ # Process a project into a separate output directory, honoring .gitignore
295
+ tritobol . --output optimized --report optimization-report.json
296
+
297
+ # Explicitly mask detected credential-shaped values
298
+ tritobol prompt.txt --sensitive-data mask
299
+
300
+ # Preview the changes without writing an optimized file
301
+ tritobol prompt.txt --diff
302
+ ```
303
+
304
+ If `-o` is omitted, only the optimized text goes to stdout. Warnings and token
305
+ statistics go to stderr, so shell pipelines remain clean.
306
+ `--diff` instead writes a unified diff to stdout and never writes an output
307
+ file; it accepts one file or stdin, and cannot be combined with `-o` or project
308
+ directory processing. Review the diff before explicitly running again with
309
+ `-o optimized.txt` to save a separate copy.
310
+
311
+ Project mode writes supported text/configuration files into a separate output
312
+ tree, copies unsupported files unchanged, skips symbolic links, and reports
313
+ per-file token savings and protected/compressed segments. Aggregate token
314
+ counts cover optimized supported text files; copied unsupported files are
315
+ listed as protected but are not token-counted. Report paths are project-relative
316
+ to avoid including the user's full local filesystem path. The sensitive scanner
317
+ applies only to files that are optimized. It recognizes common
318
+ `.gitignore` globs, directory patterns, negation, and nested ignore files; it
319
+ does not claim complete parity with every Git ignore edge case. The processing
320
+ is local and makes no network requests. Supported optimization file types are
321
+ Markdown/plain-text documents, Python, JSON, and YAML. YAML validation is
322
+ available only when PyYAML is already installed; Tritobol does not install or
323
+ download it automatically.
324
+
325
+ ## What gets trimmed
326
+
327
+ The default pipeline is deterministic and conservative:
328
+
329
+ 1. **Fenced Markdown code blocks** — short snippets and unsupported languages
330
+ are preserved; supported Python and JSON blocks are selectively compressed
331
+ and parsed before acceptance. A hard token budget can still truncate final
332
+ text; disable code handling with `Tritobol(protect_code_blocks=False)` or
333
+ `--no-protect-code-blocks`.
334
+ 2. **Invisible characters** — BOM, zero-width characters, and control bytes in
335
+ prose.
336
+ 3. **HTML comments** — `<!-- ... -->` blocks in prose.
337
+ 4. **Whitespace noise** — indentation, trailing spaces, and repeated blank lines
338
+ in prose.
339
+ 5. **Decorative separators** — lines made only of `=`, `-`, `*`, `_`, `#`, etc.
340
+ 6. **Duplicate paragraphs** — exact repeats after whitespace/case normalization.
341
+ 7. **Duplicate lines** — repeated substantive lines; the first occurrence wins.
342
+ 8. **Duplicate sentences** — repeated sentences, including inside paragraphs.
343
+ 9. **Sliding window** — only when `max_tokens` is supplied and still exceeded.
344
+
345
+ Short strings are protected from duplicate removal. The default threshold is
346
+ 24 characters for coding-oriented input and 12 for document-oriented input; you
347
+ can explicitly adjust the base threshold:
348
+
349
+ ```python
350
+ tool = Tritobol(min_duplicate_chars=40)
351
+ ```
352
+
353
+ Because only redundant material is removed, savings depend on the input. Noisy
354
+ prompts may save more than already-clean prose, which may save nothing. The
355
+ included synthetic benchmark gives reproducible measurements for its fixed
356
+ examples; those results should not be generalized to arbitrary prompts or
357
+ treated as a semantic-quality guarantee. A hard `max_tokens` budget can
358
+ produce a larger reduction by discarding earlier context.
359
+
360
+ ## Reproducible benchmark
361
+
362
+ Run the bundled local benchmark from the project root:
363
+
364
+ ```bash
365
+ python -m benchmarks.run_benchmarks
366
+ python -m benchmarks.run_benchmarks --json
367
+ ```
368
+
369
+ It uses three fixed, synthetic examples and the bundled DeepSeek V4.1 Flash
370
+ tokenizer, with no model/API calls and no hard token budget. For each example
371
+ and in aggregate it reports original and optimized token counts, saved tokens
372
+ and percentage, plus retention of explicitly listed phrases. Phrase retention
373
+ is a literal substring check, not an assessment of meaning, factual accuracy,
374
+ or model performance. This small fixture suite is useful for detecting
375
+ behavior changes against stable examples; it is not a representative corpus or
376
+ a claim about real-world savings or information preservation.
377
+
378
+ ## API surface
379
+
380
+ ```python
381
+ from tritobol import (
382
+ DeepSeekTokenizer,
383
+ HuggingFaceTokenizer,
384
+ Tritobol,
385
+ optimize_text,
386
+ )
387
+
388
+ tool = Tritobol()
389
+
390
+ tool.count_tokens(text) # exact count for the configured tokenizer
391
+ tool.trim(text) # redundancy removal only
392
+ tool.fit_window(text, max_tokens) # sliding window only
393
+ tool.optimize(text, max_tokens=None) # trim + optional window + stats
394
+
395
+ optimize_text(text, max_tokens=1024) # convenience wrapper
396
+
397
+ # Existing DeepSeekTokenizer remains available; HuggingFaceTokenizer accepts
398
+ # another compatible local tokenizer.json.
399
+ other_tool = Tritobol(
400
+ tokenizer=HuggingFaceTokenizer("path/to/tokenizer.json")
401
+ )
402
+ ```
403
+
404
+ Process a project tree locally from Python:
405
+
406
+ ```python
407
+ from pathlib import Path
408
+ from tritobol import process_path
409
+
410
+ report = process_path(
411
+ "my-project",
412
+ "optimized-project",
413
+ respect_gitignore=True,
414
+ task_type="auto",
415
+ sensitive_data="skip",
416
+ )
417
+ print(report.summary())
418
+ report.write_json(Path("optimization-report.json"))
419
+ ```
420
+
421
+ ## Tests
422
+
423
+ ```bash
424
+ python -m unittest discover -s tests -v
425
+ ```
426
+
427
+ The test cases print original token count, optimized token count, and saved
428
+ percentage for each scenario:
429
+
430
+ ```text
431
+ [low noise / whitespace] original=45 optimized=39 saved=13.3%
432
+ [duplicate paragraphs] original=112 optimized=62 saved=44.6%
433
+ [high noise / repeated blocks] original=180 optimized=42 saved=76.7%
434
+ [long text / sliding window] original=700 optimized=80 saved=88.6%
435
+ ```
436
+
437
+ A runnable demonstration is also included:
438
+
439
+ ```bash
440
+ python examples/demo.py
441
+ ```
442
+
443
+ ## Bundled tokenizer provenance
444
+
445
+ The bundled file is `tokenizer.json` from the official
446
+ [`deepseek-ai/DeepSeek-V4.1-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
447
+ checkpoint:
448
+
449
+ - Vocabulary: 129,280 tokens
450
+ - License: MIT, Copyright (c) 2023 DeepSeek
451
+
452
+ See `tritobol/data/README.md` and `tritobol/data/LICENSE.deepseek-tokenizer` for
453
+ details.
454
+
455
+ ## Analysis & details
456
+
457
+ Please click [project_overview](PROJECT_OVERVIEW.md) for analyses & details.
458
+
459
+ ## License
460
+
461
+ The tritobol source is MIT licensed. The bundled tokenizer retains DeepSeek's
462
+ MIT license notice.
463
+
464
+ ## Web application (local UI)
465
+
466
+ A local-first browser interface lives in `tritobol_app/`. It includes account
467
+ registration and sign-in; account data is stored in the installation's SQLite
468
+ database, while prompt history stays in the browser. To run it from a source
469
+ clone, install the optional app dependency and start the local server:
470
+
471
+ ```bash
472
+ python -m pip install -e ".[app]"
473
+ python -m tritobol_app
474
+ ```
475
+
476
+ On Windows, you can instead double-click `start.bat` (Python 3.10+ required).
477
+ The app binds to `127.0.0.1` and uses Flask's development server; it is for
478
+ local use only.
479
+
480
+ ### Deployment status
481
+
482
+ Public deployment, including Netlify deployment, is not ready and is deferred.
483
+ After the hackathon winner announcements, interested team members may decide
484
+ whether to continue the project and plan deployment. No public hosting is
485
+ currently available.
486
+
487
+ See [`docs/APPLICATION.md`](docs/APPLICATION.md) for more about the local app.