layoutlm-forge 0.1.0__tar.gz

layoutlm_forge-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dhruv
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

layoutlm_forge-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE
+include README.md
+include pyproject.toml
+recursive-include src/layoutlm_forge py.typed
+recursive-include docs *.md
+recursive-include examples *.py

layoutlm_forge-0.1.0/PKG-INFO

@@ -0,0 +1,313 @@
+Metadata-Version: 2.4
+Name: layoutlm-forge
+Version: 0.1.0
+Summary: Production-grade LLMOps infrastructure for context window management, token counting, document chunking, and compression
+Author: Dhruv
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/dhruv-atomic-mui21/layoutlm-forge
+Project-URL: Repository, https://github.com/dhruv-atomic-mui21/layoutlm-forge
+Project-URL: Documentation, https://github.com/dhruv-atomic-mui21/layoutlm-forge/tree/main/docs
+Project-URL: Bug Tracker, https://github.com/dhruv-atomic-mui21/layoutlm-forge/issues
+Project-URL: Changelog, https://github.com/dhruv-atomic-mui21/layoutlm-forge/releases
+Keywords: llm,tokens,tokenizer,chunking,context-window,context-management,llmops,rag,prompt-engineering,openai,anthropic
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: typer>=0.9.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: api
+Requires-Dist: fastapi>=0.95.0; extra == "api"
+Requires-Dist: uvicorn>=0.21.0; extra == "api"
+Requires-Dist: pydantic>=2.0.0; extra == "api"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-benchmark; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: httpx>=0.24.0; extra == "dev"
+Requires-Dist: fastapi>=0.95.0; extra == "dev"
+Requires-Dist: uvicorn>=0.21.0; extra == "dev"
+Requires-Dist: pydantic>=2.0.0; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs; extra == "docs"
+Requires-Dist: mkdocs-material; extra == "docs"
+Dynamic: license-file
+
+<div align="center">
+  <h1>🔥 LayoutLM Forge</h1>
+  <p><b>Production-Grade LLMOps Infrastructure for Context Window Management</b></p>
+  <p><i>Deterministic token counting · Intelligent chunking · Priority-based context assembly · Cost estimation — the foundation every AI application needs.</i></p>
+
+  [![Tests](https://github.com/dhruv-atomic-mui21/layoutlm_forge/workflows/Tests/badge.svg)](https://github.com/dhruv-atomic-mui21/layoutlm_forge/actions)
+  [![PyPI](https://img.shields.io/pypi/v/layoutlm_forge.svg)](https://pypi.org/project/layoutlm_forge/)
+  [![Python](https://img.shields.io/pypi/pyversions/layoutlm_forge.svg)](https://pypi.org/project/layoutlm_forge/)
+  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+  [![Downloads](https://img.shields.io/pypi/dm/layoutlm_forge.svg)](https://pypi.org/project/layoutlm_forge/)
+</div>
+
+---
+
+## Why LayoutLM Forge?
+
+Every production AI application eventually hits the same infrastructure problems:
+
+| Problem | Impact | LayoutLM Forge Solution |
+|---------|--------|-----------------------|
+| 🎯 Context window overflow | Silent failures, truncated responses | Priority-based assembly with overflow tracking |
+| 📊 Inaccurate token counting | Budget overruns, dropped requests | Deterministic counting via tiktoken + heuristic fallbacks |
+| 🔄 Naive text splitting | Broken semantics, degraded LLM reasoning | 5 chunking strategies (sentence, paragraph, semantic, code, fixed) |
+| 💸 Unpredictable API costs | Surprise bills, no cost governance | Pre-flight cost estimation across 15+ models |
+| 🗜️ Oversized prompts | Wasted tokens, slow responses | 4 compression strategies (extractive, truncate, middle-out, map-reduce) |
+
+## Installation
+
+```bash
+pip install layoutlm_forge
+```
+
+With API server support:
+```bash
+pip install "layoutlm_forge[api]"
+```
+
+## Quick Start
+
+### Token Counting
+
+```python
+from layoutlm_forge import TokenCounter
+
+counter = TokenCounter("gpt-4o")
+tokens = counter.count("Hello, world!")
+print(f"Tokens: {tokens}")  # Tokens: 4
+
+# Check context window fit
+fits = counter.fits_in_window("Your prompt...", reserve_output=500)
+
+# Estimate cost before sending
+cost = counter.estimate_cost("Your prompt...", direction="input")
+print(f"Cost: ${cost:.6f}")
+```
+
+### Intelligent Chunking
+
+```python
+from layoutlm_forge import DocumentChunker, ChunkStrategy
+
+chunker = DocumentChunker("gpt-4o")
+
+# Chunk respecting paragraph boundaries
+chunks = chunker.chunk(
+    long_document,
+    strategy=ChunkStrategy.PARAGRAPH,
+    max_tokens=500,
+    overlap_tokens=50,
+)
+
+# Specialized chunkers
+code_chunks = chunker.chunk_code(source_code, language="python")
+md_chunks = chunker.chunk_markdown(readme_text)
+```
+
+### Priority-Based Context Assembly
+
+The core pattern for RAG applications — guarantee critical context fits while gracefully dropping lower-priority content:
+
+```python
+from layoutlm_forge import ContextWindow, Priority
+
+window = ContextWindow("gpt-4o")
+
+# System instructions — always included
+window.add_block("You are a legal assistant.", Priority.CRITICAL, "system")
+
+# User query — high priority
+window.add_block("What is the statute of limitations?", Priority.HIGH, "query")
+
+# RAG search results — included if space permits
+window.add_block(search_result_1, Priority.MEDIUM, "rag_1")
+window.add_block(search_result_2, Priority.LOW, "rag_2")
+
+# Assemble: packs highest-priority blocks first
+prompt = window.assemble(max_tokens=4096)
+
+# See what was included/dropped
+usage = window.usage()
+print(f"Included: {usage['num_included']} blocks ({usage['included_tokens']} tokens)")
+print(f"Dropped:  {usage['num_excluded']} blocks")
+```
+
+### Cost Estimation
+
+```python
+from layoutlm_forge import CostCalculator
+
+calc = CostCalculator("gpt-4o")
+
+# Single prompt cost
+cost = calc.estimate_prompt("Your prompt text here")
+print(f"Input cost: ${cost.usd:.6f}")
+
+# Compare models
+comparison = calc.compare_models(
+    texts=["Document 1...", "Document 2..."],
+    models=["gpt-4o", "gpt-4o-mini", "claude-3.5-sonnet", "gemini-flash"],
+)
+for model, analysis in comparison.items():
+    print(f"{model}: ${analysis.total_usd:.6f} for {analysis.total_tokens} tokens")
+```
+
+### Context Compression
+
+```python
+from layoutlm_forge import ContextCompressor, CompressionStrategy
+
+compressor = ContextCompressor("gpt-4o")
+
+# Extractive: keeps most important sentences via TF-IDF scoring
+result = compressor.compress(long_text, target_tokens=200)
+print(f"Compressed: {result.original_tokens} → {result.compressed_tokens} tokens")
+print(f"Savings: {result.savings_pct:.1f}%")
+
+# Middle-out: preserves start and end, removes middle
+result = compressor.compress(log_text, target_tokens=300, strategy=CompressionStrategy.MIDDLE_OUT)
+```
+
+### Conversation Management
+
+```python
+from layoutlm_forge import ConversationManager
+
+manager = ConversationManager("gpt-4o")
+
+manager.add_message("system", "You are a helpful Python tutor.")
+manager.add_message("user", "Explain decorators")
+manager.add_message("assistant", "Decorators are...")
+# ... many more turns ...
+
+# Auto-trim older messages to fit budget, preserving system prompt
+trimmed = manager.get_context(max_tokens=4096, preserve_system=True)
+```
+
+## Supported Models
+
+| Provider | Models | Token Counting | Pricing |
+|----------|--------|:--------------:|:-------:|
+| **OpenAI** | GPT-4, GPT-4 Turbo, GPT-4o, GPT-4o-mini, GPT-3.5 Turbo | ✅ tiktoken | ✅ |
+| **Anthropic** | Claude 3 Opus, Claude 3.5 Sonnet, Claude 3 Haiku | ≈ estimate | ✅ |
+| **Google** | Gemini Pro, Gemini Flash | ≈ estimate | ✅ |
+| **Meta** | Llama 3 8B, Llama 3 70B | ≈ estimate | — |
+| **Mistral** | Mistral Large | ≈ estimate | ✅ |
+| **Cohere** | Command R+ | ≈ estimate | ✅ |
+
+Register custom models:
+```python
+from layoutlm_forge import ModelRegistry, ModelInfo, TokenizerBackend
+
+ModelRegistry.register(ModelInfo(
+    name="my-fine-tuned-model",
+    backend=TokenizerBackend.OPENAI,
+    context_window=16_384,
+    encoding_name="cl100k_base",
+    input_cost_per_1k=0.002,
+    output_cost_per_1k=0.006,
+))
+```
+
+## CLI
+
+```bash
+# Count tokens
+layoutlm_forge count "Hello world" --model gpt-4o
+
+# Chunk a document
+layoutlm_forge chunk document.md --strategy semantic --max-tokens 500
+
+# Estimate cost
+layoutlm_forge cost document.txt --model claude-3.5-sonnet
+
+# List all models
+layoutlm_forge models
+
+# Health check
+layoutlm_forge doctor
+
+# Start API server
+layoutlm_forge serve --port 8000
+
+# Interactive demo
+layoutlm_forge demo
+```
+
+## REST API
+
+Start the server and access interactive docs at `http://localhost:8000/docs`:
+
+```bash
+pip install "layoutlm_forge[api]"
+layoutlm_forge serve
+```
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | System health + version |
+| `/api/v1/tokens/count` | POST | Count tokens |
+| `/api/v1/tokens/validate` | POST | Check context window fit |
+| `/api/v1/chunks/` | POST | Chunk text |
+| `/api/v1/context/assemble` | POST | Priority-based assembly |
+| `/api/v1/compress/` | POST | Compress text |
+| `/api/v1/cost/estimate` | POST | Estimate cost |
+
+## Architecture
+
+```
+layoutlm_forge/
+├── models.py        # Model registry (15+ models, pricing, backends)
+├── tokenizer.py     # Multi-provider token counter (tiktoken + heuristics)
+├── chunker.py       # 5-strategy document chunker with overlap
+├── context.py       # Priority-based context assembly + conversation manager
+├── compressor.py    # 4-strategy compression engine (TF-IDF, middle-out, etc.)
+├── cost.py          # Cost estimation engine with model comparison
+├── cli/main.py      # Typer CLI with Rich output
+└── api/             # FastAPI server with versioned routes
+```
+
+## Docker
+
+```bash
+docker build -t layoutlm_forge .
+docker-compose up
+```
+
+## Development
+
+```bash
+git clone https://github.com/dhruv-atomic-mui21/layoutlm_forge.git
+cd layoutlm_forge
+pip install -e ".[dev]"
+pytest
+```
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow guidelines.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

layoutlm_forge-0.1.0/README.md

@@ -0,0 +1,262 @@
+<div align="center">
+  <h1>🔥 LayoutLM Forge</h1>
+  <p><b>Production-Grade LLMOps Infrastructure for Context Window Management</b></p>
+  <p><i>Deterministic token counting · Intelligent chunking · Priority-based context assembly · Cost estimation — the foundation every AI application needs.</i></p>
+
+  [![Tests](https://github.com/dhruv-atomic-mui21/layoutlm_forge/workflows/Tests/badge.svg)](https://github.com/dhruv-atomic-mui21/layoutlm_forge/actions)
+  [![PyPI](https://img.shields.io/pypi/v/layoutlm_forge.svg)](https://pypi.org/project/layoutlm_forge/)
+  [![Python](https://img.shields.io/pypi/pyversions/layoutlm_forge.svg)](https://pypi.org/project/layoutlm_forge/)
+  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+  [![Downloads](https://img.shields.io/pypi/dm/layoutlm_forge.svg)](https://pypi.org/project/layoutlm_forge/)
+</div>
+
+---
+
+## Why LayoutLM Forge?
+
+Every production AI application eventually hits the same infrastructure problems:
+
+| Problem | Impact | LayoutLM Forge Solution |
+|---------|--------|-----------------------|
+| 🎯 Context window overflow | Silent failures, truncated responses | Priority-based assembly with overflow tracking |
+| 📊 Inaccurate token counting | Budget overruns, dropped requests | Deterministic counting via tiktoken + heuristic fallbacks |
+| 🔄 Naive text splitting | Broken semantics, degraded LLM reasoning | 5 chunking strategies (sentence, paragraph, semantic, code, fixed) |
+| 💸 Unpredictable API costs | Surprise bills, no cost governance | Pre-flight cost estimation across 15+ models |
+| 🗜️ Oversized prompts | Wasted tokens, slow responses | 4 compression strategies (extractive, truncate, middle-out, map-reduce) |
+
+## Installation
+
+```bash
+pip install layoutlm_forge
+```
+
+With API server support:
+```bash
+pip install "layoutlm_forge[api]"
+```
+
+## Quick Start
+
+### Token Counting
+
+```python
+from layoutlm_forge import TokenCounter
+
+counter = TokenCounter("gpt-4o")
+tokens = counter.count("Hello, world!")
+print(f"Tokens: {tokens}")  # Tokens: 4
+
+# Check context window fit
+fits = counter.fits_in_window("Your prompt...", reserve_output=500)
+
+# Estimate cost before sending
+cost = counter.estimate_cost("Your prompt...", direction="input")
+print(f"Cost: ${cost:.6f}")
+```
+
+### Intelligent Chunking
+
+```python
+from layoutlm_forge import DocumentChunker, ChunkStrategy
+
+chunker = DocumentChunker("gpt-4o")
+
+# Chunk respecting paragraph boundaries
+chunks = chunker.chunk(
+    long_document,
+    strategy=ChunkStrategy.PARAGRAPH,
+    max_tokens=500,
+    overlap_tokens=50,
+)
+
+# Specialized chunkers
+code_chunks = chunker.chunk_code(source_code, language="python")
+md_chunks = chunker.chunk_markdown(readme_text)
+```
+
+### Priority-Based Context Assembly
+
+The core pattern for RAG applications — guarantee critical context fits while gracefully dropping lower-priority content:
+
+```python
+from layoutlm_forge import ContextWindow, Priority
+
+window = ContextWindow("gpt-4o")
+
+# System instructions — always included
+window.add_block("You are a legal assistant.", Priority.CRITICAL, "system")
+
+# User query — high priority
+window.add_block("What is the statute of limitations?", Priority.HIGH, "query")
+
+# RAG search results — included if space permits
+window.add_block(search_result_1, Priority.MEDIUM, "rag_1")
+window.add_block(search_result_2, Priority.LOW, "rag_2")
+
+# Assemble: packs highest-priority blocks first
+prompt = window.assemble(max_tokens=4096)
+
+# See what was included/dropped
+usage = window.usage()
+print(f"Included: {usage['num_included']} blocks ({usage['included_tokens']} tokens)")
+print(f"Dropped:  {usage['num_excluded']} blocks")
+```
+
+### Cost Estimation
+
+```python
+from layoutlm_forge import CostCalculator
+
+calc = CostCalculator("gpt-4o")
+
+# Single prompt cost
+cost = calc.estimate_prompt("Your prompt text here")
+print(f"Input cost: ${cost.usd:.6f}")
+
+# Compare models
+comparison = calc.compare_models(
+    texts=["Document 1...", "Document 2..."],
+    models=["gpt-4o", "gpt-4o-mini", "claude-3.5-sonnet", "gemini-flash"],
+)
+for model, analysis in comparison.items():
+    print(f"{model}: ${analysis.total_usd:.6f} for {analysis.total_tokens} tokens")
+```
+
+### Context Compression
+
+```python
+from layoutlm_forge import ContextCompressor, CompressionStrategy
+
+compressor = ContextCompressor("gpt-4o")
+
+# Extractive: keeps most important sentences via TF-IDF scoring
+result = compressor.compress(long_text, target_tokens=200)
+print(f"Compressed: {result.original_tokens} → {result.compressed_tokens} tokens")
+print(f"Savings: {result.savings_pct:.1f}%")
+
+# Middle-out: preserves start and end, removes middle
+result = compressor.compress(log_text, target_tokens=300, strategy=CompressionStrategy.MIDDLE_OUT)
+```
+
+### Conversation Management
+
+```python
+from layoutlm_forge import ConversationManager
+
+manager = ConversationManager("gpt-4o")
+
+manager.add_message("system", "You are a helpful Python tutor.")
+manager.add_message("user", "Explain decorators")
+manager.add_message("assistant", "Decorators are...")
+# ... many more turns ...
+
+# Auto-trim older messages to fit budget, preserving system prompt
+trimmed = manager.get_context(max_tokens=4096, preserve_system=True)
+```
+
+## Supported Models
+
+| Provider | Models | Token Counting | Pricing |
+|----------|--------|:--------------:|:-------:|
+| **OpenAI** | GPT-4, GPT-4 Turbo, GPT-4o, GPT-4o-mini, GPT-3.5 Turbo | ✅ tiktoken | ✅ |
+| **Anthropic** | Claude 3 Opus, Claude 3.5 Sonnet, Claude 3 Haiku | ≈ estimate | ✅ |
+| **Google** | Gemini Pro, Gemini Flash | ≈ estimate | ✅ |
+| **Meta** | Llama 3 8B, Llama 3 70B | ≈ estimate | — |
+| **Mistral** | Mistral Large | ≈ estimate | ✅ |
+| **Cohere** | Command R+ | ≈ estimate | ✅ |
+
+Register custom models:
+```python
+from layoutlm_forge import ModelRegistry, ModelInfo, TokenizerBackend
+
+ModelRegistry.register(ModelInfo(
+    name="my-fine-tuned-model",
+    backend=TokenizerBackend.OPENAI,
+    context_window=16_384,
+    encoding_name="cl100k_base",
+    input_cost_per_1k=0.002,
+    output_cost_per_1k=0.006,
+))
+```
+
+## CLI
+
+```bash
+# Count tokens
+layoutlm_forge count "Hello world" --model gpt-4o
+
+# Chunk a document
+layoutlm_forge chunk document.md --strategy semantic --max-tokens 500
+
+# Estimate cost
+layoutlm_forge cost document.txt --model claude-3.5-sonnet
+
+# List all models
+layoutlm_forge models
+
+# Health check
+layoutlm_forge doctor
+
+# Start API server
+layoutlm_forge serve --port 8000
+
+# Interactive demo
+layoutlm_forge demo
+```
+
+## REST API
+
+Start the server and access interactive docs at `http://localhost:8000/docs`:
+
+```bash
+pip install "layoutlm_forge[api]"
+layoutlm_forge serve
+```
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | System health + version |
+| `/api/v1/tokens/count` | POST | Count tokens |
+| `/api/v1/tokens/validate` | POST | Check context window fit |
+| `/api/v1/chunks/` | POST | Chunk text |
+| `/api/v1/context/assemble` | POST | Priority-based assembly |
+| `/api/v1/compress/` | POST | Compress text |
+| `/api/v1/cost/estimate` | POST | Estimate cost |
+
+## Architecture
+
+```
+layoutlm_forge/
+├── models.py        # Model registry (15+ models, pricing, backends)
+├── tokenizer.py     # Multi-provider token counter (tiktoken + heuristics)
+├── chunker.py       # 5-strategy document chunker with overlap
+├── context.py       # Priority-based context assembly + conversation manager
+├── compressor.py    # 4-strategy compression engine (TF-IDF, middle-out, etc.)
+├── cost.py          # Cost estimation engine with model comparison
+├── cli/main.py      # Typer CLI with Rich output
+└── api/             # FastAPI server with versioned routes
+```
+
+## Docker
+
+```bash
+docker build -t layoutlm_forge .
+docker-compose up
+```
+
+## Development
+
+```bash
+git clone https://github.com/dhruv-atomic-mui21/layoutlm_forge.git
+cd layoutlm_forge
+pip install -e ".[dev]"
+pytest
+```
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow guidelines.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

layoutlm_forge-0.1.0/docs/api-reference.md

@@ -0,0 +1,58 @@
+# API Reference
+
+LayoutLM Forge includes a FastAPI application to serve these tools over HTTP.
+
+By default, the server runs on port 8000. Start it via CLI:
+
+```bash
+layoutlm_forge serve --port 8000
+```
+
+## Endpoints
+
+### `GET /health`
+Returns system health, version, and the number of supported model providers.
+
+### `POST /api/v1/tokens/count`
+Count tokens for a given string.
+
+**Request:**
+```json
+{
+  "text": "Hello world!",
+  "model": "gpt-4o"
+}
+```
+
+### `POST /api/v1/cost/estimate`
+Estimate input cost for processing text.
+
+**Request:**
+```json
+{
+  "text": "Hello world!",
+  "model": "gpt-4o"
+}
+```
+
+### `POST /api/v1/chunks/`
+Chunk text with strategies like `paragraph`, `sentence`, `fixed`, `semantic`, and `code`.
+
+### `POST /api/v1/context/assemble`
+Pass a list of priority-labelled blocks.
+
+**Request:**
+```json
+{
+  "blocks": [
+    {"content": "System prompt", "priority": "CRITICAL"},
+    {"content": "Irrelevant chat", "priority": "LOW"}
+  ],
+  "max_tokens": 100
+}
+```
+
+### `POST /api/v1/compress/`
+Compress long text into a specific target budget.
+
+**Strategies**: `extractive`, `truncate`, `middle_out`, `map_reduce`

layoutlm_forge-0.1.0/docs/architecture.md

@@ -0,0 +1,18 @@
+# Architecture
+
+LayoutLM Forge is split into modular components that can be used independently or together through the API and CLI.
+
+## Core Modules
+
+1. **`layoutlm_forge.models`**: Stores model metadata, cost schema, and backend mapping (OpenAI, Anthropic, Google, etc.).
+2. **`layoutlm_forge.tokenizer`**: Uses `tiktoken` for OpenAI models and heuristic estimations for others to rapidly count tokens.
+3. **`layoutlm_forge.chunker`**: Employs Regex and token counting to securely chunk documents based on semantics (paragraphs, markdown nodes, functions/classes).
+4. **`layoutlm_forge.context`**: A Priority queue-style greedy packer. It guarantees that `CRITICAL` or `HIGH` priority blocks fit inside the window before packing `LOW` priority blocks.
+5. **`layoutlm_forge.compressor`**: Token budget optimizer. Uses heuristics like TF-IDF or simple map-reduce text truncation to forcefully fit contexts into smaller token constraints. 
+
+## Flow
+
+1. You **Chunk** a large document.
+2. You feed the chunks into the **Context** assembler, adding priority tags (so that old data is dropped if token limit is reached).
+3. If necessary, you run **Compression** on the chunks to squeeze out more space.
+4. Before issuing the API request to the LLM, you use **Cost** to estimate the total expense of your context string.