emergent-translator 1.1.0__tar.gz → 1.1.1__tar.gz

emergent_translator-1.1.1/PKG-INFO

@@ -0,0 +1,248 @@
+Metadata-Version: 2.4
+Name: emergent-translator
+Version: 1.1.1
+Summary: 60x compression efficiency for AI communication through emergent language translation
+Author-email: Emergent Language Team <hello@emergentlanguage.ai>
+Maintainer-email: Emergent Language Team <hello@emergentlanguage.ai>
+License-Expression: GPL-3.0-or-later
+Project-URL: Homepage, https://emergentlanguage.ai
+Project-URL: GitHub, https://github.com/maco144/emergent-language
+Project-URL: Documentation, https://github.com/maco144/emergent-language/wiki
+Project-URL: Repository, https://github.com/maco144/emergent-language.git
+Project-URL: Bug Tracker, https://github.com/maco144/emergent-language/issues
+Project-URL: Changelog, https://github.com/maco144/emergent-language/releases
+Project-URL: API Docs, http://149.28.33.118:8001/docs
+Project-URL: Live Demo, http://149.28.33.118:8001
+Keywords: ai,compression,emergent-language,api,translation,efficiency,machine-learning,natural-language-processing
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Communications
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn[standard]>=0.23.0
+Requires-Dist: python-multipart>=0.0.6
+Requires-Dist: websockets>=11.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: aiofiles>=23.0.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: openai>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: isort>=5.12; extra == "dev"
+Requires-Dist: flake8>=6.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Provides-Extra: monitoring
+Requires-Dist: structlog>=23.0.0; extra == "monitoring"
+Requires-Dist: prometheus-fastapi-instrumentator>=6.0.0; extra == "monitoring"
+Provides-Extra: formats
+Requires-Dist: pyyaml>=6.0; extra == "formats"
+Requires-Dist: msgpack>=1.0; extra == "formats"
+Requires-Dist: protobuf>=4.0; extra == "formats"
+Requires-Dist: pyarrow>=12.0; extra == "formats"
+Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "formats"
+Requires-Dist: tomli_w>=1.0; extra == "formats"
+Requires-Dist: pymongo>=4.0; extra == "formats"
+Requires-Dist: cbor2>=5.0; extra == "formats"
+Requires-Dist: openpyxl>=3.1; extra == "formats"
+Provides-Extra: examples
+Requires-Dist: langchain>=0.0.300; extra == "examples"
+Requires-Dist: crewai>=0.1.0; extra == "examples"
+Requires-Dist: jupyter>=1.0.0; extra == "examples"
+Requires-Dist: matplotlib>=3.0.0; extra == "examples"
+Dynamic: license-file
+
+# Emergent Language Translator
+
+**High-performance binary encoding for AI agent communication**
+
+[![PyPI](https://img.shields.io/pypi/v/emergent-translator)](https://pypi.org/project/emergent-translator/)
+[![Python](https://img.shields.io/badge/Python-3.9+-brightgreen)](https://pypi.org/project/emergent-translator/)
+[![Tests](https://img.shields.io/badge/Tests-535%20passed-green)]()
+[![License](https://img.shields.io/badge/License-GPL--3.0-blue)](LICENSE)
+
+Emergent Language Translator compresses structured AI messages into a compact binary format using learned codebooks, common-key dictionaries, and zlib. Batch encoding amortizes header overhead across messages — the more you batch, the better the ratio.
+
+## Quick Start
+
+```bash
+pip install emergent-translator
+```
+
+```python
+from emergent_translator import BatchEncoder
+
+encoder = BatchEncoder()
+
+# Encode a batch of agent messages
+messages = [
+    {"role": "user", "content": "analyze market trends", "priority": "high"},
+    {"role": "assistant", "content": "Starting analysis", "status": "active"},
+    {"role": "system", "content": "Agent coordinator online", "version": "1.0"},
+]
+
+result = encoder.encode_batch(messages)
+print(f"{len(messages)} messages: 226 bytes JSON -> {len(result.payload)} bytes binary")
+# 3 messages: 226 bytes JSON -> 141 bytes (38% reduction)
+
+# Perfect round-trip reconstruction
+decoded = encoder.decode_batch(result.payload)
+assert decoded == messages
+```
+
+## Compression Results
+
+The batch encoder uses a binary wire format with common-key/value dictionaries and zlib compression. Efficiency improves with batch size:
+
+| Workload | JSON Size | Encoded Size | Reduction |
+|----------|-----------|-------------|-----------|
+| 3 agent messages | 226 bytes | 141 bytes | 38% |
+| 10 agent messages | 750 bytes | 112 bytes | **85%** |
+| 50 agent messages | 4,880 bytes | 286 bytes | **94%** |
+
+Encoding speed: sub-millisecond (0.2ms typical).
+
+## Binary Format
+
+All payloads start with magic bytes `\xE7\xB0` followed by a version byte:
+
+```
+v1/v2: MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + PAYLOAD + CRC32(4)
+v3:    MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + CB_VERSION(2) + CB_LEN(2) + [CODEBOOK] + PAYLOAD + CRC32(4)
+```
+
+Common keys (`role`, `content`, `action`, `status`, `priority`, ...) and values (`user`, `assistant`, `system`, `high`, `low`, ...) are encoded as single-byte tokens. Remaining data is zlib-compressed.
+
+## Adaptive Codebooks
+
+The static dictionaries cover common AI communication patterns. For domain-specific traffic, train a codebook that learns your most frequent keys and values:
+
+```python
+from emergent_translator import AdaptiveCodebook, BatchEncoder
+
+# Train on observed traffic
+codebook = AdaptiveCodebook()
+for msg in training_messages:
+    codebook.observe(msg)
+codebook.rebuild(min_freq=5)
+
+# Encode with learned codebook (v3 format, codebook embedded in payload)
+encoder = BatchEncoder()
+result = encoder.encode_batch(messages, codebook=codebook.active)
+decoded = encoder.decode_batch(result.payload)  # codebook auto-extracted
+```
+
+Train a codebook from synthetic data:
+```bash
+python scripts/train_codebook.py --messages 50000 --benchmark
+```
+
+## Multi-Format Support
+
+Parse and serialize 13+ formats, then compress through the batch encoder:
+
+```python
+from emergent_translator import detect_format, get_handler, BatchEncoder
+
+fmt = detect_format("data.csv")          # "csv"
+parse_fn, serialize_fn = get_handler(fmt)
+records = parse_fn(open("data.csv").read())
+
+encoder = BatchEncoder()
+result = encoder.encode_batch(records)
+```
+
+Supported: JSON, CSV, JSONL, YAML, TOML, INI, XML, MessagePack, Protobuf, Parquet, Arrow, BSON, CBOR, XLSX.
+
+## GPU Batch Encoder
+
+For higher throughput, use the GPU-accelerated encoder (falls back to CPU with ThreadPoolExecutor if CuPy is unavailable):
+
+```python
+from emergent_translator import GPUBatchEncoder
+
+gpu_encoder = GPUBatchEncoder(num_workers=8)
+result = gpu_encoder.encode_batch(messages)
+decoded = gpu_encoder.decode_batch(result.payload)
+```
+
+## LLM Token Savings
+
+Two complementary modules for reducing token usage with LLMs like Claude:
+
+### Code Skeletonization
+
+Strip Python files to signatures + docstrings. Feed Claude the *structure* without paying for implementation lines:
+
+```python
+from emergent_translator import skeletonize_file
+
+result = skeletonize_file("my_module.py", focal=["important_func"])
+print(f"{result.original_tokens} -> {result.skeleton_tokens} tokens "
+      f"({result.token_reduction_pct:.0f}% reduction)")
+```
+
+### Claude Text Compression
+
+Compress keys and values in structured data flowing through Claude API conversations:
+
+```python
+from emergent_translator import ClaudeCompressor
+
+compressor = ClaudeCompressor()
+system = compressor.system_prompt_prefix() + "\n\nYour prompt..."
+compressed_msgs = compressor.compress_messages(messages)
+```
+
+## Project Structure
+
+```
+src/emergent_translator/    # pip-installable package
+  batch_encoder.py          # v1 batch encoder (encode/decode)
+  gpu_batch_encoder.py      # v2 GPU-accelerated encoder
+  adaptive_codebook.py      # v3 learned codebooks
+  format_handlers.py        # 13+ format parsers
+  emergent_symbols.py       # symbol encoder
+  api_server.py             # FastAPI server
+  cli.py                    # CLI tool
+scripts/                    # benchmarks, stress tests, workers
+tests/                      # 535 tests
+```
+
+## Development
+
+```bash
+git clone https://github.com/maco144/emergent-language
+cd emergent-language
+pip install -e ".[dev,formats]"
+python -m pytest tests/ -v
+```
+
+## Docker
+
+```bash
+docker build -t emergent-translator .
+docker run -p 8000:8000 emergent-translator
+```
+
+## License
+
+GPL-3.0-or-later. See [LICENSE](LICENSE) for details.
+
+Commercial licensing available — see [COMMERCIAL_LICENSE.md](COMMERCIAL_LICENSE.md).

emergent_translator-1.1.1/README.md

@@ -0,0 +1,178 @@
+# Emergent Language Translator
+
+**High-performance binary encoding for AI agent communication**
+
+[![PyPI](https://img.shields.io/pypi/v/emergent-translator)](https://pypi.org/project/emergent-translator/)
+[![Python](https://img.shields.io/badge/Python-3.9+-brightgreen)](https://pypi.org/project/emergent-translator/)
+[![Tests](https://img.shields.io/badge/Tests-535%20passed-green)]()
+[![License](https://img.shields.io/badge/License-GPL--3.0-blue)](LICENSE)
+
+Emergent Language Translator compresses structured AI messages into a compact binary format using learned codebooks, common-key dictionaries, and zlib. Batch encoding amortizes header overhead across messages — the more you batch, the better the ratio.
+
+## Quick Start
+
+```bash
+pip install emergent-translator
+```
+
+```python
+from emergent_translator import BatchEncoder
+
+encoder = BatchEncoder()
+
+# Encode a batch of agent messages
+messages = [
+    {"role": "user", "content": "analyze market trends", "priority": "high"},
+    {"role": "assistant", "content": "Starting analysis", "status": "active"},
+    {"role": "system", "content": "Agent coordinator online", "version": "1.0"},
+]
+
+result = encoder.encode_batch(messages)
+print(f"{len(messages)} messages: 226 bytes JSON -> {len(result.payload)} bytes binary")
+# 3 messages: 226 bytes JSON -> 141 bytes (38% reduction)
+
+# Perfect round-trip reconstruction
+decoded = encoder.decode_batch(result.payload)
+assert decoded == messages
+```
+
+## Compression Results
+
+The batch encoder uses a binary wire format with common-key/value dictionaries and zlib compression. Efficiency improves with batch size:
+
+| Workload | JSON Size | Encoded Size | Reduction |
+|----------|-----------|-------------|-----------|
+| 3 agent messages | 226 bytes | 141 bytes | 38% |
+| 10 agent messages | 750 bytes | 112 bytes | **85%** |
+| 50 agent messages | 4,880 bytes | 286 bytes | **94%** |
+
+Encoding speed: sub-millisecond (0.2ms typical).
+
+## Binary Format
+
+All payloads start with magic bytes `\xE7\xB0` followed by a version byte:
+
+```
+v1/v2: MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + PAYLOAD + CRC32(4)
+v3:    MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + CB_VERSION(2) + CB_LEN(2) + [CODEBOOK] + PAYLOAD + CRC32(4)
+```
+
+Common keys (`role`, `content`, `action`, `status`, `priority`, ...) and values (`user`, `assistant`, `system`, `high`, `low`, ...) are encoded as single-byte tokens. Remaining data is zlib-compressed.
+
+## Adaptive Codebooks
+
+The static dictionaries cover common AI communication patterns. For domain-specific traffic, train a codebook that learns your most frequent keys and values:
+
+```python
+from emergent_translator import AdaptiveCodebook, BatchEncoder
+
+# Train on observed traffic
+codebook = AdaptiveCodebook()
+for msg in training_messages:
+    codebook.observe(msg)
+codebook.rebuild(min_freq=5)
+
+# Encode with learned codebook (v3 format, codebook embedded in payload)
+encoder = BatchEncoder()
+result = encoder.encode_batch(messages, codebook=codebook.active)
+decoded = encoder.decode_batch(result.payload)  # codebook auto-extracted
+```
+
+Train a codebook from synthetic data:
+```bash
+python scripts/train_codebook.py --messages 50000 --benchmark
+```
+
+## Multi-Format Support
+
+Parse and serialize 13+ formats, then compress through the batch encoder:
+
+```python
+from emergent_translator import detect_format, get_handler, BatchEncoder
+
+fmt = detect_format("data.csv")          # "csv"
+parse_fn, serialize_fn = get_handler(fmt)
+records = parse_fn(open("data.csv").read())
+
+encoder = BatchEncoder()
+result = encoder.encode_batch(records)
+```
+
+Supported: JSON, CSV, JSONL, YAML, TOML, INI, XML, MessagePack, Protobuf, Parquet, Arrow, BSON, CBOR, XLSX.
+
+## GPU Batch Encoder
+
+For higher throughput, use the GPU-accelerated encoder (falls back to CPU with ThreadPoolExecutor if CuPy is unavailable):
+
+```python
+from emergent_translator import GPUBatchEncoder
+
+gpu_encoder = GPUBatchEncoder(num_workers=8)
+result = gpu_encoder.encode_batch(messages)
+decoded = gpu_encoder.decode_batch(result.payload)
+```
+
+## LLM Token Savings
+
+Two complementary modules for reducing token usage with LLMs like Claude:
+
+### Code Skeletonization
+
+Strip Python files to signatures + docstrings. Feed Claude the *structure* without paying for implementation lines:
+
+```python
+from emergent_translator import skeletonize_file
+
+result = skeletonize_file("my_module.py", focal=["important_func"])
+print(f"{result.original_tokens} -> {result.skeleton_tokens} tokens "
+      f"({result.token_reduction_pct:.0f}% reduction)")
+```
+
+### Claude Text Compression
+
+Compress keys and values in structured data flowing through Claude API conversations:
+
+```python
+from emergent_translator import ClaudeCompressor
+
+compressor = ClaudeCompressor()
+system = compressor.system_prompt_prefix() + "\n\nYour prompt..."
+compressed_msgs = compressor.compress_messages(messages)
+```
+
+## Project Structure
+
+```
+src/emergent_translator/    # pip-installable package
+  batch_encoder.py          # v1 batch encoder (encode/decode)
+  gpu_batch_encoder.py      # v2 GPU-accelerated encoder
+  adaptive_codebook.py      # v3 learned codebooks
+  format_handlers.py        # 13+ format parsers
+  emergent_symbols.py       # symbol encoder
+  api_server.py             # FastAPI server
+  cli.py                    # CLI tool
+scripts/                    # benchmarks, stress tests, workers
+tests/                      # 535 tests
+```
+
+## Development
+
+```bash
+git clone https://github.com/maco144/emergent-language
+cd emergent-language
+pip install -e ".[dev,formats]"
+python -m pytest tests/ -v
+```
+
+## Docker
+
+```bash
+docker build -t emergent-translator .
+docker run -p 8000:8000 emergent-translator
+```
+
+## License
+
+GPL-3.0-or-later. See [LICENSE](LICENSE) for details.
+
+Commercial licensing available — see [COMMERCIAL_LICENSE.md](COMMERCIAL_LICENSE.md).

{emergent_translator-1.1.0 → emergent_translator-1.1.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "emergent-translator"
-version = "1.1.0"
+version = "1.1.1"
 description = "60x compression efficiency for AI communication through emergent language translation"
 readme = "README.md"
 license = "GPL-3.0-or-later"

{emergent_translator-1.1.0 → emergent_translator-1.1.1}/src/emergent_translator/__init__.py

@@ -32,7 +32,7 @@ from .format_handlers import (
 
 from .emergent_symbols import EmergentSymbolEncoder
 
-__version__ = "1.1.0"
+__version__ = "1.1.1"
 __author__ = "Emergent Language Team"
 __email__ = "hello@emergentlanguage.ai"
 __description__ = "60x compression efficiency for AI communication"

emergent_translator-1.1.1/src/emergent_translator.egg-info/PKG-INFO

@@ -0,0 +1,248 @@
+Metadata-Version: 2.4
+Name: emergent-translator
+Version: 1.1.1
+Summary: 60x compression efficiency for AI communication through emergent language translation
+Author-email: Emergent Language Team <hello@emergentlanguage.ai>
+Maintainer-email: Emergent Language Team <hello@emergentlanguage.ai>
+License-Expression: GPL-3.0-or-later
+Project-URL: Homepage, https://emergentlanguage.ai
+Project-URL: GitHub, https://github.com/maco144/emergent-language
+Project-URL: Documentation, https://github.com/maco144/emergent-language/wiki
+Project-URL: Repository, https://github.com/maco144/emergent-language.git
+Project-URL: Bug Tracker, https://github.com/maco144/emergent-language/issues
+Project-URL: Changelog, https://github.com/maco144/emergent-language/releases
+Project-URL: API Docs, http://149.28.33.118:8001/docs
+Project-URL: Live Demo, http://149.28.33.118:8001
+Keywords: ai,compression,emergent-language,api,translation,efficiency,machine-learning,natural-language-processing
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Communications
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn[standard]>=0.23.0
+Requires-Dist: python-multipart>=0.0.6
+Requires-Dist: websockets>=11.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: aiofiles>=23.0.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: openai>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: isort>=5.12; extra == "dev"
+Requires-Dist: flake8>=6.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Provides-Extra: monitoring
+Requires-Dist: structlog>=23.0.0; extra == "monitoring"
+Requires-Dist: prometheus-fastapi-instrumentator>=6.0.0; extra == "monitoring"
+Provides-Extra: formats
+Requires-Dist: pyyaml>=6.0; extra == "formats"
+Requires-Dist: msgpack>=1.0; extra == "formats"
+Requires-Dist: protobuf>=4.0; extra == "formats"
+Requires-Dist: pyarrow>=12.0; extra == "formats"
+Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "formats"
+Requires-Dist: tomli_w>=1.0; extra == "formats"
+Requires-Dist: pymongo>=4.0; extra == "formats"
+Requires-Dist: cbor2>=5.0; extra == "formats"
+Requires-Dist: openpyxl>=3.1; extra == "formats"
+Provides-Extra: examples
+Requires-Dist: langchain>=0.0.300; extra == "examples"
+Requires-Dist: crewai>=0.1.0; extra == "examples"
+Requires-Dist: jupyter>=1.0.0; extra == "examples"
+Requires-Dist: matplotlib>=3.0.0; extra == "examples"
+Dynamic: license-file
+
+# Emergent Language Translator
+
+**High-performance binary encoding for AI agent communication**
+
+[![PyPI](https://img.shields.io/pypi/v/emergent-translator)](https://pypi.org/project/emergent-translator/)
+[![Python](https://img.shields.io/badge/Python-3.9+-brightgreen)](https://pypi.org/project/emergent-translator/)
+[![Tests](https://img.shields.io/badge/Tests-535%20passed-green)]()
+[![License](https://img.shields.io/badge/License-GPL--3.0-blue)](LICENSE)
+
+Emergent Language Translator compresses structured AI messages into a compact binary format using learned codebooks, common-key dictionaries, and zlib. Batch encoding amortizes header overhead across messages — the more you batch, the better the ratio.
+
+## Quick Start
+
+```bash
+pip install emergent-translator
+```
+
+```python
+from emergent_translator import BatchEncoder
+
+encoder = BatchEncoder()
+
+# Encode a batch of agent messages
+messages = [
+    {"role": "user", "content": "analyze market trends", "priority": "high"},
+    {"role": "assistant", "content": "Starting analysis", "status": "active"},
+    {"role": "system", "content": "Agent coordinator online", "version": "1.0"},
+]
+
+result = encoder.encode_batch(messages)
+print(f"{len(messages)} messages: 226 bytes JSON -> {len(result.payload)} bytes binary")
+# 3 messages: 226 bytes JSON -> 141 bytes (38% reduction)
+
+# Perfect round-trip reconstruction
+decoded = encoder.decode_batch(result.payload)
+assert decoded == messages
+```
+
+## Compression Results
+
+The batch encoder uses a binary wire format with common-key/value dictionaries and zlib compression. Efficiency improves with batch size:
+
+| Workload | JSON Size | Encoded Size | Reduction |
+|----------|-----------|-------------|-----------|
+| 3 agent messages | 226 bytes | 141 bytes | 38% |
+| 10 agent messages | 750 bytes | 112 bytes | **85%** |
+| 50 agent messages | 4,880 bytes | 286 bytes | **94%** |
+
+Encoding speed: sub-millisecond (0.2ms typical).
+
+## Binary Format
+
+All payloads start with magic bytes `\xE7\xB0` followed by a version byte:
+
+```
+v1/v2: MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + PAYLOAD + CRC32(4)
+v3:    MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + CB_VERSION(2) + CB_LEN(2) + [CODEBOOK] + PAYLOAD + CRC32(4)
+```
+
+Common keys (`role`, `content`, `action`, `status`, `priority`, ...) and values (`user`, `assistant`, `system`, `high`, `low`, ...) are encoded as single-byte tokens. Remaining data is zlib-compressed.
+
+## Adaptive Codebooks
+
+The static dictionaries cover common AI communication patterns. For domain-specific traffic, train a codebook that learns your most frequent keys and values:
+
+```python
+from emergent_translator import AdaptiveCodebook, BatchEncoder
+
+# Train on observed traffic
+codebook = AdaptiveCodebook()
+for msg in training_messages:
+    codebook.observe(msg)
+codebook.rebuild(min_freq=5)
+
+# Encode with learned codebook (v3 format, codebook embedded in payload)
+encoder = BatchEncoder()
+result = encoder.encode_batch(messages, codebook=codebook.active)
+decoded = encoder.decode_batch(result.payload)  # codebook auto-extracted
+```
+
+Train a codebook from synthetic data:
+```bash
+python scripts/train_codebook.py --messages 50000 --benchmark
+```
+
+## Multi-Format Support
+
+Parse and serialize 13+ formats, then compress through the batch encoder:
+
+```python
+from emergent_translator import detect_format, get_handler, BatchEncoder
+
+fmt = detect_format("data.csv")          # "csv"
+parse_fn, serialize_fn = get_handler(fmt)
+records = parse_fn(open("data.csv").read())
+
+encoder = BatchEncoder()
+result = encoder.encode_batch(records)
+```
+
+Supported: JSON, CSV, JSONL, YAML, TOML, INI, XML, MessagePack, Protobuf, Parquet, Arrow, BSON, CBOR, XLSX.
+
+## GPU Batch Encoder
+
+For higher throughput, use the GPU-accelerated encoder (falls back to CPU with ThreadPoolExecutor if CuPy is unavailable):
+
+```python
+from emergent_translator import GPUBatchEncoder
+
+gpu_encoder = GPUBatchEncoder(num_workers=8)
+result = gpu_encoder.encode_batch(messages)
+decoded = gpu_encoder.decode_batch(result.payload)
+```
+
+## LLM Token Savings
+
+Two complementary modules for reducing token usage with LLMs like Claude:
+
+### Code Skeletonization
+
+Strip Python files to signatures + docstrings. Feed Claude the *structure* without paying for implementation lines:
+
+```python
+from emergent_translator import skeletonize_file
+
+result = skeletonize_file("my_module.py", focal=["important_func"])
+print(f"{result.original_tokens} -> {result.skeleton_tokens} tokens "
+      f"({result.token_reduction_pct:.0f}% reduction)")
+```
+
+### Claude Text Compression
+
+Compress keys and values in structured data flowing through Claude API conversations:
+
+```python
+from emergent_translator import ClaudeCompressor
+
+compressor = ClaudeCompressor()
+system = compressor.system_prompt_prefix() + "\n\nYour prompt..."
+compressed_msgs = compressor.compress_messages(messages)
+```
+
+## Project Structure
+
+```
+src/emergent_translator/    # pip-installable package
+  batch_encoder.py          # v1 batch encoder (encode/decode)
+  gpu_batch_encoder.py      # v2 GPU-accelerated encoder
+  adaptive_codebook.py      # v3 learned codebooks
+  format_handlers.py        # 13+ format parsers
+  emergent_symbols.py       # symbol encoder
+  api_server.py             # FastAPI server
+  cli.py                    # CLI tool
+scripts/                    # benchmarks, stress tests, workers
+tests/                      # 535 tests
+```
+
+## Development
+
+```bash
+git clone https://github.com/maco144/emergent-language
+cd emergent-language
+pip install -e ".[dev,formats]"
+python -m pytest tests/ -v
+```
+
+## Docker
+
+```bash
+docker build -t emergent-translator .
+docker run -p 8000:8000 emergent-translator
+```
+
+## License
+
+GPL-3.0-or-later. See [LICENSE](LICENSE) for details.
+
+Commercial licensing available — see [COMMERCIAL_LICENSE.md](COMMERCIAL_LICENSE.md).