eole 0.5.0__tar.gz

eole-0.5.0/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024-Present EOLE
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

eole-0.5.0/PKG-INFO

@@ -0,0 +1,227 @@
+Metadata-Version: 2.1
+Name: eole
+Version: 0.5.0
+Summary: Open language modeling toolkit based on PyTorch
+Project-URL: Source, https://github.com/eole-nlp/eole/
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: configargparse
+Requires-Dist: ctranslate2<5,>=4
+Requires-Dist: fastapi
+Requires-Dist: fasttext-wheel
+Requires-Dist: huggingface_hub
+Requires-Dist: datasets
+Requires-Dist: numpy>=2.0
+Requires-Dist: pandas
+Requires-Dist: protobuf==3.20.1
+Requires-Dist: pyahocorasick
+Requires-Dist: pyonmttok<2,>=1.38.1
+Requires-Dist: pyyaml
+Requires-Dist: rapidfuzz
+Requires-Dist: rich
+Requires-Dist: sacrebleu
+Requires-Dist: safetensors
+Requires-Dist: sentencepiece<0.1.98,>=0.1.94
+Requires-Dist: six
+Requires-Dist: spacy
+Requires-Dist: subword-nmt>=0.3.7
+Requires-Dist: tensorboard>=2.18.0
+Requires-Dist: torch<2.11,>=2.8
+Requires-Dist: torch-optimi
+Requires-Dist: uvicorn
+Requires-Dist: waitress
+Requires-Dist: pydantic
+
+# EOLE
+
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://eole-nlp.github.io/eole)
+
+Open language modeling toolkit based on [PyTorch](https://pytorch.org) initially spun-off of OpenNMT-py
+
+
+- New !!!! - Added Torch compile and Cudagraphs - as fast as vLLM / faster than CT2 on GPU. see [results](https://github.com/eole-nlp/eole/blob/main/benchmarks/genai/README.md)
+
+Just reproduce with your own hardware:
+```
+git clone https://github.com/eole-nlp/eole
+cd eole
+pip install -e .
+cd benchmarks/genai
+EOLE_TORCH_COMPILE="1" EOLE_COMPILE_MODE="0" python generate-eole.py
+```
+First run will take 60-80 seconds to compile
+Run it a second time and see the blast.
+
+
+- New January 2026: Almost full refactor of the code: Encoders, Decoders, Adapters, Model classes, Trainer, Distributed training / Inference.
+
+We aim to maintain the **research-friendly** approach of the original project while including latest architectures (LLMs) and various other techniques.
+Our goal is to provide a comprehensive yet compact and modular codebase for experimenting with various types of language models (encoder, decoder, seq2seq).
+
+## HF Models supported
+
+- **tencent/HunyuanOCR** End-to-End OCR model by Tencent. Uses more image token vs Deepseek but smaller LM. Results are impressive. (see [recipe](https://github.com/eole-nlp/eole/tree/main/recipes/hunyuanocr))
+- **deepseek-ai/DeepSeek-OCR** For now takes any image and rescales to 1024x1024 before processing - Gundam mode not implemented yet) - pdf_ocr to mmd replicated - check recipes
+- **tencent/Hunyuan-MT-7B** SOTA NMT at WMT25, better than Towerplus-9B and EuroLLM-9B
+- **Qwen/Qwen2/3** Non VL family. Includes Qwen3-30B-A3B
+- **google/gemma-3-27b-it** All Gemma3 family - supports text and image input
+- **Mistral-3.1-24B-instruct** supports all Mistral AI models (text and image input) - includes Ministral 3, Mixtral, Mathstral
+- **meta-llama/Llama-3.X** models
+- **microsoft/Phi-2/3** models
+
+Of course you can train your own architecture (Decoder only, Encoder Only, or EncoderDecoder Model)
+
+## Latest developments
+
+- **high inference speed** using Flash Attention (decoding with in-place KVCache), Vllm RMSNorm kernel, fused MLP Gate / activation, fused KVQ Linear.
+- **prefixLM + split prompt/answer in src/tgt** optional method to feed your data
+- **Pure-BF16 Training** thanks to [Kahan Summation](https://arxiv.org/pdf/2010.06192) implemented [here](https://optimi.benjaminwarner.dev/kahan_summation/)
+- **Web-based (Google translator-like) interface** featuring the latest Hunyuan-MT-7B or EuroLLM-8B-Instruct LLM
+- **Estimator layer** which enables to rescore multiple beams in the same model. Read article [here](https://medium.com/p/05b00b271a47) and [here](https://medium.com/p/7dccfe167814)
+- **Support Hugging Face Tokenizers** for better compatiblity
+- **Replicate CometKiwi(XL/XXL)** Encoder+Estimator models
+
+---
+
+## Key Features
+
+- **Versatile Training and Inference**: Train from scratch, finetune, and infer models of various architectures including Transformer Encoder/Decoder/EncoderDecoder and RNN EncoderDecoder.
+- **Dynamic Data Transforms**: Apply on-the-fly transformations in the dataloading logic for both training and inference.
+- **Comprehensive LLM Support**: Includes converters for Llama, Mistral, Phi, Gemma ...
+- **Advanced Quantization**: Support for 8-bit and 4-bit quantization, along with LoRA adapters, with or without checkpointing, as well as mixed precision (FP16).
+- **Efficient Finetuning**: Finetune 7B and 13B models on a single RTX 24GB GPU using 4-bit quantization.
+- **Flexible Inference**: Perform inference in 4-bit or 8-bit using the same layer quantization methods as in finetuning.
+- **Tensor Parallelism**: Enable tensor parallelism for both training and inference when models exceed the memory capacity of a single GPU.
+
+## Work completed
+
+We have made significant progress in several areas:
+
+- **Configuration Management**: Streamlined through [pydantic](https://docs.pydantic.dev) models.
+- **Command Line Entry Points**: Improved using structured subparsers for better organization.
+- **Reproducible Recipes**: Provided for widely used models and tasks, ensuring consistency and reliability.
+- **Core API Simplification**: Refined around the new configuration objects for ease of use.
+- **Revamped Fast API based server**: see above example with EuroLLM-9B-Instruct
+
+---
+
+### Future Directions
+
+There are still several exciting avenues to explore:
+
+- **Documentation**: Enhance and expand the documentation for better user guidance.
+- **Test Coverage**: Improve testing to ensure code reliability and performance.
+- **Logging Enhancements**: Implement more sophisticated logging mechanisms.
+- **Broader Model Support**: Extend support to include a wider range of open models, potentially multi-modal.
+
+## Setup
+
+### Using Docker
+
+To facilitate setup and reproducibility, we provide Docker images via the GitHub Container Registry: [EOLE Docker Images](https://github.com/eole-nlp/eole/pkgs/container/eole).
+
+You can customize the workflow and build your own images based on specific needs using `build.sh` and `Dockerfile` in the `docker` directory of the repository.
+
+
+To pull the Docker image:
+```bash
+docker pull ghcr.io/eole-nlp/eole:0.4.0-torch2.9.1-ubuntu22.04-cuda12.8
+```
+
+Example one-liner to run a container and open a bash shell within it:
+```bash
+docker run --rm -it --runtime=nvidia ghcr.io/eole-nlp/eole:0.4.0-torch2.9.1-ubuntu22.04-cuda12.8
+```
+
+> **Note**: Ensure you have the [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) (formerly nvidia-docker) installed to take advantage of CUDA/GPU features.
+
+Depending on your needs, you can add various flags:
+- `-p 5000:5000`: Forward an exposed port from your container to your host.
+- `-v /some/local/directory:/some/container/directory`: Mount a local directory to a container directory.
+- `--entrypoint some_command`: Run a specific command as the container entry point (instead of the default bash shell).
+
+### Installing Locally
+
+#### Requirements
+
+- Python >= 3.10
+- PyTorch >= 2.8 < 2.10
+
+#### Installation from Source
+
+To install from source:
+```bash
+git clone https://github.com/eole-nlp/eole
+cd eole
+pip install -e .
+```
+
+#### Installation from PyPI
+
+Installation from PyPI will be available soon.
+
+#### Notes
+
+If you encounter a `MemoryError` during installation, try using `pip` with the `--no-cache-dir` option.
+
+(Optional) Some advanced features (e.g., pretrained models or specific transforms) require extra packages. Install them with:
+```bash
+pip install -r requirements.opt.txt
+```
+
+### Manual Installation of Some Dependencies
+
+#### Flash Attention
+
+To use [Flash Attention](https://github.com/Dao-AILab/flash-attention#installation-and-features), install it manually:
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+#### AWQ
+
+For inference or quantizing an AWQ model, AutoAWQ is required. Install it with:
+```bash
+pip install autoawq
+```
+
+For more details, refer to [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
+
+
+## Notes on Mixed-precision or Low precision Training
+
+Until Feb 25, we used torch optimizers with or without AMP (mixed precision) or "fusedadam" which was an old implementation of Apex/Nvidia using FP16 with dynamic loss scaling and without FP32 master weights.
+As of 0.2 "fusedadam" is deprecated and we implemented pure-BF16 training.
+
+As a result, config flags are now:
+
+For FP16-amp or BF16-amp training (using pytorch optimizers and amp implementation)
+```
+compute_dtype: fp16 or bf16
+use_amp: true
+optim: adam or adamw
+```
+Special note: even though it may not be logical, we still use the torch GradScaler in BF16-AMP. Even if the BF16 range is similar to FP32, scaling prevents from underflowing.
+We tested BF16-AMP without the GradScaler and it does not give good results.
+
+
+For pure-bf16 training (using torch-optimi and kahan summation)
+```
+compute_dtype: bf16
+use_amp: false
+optim: adam or adamw
+```
+Pure-BF16 training is faster than AMP and the memory footprint is reduced (master weights are kept in BF16 vs FP32). However Kahan Summation is not magical, results are good but not as good as AMP.
+Use this feature mainly when memory footprint is an issue with LLMs.
+
+
+---
+
+## Contributing
+
+We love contributions! Please look at issues marked with the [contributions welcome](https://github.com/eole-nlp/eole/issues?q=is%3Aissue+is%3Aopen+label%3A%22contributions+welcome%22) tag.
+
+Before raising an issue, make sure you read the requirements and the [Full Documentation](https://eole-nlp.github.io/eole). You can also check if a [Recipe](https://github.com/eole-nlp/eole/tree/main/recipes) fits your use case.
+
+Unless there is a bug, please use the [Discussions](https://github.com/eole-nlp/eole/discussions) tab to ask questions or propose new topics/features.

eole-0.5.0/README.md

@@ -0,0 +1,192 @@
+# EOLE
+
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://eole-nlp.github.io/eole)
+
+Open language modeling toolkit based on [PyTorch](https://pytorch.org) initially spun-off of OpenNMT-py
+
+
+- New !!!! - Added Torch compile and Cudagraphs - as fast as vLLM / faster than CT2 on GPU. see [results](https://github.com/eole-nlp/eole/blob/main/benchmarks/genai/README.md)
+
+Just reproduce with your own hardware:
+```
+git clone https://github.com/eole-nlp/eole
+cd eole
+pip install -e .
+cd benchmarks/genai
+EOLE_TORCH_COMPILE="1" EOLE_COMPILE_MODE="0" python generate-eole.py
+```
+First run will take 60-80 seconds to compile
+Run it a second time and see the blast.
+
+
+- New January 2026: Almost full refactor of the code: Encoders, Decoders, Adapters, Model classes, Trainer, Distributed training / Inference.
+
+We aim to maintain the **research-friendly** approach of the original project while including latest architectures (LLMs) and various other techniques.
+Our goal is to provide a comprehensive yet compact and modular codebase for experimenting with various types of language models (encoder, decoder, seq2seq).
+
+## HF Models supported
+
+- **tencent/HunyuanOCR** End-to-End OCR model by Tencent. Uses more image token vs Deepseek but smaller LM. Results are impressive. (see [recipe](https://github.com/eole-nlp/eole/tree/main/recipes/hunyuanocr))
+- **deepseek-ai/DeepSeek-OCR** For now takes any image and rescales to 1024x1024 before processing - Gundam mode not implemented yet) - pdf_ocr to mmd replicated - check recipes
+- **tencent/Hunyuan-MT-7B** SOTA NMT at WMT25, better than Towerplus-9B and EuroLLM-9B
+- **Qwen/Qwen2/3** Non VL family. Includes Qwen3-30B-A3B
+- **google/gemma-3-27b-it** All Gemma3 family - supports text and image input
+- **Mistral-3.1-24B-instruct** supports all Mistral AI models (text and image input) - includes Ministral 3, Mixtral, Mathstral
+- **meta-llama/Llama-3.X** models
+- **microsoft/Phi-2/3** models
+
+Of course you can train your own architecture (Decoder only, Encoder Only, or EncoderDecoder Model)
+
+## Latest developments
+
+- **high inference speed** using Flash Attention (decoding with in-place KVCache), Vllm RMSNorm kernel, fused MLP Gate / activation, fused KVQ Linear.
+- **prefixLM + split prompt/answer in src/tgt** optional method to feed your data
+- **Pure-BF16 Training** thanks to [Kahan Summation](https://arxiv.org/pdf/2010.06192) implemented [here](https://optimi.benjaminwarner.dev/kahan_summation/)
+- **Web-based (Google translator-like) interface** featuring the latest Hunyuan-MT-7B or EuroLLM-8B-Instruct LLM
+- **Estimator layer** which enables to rescore multiple beams in the same model. Read article [here](https://medium.com/p/05b00b271a47) and [here](https://medium.com/p/7dccfe167814)
+- **Support Hugging Face Tokenizers** for better compatiblity
+- **Replicate CometKiwi(XL/XXL)** Encoder+Estimator models
+
+---
+
+## Key Features
+
+- **Versatile Training and Inference**: Train from scratch, finetune, and infer models of various architectures including Transformer Encoder/Decoder/EncoderDecoder and RNN EncoderDecoder.
+- **Dynamic Data Transforms**: Apply on-the-fly transformations in the dataloading logic for both training and inference.
+- **Comprehensive LLM Support**: Includes converters for Llama, Mistral, Phi, Gemma ...
+- **Advanced Quantization**: Support for 8-bit and 4-bit quantization, along with LoRA adapters, with or without checkpointing, as well as mixed precision (FP16).
+- **Efficient Finetuning**: Finetune 7B and 13B models on a single RTX 24GB GPU using 4-bit quantization.
+- **Flexible Inference**: Perform inference in 4-bit or 8-bit using the same layer quantization methods as in finetuning.
+- **Tensor Parallelism**: Enable tensor parallelism for both training and inference when models exceed the memory capacity of a single GPU.
+
+## Work completed
+
+We have made significant progress in several areas:
+
+- **Configuration Management**: Streamlined through [pydantic](https://docs.pydantic.dev) models.
+- **Command Line Entry Points**: Improved using structured subparsers for better organization.
+- **Reproducible Recipes**: Provided for widely used models and tasks, ensuring consistency and reliability.
+- **Core API Simplification**: Refined around the new configuration objects for ease of use.
+- **Revamped Fast API based server**: see above example with EuroLLM-9B-Instruct
+
+---
+
+### Future Directions
+
+There are still several exciting avenues to explore:
+
+- **Documentation**: Enhance and expand the documentation for better user guidance.
+- **Test Coverage**: Improve testing to ensure code reliability and performance.
+- **Logging Enhancements**: Implement more sophisticated logging mechanisms.
+- **Broader Model Support**: Extend support to include a wider range of open models, potentially multi-modal.
+
+## Setup
+
+### Using Docker
+
+To facilitate setup and reproducibility, we provide Docker images via the GitHub Container Registry: [EOLE Docker Images](https://github.com/eole-nlp/eole/pkgs/container/eole).
+
+You can customize the workflow and build your own images based on specific needs using `build.sh` and `Dockerfile` in the `docker` directory of the repository.
+
+
+To pull the Docker image:
+```bash
+docker pull ghcr.io/eole-nlp/eole:0.4.0-torch2.9.1-ubuntu22.04-cuda12.8
+```
+
+Example one-liner to run a container and open a bash shell within it:
+```bash
+docker run --rm -it --runtime=nvidia ghcr.io/eole-nlp/eole:0.4.0-torch2.9.1-ubuntu22.04-cuda12.8
+```
+
+> **Note**: Ensure you have the [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) (formerly nvidia-docker) installed to take advantage of CUDA/GPU features.
+
+Depending on your needs, you can add various flags:
+- `-p 5000:5000`: Forward an exposed port from your container to your host.
+- `-v /some/local/directory:/some/container/directory`: Mount a local directory to a container directory.
+- `--entrypoint some_command`: Run a specific command as the container entry point (instead of the default bash shell).
+
+### Installing Locally
+
+#### Requirements
+
+- Python >= 3.10
+- PyTorch >= 2.8 < 2.10
+
+#### Installation from Source
+
+To install from source:
+```bash
+git clone https://github.com/eole-nlp/eole
+cd eole
+pip install -e .
+```
+
+#### Installation from PyPI
+
+Installation from PyPI will be available soon.
+
+#### Notes
+
+If you encounter a `MemoryError` during installation, try using `pip` with the `--no-cache-dir` option.
+
+(Optional) Some advanced features (e.g., pretrained models or specific transforms) require extra packages. Install them with:
+```bash
+pip install -r requirements.opt.txt
+```
+
+### Manual Installation of Some Dependencies
+
+#### Flash Attention
+
+To use [Flash Attention](https://github.com/Dao-AILab/flash-attention#installation-and-features), install it manually:
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+#### AWQ
+
+For inference or quantizing an AWQ model, AutoAWQ is required. Install it with:
+```bash
+pip install autoawq
+```
+
+For more details, refer to [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
+
+
+## Notes on Mixed-precision or Low precision Training
+
+Until Feb 25, we used torch optimizers with or without AMP (mixed precision) or "fusedadam" which was an old implementation of Apex/Nvidia using FP16 with dynamic loss scaling and without FP32 master weights.
+As of 0.2 "fusedadam" is deprecated and we implemented pure-BF16 training.
+
+As a result, config flags are now:
+
+For FP16-amp or BF16-amp training (using pytorch optimizers and amp implementation)
+```
+compute_dtype: fp16 or bf16
+use_amp: true
+optim: adam or adamw
+```
+Special note: even though it may not be logical, we still use the torch GradScaler in BF16-AMP. Even if the BF16 range is similar to FP32, scaling prevents from underflowing.
+We tested BF16-AMP without the GradScaler and it does not give good results.
+
+
+For pure-bf16 training (using torch-optimi and kahan summation)
+```
+compute_dtype: bf16
+use_amp: false
+optim: adam or adamw
+```
+Pure-BF16 training is faster than AMP and the memory footprint is reduced (master weights are kept in BF16 vs FP32). However Kahan Summation is not magical, results are good but not as good as AMP.
+Use this feature mainly when memory footprint is an issue with LLMs.
+
+
+---
+
+## Contributing
+
+We love contributions! Please look at issues marked with the [contributions welcome](https://github.com/eole-nlp/eole/issues?q=is%3Aissue+is%3Aopen+label%3A%22contributions+welcome%22) tag.
+
+Before raising an issue, make sure you read the requirements and the [Full Documentation](https://eole-nlp.github.io/eole). You can also check if a [Recipe](https://github.com/eole-nlp/eole/tree/main/recipes) fits your use case.
+
+Unless there is a bug, please use the [Discussions](https://github.com/eole-nlp/eole/discussions) tab to ask questions or propose new topics/features.

eole-0.5.0/eole/__init__.py

@@ -0,0 +1,11 @@
+import os
+
+__version__ = "0.5.0"
+ROOT_DIR = os.path.abspath(os.path.dirname(__file__))
+EOLE_TORCH_COMPILE = os.environ.get("EOLE_TORCH_COMPILE", "0") == "1"
+
+EOLE_COMPILE_MODE = os.environ.get("EOLE_COMPILE_MODE", "2")
+# Mode = 0 : Decoder Level - cudagraphs True
+# Mode = 1 : Decoder Level - cudagraphs False
+# Mode = 2 : Decoder Layer Level - cudagraphs True
+# Mode = 3 : Decoder Layer Level - cudagraphs False

eole-0.5.0/eole/adapters/__init__.py

@@ -0,0 +1,14 @@
+"""Module defining adapters / mm projectors."""
+
+from eole.adapters.adapters import VisionLanguageAdapter
+from eole.adapters.adapters import Gemma3MultiModalProjector
+from eole.adapters.adapters import DeepSeekOCRProjector
+from eole.adapters.adapters import HunYuanVisionPatchMerger
+
+
+str2adapter = {
+    "llava": VisionLanguageAdapter,
+    "gemma3": Gemma3MultiModalProjector,
+    "deepseekocr": DeepSeekOCRProjector,
+    "hunyuanocr": HunYuanVisionPatchMerger,
+}