eole 0.0.1__tar.gz

eole-0.0.1/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024-Present EOLE
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

eole-0.0.1/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.1
+Name: eole
+Version: 0.0.1
+Summary: Open language modeling toolkit based on PyTorch
+Project-URL: Source, https://github.com/eole-nlp/eole/
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: configargparse
+Requires-Dist: ctranslate2<5,>=4
+Requires-Dist: fasttext-wheel
+Requires-Dist: flask
+Requires-Dist: huggingface_hub
+Requires-Dist: numpy<2.0
+Requires-Dist: pandas
+Requires-Dist: pyahocorasick
+Requires-Dist: pyonmttok<2,>=1.37
+Requires-Dist: pyyaml
+Requires-Dist: rapidfuzz
+Requires-Dist: sacrebleu
+Requires-Dist: safetensors
+Requires-Dist: sentencepiece
+Requires-Dist: sentencepiece<0.1.98,>=0.1.94
+Requires-Dist: six
+Requires-Dist: spacy
+Requires-Dist: subword-nmt>=0.3.7
+Requires-Dist: tensorboard>=2.3
+Requires-Dist: torch<2.4,>=2.3
+Requires-Dist: waitress
+
+# EOLE
+
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://eole-nlp.github.io/eole)
+
+Open language modeling toolkit based on [PyTorch](https://pytorch.org).
+
+## 👷‍♂️🚧 Work in Progress
+
+[EOLE](https://github.com/eole-nlp/eole) is a spin-off of the [OpenNMT-py](https://github.com/opennmt/opennmt-py) project. We aim to maintain the research-friendly approach of the original project while updating the structure and expanding it to include new topics related to large language models (LLMs) and various other techniques. Our goal is to provide a comprehensive yet compact and modular codebase for experimenting with various types of language models (encoder, decoder, seq2seq).
+
+---
+
+### Current State
+
+We have made significant progress in several areas:
+
+- **Configuration Management**: Streamlined through [pydantic](https://docs.pydantic.dev) models.
+- **Command Line Entry Points**: Improved using structured subparsers for better organization.
+- **Reproducible Recipes**: Provided for widely used models and tasks, ensuring consistency and reliability.
+- **Core API Simplification**: Refined around the new configuration objects for ease of use.
+
+### Future Directions
+
+There are still several exciting avenues to explore:
+
+- **Further Simplification and Refactoring**: Continue enhancing the codebase for clarity and efficiency.
+- **Inference Server**: Develop a robust solution for model inference.
+- **Additional Recipes**: Expand the library of reproducible recipes.
+- **Documentation**: Enhance and expand the documentation for better user guidance.
+- **Test Coverage**: Improve testing to ensure code reliability and performance.
+- **Logging Enhancements**: Implement more sophisticated logging mechanisms.
+- **Broader Model Support**: Extend support to include a wider range of open models, potentially multi-modal.
+
+---
+
+## Key Features
+
+- **Versatile Training and Inference**: Train from scratch, finetune, and infer models of various architectures including Transformer Encoder/Decoder/EncoderDecoder and RNN EncoderDecoder.
+- **Dynamic Data Transforms**: Apply on-the-fly transformations in the dataloading logic for both training and inference.
+- **Comprehensive LLM Support**: Includes converters for Llama, Mistral, Phi, OpenLlama, Redpajama, MPT-7B, and Falcon models.
+- **Advanced Quantization**: Support for 8-bit and 4-bit quantization, along with LoRA adapters, with or without checkpointing, as well as mixed precision (FP16).
+- **Efficient Finetuning**: Finetune 7B and 13B models on a single RTX 24GB GPU using 4-bit quantization.
+- **Flexible Inference**: Perform inference in 4-bit or 8-bit using the same layer quantization methods as in finetuning.
+- **Tensor Parallelism**: Enable tensor parallelism for both training and inference when models exceed the memory capacity of a single GPU.
+
+---
+
+## Setup
+
+### Using Docker
+
+To facilitate setup and reproducibility, we provide Docker images via the GitHub Container Registry: [EOLE Docker Images](https://github.com/eole-nlp/eole/pkgs/container/eole).
+
+You can customize the workflow and build your own images based on specific needs using `build.sh` and `Dockerfile` in the `docker` directory of the repository.
+
+To pull the Docker image:
+```bash
+docker pull ghcr.io/eole-nlp/eole:0.0.1-ubuntu22.04-cuda12.1
+```
+
+Example one-liner to run a container and open a bash shell within it:
+```bash
+docker run --rm -it --runtime=nvidia ghcr.io/eole-nlp/eole:0.0.1-ubuntu22.04-cuda12.1
+```
+
+> **Note**: Ensure you have the [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) (formerly nvidia-docker) installed to take advantage of CUDA/GPU features.
+
+Depending on your needs, you can add various flags:
+- `-p 5000:5000`: Forward an exposed port from your container to your host.
+- `-v /some/local/directory:/some/container/directory`: Mount a local directory to a container directory.
+- `--entrypoint some_command`: Run a specific command as the container entry point (instead of the default bash shell).
+
+### Installing Locally
+
+#### Requirements
+
+- Python >= 3.10
+- PyTorch >= 2.3 < 2.4
+
+#### Installation from Source
+
+To install from source:
+```bash
+git clone https://github.com/eole-nlp/eole
+cd eole
+pip install -e .
+```
+
+#### Installation from PyPI
+
+Installation from PyPI will be available soon.
+
+#### Notes
+
+If you encounter a `MemoryError` during installation, try using `pip` with the `--no-cache-dir` option.
+
+(Optional) Some advanced features (e.g., pretrained models or specific transforms) require extra packages. Install them with:
+```bash
+pip install -r requirements.opt.txt
+```
+
+### Manual Installation of Some Dependencies
+
+#### Apex
+
+Apex is recommended for improved performance, especially for the legacy fusedadam optimizer and FusedRMSNorm.
+```bash
+git clone https://github.com/NVIDIA/apex
+cd apex
+pip3 install -v --no-build-isolation --config-settings --build-option="--cpp_ext --cuda_ext --deprecated_fused_adam --xentropy --fast_multihead_attn" ./
+cd ..
+```
+
+#### Flash Attention
+
+As of October 2023, Flash Attention 1 has been integrated into PyTorch v2, but using Flash Attention 2 with v2.3.1 is recommended for sliding window attention support. 
+
+For regular `position_encoding=True` or Rotary with `max_relative_positions=-1`, `eole` will attempt to use an optimized dot-product path. To use [Flash Attention](https://github.com/Dao-AILab/flash-attention#installation-and-features), install it manually:
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+If Flash Attention 2 is not installed, `F.scaled_dot_product_attention` from PyTorch 2.x will be used.
+
+For `max_relative_positions > 0` or Alibi `max_relative_positions=-2`, `eole` will use legacy code for matrix multiplications. Flash Attention and `F.scaled_dot_product_attention` offer faster performance and better GPU memory efficiency.
+
+#### AWQ
+
+For inference or quantizing an AWQ model, AutoAWQ is required. Install it with:
+```bash
+pip install autoawq
+```
+
+For more details, refer to [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
+
+---
+
+## Contributing
+
+We love contributions! Please look at issues marked with the [contributions welcome](https://github.com/eole-nlp/eole/issues?q=is%3Aissue+is%3Aopen+label%3A%22contributions+welcome%22) tag.
+
+Before raising an issue, make sure you read the requirements and the [Full Documentation](https://eole-nlp.github.io/eole). You can also check if a [Recipe](https://github.com/eole-nlp/eole/tree/main/recipes) fits your use case.
+
+Unless there is a bug, please use the [Discussions](https://github.com/eole-nlp/eole/discussions) tab to ask questions or propose new topics/features.

eole-0.0.1/README.md

@@ -0,0 +1,144 @@
+# EOLE
+
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://eole-nlp.github.io/eole)
+
+Open language modeling toolkit based on [PyTorch](https://pytorch.org).
+
+## 👷‍♂️🚧 Work in Progress
+
+[EOLE](https://github.com/eole-nlp/eole) is a spin-off of the [OpenNMT-py](https://github.com/opennmt/opennmt-py) project. We aim to maintain the research-friendly approach of the original project while updating the structure and expanding it to include new topics related to large language models (LLMs) and various other techniques. Our goal is to provide a comprehensive yet compact and modular codebase for experimenting with various types of language models (encoder, decoder, seq2seq).
+
+---
+
+### Current State
+
+We have made significant progress in several areas:
+
+- **Configuration Management**: Streamlined through [pydantic](https://docs.pydantic.dev) models.
+- **Command Line Entry Points**: Improved using structured subparsers for better organization.
+- **Reproducible Recipes**: Provided for widely used models and tasks, ensuring consistency and reliability.
+- **Core API Simplification**: Refined around the new configuration objects for ease of use.
+
+### Future Directions
+
+There are still several exciting avenues to explore:
+
+- **Further Simplification and Refactoring**: Continue enhancing the codebase for clarity and efficiency.
+- **Inference Server**: Develop a robust solution for model inference.
+- **Additional Recipes**: Expand the library of reproducible recipes.
+- **Documentation**: Enhance and expand the documentation for better user guidance.
+- **Test Coverage**: Improve testing to ensure code reliability and performance.
+- **Logging Enhancements**: Implement more sophisticated logging mechanisms.
+- **Broader Model Support**: Extend support to include a wider range of open models, potentially multi-modal.
+
+---
+
+## Key Features
+
+- **Versatile Training and Inference**: Train from scratch, finetune, and infer models of various architectures including Transformer Encoder/Decoder/EncoderDecoder and RNN EncoderDecoder.
+- **Dynamic Data Transforms**: Apply on-the-fly transformations in the dataloading logic for both training and inference.
+- **Comprehensive LLM Support**: Includes converters for Llama, Mistral, Phi, OpenLlama, Redpajama, MPT-7B, and Falcon models.
+- **Advanced Quantization**: Support for 8-bit and 4-bit quantization, along with LoRA adapters, with or without checkpointing, as well as mixed precision (FP16).
+- **Efficient Finetuning**: Finetune 7B and 13B models on a single RTX 24GB GPU using 4-bit quantization.
+- **Flexible Inference**: Perform inference in 4-bit or 8-bit using the same layer quantization methods as in finetuning.
+- **Tensor Parallelism**: Enable tensor parallelism for both training and inference when models exceed the memory capacity of a single GPU.
+
+---
+
+## Setup
+
+### Using Docker
+
+To facilitate setup and reproducibility, we provide Docker images via the GitHub Container Registry: [EOLE Docker Images](https://github.com/eole-nlp/eole/pkgs/container/eole).
+
+You can customize the workflow and build your own images based on specific needs using `build.sh` and `Dockerfile` in the `docker` directory of the repository.
+
+To pull the Docker image:
+```bash
+docker pull ghcr.io/eole-nlp/eole:0.0.1-ubuntu22.04-cuda12.1
+```
+
+Example one-liner to run a container and open a bash shell within it:
+```bash
+docker run --rm -it --runtime=nvidia ghcr.io/eole-nlp/eole:0.0.1-ubuntu22.04-cuda12.1
+```
+
+> **Note**: Ensure you have the [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) (formerly nvidia-docker) installed to take advantage of CUDA/GPU features.
+
+Depending on your needs, you can add various flags:
+- `-p 5000:5000`: Forward an exposed port from your container to your host.
+- `-v /some/local/directory:/some/container/directory`: Mount a local directory to a container directory.
+- `--entrypoint some_command`: Run a specific command as the container entry point (instead of the default bash shell).
+
+### Installing Locally
+
+#### Requirements
+
+- Python >= 3.10
+- PyTorch >= 2.3 < 2.4
+
+#### Installation from Source
+
+To install from source:
+```bash
+git clone https://github.com/eole-nlp/eole
+cd eole
+pip install -e .
+```
+
+#### Installation from PyPI
+
+Installation from PyPI will be available soon.
+
+#### Notes
+
+If you encounter a `MemoryError` during installation, try using `pip` with the `--no-cache-dir` option.
+
+(Optional) Some advanced features (e.g., pretrained models or specific transforms) require extra packages. Install them with:
+```bash
+pip install -r requirements.opt.txt
+```
+
+### Manual Installation of Some Dependencies
+
+#### Apex
+
+Apex is recommended for improved performance, especially for the legacy fusedadam optimizer and FusedRMSNorm.
+```bash
+git clone https://github.com/NVIDIA/apex
+cd apex
+pip3 install -v --no-build-isolation --config-settings --build-option="--cpp_ext --cuda_ext --deprecated_fused_adam --xentropy --fast_multihead_attn" ./
+cd ..
+```
+
+#### Flash Attention
+
+As of October 2023, Flash Attention 1 has been integrated into PyTorch v2, but using Flash Attention 2 with v2.3.1 is recommended for sliding window attention support. 
+
+For regular `position_encoding=True` or Rotary with `max_relative_positions=-1`, `eole` will attempt to use an optimized dot-product path. To use [Flash Attention](https://github.com/Dao-AILab/flash-attention#installation-and-features), install it manually:
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+If Flash Attention 2 is not installed, `F.scaled_dot_product_attention` from PyTorch 2.x will be used.
+
+For `max_relative_positions > 0` or Alibi `max_relative_positions=-2`, `eole` will use legacy code for matrix multiplications. Flash Attention and `F.scaled_dot_product_attention` offer faster performance and better GPU memory efficiency.
+
+#### AWQ
+
+For inference or quantizing an AWQ model, AutoAWQ is required. Install it with:
+```bash
+pip install autoawq
+```
+
+For more details, refer to [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
+
+---
+
+## Contributing
+
+We love contributions! Please look at issues marked with the [contributions welcome](https://github.com/eole-nlp/eole/issues?q=is%3Aissue+is%3Aopen+label%3A%22contributions+welcome%22) tag.
+
+Before raising an issue, make sure you read the requirements and the [Full Documentation](https://eole-nlp.github.io/eole). You can also check if a [Recipe](https://github.com/eole-nlp/eole/tree/main/recipes) fits your use case.
+
+Unless there is a bug, please use the [Discussions](https://github.com/eole-nlp/eole/discussions) tab to ask questions or propose new topics/features.

eole-0.0.1/eole/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.0.1"

eole-0.0.1/eole/bin/__init__.py

@@ -0,0 +1,63 @@
+import os
+import importlib
+import inspect
+
+
+class BaseBin(object):
+    """
+    Abstract base class for tools.
+    """
+
+    @classmethod
+    def add_args(cls, parser):
+        raise NotImplementedError
+
+    @classmethod
+    def run(cls, parser):
+        raise NotImplementedError
+
+
+AVAILABLE_BINS = {}
+
+
+def register_bin(name):
+    """Wrapper to automatically register available tools."""
+
+    def register_bin_cls(cls):
+        # patch to ignore standalone imports to prevent conflicts (e.g. in build docs)
+        if not (cls.__module__.startswith("eole")):
+            return cls
+        # bin_type is retrieved from bin parent dirname
+        bin_type = os.path.basename(os.path.dirname(inspect.stack()[1].filename))
+        if bin_type in AVAILABLE_BINS:
+            if (
+                name in AVAILABLE_BINS[bin_type]
+                and cls != AVAILABLE_BINS[bin_type][name]
+            ):
+                raise ValueError(
+                    f'Name "{name}" registered twice for different classes: \n'
+                    f"{AVAILABLE_BINS[bin_type][name]} != {cls}"
+                )
+            else:
+                AVAILABLE_BINS[bin_type][name] = cls
+        else:
+            AVAILABLE_BINS[bin_type] = {name: cls}
+        return cls
+
+    return register_bin_cls
+
+
+# Auto import python files in all subdirectories
+bin_dir = os.path.dirname(__file__)
+for dir in os.listdir(bin_dir):
+    abs_dir = os.path.join(os.path.dirname(__file__), dir)
+    if os.path.isdir(abs_dir):
+        for file in os.listdir(abs_dir):
+            _path = os.path.join(abs_dir, file)
+            if (
+                not file.startswith("_")
+                and not file.startswith(".")
+                and (file.endswith(".py") or os.path.isdir(_path))
+            ):
+                file_name = file[: file.find(".py")] if file.endswith(".py") else file
+                module = importlib.import_module(f"eole.bin.{dir}." + file_name)