textslinger 0.2.1__tar.gz → 0.2.4__tar.gz

{textslinger-0.2.1 → textslinger-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: textslinger
-Version: 0.2.1
+Version: 0.2.4
 Summary: TextSlinger: Fast and accurate text predictions in Python
 Author-email: Keith Vertanen <vertanen@mtu.edu>, Dylan Gaines <dgaine20@kennesaw.edu>, Soufia Bahmani <sbahmani@mtu.edu>
 License-Expression: MIT
@@ -42,6 +42,7 @@ Requires-Dist: peft~=0.14.0
 Provides-Extra: dev
 Requires-Dist: coverage>=7.0; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
 Provides-Extra: release
 Requires-Dist: twine==5.0.0; extra == "release"
 Requires-Dist: build>=1.4.0; extra == "release"
@@ -72,6 +73,13 @@ Current features:
 Our code style is whatever the Black formatter says it should be. 
 You should [configure your IDE to format using Black when you save](https://black.readthedocs.io/en/stable/integrations/editors.html).
 
+### Windows
+Windows is not supported by the library dependencies of textslinger.
+To use our library on Windows, you will need to use [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-gb/windows/wsl/).
+Please follow the [installation directions](https://learn.microsoft.com/en-gb/windows/wsl/install) and use the Linux commands below.
+You will also need to install pip to install Python libraries. Follow the 
+[pip installation directions](https://pip.pypa.io/en/stable/installation/) before proceeding.
+
 ## Setting up a Python environment
 If you don't have [Miniforge](https://conda-forge.org/download/) installed in your user account you'll first need to do that.
 
@@ -81,12 +89,17 @@ curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Minif
 zsh Miniforge3-MacOSX-arm64.sh
 ~/miniforge3/bin/conda init zsh
 ```
-To install Miniforge on Linux:
+To install Miniforge on Linux and WSL:
 ```
 curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
 bash Miniforge3-Linux-x86_64.sh
 ```
 
+When installing on WSL, enter "yes" when asked if you want to automatically initialize conda. This sets up easier access to conda in your WSL environment. You can disable the automatic start of the base environment using the following conda command after restarting your terminal:
+```
+conda config --set auto_activate_base false
+```
+
 After installing Miniforge, be sure to close your terminal and start a new one. Create an environment as follows:
 ```
 conda config --remove-key channels
@@ -97,6 +110,7 @@ conda activate textslinger
 ```
 
 ### Installation of PyTorch
+
 MacOS using Apple Silicon:
 ```
 pip install torch torchvision torchaudio
@@ -115,6 +129,18 @@ Linux without CUDA support:
 pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
 ```
 
+WSL with CUDA support (GPU driver must support installed library version or greater. Run ```nvidia-smi``` to check driver support).
+Due to a security vulnerability in earlier versions, WSL requires at least torch 2.6:
+
+```
+# CUDA 12.6
+python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
+```
+WSL without CUDA support:
+```
+python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
+```
+
 Test if the PyTorch installation worked:
 ```
 python - <<'EOF'
@@ -139,10 +165,18 @@ EOF
 ```
 
 ### Installation of libraries
-Install transformers (5.2.0 or greater required). 
+Install transformers (5.2.0 or greater required).
+
+MacOS and Linux:
 ```
 pip install transformers
 ```
+
+WSL:
+```
+python -m pip install transformers
+```
+
 Check transformers version and model support:
 ```
 python - <<'EOF'
@@ -161,20 +195,57 @@ except Exception as e:
     print("BLT model missing ❌", e)
 EOF
 ```
-Install other dependencies:
+
+#### Hugging Face Authentication
+
+You will need to log in to Hugging Face using the CLI. This CLI is installed with the `transformers` library. Please see the [Hugging Face CLI Login direction](https://huggingface.co/docs/huggingface_hub/en/guides/cli#hf-auth-login) for how to do this. We recommend [using a token](https://huggingface.co/docs/huggingface_hub/en/quick-start#authentication) so you do not need to manually enter your credentials.
+
+```
+hf auth login
+```
+
+Install other dependencies on MacOS and Linux:
 ```
 pip install pytest scipy peft psutil datasets
 
 # NOTE: increase MAX_ORDER if you plan to load n-gram models with longer context 
 MAX_ORDER=12 pip install https://github.com/kpu/kenlm/archive/master.zip
 ```
-Fix harmless warning message:
+
+Install other dependencies on WSL:
+```
+python -m pip install pytest scipy peft psutil datasets
+```
+
+If using the default WSL Ubuntu installation, the `build-essential` package is needed to build the KenLM library. You also need the `unzip` package to run the `download.sh` script later in this guide.
+```
+sudo apt-get update && sudo apt-get upgrade
+sudo apt-get install build-essential unzip
+# NOTE: increase MAX_ORDER if you plan to load n-gram models with longer context 
+export MAX_ORDER=12
+python -m pip install https://github.com/kpu/kenlm/archive/master.zip
+```
+
+Fix harmless warning message on MacOS and Linux:
 ```
 pip install --upgrade --force-reinstall setuptools
 ```
 
+Fix harmless warning message on WSL:
+
+CUDA 12.6 does not support `setuptools` versions 82.0.0+.
+
+```
+python -m pip install --upgrade --force-reinstall setuptools==81.0.0
+```
+
+Install TextSlinger itself so scripts can import the `textslinger` package from any working directory. Run this from the repository root on all platforms, the directory containing `pyproject.toml`:
+```
+python -m pip install --no-deps -e .
+```
+
 ### Testing installation
-Download assets needed by the test suite and then run it:
+Download assets needed by the test suite and then run it. If you are on WSL, check if the `assets` directory is owned by `root` using `ls -l`. If so, restart your WSL session and rercheck the ownsership.
 ```
 cd textslinger/assets
 ./download.sh
@@ -182,5 +253,10 @@ cd ..
 pytest -v -rs
 ```
 
+If you encounter an error with line endings (e.g., "\r: no such file or directory") when running `download.sh`, you can change the line endings using:
+
+```
+vim download.sh -c "set ff=unix" -c ":wq"
+```
 ---
 This material is based upon work supported by the NSF under Grant No. IIS-1909089 and IIS-2402876.

{textslinger-0.2.1 → textslinger-0.2.4}/README.md

@@ -22,6 +22,13 @@ Current features:
 Our code style is whatever the Black formatter says it should be. 
 You should [configure your IDE to format using Black when you save](https://black.readthedocs.io/en/stable/integrations/editors.html).
 
+### Windows
+Windows is not supported by the library dependencies of textslinger.
+To use our library on Windows, you will need to use [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-gb/windows/wsl/).
+Please follow the [installation directions](https://learn.microsoft.com/en-gb/windows/wsl/install) and use the Linux commands below.
+You will also need to install pip to install Python libraries. Follow the 
+[pip installation directions](https://pip.pypa.io/en/stable/installation/) before proceeding.
+
 ## Setting up a Python environment
 If you don't have [Miniforge](https://conda-forge.org/download/) installed in your user account you'll first need to do that.
 
@@ -31,12 +38,17 @@ curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Minif
 zsh Miniforge3-MacOSX-arm64.sh
 ~/miniforge3/bin/conda init zsh
 ```
-To install Miniforge on Linux:
+To install Miniforge on Linux and WSL:
 ```
 curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
 bash Miniforge3-Linux-x86_64.sh
 ```
 
+When installing on WSL, enter "yes" when asked if you want to automatically initialize conda. This sets up easier access to conda in your WSL environment. You can disable the automatic start of the base environment using the following conda command after restarting your terminal:
+```
+conda config --set auto_activate_base false
+```
+
 After installing Miniforge, be sure to close your terminal and start a new one. Create an environment as follows:
 ```
 conda config --remove-key channels
@@ -47,6 +59,7 @@ conda activate textslinger
 ```
 
 ### Installation of PyTorch
+
 MacOS using Apple Silicon:
 ```
 pip install torch torchvision torchaudio
@@ -65,6 +78,18 @@ Linux without CUDA support:
 pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
 ```
 
+WSL with CUDA support (GPU driver must support installed library version or greater. Run ```nvidia-smi``` to check driver support).
+Due to a security vulnerability in earlier versions, WSL requires at least torch 2.6:
+
+```
+# CUDA 12.6
+python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
+```
+WSL without CUDA support:
+```
+python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
+```
+
 Test if the PyTorch installation worked:
 ```
 python - <<'EOF'
@@ -89,10 +114,18 @@ EOF
 ```
 
 ### Installation of libraries
-Install transformers (5.2.0 or greater required). 
+Install transformers (5.2.0 or greater required).
+
+MacOS and Linux:
 ```
 pip install transformers
 ```
+
+WSL:
+```
+python -m pip install transformers
+```
+
 Check transformers version and model support:
 ```
 python - <<'EOF'
@@ -111,20 +144,57 @@ except Exception as e:
     print("BLT model missing ❌", e)
 EOF
 ```
-Install other dependencies:
+
+#### Hugging Face Authentication
+
+You will need to log in to Hugging Face using the CLI. This CLI is installed with the `transformers` library. Please see the [Hugging Face CLI Login direction](https://huggingface.co/docs/huggingface_hub/en/guides/cli#hf-auth-login) for how to do this. We recommend [using a token](https://huggingface.co/docs/huggingface_hub/en/quick-start#authentication) so you do not need to manually enter your credentials.
+
+```
+hf auth login
+```
+
+Install other dependencies on MacOS and Linux:
 ```
 pip install pytest scipy peft psutil datasets
 
 # NOTE: increase MAX_ORDER if you plan to load n-gram models with longer context 
 MAX_ORDER=12 pip install https://github.com/kpu/kenlm/archive/master.zip
 ```
-Fix harmless warning message:
+
+Install other dependencies on WSL:
+```
+python -m pip install pytest scipy peft psutil datasets
+```
+
+If using the default WSL Ubuntu installation, the `build-essential` package is needed to build the KenLM library. You also need the `unzip` package to run the `download.sh` script later in this guide.
+```
+sudo apt-get update && sudo apt-get upgrade
+sudo apt-get install build-essential unzip
+# NOTE: increase MAX_ORDER if you plan to load n-gram models with longer context 
+export MAX_ORDER=12
+python -m pip install https://github.com/kpu/kenlm/archive/master.zip
+```
+
+Fix harmless warning message on MacOS and Linux:
 ```
 pip install --upgrade --force-reinstall setuptools
 ```
 
+Fix harmless warning message on WSL:
+
+CUDA 12.6 does not support `setuptools` versions 82.0.0+.
+
+```
+python -m pip install --upgrade --force-reinstall setuptools==81.0.0
+```
+
+Install TextSlinger itself so scripts can import the `textslinger` package from any working directory. Run this from the repository root on all platforms, the directory containing `pyproject.toml`:
+```
+python -m pip install --no-deps -e .
+```
+
 ### Testing installation
-Download assets needed by the test suite and then run it:
+Download assets needed by the test suite and then run it. If you are on WSL, check if the `assets` directory is owned by `root` using `ls -l`. If so, restart your WSL session and rercheck the ownsership.
 ```
 cd textslinger/assets
 ./download.sh
@@ -132,5 +202,10 @@ cd ..
 pytest -v -rs
 ```
 
+If you encounter an error with line endings (e.g., "\r: no such file or directory") when running `download.sh`, you can change the line endings using:
+
+```
+vim download.sh -c "set ff=unix" -c ":wq"
+```
 ---
 This material is based upon work supported by the NSF under Grant No. IIS-1909089 and IIS-2402876.

textslinger-0.2.4/assets/download_hf_models.py

@@ -0,0 +1,23 @@
+"""Download Hugging Face models used by repository tests into the HF cache."""
+
+from huggingface_hub import snapshot_download
+
+
+MODEL_NAMES = (
+    "figmtu/opt-125m-aac",
+    "itazap/blt-1b-hf",
+    "distilbert/distilgpt2",
+    "EleutherAI/pythia-70m",
+    "EleutherAI/gpt-neo-125m",
+)
+
+
+def main() -> None:
+    for model_name in MODEL_NAMES:
+        print(f"*** Downloading Hugging Face model: {model_name}")
+        model_path = snapshot_download(repo_id=model_name)
+        print(f"    Cached at: {model_path}")
+
+
+if __name__ == "__main__":
+    main()

{textslinger-0.2.1 → textslinger-0.2.4}/pyproject.toml

@@ -52,6 +52,7 @@ dependencies = [
 dev = [
     "coverage>=7.0",
     "pytest",
+    "black"
 ]
 release = [
     "twine==5.0.0",
@@ -76,5 +77,5 @@ version = {attr = "textslinger.__version__"}
 exclude = ["examples", "scripts"]
 
 [tool.black]
-line-length = 88
+line-length = 100
 target-version = ["py310"]