hf-serve 0.1.0__tar.gz

hf_serve-0.1.0/PKG-INFO

@@ -0,0 +1,215 @@
+Metadata-Version: 2.4
+Name: hf-serve
+Version: 0.1.0
+Summary: Experimental Hugging Face API for Transformers, Diffusers and Sentence Transformers, written in Python.
+Author: Alvaro Bartolome
+Author-email: Alvaro Bartolome <alvaro.bartolome@huggingface.co>
+License-Expression: Apache-2.0
+Requires-Dist: uvicorn>=0.34.0
+Requires-Dist: fastapi>=0.115.6
+Requires-Dist: starlette>=1.0.1
+Requires-Dist: prometheus-client>=0.21.1
+Requires-Dist: torch
+Requires-Dist: torchvision
+Requires-Dist: torchcodec
+Requires-Dist: diffusers>=0.37.0
+Requires-Dist: transformers[accelerate,audio,chat-template,kernels,timm,vision]>=5.9.0
+Requires-Dist: pydantic>=2.10.5
+Requires-Dist: sentence-transformers>=5.5.1
+Requires-Dist: sentencepiece>=0.2.0
+Requires-Dist: protobuf>=5.29.3
+Requires-Dist: peft>=0.14.0
+Requires-Dist: pillow>=11.1.0
+Requires-Dist: pydub>=0.25.0
+Requires-Dist: python-multipart>=0.0.20
+Requires-Dist: python-magic>=0.4.27
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: torch>=2.5.1,<2.8.0 ; extra == 'cpu'
+Requires-Dist: torchvision ; extra == 'cpu'
+Requires-Dist: torchcodec<0.6 ; extra == 'cpu'
+Requires-Dist: torch>=2.5.1,<2.8.0 ; extra == 'cuda'
+Requires-Dist: torchvision ; extra == 'cuda'
+Requires-Dist: torchcodec<0.6 ; extra == 'cuda'
+Requires-Dist: torch>2.8.0 ; extra == 'cuda-130'
+Requires-Dist: torchvision ; extra == 'cuda-130'
+Requires-Dist: torchcodec ; extra == 'cuda-130'
+Requires-Dist: flash-attn==2.8.3 ; extra == 'flash-attn'
+Maintainer: Alvaro Bartolome, Juan Julian Cea, Erik Kaunismäki
+Maintainer-email: Alvaro Bartolome <alvaro.bartolome@huggingface.co>, Juan Julian Cea <juan.moran@huggingface.co>, Erik Kaunismäki <erik.kaunismaki@huggingface.co>
+Requires-Python: >=3.10, <3.13
+Provides-Extra: cpu
+Provides-Extra: cuda
+Provides-Extra: cuda-130
+Provides-Extra: flash-attn
+Description-Content-Type: text/markdown
+
+# `hf-serve`
+
+> [!WARNING]
+> This project is still experimental, meant to replace the former
+> [`huggingface-inference-toolkit`](https://github.com/huggingface/huggingface-inference-toolkit).
+
+## Installation
+
+First you need to setup your environment with [`uv`](https://github.com/astral-sh/uv) (or with your preferred Python environment manager).
+
+```bash
+uv venv --python 3.12
+source .venv/bin/activate
+```
+
+> [!NOTE]
+> Due to the need of `--preview-features extra-build-dependencies` to install
+> [`flash-attn`](https://github.com/Dao-AILab/flash-attention) with `uv` without
+> compiling it, but rather relying on the pre-built binaries, you need to use
+> `uv` v0.8.13 (or higher, but beware on major updates since the feature is
+> still experimental, so < v0.9.0 is recommended until stable).
+>
+> Reference: https://docs.astral.sh/uv/concepts/projects/config/#augmenting-build-dependencies
+>
+> To update `uv` once installed if `uv version` is lower than v0.8.13, simply
+> `uv self update`.
+
+Install it from the `uv.lock` file for CPU / MPS as follows:
+
+```bash
+uv sync --active --frozen --extra cpu
+```
+
+Alternatively, install it on NVIDIA CUDA 12.6 as follows:
+
+```bash
+uv sync --active --frozen --extra cuda-126 --extra flash-attn --preview-features extra-build-dependencies
+```
+
+Or if you have CUDA +13.0 then:
+
+```bash
+uv sync --active --frozen --extra cuda-130 --extra flash-attn --preview-features extra-build-dependencies
+```
+
+> [!WARNING]
+> The default registry for the NVIDIA CUDA wheels for PyTorch is set to CUDA 12.6. If
+> you want to install another PyTorch version as per the CUDA compatibility, then
+> run e.g. `uv pip install -e . --torch-backend cu128`, but note it won't be relying
+> on the `uv.lock` so some dependencies might mismatch.
+>
+> Reference: https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection
+
+```console
+$ uv run hf-serve --help
+```
+
+## Examples
+
+> [!NOTE]
+> On the examples below, given the recently introduced `extra-build-dependencies`
+> for `flash-attn` on CUDA as per https://docs.astral.sh/uv/concepts/projects/config/#build-isolation,
+> it means that you'll need to run the examples as `uv run --preview-features extra-build-dependencies ...`
+> to disable the warning:
+> ```console
+> warning: The `extra-build-dependencies` option is experimental and may change without warning. Pass `--preview-features extra-build-dependencies` to disable this warning.
+> ```
+
+### 🤏 Run `HuggingFaceTB/SmolLM3-3B` with an OpenAI API
+
+```bash
+uv run hf-serve --model-id HuggingFaceTB/SmolLM3-3B --task text-generation --dtype float16
+```
+
+> [!NOTE]
+> If you are running on an instance with NVIDIA GPU, it's recommended to install `hf-serve`
+> with `flash-attn` extra in order to benefit from accelerated inference:
+> ```bash
+> uv sync --active --frozen --extra cuda --extra flash-attn --preview-features extra-build-dependencies
+> ```
+
+### 🔵 Run `sentence-transformers/all-MiniLM-L6-v2` on Azure AI
+
+```bash
+uv run hf-serve --model-id sentence-transformers/all-MiniLM-L6-v2 --task sentence-similarity --dtype float32 --cloud azure
+```
+
+> [!WARNING]
+> Given that Azure AI Foundry and Azure ML expect the inference route to be `/score`
+> rather than `/predict`, which is the standard for Inference Endpoints API, and since
+> `/score` is a redirect to `/predict`, then we need to send the `curl` request with the
+> `-L/--location` flag so that it follows the redirect, otherwise we get an HTTP 307.
+>
+> ```bash
+> curl -L http://localhost:8080/score -H "Content-Type: application/json" -d '{"inputs":{"source_sentence":"What is Deep Learning?","sentences":["Deep Learning is...","Deep Learning is not..."]}}'
+> ```
+
+### 👂 Run `facebook/wav2vec2-base-960h` an `automatic-speech-recognition` model
+
+> [!NOTE]
+> Before running `automatic-speech-recognition` or really any of `audio-classification`
+> or `zero-shot-audio-classification` you will need to install some system dependencies
+> in advance for those to work as `ffmpeg` and `libmagic-dev`.
+
+```bash
+uv run hf-serve --model-id facebook/wav2vec2-large-960h --task automatic-speech-recognition --dtype float16
+```
+
+> [!WARNING]
+> On MacOS, if you installed `ffmpeg` via `brew`, you will need to set the following
+> environment variable in advance `DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib`
+>
+> Reference: https://github.com/pytorch/torchcodec/issues/570#issuecomment-2913609176
+
+And, then you can send a sample request as:
+
+```bash
+curl -L http://localhost:8080/predict \
+    -H "Content-Type: application/json" \
+    -d '{"inputs":"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac"}'
+```
+
+> [!WARNING]
+> Given the nature of some tasks that need to support JSON, forms, and files,
+> the `/predict` method for those is a redirect to the respective inner endpoint:
+> `/predict-json`, `/predict-form`, and `/predict-file`. Those are non-standard
+> but required to keep full compatibility with the current Hugging Face Inference
+> Endpoints API Specification, but in reality the redirect response (HTTP 307)
+> shouldn't be used as an routing route, but rather dedicated routes for those.
+
+> [!NOTE]
+> The OpenAI Audio Transcriptions API is still not yet part of `hf-serve` but it's
+> on the roadmap and it will be released soon, stay tuned!
+
+### 🔈 Run `facebook/wav2vec2-lv-60-espeak-cv-ft` (with `phonemizer` and `espeak`)
+
+> [!NOTE]
+> Some models as e.g. `facebook/wav2vec2-lv-60-espeak-cv-ft`, rely on `phonemizer` for
+> the "phonemization" of words and texts in many languages, based at the same time on
+> different Text-To-Speech (TTS) backends as e.g. `espeak-ng` which is supports a lot
+> of languages and IPA (International Phonetic Alphabet). This being said, such models
+> require custom dependencies that need to be installed beforehand as those don't come
+> as default `hf-serve` dependencies; whilst those can be installed as e.g. on MacOS:
+>
+> ```bash
+> brew install ffmpeg
+> brew install espeak
+> ```
+
+> [!WARNING]
+> Beware that when installing `ffmpeg` with `brew` on a specific
+> version as e.g. `brew install ffmpeg@7` as it will be installed as "keg-only",
+> meaning that it won't be symlinked into `/opt/homebrew`, meaning that the path
+> to the library won't be `/opt/homebrew/lib` but rather `/opt/homebrew/opt/ffmpeg/lib`
+> instead, meaning that on MacOS you'll need to set `DYLD_FALLBACK_LIBRARY_PATH`
+> to wherever the `ffmpeg` library is installed in.
+
+To run `facebook/wav2vec2-lv-60-espeak-cv-ft` on e.g. MacOS, you need to run the following:
+
+```bash
+DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps
+```
+
+Note that if you have installed another version of `ffmpeg` with `brew` as e.g. `brew install ffmpeg@7`, you should use the following command instead:
+
+```bash
+DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/opt/ffmpeg@7/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps
+```
+
+The main difference relies on the path used for `DYLD_FALLBACK_LIBRARY_PATH` which is now pointing to the exact `brew`-installed version of `ffmpeg` instead. More information on the compatibility issues with `ffmpeg`, `torchcodec` and `torch` at https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.

hf_serve-0.1.0/README.md

@@ -0,0 +1,170 @@
+# `hf-serve`
+
+> [!WARNING]
+> This project is still experimental, meant to replace the former
+> [`huggingface-inference-toolkit`](https://github.com/huggingface/huggingface-inference-toolkit).
+
+## Installation
+
+First you need to setup your environment with [`uv`](https://github.com/astral-sh/uv) (or with your preferred Python environment manager).
+
+```bash
+uv venv --python 3.12
+source .venv/bin/activate
+```
+
+> [!NOTE]
+> Due to the need of `--preview-features extra-build-dependencies` to install
+> [`flash-attn`](https://github.com/Dao-AILab/flash-attention) with `uv` without
+> compiling it, but rather relying on the pre-built binaries, you need to use
+> `uv` v0.8.13 (or higher, but beware on major updates since the feature is
+> still experimental, so < v0.9.0 is recommended until stable).
+>
+> Reference: https://docs.astral.sh/uv/concepts/projects/config/#augmenting-build-dependencies
+>
+> To update `uv` once installed if `uv version` is lower than v0.8.13, simply
+> `uv self update`.
+
+Install it from the `uv.lock` file for CPU / MPS as follows:
+
+```bash
+uv sync --active --frozen --extra cpu
+```
+
+Alternatively, install it on NVIDIA CUDA 12.6 as follows:
+
+```bash
+uv sync --active --frozen --extra cuda-126 --extra flash-attn --preview-features extra-build-dependencies
+```
+
+Or if you have CUDA +13.0 then:
+
+```bash
+uv sync --active --frozen --extra cuda-130 --extra flash-attn --preview-features extra-build-dependencies
+```
+
+> [!WARNING]
+> The default registry for the NVIDIA CUDA wheels for PyTorch is set to CUDA 12.6. If
+> you want to install another PyTorch version as per the CUDA compatibility, then
+> run e.g. `uv pip install -e . --torch-backend cu128`, but note it won't be relying
+> on the `uv.lock` so some dependencies might mismatch.
+>
+> Reference: https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection
+
+```console
+$ uv run hf-serve --help
+```
+
+## Examples
+
+> [!NOTE]
+> On the examples below, given the recently introduced `extra-build-dependencies`
+> for `flash-attn` on CUDA as per https://docs.astral.sh/uv/concepts/projects/config/#build-isolation,
+> it means that you'll need to run the examples as `uv run --preview-features extra-build-dependencies ...`
+> to disable the warning:
+> ```console
+> warning: The `extra-build-dependencies` option is experimental and may change without warning. Pass `--preview-features extra-build-dependencies` to disable this warning.
+> ```
+
+### 🤏 Run `HuggingFaceTB/SmolLM3-3B` with an OpenAI API
+
+```bash
+uv run hf-serve --model-id HuggingFaceTB/SmolLM3-3B --task text-generation --dtype float16
+```
+
+> [!NOTE]
+> If you are running on an instance with NVIDIA GPU, it's recommended to install `hf-serve`
+> with `flash-attn` extra in order to benefit from accelerated inference:
+> ```bash
+> uv sync --active --frozen --extra cuda --extra flash-attn --preview-features extra-build-dependencies
+> ```
+
+### 🔵 Run `sentence-transformers/all-MiniLM-L6-v2` on Azure AI
+
+```bash
+uv run hf-serve --model-id sentence-transformers/all-MiniLM-L6-v2 --task sentence-similarity --dtype float32 --cloud azure
+```
+
+> [!WARNING]
+> Given that Azure AI Foundry and Azure ML expect the inference route to be `/score`
+> rather than `/predict`, which is the standard for Inference Endpoints API, and since
+> `/score` is a redirect to `/predict`, then we need to send the `curl` request with the
+> `-L/--location` flag so that it follows the redirect, otherwise we get an HTTP 307.
+>
+> ```bash
+> curl -L http://localhost:8080/score -H "Content-Type: application/json" -d '{"inputs":{"source_sentence":"What is Deep Learning?","sentences":["Deep Learning is...","Deep Learning is not..."]}}'
+> ```
+
+### 👂 Run `facebook/wav2vec2-base-960h` an `automatic-speech-recognition` model
+
+> [!NOTE]
+> Before running `automatic-speech-recognition` or really any of `audio-classification`
+> or `zero-shot-audio-classification` you will need to install some system dependencies
+> in advance for those to work as `ffmpeg` and `libmagic-dev`.
+
+```bash
+uv run hf-serve --model-id facebook/wav2vec2-large-960h --task automatic-speech-recognition --dtype float16
+```
+
+> [!WARNING]
+> On MacOS, if you installed `ffmpeg` via `brew`, you will need to set the following
+> environment variable in advance `DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib`
+>
+> Reference: https://github.com/pytorch/torchcodec/issues/570#issuecomment-2913609176
+
+And, then you can send a sample request as:
+
+```bash
+curl -L http://localhost:8080/predict \
+    -H "Content-Type: application/json" \
+    -d '{"inputs":"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac"}'
+```
+
+> [!WARNING]
+> Given the nature of some tasks that need to support JSON, forms, and files,
+> the `/predict` method for those is a redirect to the respective inner endpoint:
+> `/predict-json`, `/predict-form`, and `/predict-file`. Those are non-standard
+> but required to keep full compatibility with the current Hugging Face Inference
+> Endpoints API Specification, but in reality the redirect response (HTTP 307)
+> shouldn't be used as an routing route, but rather dedicated routes for those.
+
+> [!NOTE]
+> The OpenAI Audio Transcriptions API is still not yet part of `hf-serve` but it's
+> on the roadmap and it will be released soon, stay tuned!
+
+### 🔈 Run `facebook/wav2vec2-lv-60-espeak-cv-ft` (with `phonemizer` and `espeak`)
+
+> [!NOTE]
+> Some models as e.g. `facebook/wav2vec2-lv-60-espeak-cv-ft`, rely on `phonemizer` for
+> the "phonemization" of words and texts in many languages, based at the same time on
+> different Text-To-Speech (TTS) backends as e.g. `espeak-ng` which is supports a lot
+> of languages and IPA (International Phonetic Alphabet). This being said, such models
+> require custom dependencies that need to be installed beforehand as those don't come
+> as default `hf-serve` dependencies; whilst those can be installed as e.g. on MacOS:
+>
+> ```bash
+> brew install ffmpeg
+> brew install espeak
+> ```
+
+> [!WARNING]
+> Beware that when installing `ffmpeg` with `brew` on a specific
+> version as e.g. `brew install ffmpeg@7` as it will be installed as "keg-only",
+> meaning that it won't be symlinked into `/opt/homebrew`, meaning that the path
+> to the library won't be `/opt/homebrew/lib` but rather `/opt/homebrew/opt/ffmpeg/lib`
+> instead, meaning that on MacOS you'll need to set `DYLD_FALLBACK_LIBRARY_PATH`
+> to wherever the `ffmpeg` library is installed in.
+
+To run `facebook/wav2vec2-lv-60-espeak-cv-ft` on e.g. MacOS, you need to run the following:
+
+```bash
+DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps
+```
+
+Note that if you have installed another version of `ffmpeg` with `brew` as e.g. `brew install ffmpeg@7`, you should use the following command instead:
+
+```bash
+DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/opt/ffmpeg@7/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps
+```
+
+The main difference relies on the path used for `DYLD_FALLBACK_LIBRARY_PATH` which is now pointing to the exact `brew`-installed version of `ffmpeg` instead. More information on the compatibility issues with `ffmpeg`, `torchcodec` and `torch` at https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.

hf_serve-0.1.0/pyproject.toml

@@ -0,0 +1,136 @@
+[project]
+name = "hf-serve"
+version = "0.1.0"
+description = "Experimental Hugging Face API for Transformers, Diffusers and Sentence Transformers, written in Python."
+readme = "README.md"
+license = "Apache-2.0"
+authors = [
+  { name = "Alvaro Bartolome", email = "alvaro.bartolome@huggingface.co" },
+]
+maintainers = [
+  { name = "Alvaro Bartolome", email = "alvaro.bartolome@huggingface.co" },
+  { name = "Juan Julian Cea", email = "juan.moran@huggingface.co" },
+  { name = "Erik Kaunismäki", email = "erik.kaunismaki@huggingface.co" },
+]
+requires-python = ">=3.10,<3.13"
+dependencies = [
+    "uvicorn>=0.34.0",
+    "fastapi>=0.115.6",
+    # Floor pinned to the version that fixes CVE-2026-48710 (GHSA-86qp-5c8j-p5mr),
+    # Host header poisons request.url.path. The Starlette range FastAPI allows
+    # would otherwise let a fresh resolve drop back to vulnerable 0.x or 1.0.0.
+    "starlette>=1.0.1",
+    "prometheus-client>=0.21.1",
+    "torch",
+    "torchvision",
+    # NOTE: `torchcodec` as opposed to `torch` won't install the CUDA-compatible
+    # version by default but rather the CPU-only one.
+    # Reference: https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec
+    "torchcodec",
+    # NOTE: `diffusers` won't complain, but support for `transformers` 5.0.0+
+    # is still work in progress as per https://github.com/huggingface/diffusers/pull/12976
+    "diffusers>=0.37.0",
+    # NOTE: `torch-speech` is now `audio` in Transformers 5.0+ and `phonemizer`
+    # is included as a default dependency instead of an `audio` specific one.
+    # Anyway, `audio` models might need some system dependencies e.g. `espeak-ng`
+    "transformers[accelerate,audio,chat_template,kernels,timm,vision]>=5.9.0",
+    "pydantic>=2.10.5",
+    # NOTE: `optimum[onnxruntime]` seems to have a cap on `transformers<4.53.0`
+    # NOTE: `optimum` for ONNX recently migrated to `optimum-onnx` at https://github.com/huggingface/optimum-onnx
+    # but still has the cap on `transformers<4.56.0` which prevents us from relying on `main`
+    # "sentence-transformers[onnx]>=5.0.0",
+    # NOTE: for GPU that'd be the following
+    # "sentence-transformers[onnx-gpu]>=5.0.0",
+    "sentence-transformers>=5.5.1",
+    "sentencepiece>=0.2.0",
+    "protobuf>=5.29.3",
+    "peft>=0.14.0",
+    "pillow>=11.1.0",
+    # NOTE: `pydub` is required for `automatic-speech-recognition` to parse the
+    # audio bytes and calculate the audio duration in seconds
+    "pydub>=0.25.0",
+    "python-multipart>=0.0.20",
+    "python-magic>=0.4.27",
+    "pandas>=2.3.3",
+]
+
+[project.optional-dependencies]
+cpu = ["torch>=2.5.1,<2.8.0", "torchvision", "torchcodec<0.6"]
+cuda = ["torch>=2.5.1,<2.8.0", "torchvision", "torchcodec<0.6"]
+cuda-130 = ["torch>2.8.0", "torchvision", "torchcodec"]
+flash-attn = ["flash-attn==2.8.3"]
+
+# causal-1dconv = ["causal_conv1d==1.5.2"]
+# NOTE: `flash-linear-attention` has a lock on `transformers`<4.5.6.0
+# flash-linear-attention = ["flash-linear-attention==0.3.2"]
+
+# NOTE: useful for Intel `auto-round` quants, most likely can be installed on the fly if needed (?)
+# e.g. https://huggingface.co/Intel/Qwen3-Next-80B-A3B-Instruct-int4-mixed-AutoRound/tree/main
+# Reference: https://github.com/intel/auto-round
+# quantization = ["auto-round>=0.5.0", "gptqmodel==4.2.5", "bitsandbytes>=0.47.0"]
+
+[build-system]
+requires = ["uv_build>=0.9.26,<0.10.0"]
+build-backend = "uv_build"
+
+[project.scripts]
+hf-serve = "hf_serve.cli:main"
+
+[tool.ruff]
+line-length = 112
+
+[tool.ruff.lint]
+select = ["I"]
+
+[tool.ruff.lint.isort]
+force-single-line = false
+
+[tool.uv]
+conflicts = [
+    [
+      { extra = "cpu" },
+      { extra = "cuda" },
+      { extra = "cuda-130" },
+    ],
+]
+
+[tool.uv.sources]
+torch = [
+  { index = "pytorch-cpu", extra = "cpu" },
+  { index = "pytorch-cu126", extra = "cuda" },
+  { index = "pytorch-cu130", extra = "cuda-130" },
+]
+torchvision = [
+  { index = "pytorch-cpu", extra = "cpu" },
+  { index = "pytorch-cu126", extra = "cuda" },
+  { index = "pytorch-cu130", extra = "cuda-130" },
+]
+torchcodec = [
+  { index = "pytorch-cpu", extra = "cpu" },
+  { index = "pytorch-cu126", extra = "cuda" },
+  { index = "pytorch-cu130", extra = "cuda-130" },
+]
+
+[[tool.uv.index]]
+name = "pytorch-cpu"
+url = "https://download.pytorch.org/whl/cpu"
+explicit = true
+
+[[tool.uv.index]]
+name = "pytorch-cu126"
+url = "https://download.pytorch.org/whl/cu126"
+explicit = true
+
+[[tool.uv.index]]
+name = "pytorch-cu130"
+url = "https://download.pytorch.org/whl/cu130"
+explicit = true
+
+[tool.uv.extra-build-dependencies]
+flash-attn = [{ requirement = "torch", match-runtime = true }]
+# causal-1dconv = [{ requirement = "torch", match-runtime = true }]
+# flash-linear-attention = [{ requirement = "torch", match-runtime = true }]
+# gptqmodel = [{ requirement = "torch", match-runtime = true }]
+
+[tool.uv.extra-build-variables]
+flash-attn = { FLASH_ATTENTION_SKIP_CUDA_BUILD = "TRUE" }

hf_serve-0.1.0/src/hf_serve/cli.py

@@ -0,0 +1,141 @@
+import argparse
+import os
+from typing import get_args
+
+from hf_serve.cli_utils import strtobool
+from hf_serve.compatibility.backwards import ensure_backwards_compatibility
+from hf_serve.server import launch
+from hf_serve.types.task import TaskTypes
+
+# NOTE: required in order for the actual values for the environment variables
+# to be set before the defaults are provided to the `argparse` arguments
+ensure_backwards_compatibility()
+
+parser = argparse.ArgumentParser(description="Hugging Face Serve API")
+
+parser.add_argument(
+    "--host",
+    type=str,
+    default=os.getenv("HOST", None) or "0.0.0.0",
+    required=False,
+    help="The host into which the FastAPI API will be deployed to, defaults to 0.0.0.0. It can also be set via the environment variable `HOST`.",
+)
+
+parser.add_argument(
+    "--port",
+    type=int,
+    default=os.getenv("PORT", None) or 8080,
+    required=False,
+    help="The port in which the FastAPI API will listen to, defaults to 8080. It can also be set via the environment variable `PORT`.",
+)
+
+# NOTE: only one of `--model-id` or `--model-dir` should be provided
+parser.add_argument(
+    "--model-id",
+    type=str,
+    default=os.getenv("MODEL_ID", None),
+    help="The model ID on the Hugging Face Hub. It can also be set via the environment variable `MODEL_ID`.",
+)
+
+parser.add_argument(
+    "--model-dir",
+    type=str,
+    default=os.getenv("MODEL_DIR", None),
+    help="A local directory that contains a Hugging Face compatible model. It can also be set via the environment variable `MODEL_DIR`.",
+)
+
+parser.add_argument(
+    "--revision",
+    type=str,
+    default=os.getenv("REVISION", None) or None,
+    help="The revision of model on the Hugging Face Hub. It can also be set via the environment variable `REVISION`.",
+)
+
+parser.add_argument(
+    "--task",
+    type=str,
+    default=os.getenv("TASK", None),
+    choices=get_args(TaskTypes),
+    help="Any of the supported tasks for either Transformers, Diffusers, or Sentence Transformers. It can also be set via the environment variable `TASK`.",
+)
+
+parser.add_argument(
+    "--device",
+    type=str,
+    default=os.getenv("DEVICE", None) or "auto",
+    choices=["auto", "balanced", "cuda", "cpu", "mps"],
+    required=False,
+    help="The device on which the model weights will be loaded into, defaults to auto that selects an accelerator if available, otherwise it falls back to the CPU. It can also be set via the environment variable `DEVICE`.",
+)
+
+parser.add_argument(
+    "--dtype",
+    type=str,
+    # NOTE: This might seem weird, but if `DTYPE=""` then it won't be None,
+    # hence its value will be `""` but since we don't want that, we need to add
+    # the check `or None` to make sure that if `""` is set, then we default to
+    # `None`
+    default=os.getenv("DTYPE", None) or None,
+    choices=["float32", "float16", "bfloat16", "float8", "int8", "int4"],
+    required=False,
+    help="The PyTorch dtype in which the model weights will be loaded, defaults to None meaning that the default dtype for the given model will be used i.e., the dtype in which the model weights are available. It can also be set via the environment variable `DTYPE`.",
+)
+
+parser.add_argument(
+    "--trust-remote-code",
+    # TODO(alvarobartt): Eventually review this, as I'm not super happy with this line here
+    default=strtobool(os.getenv("TRUST_REMOTE_CODE", None)) or False,
+    action="store_true",
+    required=False,
+    help="Whether to trust remote code execution from the Hugging Face Hub, more information at https://huggingface.co/docs/transformers/v4.57.3/en/models#custom-models. Defaults to False, as it's not recommended unless required.",
+)
+
+# TODO(juanjucm): validate accepted_mimetypes values based on the task.
+# Check processor's accepted file formats (e.g. ffmpeg for audio (https://www.ffmpeg.org/general.html#File-Formats))
+parser.add_argument(
+    "--accepted-mimetypes",
+    type=str,
+    default=os.getenv("ACCEPTED_MIMETYPES", None) or None,
+    required=False,
+    help="A comma-separated list of accepted MIME types for file uploads. By default, each task will have all valid MIME types (e.g. audio/* for audio tasks, image/* for image tasks). It can also be set via the environment variable `ACCEPTED_MIMETYPES`.",
+)
+
+parser.add_argument(
+    "--max-file-size",
+    type=int,
+    # TODO: Maybe remove the `or None` and instead of checking that `if max_file_size is not None`
+    # we could just `if not max_file_size` (which handles `""`)
+    default=os.getenv("MAX_FILE_SIZE", None) or None,
+    required=False,
+    help="The maximum file size in bytes for file uploads (e.g 10485760 for 10MB). By default, no file size limit is considered. It can also be set via the environment variable `MAX_FILE_SIZE`.",
+)
+
+parser.add_argument(
+    "--cloud",
+    type=str,
+    default=os.getenv("CLOUD", None) or None,
+    choices=["azure", "google"],
+    required=False,
+    help="To be defined when deploying on a cloud provider to ensure that it's compatible with the provider expectations e.g. `/score` route needs to be exposed for Azure AI Foundry and Azure ML deployments (among others); or e.g. `instances` needs to be a list of inputs for Vertex AI (among others).",
+)
+
+
+def main() -> None:
+    args = parser.parse_args()
+
+    launch(
+        host=args.host,
+        port=args.port,
+        model_id=args.model_id,
+        model_dir=args.model_dir,
+        revision=args.revision,
+        task=args.task,
+        device=args.device,
+        dtype=args.dtype,
+        trust_remote_code=args.trust_remote_code,
+        # TODO(juanjucm): This can most likely be a list, and it will automatically be formatted this way
+        # without having to handle that here
+        accepted_mimetypes=args.accepted_mimetypes.split(",") if args.accepted_mimetypes else None,
+        max_file_size=args.max_file_size,
+        cloud=args.cloud,
+    )

hf_serve-0.1.0/src/hf_serve/cli_utils.py

@@ -0,0 +1,5 @@
+from typing import Optional
+
+
+def strtobool(value: Optional[str]) -> bool:
+    return True if value is not None and value.lower() in {"y", "yes", "t", "true", "on", "1"} else False