floudsonnx 1.0.0__tar.gz

floudsonnx-1.0.0/LICENSE

@@ -0,0 +1,17 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+Copyright (c) 2026 Goutam Malakar
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

floudsonnx-1.0.0/PKG-INFO

@@ -0,0 +1,531 @@
+Metadata-Version: 2.4
+Name: floudsonnx
+Version: 1.0.0
+Summary: Ollama-style ONNX model store and runtime - pull, cache, and serve ORT sessions
+Author: Goutam Malakar
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/gmalakar/floudsonnx
+Project-URL: Issues, https://github.com/gmalakar/floudsonnx/issues
+Keywords: onnx,onnxruntime,model-serving,huggingface,inference,flouds
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: <3.13,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: onnxruntime>=1.20.1
+Requires-Dist: transformers<4.58.0,>=4.44.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: numpy<1.27,>=1.26.4
+Provides-Extra: export
+Requires-Dist: flouds-model-exporter>=1.0.1; extra == "export"
+Provides-Extra: seq2seq
+Requires-Dist: optimum[onnxruntime]>=1.22.0; extra == "seq2seq"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110.0; extra == "server"
+Requires-Dist: uvicorn[standard]>=0.29.0; extra == "server"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
+Requires-Dist: pre-commit==4.5.1; extra == "dev"
+Requires-Dist: black==24.1.1; extra == "dev"
+Requires-Dist: isort==5.13.2; extra == "dev"
+Requires-Dist: flake8==7.0.0; extra == "dev"
+Requires-Dist: flake8-bugbear; extra == "dev"
+Requires-Dist: flake8-comprehensions; extra == "dev"
+Requires-Dist: mypy==1.8.0; extra == "dev"
+Requires-Dist: pyright==1.1.410; extra == "dev"
+Requires-Dist: bandit==1.7.5; extra == "dev"
+Requires-Dist: pbr; extra == "dev"
+Provides-Extra: all
+Requires-Dist: floudsonnx[export,seq2seq,server]; extra == "all"
+Dynamic: license-file
+
+# floudsonnx
+
+[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12-blue.svg)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)
+
+`floudsonnx` is a small ONNX model store and runtime for Python. It can pull
+Hugging Face models into a local ONNX store, cache runtime sessions, and load
+models as either `onnxruntime.InferenceSession` or Optimum
+`ORTModelForSeq2SeqLM` objects.
+
+The package is designed for applications that want an Ollama-like local model
+store, but for ONNX artifacts.
+
+## Important: Hugging Face Models Must Be Converted To ONNX
+
+`floudsonnx` runs ONNX models. Hugging Face models must be exported to ONNX
+before they can be loaded as runtime sessions.
+
+Install the `export` extra when you want `floudsonnx` to convert Hugging Face
+models automatically:
+
+```bash
+pip install "floudsonnx[export]"
+```
+
+That extra installs `flouds-model-exporter`, which is required for converting
+Hugging Face models to ONNX. Without it, `floudsonnx` can still load models
+that already exist in the local ONNX store, but it cannot pull/export new
+Hugging Face models.
+
+For private or gated Hugging Face models, provide a token with the standard
+Hugging Face Hub variable:
+
+```bash
+export HUGGINGFACE_HUB_TOKEN="hf_xxx_your_token"
+```
+
+`HUGGINGFACE_HUB_TOKEN` is read by the Hugging Face/exporter stack during
+pull/export.
+
+You can also pass a token per call:
+
+```python
+from floudsonnx import pull
+
+pull("org/private-model", model_for="fe", hf_token="hf_xxx_your_token")
+```
+
+Never commit real Hugging Face tokens to source control.
+
+## Features
+
+- Local model store under `~/.flouds/models` by default.
+- Auto-export from Hugging Face through the optional `flouds-model-exporter`
+  integration.
+- Runtime loading for feature extraction, sequence classification, ranker,
+  seq2seq, and LLM-style model categories.
+- Thread-safe session caching for ONNX Runtime and seq2seq models.
+- Tokenizer loading and caching.
+- Python API, CLI, and optional FastAPI server.
+- Manifest files for locally stored models.
+
+## Installation
+
+`floudsonnx` supports Python `3.11` and `3.12`.
+
+```bash
+pip install floudsonnx
+```
+
+The core install loads models that are already exported to ONNX. Install extras
+for export, seq2seq, or server support:
+
+```bash
+pip install "floudsonnx[export]"          # auto-export via flouds-model-exporter
+pip install "floudsonnx[seq2seq]"         # Optimum ORTModelForSeq2SeqLM support
+pip install "floudsonnx[server]"          # FastAPI + Uvicorn HTTP server
+pip install "floudsonnx[all]"             # export + seq2seq + server
+```
+
+Core dependencies:
+
+- `onnxruntime`
+- `transformers`
+- `pydantic`
+- `numpy`
+
+Optional extras:
+
+- `export`: `flouds-model-exporter>=1.0.1`
+- `seq2seq`: `optimum[onnxruntime]`
+- `server`: `fastapi`, `uvicorn[standard]`
+
+For feature-extraction, classification, and ranker exports, prefer
+`library="transformers"` to avoid `sentence-transformers` auto-detection in the
+exporter.
+
+### Store Location And `ONNX_PATH`
+
+`floudsonnx` stores models under `~/.flouds/models` by default. Override the
+store root with `FloudsOnnxSettings`:
+
+```python
+from floudsonnx import FloudsOnnxClient, FloudsOnnxSettings
+
+client = FloudsOnnxClient(
+    FloudsOnnxSettings(onnx_path="/path/to/onnx/models")
+)
+```
+
+When `floudsonnx` calls `flouds-model-exporter`, it sets the raw `ONNX_PATH`
+environment variable internally so exported files land in the selected
+`floudsonnx` store.
+
+If you run `flouds-model-exporter` directly, use its native `ONNX_PATH`
+variable:
+
+```bash
+export ONNX_PATH="/path/to/onnx/models"
+flouds-export export --model-name t5-small --model-for s2s --task seq2seq-lm
+```
+
+## Quickstart
+
+This example pulls a feature-extraction model, loads it, tokenizes input text,
+builds the ONNX Runtime input feed from the session's actual inputs, and runs
+inference.
+
+```python
+import numpy as np
+
+from floudsonnx import create_model
+
+model = create_model(
+    "sentence-transformers/all-MiniLM-L6-v2",
+    model_for="fe",
+    task="feature-extraction",
+    library="transformers",
+    normalize_embeddings=True,
+)
+
+encoded = model.tokenizer(
+    ["Hello world"],
+    return_tensors="np",
+    padding=True,
+    truncation=True,
+    max_length=64,
+)
+
+session_inputs = {item.name for item in model.session.get_inputs()}
+feed = {name: encoded[name].astype(np.int64) for name in session_inputs if name in encoded}
+
+# Some exported encoder models require token_type_ids even when the tokenizer
+# does not return them.
+for missing_name in session_inputs - set(feed):
+    feed[missing_name] = np.zeros_like(next(iter(feed.values())))
+
+outputs = model.run(None, feed)
+print(outputs[0].shape)
+```
+
+The `create_model()` helper pulls/exports the model if needed, then loads it.
+Use `load_model()` when the model is already present on disk and you do not
+want auto-export.
+
+## Seq2Seq Example
+
+Seq2seq loading requires the `seq2seq` extra. Pulling from Hugging Face also
+requires the `export` extra.
+
+```python
+from floudsonnx import create_model
+
+model = create_model("t5-small", model_for="s2s", task="seq2seq-lm")
+
+encoded = model.tokenizer(
+    ["summarize: The quick brown fox jumps over the lazy dog."],
+    return_tensors="pt",
+    truncation=True,
+    max_length=64,
+)
+
+tokens = model.seq2seq_model.generate(
+    input_ids=encoded["input_ids"],
+    max_new_tokens=32,
+)
+
+print(model.tokenizer.decode(tokens[0], skip_special_tokens=True))
+```
+
+## Model Types
+
+| `model_for` | Default export task | Runtime strategy |
+|---|---|---|
+| `fe` | `feature-extraction` | `onnxruntime.InferenceSession` |
+| `sc` | `text-classification` | `onnxruntime.InferenceSession` |
+| `ranker` | `text-classification` | `onnxruntime.InferenceSession` |
+| `s2s` | `seq2seq-lm` | `ORTModelForSeq2SeqLM` |
+| `llm` | `text-generation-with-past` | `onnxruntime.InferenceSession` by default, or `ORTModelForSeq2SeqLM` when configured with `use_seq2seqlm=True` |
+
+## Python API
+
+Top-level convenience functions:
+
+```python
+from floudsonnx import (
+    create_model,
+    list_models,
+    load_model,
+    pull,
+    remove_model,
+)
+```
+
+Available top-level functions:
+
+| Function | Description |
+|---|---|
+| `create_model(model_name, model_for="fe", **kwargs)` | Pull/export if needed, then load the model. |
+| `load_model(model_name, model_for="fe")` | Load an existing local model without auto-export. |
+| `pull(model_name, model_for="fe", **kwargs)` | Export/store the model and return its manifest. |
+| `list_models()` | Return local `ModelManifest` objects. |
+| `remove_model(model_name, model_for="fe")` | Delete the local model and evict cached sessions. |
+
+For explicit settings and cache control, use `FloudsOnnxClient`:
+
+```python
+from floudsonnx import FloudsOnnxClient, FloudsOnnxSettings
+
+settings = FloudsOnnxSettings(
+    home_dir="/data/flouds",
+    session_provider="CPUExecutionProvider",
+)
+client = FloudsOnnxClient(settings)
+
+manifest = client.pull(
+    "BAAI/bge-base-en-v1.5",
+    model_for="fe",
+    task="feature-extraction",
+    library="transformers",
+    normalize_embeddings=True,
+)
+
+model = client.load_model("BAAI/bge-base-en-v1.5", model_for="fe")
+print(model.session_strategy)
+
+client.unload("BAAI/bge-base-en-v1.5", model_for="fe")
+client.remove("BAAI/bge-base-en-v1.5", model_for="fe")
+```
+
+`FloudsOnnxClient` methods:
+
+- `pull(...)`
+- `list()`
+- `remove(model_name, model_for="fe")`
+- `create_model(...)`
+- `load_model(model_name, model_for="fe")`
+- `reload(model_name, model_for="fe")`
+- `unload(model_name, model_for="fe")`
+- `is_loaded(model_name, model_for="fe")`
+- `cache_stats()`
+
+## LoadedModel
+
+`create_model()` and `load_model()` return a `LoadedModel` with:
+
+- `model_name`
+- `model_for`
+- `model_dir`
+- `config`
+- `tokenizer`
+- `session_strategy`
+- `session` for `onnxruntime.InferenceSession` models
+- `seq2seq_model` for `ORTModelForSeq2SeqLM` models
+- `is_seq2seq`
+- `run(output_names, input_feed, run_options=None)`
+
+For encoder/classification/ranker models, call `model.run(...)`.
+
+For seq2seq models, call `model.seq2seq_model.generate(...)`.
+
+## CLI
+
+```bash
+floudsonnx --help
+```
+
+Commands:
+
+```bash
+# Pull/export a model to the local store
+floudsonnx pull sentence-transformers/all-MiniLM-L6-v2 --for fe --task feature-extraction
+
+# Disable ONNX optimization during pull
+floudsonnx pull t5-small --for s2s --task seq2seq-lm --no-optimize
+
+# List locally stored models
+floudsonnx list
+
+# Show manifest JSON
+floudsonnx info sentence-transformers/all-MiniLM-L6-v2 --for fe
+
+# Remove from local store
+floudsonnx remove sentence-transformers/all-MiniLM-L6-v2 --for fe
+
+# Evict and reload from disk
+floudsonnx reload sentence-transformers/all-MiniLM-L6-v2 --for fe
+
+# Show session cache stats
+floudsonnx stats
+
+# Start the optional HTTP server
+floudsonnx serve --host 127.0.0.1 --port 19720
+```
+
+Current `pull` CLI options:
+
+- `--for`
+- `--task`
+- `--optimize`
+- `--no-optimize`
+- `--optimization-level`
+- `--opset-version`
+- `--device`
+- `--framework`
+- `--library`
+- `--normalize-embeddings`
+- `--force`
+- `--trust-remote-code`
+- `--use-external-data-format`
+- `--use-subprocess`
+- `--use-fallback-if-failed`
+- `--merge`
+- `--skip-validator`
+- `--hf-token`
+
+Example:
+
+```bash
+floudsonnx pull sentence-transformers/all-MiniLM-L6-v2 \
+  --for fe \
+  --task feature-extraction \
+  --library transformers \
+  --normalize-embeddings
+```
+
+## Optional HTTP Server
+
+Install the server extra:
+
+```bash
+pip install "floudsonnx[server]"
+```
+
+Start the server:
+
+```bash
+floudsonnx serve --host 127.0.0.1 --port 19720
+```
+
+Routes:
+
+| Method | Path | Description |
+|---|---|---|
+| `GET` | `/health` | Health check. |
+| `GET` | `/api/v1/models` | List local model manifests. |
+| `GET` | `/api/v1/models/{name}?model_for=fe` | Get one manifest. |
+| `POST` | `/api/v1/models/pull` | Pull/export a model. |
+| `POST` | `/api/v1/models/load` | Load a local model into cache. |
+| `POST` | `/api/v1/models/reload` | Evict and reload a model. |
+| `POST` | `/api/v1/models/unload` | Evict a model from memory. |
+| `DELETE` | `/api/v1/models/{name}?model_for=fe` | Remove a model from disk. |
+| `GET` | `/api/v1/stats` | Return cache statistics. |
+
+Pull request body:
+
+```json
+{
+  "model_name": "sentence-transformers/all-MiniLM-L6-v2",
+  "model_for": "fe",
+  "task": "feature-extraction",
+  "force": false,
+  "optimize": false,
+  "trust_remote_code": false,
+  "use_external_data_format": false,
+  "use_fallback_if_failed": false,
+  "hf_token": null
+}
+```
+
+Load/reload/unload request body:
+
+```json
+{
+  "model_name": "sentence-transformers/all-MiniLM-L6-v2",
+  "model_for": "fe"
+}
+```
+
+## Configuration
+
+Configure `floudsonnx` with `FloudsOnnxSettings` when creating a client:
+
+```python
+from floudsonnx import FloudsOnnxClient, FloudsOnnxSettings
+
+client = FloudsOnnxClient(
+    FloudsOnnxSettings(
+        home_dir="/data/flouds",
+        onnx_path="/data/flouds/models",
+        session_provider="CPUExecutionProvider",
+        encoder_cache_max=5,
+        decoder_cache_max=5,
+        seq2seq_cache_max=3,
+    )
+)
+```
+
+Additional variables:
+
+| Environment variable | Used by | Description |
+|---|---|---|
+| `HUGGINGFACE_HUB_TOKEN` | Hugging Face Hub / `flouds-model-exporter` | Standard token variable for private or gated Hugging Face models. |
+| `ONNX_PATH` | `flouds-model-exporter` | Native exporter output root. `floudsonnx` sets this internally during export; configure the `floudsonnx` store with `FloudsOnnxSettings`. |
+
+## Local Store Layout
+
+Default layout:
+
+```text
+~/.flouds/
+`-- models/
+    |-- fe/
+    |   `-- all-MiniLM-L6-v2/
+    |       |-- model.onnx
+    |       |-- model_optimized.onnx
+    |       |-- tokenizer.json
+    |       `-- manifest.json
+    |-- s2s/
+    |-- sc/
+    |-- ranker/
+    `-- llm/
+```
+
+Each model directory contains a `manifest.json` with the model name, model
+type, export options, selected runtime strategy, discovered ONNX files, and
+model config.
+
+## Development
+
+```bash
+pip install -r requirements-dev.txt
+pip install -e ".[export,seq2seq,server]"
+pre-commit install
+pytest tests/unit -v
+```
+
+Useful checks:
+
+```bash
+python tools/check_dependency_sync.py
+black --check src tests tools
+isort --check-only src tests tools
+flake8 src tests tools
+mypy src/floudsonnx
+pyright src/floudsonnx
+python -m build
+twine check dist/*
+```
+
+Integration tests download and export real models:
+
+```bash
+pytest tests/integration -m integration -v
+```
+
+## Release
+
+Releases are tag-driven through GitHub Actions. See
+[`docs/RELEASE_PROCESS.md`](docs/RELEASE_PROCESS.md).
+
+## License
+
+Apache-2.0. See [`LICENSE`](LICENSE).