hutash-inference 0.2.5__tar.gz

hutash_inference-0.2.5/PKG-INFO

@@ -0,0 +1,13 @@
+Metadata-Version: 2.4
+Name: hutash-inference
+Version: 0.2.5
+Summary: Shared HTTP inference framework for Hutash models
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/Appsork/hutash
+Requires-Python: >=3.10
+Requires-Dist: fastapi
+Requires-Dist: uvicorn
+Requires-Dist: python-multipart
+Requires-Dist: accelerate>=0.28.0
+Provides-Extra: image
+Requires-Dist: pillow>=10.0.0; extra == "image"

hutash_inference-0.2.5/README.md

@@ -0,0 +1,57 @@
+# hutash_inference
+
+Shared inference framework for Hutash model packages. A model package
+supplies `manifest.yaml` and an `inference.py` with a `load()` and one
+method per declared capability; this package supplies everything else —
+FastAPI app, routing, input parsing, output encoding, park/unpark, and
+the hardware workarounds below.
+
+## Hardware notes
+
+### GTX 16xx (compute capability 7.5) — fp16 convolution
+
+GTX 16xx (compute capability 7.5) has a known fp16 convolution defect in
+cuDNN. hutash_inference auto-disables cuDNN on these cards. This affects
+image and video model generation only. LLMs and audio models are
+unaffected.
+
+The failure is silent rather than loud: cuDNN's half-precision
+convolution returns NaN, the NaN propagates through the UNet into the
+latents, the VAE decodes NaN to zeros, and the model answers `200 OK`
+with a completely black image. fp16 *matmul* on the same card is correct,
+which is why text and audio stacks never notice.
+
+Measured on a GTX 1650, one 320×64×64 convolution:
+
+| Path | absmax |
+|------|--------|
+| fp16 conv | `nan` |
+| fp32 conv | 244.98 |
+| fp16 conv, cuDNN off | 245.00 |
+
+The last two agreeing is the reason for the fix: the arithmetic is
+correct and only cuDNN's kernel is wrong, so disabling cuDNN restores
+fp16 rather than giving precision up. Forcing fp32 would also work
+numerically but doubles weight residency — roughly 3.4 GB for a Stable
+Diffusion 1.5 UNet alone — which does not fit the 4 GB cards this
+mostly affects.
+
+Applied once per process from `Inference.__init__`, before any model's
+`load()` runs, since it changes how every convolution afterwards is
+computed. See `_apply_cudnn_fp16_workaround` in `base.py`.
+
+**Known false positive.** The check is compute capability `(7, 5)`
+exactly, which also matches the RTX 20 series (2060/2070/2080). Those
+cards have tensor cores and correct fp16, so they lose cuDNN's faster
+convolution kernels for no reason — a speed cost on a healthy card,
+never a wrong result. Narrowing it means probing one convolution for
+NaN at startup instead of reading a capability tuple.
+
+## Low VRAM mode
+
+When the engine starts a model with a VRAM budget it sets
+`HUTASH_VRAM_BUDGET_MB` and `HUTASH_MODEL_FRAMEWORK` in the model
+process. `server.py` translates the budget into whatever knob the
+framework understands — `device_map`/`max_memory` for HuggingFace,
+`n_gpu_layers` for llama.cpp — so a model's `inference.py` needs no
+Low-VRAM code of its own.

hutash_inference-0.2.5/__init__.py

@@ -0,0 +1,65 @@
+"""Aura Inference Library
+
+Shared framework for Aura model containers. Provides:
+- Inference base class that all model inference classes inherit from
+- @capability decorator for marking methods as HTTP endpoint handlers
+- FastAPI server boilerplate
+- I/O helpers (audio, image, video, file) — `encode_image` for image output,
+  `sniff_audio_suffix` / `sniff_image_format` / `sniff_video_format` for
+  what arrived
+- Validation (model.yaml / manifest.json <-> inference.py compatibility)
+
+See docs/strategy/model-ssot-architecture.md for architecture details.
+"""
+
+from hutash_inference.base import (
+    IMAGE_FORMATS,
+    IMAGE_MEDIA_TYPES,
+    VIDEO_FORMATS,
+    VIDEO_INPUT_FORMATS,
+    VIDEO_MEDIA_TYPES,
+    Inference,
+    capability,
+    encode_image,
+    resolve_local_weights_dir,
+    sniff_audio_suffix,
+    sniff_image_format,
+    sniff_video_format,
+)
+from hutash_inference.errors import (
+    HutashInferenceError,
+    ValidationError,
+    ModelLoadError,
+    GenerationError,
+)
+
+# Kept in step with pyproject.toml BY HAND. 0.2.3 is UNRELEASED: PyPI serves
+# 0.2.2, and every model pipeline installs `hutash-inference>=0.2.2` from PyPI,
+# so a container runs the published release and not this tree. The source being
+# ahead of the release is normal.
+#
+# This directory is the ONE tree of record as of 2026-09-07. It was previously
+# two: an in-app copy under hutash-app/ that carried the image and video work,
+# and this one, which carried `sniff_audio_suffix` and nothing else new. They
+# were merged here and the in-app copy deleted, so publishing from anywhere
+# else is no longer possible.
+__version__ = "0.2.5"
+
+__all__ = [
+    "Inference",
+    "capability",
+    "resolve_local_weights_dir",
+    "encode_image",
+    "sniff_audio_suffix",
+    "sniff_image_format",
+    "sniff_video_format",
+    "IMAGE_FORMATS",
+    "IMAGE_MEDIA_TYPES",
+    "VIDEO_FORMATS",
+    "VIDEO_INPUT_FORMATS",
+    "VIDEO_MEDIA_TYPES",
+    "HutashInferenceError",
+    "ValidationError",
+    "ModelLoadError",
+    "GenerationError",
+]