@hypit/hypit 0.1.14 → 0.2.0

package/services/whisperx/README.md

@@ -19,7 +19,9 @@ semantic projection combines this evidence with one explicit Script Segment late
 ## Install
 
 For an ordinary installed Distribution, select the local WhisperX Endpoint and run
-`hypit programs up --endpoint <instance>`. The Runtime creates or reconciles the cold environment in
+`hypit programs prepare --endpoint <instance>` after setting `alignmentLanguages` in its Profile.
+`hypit programs up --endpoint <instance>` also prepares resources and starts the service.
+The Runtime creates or reconciles the cold environment in
 the machine Program Home and reuses a running service across projects and sessions. The commands below are contributor/operator
 diagnostics for a deliberately managed deployment:
 
@@ -28,20 +30,28 @@ WhisperX 3.8.6 supports Python 3.10 through 3.13. The checked-in lock selects Py
 ```bash
 uv python install 3.13
 uv sync --project services/whisperx --frozen
-uv run --project services/whisperx --frozen hypit-whisperx-prepare
-uv run --project services/whisperx --frozen hypit-whisperx-check
+HYPIT_WHISPERX_ALIGNMENT_LANGUAGES="zh en" services/whisperx/.venv/bin/hypit-whisperx-prepare
+HYPIT_WHISPERX_ALIGNMENT_LANGUAGES="zh en" services/whisperx/.venv/bin/hypit-whisperx-check --models
 ```
 
-The first model start may download ASR weights; first use of a language can download its alignment
-model during the request. Production should put the relevant
-Hugging Face cache on persistent storage. `hypit-whisperx-prepare` separately installs NLTK's
-`punkt_tab` sentence data through NLTK's own downloader. This resource is required by WhisperX
-alignment and is prepared explicitly before the warm service starts, never inside an inference request.
+`hypit-whisperx-prepare` explicitly prepares the selected ASR model, language alignment weights and
+NLTK sentence data. `hypit-whisperx-check --models` reads those resources without downloading.
+The service loads prepared resources only. Hugging Face calls use local-files-only APIs; the
+pinned WhisperX torchaudio and NLTK branches have no equivalent switch, so this dedicated inference
+process replaces their downloader entry points with an error. Preparation runs in a separate process
+and retains their native downloaders. These adaptations belong here, not in Runtime or Core.
+The default Pyannote VAD checkpoint ships inside the pinned WhisperX wheel.
+
+Native HF and torch caches are reused by default. An explicit model cache root selects its
+`huggingface/` and `torch/` subdirectories for both preparation and inference. No receipts, hashes or
+parallel resource inventory are maintained. Resource checks read model/tokenizer files and the
+selected language's sentence data. The example shell paths are POSIX; Windows environments use
+`.venv\Scripts\<command>.exe`.
 
 ## Run
 
 ```bash
-uv run --project services/whisperx --frozen hypit-whisperx-service
+services/whisperx/.venv/bin/hypit-whisperx-service
 curl http://127.0.0.1:8765/health
 ```
 
@@ -65,7 +75,9 @@ Configuration is deployment state:
 | `HYPIT_WHISPERX_COMPUTE` | `int8` on CPU | CTranslate2 compute type |
 | `HYPIT_WHISPERX_BATCH_SIZE` | `8` | bounded ASR batch size |
 | `HYPIT_WHISPERX_INPUT_ROOTS` | OS temp directory | path-separated roots the service may read |
-| `HYPIT_WHISPERX_NLTK_DATA` | user SVML cache | prepared, identity-checked NLTK data root |
+| `HYPIT_WHISPERX_NLTK_DATA` | user Hypit cache | selected prepared NLTK data root |
+| `HYPIT_WHISPERX_ALIGNMENT_LANGUAGES` | empty | space-separated languages to prepare; no implicit downloads for others |
+| `HYPIT_WHISPERX_MODEL_CACHE` | upstream caches | optional root for HF and torch model resources |
 | `HYPIT_WHISPERX_MAX_REQUEST_BYTES` | `65536` | HTTP JSON bound |
 | `HYPIT_WHISPERX_MAX_AUDIO_BYTES` | `536870912` | staged canonical WAV bound |
 
@@ -74,9 +86,30 @@ WhisperX version. A mismatch fails before transcription results are accepted.
 
 The service logs ASR loading, transcription, language-model loading and word alignment where those
 operations run. Completion entries include elapsed times. A loading entry means the library call
-has begun and may include a weight download; the download client supplies any transfer progress.
+has begun reading prepared local resources; downloads appear only in preparation logs.
 Transcripts and audio content are not included in these service progress entries. `/health` answers
-after ASR loading, so a healthy service may still prepare a language model on its first request.
+after ASR loading; it does not report all language caches as ready. A first request can load a
+prepared aligner into memory, but cannot download it.
+
+## Preparing sentence data through a proxy
+
+NLTK's downloader refuses a proxied request unless the operator explicitly trusts that proxy:
+it cannot enforce its direct-connection address checks through a proxy. This can stop preparation
+while fetching the NLTK data index, even when uv and Hugging Face downloads work.
+
+For a proxy you trust, scope NLTK's native opt-in to the explicit preparation command:
+
+```bash
+NLTK_ALLOW_PROXIED_URLOPEN=1 hypit programs prepare --endpoint whisperx.local
+```
+
+Use your selected Endpoint instance id. On PowerShell, set `$env:NLTK_ALLOW_PROXIED_URLOPEN = "1"`
+for that preparation session, then restore its previous value. The preparation process already
+inherits its caller's environment; no Hypit-specific proxy flag is needed. This setting authorizes
+NLTK to use the configured proxy; it neither selects a mirror nor enables inference downloads.
+Without that trust choice, use a deliberately selected direct route or supply the sentence data
+in the selected NLTK data directory before preparation. Hypit does not change the choice automatically.
+Existing usable sentence data is read locally without refreshing the download index.
 
 ## Package preparation
 

package/services/whisperx/src/hypit_whisperx_service/application.py

@@ -8,6 +8,7 @@ from . import PROTOCOL, SERVICE_VERSION
 from .audio import AudioInputError, CanonicalAudio, read_canonical_audio
 from .config import ServiceConfig
 from .engine import InferenceBusyError, InferenceInputError, normalize_language
+from .resources import UnpreparedResourceError
 
 
 class AlignmentEngine(Protocol):
@@ -82,6 +83,8 @@ class WhisperXApplication:
             raise RequestError(400, "INVALID_INPUT", str(error)) from error
         except InferenceBusyError as error:
             raise RequestError(503, "BUSY", str(error)) from error
+        except UnpreparedResourceError as error:
+            raise RequestError(503, "RESOURCE_NOT_PREPARED", str(error)) from error
         return _json_response(200, result)
 
     def error(self, error: RequestError) -> ApplicationResponse:

package/services/whisperx/src/hypit_whisperx_service/check.py

@@ -3,6 +3,7 @@ from __future__ import annotations
 import importlib.metadata as metadata
 import json
 import sys
+import argparse
 
 from . import PROTOCOL, SERVICE_VERSION
 from .config import ServiceConfig
@@ -12,6 +13,9 @@ from .resources import assert_punkt_tab
 def main() -> None:
     if not ((3, 10) <= sys.version_info[:2] < (3, 14)):
         raise RuntimeError(f"WhisperX service requires Python >=3.10,<3.14; got {sys.version.split()[0]}")
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--models", action="store_true", help="Check selected cached models without downloading")
+    arguments = parser.parse_args()
     config = ServiceConfig.from_environment()
     assert_punkt_tab(config.nltk_data_root)
     import numpy  # noqa: F401
@@ -26,6 +30,9 @@ def main() -> None:
     }
     if versions["whisperx"] != "3.8.6":
         raise RuntimeError(f"expected whisperx 3.8.6, got {versions['whisperx']}")
+    if arguments.models:
+        from .models import check_models
+        check_models(config)
     print(json.dumps({
         "ok": True,
         "protocol": PROTOCOL,

package/services/whisperx/src/hypit_whisperx_service/config.py

@@ -31,6 +31,8 @@ class ServiceConfig:
     nltk_data_root: Path
     max_request_bytes: int
     max_audio_bytes: int
+    model_cache: Path | None
+    alignment_languages: tuple[str, ...]
 
     @classmethod
     def from_environment(cls, environment: Mapping[str, str] | None = None) -> "ServiceConfig":
@@ -57,7 +59,12 @@ class ServiceConfig:
         if not roots:
             raise ValueError("HYPIT_WHISPERX_INPUT_ROOTS must contain at least one path")
 
+        languages = tuple(dict.fromkeys(env.get("HYPIT_WHISPERX_ALIGNMENT_LANGUAGES", "").split()))
+        if any(not language.isascii() or not language.isalpha() or language != language.lower() for language in languages):
+            raise ValueError("HYPIT_WHISPERX_ALIGNMENT_LANGUAGES must contain lowercase language codes")
         return cls(
+            model_cache=Path(env["HYPIT_WHISPERX_MODEL_CACHE"]).expanduser().resolve() if env.get("HYPIT_WHISPERX_MODEL_CACHE") else None,
+            alignment_languages=languages,
             port=_positive_integer(env.get("HYPIT_WHISPERX_PORT", "8765"), "HYPIT_WHISPERX_PORT", 65535),
             model=model,
             device=device,

package/services/whisperx/src/hypit_whisperx_service/engine.py

@@ -10,7 +10,8 @@ from typing import Any, Mapping
 
 from .audio import CanonicalAudio
 from .config import ServiceConfig
-from .resources import assert_punkt_tab
+from .resources import assert_punkt_tab, assert_sentence_data
+from .models import asr_path, alignment_path, alignment_selection, hf_cache, torch_cache, use_local_resources_only
 
 logger = logging.getLogger("hypit.whisperx")
 
@@ -91,6 +92,7 @@ def normalize_alignment(language: str, raw: Mapping[str, Any]) -> dict[str, obje
 
 class WhisperXEngine:
     def __init__(self, config: ServiceConfig):
+        use_local_resources_only()
         assert_punkt_tab(config.nltk_data_root)
         os.environ["NLTK_DATA"] = str(config.nltk_data_root)
         try:
@@ -109,11 +111,12 @@ class WhisperXEngine:
         self._inference_lock = threading.Lock()
         self._alignment_models: dict[str, tuple[object, object]] = {}
         started = time.monotonic()
-        logger.info("loading ASR model=%s; uncached weights may download during this call", config.model)
+        logger.info("loading prepared ASR model=%s", config.model)
         self._asr = whisperx_module.load_model(
-            config.model,
+            asr_path(config),
             config.device,
             compute_type=config.compute,
+            local_files_only=True,
         )
         logger.info("ASR model ready in %.1fs", time.monotonic() - started)
         self._whisperx_version = metadata.version("whisperx")
@@ -131,10 +134,18 @@ class WhisperXEngine:
         model = self._alignment_models.get(language)
         if model is None:
             started = time.monotonic()
-            logger.info("loading alignment model for language=%s; uncached weights may download during this call", language)
+            logger.info("loading prepared alignment model for language=%s", language)
+            try:
+                kind, selected = alignment_path(self._config, language)
+            except ValueError as error:
+                raise InferenceInputError(str(error)) from error
+            assert_sentence_data(self._config.nltk_data_root, language)
             loaded = self._whisperx.load_align_model(
                 language_code=language,
                 device=self._config.device,
+                model_name=selected,
+                model_dir=str(torch_cache(self._config)) if kind == "torchaudio" else hf_cache(self._config),
+                model_cache_only=True,
             )
             if not isinstance(loaded, tuple) or len(loaded) != 2:
                 raise RuntimeError("WhisperX returned an invalid alignment model")
@@ -148,6 +159,11 @@ class WhisperXEngine:
             raise InferenceInputError(
                 f"model {self._config.model!r} is English-only and cannot transcribe {language!r}"
             )
+        if language is not None:
+            try:
+                alignment_selection(language)
+            except ValueError as error:
+                raise InferenceInputError(str(error)) from error
         samples = self._numpy.frombuffer(audio.pcm_s16le, dtype="<i2").astype(self._numpy.float32)
         samples /= 32768.0
         samples = self._numpy.ascontiguousarray(samples)
@@ -162,7 +178,7 @@ class WhisperXEngine:
                 language=language,
             )
             logger.info("transcription completed in %.1fs", time.monotonic() - started)
-            detected = transcription.get("language") or language or "en"
+            detected = transcription.get("language") or language
             if not isinstance(detected, str) or not detected.strip():
                 raise RuntimeError("WhisperX did not return a valid language")
             segments = transcription.get("segments", [])

package/services/whisperx/src/hypit_whisperx_service/models.py

@@ -0,0 +1,131 @@
+"""WhisperX-owned model selection and explicit preparation.
+
+The pinned upstream libraries own their model tables and cache formats. The service
+uses those same caches, but has no authority to fetch missing execution resources.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+from urllib.parse import urlparse
+from typing import TYPE_CHECKING
+from .resources import UnpreparedResourceError
+
+if TYPE_CHECKING:
+    from .config import ServiceConfig
+
+
+def missing_resource(subject: str) -> UnpreparedResourceError:
+    return UnpreparedResourceError(f"{subject} is not prepared; run hypit programs prepare --endpoint <your-whisperx-endpoint> with the selected model and alignmentLanguages")
+
+
+def forbid_inference_downloads(*args: object, **kwargs: object) -> None:
+    raise missing_resource("A WhisperX execution resource")
+
+
+def use_local_resources_only() -> None:
+    # This is the dedicated inference process, not the preparation process or Host.
+    # WhisperX 3.8.6 has no no-download option for its torchaudio and NLTK branches.
+    # Refuse at their download boundary as well as using HF's local-files-only API.
+    import nltk
+    import torch.hub
+    nltk.download = forbid_inference_downloads
+    torch.hub.download_url_to_file = forbid_inference_downloads
+
+
+def hf_cache(config: ServiceConfig) -> str | None:
+    return str(config.model_cache / "huggingface") if config.model_cache is not None else None
+
+
+def torch_cache(config: ServiceConfig) -> Path:
+    if config.model_cache is not None:
+        return config.model_cache / "torch"
+    import torch.hub
+    return Path(torch.hub.get_dir()) / "checkpoints"
+
+
+def alignment_selection(language: str) -> tuple[str, str]:
+    from whisperx.alignment import DEFAULT_ALIGN_MODELS_HF, DEFAULT_ALIGN_MODELS_TORCH
+    if language in DEFAULT_ALIGN_MODELS_TORCH:
+        return "torchaudio", DEFAULT_ALIGN_MODELS_TORCH[language]
+    if language in DEFAULT_ALIGN_MODELS_HF:
+        return "huggingface", DEFAULT_ALIGN_MODELS_HF[language]
+    raise ValueError(f"WhisperX has no default alignment model for language {language!r}")
+
+
+def asr_path(config: ServiceConfig, *, prepare: bool = False) -> str:
+    if prepare:
+        try:
+            return asr_path(config)
+        except RuntimeError:
+            pass
+    if Path(config.model).is_dir():
+        path = Path(config.model).resolve()
+    else:
+        from faster_whisper.utils import download_model
+        try:
+            path = Path(download_model(config.model, cache_dir=hf_cache(config), local_files_only=not prepare))
+        except Exception as error:
+            if prepare:
+                raise
+            raise missing_resource(f"ASR model {config.model!r}") from error
+    for name in ("model.bin", "config.json", "tokenizer.json"):
+        if not (path / name).is_file():
+            raise missing_resource(f"ASR model {config.model!r}: {name}")
+    return str(path)
+
+
+def alignment_path(config: ServiceConfig, language: str) -> tuple[str, str]:
+    kind, name = alignment_selection(language)
+    if kind == "torchaudio":
+        import torchaudio
+        bundle = getattr(torchaudio.pipelines, name)
+        # The pinned torchaudio bundle owns its checkpoint URL. Its cache uses
+        # that URL's basename; do not duplicate the language or URL table here.
+        checkpoint = torch_cache(config) / Path(urlparse(bundle._path).path).name
+        if not checkpoint.is_file():
+            raise missing_resource(f"Alignment model {name!r} for {language}")
+        return kind, name
+    from huggingface_hub import snapshot_download
+    try:
+        path = Path(snapshot_download(name, cache_dir=hf_cache(config), local_files_only=True))
+    except Exception as error:
+        raise missing_resource(f"Alignment model {name!r} for {language}") from error
+    weights = any((path / item).is_file() for item in (
+        "model.safetensors", "pytorch_model.bin", "model.safetensors.index.json", "pytorch_model.bin.index.json",
+    ))
+    if not weights or not (path / "config.json").is_file():
+        raise missing_resource(f"Alignment model {name!r} for {language}")
+    # Actually read tokenizer/processor resources, not only the cache directory.
+    from transformers import Wav2Vec2Processor
+    try:
+        Wav2Vec2Processor.from_pretrained(str(path), local_files_only=True)
+    except (OSError, ValueError) as error:
+        raise missing_resource(f"Alignment processor {name!r} for {language}") from error
+    return kind, str(path)
+
+
+def check_models(config: ServiceConfig) -> None:
+    asr_path(config)
+    from .resources import assert_sentence_data
+    for language in config.alignment_languages:
+        alignment_path(config, language)
+        assert_sentence_data(config.nltk_data_root, language)
+
+
+def prepare_models(config: ServiceConfig) -> None:
+    print(f"Preparing ASR model {config.model}", flush=True)
+    asr_path(config, prepare=True)
+    from whisperx import load_align_model
+    for language in config.alignment_languages:
+        kind, name = alignment_selection(language)
+        try:
+            alignment_path(config, language)
+        except RuntimeError:
+            print(f"Preparing alignment model {name} for {language}", flush=True)
+            cache = str(torch_cache(config)) if kind == "torchaudio" else hf_cache(config)
+            if cache is not None:
+                Path(cache).mkdir(parents=True, exist_ok=True)
+            # Upstream downloads and loads the exact selected checkpoint; the
+            # returned model is discarded here, not kept in the service's RAM.
+            load_align_model(language_code=language, device="cpu", model_dir=cache)
+            alignment_path(config, language)

package/services/whisperx/src/hypit_whisperx_service/prepare.py

@@ -3,17 +3,35 @@ from __future__ import annotations
 import argparse
 from pathlib import Path
 
-from .resources import default_nltk_data_root, prepare_punkt_tab
+from .resources import prepare_punkt_tab, assert_sentence_data
+from .config import ServiceConfig
+from .models import prepare_models, alignment_selection
+from dataclasses import replace
 
 
 def main() -> None:
-    parser = argparse.ArgumentParser(description="Install NLTK data required by Hypit WhisperX")
+    config = ServiceConfig.from_environment()
+    parser = argparse.ArgumentParser(description="Prepare the selected WhisperX models and sentence data")
     parser.add_argument(
         "--nltk-data",
         type=Path,
-        default=default_nltk_data_root(),
+        default=config.nltk_data_root,
         help="NLTK data root (defaults to the SVML user cache)",
     )
     arguments = parser.parse_args()
+    config = replace(config, nltk_data_root=arguments.nltk_data)
+    for language in config.alignment_languages:
+        if config.model.endswith(".en") and language != "en":
+            raise ValueError(f"ASR model {config.model!r} is English-only; select a multilingual model to prepare {language}")
+        alignment_selection(language)
     path = prepare_punkt_tab(arguments.nltk_data)
+    for language in config.alignment_languages:
+        try:
+            assert_sentence_data(arguments.nltk_data, language)
+        except RuntimeError:
+            import nltk
+            if not nltk.download("punkt_tab", download_dir=str(arguments.nltk_data), quiet=False, force=True, raise_on_error=True):
+                raise RuntimeError("NLTK could not prepare sentence data")
+            assert_sentence_data(arguments.nltk_data, language)
+    prepare_models(config)
     print(path)

package/services/whisperx/src/hypit_whisperx_service/resources.py

@@ -3,6 +3,10 @@ from __future__ import annotations
 from pathlib import Path
 
 
+class UnpreparedResourceError(RuntimeError):
+    """A selected local execution resource needs explicit preparation."""
+
+
 def default_nltk_data_root() -> Path:
     return Path.home() / ".cache" / "hypit" / "whisperx" / "nltk_data"
 
@@ -11,12 +15,11 @@ def assert_punkt_tab(root: Path) -> None:
     import nltk
 
     location = str(root.expanduser().resolve())
-    if location not in nltk.data.path:
-        nltk.data.path.insert(0, location)
+    nltk.data.path[:] = [location]
     try:
         nltk.data.find("tokenizers/punkt_tab", paths=[location])
     except LookupError as error:
-        raise RuntimeError(
+        raise UnpreparedResourceError(
             "NLTK punkt_tab data is unavailable; run `hypit-whisperx-prepare`"
         ) from error
 
@@ -39,3 +42,16 @@ def prepare_punkt_tab(root: Path) -> Path:
         raise RuntimeError("NLTK could not install punkt_tab")
     assert_punkt_tab(root)
     return target
+
+
+def assert_sentence_data(root: Path, language: str) -> None:
+    from whisperx.utils import PUNKT_LANGUAGES
+    from nltk.tokenize.punkt import load_punkt_params
+    from nltk.data import FileSystemPathPointer
+    # Use the same sentence tokenizer selection as the pinned WhisperX version.
+    name = PUNKT_LANGUAGES.get(language, "english")
+    try:
+        assert_punkt_tab(root)
+        load_punkt_params(FileSystemPathPointer(str(root / "tokenizers" / "punkt_tab" / name)))
+    except (OSError, ValueError) as error:
+        raise UnpreparedResourceError(f"NLTK sentence data for {language} ({name}) is unavailable; run hypit-whisperx-prepare") from error

package/services/yt-dlp/README.md

@@ -1,19 +1,18 @@
 # Pinned yt-dlp
 
-This project holds the locked Python downloader used by `hypit media fetch`. `uv` installs the
-selected version and its dependencies when invoked. Site support follows that version's extractors;
-the downloaded file preserves the source used by the production.
+This project owns the locked downloader and its EJS solver dependencies. Prepare it explicitly with
+`hypit media prepare-fetch`; `hypit media fetch` only uses the resulting executable and never runs uv.
+The preparation command reports the executable path under the machine Host state. Use that exact
+path for `--version`, `--help` or deliberately chosen site-specific options. On Windows it is an
+`.exe` inside the virtual environment's `Scripts` directory.
 
-It is invoked one command at a time rather than run as a service:
+For a contributor-managed environment, run `uv sync --project services/yt-dlp --frozen` explicitly,
+then invoke `.venv/bin/yt-dlp` (Windows: `.venv\Scripts\yt-dlp.exe`) from that project.
+Direct invocation accepts upstream options and config; the Hypit fetch wrapper instead ignores
+user config/plugins, disables updates and remote components, and uses its calling Node executable
+for JavaScript challenges. The locked extras supply the solver before fetching begins.
 
-```bash
-uv run --project "/path/to/hypit/services/yt-dlp" --frozen yt-dlp --version
-uv run --project "/path/to/hypit/services/yt-dlp" --frozen yt-dlp --help
-```
-
-Replace the project path with this directory in the installed Hypit Distribution. Direct invocation
-accepts yt-dlp's own options, including source-specific format or authentication settings. Choose
-those for the actual download. `ffmpeg` on PATH merges separate video and audio streams.
-
-`packages/yt-dlp` is the Node wrapper used by Video CLI. It stages a download and saves the requested
-project file; later commands read that file.
+`ffmpeg` on PATH merges separate video and audio streams; the CLI also requires `ffprobe` for the
+saved-file report. These tools are supplied by the operator's package manager, never installed by
+fetch. Site support follows the selected yt-dlp extractors. A source requiring authentication or
+additional options can use an explicit direct invocation and then supply the local file to Hypit.

package/services/yt-dlp/pyproject.toml

@@ -6,7 +6,7 @@ readme = "README.md"
 requires-python = ">=3.10,<3.14"
 license = { text = "Hypit Open Source License (Apache-2.0 with additional conditions); see LICENSE at the repository root" }
 dependencies = [
-  "yt-dlp==2026.8.19",
+  "yt-dlp[default]==2026.8.19",
 ]
 
 [tool.uv]