lumen-resources 0.2.0__tar.gz

lumen_resources-0.2.0/.gitignore

@@ -0,0 +1,64 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+PIPFILE.lock
+
+# Virtual environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Testing
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+.tox/
+.nox/
+
+# IDEs
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+.vscode/
+
+# Project specific
+.cache/
+models/
+*.npz
+*.onnx
+*.rknn
+*.pt
+
+# Temporary files
+*.tmp
+*.bak
+*.log
+
+# UV - keep lock file for reproducible builds
+.uv/

lumen_resources-0.2.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: lumen-resources
+Version: 0.2.0
+Summary: Unified model resource management for Lumen ML services
+Author-email: EdwinZhanCN <support@lumilio.org>
+License: MIT
+Project-URL: Homepage, https://github.com/EdwinZhanCN/Lumen
+Project-URL: Issues, https://github.com/EdwinZhanCN/Lumen/issues
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: huggingface_hub>=0.20.0
+Requires-Dist: modelscope>=1.11.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: jsonschema>=4.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: datamodel-code-generator[http]>=0.25.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0.0; extra == "dev"
+Provides-Extra: config
+Requires-Dist: pyyaml>=6.0.0; extra == "config"
+
+# Lumen Resources
+
+Lightweight tooling for shipping Lumen ML services. This package centralizes how models are described, validated, downloaded, and cached so every service (CLIP, face, etc.) follows the same playbook—whether weights live on Hugging Face, ModelScope, or a private registry.
+
+## Why use it?
+
+- **Single source of truth** – YAML configs describing deployments, devices, runtimes, and model aliases.
+- **Schema-backed validation** – JSON Schema plus Pydantic to catch errors before runtime.
+- **Cross-platform downloads** – Intelligent routing between Hugging Face and ModelScope with caching/resume support.
+- **CLI + Python API** – Automate in CI or embed in service bootstraps.
+- **Result schemas** – Typed response validators (`EmbeddingV1`, `FaceV1`, `LabelsV1`) for downstream services.
+
+## Installation
+
+```bash
+# project install
+pip install "lumen-resources @ git+https://github.com/EdwinZhanCN/Lumen.git@main#subdirectory=lumen-resources"
+
+# dev install
+git clone https://github.com/EdwinZhanCN/Lumen.git
+cd Lumen/lumen-resources
+pip install -e ".[dev,config]"
+```
+
+Optional extras depending on your targets:
+
+```bash
+pip install huggingface_hub
+pip install modelscope
+pip install torch torchvision
+pip install onnxruntime
+```
+
+## Usage
+
+### CLI
+
+```bash
+# download everything defined in config.yaml
+lumen-resources download config.yaml
+
+# strict config validation
+lumen-resources validate config.yaml
+
+# validate a model_info.json
+lumen-resources validate-model-info path/to/model_info.json
+
+# inspect cache contents (defaults to ~/.lumen/)
+lumen-resources list ~/.lumen/
+```
+
+### Python API
+
+```python
+from lumen_resources import (
+    load_and_validate_config,
+    Downloader,
+    load_and_validate_model_info,
+    EmbeddingV1,
+)
+
+config = load_and_validate_config("config.yaml")
+downloader = Downloader(config, verbose=True)
+results = downloader.download_all(force=False)
+
+model_info = load_and_validate_model_info("model_info.json")
+```
+
+## Configuration essentials
+
+```yaml
+metadata:
+  region: "other"      # or "cn" to prefer ModelScope
+  cache_dir: "~/.lumen/models"
+
+deployment:
+  mode: "single"       # or "hub"
+  service: "clip"
+
+services:
+  clip:
+    enabled: true
+    package: "lumen_clip"
+    backend_settings:
+      device: "cuda"
+      batch_size: 16
+      onnx_providers: ["CUDAExecutionProvider", "CPUExecutionProvider"]
+    models:
+      default:
+        model: "ViT-B-32"
+        runtime: "torch"
+      fp16:
+        model: "ViT-B-32"
+        runtime: "onnx"
+```
+
+- `metadata.region` decides whether downloads prefer ModelScope or Hugging Face.
+- `backend_settings` lets you declare execution providers, batch sizes, devices, etc.
+- Each entry in `models` becomes a cache namespace (`clip/default`, `clip/fp16`, …).
+
+## Reference
+
+- Source: `src/lumen_resources/`
+  - `lumen_config.py` – Typed config models
+  - `downloader.py` – Platform abstraction + caching
+  - `cli.py` – Command entrypoint
+  - `result_schemas/` – Response validators
+- Docs: https://doc.lumilio.org
+- Issues & support: open a ticket in the main Lumen monorepo.

lumen_resources-0.2.0/README.md

@@ -0,0 +1,109 @@
+# Lumen Resources
+
+Lightweight tooling for shipping Lumen ML services. This package centralizes how models are described, validated, downloaded, and cached so every service (CLIP, face, etc.) follows the same playbook—whether weights live on Hugging Face, ModelScope, or a private registry.
+
+## Why use it?
+
+- **Single source of truth** – YAML configs describing deployments, devices, runtimes, and model aliases.
+- **Schema-backed validation** – JSON Schema plus Pydantic to catch errors before runtime.
+- **Cross-platform downloads** – Intelligent routing between Hugging Face and ModelScope with caching/resume support.
+- **CLI + Python API** – Automate in CI or embed in service bootstraps.
+- **Result schemas** – Typed response validators (`EmbeddingV1`, `FaceV1`, `LabelsV1`) for downstream services.
+
+## Installation
+
+```bash
+# project install
+pip install "lumen-resources @ git+https://github.com/EdwinZhanCN/Lumen.git@main#subdirectory=lumen-resources"
+
+# dev install
+git clone https://github.com/EdwinZhanCN/Lumen.git
+cd Lumen/lumen-resources
+pip install -e ".[dev,config]"
+```
+
+Optional extras depending on your targets:
+
+```bash
+pip install huggingface_hub
+pip install modelscope
+pip install torch torchvision
+pip install onnxruntime
+```
+
+## Usage
+
+### CLI
+
+```bash
+# download everything defined in config.yaml
+lumen-resources download config.yaml
+
+# strict config validation
+lumen-resources validate config.yaml
+
+# validate a model_info.json
+lumen-resources validate-model-info path/to/model_info.json
+
+# inspect cache contents (defaults to ~/.lumen/)
+lumen-resources list ~/.lumen/
+```
+
+### Python API
+
+```python
+from lumen_resources import (
+    load_and_validate_config,
+    Downloader,
+    load_and_validate_model_info,
+    EmbeddingV1,
+)
+
+config = load_and_validate_config("config.yaml")
+downloader = Downloader(config, verbose=True)
+results = downloader.download_all(force=False)
+
+model_info = load_and_validate_model_info("model_info.json")
+```
+
+## Configuration essentials
+
+```yaml
+metadata:
+  region: "other"      # or "cn" to prefer ModelScope
+  cache_dir: "~/.lumen/models"
+
+deployment:
+  mode: "single"       # or "hub"
+  service: "clip"
+
+services:
+  clip:
+    enabled: true
+    package: "lumen_clip"
+    backend_settings:
+      device: "cuda"
+      batch_size: 16
+      onnx_providers: ["CUDAExecutionProvider", "CPUExecutionProvider"]
+    models:
+      default:
+        model: "ViT-B-32"
+        runtime: "torch"
+      fp16:
+        model: "ViT-B-32"
+        runtime: "onnx"
+```
+
+- `metadata.region` decides whether downloads prefer ModelScope or Hugging Face.
+- `backend_settings` lets you declare execution providers, batch sizes, devices, etc.
+- Each entry in `models` becomes a cache namespace (`clip/default`, `clip/fp16`, …).
+
+## Reference
+
+- Source: `src/lumen_resources/`
+  - `lumen_config.py` – Typed config models
+  - `downloader.py` – Platform abstraction + caching
+  - `cli.py` – Command entrypoint
+  - `result_schemas/` – Response validators
+- Docs: https://doc.lumilio.org
+- Issues & support: open a ticket in the main Lumen monorepo.

lumen_resources-0.2.0/docs/examples/clip_torch_cn.yaml

@@ -0,0 +1,37 @@
+# Lumen Services Configuration - Single Service Mode
+# yaml-language-server: $schema=https://doc.lumilio.org/schema/config-schema.yaml
+
+metadata:
+    version: "1.0.0"
+    region: "cn" # "cn" for ModelScope, "other" for HuggingFace
+    cache_dir: "~/Lumen-Resources"
+
+deployment:
+    mode: "single"
+    service: "clip"
+
+server:
+    port: 50051
+    host: "0.0.0.0"
+    mdns:
+        enabled: true
+        service_name: "lumen-clip"
+
+services:
+    # CLIP Service
+    clip:
+        enabled: true
+        package: "lumen_clip"
+        import:
+            registry_class: "lumen_clip.service_registry.CLIPService"
+            add_to_server: "lumen_clip.proto.ml_service_pb2_grpc.add_InferenceServicer_to_server"
+        backend_settings:
+            device: "mps"
+            onnx_providers:
+                - "CoreMLExecutionProvider"
+            batch_size: 8
+        models:
+            general:
+                model: "MobileCLIP2-S2" # CN-CLIP_ViT-L-14
+                runtime: torch
+                dataset: ImageNet_1k

lumen_resources-0.2.0/docs/examples/hub-service.yaml

@@ -0,0 +1,69 @@
+# Lumen Services Configuration - Hub Mode
+# yaml-language-server: $schema=https://doc.lumilio.org/schema/config-schema.yaml
+
+metadata:
+    version: "1.0.0"
+    region: "other" # "cn" for ModelScope, "other" for HuggingFace
+    cache_dir: "/opt/lumen/models"
+
+deployment:
+    mode: "hub"
+    services: ["clip", "face"]
+
+server:
+    port: 50051
+    host: "0.0.0.0"
+    mdns:
+        enabled: true
+        service_name: "lumen-hub"
+
+services:
+    # CLIP Service
+    clip:
+        enabled: true
+        package: "lumen_clip"
+        import:
+            registry_class: "lumen_clip.service_registry.ClipService"
+            add_to_server: "lumen_clip.proto.ml_service_pb2_grpc.add_InferenceServicer_to_server"
+
+        models:
+            # Default general-purpose CLIP model
+            default:
+                model: "ViT-B-32"
+                runtime: "torch"
+                dataset: "ImageNet_1k"
+
+            # Mobile-optimized model
+            mobile:
+                model: "MobileCLIP-S2"
+                runtime: "onnx"
+
+    # Face Recognition Service
+    face:
+        enabled: true
+        package: "lumen_face"
+        import:
+            registry_class: "lumen_face.service_registry.FaceService"
+            add_to_server: "lumen_face.proto.ml_service_pb2_grpc.add_FaceServicer_to_server"
+
+        models:
+            default:
+                model: "FaceNet-512"
+                runtime: "onnx"
+
+            mobile:
+                model: "MobileFaceNet"
+                runtime: "onnx"
+
+    # OCR Service (disabled, but configured)
+    ocr:
+        enabled: false
+        package: "lumen_ocr"
+        import:
+            registry_class: "lumen_ocr.service_registry.OcrService"
+            add_to_server: "lumen_ocr.proto.ml_service_pb2_grpc.add_OcrServicer_to_server"
+
+        models:
+            default:
+                model: "PaddleOCR-v4"
+                runtime: "onnx"

lumen_resources-0.2.0/docs/examples/model_info_template.json

@@ -0,0 +1,48 @@
+{
+    "name": "CN-CLIP_ViT-L-14",
+    "version": "1.0.0",
+    "description": "OFA-Sys/chinese-clip-vit-large-patch14 Chinese CLIP model for chinese-specific tasks",
+    "model_type": "chinese_clip",
+    "embedding_dim": 768,
+    "source": {
+        "format": "huggingface",
+        "repo_id": "OFA-Sys/chinese-clip-vit-large-patch14"
+    },
+    "runtimes": {
+        "torch": {
+            "available": true,
+            "files": ["torch/model.pt"]
+        },
+        "onnx": {
+            "available": true,
+            "files": [
+                "onnx/text.fp32.onnx",
+                "onnx/text.fp16.onnx",
+                "onnx/vision.fp32.onnx",
+                "onnx/vision.fp16.onnx"
+            ]
+        },
+        "rknn": {
+            "available": true,
+            "devices": ["rk3566", "rk3588"],
+            "files": {
+                "rk3566": [
+                    "rknn/rk3566/text.fp16.rknn",
+                    "rknn/rk3566/vision.fp16.rknn"
+                ],
+                "rk3588": [
+                    "rknn/rk3588/text.fp16.rknn",
+                    "rknn/rk3588/vision.fp16.rknn"
+                ]
+            }
+        }
+    },
+    "datasets": {
+        "ImageNet_1k": "ImageNet_1k.npy"
+    },
+    "metadata": {
+        "license": "Apache License 2.0",
+        "author": "Lumilio Photos",
+        "created_at": "2025-10-16"
+    }
+}

lumen_resources-0.2.0/docs/examples/single-service.yaml

@@ -0,0 +1,40 @@
+# Lumen Services Configuration - Single Service Mode
+# yaml-language-server: $schema=https://doc.lumilio.org/schema/config-schema.yaml
+
+metadata:
+    version: "1.0.0"
+    region: "other" # "cn" for ModelScope, "other" for HuggingFace
+    cache_dir: "~/.lumen/models"
+
+deployment:
+    mode: "single"
+    service: "clip"
+
+server:
+    port: 50051
+    host: "0.0.0.0"
+    mdns:
+        enabled: true
+        service_name: "lumen-clip"
+
+services:
+    # CLIP Service
+    clip:
+        enabled: true
+        package: "lumen_clip"
+        import:
+            registry_class: "lumen_clip.service_registry.ClipService"
+            add_to_server: "lumen_clip.proto.ml_service_pb2_grpc.add_InferenceServicer_to_server"
+
+        models:
+            # Default general-purpose CLIP model
+            default:
+                model: "ViT-B-32"
+                runtime: "torch"
+                dataset: "ImageNet_1k"
+
+            # Chinese CLIP model
+            cn_clip:
+                model: "CN-CLIP-ViT-B-16"
+                runtime: "torch"
+                dataset: "ImageNet_1k"

lumen_resources-0.2.0/pyproject.toml

@@ -0,0 +1,57 @@
+[project]
+name = "lumen-resources"
+dynamic = ["version"]
+description = "Unified model resource management for Lumen ML services"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "EdwinZhanCN", email = "support@lumilio.org" }
+]
+dependencies = [
+    "huggingface_hub>=0.20.0",
+    "modelscope>=1.11.0",
+    "pydantic>=2.0.0",
+    "jsonschema>=4.0.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/EdwinZhanCN/Lumen"
+Issues = "https://github.com/EdwinZhanCN/Lumen/issues"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+    "black>=23.0.0",
+    "ruff>=0.1.0",
+    "datamodel-code-generator[http]>=0.25.0",
+    "pre-commit>=3.0.0",
+]
+config = ["pyyaml>=6.0.0"]
+
+[project.scripts]
+lumen-resources = "lumen_resources.cli:main"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "N"]
+ignore = []
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_functions = ["test_*"]
+
+[build-system]
+requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools_scm]
+fallback_version = "0.0.0"
+search_parent_directories = true
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"lumen_resources" = ["schemas/*.yaml", "schemas/*.json"]

lumen_resources-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

lumen_resources-0.2.0/src/lumen_resources/__init__.py

@@ -0,0 +1,89 @@
+"""Lumen Resources - Unified Model Resource Management.
+
+Configuration-driven tool for managing ML model resources with production-grade
+YAML configuration, JSON Schema validation, and Pydantic models. Provides a
+unified interface for downloading models from multiple platforms including
+HuggingFace Hub and ModelScope.
+
+This package offers:
+- Configuration-driven YAML setup for ML model resources
+- Multi-platform support (HuggingFace Hub, ModelScope)
+- Runtime flexibility (ONNX, PyTorch, TensorFlow, RKNN)
+- Production-grade validation using JSON Schema and Pydantic
+- CLI interface for command-line operations
+- Programmatic API for integration into other applications
+
+Example:
+    >>> from lumen_resources import load_and_validate_config, Downloader
+    >>>
+    >>> # Load and validate configuration
+    >>> config = load_and_validate_config("config.yaml")
+    >>>
+    >>> # Download models
+    >>> downloader = Downloader(config, verbose=True)
+    >>> results = downloader.download_all()
+    >>>
+    >>> # Check results
+    >>> for model_type, result in results.items():
+    ...     if result.success:
+    ...         print(f"Downloaded: {model_type} to {result.model_path}")
+    ...     else:
+    ...         print(f"Failed: {model_type} - {result.error}")
+
+The package follows a layered architecture:
+- Configuration layer: Pydantic models for type-safe config handling
+- Validation layer: JSON Schema and Pydantic validation
+- Platform layer: Unified interface for different model repositories
+- Download layer: Efficient model downloading with validation
+- CLI layer: User-friendly command-line interface
+"""
+
+from .lumen_config import LumenConfig, Runtime, Region
+from .downloader import Downloader, DownloadResult
+from .exceptions import (
+    ResourceError,
+    ConfigError,
+    DownloadError,
+    PlatformUnavailableError,
+    ValidationError,
+    ModelInfoError,
+)
+from .lumen_config_validator import load_and_validate_config
+
+from .model_info import ModelInfo, Source, Runtimes, Metadata
+from .model_info_validator import load_and_validate_model_info
+from .result_schemas import (
+    EmbeddingV1,
+    FaceV1,
+    LabelsV1
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # Configuration
+    "LumenConfig",
+    "Runtime",
+    "Region",
+    "load_and_validate_config",
+    # Model Info
+    "ModelInfo",
+    "Source",
+    "Runtimes",
+    "Metadata",
+    "load_and_validate_model_info",
+    # Response Validation
+    "FaceV1",
+    "EmbeddingV1",
+    "LabelsV1",
+    # Downloader
+    "Downloader",
+    "DownloadResult",
+    # Exceptions
+    "ResourceError",
+    "ConfigError",
+    "DownloadError",
+    "PlatformUnavailableError",
+    "ValidationError",
+    "ModelInfoError",
+]