vlm-engine 0.1.0__tar.gz

vlm_engine-0.1.0/PKG-INFO

@@ -0,0 +1,309 @@
+Metadata-Version: 2.4
+Name: vlm_engine
+Version: 0.1.0
+Summary: Advanced Vision-Language Model Engine for content tagging
+Home-page: https://github.com/Haven-hvn/haven-vlm-engine-package
+Author: HAVEN Network
+Author-email: officialhavennetwork@gmail.com
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic
+Requires-Dist: numpy
+Requires-Dist: torch
+Requires-Dist: torchvision
+Requires-Dist: aiohttp
+Requires-Dist: pyyaml
+Requires-Dist: opencv-python
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# VLM Engine
+
+A high-performance Python package for Vision-Language Model (VLM) based content tagging and analysis. This package provides an advanced implementation for automatic content detection and tagging, delivering superior accuracy compared to traditional image classification methods.
+
+## Features
+
+- **Remote VLM Integration**: Connects to any OpenAI-compatible VLM endpoint (no local model loading required)
+- **Context-Aware Detection**: Leverages Vision-Language Models' understanding of visual relationships for accurate content tagging
+- **Flexible Architecture**: Modular pipeline system with configurable models and processing stages
+- **Asynchronous Processing**: Built on asyncio for efficient video and image processing
+- **Customizable Tag Sets**: Easy configuration of detection categories and confidence thresholds
+- **Production Ready**: Includes retry logic, error handling, and comprehensive logging
+
+## Installation
+
+### From PyPI (when published)
+```bash
+pip install vlm-engine
+```
+
+### From Source
+```bash
+git clone https://github.com/Haven-hvn/haven-vlm-engine-package.git
+cd vlm-engine-package
+pip install -e .
+```
+
+### Requirements
+- Python 3.8+
+- **Sufficient RAM**: Video preprocessing loads entire videos into memory (not GPU memory)
+- Compatible VLM server endpoint:
+  - Remote OpenAI-compatible API (recommended)
+  - Local server using [LM Studio](https://lmstudio.ai/)
+  - Haven's custom VLM available at [https://havenmodels.orbiter.website/](https://havenmodels.orbiter.website/)
+
+## Quick Start
+
+```python
+import asyncio
+from vlm_engine import VLMEngine
+from vlm_engine.config_models import EngineConfig, ModelConfig
+
+# Configure the engine
+config = EngineConfig(
+    active_ai_models=["vlm_nsfw_model"],
+    models={
+        "vlm_nsfw_model": ModelConfig(
+            type="vlm_model",
+            model_id="HuggingFaceTB/SmolVLM-Instruct",
+            api_base_url="http://localhost:7045",
+            tag_list=["tag1", "tag2", "tag3"]  # Your custom tags
+        )
+    }
+)
+
+# Initialize and use
+async def main():
+    engine = VLMEngine(config)
+    await engine.initialize()
+    
+    results = await engine.process_video(
+        "path/to/video.mp4",
+        frame_interval=2.0,
+        threshold=0.5
+    )
+    print(f"Detected tags: {results}")
+
+asyncio.run(main())
+```
+
+## Architecture
+
+### Core Components
+
+1. **VLMEngine**: Main entry point for the package
+   - Manages model initialization and pipeline execution
+   - Handles asynchronous processing of videos and images
+
+2. **VLMClient**: OpenAI-compatible API client
+   - Supports any VLM with chat completions endpoint
+   - Includes retry logic with exponential backoff and jitter
+   - Handles image encoding and prompt formatting
+
+3. **Pipeline System**: Flexible processing pipeline
+   - Modular design allows custom processing stages
+   - Built-in support for preprocessing, analysis, and postprocessing
+   - Configurable through YAML or Python objects
+
+4. **Model Management**: Dynamic model loading
+   - Supports multiple model types (VLM, preprocessors, postprocessors)
+   - Lazy loading for efficient resource usage
+   - Thread-safe model access
+
+## Configuration
+
+### Basic Configuration
+
+```python
+from vlm_engine.config_models import EngineConfig, ModelConfig, PipelineConfig
+
+config = EngineConfig(
+    active_ai_models=["my_vlm_model"],
+    models={
+        "my_vlm_model": ModelConfig(
+            type="vlm_model",
+            model_id="model-name",
+            api_base_url="http://localhost:8000",
+            tag_list=["action1", "action2", "action3"],
+            max_new_tokens=128,
+            request_timeout=70,
+            vlm_detected_tag_confidence=0.99
+        )
+    },
+    pipelines={
+        "video_pipeline": PipelineConfig(
+            inputs=["video_path", "frame_interval"],
+            output="results",
+            models=[{"name": "my_vlm_model", "inputs": ["frame"], "outputs": "tags"}]
+        )
+    }
+)
+```
+
+### Advanced Configuration
+
+The package supports complex configurations including:
+- Multiple models in a pipeline
+- Custom preprocessing and postprocessing stages
+- Category-specific settings (thresholds, durations, etc.)
+- Batch processing configurations
+
+See the [examples](examples/) directory for detailed configuration examples.
+
+## API Reference
+
+### VLMEngine
+
+```python
+class VLMEngine:
+    def __init__(self, config: EngineConfig)
+    async def initialize()
+    async def process_video(video_path: str, **kwargs) -> Dict[str, Any]
+```
+
+### Processing Parameters
+
+- `video_path`: Path to the video file
+- `frame_interval`: Seconds between frame samples (default: 0.5)
+- `threshold`: Confidence threshold for tag detection (default: 0.5)
+- `return_timestamps`: Include timestamp information (default: True)
+- `return_confidence`: Include confidence scores (default: True)
+
+## Performance Optimization
+
+### Memory Requirements
+- **Important**: Video preprocessing loads the entire video into system RAM (not GPU memory)
+- Ensure sufficient RAM for your video sizes (e.g., a 1GB video may require 4-8GB of available RAM)
+- Consider processing videos in segments for very large files
+
+### API Optimization
+- Configure retry settings based on your VLM server's capacity
+- Adjust `max_new_tokens` to balance speed vs accuracy
+- Use appropriate `frame_interval` to reduce processing time and API calls
+
+### Processing Speed
+- Increase `frame_interval` to sample fewer frames (faster but less accurate)
+- Use batch processing when your VLM endpoint supports it
+- Consider running multiple VLM instances for parallel processing
+
+## Extending the Package
+
+### Custom Models
+
+Create custom model classes by inheriting from the base Model class:
+
+```python
+from vlm_engine.models import Model
+
+class CustomModel(Model):
+    async def process(self, inputs):
+        # Your custom processing logic
+        return results
+```
+
+### Custom Pipelines
+
+Define custom pipelines for specific use cases:
+
+```python
+custom_pipeline = PipelineConfig(
+    inputs=["image_path"],
+    output="analysis",
+    models=[
+        {"name": "preprocessor", "inputs": ["image_path"], "outputs": "processed_image"},
+        {"name": "analyzer", "inputs": ["processed_image"], "outputs": "analysis"}
+    ]
+)
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Connection Errors**
+   - Ensure your VLM server is running and accessible
+   - Check the `api_base_url` configuration
+   - Verify firewall settings
+
+2. **GPU Memory Errors**
+   - Reduce batch size or frame interval
+   - Ensure proper CUDA installation
+   - Check GPU memory availability
+
+3. **Slow Processing**
+   - Increase frame interval for faster processing
+   - Use GPU acceleration if available
+   - Optimize VLM server settings
+
+### Logging
+
+Enable debug logging for troubleshooting:
+
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
+
+### Development Setup
+
+```bash
+git clone https://github.com/yourusername/vlm-engine.git
+cd vlm-engine
+pip install -e ".[dev]"
+```
+
+### Running Tests
+
+```bash
+pytest tests/
+```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details.
+
+## Acknowledgments
+
+- Built on top of modern Python async patterns
+- Inspired by production ML serving architectures
+- Haven's custom VLM models trained using [SmolVLM-Finetune](https://github.com/Haven-hvn/SmolVLM-Finetune) - Model Download found on [https://havenmodels.orbiter.website/](https://havenmodels.orbiter.website/)
+
+- Designed for integration with OpenAI-compatible VLM endpoints
+
+## Support
+
+For issues and feature requests, please use the [GitHub issue tracker](https://github.com/Haven-hvn/haven-vlm-engine-package/issues).
+
+For questions and discussions, join our community:
+- Discord: [Link to Discord](https://discord.gg/57mPMDfQew)
+
+---
+
+**Note**: This package requires an OpenAI-compatible VLM endpoint. Options include:
+
+### Remote Services
+- Any OpenAI-compatible API endpoint 
+- Akash deployment - https://github.com/Haven-hvn/haven-inference
+
+### Local Setup
+- [LM Studio](https://lmstudio.ai/) - Easy local VLM hosting with OpenAI-compatible API
+
+The package **does not** load VLM models directly - it communicates with external VLM services via API.

vlm_engine-0.1.0/README.md

@@ -0,0 +1,276 @@
+# VLM Engine
+
+A high-performance Python package for Vision-Language Model (VLM) based content tagging and analysis. This package provides an advanced implementation for automatic content detection and tagging, delivering superior accuracy compared to traditional image classification methods.
+
+## Features
+
+- **Remote VLM Integration**: Connects to any OpenAI-compatible VLM endpoint (no local model loading required)
+- **Context-Aware Detection**: Leverages Vision-Language Models' understanding of visual relationships for accurate content tagging
+- **Flexible Architecture**: Modular pipeline system with configurable models and processing stages
+- **Asynchronous Processing**: Built on asyncio for efficient video and image processing
+- **Customizable Tag Sets**: Easy configuration of detection categories and confidence thresholds
+- **Production Ready**: Includes retry logic, error handling, and comprehensive logging
+
+## Installation
+
+### From PyPI (when published)
+```bash
+pip install vlm-engine
+```
+
+### From Source
+```bash
+git clone https://github.com/Haven-hvn/haven-vlm-engine-package.git
+cd vlm-engine-package
+pip install -e .
+```
+
+### Requirements
+- Python 3.8+
+- **Sufficient RAM**: Video preprocessing loads entire videos into memory (not GPU memory)
+- Compatible VLM server endpoint:
+  - Remote OpenAI-compatible API (recommended)
+  - Local server using [LM Studio](https://lmstudio.ai/)
+  - Haven's custom VLM available at [https://havenmodels.orbiter.website/](https://havenmodels.orbiter.website/)
+
+## Quick Start
+
+```python
+import asyncio
+from vlm_engine import VLMEngine
+from vlm_engine.config_models import EngineConfig, ModelConfig
+
+# Configure the engine
+config = EngineConfig(
+    active_ai_models=["vlm_nsfw_model"],
+    models={
+        "vlm_nsfw_model": ModelConfig(
+            type="vlm_model",
+            model_id="HuggingFaceTB/SmolVLM-Instruct",
+            api_base_url="http://localhost:7045",
+            tag_list=["tag1", "tag2", "tag3"]  # Your custom tags
+        )
+    }
+)
+
+# Initialize and use
+async def main():
+    engine = VLMEngine(config)
+    await engine.initialize()
+    
+    results = await engine.process_video(
+        "path/to/video.mp4",
+        frame_interval=2.0,
+        threshold=0.5
+    )
+    print(f"Detected tags: {results}")
+
+asyncio.run(main())
+```
+
+## Architecture
+
+### Core Components
+
+1. **VLMEngine**: Main entry point for the package
+   - Manages model initialization and pipeline execution
+   - Handles asynchronous processing of videos and images
+
+2. **VLMClient**: OpenAI-compatible API client
+   - Supports any VLM with chat completions endpoint
+   - Includes retry logic with exponential backoff and jitter
+   - Handles image encoding and prompt formatting
+
+3. **Pipeline System**: Flexible processing pipeline
+   - Modular design allows custom processing stages
+   - Built-in support for preprocessing, analysis, and postprocessing
+   - Configurable through YAML or Python objects
+
+4. **Model Management**: Dynamic model loading
+   - Supports multiple model types (VLM, preprocessors, postprocessors)
+   - Lazy loading for efficient resource usage
+   - Thread-safe model access
+
+## Configuration
+
+### Basic Configuration
+
+```python
+from vlm_engine.config_models import EngineConfig, ModelConfig, PipelineConfig
+
+config = EngineConfig(
+    active_ai_models=["my_vlm_model"],
+    models={
+        "my_vlm_model": ModelConfig(
+            type="vlm_model",
+            model_id="model-name",
+            api_base_url="http://localhost:8000",
+            tag_list=["action1", "action2", "action3"],
+            max_new_tokens=128,
+            request_timeout=70,
+            vlm_detected_tag_confidence=0.99
+        )
+    },
+    pipelines={
+        "video_pipeline": PipelineConfig(
+            inputs=["video_path", "frame_interval"],
+            output="results",
+            models=[{"name": "my_vlm_model", "inputs": ["frame"], "outputs": "tags"}]
+        )
+    }
+)
+```
+
+### Advanced Configuration
+
+The package supports complex configurations including:
+- Multiple models in a pipeline
+- Custom preprocessing and postprocessing stages
+- Category-specific settings (thresholds, durations, etc.)
+- Batch processing configurations
+
+See the [examples](examples/) directory for detailed configuration examples.
+
+## API Reference
+
+### VLMEngine
+
+```python
+class VLMEngine:
+    def __init__(self, config: EngineConfig)
+    async def initialize()
+    async def process_video(video_path: str, **kwargs) -> Dict[str, Any]
+```
+
+### Processing Parameters
+
+- `video_path`: Path to the video file
+- `frame_interval`: Seconds between frame samples (default: 0.5)
+- `threshold`: Confidence threshold for tag detection (default: 0.5)
+- `return_timestamps`: Include timestamp information (default: True)
+- `return_confidence`: Include confidence scores (default: True)
+
+## Performance Optimization
+
+### Memory Requirements
+- **Important**: Video preprocessing loads the entire video into system RAM (not GPU memory)
+- Ensure sufficient RAM for your video sizes (e.g., a 1GB video may require 4-8GB of available RAM)
+- Consider processing videos in segments for very large files
+
+### API Optimization
+- Configure retry settings based on your VLM server's capacity
+- Adjust `max_new_tokens` to balance speed vs accuracy
+- Use appropriate `frame_interval` to reduce processing time and API calls
+
+### Processing Speed
+- Increase `frame_interval` to sample fewer frames (faster but less accurate)
+- Use batch processing when your VLM endpoint supports it
+- Consider running multiple VLM instances for parallel processing
+
+## Extending the Package
+
+### Custom Models
+
+Create custom model classes by inheriting from the base Model class:
+
+```python
+from vlm_engine.models import Model
+
+class CustomModel(Model):
+    async def process(self, inputs):
+        # Your custom processing logic
+        return results
+```
+
+### Custom Pipelines
+
+Define custom pipelines for specific use cases:
+
+```python
+custom_pipeline = PipelineConfig(
+    inputs=["image_path"],
+    output="analysis",
+    models=[
+        {"name": "preprocessor", "inputs": ["image_path"], "outputs": "processed_image"},
+        {"name": "analyzer", "inputs": ["processed_image"], "outputs": "analysis"}
+    ]
+)
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Connection Errors**
+   - Ensure your VLM server is running and accessible
+   - Check the `api_base_url` configuration
+   - Verify firewall settings
+
+2. **GPU Memory Errors**
+   - Reduce batch size or frame interval
+   - Ensure proper CUDA installation
+   - Check GPU memory availability
+
+3. **Slow Processing**
+   - Increase frame interval for faster processing
+   - Use GPU acceleration if available
+   - Optimize VLM server settings
+
+### Logging
+
+Enable debug logging for troubleshooting:
+
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
+
+### Development Setup
+
+```bash
+git clone https://github.com/yourusername/vlm-engine.git
+cd vlm-engine
+pip install -e ".[dev]"
+```
+
+### Running Tests
+
+```bash
+pytest tests/
+```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details.
+
+## Acknowledgments
+
+- Built on top of modern Python async patterns
+- Inspired by production ML serving architectures
+- Haven's custom VLM models trained using [SmolVLM-Finetune](https://github.com/Haven-hvn/SmolVLM-Finetune) - Model Download found on [https://havenmodels.orbiter.website/](https://havenmodels.orbiter.website/)
+
+- Designed for integration with OpenAI-compatible VLM endpoints
+
+## Support
+
+For issues and feature requests, please use the [GitHub issue tracker](https://github.com/Haven-hvn/haven-vlm-engine-package/issues).
+
+For questions and discussions, join our community:
+- Discord: [Link to Discord](https://discord.gg/57mPMDfQew)
+
+---
+
+**Note**: This package requires an OpenAI-compatible VLM endpoint. Options include:
+
+### Remote Services
+- Any OpenAI-compatible API endpoint 
+- Akash deployment - https://github.com/Haven-hvn/haven-inference
+
+### Local Setup
+- [LM Studio](https://lmstudio.ai/) - Easy local VLM hosting with OpenAI-compatible API
+
+The package **does not** load VLM models directly - it communicates with external VLM services via API.

vlm_engine-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vlm_engine-0.1.0/setup.py

@@ -0,0 +1,32 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="vlm_engine",
+    version="0.1.0",
+    description="Advanced Vision-Language Model Engine for content tagging",
+    long_description=open("README.md").read(),
+    long_description_content_type="text/markdown",
+    author="HAVEN Network",
+    author_email="officialhavennetwork@gmail.com",
+    url="https://github.com/Haven-hvn/haven-vlm-engine-package",
+    packages=find_packages(),
+    install_requires=[
+        "pydantic",
+        "numpy",
+        "torch",
+        "torchvision",
+        "aiohttp",
+        "pyyaml",
+        "opencv-python"
+    ],
+    classifiers=[
+        "Development Status :: 3 - Alpha",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: MIT License",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+    ],
+    python_requires=">=3.8",
+)

vlm_engine-0.1.0/vlm_engine/__init__.py

@@ -0,0 +1,11 @@
+"""HAVEN VLM Engine: Advanced Content Tagging System
+
+This package provides an advanced Vision-Language Model implementation
+for automatic content tagging, delivering superior accuracy compared
+to traditional image classification methods.
+"""
+
+from .engine import VLMEngine
+
+__all__ = ["VLMEngine"]
+__version__ = "0.1.0"