fount-vlm-nell-02 0.3.11__tar.gz

fount_vlm_nell_02-0.3.11/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright © 2025 Prince Canuma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fount_vlm_nell_02-0.3.11/PKG-INFO

@@ -0,0 +1,418 @@
+Metadata-Version: 2.4
+Name: fount-vlm-nell-02
+Version: 0.3.11
+Summary: fork of mlx-vlm for fount
+License: MIT
+Project-URL: Homepage, https://github.com/Blaizzy/mlx-vlm
+Project-URL: Repository, https://github.com/Blaizzy/mlx-vlm
+Project-URL: Issues, https://github.com/Blaizzy/mlx-vlm/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mlx>=0.30.0
+Requires-Dist: datasets>=2.19.1
+Requires-Dist: tqdm>=4.66.2
+Requires-Dist: transformers>=5.0.0rc3
+Requires-Dist: mlx-lm>=0.30.5
+Requires-Dist: Pillow>=10.3.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: fastapi>=0.95.1
+Requires-Dist: soundfile>=0.13.1
+Requires-Dist: opencv-python>=4.12.0.88
+Requires-Dist: numpy
+Requires-Dist: uvicorn
+Provides-Extra: ui
+Requires-Dist: gradio>=5.19.0; extra == "ui"
+Provides-Extra: torch
+Requires-Dist: torch; extra == "torch"
+Requires-Dist: torchvision; extra == "torch"
+Requires-Dist: einops; extra == "torch"
+Requires-Dist: blobfile; extra == "torch"
+Requires-Dist: tiktoken; extra == "torch"
+Provides-Extra: cuda
+Requires-Dist: mlx-cuda; extra == "cuda"
+Provides-Extra: cpu
+Requires-Dist: mlx-cpu; extra == "cpu"
+Dynamic: license-file
+
+[![Upload Python Package](https://github.com/Blaizzy/mlx-vlm/actions/workflows/python-publish.yml/badge.svg)](https://github.com/Blaizzy/mlx-vlm/actions/workflows/python-publish.yml)
+# MLX-VLM
+
+MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (VLMs with audio and video support) on your Mac using MLX.
+
+## Table of Contents
+- [Installation](#installation)
+- [Usage](#usage)
+  - [Command Line Interface (CLI)](#command-line-interface-cli)
+  - [Chat UI with Gradio](#chat-ui-with-gradio)
+  - [Python Script](#python-script)
+- [Multi-Image Chat Support](#multi-image-chat-support)
+  - [Supported Models](#supported-models)
+  - [Usage Examples](#usage-examples)
+- [Model-Specific Documentation](#model-specific-documentation)
+- [Fine-tuning](#fine-tuning)
+
+## Model-Specific Documentation
+
+Some models have detailed documentation with prompt formats, examples, and best practices:
+
+| Model | Documentation |
+|-------|---------------|
+| DeepSeek-OCR | [Docs](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/deepseekocr/README.md) |
+| DeepSeek-OCR-2 | [Docs](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/deepseekocr_2/README.md) |
+| GLM-OCR | [Docs](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/glm_ocr/README.md) |
+
+## Installation
+
+The easiest way to get started is to install the `mlx-vlm` package using pip:
+
+```sh
+pip install -U mlx-vlm
+```
+
+## Usage
+
+### Command Line Interface (CLI)
+
+Generate output from a model using the CLI:
+
+```sh
+# Text generation
+mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"
+
+# Image generation
+mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg
+
+# Audio generation (New)
+mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav
+
+# Multi-modal generation (Image + Audio)
+mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav
+```
+
+### Chat UI with Gradio
+
+Launch a chat interface using Gradio:
+
+```sh
+mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bit
+```
+
+### Python Script
+
+Here's an example of how to use MLX-VLM in a Python script:
+
+```python
+import mlx.core as mx
+from mlx_vlm import load, generate
+from mlx_vlm.prompt_utils import apply_chat_template
+from mlx_vlm.utils import load_config
+
+# Load the model
+model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
+model, processor = load(model_path)
+config = load_config(model_path)
+
+# Prepare input
+image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
+# image = [Image.open("...")] can also be used with PIL.Image.Image objects
+prompt = "Describe this image."
+
+# Apply chat template
+formatted_prompt = apply_chat_template(
+    processor, config, prompt, num_images=len(image)
+)
+
+# Generate output
+output = generate(model, processor, formatted_prompt, image, verbose=False)
+print(output)
+```
+
+#### Audio Example
+
+```python
+from mlx_vlm import load, generate
+from mlx_vlm.prompt_utils import apply_chat_template
+from mlx_vlm.utils import load_config
+
+# Load model with audio support
+model_path = "mlx-community/gemma-3n-E2B-it-4bit"
+model, processor = load(model_path)
+config = model.config
+
+# Prepare audio input
+audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
+prompt = "Describe what you hear in these audio files."
+
+# Apply chat template with audio
+formatted_prompt = apply_chat_template(
+    processor, config, prompt, num_audios=len(audio)
+)
+
+# Generate output with audio
+output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
+print(output)
+```
+
+#### Multi-Modal Example (Image + Audio)
+
+```python
+from mlx_vlm import load, generate
+from mlx_vlm.prompt_utils import apply_chat_template
+from mlx_vlm.utils import load_config
+
+# Load multi-modal model
+model_path = "mlx-community/gemma-3n-E2B-it-4bit"
+model, processor = load(model_path)
+config = model.config
+
+# Prepare inputs
+image = ["/path/to/image.jpg"]
+audio = ["/path/to/audio.wav"]
+prompt = ""
+
+# Apply chat template
+formatted_prompt = apply_chat_template(
+    processor, config, prompt,
+    num_images=len(image),
+    num_audios=len(audio)
+)
+
+# Generate output
+output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
+print(output)
+```
+
+### Server (FastAPI)
+
+Start the server:
+```sh
+mlx_vlm.server --port 8080
+
+# With trust remote code enabled (required for some models)
+mlx_vlm.server --trust-remote-code
+```
+
+#### Server Options
+
+- `--host`: Host address (default: `0.0.0.0`)
+- `--port`: Port number (default: `8080`)
+- `--trust-remote-code`: Trust remote code when loading models from Hugging Face Hub
+
+You can also set trust remote code via environment variable:
+```sh
+MLX_TRUST_REMOTE_CODE=true mlx_vlm.server
+```
+
+The server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).
+
+#### Available Endpoints
+
+- `/models` - List models available locally
+- `/chat/completions` - OpenAI-compatible chat-style interaction endpoint with support for images, audio, and text
+- `/responses` - OpenAI-compatible responses endpoint
+- `/health` - Check server status
+- `/unload` - Unload current model from memory
+
+#### Usage Examples
+
+##### List available models
+
+```sh
+curl "http://localhost:8080/models"
+```
+
+##### Text Input
+
+```sh
+curl -X POST "http://localhost:8080/chat/completions" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
+    "messages": [
+      {
+        "role": "user",
+        "content": "Hello, how are you",
+      }
+    ],
+    "stream": true,
+    "max_tokens": 100
+  }'
+```
+
+##### Image Input
+
+```sh
+curl -X POST "http://localhost:8080/chat/completions" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mlx-community/Qwen2.5-VL-32B-Instruct-8bit",
+    [
+      {
+        "role": "system",
+        "content": "You are a helpful assistant."
+      },
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": This is today's chart for energy demand in California. Can you provide an analysis of the chart and comment on the implications for renewable energy in California?"
+          },
+          {
+            "type": "input_image",
+            "image_url": "/path/to/repo/examples/images/renewables_california.png"
+          }
+        ]
+      }
+    ],
+    "stream": true,
+    "max_tokens": 1000
+  }'
+```
+
+##### Audio Support (New)
+```sh
+curl -X POST "http://localhost:8080/generate" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mlx-community/gemma-3n-E2B-it-4bit",
+    "messages": [
+      {
+        "role": "user",
+        "content": [
+          { "type": "text", "text": "Describe what you hear in these audio files" },
+          {"type": "input_audio", "input_audio": "/path/to/audio1.wav"}
+          {"type": "input_audio", "input_audio": "https://example.com/audio2.mp3"}
+        ]
+      }
+    ],
+    "stream": true,
+    "max_tokens": 500
+  }'
+```
+
+##### Multi-Modal (Image + Audio)
+```sh
+curl -X POST "http://localhost:8080/generate" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mlx-community/gemma-3n-E2B-it-4bit",
+    "messages": [
+      {
+        "role": "user",
+        "content": [
+          {"type": "input_image", "image_url": "/path/to/image.jpg"},
+          {"type": "input_audio", "input_audio": "/path/to/audio.wav"}
+        ]
+      }
+    ],
+    "max_tokens": 100
+  }'
+```
+
+##### Responses Endpoint
+```sh
+curl -X POST "http://localhost:8080/responses" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
+    "messages": [
+      {
+        "role": "user",
+        "content": [
+          {"type": "input_text", "text": "What is in this image?"},
+          {"type": "input_image", "image_url": "/path/to/image.jpg"}
+        ]
+      }
+    ],
+    "max_tokens": 100
+  }'
+```
+
+#### Request Parameters
+
+- `model`: Model identifier (required)
+- `messages`: Chat messages for chat/OpenAI endpoints
+- `max_tokens`: Maximum tokens to generate
+- `temperature`: Sampling temperature
+- `top_p`: Top-p sampling parameter
+- `stream`: Enable streaming responses
+
+
+## Multi-Image Chat Support
+
+MLX-VLM supports analyzing multiple images simultaneously with select models. This feature enables more complex visual reasoning tasks and comprehensive analysis across multiple images in a single conversation.
+
+
+### Usage Examples
+
+#### Python Script
+
+```python
+from mlx_vlm import load, generate
+from mlx_vlm.prompt_utils import apply_chat_template
+from mlx_vlm.utils import load_config
+
+model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
+model, processor = load(model_path)
+config = model.config
+
+images = ["path/to/image1.jpg", "path/to/image2.jpg"]
+prompt = "Compare these two images."
+
+formatted_prompt = apply_chat_template(
+    processor, config, prompt, num_images=len(images)
+)
+
+output = generate(model, processor, formatted_prompt, images, verbose=False)
+print(output)
+```
+
+#### Command Line
+
+```sh
+mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Compare these images" --image path/to/image1.jpg path/to/image2.jpg
+```
+
+## Video Understanding
+
+MLX-VLM also supports video analysis such as captioning, summarization, and more, with select models.
+
+### Supported Models
+
+The following models support video chat:
+
+1. Qwen2-VL
+2. Qwen2.5-VL
+3. Idefics3
+4. LLaVA
+
+With more coming soon.
+
+### Usage Examples
+
+#### Command Line
+```sh
+mlx_vlm.video_generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Describe this video" --video path/to/video.mp4 --max-pixels 224 224 --fps 1.0
+```
+
+
+These examples demonstrate how to use multiple images with MLX-VLM for more complex visual reasoning tasks.
+
+# Fine-tuning
+
+MLX-VLM supports fine-tuning models with LoRA and QLoRA.
+
+## LoRA & QLoRA
+
+To learn more about LoRA, please refer to the [LoRA.md](./mlx_vlm/LORA.MD) file.