mlx-omni-server 0.1.0__tar.gz

mlx_omni_server-0.1.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.1
+Name: mlx-omni-server
+Version: 0.1.0
+Summary: 
+Home-page: https://github.com/madroidmaq/mlx-omni-server
+License: MIT
+Keywords: mlx,ai,agi,aigc,server,openai,tts,stt
+Author: madroid
+Author-email: madroidmaq@gmail.com
+Requires-Python: >=3.11,<4.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: diffusionkit (>=0.5.1,<0.6.0)
+Requires-Dist: f5-tts-mlx (>=0.1.7,<0.2.0)
+Requires-Dist: fastapi (>=0.115.4,<0.116.0)
+Requires-Dist: huggingface-hub (>=0.26.2,<0.27.0)
+Requires-Dist: mlx-lm (>=0.20.0,<0.21.0)
+Requires-Dist: mlx-whisper (>=0.4.1,<0.5.0)
+Requires-Dist: numba (>=0.57.0)
+Requires-Dist: pydantic (>=2.9.2,<3.0.0)
+Requires-Dist: python-multipart (>=0.0.17,<0.0.18)
+Requires-Dist: sse-starlette (>=2.1.3,<3.0.0)
+Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
+Project-URL: Repository, https://github.com/madroidmaq/mlx-omni-server
+Description-Content-Type: text/markdown
+
+# MLX Omni Server
+
+MLX Omni Server is a local inference server powered by Apple's MLX framework, specifically designed for Apple Silicon (M-series) chips. It implements
+OpenAI-compatible API endpoints, enabling seamless integration with existing OpenAI SDK clients while leveraging the power of local ML inference.
+
+## Features
+
+- 🚀 **Apple Silicon Optimized**: Built on MLX framework, optimized for M1/M2/M3/M4 series chips
+- 🔌 **OpenAI API Compatible**: Drop-in replacement for OpenAI API endpoints
+- 🎯 **Multiple AI Capabilities**:
+    - Audio Processing:
+        - Text-to-Speech (TTS)
+        - Speech-to-Text (STT/ASR)
+    - Chat Completion
+    - Image Generation
+- ⚡ **High Performance**: Local inference with hardware acceleration
+- 🔐 **Privacy-First**: All processing happens locally on your machine
+- 🛠 **SDK Support**: Works with official OpenAI SDK and other compatible clients
+
+## Support API Endpoints
+
+The server implements OpenAI-compatible endpoints:
+
+- [Chat](https://platform.openai.com/docs/api-reference/chat)
+    - 🚧 `/v1/chat/completions` - Chat completions
+- [Audio](https://platform.openai.com/docs/api-reference/audio)
+    - ✅ `/v1/audio/speech` - Text-to-Speech
+    - ✅ `/v1/audio/transcriptions` - Speech-to-Text
+- [Models](https://platform.openai.com/docs/api-reference/models/list)
+    - ✅ `/v1/models` - List models
+    - ✅ `/v1/models/{model}` - Retrieve or Delete model
+- [Images](https://platform.openai.com/docs/api-reference/images)
+    - ✅ `/v1/images/generations` - Image generation
+
+## Installation
+
+```bash
+# Install using pip
+pip install mlx-omni-server
+
+# Or install using poetry
+poetry add mlx-omni-server
+```
+
+## Quick Start
+
+1. Start the server:
+
+```bash
+# If installed via pip as a package
+mlx-omni-server start
+
+# If installed via poetry (recommended during development)
+poetry run start
+```
+
+2. Use with OpenAI SDK:
+
+```python
+from openai import OpenAI
+
+# Configure client to use local server
+client = OpenAI(
+    base_url="http://localhost:10240/v1",  # Point to local server
+    api_key="not-needed"  # API key is not required for local server
+)
+
+# Text-to-Speech Example
+response = client.audio.speech.create(
+    model="lucasnewman/f5-tts-mlx",
+    input="Hello, welcome to MLX Omni Server!"
+)
+
+# Speech-to-Text Example
+audio_file = open("speech.mp3", "rb")
+transcript = client.audio.transcriptions.create(
+    model="mlx-community/whisper-large-v3-turbo",
+    file=audio_file
+)
+
+# Chat Completion Example
+chat_completion = client.chat.completions.create(
+    model="meta-llama/Llama-3.2-3B-Instruct",
+    messages=[
+        {"role": "user", "content": "What can you do?"}
+    ]
+)
+
+# Image Generation Example
+image_response = client.images.generate(
+    model="argmaxinc/mlx-FLUX.1-schnell",
+    prompt="A serene landscape with mountains and a lake",
+    n=1,
+    size="512x512"
+)
+```
+
+You can view more examples in [examples](examples).
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to
+change.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- Built with [MLX](https://github.com/ml-explore/mlx) by Apple
+- API design inspired by [OpenAI](https://openai.com)
+- Uses [FastAPI](https://fastapi.tiangolo.com/) for the server implementation
+- Text-to-Speech powered by [lucasnewman/f5-tts-mlx](https://github.com/lucasnewman/f5-tts-mlx)
+- Speech-to-Text powered by [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms/mlx_lm)
+
+## Disclaimer
+
+This project is not affiliated with or endorsed by OpenAI or Apple. It's an independent implementation that provides OpenAI-compatible APIs using
+Apple's MLX framework.
+

mlx_omni_server-0.1.0/README.md

@@ -0,0 +1,120 @@
+# MLX Omni Server
+
+MLX Omni Server is a local inference server powered by Apple's MLX framework, specifically designed for Apple Silicon (M-series) chips. It implements
+OpenAI-compatible API endpoints, enabling seamless integration with existing OpenAI SDK clients while leveraging the power of local ML inference.
+
+## Features
+
+- 🚀 **Apple Silicon Optimized**: Built on MLX framework, optimized for M1/M2/M3/M4 series chips
+- 🔌 **OpenAI API Compatible**: Drop-in replacement for OpenAI API endpoints
+- 🎯 **Multiple AI Capabilities**:
+    - Audio Processing:
+        - Text-to-Speech (TTS)
+        - Speech-to-Text (STT/ASR)
+    - Chat Completion
+    - Image Generation
+- ⚡ **High Performance**: Local inference with hardware acceleration
+- 🔐 **Privacy-First**: All processing happens locally on your machine
+- 🛠 **SDK Support**: Works with official OpenAI SDK and other compatible clients
+
+## Support API Endpoints
+
+The server implements OpenAI-compatible endpoints:
+
+- [Chat](https://platform.openai.com/docs/api-reference/chat)
+    - 🚧 `/v1/chat/completions` - Chat completions
+- [Audio](https://platform.openai.com/docs/api-reference/audio)
+    - ✅ `/v1/audio/speech` - Text-to-Speech
+    - ✅ `/v1/audio/transcriptions` - Speech-to-Text
+- [Models](https://platform.openai.com/docs/api-reference/models/list)
+    - ✅ `/v1/models` - List models
+    - ✅ `/v1/models/{model}` - Retrieve or Delete model
+- [Images](https://platform.openai.com/docs/api-reference/images)
+    - ✅ `/v1/images/generations` - Image generation
+
+## Installation
+
+```bash
+# Install using pip
+pip install mlx-omni-server
+
+# Or install using poetry
+poetry add mlx-omni-server
+```
+
+## Quick Start
+
+1. Start the server:
+
+```bash
+# If installed via pip as a package
+mlx-omni-server start
+
+# If installed via poetry (recommended during development)
+poetry run start
+```
+
+2. Use with OpenAI SDK:
+
+```python
+from openai import OpenAI
+
+# Configure client to use local server
+client = OpenAI(
+    base_url="http://localhost:10240/v1",  # Point to local server
+    api_key="not-needed"  # API key is not required for local server
+)
+
+# Text-to-Speech Example
+response = client.audio.speech.create(
+    model="lucasnewman/f5-tts-mlx",
+    input="Hello, welcome to MLX Omni Server!"
+)
+
+# Speech-to-Text Example
+audio_file = open("speech.mp3", "rb")
+transcript = client.audio.transcriptions.create(
+    model="mlx-community/whisper-large-v3-turbo",
+    file=audio_file
+)
+
+# Chat Completion Example
+chat_completion = client.chat.completions.create(
+    model="meta-llama/Llama-3.2-3B-Instruct",
+    messages=[
+        {"role": "user", "content": "What can you do?"}
+    ]
+)
+
+# Image Generation Example
+image_response = client.images.generate(
+    model="argmaxinc/mlx-FLUX.1-schnell",
+    prompt="A serene landscape with mountains and a lake",
+    n=1,
+    size="512x512"
+)
+```
+
+You can view more examples in [examples](examples).
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to
+change.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- Built with [MLX](https://github.com/ml-explore/mlx) by Apple
+- API design inspired by [OpenAI](https://openai.com)
+- Uses [FastAPI](https://fastapi.tiangolo.com/) for the server implementation
+- Text-to-Speech powered by [lucasnewman/f5-tts-mlx](https://github.com/lucasnewman/f5-tts-mlx)
+- Speech-to-Text powered by [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms/mlx_lm)
+
+## Disclaimer
+
+This project is not affiliated with or endorsed by OpenAI or Apple. It's an independent implementation that provides OpenAI-compatible APIs using
+Apple's MLX framework.

mlx_omni_server-0.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+[tool.poetry]
+name = "mlx-omni-server"
+version = "0.1.0"
+description = ""
+authors = ["madroid <madroidmaq@gmail.com>"]
+readme = "README.md"
+license = "MIT"
+repository = "https://github.com/madroidmaq/mlx-omni-server"
+keywords = ["mlx", "ai", "agi", "aigc", "server", "openai", "tts", "stt"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3.11",
+]
+packages = [{ include = "mlx_omni_server", from = "src" }]
+
+[tool.poetry.dependencies]
+python = "^3.11"
+fastapi = "^0.115.4"
+python-multipart = "^0.0.17"
+pydantic = "^2.9.2"
+f5-tts-mlx = "^0.1.7"
+uvicorn = "^0.32.0"
+numba = ">=0.57.0"
+mlx-whisper = "^0.4.1"
+mlx-lm = "^0.20.0"
+huggingface-hub = "^0.26.2"
+diffusionkit = "^0.5.1"
+sse-starlette = "^2.1.3"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^8.3.3"
+httpx = "^0.27.2"
+pre-commit = "^4.0.1"
+black = "^24.10.0"
+isort = "^5.13.2"
+
+[tool.poetry.scripts]
+start = "mlx_omni_server.main:start"
+mlx-omni-server = "mlx_omni_server.main:start"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

mlx_omni_server-0.1.0/src/mlx_omni_server/api/endpoints/chat.py

@@ -0,0 +1,44 @@
+import json
+from typing import Generator
+
+from fastapi import APIRouter
+from fastapi.responses import JSONResponse, StreamingResponse
+
+from ...schemas.chat_schema import ChatCompletionRequest, ChatCompletionResponse
+from ...services.chat.models import load_model
+from ...services.chat_service import ChatService
+
+router = APIRouter(tags=["chat—completions"])
+
+
+@router.post("/chat/completions", response_model=ChatCompletionResponse)
+@router.post("/v1/chat/completions", response_model=ChatCompletionResponse)
+async def create_chat_completion(request: ChatCompletionRequest):
+    """Create a chat completion"""
+
+    chat_service = _create_chat_service(request.model)
+
+    if not request.stream:
+        completion = await chat_service.generate_completion(request)
+        return JSONResponse(content=completion.model_dump(exclude_none=True))
+
+    async def event_generator() -> Generator[str, None, None]:
+        async for chunk in chat_service.generate_stream(request):
+            if chunk.choices[0].finish_reason == "stop":
+                yield "data: [DONE]\n\n"
+            else:
+                yield f"data: {json.dumps(chunk.model_dump(exclude_none=True))}\n\n"
+
+    return StreamingResponse(
+        event_generator(),
+        media_type="text/event-stream",
+        headers={
+            "Cache-Control": "no-cache",
+            "Connection": "keep-alive",
+        },
+    )
+
+
+def _create_chat_service(model_id: str):
+    model = load_model(model_id)
+    return ChatService(model)

mlx_omni_server-0.1.0/src/mlx_omni_server/api/endpoints/images.py

@@ -0,0 +1,29 @@
+import time
+
+from fastapi import APIRouter, HTTPException
+
+from ...schemas.images_schema import ImageGenerationRequest, ImageGenerationResponse
+from ...services.images_service import ImagesService
+
+router = APIRouter(tags=["images"])
+
+
+@router.post("/images/generations")
+@router.post("/v1/images/generations")
+async def create_image(request: ImageGenerationRequest) -> ImageGenerationResponse:
+    """
+    Creates an image given a prompt.
+    """
+    try:
+        service = ImagesService()
+
+        # Generate images
+        images = service.generate_images(request)
+
+        # Create response
+        return ImageGenerationResponse(created=int(time.time()), data=images)
+
+    except ValueError as ve:
+        raise HTTPException(status_code=400, detail=str(ve))
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))

mlx_omni_server-0.1.0/src/mlx_omni_server/api/endpoints/models.py

@@ -0,0 +1,65 @@
+from fastapi import APIRouter, HTTPException, Request
+
+from ...schemas.models_schema import Model, ModelDeletion, ModelList
+from ...services.models_service import ModelsService
+
+router = APIRouter(tags=["models"])
+models_service = ModelsService()
+
+
+def extract_model_id_from_path(request: Request) -> str:
+    """Extract full model ID from request path"""
+    path = request.url.path
+    prefix = "/v1/models/" if "/v1/models/" in path else "/models/"
+    return path[len(prefix) :]
+
+
+def handle_model_error(e: Exception) -> None:
+    """Handle model-related errors and raise appropriate HTTP exceptions"""
+    if isinstance(e, ValueError):
+        raise HTTPException(status_code=404, detail=str(e))
+    print(f"Error processing request: {str(e)}")
+    raise HTTPException(status_code=500, detail=str(e))
+
+
+@router.get("/models", response_model=ModelList)
+@router.get("/v1/models", response_model=ModelList)
+async def list_models() -> ModelList:
+    """
+    Lists the currently available models, and provides basic information about each one
+    such as the owner and availability.
+    """
+    try:
+        return models_service.list_models()
+    except Exception as e:
+        handle_model_error(e)
+
+
+@router.get("/models/{model_id:path}", response_model=Model)
+@router.get("/v1/models/{model_id:path}", response_model=Model)
+async def get_model(request: Request) -> Model:
+    """
+    Retrieves a model instance, providing basic information about the model such as
+    the owner and permissioning.
+    """
+    try:
+        model_id = extract_model_id_from_path(request)
+        model = models_service.get_model(model_id)
+        if model is None:
+            raise ValueError(f"Model '{model_id}' not found")
+        return model
+    except Exception as e:
+        handle_model_error(e)
+
+
+@router.delete("/models/{model_id:path}", response_model=ModelDeletion)
+@router.delete("/v1/models/{model_id:path}", response_model=ModelDeletion)
+async def delete_model(request: Request) -> ModelDeletion:
+    """
+    Delete a fine-tuned model from local cache.
+    """
+    try:
+        model_id = extract_model_id_from_path(request)
+        return models_service.delete_model(model_id)
+    except Exception as e:
+        handle_model_error(e)

mlx_omni_server-0.1.0/src/mlx_omni_server/api/endpoints/stt.py

@@ -0,0 +1,40 @@
+from fastapi import APIRouter, Depends, HTTPException
+from fastapi.responses import JSONResponse, Response
+from starlette.responses import PlainTextResponse
+
+from mlx_omni_server.schemas.stt_schema import (
+    ResponseFormat,
+    STTRequestForm,
+    TranscriptionResponse,
+)
+from mlx_omni_server.services.stt_service import STTService
+
+router = APIRouter(tags=["speech-to-text"])
+
+
+@router.post("/audio/transcriptions", response_model=TranscriptionResponse)
+@router.post("/v1/audio/transcriptions", response_model=TranscriptionResponse)
+async def create_transcription(request: STTRequestForm = Depends()):
+    """
+    Transcribe audio file to text.
+    """
+    stt_service = STTService()
+    try:
+        result = await stt_service.transcribe(request)
+
+        # Return appropriate response based on format
+        if request.response_format == ResponseFormat.TEXT:
+            return PlainTextResponse(content=result)
+        elif request.response_format in (ResponseFormat.SRT, ResponseFormat.VTT):
+            return Response(
+                content=result,
+                media_type="text/plain",
+                headers={
+                    "Content-Disposition": f'attachment; filename="transcription.{request.response_format.value.lower()}"'
+                },
+            )
+        else:  # JSON and VERBOSE_JSON
+            return JSONResponse(content=result)
+
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))

mlx_omni_server-0.1.0/src/mlx_omni_server/api/endpoints/tts.py

@@ -0,0 +1,48 @@
+import io
+
+from fastapi import APIRouter, HTTPException
+from fastapi.responses import StreamingResponse
+
+from mlx_omni_server.schemas.tts_schema import AudioFormat, TTSRequest
+from mlx_omni_server.services.tts_service import TTSService
+
+router = APIRouter(tags=["text-to-speech"])
+
+
+@router.post("/audio/speech")
+@router.post("/v1/audio/speech")
+async def create_speech(request: TTSRequest):
+    """
+    Generate audio from input text.
+
+    Returns:
+        StreamingResponse: Audio file content in the requested format
+    """
+    tts_service = TTSService()
+
+    try:
+        audio_content = await tts_service.generate_speech(
+            request=request,
+        )
+
+        # Create content type mapping
+        content_type_mapping = {
+            AudioFormat.MP3: "audio/mpeg",
+            AudioFormat.OPUS: "audio/opus",
+            AudioFormat.AAC: "audio/aac",
+            AudioFormat.FLAC: "audio/flac",
+            AudioFormat.WAV: "audio/wav",
+            AudioFormat.PCM: "audio/pcm",
+        }
+
+        # Create response
+        return StreamingResponse(
+            io.BytesIO(audio_content),
+            media_type=content_type_mapping[request.response_format],
+            headers={
+                "Content-Disposition": f'attachment; filename="speech.{request.response_format.value}"'
+            },
+        )
+
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))

mlx_omni_server-0.1.0/src/mlx_omni_server/api/routers.py

@@ -0,0 +1,10 @@
+from fastapi import APIRouter
+
+from .endpoints import chat, images, models, stt, tts
+
+api_router = APIRouter()
+api_router.include_router(stt.router)
+api_router.include_router(tts.router)
+api_router.include_router(models.router)
+api_router.include_router(images.router)
+api_router.include_router(chat.router)

mlx_omni_server-0.1.0/src/mlx_omni_server/main.py

@@ -0,0 +1,23 @@
+import logging
+
+import uvicorn
+from fastapi import FastAPI
+
+from .api.routers import api_router
+from .middleware.logging import RequestResponseLoggingMiddleware
+
+app = FastAPI(title="MLX Omni Server")
+
+# Add request/response logging middleware with custom levels
+app.add_middleware(
+    RequestResponseLoggingMiddleware,
+    request_level=logging.DEBUG,
+    response_level=logging.DEBUG,
+    # exclude_paths=["/health"]
+)
+
+app.include_router(api_router)
+
+
+def start():
+    uvicorn.run("mlx_omni_server.main:app", host="0.0.0.0", port=10240, reload=True)