pi-llm-server 1.1.2__tar.gz

pi_llm_server-1.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 布树辉
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pi_llm_server-1.1.2/PKG-INFO

@@ -0,0 +1,505 @@
+Metadata-Version: 2.4
+Name: pi-llm-server
+Version: 1.1.2
+Summary: 统一 LLM 服务网关 - 集成 Embedding、ASR、Reranker、MinerU 服务
+Author-email: PI-Lab Team <bushuhui@foxmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/bushuhui/pi-llm-server
+Project-URL: Documentation, https://github.com/bushuhui/pi-llm-server
+Project-URL: Repository, https://github.com/bushuhui/pi-llm-server
+Project-URL: Changelog, https://github.com/bushuhui/pi-llm-server/blob/main/CHANGELOG.md
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: uvicorn[standard]>=0.30.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: starlette>=0.40.0
+Requires-Dist: python-multipart>=0.0.18
+Requires-Dist: pydantic>=2.8.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: numpy>=2.0.0
+Requires-Dist: pandas>=2.2.0
+Requires-Dist: scipy>=1.13.0
+Requires-Dist: scikit-learn>=1.5.0
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: orjson>=3.9.0
+Requires-Dist: supervisor>=4.2.0
+Requires-Dist: setproctitle>=1.3.0
+Provides-Extra: vllm
+Requires-Dist: vllm<1.0.0,>=0.14.0; extra == "vllm"
+Provides-Extra: embedding
+Requires-Dist: sentence-transformers>=5.0.0; extra == "embedding"
+Requires-Dist: transformers>=4.50.0; extra == "embedding"
+Requires-Dist: accelerate>=1.0.0; extra == "embedding"
+Requires-Dist: torch>=2.0.0; extra == "embedding"
+Provides-Extra: reranker
+Requires-Dist: transformers>=4.50.0; extra == "reranker"
+Requires-Dist: accelerate>=1.0.0; extra == "reranker"
+Requires-Dist: torch>=2.0.0; extra == "reranker"
+Provides-Extra: asr
+Requires-Dist: qwen-asr[vllm]>=0.0.5; extra == "asr"
+Requires-Dist: silero-vad>=5.1.0; extra == "asr"
+Requires-Dist: librosa>=0.10.0; extra == "asr"
+Requires-Dist: soundfile>=0.12.0; extra == "asr"
+Requires-Dist: pydub>=0.25.0; extra == "asr"
+Requires-Dist: onnxruntime-gpu>=1.16.0; extra == "asr"
+Provides-Extra: mineru
+Requires-Dist: mineru>=2.0.0; extra == "mineru"
+Requires-Dist: mineru-vl-utils>=0.1.0; extra == "mineru"
+Requires-Dist: qwen-vl-utils>=0.0.10; extra == "mineru"
+Requires-Dist: pdfminer-six>=20240706; extra == "mineru"
+Requires-Dist: pdftext>=0.6.0; extra == "mineru"
+Requires-Dist: pypdf>=5.0.0; extra == "mineru"
+Requires-Dist: pypdfium2>=4.30.0; extra == "mineru"
+Requires-Dist: gradio-pdf>=0.0.20; extra == "mineru"
+Requires-Dist: doclayout-yolo>=0.0.4; extra == "mineru"
+Requires-Dist: ultralytics>=8.0.0; extra == "mineru"
+Requires-Dist: ftfy>=6.0.0; extra == "mineru"
+Requires-Dist: pyclipper>=1.2.0; extra == "mineru"
+Requires-Dist: omegaconf>=2.3.0; extra == "mineru"
+Requires-Dist: pillow>=10.0.0; extra == "mineru"
+Requires-Dist: opencv-python-headless>=4.10.0; extra == "mineru"
+Requires-Dist: scikit-image>=0.25.0; extra == "mineru"
+Requires-Dist: albumentations>=2.0.0; extra == "mineru"
+Requires-Dist: einops>=0.8.0; extra == "mineru"
+Requires-Dist: shapely>=2.0.0; extra == "mineru"
+Provides-Extra: models
+Requires-Dist: transformers>=4.50.0; extra == "models"
+Requires-Dist: accelerate>=1.0.0; extra == "models"
+Requires-Dist: sentence-transformers>=5.0.0; extra == "models"
+Requires-Dist: huggingface-hub>=0.30.0; extra == "models"
+Requires-Dist: tokenizers>=0.20.0; extra == "models"
+Requires-Dist: sentencepiece>=0.2.0; extra == "models"
+Requires-Dist: safetensors>=0.4.0; extra == "models"
+Provides-Extra: api
+Requires-Dist: openai>=1.40.0; extra == "api"
+Requires-Dist: anthropic>=0.40.0; extra == "api"
+Requires-Dist: dashscope>=1.20.0; extra == "api"
+Requires-Dist: modelscope>=1.15.0; extra == "api"
+Provides-Extra: monitoring
+Requires-Dist: ray>=2.10.0; extra == "monitoring"
+Requires-Dist: prometheus-fastapi-instrumentator>=7.0.0; extra == "monitoring"
+Requires-Dist: sse-starlette>=2.0.0; extra == "monitoring"
+Requires-Dist: tiktoken>=0.7.0; extra == "monitoring"
+Provides-Extra: utils
+Requires-Dist: numpy>=2.0.0; extra == "utils"
+Requires-Dist: pandas>=2.2.0; extra == "utils"
+Requires-Dist: scipy>=1.13.0; extra == "utils"
+Requires-Dist: scikit-learn>=1.5.0; extra == "utils"
+Requires-Dist: loguru>=0.7.0; extra == "utils"
+Requires-Dist: psutil>=5.9.0; extra == "utils"
+Requires-Dist: orjson>=3.9.0; extra == "utils"
+Requires-Dist: einops>=0.8.0; extra == "utils"
+Requires-Dist: shapely>=2.0.0; extra == "utils"
+Requires-Dist: magika>=1.0.0; extra == "utils"
+Requires-Dist: fast-langdetect>=0.2.0; extra == "utils"
+Requires-Dist: json-repair>=0.50.0; extra == "utils"
+Requires-Dist: supervisor>=4.2.0; extra == "utils"
+Requires-Dist: setproctitle>=1.3.0; extra == "utils"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
+Requires-Dist: black>=24.0.0; extra == "dev"
+Requires-Dist: ruff>=0.5.0; extra == "dev"
+Requires-Dist: mypy>=1.10.0; extra == "dev"
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0.0; extra == "mcp"
+Provides-Extra: all
+Requires-Dist: pi-llm-server[api,asr,dev,embedding,mcp,mineru,models,monitoring,reranker,utils,vllm]; extra == "all"
+Dynamic: license-file
+
+# PI-LLM-Server - 统一 LLM 服务网关
+
+> 为 OpenClaw、Claude Code 等 AI 编程助手提供本地化的 Embedding、ASR、Reranker、OCR 等服务
+
+**问题背景**: 阿里云等 Coding Plan 产品提供了大模型 API，但未包含 Embedding、Reranker、ASR、OCR 等辅助服务。这些模型通常较小，可本地部署以获得更低延迟和更好的隐私保护。
+
+**解决方案**: PI-LLM-Server 统一管理多种本地服务，提供标准化 API 网关，为 AI 编程助手提供一站式服务接入。
+
+---
+
+## 目录
+
+- [项目目的](#项目目的)
+- [快速开始](#快速开始)
+- [模型下载](#模型下载)
+- [使用方法](#使用方法)
+- [配置说明](#配置说明)
+- [API 文档](#api-文档)
+- [关联项目](#关联项目)
+
+---
+
+## 项目目的
+
+PI-LLM-Server 旨在解决以下问题：
+
+1. **服务碎片化**: Embedding、ASR、Reranker、OCR 等服务分散部署，管理复杂
+2. **API 不统一**: 各服务接口风格不一致，集成成本高
+3. **缺少队列管理**: 并发请求可能导致显存溢出或服务崩溃
+4. **缺乏健康监控**: 服务异常时无法自动发现和恢复
+
+通过本项目，您可以：
+
+- 统一管理多个 AI 子服务，提供一致的服务入口
+- 配置请求队列，防止并发过载
+- 启用 API 访问控制，保护服务安全
+- 实时监控服务健康状态
+- 为 OpenClaw、Claude Code 等工具提供本地化服务支持
+
+---
+
+## 快速开始
+
+### 1. 环境准备
+
+详细的 conda 安装可以参考： [安装Python环境](https://github.com/bushuhui/machinelearning_notebook/blob/master/references_tips/InstallPython.md)
+
+```bash
+# 创建 Conda 环境
+conda create -n pi-llm-server python=3.13
+conda activate pi-llm-server
+```
+
+### 2. 安装项目
+
+```bash
+# 方式 1: 进入项目目录安装（推荐）
+cd pi-llm-server
+pip install -e ".[all]"
+
+# 方式 2: 只安装核心服务（按需选择）
+pip install -e ".[embedding,reranker,asr,mineru]"
+```
+
+### 3. 系统依赖
+
+#### CUDA Toolkit 安装（需要 GPU 时使用）
+
+```bash
+# 访问 NVIDIA CUDA 下载页面
+# https://developer.nvidia.com/cuda-toolkit-archive
+
+# 或使用快捷链接（CUDA 12.8）
+# https://developer.nvidia.com/cuda-12-8-1-download-archive
+
+# 安装后创建符号链接（如需要）
+cd /usr/bin
+sudo ln -s /usr/local/cuda-12.8/bin/nvcc nvcc
+
+# 设置环境变量，把如下的内容放入 /etc/bash.bashrc (如果使用的是bash，其他的sh类似)
+export CUDA_HOME=/usr/local/cuda-12.8
+export PATH=$CUDA_HOME/bin:$CUDA_HOME/nvvm/bin:$PATH
+export CPLUS_INCLUDE_PATH=$CUDA_HOME/include:$CPLUS_INCLUDE_PATH
+export LIBRARY_PATH=$CUDA_HOME/lib64:$LIBRARY_PATH
+export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
+```
+
+
+---
+
+## 模型下载
+
+### 方式 1: 使用 ModelScope 下载（推荐国内用户）
+
+```bash
+# Embedding 模型
+modelscope download --model unsloth/Qwen3-Embedding-0.6B
+
+# ASR 模型
+modelscope download --model Qwen/Qwen3-ASR-1.7B
+
+# Reranker 模型
+modelscope download --model Qwen/Qwen3-Reranker-0.6B
+```
+
+### 方式 2: 模型存储位置
+
+ModelScope 模型默认下载到：
+```
+~/.cache/modelscope/hub/models/<组织>/<模型名>
+```
+
+例如：
+- Embedding: `~/.cache/modelscope/hub/models/unsloth/Qwen3-Embedding-4B`
+- ASR: `~/.cache/modelscope/hub/models/Qwen/Qwen3-ASR-1.7B`
+- Reranker: `~/.cache/modelscope/hub/models/Qwen/Qwen3-Reranker-0.6B`
+
+### 方式 3: 使用 HuggingFace（需要网络条件）
+
+```bash
+# 使用 huggingface-cli 下载
+huggingface-cli download unsloth/Qwen3-Embedding-4B --local-dir ~/.cache/huggingface/hub/models/unsloth/Qwen3-Embedding-4B
+```
+
+---
+
+## 使用方法
+
+### 1. 配置目录结构
+
+首次运行时会自动创建配置文件，或手动创建：
+
+```bash
+# 创建配置目录
+mkdir -p ~/.config/pi-llm-server
+
+# 复制示例配置
+cp examples/config.example.yaml ~/.config/pi-llm-server/config.yaml
+```
+
+### 2. 编辑配置文件
+
+编辑 `~/.config/pi-llm-server/config.yaml`，主要配置项：
+
+```yaml
+server:
+  host: "0.0.0.0"
+  port: 8090
+
+auth:
+  enabled: true
+  tokens:
+    - "your-api-token-here"
+
+services:
+  embedding:
+    enabled: true
+    base_url: "http://127.0.0.1:8091"
+  asr:
+    enabled: true
+    base_url: "http://127.0.0.1:8092"
+  reranker:
+    enabled: true
+    base_url: "http://127.0.0.1:8093"
+  mineru:
+    enabled: true
+    base_url: "http://127.0.0.1:8094"
+```
+
+### 3. 启动服务
+
+#### 方式 A: 启动统一网关（推荐）
+
+```bash
+# 使用命令行工具启动网关服务
+pi-llm-server
+
+# 或指定配置
+pi-llm-server --config ~/.config/pi-llm-server/config.yaml --port 8090
+
+# 后台运行
+nohup pi-llm-server > ~/.cache/pi-llm-server/logs/gateway.log 2>&1 &
+```
+
+#### 方式 B: 启动子服务
+
+```bash
+# Embedding 服务
+python pi_llm_server/launcher/embedding_server.py --model-path ~/.cache/modelscope/hub/models/unsloth/Qwen3-Embedding-0.6B --device cpu
+
+# ASR 服务（需要 GPU）
+python pi_llm_server/launcher/asr_server.py --model-path ~/.cache/modelscope/hub/models/Qwen/Qwen3-ASR-1.7B
+
+# Reranker 服务
+python pi_llm_server/launcher/reranker_server.py --model-path ~/.cache/modelscope/hub/models/Qwen/Qwen3-Reranker-0.6B --device cpu
+
+# MinerU 服务（需要在 MinerU 环境中）
+# 编辑 mineru_server.sh 配置后启动
+./mineru_server.sh start
+```
+
+#### 方式 C: 使用服务管理器
+
+```bash
+# 启动所有服务
+python pi_llm_server/launcher/service_manager.py start --all
+
+# 查看服务状态
+python pi_llm_server/launcher/service_manager.py status
+
+# 停止所有服务
+python pi_llm_server/launcher/service_manager.py stop --all
+```
+
+### 4. 验证服务
+
+```bash
+# 健康检查
+curl http://localhost:8090/health
+
+# 列出可用模型
+curl http://localhost:8090/v1/models
+
+# 生成 Embedding
+curl -X POST http://localhost:8090/v1/embeddings \
+  -H "Authorization: Bearer sk-your-token" \
+  -H "Content-Type: application/json" \
+  -d '{"input": "你好，世界！"}'
+```
+
+---
+
+## 配置说明
+
+### 配置文件位置
+
+- 默认路径：`~/.config/pi-llm-server/config.yaml`
+- 可通过 `--config` 参数指定其他路径
+
+### 主要配置项
+
+| 配置项 | 说明 | 默认值 |
+|--------|------|--------|
+| `server.host` | 监听地址 | `0.0.0.0` |
+| `server.port` | 监听端口 | `8090` |
+| `server.workers` | 工作进程数 | `4` |
+| `server.log_level` | 日志级别 | `info` |
+| `auth.enabled` | 是否启用认证 | `true` |
+| `auth.tokens` | 有效 Token 列表 | `[]` |
+| `queue.enabled` | 是否启用队列 | `true` |
+| `services.*.enabled` | 是否启用子服务 | `true` |
+| `services.*.base_url` | 子服务地址 | 需配置 |
+
+### 队列配置策略
+
+| 服务 | 并发数 | 队列大小 | 超时 (秒) | 说明 |
+|------|--------|----------|-----------|------|
+| embedding | 4 | 200 | 60 | CPU 多核并行 |
+| reranker | 4 | 200 | 120 | CPU 多核并行 |
+| asr | 1 | 50 | 600 | GPU 推理顺序处理 |
+| mineru | 1 | 20 | 1800 | PDF 解析耗时 |
+
+### 端口分配
+
+| 服务 | 端口 | 说明 |
+|------|------|------|
+| 统一网关 | 8090 | 主入口 |
+| Embedding | 8091 | 文本向量化 |
+| ASR | 8092 | 语音识别 |
+| Reranker | 8093 | 文档重排序 |
+| MinerU | 8094 | PDF 解析 |
+
+---
+
+## API 文档
+
+### 端点列表
+
+| 端点 | 方法 | 说明 | 认证 |
+|------|------|------|------|
+| `/` | GET | 欢迎信息 | 否 |
+| `/health` | GET | 健康检查 | 否 |
+| `/status` | GET | 详细状态 | 是 |
+| `/v1/models` | GET | 可用模型列表 | 可选 |
+| `/v1/embeddings` | POST | 生成 Embedding | 是 |
+| `/v1/rerank` | POST | 文档重排序 | 是 |
+| `/v1/audio/transcriptions` | POST | 语音转文字 | 是 |
+| `/v1/ocr/parser` | POST | PDF 解析 | 是 |
+| `/docs` | GET | Swagger 文档 | 否 |
+
+### 使用示例
+
+详见 [doc/README_services.md](doc/README_services.md)
+
+---
+
+## 关联项目
+
+### VLLM
+
+[VLLM](https://github.com/vllm-project/vllm) 是一个高吞吐、大吞吐量的 LLM 推理和服务引擎。
+
+- **关系**: PI-LLM-Server 使用 VLLM 作为 ASR 服务的推理后端
+- **区别**: VLLM 专注于 LLM 推理引擎，PI-LLM-Server 专注于服务集成和统一管理
+- **协作**: 可以结合使用，VLLM 提供底层推理能力，PI-LLM-Server 提供上层服务编排
+
+### LocalAI
+
+[LocalAI](https://github.com/mudler/LocalAI) 是一个开源的 OpenAI API 替代品，支持多种模型。
+
+- **关系**: LocalAI 也是提供本地化 AI 服务的项目
+- **区别**:
+  - LocalAI 是"All-in-One"的大而全方案，支持更多模型类型
+  - PI-LLM-Server 更轻量，专注于 Embedding、ASR、Reranker、OCR 等辅助服务
+  - PI-LLM-Server 更适合作为其他服务的补充，而非替代
+- **协作**: 可以与 LocalAI 并存，各自负责不同的服务场景
+
+### 阿里云百炼/通义灵码
+
+- **背景**: 提供大模型 API，但缺少 Embedding、Reranker 等辅助服务
+- **PI-LLM-Server 定位**: 补充这些本地可部署的小模型服务，提供更快的响应速度
+- **优势**:
+  - 本地部署，零网络延迟
+  - 数据隐私，敏感信息不出内网
+  - 成本更低，无需调用付费 API
+
+### 典型架构
+
+```
+┌─────────────────┐
+│  AI 编程助手     │
+│ OpenClaw/Code   │
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ PI-LLM-Server   │ ← 统一网关 (8090)
+│  网关服务        │
+└────────┬────────┘
+         │
+    ┌────┼────┬─────────┬──────────┐
+    ▼    ▼    ▼         ▼          ▼
+┌────────┐ ┌─────┐ ┌────────┐ ┌────────┐
+│Embedding│ │ ASR │ │Reranker│ │ MinerU │
+│ :8091  │ │:8092│ │ :8093  │ │ :8094  │
+└────────┘ └─────┘ └────────┘ └────────┘
+     │        │        │          │
+     ▼        ▼        ▼          ▼
+┌─────────┐ ┌─────┐ ┌────────┐ ┌────────┐
+│Qwen3-   │ │Qwen3│ │Qwen3-  │ │MinerU  │
+│Embedding│ │-ASR │ │Reranker│ │VLM     │
+└─────────┘ └─────┘ └────────┘ └────────┘
+```
+
+---
+
+## 故障排查
+
+### 常见问题
+
+1. **服务启动失败**: 检查端口是否被占用，使用 `netstat -tlnp | grep 8090` 查看
+
+2. **模型加载失败**: 确认模型路径正确，检查 `~/.cache/modelscope/hub/models/` 目录
+
+3. **GPU 显存不足**: 调整 `gpu_memory_utilization` 参数，降低显存使用率
+
+4. **CUDA 版本不匹配**: 检查 PyTorch CUDA 版本与系统 CUDA 是否一致
+
+### 日志位置
+
+```bash
+# 网关日志
+~/.cache/pi-llm-server/logs/gateway.log
+
+# 子服务日志
+~/.cache/pi-llm-server/logs/<service>.log
+```
+
+---
+
+## License
+
+MIT License
+