xiaoshiai-hub 0.1.3__tar.gz → 1.0.0__tar.gz

xiaoshiai_hub-1.0.0/PKG-INFO

@@ -0,0 +1,473 @@
+Metadata-Version: 2.4
+Name: xiaoshiai-hub
+Version: 1.0.0
+Summary: Python SDK for XiaoShi AI Hub - Upload, download, and manage AI models and datasets with xpai-enc encryption support
+Home-page: https://github.com/poxiaoyun/moha-sdk
+Author: XiaoShi AI
+Author-email: XiaoShi AI <support@xiaoshiai.cn>
+Maintainer: XiaoShi AI Team
+Maintainer-email: XiaoShi AI Team <support@xiaoshiai.cn>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/poxiaoyun/moha-sdk
+Project-URL: Repository, https://github.com/poxiaoyun/moha-sdk
+Project-URL: Issues, https://github.com/poxiaoyun/moha-sdk/issues
+Keywords: xiaoshi,ai-hub,machine-learning,deep-learning,model-hub,dataset,encryption,xpai-enc,sdk,api-client,model-encryption
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.20.0
+Requires-Dist: tqdm>=4.62.0
+Provides-Extra: upload
+Requires-Dist: gitpython>=3.1.0; extra == "upload"
+Provides-Extra: encryption
+Provides-Extra: dev
+Requires-Dist: pytest>=6.0; extra == "dev"
+Requires-Dist: pytest-cov>=2.0; extra == "dev"
+Requires-Dist: pytest-mock>=3.6.0; extra == "dev"
+Requires-Dist: black>=22.0; extra == "dev"
+Requires-Dist: flake8>=4.0; extra == "dev"
+Requires-Dist: mypy>=0.950; extra == "dev"
+Requires-Dist: isort>=5.10.0; extra == "dev"
+Requires-Dist: build>=0.8.0; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Requires-Dist: gitpython>=3.1.0; extra == "dev"
+Provides-Extra: all
+Requires-Dist: gitpython>=3.1.0; extra == "all"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: maintainer
+Dynamic: requires-python
+
+# XiaoShi AI Hub Python SDK
+
+[![PyPI version](https://badge.fury.io/py/xiaoshiai-hub.svg)](https://badge.fury.io/py/xiaoshiai-hub)
+[![Python Support](https://img.shields.io/pypi/pyversions/xiaoshiai-hub.svg)](https://pypi.org/project/xiaoshiai-hub/)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+XiaoShi AI Hub Python SDK 是一个功能强大的 Python 库，用于与 XiaoShi AI Hub 平台进行交互。它提供了简单易用的 API，支持模型和数据集的上传、下载，并集成了 [xpai-enc](https://github.com/poxiaoyun/xpai-enc) 提供的透明加密功能。
+
+
+## ✨ 特性
+
+- 🚀 **简单易用** - 类似 Hugging Face Hub 的 API 设计，上手即用
+- 📥 **下载功能** - 支持下载单个文件或整个仓库
+- 📤 **上传功能** - 支持上传文件和文件夹到仓库
+- 🔐 **智能加密** - 集成 xpai-enc，自动加密大型模型文件（≥5MB 的 .safetensors、.bin、.pt、.pth、.ckpt 文件）
+- 🎯 **模式匹配** - 支持使用 allow/ignore 模式过滤文件
+- 📊 **进度显示** - 下载和上传时显示进度条
+- 🔑 **多种认证** - 支持用户名/密码和 Token 认证
+- 🌐 **环境变量配置** - 灵活的 Hub URL 配置
+- 💾 **缓存支持** - 高效的文件缓存机制
+- 🔍 **类型提示** - 完整的类型注解，IDE 友好
+- ✅ **仓库验证** - 上传前自动检查仓库是否存在
+
+## 📦 安装
+
+### 基础安装
+
+```bash
+pip install xiaoshiai-hub
+```
+
+### 完整安装（包含加密功能）
+
+```bash
+# 安装 SDK
+pip install xiaoshiai-hub
+
+# 安装 xpai-enc（用于加密功能）
+pip install git+https://github.com/poxiaoyun/xpai-enc.git
+```
+
+## 🚀 快速开始
+
+### 下载单个文件
+
+```python
+from xiaoshiai_hub import moha_hub_download
+
+# 下载单个文件
+file_path = moha_hub_download(
+    repo_id="demo/demo",
+    filename="config.yaml",
+    repo_type="models",  # 或 "datasets"
+    username="your-username",
+    password="your-password",
+)
+print(f"文件已下载到: {file_path}")
+```
+
+### 下载整个仓库
+
+```python
+from xiaoshiai_hub import snapshot_download
+
+# 下载整个仓库
+repo_path = snapshot_download(
+    repo_id="demo/demo",
+    repo_type="models",
+    username="your-username",
+    password="your-password",
+)
+print(f"仓库已下载到: {repo_path}")
+```
+
+### 使用过滤器下载
+
+```python
+from xiaoshiai_hub import snapshot_download
+
+# 只下载 YAML 和 Markdown 文件
+repo_path = snapshot_download(
+    repo_id="demo/demo",
+    allow_patterns=["*.yaml", "*.yml", "*.md"],
+    ignore_patterns=[".git*", "*.log"],
+    username="your-username",
+    password="your-password",
+)
+```
+
+### 上传文件
+
+```python
+from xiaoshiai_hub import upload_file
+
+# 上传单个文件
+result = upload_file(
+    path_file="./config.yaml",
+    path_in_repo="config.yaml",
+    repo_id="demo/my-model",
+    repo_type="models",
+    commit_message="Upload config file",
+    username="your-username",
+    password="your-password",
+)
+print(f"上传成功: {result}")
+```
+
+### 上传文件夹
+
+```python
+from xiaoshiai_hub import upload_folder
+
+# 上传整个文件夹
+result = upload_folder(
+    folder_path="./my_model",
+    repo_id="demo/my-model",
+    repo_type="models",
+    commit_message="Upload model files",
+    ignore_patterns=["*.log", ".git*"],  # 忽略这些文件
+    username="your-username",
+    password="your-password",
+)
+print(f"上传成功: {result}")
+```
+
+### 加密上传
+
+SDK 会自动加密大型模型文件（≥5MB 的 .safetensors、.bin、.pt、.pth、.ckpt 文件）：
+
+```python
+from xiaoshiai_hub import upload_file
+
+# 上传文件，大型模型文件会自动加密
+result = upload_file(
+    path_file="./model.safetensors",  # 如果 ≥5MB，会自动加密
+    path_in_repo="model.safetensors",
+    repo_id="demo/my-model",
+    repo_type="models",
+    encryption_password="your-secure-password",  # 设置加密密码
+    username="your-username",
+    password="your-password",
+)
+```
+
+### 上传文件夹（自动加密大文件）
+
+```python
+from xiaoshiai_hub import upload_folder
+
+# 上传文件夹，大型模型文件会自动加密
+result = upload_folder(
+    folder_path="./my_model",
+    repo_id="demo/my-model",
+    repo_type="models",
+    encryption_password="your-secure-password",  # 大文件会自动加密
+    ignore_patterns=["*.log", ".git*"],
+    username="your-username",
+    password="your-password",
+)
+# 加密清单文件 xpai_encryption_manifest.enc 会自动上传
+```
+
+### 使用 HubClient API
+
+```python
+from xiaoshiai_hub import HubClient
+
+# 创建客户端
+client = HubClient(
+    username="your-username",
+    password="your-password",
+)
+
+# 获取仓库信息
+repo_info = client.get_repository_info("demo", "models", "my-model")
+print(f"仓库名称: {repo_info.name}")
+print(f"组织: {repo_info.organization}")
+
+# 列出分支
+branches = client.list_branches("demo", "models", "my-model")
+for branch in branches:
+    print(f"分支: {branch.name} (commit: {branch.commit_sha})")
+
+# 浏览仓库内容
+content = client.get_repository_content("demo", "models", "my-model", "main")
+for entry in content.entries:
+    print(f"{entry.type}: {entry.name}")
+```
+
+## 🔐 加密功能
+
+SDK 集成了 [xpai-enc](https://github.com/poxiaoyun/xpai-enc) 提供的智能加密功能。
+
+### 自动加密规则
+
+上传时，SDK 会自动加密符合以下条件的文件：
+
+1. **文件大小** ≥ 5MB
+2. **文件扩展名**为：`.safetensors`、`.bin`、`.pt`、`.pth`、`.ckpt`
+
+小文件和其他类型的文件（如配置文件、README 等）不会被加密，保持可读性。
+
+### 加密清单文件
+
+加密后会自动生成 `xpai_encryption_manifest.enc` 清单文件，记录哪些文件被加密了。此文件会自动上传到仓库。
+
+### 使用加密功能
+
+```python
+from xiaoshiai_hub import upload_folder
+
+# 上传文件夹，自动加密大型模型文件
+result = upload_folder(
+    folder_path="./llama-7b",
+    repo_id="demo/llama-7b",
+    encryption_password="my-secure-password-123",  # 设置加密密码
+    username="your-username",
+    password="your-password",
+)
+
+# 文件夹中的大型模型文件（如 model.safetensors）会被自动加密
+# 小文件（如 config.json、README.md）保持原样
+# 加密清单 xpai_encryption_manifest.enc 会自动上传
+```
+
+### 临时目录管理
+
+上传时可以指定临时目录用于存放加密文件：
+
+```python
+result = upload_folder(
+    folder_path="./my_model",
+    repo_id="demo/my-model",
+    encryption_password="password",
+    temp_dir="/tmp/encrypted_files",  # 指定临时目录
+    username="your-username",
+    password="your-password",
+)
+# 如果不指定 temp_dir，会自动创建临时目录并在上传后清理
+```
+
+## ⚙️ 配置
+
+### 环境变量
+
+```bash
+# Hub 服务端点
+export MOHA_ENDPOINT="https://your-hub-url.com/moha"
+```
+
+## � 使用场景
+
+### 场景 1: 上传开源模型到私有 Hub
+
+```python
+from xiaoshiai_hub import upload_folder
+
+# 上传 Hugging Face 下载的模型到私有 Hub
+result = upload_folder(
+    folder_path="~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf",
+    repo_id="myorg/llama-2-7b",
+    repo_type="models",
+    commit_message="Upload Llama 2 7B model",
+    username="your-username",
+    password="your-password",
+)
+```
+
+### 场景 2: 加密上传敏感模型
+
+```python
+from xiaoshiai_hub import upload_folder
+
+# 上传模型并加密大文件
+result = upload_folder(
+    folder_path="./proprietary-model",
+    repo_id="myorg/proprietary-model",
+    encryption_password="super-secret-password",  # 大文件自动加密
+    ignore_patterns=["*.log", "checkpoints/"],
+    username="your-username",
+    password="your-password",
+)
+```
+
+### 场景 3: 批量下载数据集
+
+```python
+from xiaoshiai_hub import snapshot_download
+
+# 下载整个数据集
+dataset_path = snapshot_download(
+    repo_id="myorg/my-dataset",
+    repo_type="datasets",
+    allow_patterns=["*.parquet", "*.json"],  # 只下载数据文件
+    ignore_patterns=["*.md"],  # 忽略文档
+    username="your-username",
+    password="your-password",
+)
+```
+
+### 场景 4: 检查仓库是否存在
+
+```python
+from xiaoshiai_hub import HubClient
+from xiaoshiai_hub.exceptions import RepositoryNotFoundError
+
+client = HubClient(username="your-username", password="your-password")
+
+try:
+    repo_info = client.get_repository_info("myorg", "models", "my-model")
+    print(f"仓库存在: {repo_info.name}")
+except RepositoryNotFoundError:
+    print("仓库不存在，请先创建")
+```
+
+
+## ⚠️ 重要说明
+
+### 仓库必须先创建
+
+在上传文件或文件夹之前，必须先在 Hub 上创建仓库。SDK 会自动检查仓库是否存在：
+
+```python
+from xiaoshiai_hub import upload_file
+from xiaoshiai_hub.exceptions import RepositoryNotFoundError
+
+try:
+    result = upload_file(
+        path_file="./model.bin",
+        path_in_repo="model.bin",
+        repo_id="myorg/my-model",
+        username="your-username",
+        password="your-password",
+    )
+except RepositoryNotFoundError as e:
+    print(f"错误: {e}")
+    print("请先在 Hub 上创建仓库")
+```
+
+### 加密文件的大小和类型限制
+
+只有满足以下条件的文件才会被加密：
+
+1. **文件大小** ≥ 5MB
+2. **文件扩展名**为：`.safetensors`、`.bin`、`.pt`、`.pth`、`.ckpt`
+
+其他文件保持原样，不会被加密。
+
+### 临时文件清理
+
+使用 `encryption_password` 时，SDK 会创建临时目录存放加密文件。上传完成后会自动清理，但如果上传失败，可能需要手动清理临时目录。
+
+## 🔧 开发
+
+### 设置开发环境
+
+```bash
+# 克隆仓库
+git clone https://github.com/poxiaoyun/moha-sdk.git
+cd moha-sdk
+
+# 创建虚拟环境
+python -m venv venv
+source venv/bin/activate  # Linux/macOS
+# 或
+venv\Scripts\activate  # Windows
+
+# 安装依赖
+pip install -r requirements.txt
+
+# 安装 xpai-enc（用于加密功能）
+pip install git+https://github.com/poxiaoyun/xpai-enc.git
+```
+
+## 🔗 相关项目
+
+- **[xpai-enc](https://github.com/poxiaoyun/xpai-enc)** - 模型文件加密/解密工具，提供透明的加密支持
+
+## 📝 更新日志
+
+### v0.1.3 (最新)
+
+- ✅ 添加仓库存在性检查（上传前自动验证）
+- ✅ 集成 xpai-enc 加密功能
+- ✅ 自动加密大型模型文件（≥5MB）
+- ✅ 自动生成和上传加密清单文件
+- ✅ 改进错误处理和提示信息
+
+### v0.1.0
+
+- 🎉 初始版本发布
+- ✅ 基础下载功能
+- ✅ 基础上传功能
+- ✅ HubClient API
+
+## 🤝 贡献
+
+欢迎贡献！请随时提交 Issue 或 Pull Request。
+
+### 贡献指南
+
+1. Fork 本仓库
+2. 创建特性分支 (`git checkout -b feature/amazing-feature`)
+3. 提交更改 (`git commit -m 'Add some amazing feature'`)
+4. 推送到分支 (`git push origin feature/amazing-feature`)
+5. 开启 Pull Request
+
+## 📄 许可证
+
+本项目采用 Apache 2.0 许可证 - 详见 [LICENSE](LICENSE) 文件
+
+## 💬 支持
+
+如有问题或需要帮助，请：
+
+1. 查看文档和示例
+2. 搜索或创建 [Issue](https://github.com/poxiaoyun/moha-sdk/issues)
+3. 联系维护者