gsnet-sdk 0.1.0__tar.gz

gsnet_sdk-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.1
+Name: gsnet-sdk
+Version: 0.1.0
+Summary: Python SDK and CLI for the GSnet platform
+Home-page: https://gsnet.pcl.ac.cn/OpenIOSSG/gsnet-sdk
+Author: chenzh05
+Author-email: chenzh.ds@outlook.com
+License: MIT
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: requests
+Requires-Dist: tqdm
+Requires-Dist: deprecated
+Requires-Dist: pwinput
+Requires-Dist: httpx
+Requires-Dist: rich
+Requires-Dist: aiofiles
+Requires-Dist: h2
+Requires-Dist: cryptography>=42.0.0
+
+# GSnet
+
+> PYPI package for GSnet平台。
+
+## 安装
+
+_适配 python3.8 及以上版本_
+
+```bash
+pip install gsnet-sdk
+```
+
+PyPI 发布名为 `gsnet-sdk`，Python 导入名和命令行名称均为 `gsnet`：
+
+```python
+import gsnet
+```
+
+```bash
+gsnet --help
+```
+
+首次发布版本为 `0.1.0`，后续版本规则与发布步骤见 [版本与发布指南](doc/version-guide.md)。
+
+## 功能介绍
+
+- 支持进行本地上传与下载文件到GSnet平台
+- 支持代码调用函数，可在本地Python脚本或notebook使用；支持终端命令行，方便在服务器上使用。
+- 具体请参考 [帮助文档](doc/sdk.md)
+
+## 使用示例
+
+以 [MNIST 示例数据集](https://gsnet.pcl.ac.cn/datasets/detail/OpenIOSSG/MnistDataset_mindspore)为例。
+
+- `python` 代码中下载（可直接复制到 python 文件或 notebook 中运行）
+
+```python
+from gsnet import gsnet_download_file
+
+# 下载整个数据集；只下载部分文件时用 filename（需带远端子目录，多个文件传列表）
+#   下载单个文件加上：filename="train/train-images-idx3-ubyte"
+gsnet_download_file(
+    repo_id="OpenIOSSG/MnistDataset_mindspore",  # 必填：拥有者/数据集名
+    local_dir="./downloads",                     # 选填：本地保存目录，默认当前目录下的 `MnistDataset_mindspore/`
+    #filename="train/train-images-idx3-ubyte"     # 选填：下载指定文件，不填默认下载整个数据集
+)
+```
+
+- 上述代码调用对应的终端命令（可直接复制到终端命令行中运行）
+
+```bash
+gsnet dataset download OpenIOSSG/MnistDataset_mindspore --local_dir ./downloads
+```
+
+- 执行结果：实时显示整体进度与各文件进度，并按远端目录结构保存到本地
+
+```bash
+数据集 `OpenIOSSG/MnistDataset_mindspore` 获取文件列表中……
+- 云端共找到 5 个文件。
+- 开始下载 5 个文件……
+- 本地保存目录 D:/MnistDataset_mindspore/downloads
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  10 workers | 已处理文件: 5/5 -------------- 100% 60.1MiB/s 52.4MiB 0:00:00 │
+└─────────────────────────────────────────────────────────────────────────────┘
+ README.md                 -------------------- 100% ?         193bytes 0:00:00
+ (…)t10k-images-idx3-ubyte -------------------- 100% 44.8MiB/s 7.5MiB   0:00:00
+ (…)t10k-labels-idx1-ubyte -------------------- 100% ?         9.8KiB   0:00:00
+ (…)rain-images-idx3-ubyte -------------------- 100% 90.2MiB/s 44.9MiB  0:00:00
+ (…)rain-labels-idx1-ubyte -------------------- 100% 55.7MiB/s 58.6KiB  0:00:00
+```
+
+下载完成后的本地目录结构（与远端 `test/`、`train/` 一致）：
+
+```text
+downloads/
+├── README.md                        193 B
+├── test/
+│   ├── t10k-images-idx3-ubyte      7.84 MB
+│   └── t10k-labels-idx1-ubyte      9.78 KB
+└── train/
+    ├── train-images-idx3-ubyte    44.86 MB
+    └── train-labels-idx1-ubyte   58.60 KB
+```
+
+- 只下载部分文件：`-f` 
+
+```bash
+# 单个文件 -> ./MnistDataset_mindspore/train/train-images-idx3-ubyte
+gsnet dataset download OpenIOSSG/MnistDataset_mindspore -f train/train-images-idx3-ubyte
+
+# 多个文件（逗号隔开） -> ./MnistDataset_mindspore/test/t10k-images-idx3-ubyte 与 .../test/t10k-labels-idx1-ubyte
+gsnet dataset download OpenIOSSG/MnistDataset_mindspore -f test/t10k-images-idx3-ubyte,test/t10k-labels-idx1-ubyte
+```
+
+- 更多使用指南（上传、受控数据集、权限与常见报错等）请参考 [帮助文档](doc/sdk.md)

gsnet_sdk-0.1.0/README.md

@@ -0,0 +1,95 @@
+# GSnet
+
+> PYPI package for GSnet平台。
+
+## 安装
+
+_适配 python3.8 及以上版本_
+
+```bash
+pip install gsnet-sdk
+```
+
+PyPI 发布名为 `gsnet-sdk`，Python 导入名和命令行名称均为 `gsnet`：
+
+```python
+import gsnet
+```
+
+```bash
+gsnet --help
+```
+
+首次发布版本为 `0.1.0`，后续版本规则与发布步骤见 [版本与发布指南](doc/version-guide.md)。
+
+## 功能介绍
+
+- 支持进行本地上传与下载文件到GSnet平台
+- 支持代码调用函数，可在本地Python脚本或notebook使用；支持终端命令行，方便在服务器上使用。
+- 具体请参考 [帮助文档](doc/sdk.md)
+
+## 使用示例
+
+以 [MNIST 示例数据集](https://gsnet.pcl.ac.cn/datasets/detail/OpenIOSSG/MnistDataset_mindspore)为例。
+
+- `python` 代码中下载（可直接复制到 python 文件或 notebook 中运行）
+
+```python
+from gsnet import gsnet_download_file
+
+# 下载整个数据集；只下载部分文件时用 filename（需带远端子目录，多个文件传列表）
+#   下载单个文件加上：filename="train/train-images-idx3-ubyte"
+gsnet_download_file(
+    repo_id="OpenIOSSG/MnistDataset_mindspore",  # 必填：拥有者/数据集名
+    local_dir="./downloads",                     # 选填：本地保存目录，默认当前目录下的 `MnistDataset_mindspore/`
+    #filename="train/train-images-idx3-ubyte"     # 选填：下载指定文件，不填默认下载整个数据集
+)
+```
+
+- 上述代码调用对应的终端命令（可直接复制到终端命令行中运行）
+
+```bash
+gsnet dataset download OpenIOSSG/MnistDataset_mindspore --local_dir ./downloads
+```
+
+- 执行结果：实时显示整体进度与各文件进度，并按远端目录结构保存到本地
+
+```bash
+数据集 `OpenIOSSG/MnistDataset_mindspore` 获取文件列表中……
+- 云端共找到 5 个文件。
+- 开始下载 5 个文件……
+- 本地保存目录 D:/MnistDataset_mindspore/downloads
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  10 workers | 已处理文件: 5/5 -------------- 100% 60.1MiB/s 52.4MiB 0:00:00 │
+└─────────────────────────────────────────────────────────────────────────────┘
+ README.md                 -------------------- 100% ?         193bytes 0:00:00
+ (…)t10k-images-idx3-ubyte -------------------- 100% 44.8MiB/s 7.5MiB   0:00:00
+ (…)t10k-labels-idx1-ubyte -------------------- 100% ?         9.8KiB   0:00:00
+ (…)rain-images-idx3-ubyte -------------------- 100% 90.2MiB/s 44.9MiB  0:00:00
+ (…)rain-labels-idx1-ubyte -------------------- 100% 55.7MiB/s 58.6KiB  0:00:00
+```
+
+下载完成后的本地目录结构（与远端 `test/`、`train/` 一致）：
+
+```text
+downloads/
+├── README.md                        193 B
+├── test/
+│   ├── t10k-images-idx3-ubyte      7.84 MB
+│   └── t10k-labels-idx1-ubyte      9.78 KB
+└── train/
+    ├── train-images-idx3-ubyte    44.86 MB
+    └── train-labels-idx1-ubyte   58.60 KB
+```
+
+- 只下载部分文件：`-f` 
+
+```bash
+# 单个文件 -> ./MnistDataset_mindspore/train/train-images-idx3-ubyte
+gsnet dataset download OpenIOSSG/MnistDataset_mindspore -f train/train-images-idx3-ubyte
+
+# 多个文件（逗号隔开） -> ./MnistDataset_mindspore/test/t10k-images-idx3-ubyte 与 .../test/t10k-labels-idx1-ubyte
+gsnet dataset download OpenIOSSG/MnistDataset_mindspore -f test/t10k-images-idx3-ubyte,test/t10k-labels-idx1-ubyte
+```
+
+- 更多使用指南（上传、受控数据集、权限与常见报错等）请参考 [帮助文档](doc/sdk.md)

gsnet_sdk-0.1.0/pyproject.toml

@@ -0,0 +1,8 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.black]
+line-length = 88
+target-version = ['py38']
+include = '\.pyi?$'

gsnet_sdk-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

gsnet_sdk-0.1.0/setup.py

@@ -0,0 +1,28 @@
+from setuptools import find_packages, setup
+
+with open("README.md", "r", encoding="utf-8") as f:
+    long_description = f.read()
+
+setup(
+    name="gsnet-sdk",
+    version="0.1.0",
+    description="Python SDK and CLI for the GSnet platform",
+    package_dir={"": "src"},
+    packages=find_packages(where="src"),
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://gsnet.pcl.ac.cn/OpenIOSSG/gsnet-sdk",
+    author="chenzh05",
+    author_email="chenzh.ds@outlook.com",
+    license="MIT",
+    classifiers=[
+        "License :: OSI Approved :: MIT License",
+        "Intended Audience :: Developers",
+        "Topic :: Software Development :: Build Tools",
+        "Programming Language :: Python :: 3.8",
+        "Operating System :: OS Independent",
+    ],
+    entry_points={"console_scripts": ["gsnet = gsnet.commands.gsnet_cli:main"]},
+    install_requires=["requests", "tqdm", "deprecated", "pwinput", "httpx", "rich", "aiofiles", "h2", "cryptography>=42.0.0"],
+    python_requires=">=3.8",
+)

gsnet_sdk-0.1.0/src/gsnet/__init__.py

@@ -0,0 +1,32 @@
+import asyncio
+import os
+import sys
+
+from gsnet._login import login, logout, whoami
+from gsnet.refactor.constants import BOLD, DIM, RESET
+from gsnet.refactor.core import GSnetRouter
+from gsnet.refactor.plugins.log import setup_logging
+from gsnet.refactor.sdk import notice, gsnet_download_file, gsnet_upload_file
+
+from .downloader import download_file, download_model, download_model_file
+from .uploader import upload_file, upload_model, upload_model_file
+
+setup_logging()
+
+# if windows machine, do not set uvloop
+if os.name == "nt":
+    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
+else:
+    try:
+        import uvloop
+    except ImportError:
+        import subprocess
+
+        print(f"{DIM}Module {BOLD}`uvloop`{RESET} {DIM}undetected, installing...{RESET}")
+        subprocess.check_call(
+            [sys.executable, "-m", "pip", "install", "-qq", "uvloop"],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+        import uvloop
+    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

gsnet_sdk-0.1.0/src/gsnet/_dataclass.py

@@ -0,0 +1,404 @@
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass, field
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from .utils import parse_time
+
+
+@dataclass
+class UserInfo:
+    """
+    User Info Dataclass
+
+    Attributes:
+        username (str):
+            username for GSnet links and repo_id
+        id (int):
+            unique number id of user
+        login (str):
+            username used for login purpose
+        subject_name (str):
+            self-defined name for display
+        email (str):
+            registered email address
+        avatar_url (str):
+            user avatar url
+        language (str):
+            user default language
+        is_admin (bool):
+            whether the user is admin or not
+        is_active (bool):
+            whether the user is active or not
+        last_login (str):
+            last login time
+        created (str):
+            created time
+    """
+
+    username: str
+    id: Optional[int]
+    login: Optional[str]
+    subject_name: Optional[str]
+    email: Optional[str]
+    avatar_url: Optional[str]
+    language: Optional[str]
+    is_admin: Optional[bool]
+    is_active: Optional[bool]
+    last_login: Optional[str]
+    created: Optional[str]
+
+
+@dataclass
+class RepoPermission:
+    """
+    Permission Info Dataclass
+
+    Attributes:
+        admin (bool):
+            whether the user is admin or not
+        push (bool):
+            whether the user can push to the repository or not
+        pull (bool):
+            whether the user can pull from the repository or not
+    """
+
+    admin: bool
+    push: bool
+    pull: bool
+
+
+@dataclass
+class RepoInfo:
+    """
+    Repo Info Dataclass
+
+    Attributes:
+        id (int):
+            repo id
+        owner (UserInfo):
+            owner info
+        name (str):
+            repo name
+        repo_id (str):
+            repo id in `Username/Reponame` format
+        size (int):
+            repo size
+        html_url (str):
+            repo url
+        permissions (RepoPermission):
+            repo permissions
+        created_at (str):
+            created time
+        updated_at (str):
+            updated time
+    """
+
+    id: int
+    owner: UserInfo
+    name: str
+    repo_id: str
+    size: Optional[int]
+    html_url: Optional[str]
+    permissions: Optional[RepoPermission]
+    created_at: Optional[datetime]
+    updated_at: Optional[datetime]
+
+    def __init__(self, **kwargs):
+        self.id = kwargs.pop("id")
+        self.owner = UserInfo(**kwargs.pop("owner"))
+        self.name = kwargs.pop("name")
+        self.repo_id = kwargs.pop("subject_name")
+
+        self.size = kwargs.pop("size", None)
+        self.html_url = kwargs.pop("html_url", None)
+        permissions = kwargs.pop("permissions", None)
+        self.permissions = RepoPermission(**permissions) if permissions else None
+
+        created_at = kwargs.pop("created_at", None)
+        self.created_at = parse_time(created_at) if created_at else None
+        updated_at = kwargs.pop("updated_at", None)
+        self.updated_at = parse_time(updated_at) if updated_at else None
+
+
+@dataclass
+class DatasetFile:
+    """TODO"""
+
+    uuid: str
+    name: str
+    size: int
+    type: int
+    id: int
+    datasetId: int
+    repo_id: str
+    uploaderId: Optional[int]
+    isPrivate: Optional[bool]
+    decompressState: Optional[int]
+
+    def __init__(self, **kwargs):
+        self.uuid = kwargs.pop("uuid")
+        self.name = kwargs.pop("name")
+        self.size = kwargs.pop("size")
+        self.type = kwargs.pop("type")
+        self.id = kwargs.pop("id")
+        self.datasetId = kwargs.pop("datasetId")
+        self.repo_id = kwargs.pop("repo_id")
+
+        self.uploaderId = kwargs.pop("uploaderId", None)
+        self.isPrivate = kwargs.pop("isPrivate", None)
+        self.decompressState = kwargs.pop("decompressState", None)
+
+
+@dataclass
+class DatasetInfo:
+    """TODO"""
+
+    id: str
+    userId: int
+    repoId: int
+    title: str
+    repo_id: str
+    attachments: Optional[List[DatasetFile]]
+    status: Optional[int]
+
+    def __init__(self, **kwargs):
+        self.id = kwargs.pop("id")
+        self.userId = kwargs.pop("userId")
+        self.repoId = kwargs.pop("repoId")
+        self.title = kwargs.pop("title")
+        self.repo_id = kwargs.pop("repo_id")
+
+        attachments = kwargs.pop("attachments", [])
+        if attachments:
+            for data in attachments:
+                data.update({"repo_id": self.repo_id})
+            self.attachments = [DatasetFile(**data) for data in attachments]
+        else:
+            self.attachments = None
+
+        self.status = kwargs.pop("status", None)
+        self.recommend = kwargs.pop("recommend", None)
+
+
+@dataclass
+class FileChunkInfo:
+    """
+    File Chunk Info Dataclass
+
+    API Response (dict):
+        - dataset "/attachments/get_chunks"
+            "uuid":        fileChunk.UUID,
+            "uploaded":    strconv.Itoa(fileChunk.IsUploaded),
+            "uploadID":    fileChunk.UploadID,
+            "chunks":      string(chunks),
+            "attachID":    "0",
+            "datasetID":   "0",
+            "fileName":    "",
+            "datasetName": "",
+
+        - model "/attachments/model/get_chunks"
+            "uuid":      fileChunk.UUID,
+            "uploaded":  strconv.Itoa(fileChunk.IsUploaded),
+            "uploadID":  fileChunk.UploadID,
+            "chunks":    string(chunks),
+            "attachID":  "0",
+            "modeluuid": modeluuid,
+            "datasetID": "0",
+            "fileName":  "",
+            "modelName": "",
+
+        - error response
+            "result_code": "-1",
+            "msg":         errStr,
+
+    Attributes:
+        result_code (int):
+            return 0 when success, otherwise return -1 with error message
+        msg (str):
+            return plain string error message, return "success" when None
+        uploaded (Optional[bool]):
+            whether the file is uploaded or not
+        attachID (Optional[int]):
+            attachment id
+        uploaded_chunks (Optional[list]):
+            list of uploaded chunks
+        uploadID (Optional[str]):
+            upload id
+        uuid (Optional[str]):
+            file uuid
+        repo_id (Optional[str]):
+            repo id in `Username/Reponame` format
+        dataset_or_model_name (Optional[str]):
+            dataset or model name
+    """
+
+    result_code: int
+    msg: str
+    uploaded: Optional[bool]
+    md5: Optional[str]
+    fileName: Optional[str]
+    attachID: Optional[int]
+    uploaded_chunks: Optional[list]
+    uploadID: Optional[str]
+    uuid: Optional[str]
+    repo_id: Optional[str]
+    dataset_or_model_name: Optional[str]
+    realName: Optional[str]
+    datasetID: Optional[str]
+    upload_mode: Optional[str]
+
+    def __init__(self, **kwargs):
+        result_code = kwargs.pop("result_code", 0)
+        self.result_code = int(result_code)
+        self.msg = kwargs.pop("msg", "success")
+
+        uploaded = kwargs.pop("uploaded", "")
+        self.uploaded = True if uploaded == "1" else False
+        self.attachID = kwargs.pop("attachID", None)
+
+        raw_chunks = kwargs.pop("chunks", "")
+        uploaded_chunks = [i for i in raw_chunks.split(",") if i != ""]
+        self.uploaded_chunks = uploaded_chunks
+
+        uploadID = kwargs.pop("uploadID", "")
+        self.uploadID = uploadID if uploadID != "" else None
+        uuid = kwargs.pop("uuid", "")
+        self.uuid = uuid if uuid != "" else None
+
+        repo = kwargs.pop("repoName", None)
+        owner = kwargs.pop("repoOwner", None)
+        self.repo_id = owner + "/" + repo if owner and repo else None
+
+        if kwargs.pop("upload_mode") == "dataset":
+            self.dataset_or_model_name = kwargs.pop("datasetName", "")
+        else:
+            self.dataset_or_model_name = kwargs.pop("modelName", "")
+
+        self.fileName = kwargs.pop("fileName", "")
+        self.realName = self.fileName
+        self.datasetID = kwargs.pop("datasetID", "")
+        self.md5 = kwargs.pop("md5", None)
+        self.upload_mode = kwargs.pop("upload_mode", None)
+
+
+@dataclass
+class NewMultipart:
+    """TODO"""
+
+    result_code: int
+    msg: str
+    uuid: Optional[str]
+    uploadID: Optional[str]
+
+    def __init__(self, **kwargs):
+        result_code = kwargs.pop("result_code", 0)
+        self.result_code = int(result_code)
+        self.msg = kwargs.pop("msg", "success")
+        self.uuid = kwargs.pop("uuid", None)
+        self.uploadID = kwargs.pop("uploadID", None)
+
+
+@dataclass
+class MultipartUrl:
+    """TODO"""
+
+    url: Optional[str] = None
+    msg: Optional[str] = None
+
+
+@dataclass
+class ModelFile:
+    FileName: Optional[str]
+    ModTime: Optional[str]
+    IsDir: Optional[bool]
+    Size: Optional[int]
+    ParenDir: Optional[str]
+    UUID: Optional[str]
+    RelativePath: Optional[str]
+    repo_id: Optional[str]
+    model_id: Optional[str]
+    IsSupportPreview: Optional[str] = ""
+
+
+@dataclass
+class ModelInfo:
+    id: Optional[str]
+    name: Optional[str]
+    size: Optional[int]
+    path: Optional[str]
+    repo_id: Optional[str]
+    repoId: Optional[int]
+    userId: Optional[int]
+    userName: Optional[str]
+    isCanOper: Optional[bool]
+    isCanDelete: Optional[bool]
+    isCanDownload: Optional[bool]
+    repoName: Optional[str]
+    repoOwnerName: Optional[str]
+    modelFileList: Optional[List[ModelFile]]
+    modelType: Optional[int]
+
+    def __init__(self, **kwargs):
+        self.id = kwargs.pop("id", None)
+        self.name = kwargs.pop("name", None)
+        self.size = kwargs.pop("size", None)
+        self.path = kwargs.pop("path", None)
+        self.repo_id = kwargs.pop("repo_id", None)
+        self.repoId = kwargs.pop("repoId", None)
+        self.userId = kwargs.pop("userId", None)
+        self.userName = kwargs.pop("userName", None)
+        self.isCanOper = kwargs.pop("isCanOper", None)
+        self.isCanDelete = kwargs.pop("isCanDelete", None)
+        self.isCanDownload = kwargs.pop("isCanDownload", None)
+        self.repoName = kwargs.pop("repoName", None)
+        self.repoOwnerName = kwargs.pop("repoOwnerName", None)
+
+        modelFileList = kwargs.pop("modelFileList", [])
+        if modelFileList:
+            for data in modelFileList:
+                data.update({"repo_id": self.repo_id, "model_id": self.id})
+            self.modelFileList = [ModelFile(**data) for data in modelFileList]
+        else:
+            self.modelFileList = None
+
+        self.modelType = kwargs.pop("modelType", None)
+
+
+@dataclass
+class ModelCreate:
+    code: str
+    msg: Optional[str] = None
+    id: Optional[int] = None
+
+
+@dataclass
+class BasicResp:
+    code: int
+    msg: str
+    data: any
+
+
+@dataclass
+class CreateRepoOption:
+    name: str
+    alias: Optional[str] = None
+    description: Optional[str] = None
+    private: bool = False
+    issue_labels: Optional[str] = None
+    auto_init: bool = False
+    gitignores: Optional[str] = None
+    license: Optional[str] = None
+    readme: Optional[str] = None
+    default_branch: Optional[str] = None
+
+    def __post_init__(self):
+        if len(self.name) > 100:
+            raise ValueError("Name must be at most 100 characters long")
+        if self.alias and len(self.alias) > 100:
+            raise ValueError("Alias must be at most 100 characters long")
+        if self.description and len(self.description) > 255:
+            raise ValueError("Description must be at most 255 characters long")
+        if self.default_branch and len(self.default_branch) > 100:
+            raise ValueError("Default branch must be at most 100 characters long")