jishu-hub 0.1.0__tar.gz

jishu_hub-0.1.0/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: jishu-hub
+Version: 0.1.0
+Summary: jishu Hub CLI for secure dataset and label transfer
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.27
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13.7
+Requires-Dist: cryptography>=42
+
+# jishu-hub
+
+安全传输 CLI：在**已授权设备**上上传/下载数据集、标注数据，以及拉取评测报告。
+
+---
+
+## 在其他机器上的使用流程
+
+每台新机器都要由根超管批准一次。密钥在本机生成，**不能**把旧机器的 `~/.jishu` 整包拷过去当「换机」。
+
+### 0. 目标机取机器码 → 根超管录入
+
+无一次性注册码。设备准入由超管按「用户 + 机器码」批准。
+
+1. 在**目标机器**安装 CLI 后执行：
+
+```bash
+jishu device-code
+```
+
+将 64 位机器码发给管理员（工单 / 消息）。
+
+2. 根超管在 **后台管理 → 用户列表** 点「录入机器」：粘贴机器码、选填备注 → 批准。
+3. 管理员告知 **Hub 网关地址**（侧信道，勿写入公开文档）。
+
+设备与批准吊销在 **后台管理 → 设备/Hub**。
+
+### 1. 目标机器：环境
+
+- Python ≥ 3.10
+- 能访问 Hub 网关（HTTPS；生产环境通常要求客户端证书 / mTLS，由网关终止）
+- 安装 CLI（**尚未上架公网 PyPI**，`pip install jishu-hub` 会失败）：
+
+```bash
+# 从本仓库开发安装（当前推荐）
+cd /path/to/data_management
+pip install -e ./cli
+
+# 或安装已构建的 wheel（内网制品库 / 构建机产出）
+# pip install /path/to/jishu_hub-0.1.0-py3-none-any.whl
+# pip install jishu-hub --index-url https://<私有索引>
+```
+
+可选：固定网关环境变量：
+
+```bash
+export JISHU_API_URL="https://<hub-host>"   # Linux/macOS
+# Windows PowerShell: $env:JISHU_API_URL="https://<hub-host>"
+```
+
+### 2. 登录（首次自动激活本机）
+
+管理员已录入本机机器码后：
+
+```bash
+jishu login --gateway https://<hub-host> --username <平台用户名>
+# 会提示密码；首次会自动 claim 设备证书，之后只需 login
+```
+
+成功后本机会生成：
+
+| 位置 | 内容 |
+|---|---|
+| `~/.jishu/`（或 `JISHU_HOME`） | 网关、device_id、证书指纹等 |
+| 本机密钥材料 | 私钥 + 设备证书（私钥不出机） |
+
+验证：
+
+```bash
+jishu whoami
+```
+
+### 3. 常用命令
+
+```bash
+jishu dataset upload <本地目录> --dataset-id <id> --version <ver>
+jishu dataset download --dataset-id <id> --version <ver> --out <目录>
+jishu label upload <标注目录> --project-id <id>
+jishu eval reports list
+```
+
+上传/下载权限由平台账号 ACL 决定，与网页一致。
+
+### 4. 换机 / 吊销
+
+- 换机：新机 `device-code` → 超管重新录入 → 新机 `login`
+- 吊销：超管在 **设备/Hub** 吊销批准或设备后，该机无法再 login / 传输

jishu_hub-0.1.0/README.md

@@ -0,0 +1,87 @@
+# jishu-hub
+
+安全传输 CLI：在**已授权设备**上上传/下载数据集、标注数据，以及拉取评测报告。
+
+---
+
+## 在其他机器上的使用流程
+
+每台新机器都要由根超管批准一次。密钥在本机生成，**不能**把旧机器的 `~/.jishu` 整包拷过去当「换机」。
+
+### 0. 目标机取机器码 → 根超管录入
+
+无一次性注册码。设备准入由超管按「用户 + 机器码」批准。
+
+1. 在**目标机器**安装 CLI 后执行：
+
+```bash
+jishu device-code
+```
+
+将 64 位机器码发给管理员（工单 / 消息）。
+
+2. 根超管在 **后台管理 → 用户列表** 点「录入机器」：粘贴机器码、选填备注 → 批准。
+3. 管理员告知 **Hub 网关地址**（侧信道，勿写入公开文档）。
+
+设备与批准吊销在 **后台管理 → 设备/Hub**。
+
+### 1. 目标机器：环境
+
+- Python ≥ 3.10
+- 能访问 Hub 网关（HTTPS；生产环境通常要求客户端证书 / mTLS，由网关终止）
+- 安装 CLI（**尚未上架公网 PyPI**，`pip install jishu-hub` 会失败）：
+
+```bash
+# 从本仓库开发安装（当前推荐）
+cd /path/to/data_management
+pip install -e ./cli
+
+# 或安装已构建的 wheel（内网制品库 / 构建机产出）
+# pip install /path/to/jishu_hub-0.1.0-py3-none-any.whl
+# pip install jishu-hub --index-url https://<私有索引>
+```
+
+可选：固定网关环境变量：
+
+```bash
+export JISHU_API_URL="https://<hub-host>"   # Linux/macOS
+# Windows PowerShell: $env:JISHU_API_URL="https://<hub-host>"
+```
+
+### 2. 登录（首次自动激活本机）
+
+管理员已录入本机机器码后：
+
+```bash
+jishu login --gateway https://<hub-host> --username <平台用户名>
+# 会提示密码；首次会自动 claim 设备证书，之后只需 login
+```
+
+成功后本机会生成：
+
+| 位置 | 内容 |
+|---|---|
+| `~/.jishu/`（或 `JISHU_HOME`） | 网关、device_id、证书指纹等 |
+| 本机密钥材料 | 私钥 + 设备证书（私钥不出机） |
+
+验证：
+
+```bash
+jishu whoami
+```
+
+### 3. 常用命令
+
+```bash
+jishu dataset upload <本地目录> --dataset-id <id> --version <ver>
+jishu dataset download --dataset-id <id> --version <ver> --out <目录>
+jishu label upload <标注目录> --project-id <id>
+jishu eval reports list
+```
+
+上传/下载权限由平台账号 ACL 决定，与网页一致。
+
+### 4. 换机 / 吊销
+
+- 换机：新机 `device-code` → 超管重新录入 → 新机 `login`
+- 吊销：超管在 **设备/Hub** 吊销批准或设备后，该机无法再 login / 传输

jishu_hub-0.1.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "jishu-hub"
+version = "0.1.0"
+description = "jishu Hub CLI for secure dataset and label transfer"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+  "httpx>=0.27",
+  "typer>=0.12",
+  "rich>=13.7",
+  "cryptography>=42",
+]
+
+[project.scripts]
+jishu = "jishu_hub.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

jishu_hub-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

jishu_hub-0.1.0/src/jishu_hub/__init__.py

@@ -0,0 +1,3 @@
+"""jishu-hub CLI client."""
+
+__version__ = "0.1.0"

jishu_hub-0.1.0/src/jishu_hub/__main__.py

@@ -0,0 +1,4 @@
+from jishu_hub.cli import main
+
+if __name__ == "__main__":
+    main()

jishu_hub-0.1.0/src/jishu_hub/cli.py

@@ -0,0 +1,335 @@
+"""jishu CLI 入口。"""
+from __future__ import annotations
+
+import zipfile
+from pathlib import Path
+from typing import Optional
+
+import typer
+from rich import print as rprint
+
+from jishu_hub import __version__, store
+from jishu_hub.device_crypto import build_csr, device_fp_hash, save_device_cert
+from jishu_hub.label_validate import validate_label_dir
+from jishu_hub.rpc import HubClient, HubClientError
+from jishu_hub.transfer_io import download_dataset, upload_directory
+
+app = typer.Typer(add_completion=False, no_args_is_help=True, help="jishu Hub CLI")
+dataset_app = typer.Typer(help="数据集上传/下载")
+label_app = typer.Typer(help="智能标注")
+eval_app = typer.Typer(help="评测报告")
+app.add_typer(dataset_app, name="dataset")
+app.add_typer(label_app, name="label")
+reports_app = typer.Typer(help="报告")
+eval_app.add_typer(reports_app, name="reports")
+app.add_typer(eval_app, name="eval")
+
+
+def _client(gateway: str | None = None) -> HubClient:
+    return HubClient(gateway=gateway)
+
+
+@app.command("device-code")
+def device_code_cmd() -> None:
+    """打印本机机器码，提交给根超管在后台录入。"""
+    fp = device_fp_hash()
+    rprint(fp)
+    rprint("[dim]将上述机器码发给管理员；录入后在本机执行 jishu login --gateway <url>[/dim]")
+
+
+def _ensure_device_claimed(username: str, gateway: str | None) -> None:
+    """若本地尚无设备证书，则按超管批准自动 claim。"""
+    device = store.load_json("device")
+    cert = store.load_secret("device_cert")
+    if device.get("device_id") and cert:
+        return
+    gw = (gateway or store.gateway() or "").rstrip("/")
+    if not gw:
+        raise SystemExit("首次登录需指定网关：jishu login --gateway https://<hub-host>")
+    store.save_json("gateway", {"gateway_url": gw})
+    csr = build_csr()
+    fp = device_fp_hash()
+    client = HubClient(gateway=gw)
+    try:
+        result = client.claim(username, csr, fp)
+    except HubClientError as e:
+        rprint(f"[red]device claim failed:[/red] {e.code}")
+        if e.code == "machine_not_approved":
+            rprint(
+                "[yellow]本机尚未被管理员批准。请执行 jishu device-code 并将机器码发给根超管录入。[/yellow]"
+            )
+        raise typer.Exit(1)
+    save_device_cert(result.get("cert_pem") or "")
+    store.save_json(
+        "device",
+        {
+            "device_id": result.get("device_id"),
+            "cert_fingerprint": result.get("cert_fingerprint"),
+            "device_fp_hash": fp,
+            "scopes": result.get("scopes") or [],
+            "cli_min_version": result.get("cli_min_version"),
+        },
+    )
+    pins = result.get("tls_pins") or []
+    store.save_json(
+        "gateway",
+        {"gateway_url": result.get("gateway_url") or gw, "tls_pins": pins},
+    )
+    rprint(f"[green]device claimed[/green] device={result.get('device_id')}")
+
+
+@app.command("login")
+def login_cmd(
+    username: str = typer.Option(..., prompt=True),
+    password: str = typer.Option(..., prompt=True, hide_input=True),
+    gateway: Optional[str] = typer.Option(
+        None,
+        "--gateway",
+        "-g",
+        envvar="JISHU_API_URL",
+        help="Hub 网关 URL（首次登录或本机未配置时必填）",
+    ),
+) -> None:
+    """登录；若本机尚未激活，会先按管理员录入的机器码自动 claim。"""
+    try:
+        _ensure_device_claimed(username, gateway)
+        client = _client(gateway)
+        out = client.login(username, password)
+    except HubClientError as e:
+        rprint(f"[red]login failed:[/red] {e.code}")
+        raise typer.Exit(1)
+    rprint(f"[green]logged in[/green] scopes={out.get('scope')}")
+
+
+@app.command("logout")
+def logout_cmd() -> None:
+    try:
+        client = _client()
+        client.rpc("auth.logout", {})
+    except Exception:
+        pass
+    store.delete_secret("access_token")
+    store.delete_secret("refresh_token")
+    rprint("logged out")
+
+
+@app.command("whoami")
+def whoami_cmd() -> None:
+    try:
+        me = _client().rpc("auth.me", {})
+    except HubClientError as e:
+        rprint(f"[red]{e.code}[/red]")
+        raise typer.Exit(1)
+    rprint(me)
+
+
+@dataset_app.command("upload")
+def dataset_upload(
+    local_dir: Path = typer.Argument(..., exists=True, file_okay=False),
+    dataset_id: str = typer.Option(..., "--id"),
+    version: Optional[str] = typer.Option(
+        None, "--version", help="版本标签（分支指针）；不存在则创建"
+    ),
+    ref: Optional[str] = typer.Option(None, "--ref", help="与 --version 同义"),
+) -> None:
+    ver = (version or ref or "").strip()
+    if not ver:
+        rprint("[red]need --version or --ref[/red]")
+        raise typer.Exit(2)
+    try:
+        client = _client()
+        upload_directory(
+            client,
+            local_dir,
+            dataset_id=dataset_id,
+            version=ver,
+        )
+    except HubClientError as e:
+        rprint(f"[red]upload failed:[/red] {e.code} {e.message}")
+        raise typer.Exit(1)
+    rprint("[green]upload done[/green]")
+
+
+@dataset_app.command("download")
+def dataset_download_cmd(
+    dataset_id: str = typer.Option(..., "--id"),
+    version: str = typer.Option(..., "--version"),
+    out: Path = typer.Option(Path("./download"), "--out"),
+    paths: Optional[str] = typer.Option(None, "--paths", help="逗号分隔相对路径"),
+    incremental: bool = typer.Option(True, "--incremental/--full"),
+) -> None:
+    path_list = [p.strip() for p in (paths or "").split(",") if p.strip()]
+    try:
+        download_dataset(
+            _client(),
+            dataset_id=dataset_id,
+            version=version,
+            out_dir=out,
+            paths=path_list or None,
+            incremental=incremental,
+        )
+    except HubClientError as e:
+        rprint(f"[red]download failed:[/red] {e.code}")
+        raise typer.Exit(1)
+    except ValueError as e:
+        rprint(f"[red]path error:[/red] {e}")
+        raise typer.Exit(1)
+    rprint(f"[green]downloaded to[/green] {out}")
+
+
+@dataset_app.command("pull")
+def dataset_pull(
+    dataset_id: str = typer.Option(..., "--id"),
+    version: str = typer.Option(..., "--version"),
+    out: Path = typer.Option(Path("./download"), "--out"),
+) -> None:
+    """download 别名。"""
+    dataset_download_cmd(dataset_id=dataset_id, version=version, out=out, paths=None, incremental=True)
+
+
+@label_app.command("validate")
+def label_validate_cmd(
+    local_dir: Path = typer.Argument(..., exists=True, file_okay=False),
+    project: str = typer.Option("", "--project", help="标注项目 id，可选拉 upload-spec"),
+) -> None:
+    media_type = "image"
+    if project:
+        try:
+            spec = _client().rpc("label.upload_spec", {"label_id": project})
+            media_type = str(spec.get("media_type") or "image")
+        except HubClientError as e:
+            rprint(f"[yellow]upload_spec skipped:[/yellow] {e.code}")
+    errors = validate_label_dir(local_dir, media_type=media_type)
+    if errors:
+        for e in errors:
+            rprint(f"[red]- {e}[/red]")
+        raise typer.Exit(1)
+    rprint("[green]validate ok[/green]")
+
+
+@label_app.command("upload")
+def label_upload_cmd(
+    local_dir: Path = typer.Argument(..., exists=True, file_okay=False),
+    project: str = typer.Option(..., "--project"),
+) -> None:
+    errors = validate_label_dir(local_dir)
+    if errors:
+        for e in errors:
+            rprint(f"[red]- {e}[/red]")
+        raise typer.Exit(1)
+
+    # pack zip like UI: assets + annotations if present, else wrap flat
+    import tempfile
+
+    client = _client()
+    with tempfile.TemporaryDirectory() as td:
+        zip_path = Path(td) / "label.zip"
+        with zipfile.ZipFile(zip_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
+            for p in local_dir.rglob("*"):
+                if p.is_file() and not p.is_symlink():
+                    arc = p.relative_to(local_dir).as_posix()
+                    if ".." in arc.split("/"):
+                        raise typer.Exit(1)
+                    zf.write(p, arcname=arc)
+        size = zip_path.stat().st_size
+        payload = {
+            "label_id": project,
+            "filename": "label.zip",
+            "file_size": size,
+        }
+        try:
+            init = client.rpc(
+                "label.archive.init",
+                payload,
+                with_challenge=True,
+                resource_id=project,
+            )
+            import httpx
+
+            parts = init.get("parts") or []
+            part_size = int(init.get("part_size") or size)
+            done = []
+            with zip_path.open("rb") as f, httpx.Client(timeout=300.0) as http:
+                for pinfo in parts:
+                    pn = int(pinfo["part_number"])
+                    start = (pn - 1) * part_size
+                    end = min(start + part_size, size)
+                    f.seek(start)
+                    chunk = f.read(end - start)
+                    r = http.put(pinfo["url"], content=chunk)
+                    r.raise_for_status()
+                    etag = (r.headers.get("ETag") or "").strip('"')
+                    done.append({"part_number": pn, "etag": etag})
+            client.rpc(
+                "label.archive.complete",
+                {
+                    "transfer_session_id": init["transfer_session_id"],
+                    "parts": done,
+                },
+            )
+            batch = init.get("batch_id")
+            if batch:
+                st = client.rpc(
+                    "label.archive.status",
+                    {"label_id": project, "batch_id": batch},
+                )
+                rprint(st)
+        except HubClientError as e:
+            rprint(f"[red]label upload failed:[/red] {e.code}")
+            raise typer.Exit(1)
+    rprint("[green]label upload submitted[/green]")
+
+
+@reports_app.command("list")
+def eval_reports_list(
+    scope: str = typer.Option("mine", "--scope"),
+    team_id: Optional[int] = typer.Option(None, "--team-id"),
+) -> None:
+    payload: dict = {"scope": scope}
+    if team_id is not None:
+        payload["team_id"] = team_id
+    try:
+        out = _client().rpc("eval.reports.list", payload)
+    except HubClientError as e:
+        rprint(f"[red]{e.code}[/red]")
+        raise typer.Exit(1)
+    rprint(out)
+
+
+@reports_app.command("pull")
+def eval_reports_pull(
+    report_name: str = typer.Argument(...),
+    out: Path = typer.Option(Path("./report.json"), "--out"),
+    scope: str = typer.Option("mine", "--scope"),
+    team_id: Optional[int] = typer.Option(None, "--team-id"),
+) -> None:
+    payload: dict = {"report_name": report_name, "scope": scope}
+    if team_id is not None:
+        payload["team_id"] = team_id
+    try:
+        out_data = _client().rpc("eval.reports.pull_meta", payload)
+    except HubClientError as e:
+        rprint(f"[red]{e.code}[/red]")
+        raise typer.Exit(1)
+    import json
+
+    out.write_text(json.dumps(out_data, ensure_ascii=False, indent=2), encoding="utf-8")
+    rprint(f"[green]wrote[/green] {out}")
+
+
+@app.callback()
+def main_callback(
+    version: bool = typer.Option(False, "--version", help="Show version"),
+) -> None:
+    if version:
+        rprint(__version__)
+        raise typer.Exit()
+
+
+# Typer wants `app` callable for console_scripts — expose as main
+def main() -> None:
+    app()
+
+
+if __name__ == "__main__":
+    main()

jishu_hub-0.1.0/src/jishu_hub/device_crypto.py

@@ -0,0 +1,70 @@
+"""设备密钥：本地生成，私钥不离开本机。"""
+from __future__ import annotations
+
+import base64
+import hashlib
+import json
+import platform
+import uuid
+from pathlib import Path
+
+from cryptography import x509
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import padding, rsa
+from cryptography.x509.oid import NameOID
+
+from jishu_hub import store
+
+
+def device_fp_hash() -> str:
+    raw = "|".join(
+        [
+            platform.node(),
+            platform.system(),
+            platform.machine(),
+            str(uuid.getnode()),
+            str(Path.home()),
+        ]
+    )
+    return hashlib.sha256(raw.encode("utf-8")).hexdigest()
+
+
+def ensure_key_pair() -> rsa.RSAPrivateKey:
+    pem = store.load_secret("device_key")
+    if pem:
+        return serialization.load_pem_private_key(pem.encode("utf-8"), password=None)  # type: ignore[return-value]
+    key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
+    store.save_secret(
+        "device_key",
+        key.private_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PrivateFormat.PKCS8,
+            encryption_algorithm=serialization.NoEncryption(),
+        ).decode("utf-8"),
+    )
+    return key
+
+
+def build_csr(device_cn: str = "jishu-device") -> str:
+    key = ensure_key_pair()
+    csr = (
+        x509.CertificateSigningRequestBuilder()
+        .subject_name(
+            x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, device_cn[:64])])
+        )
+        .sign(key, hashes.SHA256())
+    )
+    return csr.public_bytes(serialization.Encoding.PEM).decode("utf-8")
+
+
+def sign_challenge_message(message_obj: dict) -> str:
+    key = ensure_key_pair()
+    msg = json.dumps(message_obj, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode(
+        "utf-8"
+    )
+    sig = key.sign(msg, padding.PKCS1v15(), hashes.SHA256())
+    return base64.b64encode(sig).decode("ascii")
+
+
+def save_device_cert(cert_pem: str) -> None:
+    store.save_secret("device_cert", cert_pem)