instance-repo 0.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- instance_repo-0.7.0/PKG-INFO +96 -0
- instance_repo-0.7.0/README.md +83 -0
- instance_repo-0.7.0/instance_repo/__init__.py +32 -0
- instance_repo-0.7.0/instance_repo/cache.py +57 -0
- instance_repo-0.7.0/instance_repo/cli.py +401 -0
- instance_repo-0.7.0/instance_repo/clients/__init__.py +8 -0
- instance_repo-0.7.0/instance_repo/clients/config.py +103 -0
- instance_repo-0.7.0/instance_repo/clients/datasets.py +156 -0
- instance_repo-0.7.0/instance_repo/clients/instances.py +255 -0
- instance_repo-0.7.0/instance_repo/clients/reports.py +53 -0
- instance_repo-0.7.0/instance_repo/clients/scaffold.py +105 -0
- instance_repo-0.7.0/instance_repo/clients/versions.py +448 -0
- instance_repo-0.7.0/instance_repo/concurrency.py +72 -0
- instance_repo-0.7.0/instance_repo/content.py +52 -0
- instance_repo-0.7.0/instance_repo/endpoints.py +92 -0
- instance_repo-0.7.0/instance_repo/errors.py +172 -0
- instance_repo-0.7.0/instance_repo/image.py +100 -0
- instance_repo-0.7.0/instance_repo/loader.py +11 -0
- instance_repo-0.7.0/instance_repo/models.py +446 -0
- instance_repo-0.7.0/instance_repo/paths.py +155 -0
- instance_repo-0.7.0/instance_repo/release.py +162 -0
- instance_repo-0.7.0/instance_repo/repo.py +279 -0
- instance_repo-0.7.0/instance_repo/retry.py +81 -0
- instance_repo-0.7.0/instance_repo/store/__init__.py +19 -0
- instance_repo-0.7.0/instance_repo/store/acr.py +147 -0
- instance_repo-0.7.0/instance_repo/store/base.py +70 -0
- instance_repo-0.7.0/instance_repo/store/oss.py +247 -0
- instance_repo-0.7.0/instance_repo/store/registry.py +118 -0
- instance_repo-0.7.0/instance_repo/transport.py +372 -0
- instance_repo-0.7.0/instance_repo/validate.py +52 -0
- instance_repo-0.7.0/instance_repo.egg-info/PKG-INFO +96 -0
- instance_repo-0.7.0/instance_repo.egg-info/SOURCES.txt +36 -0
- instance_repo-0.7.0/instance_repo.egg-info/dependency_links.txt +1 -0
- instance_repo-0.7.0/instance_repo.egg-info/entry_points.txt +2 -0
- instance_repo-0.7.0/instance_repo.egg-info/requires.txt +10 -0
- instance_repo-0.7.0/instance_repo.egg-info/top_level.txt +1 -0
- instance_repo-0.7.0/pyproject.toml +46 -0
- instance_repo-0.7.0/setup.cfg +4 -0
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: instance-repo
|
|
3
|
+
Version: 0.7.0
|
|
4
|
+
Summary: InstanceRepo SDK — 评测 Instance 统一存储客户端(SDK-only;控制面由 apiserver 承载)
|
|
5
|
+
Requires-Python: >=3.10
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: tomli>=1.1.0; python_version < "3.11"
|
|
8
|
+
Provides-Extra: oss
|
|
9
|
+
Requires-Dist: oss2>=2.19; extra == "oss"
|
|
10
|
+
Provides-Extra: dev
|
|
11
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
12
|
+
Requires-Dist: coverage>=7; extra == "dev"
|
|
13
|
+
|
|
14
|
+
# instance-repo
|
|
15
|
+
|
|
16
|
+
InstanceRepo Python SDK 与 `irepo` 命令行工具——评测 Instance 的统一存储客户端。
|
|
17
|
+
|
|
18
|
+
SDK 只负责客户端逻辑;控制面(元数据、鉴权、STS 凭据签发)由 apiserver 承载,数据面
|
|
19
|
+
(OSS 读写、镜像推拉)经 apiserver 下发的临时凭据直连。SDK **不内置**任何 apiserver
|
|
20
|
+
域名、环境名、OSS bucket / endpoint / region、ACR registry / namespace 等基础设施信息,
|
|
21
|
+
这些值一律在运行时由调用方通过环境变量或构造参数注入(见下文“配置”)。
|
|
22
|
+
|
|
23
|
+
Python SDK 与 Go SDK 功能、错误码、行为逐字对齐。
|
|
24
|
+
|
|
25
|
+
## 安装
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
pip install instance-repo # SDK + irepo CLI(Python ≥ 3.10)
|
|
29
|
+
pip install "instance-repo[oss]" # 需要数据面 OSS 读写时,附带 oss2
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
镜像推拉依赖外部二进制 `skopeo`;未安装时相关操作会被跳过(`push_image=False` / `--no-image`)。
|
|
33
|
+
|
|
34
|
+
## 配置
|
|
35
|
+
|
|
36
|
+
所有基础设施地址在运行时注入,优先级:构造参数 > 环境变量 > 空(用到时才校验并报错)。
|
|
37
|
+
|
|
38
|
+
控制面:
|
|
39
|
+
|
|
40
|
+
- `INSTANCEREPO_API_BASE`:apiserver 根地址(必填,无内置默认)
|
|
41
|
+
- `IR_API_ENV`:下发的 `Env` 请求头值,apiserver 据此路由(原样下发,无别名映射)
|
|
42
|
+
- `INSTANCEREPO_TOKEN`:身份 token(`X-API-Key`)
|
|
43
|
+
- `IR_NETWORK`:数据面走 `internal`(默认)还是 `public`(VPC 外须设 `public`)
|
|
44
|
+
|
|
45
|
+
数据面(按需,仅在真正读写 OSS / 镜像时才需要):
|
|
46
|
+
|
|
47
|
+
- `IR_OSS_BUCKET`、`IR_ACR_HOST`、`IR_ACR_NAMESPACE`
|
|
48
|
+
- `IR_OSS_PREFIX`:bucket 内对象目录前缀,协议常量,默认 `swe/datasets`
|
|
49
|
+
- `IR_SCAFFOLD_BUCKET`、`IR_SCAFFOLD_ROOT`:scaffold 包存储位置
|
|
50
|
+
|
|
51
|
+
`profile` 名仅作标签,不再对应任何内置取值;也可用 `profile_overrides={...}` 传入等价字段。
|
|
52
|
+
|
|
53
|
+
## 快速开始
|
|
54
|
+
|
|
55
|
+
```python
|
|
56
|
+
from instance_repo import Repo
|
|
57
|
+
|
|
58
|
+
repo = Repo() # 从环境变量读取 api_base / token / 数据面配置
|
|
59
|
+
|
|
60
|
+
# 控制面元数据操作(不需要数据面配置)
|
|
61
|
+
datasets = repo.datasets.list()
|
|
62
|
+
versions = repo.versions.list("alibaba/mybench")
|
|
63
|
+
state = repo.versions.status("alibaba/mybench", "v1")
|
|
64
|
+
|
|
65
|
+
# 身份自查:返回当前 token 对应工号,用于拼 user-data 前缀 swe/user-data/users/{uid}/
|
|
66
|
+
uid = repo.whoami()
|
|
67
|
+
|
|
68
|
+
# 用户输入数据(插件 / 数据合成输入)
|
|
69
|
+
repo.upload_user_data(uid, "plugin.bin", "plugins/plugin.bin")
|
|
70
|
+
keys = repo.list_user_data(uid)
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
未配置 `INSTANCEREPO_API_BASE` 时,发起控制面请求会抛出明确的 `SchemaError` 引导设置该变量;
|
|
74
|
+
数据面操作在真正用到某字段(如 `oss_bucket`)时才校验,纯控制面操作不受影响。
|
|
75
|
+
|
|
76
|
+
## CLI
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
irepo --help
|
|
80
|
+
# 子命令:validate / push / pull / list / get / publish / grant / feedback
|
|
81
|
+
# report / whoami / user-data / create / claim / version
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
跨环境上架的目标数据面地址通过参数显式提供(SDK 不内置任何桶/仓地址):
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
irepo publish alibaba/mybench/v1 \
|
|
88
|
+
--target-bucket <your-oss-bucket> \
|
|
89
|
+
--target-acr <your-registry-host> --target-acr-namespace <your-namespace>
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## 版本与文档
|
|
93
|
+
|
|
94
|
+
版本号在 `pyproject.toml` / `instance_repo.__version__` / Go `version.go` /
|
|
95
|
+
conformance golden 四处保持一致。各环境的真实配置取值、端到端示例与错误码对照,见团队内部
|
|
96
|
+
部署文档。
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# instance-repo
|
|
2
|
+
|
|
3
|
+
InstanceRepo Python SDK 与 `irepo` 命令行工具——评测 Instance 的统一存储客户端。
|
|
4
|
+
|
|
5
|
+
SDK 只负责客户端逻辑;控制面(元数据、鉴权、STS 凭据签发)由 apiserver 承载,数据面
|
|
6
|
+
(OSS 读写、镜像推拉)经 apiserver 下发的临时凭据直连。SDK **不内置**任何 apiserver
|
|
7
|
+
域名、环境名、OSS bucket / endpoint / region、ACR registry / namespace 等基础设施信息,
|
|
8
|
+
这些值一律在运行时由调用方通过环境变量或构造参数注入(见下文“配置”)。
|
|
9
|
+
|
|
10
|
+
Python SDK 与 Go SDK 功能、错误码、行为逐字对齐。
|
|
11
|
+
|
|
12
|
+
## 安装
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
pip install instance-repo # SDK + irepo CLI(Python ≥ 3.10)
|
|
16
|
+
pip install "instance-repo[oss]" # 需要数据面 OSS 读写时,附带 oss2
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
镜像推拉依赖外部二进制 `skopeo`;未安装时相关操作会被跳过(`push_image=False` / `--no-image`)。
|
|
20
|
+
|
|
21
|
+
## 配置
|
|
22
|
+
|
|
23
|
+
所有基础设施地址在运行时注入,优先级:构造参数 > 环境变量 > 空(用到时才校验并报错)。
|
|
24
|
+
|
|
25
|
+
控制面:
|
|
26
|
+
|
|
27
|
+
- `INSTANCEREPO_API_BASE`:apiserver 根地址(必填,无内置默认)
|
|
28
|
+
- `IR_API_ENV`:下发的 `Env` 请求头值,apiserver 据此路由(原样下发,无别名映射)
|
|
29
|
+
- `INSTANCEREPO_TOKEN`:身份 token(`X-API-Key`)
|
|
30
|
+
- `IR_NETWORK`:数据面走 `internal`(默认)还是 `public`(VPC 外须设 `public`)
|
|
31
|
+
|
|
32
|
+
数据面(按需,仅在真正读写 OSS / 镜像时才需要):
|
|
33
|
+
|
|
34
|
+
- `IR_OSS_BUCKET`、`IR_ACR_HOST`、`IR_ACR_NAMESPACE`
|
|
35
|
+
- `IR_OSS_PREFIX`:bucket 内对象目录前缀,协议常量,默认 `swe/datasets`
|
|
36
|
+
- `IR_SCAFFOLD_BUCKET`、`IR_SCAFFOLD_ROOT`:scaffold 包存储位置
|
|
37
|
+
|
|
38
|
+
`profile` 名仅作标签,不再对应任何内置取值;也可用 `profile_overrides={...}` 传入等价字段。
|
|
39
|
+
|
|
40
|
+
## 快速开始
|
|
41
|
+
|
|
42
|
+
```python
|
|
43
|
+
from instance_repo import Repo
|
|
44
|
+
|
|
45
|
+
repo = Repo() # 从环境变量读取 api_base / token / 数据面配置
|
|
46
|
+
|
|
47
|
+
# 控制面元数据操作(不需要数据面配置)
|
|
48
|
+
datasets = repo.datasets.list()
|
|
49
|
+
versions = repo.versions.list("alibaba/mybench")
|
|
50
|
+
state = repo.versions.status("alibaba/mybench", "v1")
|
|
51
|
+
|
|
52
|
+
# 身份自查:返回当前 token 对应工号,用于拼 user-data 前缀 swe/user-data/users/{uid}/
|
|
53
|
+
uid = repo.whoami()
|
|
54
|
+
|
|
55
|
+
# 用户输入数据(插件 / 数据合成输入)
|
|
56
|
+
repo.upload_user_data(uid, "plugin.bin", "plugins/plugin.bin")
|
|
57
|
+
keys = repo.list_user_data(uid)
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
未配置 `INSTANCEREPO_API_BASE` 时,发起控制面请求会抛出明确的 `SchemaError` 引导设置该变量;
|
|
61
|
+
数据面操作在真正用到某字段(如 `oss_bucket`)时才校验,纯控制面操作不受影响。
|
|
62
|
+
|
|
63
|
+
## CLI
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
irepo --help
|
|
67
|
+
# 子命令:validate / push / pull / list / get / publish / grant / feedback
|
|
68
|
+
# report / whoami / user-data / create / claim / version
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
跨环境上架的目标数据面地址通过参数显式提供(SDK 不内置任何桶/仓地址):
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
irepo publish alibaba/mybench/v1 \
|
|
75
|
+
--target-bucket <your-oss-bucket> \
|
|
76
|
+
--target-acr <your-registry-host> --target-acr-namespace <your-namespace>
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## 版本与文档
|
|
80
|
+
|
|
81
|
+
版本号在 `pyproject.toml` / `instance_repo.__version__` / Go `version.go` /
|
|
82
|
+
conformance golden 四处保持一致。各环境的真实配置取值、端到端示例与错误码对照,见团队内部
|
|
83
|
+
部署文档。
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
"""instance_repo — 评测 Instance 统一存储 SDK(Python)。
|
|
2
|
+
|
|
3
|
+
唯一入口:所有 OSS/ACR 操作经 SDK,先向 apiserver 换 STS 再直连。控制面
|
|
4
|
+
(鉴权/元数据/STS/审计/release)全在 apiserver。
|
|
5
|
+
"""
|
|
6
|
+
from .repo import Repo
|
|
7
|
+
from .loader import load_instances
|
|
8
|
+
from .models import (
|
|
9
|
+
DatasetSeries, DatasetVersion, DatasetInstance, ManifestEntry,
|
|
10
|
+
OwnerReference, Feedback, Identity, PushResult,
|
|
11
|
+
RolloutReport, RolloutJob, ScaffoldConfig,
|
|
12
|
+
canonical_json, parse_feedbacks, append_feedback,
|
|
13
|
+
)
|
|
14
|
+
from .errors import (
|
|
15
|
+
InstanceRepoError, LayoutError, SchemaError, OwnerRefMissing, ImageMissing,
|
|
16
|
+
Immutable, DigestMismatch, Forbidden, NotVisible, TargetNotAllowed,
|
|
17
|
+
NetworkDenied, CredExpired, TransportError,
|
|
18
|
+
)
|
|
19
|
+
|
|
20
|
+
__version__ = "0.7.0"
|
|
21
|
+
|
|
22
|
+
__all__ = [
|
|
23
|
+
"Repo", "load_instances",
|
|
24
|
+
"DatasetSeries", "DatasetVersion", "DatasetInstance", "ManifestEntry",
|
|
25
|
+
"OwnerReference", "Feedback", "Identity", "PushResult",
|
|
26
|
+
"RolloutReport", "RolloutJob", "ScaffoldConfig",
|
|
27
|
+
"canonical_json", "parse_feedbacks", "append_feedback",
|
|
28
|
+
"InstanceRepoError", "LayoutError", "SchemaError", "OwnerRefMissing",
|
|
29
|
+
"ImageMissing", "Immutable", "DigestMismatch", "Forbidden", "NotVisible",
|
|
30
|
+
"TargetNotAllowed", "NetworkDenied", "CredExpired", "TransportError",
|
|
31
|
+
"__version__",
|
|
32
|
+
]
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
"""cache.py — 线程安全 TTL 缓存(复用 STS 凭据与 oss2 Bucket)。
|
|
2
|
+
|
|
3
|
+
按 key 缓存 value + 过期 epoch;命中要求 now < expiry - safety。expiry=None 视为不过期。
|
|
4
|
+
STS 凭据按 (resource,bucket,prefix)/(acr,host) 缓存、按签发返回的 Expiration 失效;
|
|
5
|
+
oss2 Bucket 按 (bucket,prefix,security_token) 缓存以复用底层连接池。
|
|
6
|
+
"""
|
|
7
|
+
from __future__ import annotations
|
|
8
|
+
|
|
9
|
+
import threading
|
|
10
|
+
import time
|
|
11
|
+
from datetime import datetime, timezone
|
|
12
|
+
from typing import Any
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
def parse_expiry(rfc3339: str | None) -> float | None:
|
|
16
|
+
"""RFC3339(如 ``2026-08-06T16:32:26Z``)→ epoch 秒;空/不可解析返回 None。
|
|
17
|
+
|
|
18
|
+
无时区信息(naive)的字符串按 UTC 解释,避免被本地时区偏移影响过期判断。
|
|
19
|
+
"""
|
|
20
|
+
if not rfc3339:
|
|
21
|
+
return None
|
|
22
|
+
s = rfc3339.strip().replace("Z", "+00:00")
|
|
23
|
+
try:
|
|
24
|
+
dt = datetime.fromisoformat(s)
|
|
25
|
+
except ValueError:
|
|
26
|
+
return None
|
|
27
|
+
if dt.tzinfo is None:
|
|
28
|
+
dt = dt.replace(tzinfo=timezone.utc)
|
|
29
|
+
return dt.timestamp()
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
class TTLCache:
|
|
33
|
+
"""线程安全 TTL 缓存。"""
|
|
34
|
+
|
|
35
|
+
def __init__(self, *, safety: float = 60.0) -> None:
|
|
36
|
+
self._d: dict[Any, tuple[Any, float | None]] = {}
|
|
37
|
+
self._lock = threading.Lock()
|
|
38
|
+
self._safety = safety
|
|
39
|
+
|
|
40
|
+
def get(self, key: Any) -> Any | None:
|
|
41
|
+
with self._lock:
|
|
42
|
+
item = self._d.get(key)
|
|
43
|
+
if item is None:
|
|
44
|
+
return None
|
|
45
|
+
value, exp = item
|
|
46
|
+
if exp is not None and time.time() >= exp - self._safety:
|
|
47
|
+
self._d.pop(key, None)
|
|
48
|
+
return None
|
|
49
|
+
return value
|
|
50
|
+
|
|
51
|
+
def put(self, key: Any, value: Any, expiry: float | None) -> None:
|
|
52
|
+
with self._lock:
|
|
53
|
+
self._d[key] = (value, expiry)
|
|
54
|
+
|
|
55
|
+
def clear(self) -> None:
|
|
56
|
+
with self._lock:
|
|
57
|
+
self._d.clear()
|
|
@@ -0,0 +1,401 @@
|
|
|
1
|
+
"""cli.py — irepo 命令行入口。
|
|
2
|
+
|
|
3
|
+
子命令:validate / push / list / get / pull / publish / grant / feedback / report /
|
|
4
|
+
create / claim / version。纯参数解析 + 分发到 Repo;便于单测(main(argv, repo=注入))。
|
|
5
|
+
"""
|
|
6
|
+
from __future__ import annotations
|
|
7
|
+
|
|
8
|
+
import argparse
|
|
9
|
+
import json
|
|
10
|
+
import sys
|
|
11
|
+
|
|
12
|
+
from .errors import InstanceRepoError
|
|
13
|
+
from .models import OwnerReference, Feedback, RolloutReport, DatasetSeries
|
|
14
|
+
from .repo import Repo
|
|
15
|
+
|
|
16
|
+
_STATUS_CHOICES = ["draft", "published", "deprecated", "archived"]
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
def build_parser() -> argparse.ArgumentParser:
|
|
20
|
+
ap = argparse.ArgumentParser(prog="irepo", description="InstanceRepo CLI")
|
|
21
|
+
ap.add_argument("--profile", default="default",
|
|
22
|
+
help="数据面 profile 标签(仅作标识;OSS bucket / ACR host 等经 "
|
|
23
|
+
"IR_OSS_BUCKET / IR_ACR_HOST 等环境变量提供,见部署文档)")
|
|
24
|
+
ap.add_argument("--api-env", default=None,
|
|
25
|
+
help="下发的 Env 请求头值(服务端据此路由;亦可用 IR_API_ENV);"
|
|
26
|
+
"apiserver 根地址用 INSTANCEREPO_API_BASE 提供")
|
|
27
|
+
sub = ap.add_subparsers(dest="cmd", required=True)
|
|
28
|
+
|
|
29
|
+
v = sub.add_parser("validate", help="本地校验 Instance 目录")
|
|
30
|
+
v.add_argument("dir")
|
|
31
|
+
|
|
32
|
+
p = sub.add_parser("push", help="上传 Instance(元数据先行)")
|
|
33
|
+
p.add_argument("dir")
|
|
34
|
+
p.add_argument("--dataset", required=True)
|
|
35
|
+
p.add_argument("--version", required=True)
|
|
36
|
+
p.add_argument("--owner-ref", required=True, help="OwnerReference JSON")
|
|
37
|
+
p.add_argument("--split", default="default")
|
|
38
|
+
p.add_argument("--no-image", action="store_true")
|
|
39
|
+
p.add_argument("--overwrite", action="store_true")
|
|
40
|
+
p.add_argument("--no-register", action="store_true",
|
|
41
|
+
help="跳过 DB 写入(register/commit),只传 OSS,之后由 ingest 扫描入库;"
|
|
42
|
+
"apiserver 未实现实例写端点时必须用它")
|
|
43
|
+
p.add_argument("--no-verify", action="store_true",
|
|
44
|
+
help="跳过上传后的回读校验(默认校验 OSS 对象/ACR 镜像确已落库)")
|
|
45
|
+
|
|
46
|
+
lst = sub.add_parser("list", help="列出 dataset/version 下实例")
|
|
47
|
+
lst.add_argument("ref", help="{L1}/{L2}/{version}")
|
|
48
|
+
lst.add_argument("--split", default=None,
|
|
49
|
+
help="按 split 过滤;缺省为跨 split 检索")
|
|
50
|
+
|
|
51
|
+
pl = sub.add_parser("pull", help="拉取并解包(digest 校验)")
|
|
52
|
+
pl.add_argument("ref", help="{L1}/{L2}/{version}/{id}")
|
|
53
|
+
pl.add_argument("-o", "--out", default="./work")
|
|
54
|
+
pl.add_argument("--split", default="default",
|
|
55
|
+
help="实例所在 split(写入时用了非默认 split 就必须带上)")
|
|
56
|
+
|
|
57
|
+
pub = sub.add_parser("publish", help="上架到生产(benchmark-release)")
|
|
58
|
+
pub.add_argument("ref", help="{L1}/{L2}/{version}")
|
|
59
|
+
pub.add_argument("--instances", default=None,
|
|
60
|
+
help="逗号分隔的 KEEP 实例 id;granularity=instance 或 images=on 时必填")
|
|
61
|
+
pub.add_argument("--target", default="", help="目标 profile 标签(仅作标识)")
|
|
62
|
+
pub.add_argument("--target-bucket", default=None,
|
|
63
|
+
help="目标 OSS bucket(跨环境上架必填;SDK 不内置任何桶地址)")
|
|
64
|
+
pub.add_argument("--target-acr", default=None, help="目标 ACR host(images=on 时用)")
|
|
65
|
+
pub.add_argument("--target-acr-namespace", default=None, help="目标 ACR namespace")
|
|
66
|
+
pub.add_argument("--granularity", choices=["dataset", "instance"], default="dataset")
|
|
67
|
+
pub.add_argument("--images", choices=["off", "on"], default="off")
|
|
68
|
+
pub.add_argument("--overwrite", action="store_true")
|
|
69
|
+
pub.add_argument("--wait-approval", action="store_true", help="等待人工审批直到终态")
|
|
70
|
+
pub.add_argument("--description", default="")
|
|
71
|
+
pub.add_argument("--request-json", default=None,
|
|
72
|
+
help="逃生舱:直接透传 workflow 请求体 JSON 文件(给定时忽略上面 typed 参数)")
|
|
73
|
+
|
|
74
|
+
g = sub.add_parser("grant", help="授权")
|
|
75
|
+
g.add_argument("dataset")
|
|
76
|
+
g.add_argument("--user", required=True)
|
|
77
|
+
g.add_argument("--role", required=True)
|
|
78
|
+
|
|
79
|
+
fb = sub.add_parser("feedback", help="回写反馈(append)")
|
|
80
|
+
fb.add_argument("ref", help="{L1}/{L2}/{version}/{id}")
|
|
81
|
+
fb.add_argument("--type", required=True)
|
|
82
|
+
fb.add_argument("--id", required=True)
|
|
83
|
+
fb.add_argument("--score", type=float, required=True)
|
|
84
|
+
fb.add_argument("--message", default="")
|
|
85
|
+
|
|
86
|
+
rp = sub.add_parser("report", help="质检 report 存取(OSS)")
|
|
87
|
+
rp.add_argument("action", choices=["get", "put", "assemble"])
|
|
88
|
+
rp.add_argument("ref", help="get/put: {L1}/{L2}/{version}/{id};assemble: {L1}/{L2}/{version}")
|
|
89
|
+
rp.add_argument("--file", default=None, help="put: 输入 report.jsonl;get: 输出(缺省打印);assemble: 输出 jsonl")
|
|
90
|
+
rp.add_argument("--instances", default=None, help="assemble: 逗号分隔的实例 id")
|
|
91
|
+
rp.add_argument("--overwrite", action="store_true")
|
|
92
|
+
|
|
93
|
+
sub.add_parser("whoami", help="打印当前 apikey 对应的工号(user_id)")
|
|
94
|
+
|
|
95
|
+
ud = sub.add_parser("user-data", help="用户输入数据(插件/合成输入) put/get/ls")
|
|
96
|
+
ud.add_argument("action", choices=["put", "get", "ls"])
|
|
97
|
+
ud.add_argument("pos", nargs="*",
|
|
98
|
+
help="put: <local> <remote>;get: <remote> <local>;ls: [subpath]")
|
|
99
|
+
ud.add_argument("--uid", default=None,
|
|
100
|
+
help="目标 uid(缺省=自己 whoami;系统白名单读他人时指定)")
|
|
101
|
+
ud.add_argument("--overwrite", action="store_true", help="put 时覆盖已存在对象")
|
|
102
|
+
|
|
103
|
+
cr = sub.add_parser("create", help="创建 DatasetSeries")
|
|
104
|
+
cr.add_argument("dataset", help="{L1}/{L2}")
|
|
105
|
+
cr.add_argument("--benchmark-id", default=None,
|
|
106
|
+
help="绑定的 benchmark source_id(默认必填;apiserver 据它派生 teacher)")
|
|
107
|
+
cr.add_argument("--visibility", choices=["public", "private"], default="private")
|
|
108
|
+
cr.add_argument("--display-name", default=None)
|
|
109
|
+
cr.add_argument("--description", default=None)
|
|
110
|
+
cr.add_argument("--default-scaffold", default=None)
|
|
111
|
+
cr.add_argument("--owner", default=None, help="归属人工号(缺省取调用方)")
|
|
112
|
+
cr.add_argument("--owner-team-id", default=None)
|
|
113
|
+
cr.add_argument("--workspace-id", default=None, help="归属 workspace/queue(可选)")
|
|
114
|
+
cr.add_argument("--no-require-benchmark", action="store_true",
|
|
115
|
+
help="跳过 benchmark_id 本地必填校验(仅服务端关掉强制绑定时用)")
|
|
116
|
+
|
|
117
|
+
cl = sub.add_parser("claim", help="认领 dataset(成为 admin)")
|
|
118
|
+
cl.add_argument("dataset_id", help="dataset 业务 ID(ds- 前缀)")
|
|
119
|
+
cl.add_argument("--workspace-id", required=True, help="归属 workspace/queue ID")
|
|
120
|
+
cl.add_argument("--benchmark-id", default=None,
|
|
121
|
+
help="绑定的 benchmark source_id(默认必填;apiserver 据它派生 teacher)")
|
|
122
|
+
cl.add_argument("--owner", default=None,
|
|
123
|
+
help="归属人工号(缺省取调用方;已有非空 owner 不会被覆盖)")
|
|
124
|
+
cl.add_argument("--no-require-benchmark", action="store_true",
|
|
125
|
+
help="跳过 benchmark_id 本地必填校验(仅服务端关掉强制绑定时用)")
|
|
126
|
+
|
|
127
|
+
gt = sub.add_parser("get", help="取单个实例元数据")
|
|
128
|
+
gt.add_argument("ref", help="{L1}/{L2}/{version}/{id}")
|
|
129
|
+
gt.add_argument("--split", default=None,
|
|
130
|
+
help="实例所在 split;缺省为跨 split 检索")
|
|
131
|
+
|
|
132
|
+
ver = sub.add_parser("version", help="版本管理(create/list/status/update/set-run-type)")
|
|
133
|
+
ver.add_argument("action",
|
|
134
|
+
choices=["create", "list", "status", "update", "set-run-type"])
|
|
135
|
+
ver.add_argument("ref",
|
|
136
|
+
help="create/list: {L1}/{L2};"
|
|
137
|
+
"status/update/set-run-type: {L1}/{L2}/{version}")
|
|
138
|
+
ver.add_argument("--version", default=None, help="create 用:版本号")
|
|
139
|
+
ver.add_argument("--storage-path", default=None,
|
|
140
|
+
help="create 用:oss://{bucket}/{prefix}")
|
|
141
|
+
ver.add_argument("--storage-type", choices=["oss"], default="oss")
|
|
142
|
+
ver.add_argument("--splits", default=None, help="逗号分隔 split 名")
|
|
143
|
+
ver.add_argument("--status", choices=_STATUS_CHOICES, default=None,
|
|
144
|
+
help="update 用:版本状态流转")
|
|
145
|
+
ver.add_argument("--run-type", choices=["train", "eval"], default=None,
|
|
146
|
+
help="version 级用途标签")
|
|
147
|
+
ver.add_argument("--split-run-types", default=None,
|
|
148
|
+
help="split 级打标,格式 name=train,name2=eval(与 --run-type 互斥)")
|
|
149
|
+
return ap
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
def _split_ref(ref: str, parts: int) -> list[str]:
|
|
153
|
+
segs = ref.split("/")
|
|
154
|
+
if len(segs) != parts:
|
|
155
|
+
raise SystemExit(f"ref must have {parts} segments, got {ref!r}")
|
|
156
|
+
return segs
|
|
157
|
+
|
|
158
|
+
|
|
159
|
+
def _parse_split_run_types(s: str) -> list[dict]:
|
|
160
|
+
"""解析 CLI 的 ``name=train,name2=eval`` 格式为 [{name, run_type}] 列表。
|
|
161
|
+
|
|
162
|
+
run_type 合法性交由 SDK versions.update 校验(会列出合法值)。"""
|
|
163
|
+
out: list[dict] = []
|
|
164
|
+
for pair in s.split(","):
|
|
165
|
+
pair = pair.strip()
|
|
166
|
+
if not pair:
|
|
167
|
+
continue
|
|
168
|
+
if "=" not in pair:
|
|
169
|
+
raise SystemExit(
|
|
170
|
+
f"--split-run-types 每项须为 name=run_type,got {pair!r}")
|
|
171
|
+
name, rt = pair.split("=", 1)
|
|
172
|
+
out.append({"name": name.strip(), "run_type": rt.strip()})
|
|
173
|
+
return out
|
|
174
|
+
|
|
175
|
+
|
|
176
|
+
def main(argv=None, *, repo: Repo | None = None, out=None) -> int:
|
|
177
|
+
args = build_parser().parse_args(argv)
|
|
178
|
+
out = out or sys.stdout
|
|
179
|
+
r = repo or Repo(profile=args.profile, api_env=getattr(args, "api_env", None))
|
|
180
|
+
|
|
181
|
+
if args.cmd == "validate":
|
|
182
|
+
rep = r.instances.validate(args.dir)
|
|
183
|
+
print(f"OK {rep.instance_id} (metadata keys: {sorted(rep.metadata)})", file=out)
|
|
184
|
+
return 0
|
|
185
|
+
|
|
186
|
+
if args.cmd == "push":
|
|
187
|
+
owner = OwnerReference.from_json(args.owner_ref)
|
|
188
|
+
res = r.instances.push(args.dir, args.dataset, args.version,
|
|
189
|
+
owner_reference=owner, split=args.split,
|
|
190
|
+
push_image=not args.no_image,
|
|
191
|
+
overwrite=args.overwrite,
|
|
192
|
+
register=not args.no_register,
|
|
193
|
+
verify=not args.no_verify)
|
|
194
|
+
print(f"{res.instance_id} uploaded={res.uploaded} "
|
|
195
|
+
f"image_pushed={res.image_pushed}", file=out)
|
|
196
|
+
return 0
|
|
197
|
+
|
|
198
|
+
if args.cmd == "list":
|
|
199
|
+
l1, l2, version = _split_ref(args.ref, 3)
|
|
200
|
+
for inst in r.instances.list(f"{l1}/{l2}", version, args.split):
|
|
201
|
+
print(f"{inst.instance_id}\t{inst.difficulty}\t{inst.docker_image}", file=out)
|
|
202
|
+
return 0
|
|
203
|
+
|
|
204
|
+
if args.cmd == "pull":
|
|
205
|
+
l1, l2, version, iid = _split_ref(args.ref, 4)
|
|
206
|
+
path = r.instances.pull(f"{l1}/{l2}", version, iid, args.out,
|
|
207
|
+
split=args.split)
|
|
208
|
+
print(str(path), file=out)
|
|
209
|
+
return 0
|
|
210
|
+
|
|
211
|
+
if args.cmd == "publish":
|
|
212
|
+
l1, l2, version = _split_ref(args.ref, 3)
|
|
213
|
+
if args.request_json:
|
|
214
|
+
with open(args.request_json, encoding="utf-8") as f:
|
|
215
|
+
body = json.load(f)
|
|
216
|
+
wf = r.versions.publish(f"{l1}/{l2}", version, request_body=body,
|
|
217
|
+
wait_approval=args.wait_approval)
|
|
218
|
+
else:
|
|
219
|
+
insts = ([s.strip() for s in args.instances.split(",") if s.strip()]
|
|
220
|
+
if args.instances else None)
|
|
221
|
+
# 目标数据面配置由 CLI 显式提供(SDK 不内置任何环境的桶/仓地址)。
|
|
222
|
+
tgt_ovr = {}
|
|
223
|
+
if args.target_bucket:
|
|
224
|
+
tgt_ovr["oss_bucket"] = args.target_bucket
|
|
225
|
+
if args.target_acr:
|
|
226
|
+
tgt_ovr["acr_host"] = args.target_acr
|
|
227
|
+
if args.target_acr_namespace:
|
|
228
|
+
tgt_ovr["acr_namespace"] = args.target_acr_namespace
|
|
229
|
+
wf = r.versions.publish(
|
|
230
|
+
f"{l1}/{l2}", version, instances=insts, target=args.target,
|
|
231
|
+
target_overrides=(tgt_ovr or None),
|
|
232
|
+
granularity=args.granularity, images=args.images,
|
|
233
|
+
overwrite=args.overwrite, wait_approval=args.wait_approval,
|
|
234
|
+
description=args.description)
|
|
235
|
+
print(f"workflow {wf.get('id','')} status={wf.get('status','')}", file=out)
|
|
236
|
+
return 0
|
|
237
|
+
|
|
238
|
+
if args.cmd == "whoami":
|
|
239
|
+
print(r.whoami(), file=out)
|
|
240
|
+
return 0
|
|
241
|
+
|
|
242
|
+
if args.cmd == "user-data":
|
|
243
|
+
uid = args.uid or r.whoami()
|
|
244
|
+
if args.action == "put":
|
|
245
|
+
if len(args.pos) < 2:
|
|
246
|
+
raise SystemExit("user-data put 需要 <local> <remote>")
|
|
247
|
+
res = r.upload_user_data(uid, args.pos[0], args.pos[1],
|
|
248
|
+
overwrite=args.overwrite)
|
|
249
|
+
tag = "skipped (exists): " if res.get("skipped") else "put: "
|
|
250
|
+
print(tag + res.get("key", ""), file=out)
|
|
251
|
+
return 0
|
|
252
|
+
if args.action == "get":
|
|
253
|
+
if len(args.pos) < 2:
|
|
254
|
+
raise SystemExit("user-data get 需要 <remote> <local>")
|
|
255
|
+
path = r.download_user_data(uid, args.pos[0], args.pos[1])
|
|
256
|
+
print(f"got: {path}", file=out)
|
|
257
|
+
return 0
|
|
258
|
+
# ls
|
|
259
|
+
subpath = args.pos[0] if args.pos else ""
|
|
260
|
+
for k in r.list_user_data(uid, subpath):
|
|
261
|
+
print(k, file=out)
|
|
262
|
+
return 0
|
|
263
|
+
|
|
264
|
+
if args.cmd == "grant":
|
|
265
|
+
r.datasets.grant(args.dataset, args.user, args.role)
|
|
266
|
+
print(f"granted {args.role} to {args.user} on {args.dataset}", file=out)
|
|
267
|
+
return 0
|
|
268
|
+
|
|
269
|
+
if args.cmd == "report":
|
|
270
|
+
if args.action == "assemble":
|
|
271
|
+
l1, l2, version = _split_ref(args.ref, 3)
|
|
272
|
+
insts = [s.strip() for s in (args.instances or "").split(",") if s.strip()]
|
|
273
|
+
dest = args.file or "train.jsonl"
|
|
274
|
+
path = r.reports.assemble_jsonl(f"{l1}/{l2}", version, insts, dest)
|
|
275
|
+
print(str(path), file=out)
|
|
276
|
+
return 0
|
|
277
|
+
l1, l2, version, iid = _split_ref(args.ref, 4)
|
|
278
|
+
if args.action == "put":
|
|
279
|
+
with open(args.file, encoding="utf-8") as f:
|
|
280
|
+
rpt = RolloutReport.from_jsonl_line(f.read().strip())
|
|
281
|
+
# ref 与 report 内容必须一致,否则会把 report 写到与 ref 不符的 key(静默错位)
|
|
282
|
+
if rpt.datasetname != f"{l1}/{l2}" or rpt.instanceid != iid:
|
|
283
|
+
raise SystemExit(
|
|
284
|
+
f"ref {args.ref} 与 report 内容不一致 "
|
|
285
|
+
f"(report={rpt.datasetname}/{rpt.instanceid})")
|
|
286
|
+
r.reports.put(rpt, version, overwrite=args.overwrite)
|
|
287
|
+
print(f"report put {rpt.instanceid}", file=out)
|
|
288
|
+
return 0
|
|
289
|
+
# get
|
|
290
|
+
rpt = r.reports.get(f"{l1}/{l2}", version, iid)
|
|
291
|
+
line = rpt.to_jsonl_line()
|
|
292
|
+
if args.file:
|
|
293
|
+
with open(args.file, "w", encoding="utf-8") as f:
|
|
294
|
+
f.write(line + "\n")
|
|
295
|
+
print(str(args.file), file=out)
|
|
296
|
+
else:
|
|
297
|
+
print(line, file=out)
|
|
298
|
+
return 0
|
|
299
|
+
|
|
300
|
+
if args.cmd == "feedback":
|
|
301
|
+
l1, l2, version, iid = _split_ref(args.ref, 4)
|
|
302
|
+
fb = Feedback(type=args.type, id=args.id, score=args.score, message=args.message)
|
|
303
|
+
r.instances.add_feedback(f"{l1}/{l2}", version, iid, fb)
|
|
304
|
+
print(f"feedback appended to {iid}", file=out)
|
|
305
|
+
return 0
|
|
306
|
+
|
|
307
|
+
if args.cmd == "create":
|
|
308
|
+
l1, l2 = _split_ref(args.dataset, 2)
|
|
309
|
+
# teacher 由 apiserver 从 benchmark 派生,CLI 不再提供 teacher 参数。
|
|
310
|
+
series = DatasetSeries(
|
|
311
|
+
dataset_name=args.dataset, dataset_l1=l1, dataset_l2=l2,
|
|
312
|
+
benchmark_id=args.benchmark_id,
|
|
313
|
+
visibility=args.visibility,
|
|
314
|
+
display_name=args.display_name,
|
|
315
|
+
description=args.description,
|
|
316
|
+
default_scaffold=args.default_scaffold,
|
|
317
|
+
owner=args.owner,
|
|
318
|
+
owner_team_id=args.owner_team_id,
|
|
319
|
+
workspace_id=args.workspace_id)
|
|
320
|
+
created = r.datasets.create(
|
|
321
|
+
series, require_benchmark=not args.no_require_benchmark)
|
|
322
|
+
print(f"created {created.dataset_name} (dataset_id={created.dataset_id})",
|
|
323
|
+
file=out)
|
|
324
|
+
return 0
|
|
325
|
+
|
|
326
|
+
if args.cmd == "claim":
|
|
327
|
+
res = r.datasets.claim(
|
|
328
|
+
args.dataset_id, args.workspace_id, args.benchmark_id or "",
|
|
329
|
+
owner=args.owner,
|
|
330
|
+
require_benchmark=not args.no_require_benchmark)
|
|
331
|
+
print(f"claimed {res.get('dataset_id', args.dataset_id)} "
|
|
332
|
+
f"by {res.get('user_id', '')}", file=out)
|
|
333
|
+
return 0
|
|
334
|
+
|
|
335
|
+
if args.cmd == "get":
|
|
336
|
+
l1, l2, version, iid = _split_ref(args.ref, 4)
|
|
337
|
+
inst = r.instances.get(f"{l1}/{l2}", version, iid, args.split)
|
|
338
|
+
# 实例不存在时库层返回 None(apiserver 92003),显式区分"查不到"与"字段为空"。
|
|
339
|
+
if inst is None:
|
|
340
|
+
raise SystemExit(f"instance {args.ref} not found")
|
|
341
|
+
print(f"{inst.instance_id}\t{inst.difficulty}\t{inst.docker_image}", file=out)
|
|
342
|
+
return 0
|
|
343
|
+
|
|
344
|
+
if args.cmd == "version":
|
|
345
|
+
srt = (_parse_split_run_types(args.split_run_types)
|
|
346
|
+
if args.split_run_types else None)
|
|
347
|
+
if args.action in ("create", "list"):
|
|
348
|
+
l1, l2 = _split_ref(args.ref, 2)
|
|
349
|
+
dataset = f"{l1}/{l2}"
|
|
350
|
+
if args.action == "list":
|
|
351
|
+
for v in r.versions.list(dataset):
|
|
352
|
+
print(f"{v.version}\t{v.status}\t{v.storage_path}", file=out)
|
|
353
|
+
return 0
|
|
354
|
+
if not args.version or not args.storage_path:
|
|
355
|
+
raise SystemExit("version create 需要 --version 与 --storage-path")
|
|
356
|
+
splits = ([s.strip() for s in args.splits.split(",") if s.strip()]
|
|
357
|
+
if args.splits else None)
|
|
358
|
+
dv = r.versions.create(dataset, args.version,
|
|
359
|
+
storage_path=args.storage_path, splits=splits,
|
|
360
|
+
status=args.status or "draft",
|
|
361
|
+
storage_type=args.storage_type)
|
|
362
|
+
print(f"created version {dv.dataset}/{dv.version} status={dv.status}",
|
|
363
|
+
file=out)
|
|
364
|
+
return 0
|
|
365
|
+
l1, l2, version = _split_ref(args.ref, 3)
|
|
366
|
+
dataset = f"{l1}/{l2}"
|
|
367
|
+
if args.action == "status":
|
|
368
|
+
print(r.versions.status(dataset, version), file=out)
|
|
369
|
+
return 0
|
|
370
|
+
if args.action == "set-run-type":
|
|
371
|
+
r.versions.set_run_type(dataset, version,
|
|
372
|
+
run_type=args.run_type, split_run_types=srt)
|
|
373
|
+
print(f"run_type set on {dataset}/{version}", file=out)
|
|
374
|
+
return 0
|
|
375
|
+
# update
|
|
376
|
+
splits = ([s.strip() for s in args.splits.split(",") if s.strip()]
|
|
377
|
+
if args.splits else None)
|
|
378
|
+
r.versions.update(dataset, version, status=args.status,
|
|
379
|
+
splits=splits, run_type=args.run_type,
|
|
380
|
+
split_run_types=srt)
|
|
381
|
+
print(f"updated {dataset}/{version}", file=out)
|
|
382
|
+
return 0
|
|
383
|
+
|
|
384
|
+
return 2 # pragma: no cover - argparse 已保证 cmd 合法
|
|
385
|
+
|
|
386
|
+
|
|
387
|
+
def cli_main(argv=None) -> int:
|
|
388
|
+
"""入口包装:把 SDK 异常收敛成单行报错,不向用户抛 Python traceback。
|
|
389
|
+
|
|
390
|
+
与 Go CLI 的 ``irepo: error: …`` 输出对齐。``main()`` 保持原样抛异常,
|
|
391
|
+
便于单测直接断言异常类型。
|
|
392
|
+
"""
|
|
393
|
+
try:
|
|
394
|
+
return main(argv)
|
|
395
|
+
except InstanceRepoError as e:
|
|
396
|
+
print(f"irepo: error: {e}", file=sys.stderr)
|
|
397
|
+
return 1
|
|
398
|
+
|
|
399
|
+
|
|
400
|
+
if __name__ == "__main__": # pragma: no cover
|
|
401
|
+
raise SystemExit(cli_main())
|