@zhuoyuezs/ml-platform 0.1.4 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CI-ARTIFACT.md +21 -0
- package/DEVELOPMENT.md +7 -5
- package/checksums.json +13 -13
- package/package.json +6 -3
- package/release.json +7 -7
- package/runtime/business-client/README.md +17 -0
- package/runtime/business-client/package-lock.json +2 -2
- package/runtime/business-client/package.json +4 -2
- package/runtime/business-client/src/cli.js +50 -14
- package/runtime/business-client/src/http.js +46 -0
- package/skills/feature-management/SKILL.md +28 -0
- package/skills/feature-management/references/platform-capability-guide.md +29 -0
package/CI-ARTIFACT.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# CI npm artifact
|
|
2
|
+
|
|
3
|
+
每次 `dev`/`main` 分支更新、合并请求或从 GitLab UI 手动运行的 Pipeline 都会生成完整的
|
|
4
|
+
`@zhuoyuezs/ml-platform` tarball。
|
|
5
|
+
GitLab pipeline 的 `npm:ml-platform-release` job 会将 `business-client-js` 放入正式包的
|
|
6
|
+
`runtime/business-client`,并上传已验收的 `.tgz` 与 `.sha256` 文件。
|
|
7
|
+
|
|
8
|
+
创建与 `npm/ml-platform/package.json` 版本一致的 SemVer tag(例如 `v0.1.5`)后,
|
|
9
|
+
同一个已验收归档会自动发布到 npmjs。发布 job 需要在 GitLab CI/CD Variables 中配置
|
|
10
|
+
受保护且 masked 的 `NPM_TOKEN`;普通分支、手动 Pipeline 和 Merge Request 不会发布 npm。
|
|
11
|
+
|
|
12
|
+
下载 artifact 后只需要 Node.js 18+:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
sha256sum -c zhuoyuezs-ml-platform-*.tgz.sha256
|
|
16
|
+
npm install --global ./zhuoyuezs-ml-platform-*.tgz
|
|
17
|
+
ml-platform version
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
运行已安装的 CLI 不需要 Python、`uv`、Git 仓库或平台源码。`uv` 只在 CI 构建阶段用于
|
|
21
|
+
执行现有的 release staging 脚本。
|
package/DEVELOPMENT.md
CHANGED
|
@@ -146,8 +146,9 @@ Host 副本、日志、会话数据、attic、凭证和实验文件。
|
|
|
146
146
|
- Codex project/user scope 和显式 Skills 目录;
|
|
147
147
|
- 最终归档敏感文件和重复 client 检查。
|
|
148
148
|
|
|
149
|
-
完成标准:同一个不可变 `.tgz`
|
|
150
|
-
|
|
149
|
+
完成标准:同一个不可变 `.tgz` 通过全部验收,并作为唯一允许发布的输入。GitLab CI 仅在
|
|
150
|
+
SemVer tag(`vX.Y.Z` 或 `X.Y.Z`)与 `package.json.version` 一致、且配置了 `NPM_TOKEN` 时
|
|
151
|
+
自动执行 npmjs 发布;dev、main 和 Merge Request pipeline 只生成并保存 artifact。
|
|
151
152
|
|
|
152
153
|
## 4. package.json
|
|
153
154
|
|
|
@@ -260,9 +261,10 @@ npx @zhuoyuezs/ml-platform@<version> install
|
|
|
260
261
|
- 不执行 GitHub Release 或其他未授权的外部发布;
|
|
261
262
|
- candidate 验证后不得用相同版本重新构建不同内容。
|
|
262
263
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
264
|
+
受控 GitLab CI 使用 masked/protected 的 `NPM_TOKEN` 临时配置 npm auth,并发布 builder 生成的
|
|
265
|
+
同一个 `.tgz`;token 不写入仓库或归档。本地登录不能替代 CI provenance,当前 token 发布也不
|
|
266
|
+
宣称包具有 provenance 来源证明。后续可迁移到 npm Trusted Publishing 和 OIDC,并在受支持的
|
|
267
|
+
公开 CI 环境执行 `npm publish --provenance`。
|
|
266
268
|
|
|
267
269
|
## 9. 参考与取舍
|
|
268
270
|
|
package/checksums.json
CHANGED
|
@@ -2,18 +2,18 @@
|
|
|
2
2
|
"files": [
|
|
3
3
|
{
|
|
4
4
|
"path": "runtime/business-client/README.md",
|
|
5
|
-
"sha256": "sha256:
|
|
6
|
-
"size_bytes":
|
|
5
|
+
"sha256": "sha256:72b0cbbb18a7d0f1bca5d97706588fbebcd2781833ae023bae0317be13342601",
|
|
6
|
+
"size_bytes": 1880
|
|
7
7
|
},
|
|
8
8
|
{
|
|
9
9
|
"path": "runtime/business-client/package-lock.json",
|
|
10
|
-
"sha256": "sha256:
|
|
10
|
+
"sha256": "sha256:d5e7102ec51df3192ae901908f36cf0380aba5a0421b9824877403ab034d6740",
|
|
11
11
|
"size_bytes": 381
|
|
12
12
|
},
|
|
13
13
|
{
|
|
14
14
|
"path": "runtime/business-client/package.json",
|
|
15
|
-
"sha256": "sha256:
|
|
16
|
-
"size_bytes":
|
|
15
|
+
"sha256": "sha256:b784fa27db6e40f72da11b9bca007155146174bc33edc4511c61d4ea4af93cf1",
|
|
16
|
+
"size_bytes": 508
|
|
17
17
|
},
|
|
18
18
|
{
|
|
19
19
|
"path": "runtime/business-client/src/catalog.js",
|
|
@@ -22,8 +22,8 @@
|
|
|
22
22
|
},
|
|
23
23
|
{
|
|
24
24
|
"path": "runtime/business-client/src/cli.js",
|
|
25
|
-
"sha256": "sha256:
|
|
26
|
-
"size_bytes":
|
|
25
|
+
"sha256": "sha256:f2634e346a4c5bdc42c81e316791906f809b768a77672e36d29cc51410c010ed",
|
|
26
|
+
"size_bytes": 29818
|
|
27
27
|
},
|
|
28
28
|
{
|
|
29
29
|
"path": "runtime/business-client/src/config.js",
|
|
@@ -32,13 +32,13 @@
|
|
|
32
32
|
},
|
|
33
33
|
{
|
|
34
34
|
"path": "runtime/business-client/src/http.js",
|
|
35
|
-
"sha256": "sha256:
|
|
36
|
-
"size_bytes":
|
|
35
|
+
"sha256": "sha256:e515b39f19e7b5e7d6c2388e9cf2d8e2c9fd18b585bb53bd4f8eeeb7412af81c",
|
|
36
|
+
"size_bytes": 15192
|
|
37
37
|
},
|
|
38
38
|
{
|
|
39
39
|
"path": "skills/feature-management/SKILL.md",
|
|
40
|
-
"sha256": "sha256:
|
|
41
|
-
"size_bytes":
|
|
40
|
+
"sha256": "sha256:7b58cc4ff2a7b83a29a6dbf9ad54e9e4a02efd332cd01c5d8fb83020b51ea506",
|
|
41
|
+
"size_bytes": 29676
|
|
42
42
|
},
|
|
43
43
|
{
|
|
44
44
|
"path": "skills/feature-management/agents/openai.yaml",
|
|
@@ -107,8 +107,8 @@
|
|
|
107
107
|
},
|
|
108
108
|
{
|
|
109
109
|
"path": "skills/feature-management/references/platform-capability-guide.md",
|
|
110
|
-
"sha256": "sha256:
|
|
111
|
-
"size_bytes":
|
|
110
|
+
"sha256": "sha256:40d0f43a30688c5852ef09203ec35b4dc2c6b9db4bc7dd26dfae52e12d10e9ee",
|
|
111
|
+
"size_bytes": 4257
|
|
112
112
|
}
|
|
113
113
|
],
|
|
114
114
|
"schema_version": "data_platform.ml_platform_checksums/v1"
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@zhuoyuezs/ml-platform",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.5",
|
|
4
4
|
"description": "ML 数据平台 business CLI 与 Agent Skills 管理工具",
|
|
5
5
|
"bin": {
|
|
6
6
|
"ml-platform": "scripts/main.js"
|
|
@@ -13,10 +13,13 @@
|
|
|
13
13
|
"release.json",
|
|
14
14
|
"checksums.json",
|
|
15
15
|
"README.md",
|
|
16
|
-
"DEVELOPMENT.md"
|
|
16
|
+
"DEVELOPMENT.md",
|
|
17
|
+
"CI-ARTIFACT.md"
|
|
17
18
|
],
|
|
18
19
|
"scripts": {
|
|
19
|
-
"test": "node --test test/*.test.js"
|
|
20
|
+
"test": "node --test test/*.test.js",
|
|
21
|
+
"api:check": "node ../../scripts/check_api_contract.js",
|
|
22
|
+
"pack": "npm pack"
|
|
20
23
|
},
|
|
21
24
|
"engines": {
|
|
22
25
|
"node": ">=18"
|
package/release.json
CHANGED
|
@@ -13,8 +13,8 @@
|
|
|
13
13
|
"entrypoint": "src/cli.js",
|
|
14
14
|
"name": "ml-platform",
|
|
15
15
|
"path": "runtime/business-client",
|
|
16
|
-
"sha256": "sha256:
|
|
17
|
-
"version": "0.
|
|
16
|
+
"sha256": "sha256:b509f9ccdedc4aa5d4d01c77609e4a16020ac6f74c62efc3af861877d897fe94",
|
|
17
|
+
"version": "0.7.0"
|
|
18
18
|
},
|
|
19
19
|
"policy_sha256": "sha256:02fbde0134696b78c0c365d6e0b76c9814227208da5b2b3d4e5dbd82dc4a3606",
|
|
20
20
|
"public_schema_versions": [
|
|
@@ -25,7 +25,7 @@
|
|
|
25
25
|
"ml_data_platform.feature_set/v1",
|
|
26
26
|
"ml_data_platform.dataset_manifest/v1"
|
|
27
27
|
],
|
|
28
|
-
"release_version": "0.1.
|
|
28
|
+
"release_version": "0.1.5",
|
|
29
29
|
"runtime_requirements": {
|
|
30
30
|
"node": ">=18",
|
|
31
31
|
"os": [
|
|
@@ -37,10 +37,10 @@
|
|
|
37
37
|
"skills": {
|
|
38
38
|
"feature-management": {
|
|
39
39
|
"path": "skills/feature-management",
|
|
40
|
-
"requires_cli": ">=0.
|
|
41
|
-
"revision": "0.1.
|
|
42
|
-
"sha256": "sha256:
|
|
40
|
+
"requires_cli": ">=0.7.0 <0.8.0",
|
|
41
|
+
"revision": "0.1.5",
|
|
42
|
+
"sha256": "sha256:ea7ef622f10300a3b5ee200adcf10070d18fadfb899d1bf2cc031ef42ffc5c03"
|
|
43
43
|
}
|
|
44
44
|
},
|
|
45
|
-
"source_commit": "
|
|
45
|
+
"source_commit": "d18a5b535a2a6c41fe4732e8e73739f2728c1e58"
|
|
46
46
|
}
|
|
@@ -25,3 +25,20 @@ ml-platform build-registered-dataset <dataset_id> <dataset_version> \
|
|
|
25
25
|
|
|
26
26
|
`resolve-manifest` 和 `build-dataset <manifest.json>` 仍用于尚未注册的本地契约。Catalog
|
|
27
27
|
normalizer 保留并校验非默认 `project`;`apply --project` 与 Catalog 内项目不一致时拒绝执行。
|
|
28
|
+
|
|
29
|
+
Registry 资产详情和版本发现使用与 `dev` OpenAPI 一致的命令:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
ml-platform get-parameter <name> <version> --project <project>
|
|
33
|
+
ml-platform get-feature <name> <version> --project <project>
|
|
34
|
+
ml-platform list-feature-versions <name> --project <project> --all
|
|
35
|
+
ml-platform list-jobs --project <project> --status succeeded --all
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
仓库中的 `api-spec/latest.json` 从 `origin/dev` 生成。同步或验证契约时运行:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
uv run python scripts/generate_openapi_schema.py --ref origin/dev
|
|
42
|
+
uv run python scripts/generate_openapi_schema.py --ref origin/dev --check
|
|
43
|
+
npm run api:check --prefix business-client-js
|
|
44
|
+
```
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@zhuoyuezs/ml-platform-business-client",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"lockfileVersion": 3,
|
|
5
5
|
"requires": true,
|
|
6
6
|
"packages": {
|
|
7
7
|
"": {
|
|
8
8
|
"name": "@zhuoyuezs/ml-platform-business-client",
|
|
9
|
-
"version": "0.
|
|
9
|
+
"version": "0.7.0",
|
|
10
10
|
"license": "UNLICENSED",
|
|
11
11
|
"bin": {
|
|
12
12
|
"ml-platform": "src/cli.js"
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@zhuoyuezs/ml-platform-business-client",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"private": true,
|
|
5
5
|
"description": "JavaScript parity implementation of the ML Platform business client",
|
|
6
6
|
"bin": {
|
|
@@ -11,7 +11,9 @@
|
|
|
11
11
|
"README.md"
|
|
12
12
|
],
|
|
13
13
|
"scripts": {
|
|
14
|
-
"test": "node --test test/*.test.js"
|
|
14
|
+
"test": "node --test test/*.test.js",
|
|
15
|
+
"api:check": "node ../scripts/check_api_contract.js",
|
|
16
|
+
"pack": "npm pack"
|
|
15
17
|
},
|
|
16
18
|
"engines": {
|
|
17
19
|
"node": ">=18"
|
|
@@ -14,6 +14,8 @@ const BUSINESS_COMMANDS = new Set([
|
|
|
14
14
|
"configure", "show-config", "health", "list-parameters", "list-operators", "list-features",
|
|
15
15
|
"list-feature-sets", "list-datasets", "get-dataset", "list-jobs", "resolve-manifest",
|
|
16
16
|
"resolve-dataset", "build-dataset", "build-registered-dataset",
|
|
17
|
+
"get-parameter", "get-feature", "get-feature-set", "get-operator",
|
|
18
|
+
"list-parameter-versions", "list-feature-versions", "list-feature-set-versions", "list-operator-versions",
|
|
17
19
|
"add-parameter", "add-feature", "add-feature-set", "add-operator", "publish-operator",
|
|
18
20
|
"add-dataset", "update-dataset", "delete-dataset", "list-dataset-artifacts",
|
|
19
21
|
"register-operator", "show-operator-specs", "delete-operator", "run-operator",
|
|
@@ -29,11 +31,19 @@ const COMMAND_USAGE = {
|
|
|
29
31
|
"list-projects": "list-projects",
|
|
30
32
|
"get-project": "get-project NAME",
|
|
31
33
|
"delete-project": "delete-project NAME",
|
|
32
|
-
"list-parameters": "list-parameters [--project PROJECT] [-q QUERY] [--limit N] [--offset N]",
|
|
33
|
-
"list-operators": "list-operators [--project PROJECT] [-q QUERY] [--limit N] [--offset N]",
|
|
34
|
-
"list-features": "list-features [--project PROJECT] [-q QUERY] [--limit N] [--offset N]",
|
|
35
|
-
"list-feature-sets": "list-feature-sets [--project PROJECT] [-q QUERY] [--limit N] [--offset N]",
|
|
36
|
-
"list-datasets": "list-datasets [--project PROJECT] [-q QUERY] [--limit N] [--offset N] [--include-deleted]",
|
|
34
|
+
"list-parameters": "list-parameters [--project PROJECT] [--name NAME] [--version VERSION] [-q QUERY] [--limit N] [--offset N] [--all]",
|
|
35
|
+
"list-operators": "list-operators [--project PROJECT] [--name NAME] [--version VERSION] [-q QUERY] [--limit N] [--offset N] [--all]",
|
|
36
|
+
"list-features": "list-features [--project PROJECT] [--name NAME] [--version VERSION] [-q QUERY] [--limit N] [--offset N] [--all]",
|
|
37
|
+
"list-feature-sets": "list-feature-sets [--project PROJECT] [--name NAME] [--version VERSION] [-q QUERY] [--limit N] [--offset N] [--all]",
|
|
38
|
+
"list-datasets": "list-datasets [--project PROJECT] [--dataset-id ID] [--dataset-version VERSION] [-q QUERY] [--limit N] [--offset N] [--include-deleted] [--all]",
|
|
39
|
+
"get-parameter": "get-parameter NAME VERSION [--project PROJECT]",
|
|
40
|
+
"get-feature": "get-feature NAME VERSION [--project PROJECT]",
|
|
41
|
+
"get-feature-set": "get-feature-set NAME VERSION [--project PROJECT]",
|
|
42
|
+
"get-operator": "get-operator NAME VERSION [--project PROJECT]",
|
|
43
|
+
"list-parameter-versions": "list-parameter-versions NAME [--project PROJECT] [--limit N] [--offset N] [--all]",
|
|
44
|
+
"list-feature-versions": "list-feature-versions NAME [--project PROJECT] [--limit N] [--offset N] [--all]",
|
|
45
|
+
"list-feature-set-versions": "list-feature-set-versions NAME [--project PROJECT] [--limit N] [--offset N] [--all]",
|
|
46
|
+
"list-operator-versions": "list-operator-versions NAME [--project PROJECT] [--limit N] [--offset N] [--all]",
|
|
37
47
|
"add-parameter": "add-parameter SPEC_JSON",
|
|
38
48
|
"add-feature": "add-feature SPEC_JSON",
|
|
39
49
|
"add-feature-set": "add-feature-set SPEC_JSON",
|
|
@@ -44,7 +54,7 @@ const COMMAND_USAGE = {
|
|
|
44
54
|
"delete-dataset": "delete-dataset DATASET_ID DATASET_VERSION [--project PROJECT]",
|
|
45
55
|
"list-dataset-artifacts": "list-dataset-artifacts [--project PROJECT] [--dataset-id ID] [--dataset-version VERSION] [--include-deleted] [--limit N] [--offset N]",
|
|
46
56
|
"get-dataset": "get-dataset DATASET_ID DATASET_VERSION [--project PROJECT] [--include-deleted]",
|
|
47
|
-
"list-jobs": "list-jobs [-q QUERY] [--limit N] [--offset N]",
|
|
57
|
+
"list-jobs": "list-jobs [filters] [-q QUERY] [--limit N] [--offset N] [--all]",
|
|
48
58
|
"resolve-manifest": "resolve-manifest MANIFEST [--out PATH]",
|
|
49
59
|
"resolve-dataset": "resolve-dataset DATASET_ID DATASET_VERSION [--project PROJECT] [--out PATH]",
|
|
50
60
|
"build-dataset": "build-dataset MANIFEST [--engine chronon] [--source-mode direct] [--upload-chronon-metadata] [--partition-duration DURATION] [--max-parallelism N] [--no-resume] [--wait] [--poll-interval SECONDS] [--wait-timeout-seconds SECONDS]",
|
|
@@ -58,7 +68,7 @@ const COMMAND_USAGE = {
|
|
|
58
68
|
"delete-operator": "delete-operator NAME VERSION [--project PROJECT]",
|
|
59
69
|
"run-operator": "run-operator NAME VERSION [--project PROJECT] [--config JSON]",
|
|
60
70
|
"get-dataset-artifact": "get-dataset-artifact DATASET_ID MANIFEST_HASH [--project PROJECT]",
|
|
61
|
-
"download-dataset-artifact": "download-dataset-artifact DATASET_ID MANIFEST_HASH [--project PROJECT] --out-dir PATH [--force]",
|
|
71
|
+
"download-dataset-artifact": "download-dataset-artifact DATASET_ID MANIFEST_HASH [--project PROJECT] --out-dir PATH [--file PATH] [--force]",
|
|
62
72
|
"fetch-inference-data": "fetch-inference-data MANIFEST --cutoff-time TIMESTAMP [--max-workers N] [--max-source-lag-hours HOURS] [--no-validate-freshness] [--allow-missing]",
|
|
63
73
|
apply: "apply CATALOG_DIRECTORY [--dry-run] [--project PROJECT]",
|
|
64
74
|
};
|
|
@@ -245,17 +255,39 @@ async function runBusinessCli(argv) {
|
|
|
245
255
|
} else if (options.command === "get-dataset") {
|
|
246
256
|
const datasetId = positional(rest, "dataset id"); const datasetVersion = positional(rest, "dataset version"); const project = take(rest, "--project", "default");
|
|
247
257
|
result = await registeredDataset(client(options), project, datasetId, datasetVersion, boolean(rest, "--include-deleted"));
|
|
248
|
-
}
|
|
249
|
-
|
|
258
|
+
} else if (["get-parameter", "get-feature", "get-feature-set", "get-operator"].includes(options.command)) {
|
|
259
|
+
const resource = options.command.slice(4);
|
|
260
|
+
const endpoint = resource === "feature-set" ? "feature-sets" : `${resource}s`;
|
|
261
|
+
const name = positional(rest, `${resource} name`);
|
|
262
|
+
const version = positional(rest, `${resource} version`);
|
|
263
|
+
const project = take(rest, "--project", "default");
|
|
264
|
+
result = await client(options).get(`/${endpoint}/${encodeURIComponent(project)}/${encodeURIComponent(name)}/${encodeURIComponent(version)}`);
|
|
265
|
+
} else if (["list-parameter-versions", "list-feature-versions", "list-feature-set-versions", "list-operator-versions"].includes(options.command)) {
|
|
266
|
+
const resource = options.command.replace(/^list-/, "").replace(/-versions$/, "");
|
|
267
|
+
const endpoint = resource === "feature-set" ? "feature-sets" : `${resource}s`;
|
|
268
|
+
const name = positional(rest, `${resource} name`);
|
|
269
|
+
const project = take(rest, "--project", "default");
|
|
270
|
+
const allPages = boolean(rest, "--all");
|
|
271
|
+
const rawLimit = take(rest, "--limit"); const rawOffset = take(rest, "--offset");
|
|
272
|
+
if (allPages && (rawLimit !== undefined || rawOffset !== undefined)) throw new Error("--all cannot be combined with --limit or --offset");
|
|
273
|
+
if (allPages) result = await fetchAll(client(options), `/${endpoint}/${encodeURIComponent(project)}/${encodeURIComponent(name)}`);
|
|
274
|
+
else { const limit = Number(rawLimit ?? 50); const offset = Number(rawOffset ?? 0); if (!Number.isInteger(limit) || limit < 1 || limit > 500) throw new Error("limit must be between 1 and 500"); if (!Number.isInteger(offset) || offset < 0) throw new Error("offset must be >= 0"); result = await client(options).get(`/${endpoint}/${encodeURIComponent(project)}/${encodeURIComponent(name)}`, { limit, offset }); }
|
|
275
|
+
} else if (options.command.startsWith("list-") && !["list-jobs", "list-dataset-artifacts"].includes(options.command)) {
|
|
250
276
|
const endpoints = { "list-parameters": "/parameters", "list-operators": "/operators", "list-features": "/features", "list-feature-sets": "/feature-sets", "list-datasets": "/datasets" };
|
|
251
277
|
const project = take(rest, "--project");
|
|
278
|
+
const name = take(rest, "--name"); const version = take(rest, "--version");
|
|
252
279
|
const query = take(rest, "--q", take(rest, "-q")); const includeDeleted = options.command === "list-datasets" && boolean(rest, "--include-deleted"); const rawLimit = take(rest, "--limit"); const rawOffset = take(rest, "--offset");
|
|
253
|
-
|
|
254
|
-
|
|
280
|
+
const allPages = boolean(rest, "--all");
|
|
281
|
+
if (allPages && (rawLimit !== undefined || rawOffset !== undefined)) throw new Error("--all cannot be combined with --limit or --offset");
|
|
282
|
+
if (allPages) result = await fetchAll(client(options), endpoints[options.command], { q: query, project, name, version, include_deleted: includeDeleted });
|
|
283
|
+
else { const limit = Number(rawLimit ?? 50); const offset = Number(rawOffset ?? 0); if (!Number.isInteger(limit) || limit < 1 || limit > 500) throw new Error("limit must be between 1 and 500"); if (!Number.isInteger(offset) || offset < 0) throw new Error("offset must be >= 0"); result = await client(options).get(endpoints[options.command], { q: query, limit, offset, project, name, version, include_deleted: includeDeleted }); }
|
|
255
284
|
} else if (options.command === "list-jobs") {
|
|
256
285
|
const query = take(rest, "--q", take(rest, "-q")); const rawLimit = take(rest, "--limit"); const rawOffset = take(rest, "--offset");
|
|
257
|
-
|
|
258
|
-
|
|
286
|
+
const filters = {}; for (const field of ["project", "dataset-id", "dataset-version", "manifest-hash", "status", "parent-job-id", "created-from", "created-to", "updated-from", "updated-to"]) filters[field.replaceAll("-", "_")] = take(rest, `--${field}`);
|
|
287
|
+
const allPages = boolean(rest, "--all");
|
|
288
|
+
if (allPages && (rawLimit !== undefined || rawOffset !== undefined)) throw new Error("--all cannot be combined with --limit or --offset");
|
|
289
|
+
if (allPages) result = await fetchAll(client(options), "/jobs", { q: query, ...filters });
|
|
290
|
+
else { const limit = Number(rawLimit ?? 50); const offset = Number(rawOffset ?? 0); if (!Number.isInteger(limit) || limit < 1 || limit > 500) throw new Error("limit must be between 1 and 500"); if (!Number.isInteger(offset) || offset < 0) throw new Error("offset must be >= 0"); result = await client(options).get("/jobs", { q: query, limit, offset, ...filters }); }
|
|
259
291
|
}
|
|
260
292
|
else if (options.command === "resolve-manifest") {
|
|
261
293
|
const manifest = positional(rest, "manifest");
|
|
@@ -314,7 +346,11 @@ async function runBusinessCli(argv) {
|
|
|
314
346
|
} else if (options.command === "download-dataset-artifact") {
|
|
315
347
|
const dataset = positional(rest, "dataset id"); const hash = positional(rest, "manifest hash"); const project = take(rest, "--project", "default"); const outDir = take(rest, "--out-dir");
|
|
316
348
|
if (!outDir) throw new Error("--out-dir is required");
|
|
317
|
-
|
|
349
|
+
const files = [];
|
|
350
|
+
while (rest.includes("--file")) files.push(take(rest, "--file"));
|
|
351
|
+
result = files.length
|
|
352
|
+
? await client(options).downloadDatasetArtifactFiles(project, dataset, hash, files, resolvedPath(outDir), boolean(rest, "--force"))
|
|
353
|
+
: await client(options).downloadDatasetArtifact(project, dataset, hash, resolvedPath(outDir), boolean(rest, "--force"));
|
|
318
354
|
} else if (options.command === "fetch-inference-data") {
|
|
319
355
|
const manifest = positional(rest, "manifest"); const cutoffTime = take(rest, "--cutoff-time"); if (!cutoffTime) throw new Error("--cutoff-time is required");
|
|
320
356
|
const realtime = {}; const maxWorkers = take(rest, "--max-workers"); const maxLag = take(rest, "--max-source-lag-hours"); if (maxWorkers !== undefined) { realtime.max_workers = integer(maxWorkers, "--max-workers"); if (realtime.max_workers < 1) throw new Error("--max-workers must be at least 1"); } if (maxLag !== undefined) { realtime.max_source_lag_hours = finiteNumber(maxLag, "--max-source-lag-hours"); if (realtime.max_source_lag_hours < 0) throw new Error("--max-source-lag-hours must be non-negative"); } if (boolean(rest, "--no-validate-freshness")) realtime.validate_freshness = false; if (boolean(rest, "--allow-missing")) realtime.allow_missing = true;
|
|
@@ -95,6 +95,52 @@ class PlatformApiClient {
|
|
|
95
95
|
fs.rmSync(temporaryDir, { recursive: true, force: true });
|
|
96
96
|
}
|
|
97
97
|
}
|
|
98
|
+
|
|
99
|
+
async downloadDatasetArtifactFiles(project, datasetId, manifestHash, relativePaths, outputDir, force = false) {
|
|
100
|
+
const files = [...new Set((relativePaths || []).map((value) => String(value)))];
|
|
101
|
+
if (!files.length) throw new Error("at least one artifact file is required");
|
|
102
|
+
for (const relative of files) {
|
|
103
|
+
const normalized = relative.replace(/\\/g, "/");
|
|
104
|
+
const parts = normalized.split("/");
|
|
105
|
+
if (!normalized || normalized.startsWith("/") || normalized.endsWith("/") || parts.includes("..")) {
|
|
106
|
+
throw new Error(`artifact path escapes output directory: ${JSON.stringify(relative)}`);
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
const targetRoot = path.resolve(expandHome(outputDir));
|
|
110
|
+
if (fs.existsSync(targetRoot) && fs.readdirSync(targetRoot).length) {
|
|
111
|
+
if (!force) throw new Error(`download output directory is not empty: ${targetRoot}`);
|
|
112
|
+
fs.rmSync(targetRoot, { recursive: true, force: true });
|
|
113
|
+
}
|
|
114
|
+
fs.mkdirSync(targetRoot, { recursive: true });
|
|
115
|
+
try {
|
|
116
|
+
for (const relative of files) {
|
|
117
|
+
const normalized = relative.replace(/\\/g, "/");
|
|
118
|
+
const parts = normalized.split("/");
|
|
119
|
+
const destination = path.resolve(targetRoot, ...parts);
|
|
120
|
+
if (!normalized || normalized.startsWith("/") || normalized.endsWith("/")
|
|
121
|
+
|| parts.includes("..") || destination !== targetRoot && !destination.startsWith(`${targetRoot}${path.sep}`)) {
|
|
122
|
+
throw new Error(`artifact path escapes output directory: ${JSON.stringify(relative)}`);
|
|
123
|
+
}
|
|
124
|
+
fs.mkdirSync(path.dirname(destination), { recursive: true });
|
|
125
|
+
const endpoint = `/dataset-artifacts/${encodeURIComponent(project)}/${encodeURIComponent(datasetId)}/${encodeURIComponent(manifestHash)}/files/${parts.map(encodeURIComponent).join("/")}`;
|
|
126
|
+
const controller = new AbortController();
|
|
127
|
+
const timer = setTimeout(() => controller.abort(), this.timeoutMs);
|
|
128
|
+
try {
|
|
129
|
+
const response = await fetch(this.url(endpoint), { headers: { Accept: "application/octet-stream" }, signal: controller.signal });
|
|
130
|
+
if (!response.ok) {
|
|
131
|
+
let detail = response.statusText;
|
|
132
|
+
try { detail = JSON.parse(await response.text())?.detail ?? detail; } catch (_) { /* status text */ }
|
|
133
|
+
throw new Error(`artifact file download failed: HTTP ${response.status}: ${detail}`);
|
|
134
|
+
}
|
|
135
|
+
fs.writeFileSync(destination, Buffer.from(await response.arrayBuffer()), { flag: "wx" });
|
|
136
|
+
} finally { clearTimeout(timer); }
|
|
137
|
+
}
|
|
138
|
+
} catch (error) {
|
|
139
|
+
fs.rmSync(targetRoot, { recursive: true, force: true });
|
|
140
|
+
throw error;
|
|
141
|
+
}
|
|
142
|
+
return { project, dataset_id: datasetId, manifest_hash: manifestHash, output_dir: targetRoot, file_count: files.length, files: files.sort() };
|
|
143
|
+
}
|
|
98
144
|
}
|
|
99
145
|
|
|
100
146
|
function contentLength(response) {
|
|
@@ -14,6 +14,34 @@ Operator ------> Feature -> FeatureSet -> DatasetManifest -> DatasetArtifact
|
|
|
14
14
|
|
|
15
15
|
Treat `Recipe` and public `Feature.compute` as removed. Treat `computation_hash` as internal execution metadata, never as a business-managed asset.
|
|
16
16
|
|
|
17
|
+
## Platform Role And Evidence Boundaries
|
|
18
|
+
|
|
19
|
+
The ML Platform is a deterministic data-contract and data-delivery platform. It
|
|
20
|
+
registers versioned source Parameters, runs approved deterministic Operators,
|
|
21
|
+
orders single-column Features into FeatureSets, resolves DatasetManifests, and
|
|
22
|
+
materializes batch DatasetArtifacts or serves one causal realtime inference row.
|
|
23
|
+
It records versions, lineage, quality, missingness, freshness, and execution
|
|
24
|
+
metadata so an algorithm project can consume data without owning source routing
|
|
25
|
+
or cache details.
|
|
26
|
+
|
|
27
|
+
It does not train, evaluate, or serve models; own complete business feature
|
|
28
|
+
engineering; provision arbitrary source tables; or administer Kubernetes and
|
|
29
|
+
platform services. Read
|
|
30
|
+
[references/platform-capability-guide.md](references/platform-capability-guide.md)
|
|
31
|
+
when the user needs the detailed platform boundary or evidence-level explanation.
|
|
32
|
+
|
|
33
|
+
Keep these evidence rules active in every workflow:
|
|
34
|
+
|
|
35
|
+
- Platform defaults and capabilities are not business evidence.
|
|
36
|
+
- Dry-run, publish, resolve, build, metadata inspection, downloaded-file
|
|
37
|
+
inspection, and realtime fetch are separate evidence checkpoints; success at
|
|
38
|
+
one checkpoint does not imply success at the next.
|
|
39
|
+
- `validation.ok=true` does not by itself prove source semantics, causal formula
|
|
40
|
+
correctness, Parquet contents, or numeric parity.
|
|
41
|
+
- Dry-run, publish, and resolve may not detect a missing source relation. Do not
|
|
42
|
+
report source-backed success until build/fetch returns the required evidence,
|
|
43
|
+
and never rename a Parameter or remove fields merely to bypass a failed read.
|
|
44
|
+
|
|
17
45
|
## Load The Right Context
|
|
18
46
|
|
|
19
47
|
1. Resolve the Skill root as the directory containing this `SKILL.md`.
|
|
@@ -3,6 +3,35 @@
|
|
|
3
3
|
Use this guide to explain what the platform can execute. It is not business
|
|
4
4
|
evidence. A platform default never answers an unresolved business question.
|
|
5
5
|
|
|
6
|
+
## Platform In One Paragraph
|
|
7
|
+
|
|
8
|
+
The platform is a versioned data-contract, registry, execution, and delivery
|
|
9
|
+
layer for algorithm projects. It turns confirmed source Parameters and approved
|
|
10
|
+
deterministic Operators into ordered Features and reproducible DatasetManifests,
|
|
11
|
+
then either materializes a batch DatasetArtifact or serves one causal realtime
|
|
12
|
+
read. It owns source adapters, quality/missingness/freshness evidence, lineage,
|
|
13
|
+
replay metadata, and Job execution; it does not own model training, model
|
|
14
|
+
evaluation, model serving, source-table creation, or Kubernetes administration.
|
|
15
|
+
|
|
16
|
+
The following distinctions are part of the platform contract:
|
|
17
|
+
|
|
18
|
+
- A `Parameter` is a public source-data contract, not an arbitrary raw table or
|
|
19
|
+
a model feature.
|
|
20
|
+
- An `Operator` is versioned executable code; a `Feature` is exactly one output
|
|
21
|
+
column; a `FeatureSet` is the consumer-visible column order.
|
|
22
|
+
- A `DatasetManifest` is a versioned request for time range, read policy,
|
|
23
|
+
rowsets, missing/endpoint rules, prediction, and one FeatureSet.
|
|
24
|
+
- A `DatasetArtifact` is an output of an authorized build, not evidence that the
|
|
25
|
+
source contract or business formula was semantically approved.
|
|
26
|
+
- Realtime inference is a causal cutoff read. It must not run a batch build,
|
|
27
|
+
write Parquet, or publish a DatasetArtifact.
|
|
28
|
+
|
|
29
|
+
Dry-run and registry resolution are structural evidence. They may not detect a
|
|
30
|
+
missing source relation; source-backed success must be established by the build
|
|
31
|
+
or realtime fetch response. Artifact metadata is separate from downloaded-file
|
|
32
|
+
evidence: schema, column order, file hashes, and numeric parity require the
|
|
33
|
+
complete files.
|
|
34
|
+
|
|
6
35
|
## Public Assets
|
|
7
36
|
|
|
8
37
|
| Asset | Stores | Use when | Do not use for |
|