@zhuoyuezs/ml-platform 0.1.1 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DEVELOPMENT.md +96 -9
- package/README.md +89 -38
- package/checksums.json +37 -32
- package/package.json +5 -1
- package/release-policy.json +10 -0
- package/release.json +13 -9
- package/runtime/business-client/README.md +13 -0
- package/runtime/business-client/package-lock.json +2 -2
- package/runtime/business-client/package.json +1 -1
- package/runtime/business-client/src/catalog.js +40 -17
- package/runtime/business-client/src/cli.js +150 -26
- package/runtime/business-client/src/config.js +6 -2
- package/runtime/business-client/src/http.js +88 -17
- package/scripts/lib.js +98 -4
- package/scripts/main.js +96 -11
- package/skills/feature-management/SKILL.md +187 -18
- package/skills/feature-management/assets/catalog-template/datasets/example_temperature_training.v1.json +16 -0
- package/skills/feature-management/assets/catalog-template/feature_sets/example_temperature_core.v1.json +1 -0
- package/skills/feature-management/assets/catalog-template/features/example_temperature_mean_5m.v1.json +9 -2
- package/skills/feature-management/assets/catalog-template/operator_package/tests/test_operator.py +51 -24
- package/skills/feature-management/assets/catalog-template/operators/example_temperature_features.v1.json +16 -1
- package/skills/feature-management/assets/catalog-template/parameters/example_temperature.v1.json +1 -0
- package/skills/feature-management/references/commands.md +66 -4
- package/skills/feature-management/references/contracts.md +43 -6
- package/skills/feature-management/references/operator-authoring.md +15 -7
- package/skills/feature-management/references/platform-capability-guide.md +44 -0
|
@@ -19,12 +19,16 @@ Treat `Recipe` and public `Feature.compute` as removed. Treat `computation_hash`
|
|
|
19
19
|
1. Resolve the Skill root as the directory containing this `SKILL.md`.
|
|
20
20
|
2. Read [references/contracts.md](references/contracts.md) before creating or changing JSON assets.
|
|
21
21
|
3. Read [references/operator-authoring.md](references/operator-authoring.md) whenever creating or changing Operator code or a wheel.
|
|
22
|
-
4. Read [references/commands.md](references/commands.md) before
|
|
22
|
+
4. Read [references/commands.md](references/commands.md) immediately before a platform CLI action: discovery, validation, publication, build, realtime fetch, or artifact download. A local-only draft that is forbidden to call the CLI does not need this reference.
|
|
23
23
|
5. Use [assets/catalog-template](assets/catalog-template) as a copyable starting point for a new end-to-end catalog. Rename every `example_*` identifier and update every referenced path before validation.
|
|
24
|
+
6. Read [references/platform-capability-guide.md](references/platform-capability-guide.md) only when selecting an asset type or explaining a platform capability. Do not load it merely to write a confirmed local draft; it is never evidence for unresolved business semantics.
|
|
24
25
|
|
|
25
26
|
## Initialize The Client
|
|
26
27
|
|
|
27
|
-
Use the `ml-platform` executable on `PATH` for every platform command. The formal
|
|
28
|
+
Use the `ml-platform` executable on `PATH` for every platform command. The formal npm release installs
|
|
29
|
+
the CLI and this pure Skill together. The Skill does not contain a second CLI runtime, launcher, or
|
|
30
|
+
installer. During repository development, provide the current JavaScript CLI through an isolated test
|
|
31
|
+
`PATH`.
|
|
28
32
|
|
|
29
33
|
Before any platform operation, verify the executable and its version:
|
|
30
34
|
|
|
@@ -36,17 +40,32 @@ ml-platform version
|
|
|
36
40
|
If the executable is missing or cannot start, stop and report that the ML Platform release installation is incomplete. Do not download a client, search for a repository checkout, or fall back to another executable.
|
|
37
41
|
|
|
38
42
|
Before any server-profile discovery, dry-run, publication, build, or artifact
|
|
39
|
-
operation,
|
|
43
|
+
operation, inspect the effective API configuration:
|
|
40
44
|
|
|
41
45
|
```bash
|
|
42
|
-
ml-platform
|
|
46
|
+
ml-platform show-config
|
|
43
47
|
ml-platform --profile server health
|
|
44
48
|
```
|
|
45
49
|
|
|
46
|
-
Use `
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
+
Use the global `--profile server` form shown above, before the business
|
|
51
|
+
command. Do not append `--profile` after a business command. Before discovery,
|
|
52
|
+
check the installed release with `ml-platform --help` and the relevant
|
|
53
|
+
subcommand's `--help`; CLI releases may differ in supported flags. In the
|
|
54
|
+
deployed 0.6.x CLI, registry `list-*` commands emit JSON by default and do not
|
|
55
|
+
accept a `--json` flag.
|
|
56
|
+
|
|
57
|
+
If the installed release rejects the global `--profile server` form shown above,
|
|
58
|
+
record the CLI syntax error and inspect `show-config`/subcommand help before
|
|
59
|
+
retrying the same read-only command without that flag when supported. Do not
|
|
60
|
+
switch clients or classify a syntax error as an API outage; keep each command
|
|
61
|
+
separate so CLI compatibility remains distinguishable from network health.
|
|
62
|
+
|
|
63
|
+
The effective API target is resolved in this order: explicit `--api-url`,
|
|
64
|
+
`ML_PLATFORM_API_URL`, then the saved value written by `ml-platform configure`.
|
|
65
|
+
If no target is configured, ask the user for an approved API URL before making
|
|
66
|
+
server requests. Do not hardcode an environment-specific address in a Skill,
|
|
67
|
+
catalog, or prompt. Keep database, MinIO, and source credentials out of commands;
|
|
68
|
+
use the server's managed Secrets.
|
|
50
69
|
|
|
51
70
|
Install and upgrade the CLI and Skill only through the explicit commands provided by `@zhuoyuezs/ml-platform`. Restart the Agent session after a successful Skill upgrade so the definition is reloaded. Never copy files over an installed Skill directory manually.
|
|
52
71
|
|
|
@@ -56,7 +75,7 @@ Collect only missing information. Do not invent a source table, source field, fo
|
|
|
56
75
|
|
|
57
76
|
Confirm:
|
|
58
77
|
|
|
59
|
-
- the target project (namespace) for the assets; use the built-in `default` only when the user has no dedicated project.
|
|
78
|
+
- the target project (namespace) for the assets; use the built-in `default` only when the user has no dedicated project. After the target is confirmed, write its explicit `project` field in every Parameter, Operator, Feature, FeatureSet and DatasetManifest draft; never rely on the schema default. Reads and filters use the `--project` command option;
|
|
60
79
|
- business meaning, stable asset names, owner, and intended consumers;
|
|
61
80
|
- Parameter source adapter and credential-free source mapping;
|
|
62
81
|
- feature formula, exact Parameter dependencies, windows, inclusion rules, rounding, null behavior, and output dtype;
|
|
@@ -70,12 +89,91 @@ Confirm:
|
|
|
70
89
|
|
|
71
90
|
Separate preparation from mutation. Creating files and running local tests or `apply --dry-run` does not authorize publishing a wheel, changing the Registry, or submitting a build Job.
|
|
72
91
|
|
|
92
|
+
A label such as "10-minute mean" confirms neither its input Parameter nor a
|
|
93
|
+
complete formula. Before drafting an Operator or Feature, ask for the exact
|
|
94
|
+
versioned input Parameter(s), or for an approved source contract from which
|
|
95
|
+
they will be created. Do not infer an input from the Feature name or a similar
|
|
96
|
+
existing asset.
|
|
97
|
+
|
|
98
|
+
### Semantic Asset Review Gate
|
|
99
|
+
|
|
100
|
+
Treat resource correctness as a separate gate from workflow correctness. Before
|
|
101
|
+
creating Catalog JSON, write a local semantic asset review with one row per
|
|
102
|
+
Parameter, Operator, Feature and Dataset field:
|
|
103
|
+
|
|
104
|
+
```text
|
|
105
|
+
asset_key | business_meaning | source/evidence | confirmed_by | unresolved | proposed_value
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
The review must distinguish three sources: confirmed business facts, facts
|
|
109
|
+
observed from the selected Project Registry/CLI, and agent proposals. A Skill
|
|
110
|
+
reference, schema default, similarly named old asset, or platform capability is
|
|
111
|
+
not business evidence. Do not convert a proposal into `proposed_value` without
|
|
112
|
+
explicit user confirmation or an exact match in the selected Project.
|
|
113
|
+
|
|
114
|
+
Stop with `need_business_confirmation` when any required source mapping, field,
|
|
115
|
+
unit, formula, time conversion, missing/quality policy, FeatureSet order,
|
|
116
|
+
dataset read policy, rowset/endpoint policy, owner, or approval boundary is
|
|
117
|
+
unresolved. Do not create placeholder Catalog files to make the workflow look
|
|
118
|
+
complete.
|
|
119
|
+
|
|
120
|
+
For a requested Feature, `exact Parameter dependencies` is an independent
|
|
121
|
+
required contract item. If the user names a transform or window but not its
|
|
122
|
+
input Parameter(s), include `input_parameter` in the unresolved questions and
|
|
123
|
+
stop. Do this before treating an Operator formula as complete: a time window,
|
|
124
|
+
cutoff rule, or Feature name never identifies the input.
|
|
125
|
+
|
|
126
|
+
After confirmation, show the proposed dependency closure and obtain approval of
|
|
127
|
+
the semantic asset plan before generating Catalog files. A successful local
|
|
128
|
+
schema check or remote dry-run is not semantic approval.
|
|
129
|
+
|
|
130
|
+
Before emitting or writing a draft, mechanically check that every asset carries
|
|
131
|
+
the confirmed `project` field, that every Feature input carries that same
|
|
132
|
+
`project`, and that every Operator `input_schema.parameters` reference is
|
|
133
|
+
qualified as `project/parameter:version`. The schema's `default` values are not
|
|
134
|
+
valid substitutes for an explicit non-default target. When a confirmed Feature
|
|
135
|
+
config defines window closure, cutoff inclusion, duplicate handling, or empty
|
|
136
|
+
window behavior, copy all of those fields into the Feature config rather than
|
|
137
|
+
reducing it to the window string.
|
|
138
|
+
|
|
73
139
|
## Discover Existing Assets
|
|
74
140
|
|
|
75
|
-
Check the selected profile and list existing Parameter, Operator, Feature, FeatureSet, and Dataset versions before choosing names. Scope discovery to the target project with `--project`, because the true registry key is `project/name:version` and the same `name:version` may exist independently under another project. Registry list commands are paginated; search by stable identifier or follow every page until `offset + len(items) >= total`. Reuse an exact immutable version only when its full content matches. Never reference an asset in another project; cross-project references are rejected.
|
|
141
|
+
Check the selected profile and list existing Parameter, Operator, Feature, FeatureSet, and Dataset versions before choosing names. Scope discovery to the target project with `--project`, because the true registry key is `project/name:version` and the same `name:version` may exist independently under another project. Run each `list-*` command as a separate command so one failed check cannot hide the status of the others. Do not add `--json` unless the installed subcommand help explicitly advertises it; supported releases emit machine-readable JSON by default. Registry list commands are paginated; search by stable identifier or follow every page until `offset + len(items) >= total`. Reuse an exact immutable version only when its full content matches. Never reference an asset in another project; cross-project references are rejected.
|
|
142
|
+
|
|
143
|
+
CLI readiness is a hard gate for every Registry conclusion. If `command -v`,
|
|
144
|
+
`version`, `show-config`, or server `health` fails, returns a nonzero status, or
|
|
145
|
+
does not identify the intended target, stop with `platform_not_ready`. Do not
|
|
146
|
+
list assets, select reuse versus authoring, dry-run, publish, build, fetch, or
|
|
147
|
+
report a Registry fact from prompt text, a remembered response, or a similar
|
|
148
|
+
local asset. Report the failed command and its configured target; resume only
|
|
149
|
+
after the user/environment restores the same target.
|
|
76
150
|
|
|
77
151
|
Create a new version when source semantics, formula code, config meaning, inputs, output dtype, time behavior, quality rules, or column order change. Never overwrite an immutable version or use suffixes such as `new`, `final`, or `test2`.
|
|
78
152
|
|
|
153
|
+
### Operate An Existing Registered Dataset
|
|
154
|
+
|
|
155
|
+
For diagnosis, resolve, build, Job lookup, and artifact lookup of an already
|
|
156
|
+
registered DatasetManifest, use the API through `ml-platform`. Do not search the
|
|
157
|
+
platform source checkout or a local Catalog directory for a matching JSON file.
|
|
158
|
+
|
|
159
|
+
1. Run `show-config` and `health` to record the effective API source.
|
|
160
|
+
2. Run `list-projects`, then `list-datasets --project <project>` to discover
|
|
161
|
+
candidates from the Registry.
|
|
162
|
+
3. Ask the user to confirm the exact `project`, `dataset_id`, and
|
|
163
|
+
`dataset_version` when more than one valid candidate exists. Never infer a
|
|
164
|
+
target from a similar local filename.
|
|
165
|
+
4. Run `get-dataset <dataset_id> <dataset_version> --project <project>` and
|
|
166
|
+
inspect the returned registered contract.
|
|
167
|
+
5. Run `resolve-dataset` for a read-only dependency and manifest-hash check.
|
|
168
|
+
6. Submit exactly one `build-registered-dataset ... --wait` only after the user
|
|
169
|
+
authorizes that exact registered version.
|
|
170
|
+
7. Use the returned Job ID with `get-job`, then use the returned manifest hash
|
|
171
|
+
with `get-dataset-artifact` after success.
|
|
172
|
+
|
|
173
|
+
Local `resolve-manifest` and `build-dataset <file>` remain authoring tools for a
|
|
174
|
+
new or changed Catalog contract before it is registered. They are not required
|
|
175
|
+
for operating an existing Registry version.
|
|
176
|
+
|
|
79
177
|
## Upgrade Dependency Chains
|
|
80
178
|
|
|
81
179
|
Treat a version change as a release of its affected reverse-dependency closure, not as an isolated asset bump.
|
|
@@ -111,8 +209,29 @@ Keep platform demos and V95 built-ins unchanged. Do not put unrelated business a
|
|
|
111
209
|
|
|
112
210
|
### Define Parameters
|
|
113
211
|
|
|
212
|
+
Use the platform-capability guide's asset table before choosing the resource
|
|
213
|
+
type. For each Parameter record the business signal, exact source evidence,
|
|
214
|
+
unit, time/availability semantics and missing policy in the semantic review.
|
|
215
|
+
Never infer a source table, column, measurement, tag, unit or timezone from a
|
|
216
|
+
Parameter name.
|
|
217
|
+
|
|
114
218
|
Create one stable, versioned source contract for each independently readable value. Keep credentials out of JSON. Use only supported direct adapters and explicit source identity. Put source data validity in `quality_rules`; put dataset-specific transformations in preprocess Operators.
|
|
115
219
|
|
|
220
|
+
When the approved source contract supplies SQL, copy its query text and named
|
|
221
|
+
parameters exactly into `source.sql`/`source.params`. Do not retype, beautify,
|
|
222
|
+
rename an identifier, change a join key, or "simplify" an approved query while
|
|
223
|
+
drafting. SQL whitespace may be formatted only when a reviewed formatter proves
|
|
224
|
+
the token stream is unchanged; every source identifier and `<= %(end)s` boundary
|
|
225
|
+
remains part of the Parameter contract.
|
|
226
|
+
|
|
227
|
+
For SQL Parameters, the runtime contract is a single statement: use the exact
|
|
228
|
+
`source.sql` field and a mapping under `source.params` for every named
|
|
229
|
+
placeholder other than the reserved `start` and `end` window parameters. Do not
|
|
230
|
+
use `source.parameters` (it is not a supported alias and may be silently
|
|
231
|
+
dropped by older CLI/server releases). Do not leave a trailing semicolon; the
|
|
232
|
+
deployed Worker rejects SQL containing multiple statements. Verify the resolved
|
|
233
|
+
Parameter still contains the named params before authorizing a build.
|
|
234
|
+
|
|
116
235
|
Use Parameter `rounding` when every consumer must receive the same fixed-point value. The platform applies `mode` (`half_up` or `half_even`) and `decimals` after source normalization and before quality checks, source caching, replay, and Operator execution. Prefer an unrounded source expression, do not repeat the same rounding in an Operator, and publish a new Parameter plus its affected reverse-dependency closure when the rule changes.
|
|
117
236
|
|
|
118
237
|
Do not represent a rolling mean, lag, ratio, trend, or model input as a Parameter. Those are Features.
|
|
@@ -137,7 +256,11 @@ Use a meaningful immutable `function_hash`. Change the Operator version whenever
|
|
|
137
256
|
|
|
138
257
|
### Define Features
|
|
139
258
|
|
|
140
|
-
Create one JSON file per single output column.
|
|
259
|
+
Create one JSON file per single output column. Set the confirmed `project` on the
|
|
260
|
+
Feature and on every nested Feature input; a FeatureInput without `project`
|
|
261
|
+
silently targets `default`. Bind exactly one Operator version and physical
|
|
262
|
+
`output_column`. List only the Parameter versions actually required by that
|
|
263
|
+
column's formula.
|
|
141
264
|
|
|
142
265
|
Features may share the same Operator and config. The planner will merge their exact inputs and call the Operator once per computation group. Do not duplicate every Operator input into every Feature merely to make schemas look uniform.
|
|
143
266
|
|
|
@@ -147,8 +270,41 @@ Reference immutable Feature versions in the exact consumer column order. Keep th
|
|
|
147
270
|
|
|
148
271
|
### Define The DatasetManifest
|
|
149
272
|
|
|
273
|
+
Do not use DatasetManifest as a catch-all for unresolved semantics. Confirm the
|
|
274
|
+
dataset purpose and consumer, then separately confirm the target grid, horizon,
|
|
275
|
+
read policy, explicit Parameter outputs, rowset or rowset splits, abnormal windows and
|
|
276
|
+
endpoint eligibility. Keep the semantic asset review beside the draft Catalog
|
|
277
|
+
so a reviewer can compare business decisions with generated fields.
|
|
278
|
+
|
|
150
279
|
Reference exactly one FeatureSet. Use `parameters` only for Parameter columns that must also appear explicitly in `parameter_dataset.parquet`; Feature dependencies are resolved automatically.
|
|
151
280
|
|
|
281
|
+
Map the current Manifest schema exactly rather than carrying descriptive fields
|
|
282
|
+
from Parameter, Feature, or FeatureSet into it. A target Parameter reference is
|
|
283
|
+
**nested**; never flatten its `version` or `project` onto `target`:
|
|
284
|
+
|
|
285
|
+
```json
|
|
286
|
+
"target": {
|
|
287
|
+
"parameter": {"parameter": "hot_metal_si", "version": "v1", "project": "<project>"},
|
|
288
|
+
"si_time_source": "dispatch_time",
|
|
289
|
+
"offset_minutes": 20,
|
|
290
|
+
"interpolation": {"method": "linear"}
|
|
291
|
+
}
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
`target.interpolation` is `{"method": "linear"}`, `{"method": "none"}` or
|
|
295
|
+
`{"method": "forward_fill"}` -- neither it nor `target.parameter` accepts a
|
|
296
|
+
bare string. The current DatasetManifest has no top-level `owner`,
|
|
297
|
+
`description`, `si_null_policy`, or `preserve_all_candidate_rows`. Map a
|
|
298
|
+
business statement only to an existing schema field (for example `mark_only`,
|
|
299
|
+
`endpoint_policy`, a Feature config, or a Parameter missing policy); otherwise
|
|
300
|
+
keep it in the semantic review and ask for a platform/schema change. Validate
|
|
301
|
+
the generated object against the current schema before running `apply --dry-run`.
|
|
302
|
+
|
|
303
|
+
When a confirmed missing-data policy applies to a Feature-only Parameter, put it
|
|
304
|
+
in `parameter_missing_policies`, using `parameter`, `version`, `project`, and
|
|
305
|
+
`policy`. Do not copy the `ParameterRequest` names `name` and `missing_policy`
|
|
306
|
+
into that list. Omit the section when no such policy is confirmed.
|
|
307
|
+
|
|
152
308
|
Declare one dataset-wide `prediction` contract when Features use a forecast cutoff. Operators consume `context.cutoff_times`; do not copy the same prediction horizon into every Feature config. Check the selected Operator's `input_schema.prediction` limits and satisfy an explicit-horizon requirement before publishing or building.
|
|
153
309
|
|
|
154
310
|
Use `snapshot` for reproducible training data, `as_of` for historical visibility replay, and `latest` for current inference-style reads. Use a fresh `dataset_version` when validating a new release or intentionally requesting a new immutable dataset contract.
|
|
@@ -173,11 +329,16 @@ freshness, tail-edge, gap-fill, and rowset contracts.
|
|
|
173
329
|
|
|
174
330
|
Run, in order:
|
|
175
331
|
|
|
176
|
-
1.
|
|
177
|
-
2.
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
332
|
+
1. Semantic asset review: no required unresolved field and every proposed value has a source or user confirmation.
|
|
333
|
+
2. Operator unit tests. Before building, turn every applicable item in
|
|
334
|
+
`operator-authoring.md` into a separately named test; do not let one broad
|
|
335
|
+
happy-path assertion substitute for cutoff-before/at/after, duplicate and
|
|
336
|
+
missing input, empty history, requested-output, event-time order, or
|
|
337
|
+
dtype/rounding coverage. The template is only a starting point.
|
|
338
|
+
3. Wheel build and wheel filename verification.
|
|
339
|
+
4. Catalog path, schema, dependency, immutability, and package validation with `apply --dry-run` against the same target profile intended for publication.
|
|
340
|
+
5. Compare the catalog's exact dependency closure with the intended old-to-new version mapping. Reject any affected downstream reference that still points to an old version unless its retention is explicit and justified.
|
|
341
|
+
6. A human-readable summary of planned new, unchanged, retained, and conflicting assets.
|
|
181
342
|
|
|
182
343
|
Stop on any error. Do not weaken schema validation, fabricate a missing dependency, change an existing version in place, or switch profiles to make validation pass.
|
|
183
344
|
|
|
@@ -185,7 +346,7 @@ Stop on any error. Do not weaken schema validation, fabricate a missing dependen
|
|
|
185
346
|
|
|
186
347
|
Publish only after the user explicitly approves Registry and wheel changes. Use catalog `apply` so publication follows Parameter -> Operator -> Feature -> FeatureSet -> Dataset order.
|
|
187
348
|
|
|
188
|
-
After publication, resolve every new DatasetManifest and compare its exact Parameter, Operator, Feature, and FeatureSet versions with the pre-publication mapping. Stop if an affected old key or any unexpected version remains. Resolve the manifest before building. Submit a build only when requested. For server builds, wait for the terminal Job state and download the exact artifact by `dataset_id + manifest_hash`.
|
|
349
|
+
After publication, resolve every new DatasetManifest and compare its exact Parameter, Operator, Feature, and FeatureSet versions with the pre-publication mapping. Stop if an affected old key or any unexpected version remains. Resolve the manifest before building. Submit a build only when requested. For the current partitioned materialization runtime, use `--max-parallelism 1` unless a separately verified platform capability says otherwise; do not retry or resubmit a failed Job until its error and already-completed partitions are understood. For server builds, wait for the terminal Job state and download the exact artifact by `dataset_id + manifest_hash`.
|
|
189
350
|
|
|
190
351
|
Do not delete versioned Registry assets, cancel Jobs, rebuild images, modify Kubernetes, or change service configuration as part of this workflow unless the user separately and explicitly requests that action. An empty Project may be soft-deleted only on an explicit request; rely on the server to reject deletion when resources still exist.
|
|
191
352
|
|
|
@@ -203,6 +364,14 @@ gap-filled inputs as degraded freshness, not as a normal read.
|
|
|
203
364
|
|
|
204
365
|
## Verify The Artifact
|
|
205
366
|
|
|
367
|
+
Treat Registry metadata inspection and downloaded-file inspection as separate
|
|
368
|
+
checkpoints. A successful `get-dataset-artifact` can establish the immutable
|
|
369
|
+
identity, object inventory, row count, validation summary, and lineage metadata;
|
|
370
|
+
it cannot establish Parquet schema, column order, or file-content hashes. If a
|
|
371
|
+
download times out, report whether no response/progress was observed or whether
|
|
372
|
+
some body bytes were received, preserve the metadata-only evidence, and leave
|
|
373
|
+
downloaded-file validation explicitly incomplete.
|
|
374
|
+
|
|
206
375
|
Require all of the following before reporting success:
|
|
207
376
|
|
|
208
377
|
- Job status is `succeeded`;
|
|
@@ -235,7 +404,7 @@ Require all of the following before reporting success:
|
|
|
235
404
|
windows tiling the fetch range, and any retries are accounted for;
|
|
236
405
|
- missing values and warnings are reported, not silently repaired.
|
|
237
406
|
|
|
238
|
-
The manifest hash returned by `resolve-manifest` is the artifact key: it stays
|
|
407
|
+
The manifest hash returned by `resolve-manifest` or `resolve-dataset` is the artifact key: it stays
|
|
239
408
|
the same after the build, so it is usable for cache reuse and polling before
|
|
240
409
|
submission. Distinguish the two lookup failures rather than treating both as "not
|
|
241
410
|
ready" — `422` with `invalid_artifact_key` means the key can never name an
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schema_version": "ml_data_platform.dataset_manifest/v1",
|
|
3
|
+
"project": "replace_with_project",
|
|
3
4
|
"dataset_id": "example_temperature_training",
|
|
4
5
|
"dataset_version": "v1",
|
|
5
6
|
"mode": "training",
|
|
@@ -9,7 +10,22 @@
|
|
|
9
10
|
"end": "2026-07-02T00:00:00+08:00",
|
|
10
11
|
"grid": "10min"
|
|
11
12
|
},
|
|
13
|
+
"prediction": {
|
|
14
|
+
"horizon": "0min"
|
|
15
|
+
},
|
|
16
|
+
"rowset": {
|
|
17
|
+
"strategy": "fixed_grid",
|
|
18
|
+
"grid": "10min"
|
|
19
|
+
},
|
|
12
20
|
"parameters": [],
|
|
21
|
+
"parameter_missing_policies": [
|
|
22
|
+
{
|
|
23
|
+
"parameter": "example_temperature",
|
|
24
|
+
"version": "v1",
|
|
25
|
+
"project": "replace_with_project",
|
|
26
|
+
"policy": "report_only"
|
|
27
|
+
}
|
|
28
|
+
],
|
|
13
29
|
"feature_set": {
|
|
14
30
|
"name": "example_temperature_core",
|
|
15
31
|
"version": "v1"
|
|
@@ -1,17 +1,24 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schema_version": "ml_data_platform.feature/v1",
|
|
3
|
+
"project": "replace_with_project",
|
|
3
4
|
"name": "example_temperature_mean_5m",
|
|
4
5
|
"version": "v1",
|
|
5
6
|
"inputs": [
|
|
6
7
|
{
|
|
7
8
|
"parameter": "example_temperature",
|
|
8
|
-
"version": "v1"
|
|
9
|
+
"version": "v1",
|
|
10
|
+
"project": "replace_with_project"
|
|
9
11
|
}
|
|
10
12
|
],
|
|
11
13
|
"operator": "example_temperature_features",
|
|
12
14
|
"operator_version": "v1",
|
|
13
15
|
"config": {
|
|
14
|
-
"window": "5min"
|
|
16
|
+
"window": "5min",
|
|
17
|
+
"window_closed": "right",
|
|
18
|
+
"cutoff_included": true,
|
|
19
|
+
"post_cutoff_allowed": false,
|
|
20
|
+
"duplicate_event_time": "last",
|
|
21
|
+
"empty_window": "null"
|
|
15
22
|
},
|
|
16
23
|
"output_column": "example_temperature_mean_5m",
|
|
17
24
|
"output_dtype": "float64",
|
package/skills/feature-management/assets/catalog-template/operator_package/tests/test_operator.py
CHANGED
|
@@ -9,20 +9,21 @@ from business_feature_operator_template import compute_features
|
|
|
9
9
|
|
|
10
10
|
|
|
11
11
|
class ComputeFeaturesTest(unittest.TestCase):
|
|
12
|
-
def context(
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
12
|
+
def context(
|
|
13
|
+
self,
|
|
14
|
+
timestamps: list[str] | None = None,
|
|
15
|
+
values: list[float | None] | None = None,
|
|
16
|
+
targets: list[str] | None = None,
|
|
17
|
+
requested: list[str] | None = None,
|
|
18
|
+
) -> SimpleNamespace:
|
|
19
|
+
timestamps = timestamps or [
|
|
20
|
+
"2026-07-01 09:54:00",
|
|
21
|
+
"2026-07-01 09:56:00",
|
|
22
|
+
"2026-07-01 10:00:00",
|
|
23
|
+
"2026-07-01 10:01:00",
|
|
24
|
+
]
|
|
25
|
+
values = values or [10.0, 20.0, 30.0, 40.0]
|
|
26
|
+
source = pd.DataFrame({"timestamp": pd.DatetimeIndex(timestamps), "value": values})
|
|
26
27
|
return SimpleNamespace(
|
|
27
28
|
requested_output_columns=requested or ["example_temperature_mean_5m"],
|
|
28
29
|
inputs=[
|
|
@@ -32,24 +33,50 @@ class ComputeFeaturesTest(unittest.TestCase):
|
|
|
32
33
|
)
|
|
33
34
|
],
|
|
34
35
|
metric_frames={"example_temperature:v1": source},
|
|
35
|
-
target_times=pd.DatetimeIndex(
|
|
36
|
-
["2026-07-01 10:00:00", "2026-07-01 10:02:00"]
|
|
37
|
-
),
|
|
36
|
+
target_times=pd.DatetimeIndex(targets or ["2026-07-01 10:00:00", "2026-07-01 10:02:00"]),
|
|
38
37
|
config={"window": "5min"},
|
|
39
38
|
)
|
|
40
39
|
|
|
41
|
-
def
|
|
40
|
+
def test_formula_uses_the_confirmed_mean(self) -> None:
|
|
42
41
|
result = compute_features(self.context())
|
|
43
|
-
|
|
44
|
-
self.assertEqual(
|
|
45
|
-
list(result.columns),
|
|
46
|
-
["event_time", "example_temperature_mean_5m"],
|
|
47
|
-
)
|
|
48
42
|
self.assertEqual(result["example_temperature_mean_5m"].tolist(), [25.0, 35.0])
|
|
49
43
|
|
|
44
|
+
def test_excludes_event_at_open_left_window_boundary(self) -> None:
|
|
45
|
+
result = compute_features(self.context(["2026-07-01 09:55:00", "2026-07-01 09:56:00"], [10.0, 20.0]))
|
|
46
|
+
self.assertEqual(result["example_temperature_mean_5m"].iloc[0], 20.0)
|
|
47
|
+
|
|
48
|
+
def test_includes_event_exactly_at_cutoff(self) -> None:
|
|
49
|
+
result = compute_features(self.context(["2026-07-01 10:00:00"], [30.0]))
|
|
50
|
+
self.assertEqual(result["example_temperature_mean_5m"].iloc[0], 30.0)
|
|
51
|
+
|
|
52
|
+
def test_excludes_event_after_cutoff(self) -> None:
|
|
53
|
+
result = compute_features(self.context(["2026-07-01 10:00:01"], [40.0]))
|
|
54
|
+
self.assertTrue(pd.isna(result["example_temperature_mean_5m"].iloc[0]))
|
|
55
|
+
|
|
56
|
+
def test_uses_last_value_for_duplicate_event_time(self) -> None:
|
|
57
|
+
result = compute_features(self.context(["2026-07-01 10:00:00", "2026-07-01 10:00:00"], [10.0, 30.0]))
|
|
58
|
+
self.assertEqual(result["example_temperature_mean_5m"].iloc[0], 30.0)
|
|
59
|
+
|
|
60
|
+
def test_empty_history_returns_null(self) -> None:
|
|
61
|
+
result = compute_features(self.context(["2026-07-01 09:00:00"], [10.0]))
|
|
62
|
+
self.assertTrue(pd.isna(result["example_temperature_mean_5m"].iloc[0]))
|
|
63
|
+
|
|
64
|
+
def test_requested_output_is_exactly_the_supported_subset(self) -> None:
|
|
65
|
+
result = compute_features(self.context(requested=["example_temperature_mean_5m"]))
|
|
66
|
+
self.assertEqual(list(result.columns), ["event_time", "example_temperature_mean_5m"])
|
|
67
|
+
|
|
68
|
+
def test_preserves_requested_event_time_order(self) -> None:
|
|
69
|
+
targets = ["2026-07-01 10:02:00", "2026-07-01 10:00:00"]
|
|
70
|
+
result = compute_features(self.context(targets=targets))
|
|
71
|
+
self.assertEqual(list(result["event_time"]), list(pd.DatetimeIndex(targets)))
|
|
72
|
+
|
|
73
|
+
def test_output_dtype_is_float64(self) -> None:
|
|
74
|
+
result = compute_features(self.context())
|
|
75
|
+
self.assertEqual(str(result["example_temperature_mean_5m"].dtype), "float64")
|
|
76
|
+
|
|
50
77
|
def test_rejects_unknown_requested_output(self) -> None:
|
|
51
78
|
with self.assertRaisesRegex(ValueError, "unsupported output columns"):
|
|
52
|
-
compute_features(self.context(["unknown_feature"]))
|
|
79
|
+
compute_features(self.context(requested=["unknown_feature"]))
|
|
53
80
|
|
|
54
81
|
|
|
55
82
|
if __name__ == "__main__":
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schema_version": "ml_data_platform.operator/v1",
|
|
3
|
+
"project": "replace_with_project",
|
|
3
4
|
"name": "example_temperature_features",
|
|
4
5
|
"version": "v1",
|
|
5
6
|
"type": "feature",
|
|
@@ -10,7 +11,21 @@
|
|
|
10
11
|
"code_artifact": null,
|
|
11
12
|
"input_schema": {
|
|
12
13
|
"parameters": [
|
|
13
|
-
"example_temperature:v1"
|
|
14
|
+
"replace_with_project/example_temperature:v1"
|
|
15
|
+
],
|
|
16
|
+
"prediction": {
|
|
17
|
+
"required": true,
|
|
18
|
+
"minimum_horizon": "0min",
|
|
19
|
+
"maximum_horizon": "0min"
|
|
20
|
+
},
|
|
21
|
+
"history_requirements": [
|
|
22
|
+
{
|
|
23
|
+
"anchor": "cutoff",
|
|
24
|
+
"lookback": "5min",
|
|
25
|
+
"output_columns": [
|
|
26
|
+
"example_temperature_mean_5m"
|
|
27
|
+
]
|
|
28
|
+
}
|
|
14
29
|
]
|
|
15
30
|
},
|
|
16
31
|
"output_schema": {
|
|
@@ -31,10 +31,11 @@ ml-platform --profile server show-config
|
|
|
31
31
|
ml-platform --profile server health
|
|
32
32
|
```
|
|
33
33
|
|
|
34
|
-
The
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
34
|
+
The URL is environment-specific and must be supplied by the user, an approved
|
|
35
|
+
`ML_PLATFORM_API_URL`, or the saved value from `ml-platform configure`. Do not
|
|
36
|
+
hardcode an internal address in this Skill or in a catalog. `show-config` displays
|
|
37
|
+
the effective target before discovery, dry-run, publication, build, or artifact
|
|
38
|
+
commands.
|
|
38
39
|
|
|
39
40
|
Do not place database, MinIO, or source credentials in commands when the server profile can use managed Secrets.
|
|
40
41
|
|
|
@@ -48,6 +49,15 @@ ml-platform --profile server list-feature-sets -q <stable-name>
|
|
|
48
49
|
ml-platform --profile server list-datasets -q <stable-dataset-id>
|
|
49
50
|
```
|
|
50
51
|
|
|
52
|
+
Run each command separately. Check `ml-platform --help` and the specific
|
|
53
|
+
subcommand help before adding flags. In the deployed 0.6.x CLI, `list-*`
|
|
54
|
+
commands print JSON by default and do not accept a `--json` flag.
|
|
55
|
+
If a deployed release rejects the global `--profile server` prefix, treat that
|
|
56
|
+
as CLI compatibility drift: record the syntax error, inspect `show-config` and
|
|
57
|
+
the subcommand help, then retry the same read-only command without the rejected
|
|
58
|
+
flag only if the installed help permits it. Do not report this as API/network
|
|
59
|
+
unavailability or switch to a direct HTTP client.
|
|
60
|
+
|
|
51
61
|
By default each `list-*` command auto-pages through the whole result set and
|
|
52
62
|
prints a **bare JSON array** of every matching item, so it never silently stops
|
|
53
63
|
at the server's default page size. Use `-q` for a case-insensitive identifier
|
|
@@ -154,6 +164,38 @@ The cross-store publication is not transactional. If an error occurs, report the
|
|
|
154
164
|
|
|
155
165
|
## Resolve And Build A Dataset
|
|
156
166
|
|
|
167
|
+
For an existing registered DatasetManifest, use the Registry identity only; no
|
|
168
|
+
local Catalog or Manifest path is needed:
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
ml-platform --profile server list-projects
|
|
172
|
+
ml-platform --profile server list-datasets --project <project>
|
|
173
|
+
ml-platform --profile server \
|
|
174
|
+
get-dataset <dataset_id> <dataset_version> --project <project>
|
|
175
|
+
ml-platform --profile server \
|
|
176
|
+
resolve-dataset <dataset_id> <dataset_version> --project <project>
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
After the user confirms the exact triple and authorizes one build:
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
ml-platform --profile server \
|
|
183
|
+
build-registered-dataset <dataset_id> <dataset_version> \
|
|
184
|
+
--project <project> \
|
|
185
|
+
--partition-duration 1d \
|
|
186
|
+
--max-parallelism 1 \
|
|
187
|
+
--wait
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
The command first fetches that immutable DatasetManifest from the API and sends
|
|
191
|
+
the validated response to the resolve/build endpoint. It never searches the
|
|
192
|
+
current repository for a similarly named file. Record the API source,
|
|
193
|
+
`project/dataset_id:dataset_version`, manifest hash, Job ID, terminal status,
|
|
194
|
+
and artifact reference.
|
|
195
|
+
|
|
196
|
+
When authoring a new or changed contract that has not been registered yet, use
|
|
197
|
+
the local-file commands below.
|
|
198
|
+
|
|
157
199
|
Resolve without fetching source data:
|
|
158
200
|
|
|
159
201
|
```bash
|
|
@@ -184,6 +226,17 @@ ml-platform --profile server wait-job <job_id>
|
|
|
184
226
|
ml-platform --profile server get-job <job_id>
|
|
185
227
|
```
|
|
186
228
|
|
|
229
|
+
With explicit user authorization, stop a running build through the server CLI:
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
ml-platform --profile server cancel-job <job_id>
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Cancellation is terminal for that Job; it does not change the immutable
|
|
236
|
+
DatasetManifest and it does not automatically resubmit with different
|
|
237
|
+
partition or parallelism settings. Check `get-job` after the request and
|
|
238
|
+
record the returned terminal status.
|
|
239
|
+
|
|
187
240
|
## Fetch Realtime Inference Data
|
|
188
241
|
|
|
189
242
|
Fetch one causal-cutoff row in memory:
|
|
@@ -219,6 +272,15 @@ ml-platform --profile server \
|
|
|
219
272
|
`--project` defaults to `default`; pass the owning project when the dataset lives
|
|
220
273
|
elsewhere, or the lookup returns `artifact_not_found`.
|
|
221
274
|
|
|
275
|
+
The download command writes progress to stderr after response headers arrive.
|
|
276
|
+
Interpret a timeout with no progress line as a request/header or first-byte
|
|
277
|
+
failure. Interpret a timeout after one or more `artifact download: received ...`
|
|
278
|
+
lines as an incomplete response body. In either case, report the timeout and
|
|
279
|
+
keep downloaded-file validation incomplete; do not replace the supported CLI
|
|
280
|
+
with direct HTTP or object-store access. A successful metadata query remains
|
|
281
|
+
valid metadata-only evidence, but it does not prove Parquet schema, column
|
|
282
|
+
order, or file-content hashes.
|
|
283
|
+
|
|
222
284
|
Inspect at least:
|
|
223
285
|
|
|
224
286
|
```text
|