@zhuoyuezs/ml-platform 0.2.8 → 0.2.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CI-ARTIFACT.md +2 -2
- package/checksums.json +74 -54
- package/package.json +1 -1
- package/release.json +19 -19
- package/runtime/business-client/README.md +64 -4
- package/runtime/business-client/package-lock.json +2 -2
- package/runtime/business-client/package.json +1 -1
- package/runtime/business-client/src/catalog.js +25 -6
- package/runtime/business-client/src/cli.js +177 -7
- package/runtime/business-client/src/http.js +192 -32
- package/skills/data-source-exploration/SKILL.md +12 -1
- package/skills/data-source-exploration/agents/openai.yaml +2 -2
- package/skills/data-source-exploration/references/data-platform.md +83 -0
- package/skills/feature-management/SKILL.md +29 -11
- package/skills/feature-management/agents/openai.yaml +2 -2
- package/skills/feature-management/references/commands.md +64 -7
- package/skills/feature-management/references/contracts.md +124 -6
- package/skills/feature-management/references/data-platform-mapping.md +104 -0
- package/skills/feature-management/references/operator-authoring.md +116 -8
- package/skills/feature-management/references/platform-capability-guide.md +6 -0
- package/skills/feature-management/references/supervised-datasets.md +13 -0
- package/skills/model-deployment-management/SKILL.md +7 -0
- package/skills/model-deployment-management/references/contracts.md +6 -0
- package/skills/model-deployment-management/references/operations.md +28 -1
- package/skills/model-deployment-management/references/troubleshooting.md +8 -0
- package/skills/model-lifecycle-management/SKILL.md +27 -5
- package/skills/model-lifecycle-management/agents/openai.yaml +2 -2
- package/skills/model-lifecycle-management/references/ablation.md +111 -0
- package/skills/model-lifecycle-management/references/discovery.md +18 -3
- package/skills/model-lifecycle-management/references/evaluation.md +188 -1
- package/skills/model-lifecycle-management/references/experiment-runs.md +59 -0
- package/skills/model-lifecycle-management/references/model-comparison.md +11 -9
- package/skills/model-lifecycle-management/references/packaging.md +46 -0
- package/skills/model-lifecycle-management/references/training-contracts.md +172 -1
- package/skills/model-lifecycle-management/references/training.md +60 -0
- package/skills/prediction-report-management/SKILL.md +43 -8
|
@@ -7,13 +7,25 @@ Read this file whenever a workflow creates or changes executable Operator code.
|
|
|
7
7
|
1. Package contract
|
|
8
8
|
2. Feature entrypoint contract
|
|
9
9
|
3. Causal calculation rules
|
|
10
|
-
4.
|
|
11
|
-
5.
|
|
12
|
-
6.
|
|
10
|
+
4. Online period aggregates
|
|
11
|
+
5. Required tests
|
|
12
|
+
6. Build and publication
|
|
13
|
+
7. Security boundary
|
|
13
14
|
|
|
14
15
|
## Package Contract
|
|
15
16
|
|
|
16
|
-
Build a standalone pure-Python wheel
|
|
17
|
+
Build a standalone pure-Python wheel (`py3-none-any`). The platform verifies
|
|
18
|
+
the wheel digest and only extracts it into an isolated import path; it does not
|
|
19
|
+
install `Requires-Dist` dependencies, run `pip`, or accept native code. At run
|
|
20
|
+
time an Operator can import only the Python standard library, its own wheel,
|
|
21
|
+
and packages already present in the image that executes it: dataset builds
|
|
22
|
+
(the `dataset-chronon` dependency group, which provides `pandas`, `numpy` via
|
|
23
|
+
`pandas`, `pyarrow` and `pydantic`) and API realtime reads (the `api`
|
|
24
|
+
dependency group). Do not import report-local, experiment-local, or
|
|
25
|
+
unregistered business packages, and do not depend on libraries such as `scipy`
|
|
26
|
+
or `scikit-learn` that those images do not provide; vendor small pure-Python
|
|
27
|
+
helpers into the wheel instead. `[project].dependencies` documents the expected
|
|
28
|
+
environment for local tests only; it does not make a dependency available.
|
|
17
29
|
|
|
18
30
|
Minimal `pyproject.toml`:
|
|
19
31
|
|
|
@@ -63,9 +75,19 @@ computation_hash
|
|
|
63
75
|
|
|
64
76
|
Each `metric_frames[parameter_key]` is a standard long frame with source timestamps and values. Resolve frames from `context.inputs`; do not reach into a database, API, filesystem secret, or environment credential.
|
|
65
77
|
|
|
66
|
-
Use `context.cutoff_times` for causal feature windows.
|
|
67
|
-
|
|
68
|
-
|
|
78
|
+
Use `context.cutoff_times` for causal feature windows. `context.target_times`
|
|
79
|
+
are the materialized rows' `event_time` values, which are the prediction
|
|
80
|
+
decision times, and `context.cutoff_times` equals them: a row's feature
|
|
81
|
+
cutoff is its own `event_time`. The DatasetManifest `prediction.horizon` moves
|
|
82
|
+
label availability to `label_time = event_time + horizon` (and the realtime
|
|
83
|
+
response `target_time = cutoff_time + horizon`); it does not move the Operator
|
|
84
|
+
context's feature reads earlier. Do not subtract `context.prediction_horizon`
|
|
85
|
+
from either index, and do not parse a duplicated Feature-level horizon. This
|
|
86
|
+
rule covers Operator inputs only: a DatasetManifest `endpoint_policy` evaluates
|
|
87
|
+
abnormal-window context overlap with its own `cutoff = t - h` formula, described
|
|
88
|
+
in [contracts.md](contracts.md). Resolved lineage records this as
|
|
89
|
+
`prediction_time_semantics: event_time_is_causal_cutoff` and
|
|
90
|
+
`label_time_derivation: prediction_time_plus_horizon`.
|
|
69
91
|
|
|
70
92
|
An Operator requires an explicit forecast contract whenever its formula or code
|
|
71
93
|
uses `context.cutoff_times`, or any `input_schema.history_requirements` entry uses
|
|
@@ -100,7 +122,11 @@ One Operator may return multiple physical columns. Each public Feature still bin
|
|
|
100
122
|
|
|
101
123
|
## Causal Calculation Rules
|
|
102
124
|
|
|
103
|
-
State every time rule in tests and descriptions
|
|
125
|
+
State every time rule in tests and descriptions. A formula that needs the
|
|
126
|
+
complete requested window rather than a bounded causal lookback (for example
|
|
127
|
+
full-window interpolation or smoothing) is not partition-invariant; document
|
|
128
|
+
that the Dataset must be built with `--partition-duration none
|
|
129
|
+
--max-parallelism 1` (see [commands.md](commands.md)).
|
|
104
130
|
|
|
105
131
|
- forecast horizon and cutoff derivation;
|
|
106
132
|
- whether an event exactly at cutoff is included (`<=`) or excluded (`<`);
|
|
@@ -113,6 +139,72 @@ State every time rule in tests and descriptions:
|
|
|
113
139
|
|
|
114
140
|
Never infer a rule from a Feature name alone. Require a business decision when a boundary is unspecified.
|
|
115
141
|
|
|
142
|
+
## Online Period Aggregates
|
|
143
|
+
|
|
144
|
+
Use this method when one Feature row summarizes a whole period (for example a
|
|
145
|
+
day, shift or batch window) and the Feature must also be served online through
|
|
146
|
+
`fetch-inference-data` or a model's `feature_lookup`. The realtime path gives
|
|
147
|
+
the Operator exactly one target row, `event_time = cutoff`, and only source
|
|
148
|
+
rows with `timestamp <= cutoff`, so the batch and online values agree only when
|
|
149
|
+
the formula is causal and bounded.
|
|
150
|
+
|
|
151
|
+
Row semantics:
|
|
152
|
+
|
|
153
|
+
- Anchor the row at the period end: the row at cutoff `T` summarizes the
|
|
154
|
+
period that ended at `T` (for a daily Feature, `T` is local midnight and the
|
|
155
|
+
period is the previous day). Never label a row with the period start and
|
|
156
|
+
aggregate data after it.
|
|
157
|
+
- Decide the period boundary from the source timestamp meaning, which must be
|
|
158
|
+
confirmed by the data owner (see [contracts.md](contracts.md), realtime
|
|
159
|
+
policy questions). If an hourly record is stamped at the hour start, the
|
|
160
|
+
record stamped `T` belongs to the next period, so use `[T - period, T)`. If it
|
|
161
|
+
is stamped at the hour end, the record stamped `T` closes the period, so use
|
|
162
|
+
`(T - period, T]`. Test the record exactly at `T` either way.
|
|
163
|
+
|
|
164
|
+
Contract declarations:
|
|
165
|
+
|
|
166
|
+
- `supports_online: true` on the Operator and `offline_online_supported: true`
|
|
167
|
+
on each Feature that will be served online.
|
|
168
|
+
- `input_schema.history_requirements` with `anchor: cutoff` and a lookback that
|
|
169
|
+
covers the aggregation period, any multi-period context (for example a
|
|
170
|
+
3-day rolling statistic), and alignment slack for source timestamps. For
|
|
171
|
+
example, a daily mean that also uses the two previous days needs at least
|
|
172
|
+
`3d` plus slack. This lookback is also the per-Parameter realtime read
|
|
173
|
+
window; the default realtime floor `min_lookback=1h` is far too short for a
|
|
174
|
+
period aggregate, and it is not a substitute for the declaration.
|
|
175
|
+
- `input_schema.prediction` with `required`, `minimum_horizon` and
|
|
176
|
+
`maximum_horizon` (a strict nowcast uses `true`, `0min`, `0min`).
|
|
177
|
+
- Do not use `anchor: target_time` with zero lookback for a period aggregate;
|
|
178
|
+
online that reads almost nothing.
|
|
179
|
+
|
|
180
|
+
Formula rules:
|
|
181
|
+
|
|
182
|
+
- Use only rows inside the declared lookback that are at or before the row's
|
|
183
|
+
cutoff. A row's value must not change when data after its cutoff is added,
|
|
184
|
+
or when history older than the lookback is added or removed.
|
|
185
|
+
- Freeze fitted constants in Feature `config`. Values fitted on training data,
|
|
186
|
+
such as fallback medians, clipping limits or normal ranges, must be computed
|
|
187
|
+
once offline, written into `config`, and versioned with the Feature. Do not
|
|
188
|
+
recompute them from the window at run time: the online window holds only
|
|
189
|
+
the lookback, so a recomputed value differs between batch and online and
|
|
190
|
+
between partitions. A retrain that refits these constants needs a new
|
|
191
|
+
Feature (and usually Operator) version.
|
|
192
|
+
- Fill gaps only forward in time, and bound each fill by a maximum age or a
|
|
193
|
+
frozen constant. Never backward-fill from an observation after the row's
|
|
194
|
+
cutoff. Avoid backward-filling the start of the window too: the result then
|
|
195
|
+
depends on where the window starts and breaks partition invariance.
|
|
196
|
+
- The Operator owns interior gap handling. The platform's realtime tolerances
|
|
197
|
+
(`allow_tail_edge_missing`, `allow_internal_gap_fill`) only judge how far the
|
|
198
|
+
newest source row lags the cutoff; they do not insert values. Keep one
|
|
199
|
+
missing-data rule, inside the Operator or a declared Parameter rule, so batch
|
|
200
|
+
builds and realtime reads compute the same value.
|
|
201
|
+
- Emit data-quality flags as their own output columns, each bound to its own
|
|
202
|
+
Feature: for example a valid-observation count, an "insufficient data" flag,
|
|
203
|
+
or a business state flag such as "plant stopped". A flag tells consumers
|
|
204
|
+
that a row was computed under a fallback rule; do not hide that by silently
|
|
205
|
+
filling the value. Include a flag in the model FeatureSet only if it is an
|
|
206
|
+
intended model input.
|
|
207
|
+
|
|
116
208
|
## Required Tests
|
|
117
209
|
|
|
118
210
|
Use deterministic fixtures and cover each applicable item with a separately
|
|
@@ -131,6 +223,22 @@ requested-output, event-time-order, or dtype/rounding cases below:
|
|
|
131
223
|
9. output dtype and rounding;
|
|
132
224
|
10. every dated business-rule branch.
|
|
133
225
|
|
|
226
|
+
For an online-capable Operator (`supports_online: true`), also add:
|
|
227
|
+
|
|
228
|
+
11. cutoff invariance: adding source rows after the cutoff leaves the row
|
|
229
|
+
unchanged;
|
|
230
|
+
12. lookback invariance: adding or removing history older than the declared
|
|
231
|
+
lookback leaves the row unchanged;
|
|
232
|
+
13. partition invariance: computing many cutoffs in one call equals computing
|
|
233
|
+
each cutoff alone with only its lookback window, which is what a realtime
|
|
234
|
+
read does;
|
|
235
|
+
14. fallback rows: a period with no or too few valid observations produces the
|
|
236
|
+
documented fallback value and quality flag, using the frozen constants.
|
|
237
|
+
|
|
238
|
+
After publication, compare `fetch-inference-data` at a few historical cutoffs
|
|
239
|
+
with the same rows of a batch DatasetArtifact. Any difference means the
|
|
240
|
+
Operator, its fill rule or its lookback is not causal; fix it before training.
|
|
241
|
+
|
|
134
242
|
Run package tests before building the wheel. Then inspect the wheel tag and ensure it is `py3-none-any`.
|
|
135
243
|
|
|
136
244
|
## Build And Publication
|
|
@@ -14,6 +14,12 @@ replay metadata, and data Job execution. This Skill covers that data subsystem;
|
|
|
14
14
|
use the separate model lifecycle Skill for training, evaluation and packaging.
|
|
15
15
|
Model serving, source-table creation and Kubernetes administration are outside
|
|
16
16
|
this Skill's scope. Do not present a Skill boundary as a platform capability gap.
|
|
17
|
+
Parameters may read through the platform's direct database adapters or a pinned,
|
|
18
|
+
published external data-platform Mapping. Mapping discovery and source query
|
|
19
|
+
semantics remain owned by `$data-source-exploration`; binding its approved outputs
|
|
20
|
+
to ordinary Parameters is documented in
|
|
21
|
+
[data-platform-mapping.md](data-platform-mapping.md). Kubernetes Worker image and
|
|
22
|
+
Secret configuration remain platform-operator responsibilities.
|
|
17
23
|
|
|
18
24
|
The following distinctions are part of the platform contract:
|
|
19
25
|
|
|
@@ -151,6 +151,19 @@ For a manifest split named `train`, use `["train"]`. Hand off the exact artifact
|
|
|
151
151
|
identity, DataSchema hash, FeatureSet order, label/temporal contracts, observed
|
|
152
152
|
split counts and binding together.
|
|
153
153
|
|
|
154
|
+
When the same immutable DatasetArtifact must be evaluated with different date
|
|
155
|
+
boundaries, use the TrainingRun level derived binding instead of rebuilding the
|
|
156
|
+
artifact. The supported forms are `mode: derived`, with `strategy: time_range`
|
|
157
|
+
or `chronological_ratio`. Time ranges use timezone aware half open
|
|
158
|
+
ranges for `train`, `validation` and optional `test`. The resolver freezes row identities and uses
|
|
159
|
+
the prediction time column as the sample start. It removes rows whose
|
|
160
|
+
`label_time` crosses that role's end boundary and reports the count; it never
|
|
161
|
+
moves or rewrites the label. `test` remains an independent evaluation role,
|
|
162
|
+
`chronological_ratio` keeps equal prediction timestamps together. Actual counts
|
|
163
|
+
can differ from requested ratios. The ordinary
|
|
164
|
+
`assignment_column` form remains the compatibility path for artifacts that
|
|
165
|
+
already contain a split column.
|
|
166
|
+
|
|
154
167
|
## Evidence checkpoints
|
|
155
168
|
|
|
156
169
|
SQL-source `verifiable=false` means the automatic clock analysis could not
|
|
@@ -35,6 +35,9 @@ deployment does not reach `READY` or an operation is rejected.
|
|
|
35
35
|
the model capacity and sampling grid. In `feature_lookup` mode, never send
|
|
36
36
|
`future_covariates`; the platform retrieves the complete known-future Feature
|
|
37
37
|
window visible at the cutoff. Inline mode still supplies those rows in inputs.
|
|
38
|
+
- Clustering deployments take no `inference_policy`, `horizon`, or
|
|
39
|
+
`output_window`; an optional `include_inputs: true` echoes the model input
|
|
40
|
+
row. See the clustering request in [references/operations.md](references/operations.md).
|
|
38
41
|
- Run `validate-deployment` before `create-deployment`. Validation is read-only
|
|
39
42
|
and does not authorize creation.
|
|
40
43
|
- Obtain explicit authorization for the exact deployment and operation before
|
|
@@ -43,6 +46,10 @@ deployment does not reach `READY` or an operation is rejected.
|
|
|
43
46
|
returned an ambiguous result.
|
|
44
47
|
- Treat `READY` as valid only when `ReplicasReady`, `ArtifactsVerified`, and
|
|
45
48
|
`SmokePredictionPassed` are all true for the current revision.
|
|
49
|
+
- A READY staging revision can use unified prediction and scheduled reporting.
|
|
50
|
+
Do not promote merely to enable either workflow. Promote only when the user
|
|
51
|
+
requests routed canary/production traffic; unified prediction selects the
|
|
52
|
+
deployment's current revision and does not split traffic by canary weight.
|
|
46
53
|
- Use `predict-deployment` for governed deployment verification. Do not infer
|
|
47
54
|
Kubernetes Service names from a deployment ID.
|
|
48
55
|
- Treat every prediction response `target_time` as China Standard Time in RFC
|
|
@@ -72,6 +72,12 @@ continues to carry them in `inputs.future_covariates`.
|
|
|
72
72
|
The first deployment starts at `staging`. A replacement for an existing
|
|
73
73
|
service/environment declares the active leaf in `previous_deployment_id`.
|
|
74
74
|
Production is reached only by staging to canary to production promotion.
|
|
75
|
+
This stage sequence governs routed rollout, not eligibility for unified prediction
|
|
76
|
+
or reporting: a READY staging revision is sufficient for those workflows.
|
|
77
|
+
New promotion requires `service_exposure.type=Ingress` or an explicit
|
|
78
|
+
`rollout.gateway_name`. NodePort alone does not provide weighted routing.
|
|
79
|
+
The server checks routing prerequisites before persisting the promoted revision;
|
|
80
|
+
a rejected preflight preserves the existing revision and its READY state.
|
|
75
81
|
|
|
76
82
|
`service_exposure` is optional and defaults to `{"type":"ClusterIP"}`. Set
|
|
77
83
|
`type=NodePort` only when direct node access is required. `node_port` is
|
|
@@ -39,7 +39,11 @@ ml-platform --profile server list-deployments \
|
|
|
39
39
|
--model-artifact-id <artifact-id>
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
|
|
42
|
+
列表默认按 `deployment_id` 升序;需要按生命周期时间排序时使用
|
|
43
|
+
`--sort-by created_at|updated_at --sort-order asc|desc`。
|
|
44
|
+
|
|
45
|
+
Use `--service TEXT` for a case-insensitive service-name substring; surrounding
|
|
46
|
+
whitespace is ignored and an empty value does not filter. The filter may be combined with
|
|
43
47
|
`--environment`, `--state`, `--q`, `--limit`, and `--offset`. These results are
|
|
44
48
|
ITSMP platform deployments; external model-center delivery callbacks are a
|
|
45
49
|
separate status surface.
|
|
@@ -74,6 +78,20 @@ exposure mode.
|
|
|
74
78
|
|
|
75
79
|
## Promotion And Approval
|
|
76
80
|
|
|
81
|
+
Skip this step for ordinary unified prediction or scheduled reports on a READY
|
|
82
|
+
staging deployment. For weighted traffic, use an environment's existing
|
|
83
|
+
ingress-nginx with `service_exposure.type=Ingress`, or an explicitly bound,
|
|
84
|
+
ready Gateway via `rollout.gateway_name`. Inspect the immutable DeploymentSpec
|
|
85
|
+
first; changing exposure requires a reviewed replacement deployment. Do not
|
|
86
|
+
install a Gateway merely because the default ClusterIP or NodePort deployment
|
|
87
|
+
cannot be promoted.
|
|
88
|
+
|
|
89
|
+
Promotion performs Kubernetes routing dry-run admission before saving a new
|
|
90
|
+
revision. This checks API permissions and admission, not live ingress-controller
|
|
91
|
+
health or end-to-end reachability. Continue observing reconciliation and READY
|
|
92
|
+
conditions after an accepted promotion. Canary weights apply to the routed
|
|
93
|
+
entrypoint; `predict-deployment` directly verifies the selected revision.
|
|
94
|
+
|
|
77
95
|
```bash
|
|
78
96
|
ml-platform --profile server promote-deployment <deployment-id> \
|
|
79
97
|
--stage canary --actor <actor> --reason <reason>
|
|
@@ -157,6 +175,15 @@ ml-platform --profile server predict-deployment <deployment-id> \
|
|
|
157
175
|
/absolute/path/request.json
|
|
158
176
|
```
|
|
159
177
|
|
|
178
|
+
For a clustering model (`task_type: clustering` in `GET /v1/model`), send no
|
|
179
|
+
`horizon` or `output_window`; they are rejected. The single result is
|
|
180
|
+
`prediction: {kind: "clustering", cluster_id, distance_to_center |
|
|
181
|
+
cluster_probability}` without `target_time`. Add `"include_inputs": true` to
|
|
182
|
+
receive `results[0].inputs`, the validated model input row in signature column
|
|
183
|
+
order (before the model's own transform); it defaults to `false`, is rejected
|
|
184
|
+
for forecasting models, and requires a package image that supports it. Missing
|
|
185
|
+
or non-finite retrieved inputs fail the request.
|
|
186
|
+
|
|
160
187
|
For a package with an explicit `availability_boundary`, inspect
|
|
161
188
|
`lineage.visibility` in the prediction response. It records the package's
|
|
162
189
|
boundary, `cutoff_time`, `attempt_started_at`, `knowledge_time`,
|
|
@@ -10,6 +10,14 @@ ml-platform --profile server get-deployment-events <deployment-id> \
|
|
|
10
10
|
|
|
11
11
|
Interpret conditions before considering a reconcile:
|
|
12
12
|
|
|
13
|
+
- A rejected promotion mentioning `rollout.gateway_name`, Gateway HTTP status,
|
|
14
|
+
or routing RBAC leaves the current revision unchanged. Check it with
|
|
15
|
+
`get-deployment`; if staging is READY, unified prediction and reporting remain
|
|
16
|
+
usable. Use existing Ingress for intentional canary traffic, or correct the
|
|
17
|
+
explicit Gateway binding and permissions. Do not repeatedly promote to make
|
|
18
|
+
ordinary prediction work. Historical FAILED promotion revisions are not
|
|
19
|
+
automatically repaired by this guard; inspect the active replacement before
|
|
20
|
+
planning recovery.
|
|
13
21
|
- `ReplicasReady=false`: inspect scheduler, image pull, resource quota, and Pod
|
|
14
22
|
readiness through the platform/cluster diagnostic boundary.
|
|
15
23
|
- `ArtifactsVerified=false`: the Serving `/model` identity does not match the
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: model-lifecycle-management
|
|
3
|
-
description: Manage ITSMP system-model discovery, model training, evaluation, immutable model packaging, and contract-bound inference requests. Use when a user asks to inspect platform-owned SystemModels and runtime bindings, validate or submit a training run, inspect/retry/cancel training jobs, configure or run governed evaluation, inspect evaluation evidence, validate a ModelArtifact, create and inspect a ModelPackage, or invoke a deployed model. Do not use for feature catalog authoring, dataset builds, model deployment, Kubernetes administration, or image release engineering.
|
|
3
|
+
description: Manage ITSMP system-model discovery, model training, evaluation, controlled model comparison and feature/input ablation, immutable model packaging, and contract-bound inference requests. Use when a user asks to inspect platform-owned SystemModels and runtime bindings, validate or submit a training run, run a baseline/variant ablation experiment, inspect/retry/cancel training jobs, configure or run governed evaluation, inspect evaluation evidence, validate a ModelArtifact, create and inspect a ModelPackage, or invoke a deployed model. Do not use for feature catalog authoring, dataset builds, model deployment, Kubernetes administration, or image release engineering.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Model Lifecycle Management
|
|
@@ -31,10 +31,32 @@ resources or selecting runtime bindings. It also defines Job/package preflight
|
|
|
31
31
|
and version-skew handling.
|
|
32
32
|
|
|
33
33
|
For multiple-model experiments or iterative optimization, read
|
|
34
|
-
[references/model-comparison.md](references/model-comparison.md)
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
34
|
+
[references/model-comparison.md](references/model-comparison.md) and
|
|
35
|
+
[references/experiment-runs.md](references/experiment-runs.md). Use the
|
|
36
|
+
platform's durable `ExperimentRun` to associate candidates and reconcile jobs.
|
|
37
|
+
The method still comes from this Skill; ExperimentRun does not select a winner
|
|
38
|
+
or deploy a model.
|
|
39
|
+
|
|
40
|
+
For feature or input ablation experiments, also read
|
|
41
|
+
[references/ablation.md](references/ablation.md). Ablation is a comparison
|
|
42
|
+
protocol represented by `ExperimentRun` and linked to ordinary TrainingRun and
|
|
43
|
+
EvaluationRun resources; there is no separate `AblationRun` API.
|
|
44
|
+
|
|
45
|
+
For unsupervised clustering (`task.kind=clustering`, for example
|
|
46
|
+
`kmeans_clusterer` or `gaussian_mixture_clusterer`), read the clustering
|
|
47
|
+
section of [references/training-contracts.md](references/training-contracts.md)
|
|
48
|
+
for the Run, parameter and model-file contract, and the clustering section of
|
|
49
|
+
[references/packaging.md](references/packaging.md) for packaging and
|
|
50
|
+
single-row inference without a horizon. The cluster count and preprocessing
|
|
51
|
+
have silent defaults (8 clusters for KMeans, no scaling or PCA). Ask the user
|
|
52
|
+
for them, or select them with the documented method in that reference, before
|
|
53
|
+
submitting a clustering Run.
|
|
54
|
+
|
|
55
|
+
For clustering evaluation and registration, read the clustering section of
|
|
56
|
+
[references/evaluation.md](references/evaluation.md). Use a `task_type:
|
|
57
|
+
"clustering"` EvaluationConfig with `targets: ["cluster_id"]`, and label
|
|
58
|
+
`all`/`train` rowset results as in-sample. Register only with one PASS,
|
|
59
|
+
COMPLETE EvaluationResult for that artifact.
|
|
38
60
|
|
|
39
61
|
Read [references/zero-shot-inference.md](references/zero-shot-inference.md) when
|
|
40
62
|
the user wants synchronous zero-shot forecasting from JSON or CSV. This path
|
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
interface:
|
|
2
2
|
display_name: "模型训练、评估与打包"
|
|
3
|
-
short_description: "管理 ITSMP
|
|
4
|
-
default_prompt: "使用 $model-lifecycle-management
|
|
3
|
+
short_description: "管理 ITSMP 模型训练、评估、比较与消融实验"
|
|
4
|
+
default_prompt: "使用 $model-lifecycle-management 管理模型训练、评估、比较或消融实验流程。"
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# Feature and input ablation
|
|
2
|
+
|
|
3
|
+
Use this method when the question is whether a feature, feature group,
|
|
4
|
+
covariate, input channel or preprocessing input contributes to a model result.
|
|
5
|
+
It is a controlled comparison protocol represented by an `ExperimentRun`. The
|
|
6
|
+
run links existing `TrainingRun`, `ModelArtifact`, `EvaluationRun` and
|
|
7
|
+
`EvaluationResult` facts. Read [experiment-runs.md](experiment-runs.md) for
|
|
8
|
+
execution and [model-comparison.md](model-comparison.md) for evidence rules.
|
|
9
|
+
|
|
10
|
+
## Freeze the baseline scope
|
|
11
|
+
|
|
12
|
+
Define one baseline candidate with the complete approved input FeatureSet and
|
|
13
|
+
freeze these values for the whole ablation round:
|
|
14
|
+
|
|
15
|
+
- the exact `DatasetArtifact` and `manifest_hash`;
|
|
16
|
+
- DataSchema, label columns, prediction/label time semantics and series keys;
|
|
17
|
+
- train/validation/test assignment and rowset population;
|
|
18
|
+
- evaluation config, metric versions, calculation space and slices;
|
|
19
|
+
- SystemModel, runtime release, reproducibility settings and execution policy.
|
|
20
|
+
|
|
21
|
+
The baseline is the full input set for the hypothesis being tested. A model
|
|
22
|
+
name, FeatureSet name or metric value is not enough to identify it. Store the
|
|
23
|
+
baseline and variants in one immutable ExperimentRun spec.
|
|
24
|
+
|
|
25
|
+
## Define variants
|
|
26
|
+
|
|
27
|
+
Create one candidate per removal or replacement under test. Prefer one-factor
|
|
28
|
+
variants:
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
baseline: [load, temperature, pressure, calendar]
|
|
32
|
+
remove_load: [temperature, pressure, calendar]
|
|
33
|
+
remove_weather: [load, pressure, calendar]
|
|
34
|
+
remove_calendar: [load, temperature, pressure]
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Keep the retained columns in their declared order. Give every candidate a
|
|
38
|
+
stable candidate ID, a short hypothesis, the exact removed input keys and the
|
|
39
|
+
parent baseline candidate ID in the ExperimentRun spec. Do not change hyperparameters,
|
|
40
|
+
labels, splits or evaluation policy at the same time as the ablated input.
|
|
41
|
+
If several inputs must be removed together for a single hypothesis, name that
|
|
42
|
+
group explicitly and do not interpret the result as the contribution of each
|
|
43
|
+
member separately.
|
|
44
|
+
|
|
45
|
+
For a feature-only ablation, bind each candidate to an immutable registered
|
|
46
|
+
FeatureSet that is an allowed subset/order projection of the same artifact.
|
|
47
|
+
The wide DatasetArtifact normally does not need rebuilding. Use
|
|
48
|
+
`$feature-management` to create or discover those FeatureSet versions; do not
|
|
49
|
+
edit an existing FeatureSet in place. If the hypothesis changes source data,
|
|
50
|
+
operator logic, label materialization, row membership, or feature values in
|
|
51
|
+
the artifact, create a new DatasetArtifact and start a separate comparison
|
|
52
|
+
scope instead of calling it a feature ablation.
|
|
53
|
+
|
|
54
|
+
## Execute the round
|
|
55
|
+
|
|
56
|
+
1. Create one ExperimentRun with the immutable dataset identity, evaluation
|
|
57
|
+
contract, baseline candidate and variants. Use
|
|
58
|
+
[experiment-runs.md](experiment-runs.md) for the JSON shape.
|
|
59
|
+
2. Validate all candidates before submitting any Job. Validation must show each
|
|
60
|
+
candidate FeatureSet, projected input columns, schema hash and runtime
|
|
61
|
+
identity. Validation has no training side effects.
|
|
62
|
+
3. Submit only within the user's authorized training scope. Reconcile the
|
|
63
|
+
ExperimentRun after uncertain responses; never create a replacement run to
|
|
64
|
+
retry an unknown candidate submission.
|
|
65
|
+
4. Observe every Job and validate its ModelArtifact. A failed or infrastructure
|
|
66
|
+
Job has no model-quality score and must be reported separately.
|
|
67
|
+
5. Evaluate every successful artifact on the same frozen validation rowset with
|
|
68
|
+
the same EvaluationConfig. Inspect coverage, required decisions and result
|
|
69
|
+
hashes; do not compute replacement metrics locally.
|
|
70
|
+
6. Inspect the ExperimentRun comparison on complete, comparable results. Use
|
|
71
|
+
the baseline as the reference and report absolute and relative metric
|
|
72
|
+
deltas. For metrics where lower is better, a positive candidate-minus-
|
|
73
|
+
baseline delta is degradation; follow the MetricDefinition rather than
|
|
74
|
+
assuming direction.
|
|
75
|
+
|
|
76
|
+
Example comparison request:
|
|
77
|
+
|
|
78
|
+
```json
|
|
79
|
+
{
|
|
80
|
+
"schema_version": "ml_data_platform.evaluation_result_comparison_request/v1",
|
|
81
|
+
"comparison_type": "model",
|
|
82
|
+
"result_ids": ["baseline_result", "remove_load_result"],
|
|
83
|
+
"metrics": ["regression.mae", "regression.rmse"],
|
|
84
|
+
"sort_direction": "ascending"
|
|
85
|
+
}
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Do not rank candidates with different rowsets, labels, horizons, metric
|
|
89
|
+
definitions or incomplete coverage. A candidate that fails because of a
|
|
90
|
+
runtime, image or contract problem is inconclusive about the ablated input.
|
|
91
|
+
|
|
92
|
+
## Interpret and close the round
|
|
93
|
+
|
|
94
|
+
Report, for every candidate, the baseline/variant FeatureSet identities, exact
|
|
95
|
+
removed inputs, TrainingRun/Job/ModelArtifact IDs, EvaluationResult IDs,
|
|
96
|
+
coverage, metrics, deltas and failure evidence. A small metric difference is
|
|
97
|
+
not proof that an input is useless; state the observed tolerance and whether
|
|
98
|
+
the experiment had repeated seeds or uncertainty estimates. If the result is
|
|
99
|
+
inconclusive, keep the input undecided rather than deleting it.
|
|
100
|
+
|
|
101
|
+
Only after validation comparison selects a candidate may the normal held-out
|
|
102
|
+
test procedure in `model-comparison.md` be used. The test result validates the
|
|
103
|
+
locked candidate and must not be used to choose another ablation. Packaging,
|
|
104
|
+
registration and deployment remain separate authorized gates.
|
|
105
|
+
|
|
106
|
+
The round is complete only when every submitted candidate is accounted for as
|
|
107
|
+
complete, failed, pending, unknown or skipped with a reason. Stop when the
|
|
108
|
+
declared ablation set is covered, the authorized training scope is exhausted,
|
|
109
|
+
or a shared data/runtime defect blocks the round. Do not add an automatic
|
|
110
|
+
feature deletion threshold; business owners decide the practical impact and
|
|
111
|
+
tradeoff after reviewing comparable evidence.
|
|
@@ -65,6 +65,20 @@ Jobs in lists are immutable specs; use the exact get-job command for refreshed
|
|
|
65
65
|
status. Lists do not reconcile or schedule workloads. `get-training-run` returns
|
|
66
66
|
`training_run` plus `spec_hash`; artifact/package detail returns the object.
|
|
67
67
|
|
|
68
|
+
## Download artifact files
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
ml-platform --profile server download-model-artifact ARTIFACT_ID --out-dir /tmp/model-ARTIFACT_ID [--file runtime.json]
|
|
72
|
+
ml-platform --profile server download-evaluation-result-artifact RESULT_ID --out-dir /tmp/eval-RESULT_ID [--file predictions.parquet]
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Both commands download only files indexed by the registry record (all of them
|
|
76
|
+
unless `--file` is repeated), verify each sha256 against that record, and remove
|
|
77
|
+
the output directory on any mismatch. `--project` fails closed when the
|
|
78
|
+
resource belongs to another project. Report the returned `files[].sha256` as
|
|
79
|
+
content evidence; metadata reads alone do not prove file contents. Write
|
|
80
|
+
downloads outside the repository.
|
|
81
|
+
|
|
68
82
|
## Read-only preflight
|
|
69
83
|
|
|
70
84
|
```bash
|
|
@@ -97,10 +111,11 @@ Training Run `task`, `data`, `data.artifact`, `features.feature_set`, `splits`,
|
|
|
97
111
|
config references, members and execution, plus ModelPackage requests, are also
|
|
98
112
|
structured. Unknown keys return 422 with `detail[].loc` identifying the field.
|
|
99
113
|
Keep algorithm-specific `trainer.parameters` governed by the SystemModel
|
|
100
|
-
parameter schema. A typed
|
|
114
|
+
parameter schema. A typed contract such as Chronos-2 `lora_config` or the
|
|
115
|
+
KMeans/Gaussian Mixture clustering parameters (including `preprocessing`)
|
|
101
116
|
rejects unknown fields; a SystemModel that intentionally advertises
|
|
102
|
-
`additionalProperties` remains extensible.
|
|
103
|
-
|
|
117
|
+
`additionalProperties` remains extensible. Read `additionalProperties` from
|
|
118
|
+
the exact SystemModel instead of inferring one model's policy from another.
|
|
104
119
|
|
|
105
120
|
These HTTP checks preserve accepted values and do not rewrite historical stored
|
|
106
121
|
contracts or hashes. Legacy records remain readable. A previously ignored typo
|