@zhuoyuezs/ml-platform 0.2.8 → 0.2.10

package/skills/data-source-exploration/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: data-source-exploration
-description: Discover platform-configured PostgreSQL, MySQL, SQL Server, or InfluxDB objects and run bounded structured reads for local data and feature exploration. Use when an algorithm engineer wants to inspect visible databases, schemas, tables, measurements, fields, or limited rows before defining governed feature assets. Do not use for free-form SQL, bulk export, asset publication, dataset builds, or model training.
+description: Discover platform-configured database objects or published external data-platform Mappings, and run bounded structured or Mapping snapshot reads for local data and feature exploration. Use when an algorithm engineer wants to inspect visible sources, Mapping outputs, time coverage or limited rows before defining governed feature assets. Do not use for free-form SQL, bulk export, asset publication, dataset builds, or model training.
 ---
 
 # Data Source Exploration
@@ -10,6 +10,15 @@ never request credentials in chat. The platform exposes managed connections and
 objects. Connection credentials are write-only control-plane input and are never returned by the
 platform.
 
+## External Data Platform
+
+For external published mapping discovery and bounded query/snapshot reads, follow
+[the external data platform reference](references/data-platform.md). This includes the complete
+authorized query-candidate catalog, fixed Mapping revisions, asynchronous query runs and paged
+snapshots. Keep that catalog separate from the direct database connections below; it has its own
+IDs, permissions and query semantics. Use `$feature-management` to bind confirmed outputs to
+Parameters and build features; discovery alone does not publish assets.
+
 ## Manage Connections
 
 Datasource connections are business-owned platform resources, not ConfigMap entries. An operator
@@ -48,6 +57,8 @@ Before discovery, verify the installed client and server target:
 
 ```bash
 command -v ml-platform
+ml-platform version
+ml-platform show-config
 ml-platform health
 ```
 

package/skills/data-source-exploration/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "数据源与特征探索"
-  short_description: "逐层发现已连接数据源并执行受限数据查询"
-  default_prompt: "使用 $data-source-exploration 从平台已连接数据源中发现可见库表字段，执行有限查询，并整理数据与特征假设。"
+  short_description: "发现已连接数据源或外部 Mapping，并执行有界查询"
+  default_prompt: "使用 $data-source-exploration 发现平台可见库表字段或外部数据平台已发布 Mapping，执行有界查询和快照读取，并整理覆盖率、数据与特征假设。"

package/skills/data-source-exploration/references/data-platform.md

@@ -0,0 +1,83 @@
+# Read Published Mappings From The External Data Platform
+
+Use this path when the user requests data owned by the external data platform. The modeling
+server holds its login configuration; agents never need its credentials or database DSNs.
+Discover available options with the installed `ml-platform --help`; an older CLI or unconfigured
+server does not imply permission to fall back to a physical database.
+
+1. Run `ml-platform data-platform-status` and `ml-platform list-data-platform-connections`.
+2. Inspect connection metadata and objects with `get-data-platform-connection ID` and
+   `list-data-platform-objects ID`. These are catalogs, not a physical-query interface.
+3. Prefer `list-data-platform-query-candidates` for model data discovery. It requires the
+   integration account's upstream `query:run` capability and returns its complete authorized,
+   published, enabled and ready catalog as `candidates`, `complete: true`, and `total`, without
+   reading source data. Inspect business names, domains and output labels, keys, units, roles and
+   types; filter this complete set locally. It accepts no paging or filter flags. An empty complete
+   set is an honest empty result; domains are presentation metadata, not permission grants.
+   For an older upstream that lacks this endpoint, use
+   `list-data-platform-mappings --page 1 --page-size 50 [--scope-code CODE]`, follow every page
+   using `total`, and inspect `lifecycle.run_readiness`. This catalog alone does not prove query
+   authorization. Do not fall back on a permission error or to bypass blocked mappings.
+4. Inspect `get-data-platform-mapping-revision MAPPING_ID REVISION_ID`. Confirm output keys,
+   units and time semantics. Preserve `active_revision.mapping_revision_id` for reproducibility.
+   Execution rechecks authorization and revision state; discovery does not reserve access.
+5. Write a bounded query JSON in the user's workspace, outside the platform repository:
+
+   ```json
+   {
+     "mapping": {"mapping_revision_id": "33333333-3333-4333-8333-333333333333"},
+     "window_start": "2026-10-01T00:00:00+08:00",
+     "window_end": "2026-10-01T01:00:00+08:00",
+     "max_rows": 100,
+     "execution": "async"
+   }
+   ```
+
+6. Submit with `ml-platform query-data-platform query.json`. For async submission, poll
+   `get-data-platform-query RUN_ID` until terminal. Read `result.snapshot_id` on success;
+   a successful HTTP poll alone does not establish successful execution.
+7. Read `get-data-platform-snapshot SNAPSHOT_ID --offset 0 --limit 100`. Advance offset by the
+   returned row count until it reaches `captured_rows`. An empty page before that count is an
+   incomplete read, not successful completion. Record run ID,
+   snapshot ID, mapping revision, columns and their order,
+   window, warnings and `completeness`. Partial or truncated results cannot establish complete
+   coverage. An empty result is not evidence that the source has no data outside this window.
+
+Do not resubmit uncertain POSTs automatically. Keep the returned run ID; after a timeout or
+connection failure, inspect actual task state in the data platform before another submission.
+For `SOURCE_SCHEMA_DRIFT`, ask the data owner to repair/revalidate the source mapping; never
+bypass that gate with a physical ReadSpec or SQL.
+
+For training or realtime inference, hand off confirmed outputs to `$feature-management` and its
+Mapping binding workflow. `bind-data-platform-parameters SPEC_JSON` creates/reuses ordinary immutable
+Parameters; exploration alone does not authorize that write. The adapter accepts single published
+structured, unaggregated series, or SQL ReadSpecs, with no inputs/expressions composition. SQL remains
+executed upstream. Mapping output contracts, coverage and causal semantics still require validation;
+SQL-derived values do not prove historical visibility. Duplicate event times and partial snapshots fail.
+
+Use the facade query commands for discovery and inspection. Use the Parameter adapter for dataset
+builds and realtime reads; do not copy snapshot rows into catalogs or commit credentials.
+
+## Coverage, Visibility And Failure Evidence
+
+Start with a short business-relevant window, then expand within the approved query scope when
+checking history coverage. The query facade accepts `max_rows` up to 10000 and snapshot pages up
+to 1000 rows. If a result is partial or truncated, split the time window rather than using the
+captured subset as a complete dataset. Keep the same fixed revision and inspect every window's
+warnings: source SQL may impose its own historical limit even when the snapshot is marked complete.
+Check the requested time grid, first/last timestamps, missingness, duplicates and values; a small
+successful sample does not prove the larger training window is usable.
+
+Query/read timestamps describe when data was retrieved. They do not establish when each source row
+first became available. Record whether the Mapping actually publishes a trustworthy availability
+field, and hand this evidence to `$feature-management`; do not invent it from event time or query
+completion time. Ordinary snapshot exploration can proceed without such a field, while strict
+historical `as_of` correctness remains unproven.
+
+Distinguish release/configuration failures from empty data. A missing CLI command requires checking
+the installed release; a missing route requires checking the deployed API. A `503` with
+`DATA_PLATFORM_NOT_CONFIGURED` means the server lacks integration configuration. Permission or
+schema-drift failures need resolution in the upstream account/Mapping. Timeouts and disconnected
+POSTs leave submission uncertain; inspect run state before retrying. API health alone does not prove
+upstream authentication or dataset-build Worker support. Report the failed checkpoint and target
+without exposing passwords, cookies, CSRF tokens or Secret contents.

package/skills/feature-management/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: feature-management
-description: Manage projects, versioned business feature assets and their lifecycle, dataset rowsets and endpoint eligibility, and causal realtime row or sequence-context reads through the deployed ML Platform API using the ml-platform CLI installed with the same release. Use when a business user asks an agent to manage an empty Project, define or change Parameter source contracts, author/package/register an Operator wheel, define single-column Features, order a FeatureSet, create a DatasetManifest or catalog, inspect references, soft-delete or restore a Parameter/Feature/FeatureSet version, configure training/validation/test rowsets or abnormal-data endpoint exclusion, configure realtime freshness policy, run catalog dry-run/apply, build a dataset, fetch inference data, or inspect an artifact without downloading the platform source repository. Do not use for model training, Operator/Dataset deletion, image builds, or Kubernetes deployment.
+description: Manage projects, versioned business feature assets and their lifecycle, dataset rowsets and endpoint eligibility, and causal realtime row or sequence-context reads through the deployed ML Platform API using the ml-platform CLI installed with the same release. Use when a business user asks an agent to manage an empty Project, define or change Parameter source contracts, bind published external data-platform Mapping outputs to Parameters, author/package/register an Operator wheel, define single-column Features, order a FeatureSet, create a DatasetManifest or catalog, inspect references, soft-delete or restore a Parameter/Feature/FeatureSet version, configure training/validation/test rowsets or abnormal-data endpoint exclusion, configure realtime freshness policy, run catalog dry-run/apply, build a dataset, fetch inference data, or inspect an artifact without downloading the platform source repository. Do not use for model training, Operator/Dataset deletion, image builds, or Kubernetes deployment.
 ---
 
 # Feature Management
@@ -52,7 +52,8 @@ Keep these evidence rules active in every workflow:
 4. Read [references/operator-authoring.md](references/operator-authoring.md) whenever creating or changing Operator code or a wheel.
 5. Read [references/commands.md](references/commands.md) immediately before a platform CLI action: discovery, validation, publication, build, realtime fetch, or artifact download. A local-only draft that is forbidden to call the CLI does not need this reference.
 6. Use [assets/catalog-template](assets/catalog-template) as a copyable starting point for a new end-to-end catalog. Rename every `example_*` identifier and update every referenced path before validation.
-7. Read [references/platform-capability-guide.md](references/platform-capability-guide.md) only when selecting an asset type or explaining a platform capability. Do not load it merely to write a confirmed local draft; it is never evidence for unresolved business semantics.
+7. Read [references/data-platform-mapping.md](references/data-platform-mapping.md) when binding published external data-platform Mapping outputs to Parameters.
+8. Read [references/platform-capability-guide.md](references/platform-capability-guide.md) only when selecting an asset type or explaining a platform capability. Do not load it merely to write a confirmed local draft; it is never evidence for unresolved business semantics.
 
 ## Initialize The Client
 
@@ -82,8 +83,9 @@ Use the global `--profile server` form shown above, before the business
 command. Do not append `--profile` after a business command. Before discovery,
 check the installed release with `ml-platform --help` and the relevant
 subcommand's `--help`; CLI releases may differ in supported flags. In the
-deployed 0.6.x CLI, registry `list-*` commands emit JSON by default and do not
-accept a `--json` flag.
+current CLI release line (`@zhuoyuezs/ml-platform` 0.8.x; confirm with
+`ml-platform version`), registry `list-*` commands emit JSON by default and
+reject an unknown `--json` flag.
 
 If the installed release rejects the global `--profile server` form shown above,
 record the CLI syntax error and inspect `show-config`/subcommand help before
@@ -109,14 +111,18 @@ Confirm:
 - the target project (namespace) for the assets; use the built-in `default` only when the user has no dedicated project. After the target is confirmed, write its explicit `project` field in every Parameter, Operator, Feature, FeatureSet and DatasetManifest draft; never rely on the schema default. Reads and filters use the `--project` command option;
 - business meaning, stable asset names, owner, and intended consumers;
 - Parameter source adapter (`postgresql_direct`, `mysql_direct`,
-  `sqlserver_direct`, or `influxdb_direct`) and credential-free source mapping;
+  `sqlserver_direct`, `influxdb_direct`, or `data_platform_mapping`) and credential-free
+  source mapping; for external Mappings, confirm the published revision and output keys;
 - feature formula, exact Parameter dependencies, windows, inclusion rules, rounding, null behavior, and output dtype;
 - whether an approved Operator already implements the formula;
 - FeatureSet column order;
 - dataset mode, read policy, time range, grid, prediction horizon/cutoff contract, and explicit Parameter outputs;
 - named training/validation/test rowsets, abnormal windows, endpoint-policy scope,
   and the model context lookback used to decide endpoint eligibility;
-- realtime freshness and missing-input policy when serving inference data;
+- realtime freshness and missing-input policy when serving inference data,
+  chosen per model and per Parameter from the data owner's answers on
+  timestamp meaning, arrival schedule and ingestion-time columns (see the
+  realtime policy questions in [references/contracts.md](references/contracts.md));
 - target profile/API and whether the user authorizes publish and/or build actions.
 
 Separate preparation from mutation. Creating files and running local tests or `apply --dry-run` does not authorize publishing a wheel, changing the Registry, or submitting a build Job.
@@ -260,7 +266,13 @@ unit, time/availability semantics and missing policy in the semantic review.
 Never infer a source table, column, measurement, tag, unit or timezone from a
 Parameter name.
 
-Create one stable, versioned source contract for each independently readable value. Keep credentials out of JSON. Use only supported direct adapters (`postgresql_direct`, `mysql_direct`, `sqlserver_direct`, and `influxdb_direct`) and explicit source identity. For PostgreSQL, keep `source.adapter=postgresql_direct` and put the configured alias in `source.connection_id` (for example `postgresql_bf_monitoring`); never put the connection alias in `source.adapter`. The server supplies database DSNs from managed Secrets; never add them to CLI arguments or Catalog JSON. A missing `connection_id` on a historical PostgreSQL Parameter uses the compatibility default `postgresql`. Put source data validity in `quality_rules`; put dataset-specific transformations in preprocess Operators.
+Create one stable, versioned source contract for each independently readable value. Keep credentials out of JSON. Direct database sources use `postgresql_direct`, `mysql_direct`, `sqlserver_direct`, or `influxdb_direct` and explicit source identity. For PostgreSQL, keep `source.adapter=postgresql_direct` and put the configured alias in `source.connection_id` (for example `postgresql_bf_monitoring`); never put the connection alias in `source.adapter`. The server supplies database DSNs from managed Secrets; never add them to CLI arguments or Catalog JSON. A missing `connection_id` on a historical PostgreSQL Parameter uses the compatibility default `postgresql`. Put source data validity in `quality_rules`; put dataset-specific transformations in preprocess Operators.
+
+For an approved external Mapping, use `bind-data-platform-parameters` to create or reuse
+`data_platform_mapping` Parameters and retain the returned Feature input references. Follow
+[the Mapping reference](references/data-platform-mapping.md) for revision pinning, visibility,
+read budgets and Worker acceptance. The Mapping owns its database connection and SQL; do not
+copy them into the Parameter or add a second public binding resource.
 
 When historical and live systems expose the same value, keep one Parameter and add an optional
 `online_source`. Batch/as-of reads always use `source`; realtime causal reads use `online_source`
@@ -380,9 +392,15 @@ for the cutoff formula, half-open overlap boundaries, scope behavior, and
 
 For realtime inference, declare `realtime_fetch` only when its effective policy
 must be part of the contract; leaving it unset preserves existing manifest
-hashes. Keep `allow_missing=false` unless the user explicitly accepts missing
-required inputs. Read [references/contracts.md](references/contracts.md) for the
-freshness, tail-edge, gap-fill, and rowset contracts.
+hashes. The environment defaults are strict and suit short-horizon
+forecasting; for a new model whose Features are served online, choose the
+policy deliberately and declare every field it relies on. Keep
+`allow_missing=false` unless the user explicitly accepts missing required
+inputs. Read [references/contracts.md](references/contracts.md) for the
+freshness, tail-edge, gap-fill, and rowset contracts and the policy method.
+An Operator served online, especially one that aggregates a whole period,
+must follow the Online Period Aggregates method in
+[references/operator-authoring.md](references/operator-authoring.md).
 
 ## Validate Before Publishing
 
@@ -405,7 +423,7 @@ Stop on any error. Do not weaken schema validation, fabricate a missing dependen
 
 Publish only after the user explicitly approves Registry and wheel changes. Use catalog `apply` so publication follows Parameter -> Operator -> Feature -> FeatureSet -> Dataset order.
 
-After publication, resolve every new DatasetManifest and compare its exact Parameter, Operator, Feature, and FeatureSet versions with the pre-publication mapping. Stop if an affected old key or any unexpected version remains. Resolve the manifest before building. Submit a build only when requested. Before choosing partition controls, use the build command's read-only `--plan` capability described in [commands.md](references/commands.md), inspect its estimates and warnings, and prefer platform automatic sizing over a copied daily example. Explicit overrides need source/resource evidence; preserve the submitted Job's concrete plan when resuming. For the current partitioned materialization runtime, use `--max-parallelism 1` unless a separately verified platform capability says otherwise; do not retry or resubmit a failed Job until its error and already-completed partitions are understood. For server builds, wait for the terminal Job state and download the exact artifact by `dataset_id + manifest_hash`.
+After publication, resolve every new DatasetManifest and compare its exact Parameter, Operator, Feature, and FeatureSet versions with the pre-publication mapping. Stop if an affected old key or any unexpected version remains. Resolve the manifest before building. Submit a build only when requested. Before choosing partition controls, use the build command's read-only `--plan` capability described in [commands.md](references/commands.md), inspect its estimates and warnings, and prefer platform automatic sizing over a copied daily example. Explicit overrides need source/resource evidence; preserve the submitted Job's concrete plan when resuming. For the current partitioned materialization runtime, use `--max-parallelism 1` unless a separately verified platform capability says otherwise. If any selected Operator needs the complete requested window (for example full-window interpolation or smoothing), build with `--partition-duration none --max-parallelism 1` and confirm an unpartitioned `--plan` first, because the CLI default is `auto`; do not retry or resubmit a failed Job until its error and already-completed partitions are understood. For server builds, wait for the terminal Job state and download the exact artifact by `dataset_id + manifest_hash`.
 
 Do not delete versioned Registry assets, cancel Jobs, rebuild images, modify Kubernetes, or change service configuration as part of this workflow unless the user separately and explicitly requests that action. An empty Project may be soft-deleted only on an explicit request; rely on the server to reject deletion when resources still exist.
 

package/skills/feature-management/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "特征与实时取数管理"
-  short_description: "管理版本化 Parameter、Operator、Feature、数据集与实时推理取数"
-  default_prompt: "Use $feature-management to manage versioned feature assets, build datasets, or fetch one causal realtime inference row through the platform API."
+  short_description: "绑定外部 Mapping，管理特征、数据集与实时推理取数"
+  default_prompt: "Use $feature-management to bind published external Mapping outputs to Parameters, manage versioned feature assets, build datasets, or fetch causal realtime inference data through the platform API."

package/skills/feature-management/references/commands.md

@@ -51,8 +51,9 @@ ml-platform --profile server list-datasets -q <stable-dataset-id>
 ```
 
 Run each command separately. Check `ml-platform --help` and the specific
-subcommand help before adding flags. In the deployed 0.6.x CLI, `list-*`
-commands print JSON by default and do not accept a `--json` flag.
+subcommand help before adding flags. In the current CLI release line
+(`@zhuoyuezs/ml-platform` 0.8.x; confirm with `ml-platform version`), `list-*`
+commands print JSON by default and reject an unknown `--json` flag.
 If a deployed release rejects the global `--profile server` prefix, treat that
 as CLI compatibility drift: record the syntax error, inspect `show-config` and
 the subcommand help, then retry the same read-only command without the rejected
@@ -107,8 +108,8 @@ empty project. Project commands are server-only and do not modify local storage.
 - `apply --project <project>` scopes the comparison snapshot the catalog is
   validated against (omit to compare against `default`);
 - `list-* --project <project>` filters discovery to that project;
-- `get-dataset-artifact` / `download-dataset-artifact` take the project as their
-  first positional argument (defaults to `default`).
+- `get-dataset-artifact` / `download-dataset-artifact` take `--project <project>`
+  (defaults to `default`).
 
 Before choosing names, discover within the target project: identical names in a
 different project are unrelated and must not be reused across projects to mean the
@@ -247,6 +248,25 @@ sizing is not a resource guarantee. Final merge memory is not bounded by a
 partition's estimated rows. A long prediction horizon is not itself a reason to
 use daily partitions: inspect the reported source read amplification.
 
+Partitioned execution gives each partition only its declared
+`history_requirements` lookback and label overlap. An Operator whose result for
+one row depends on the complete requested window (for example full-window
+interpolation, smoothing, normalization or ranking) is not partition-invariant,
+and no Operator field tells the planner so. Build such a manifest with
+`--partition-duration none --max-parallelism 1`, because both build commands
+default to `auto`. First run the same command with `--plan` and confirm
+`execution_spec.partition_duration` is `null`, `estimated_partition_count` is
+`1`, and the reasons report unpartitioned execution; then submit with the same
+flags. Prefer reformulating the Operator as a causal, bounded-lookback
+calculation when the business rule allows it.
+
+```bash
+ml-platform --profile server \
+  build-registered-dataset <dataset_id> <dataset_version> \
+  --project <project> \
+  --partition-duration none --max-parallelism 1 --plan
+```
+
 `--partition-duration` is an execution control and does not change the
 DatasetManifest hash. The current implementation is sequential, so keep
 `--max-parallelism 1`. Completed partitions are checkpointed and resumed by
@@ -255,8 +275,12 @@ resolved partition duration. Changing it requires a new execution, rather than
 reusing old checkpoint boundaries. Changing partition size does not authorize
 cancelling or resubmitting an already-running Job.
 
-Record the returned `job_id`, `dataset_id`, and resolved `manifest_hash`; for a registered Dataset
-also retain `registered_manifest_hash` for FeatureRetrievalSpec binding. If a separate wait is needed:
+Record the returned `job_id`, `dataset_id`, and the artifact manifest hash; for a registered Dataset
+also retain `registered_manifest_hash` for FeatureRetrievalSpec binding. A succeeded
+`--wait` or `wait-job` result carries `artifact.manifest_hash`, the DatasetArtifact key used by
+`get-dataset-artifact`, `download-dataset-artifact` and training/evaluation dataset bindings. The
+Job's top-level `manifest_hash` identifies the submitted manifest; it is not an artifact key and
+returns `artifact_not_found` there. If a separate wait is needed:
 
 ```bash
 ml-platform --profile server wait-job <job_id>
@@ -294,6 +318,23 @@ write parquet, or publish a DatasetArtifact. Treat HTTP `503` as retryable sourc
 freshness/missing-data failure and HTTP `400` as a contract error that requires a
 manifest or request change.
 
+Before choosing `realtime_fetch` lag budgets or a schedule time, observe how far
+each Parameter's source has arrived:
+
+```bash
+ml-platform --profile server get-parameter-freshness <name> <version> \
+  --project <project> [--lookback 7d] [--end 2026-10-28T00:30:00+08:00] [--online]
+```
+
+It reads `[end - lookback, end]` (end inclusive, `lookback` at most `31d`) through
+the Parameter's source adapter without cache, build, or artifact, and returns
+`latest_event_time`, `lag_hours` (the value a realtime read compares with its lag
+budget at that cutoff), `latest_value_event_time`, `missing_trailing_points`,
+`available_at_watermark`/`ingested_at_watermark` when the Parameter declares
+availability, and `source_field_checks`. `--online` reads `online_source` when
+declared. Run it at the intended schedule time on several days; one reading does
+not establish a publication pattern.
+
 For a deployment-bound row or full sequence context, prepare one JSON object
 containing `feature_retrieval`, `cutoff_time`, and `entity_keys`, then run:
 
@@ -359,7 +400,9 @@ ml-platform --profile server \
 ```
 
 `--project` defaults to `default`; pass the owning project when the dataset lives
-elsewhere, or the lookup returns `artifact_not_found`.
+elsewhere, or the lookup returns `artifact_not_found`. `<manifest_hash>` is the
+artifact manifest hash (`list-dataset-artifacts`, or `artifact.manifest_hash`
+from a succeeded build wait), not the registered manifest hash.
 
 The download command writes progress to stderr after response headers arrive.
 Interpret a timeout with no progress line as a request/header or first-byte
@@ -393,3 +436,17 @@ Use a structured Parquet reader for schema, row count, column order, and missing
 - Do not claim a successful publication from `--dry-run` output.
 - Do not claim Chronon compile/backfill from a successful first-phase Operator build.
 - Do not use `build-dataset` to serve a single realtime inference cutoff.
+
+## Bind Published Data Platform Outputs
+
+When source semantics and publication are confirmed, use the existing external Mapping directly.
+This creates ordinary immutable Parameters; it does not publish Mapping assets or execute a dataset
+build:
+
+```bash
+ml-platform bind-data-platform-parameters --help
+ml-platform bind-data-platform-parameters binding.json
+```
+
+See [data-platform-mapping.md](data-platform-mapping.md) for the request, returned Feature input
+references, visibility requirements, Worker requirements and acceptance checks.

package/skills/feature-management/references/contracts.md

@@ -139,6 +139,17 @@ supported field.
 当前严格 `as_of` 只接受 `exact` 或有合同支撑的 `upper_bound`；修改这些语义需要发布
 新的 Parameter 业务版本。
 
+`time_semantics.ingested_at_field` 和 `event_time_field` 只是标签：读取时不会查询它们，
+除非 `availability.field` 指向同一列。没有 `availability` 时，读取既没有
+`ingested_at`/`available_at` 证据，严格可见性读取（`as_of` 回放、
+`availability_boundary=cutoff_time`）也会拒绝该 Parameter。只把源表真实存在的入库列写进
+`availability`；源表没有入库时间时，保持 `availability` 为空或使用经数据 owner 确认的
+`fixed_delay`，不要声明不存在的列。发布新 Parameter 时，若 `source.connection_id` 是平台
+托管连接，平台会检查实际查询的列（`time_column`、`value_column`、metric/unit 列、
+`filters` 键、`availability.field`、`availability.revision_id_field`）：缺列直接拒绝
+（`source_fields_missing`），不存在的 `ingested_at_field`/`event_time_field` 只作为警告
+返回在 `source_field_checks` 中。已发布版本可用 `get-parameter-freshness` 查看同一检查。
+
 未来预报、排程、日历等 known-future 数据允许 `available_at < event_time`。这不是时钟错误：
 `event_time` 是未来目标时刻，`available_at` 是该修订何时已可读取。模型
 `feature_lookup` 会读取 cutoff 后的未来事件窗口，但仍严格过滤
@@ -380,8 +391,14 @@ Current schema rules:
 - Use `parameters` only to request explicit Parameter output columns.
 - Declare one dataset-wide `prediction` contract for forecast Features. `horizon`
   accepts non-negative whole-minute durations such as `0min`, `10min`, or `2h`.
-  The platform always derives `cutoff_times = target_times - horizon` and records
-  that derivation in resolved lineage; it is not a user-configurable policy.
+  For Operator inputs, each materialized row's `event_time` is its causal
+  feature cutoff (`context.cutoff_times` equals `context.target_times`), and the
+  horizon sets `label_time = event_time + horizon` without moving those reads.
+  `endpoint_policy` eligibility is separate and uses the `cutoff = t - h` rule
+  documented in its section. The platform records this in resolved
+  lineage (`prediction_time_semantics: event_time_is_causal_cutoff`,
+  `label_time_derivation: prediction_time_plus_horizon`); it is not a
+  user-configurable policy.
 
 ### Optional sections
 
@@ -442,11 +459,14 @@ attempts, rows, and elapsed time land in
 ```json
 "realtime_fetch": {
   "max_workers": 4,
+  "deadline_seconds": 30.0,
+  "chunk_days": null,
   "min_lookback": "1h",
   "validate_freshness": true,
   "max_source_lag_hours": 3.0,
   "max_source_lag_hours_by_parameter": {},
   "allow_missing": false,
+  "allow_missing_by_parameter": {},
   "allow_tail_edge_missing": true,
   "max_tail_edge_missing": 1,
   "tail_edge_missing_max_age_minutes": 65.0,
@@ -467,11 +487,109 @@ offline `source` instead.
 
 Keep the default strict: `allow_missing=false`. Tail-edge and bounded internal
 gap-fill tolerance may return a row, but the response must mark
-`freshness.degraded=true` and include validation evidence. A freshness or
+`freshness.degraded=true` and include validation evidence.
+
+Per-Parameter keys in `max_source_lag_hours_by_parameter` and
+`allow_missing_by_parameter` are full Parameter keys (`project/name:version`);
+other spellings are ignored with a validation warning. `allow_missing_by_parameter`
+overrides `allow_missing` for one Parameter in either direction and is omitted
+from the manifest while empty, so adding the field does not change existing
+hashes. A tolerated Parameter still marks the read degraded, and a deployment
+with `freshness_policy=strict` rejects degraded reads.
+
+The tail-edge and internal-gap-fill allowances are absolute lag ceilings measured
+from the cutoff, not additions to the lag budget: a source past its budget is
+tolerated when its lag is at most `max(tail_edge_missing_max_age_minutes,
+max_tail_edge_missing * expected_frequency)` or at most
+`internal_gap_fill_max_age_hours`. With the defaults (5h gap-fill ceiling) a
+Parameter budget below 5h therefore degrades rather than fails; set
+`allow_internal_gap_fill=false` (per manifest) when a budget must be hard.
+Neither allowance changes values: realtime rows use the same alignment and
+Operator code as a batch build, so these settings cannot create
+training/serving skew. Choose budgets from observed arrival with
+`get-parameter-freshness`. A freshness or
 missing-data condition that may recover returns HTTP `503`; an invalid contract
 returns HTTP `400`. Declaring `realtime_fetch` changes the manifest hash; leaving
 it unset preserves older contract hashes and uses environment fallbacks.
 
+`deadline_seconds` bounds the whole read. `chunk_days` pages long source reads
+and falls back to `source_read.chunk_days` when unset.
+
+#### Choosing a realtime fetch policy
+
+The values above are also the environment defaults (`DATA_PLATFORM_REALTIME_*`).
+They are strict and suit short-horizon forecasting on minute-level data. They
+are a process fallback, not a decision for a particular model. Choose the
+policy for each model from its data and its use, and per Parameter where
+sources differ.
+
+How the checks work:
+
+- For each Parameter, the platform takes the newest source row at or before
+  the cutoff (the watermark) and computes `lag = cutoff - watermark`.
+- The read is normal when the lag is within `max_source_lag_hours`, or within
+  the Parameter's entry in `max_source_lag_hours_by_parameter` (keyed
+  `project/name:version`).
+- A larger lag is still accepted, marked `freshness.degraded=true`, when
+  `allow_tail_edge_missing` is on and the lag is within the larger of
+  `tail_edge_missing_max_age_minutes` and `max_tail_edge_missing` times the
+  Parameter's `expected_frequency`, or when `allow_internal_gap_fill` is on and
+  the lag is within `internal_gap_fill_max_age_hours`. With the defaults, a
+  source up to 5 hours behind is accepted as degraded. Set
+  `allow_internal_gap_fill: false` when the lag budget must be a hard limit.
+- A Parameter with no rows in its window fails unless `allow_missing` is true.
+  `allow_missing` applies to every Parameter and also accepts any stale
+  source; there is no per-Parameter switch. Do not turn it on to tolerate one
+  optional input.
+- None of these settings changes values. Holes inside the lookback reach the
+  Operator unchanged, so the Operator's frozen missing-data rule
+  ([operator-authoring.md](operator-authoring.md), Online Period Aggregates)
+  is what keeps batch builds and realtime reads equal.
+
+Decide with the user:
+
+- Whether a late or missing source should fail visibly (HTTP `503`, rerun after
+  the data recovers) or return a degraded row. A long aggregate (for example a
+  daily mean) may barely change when one point is late, but a period that is
+  missing its last hours because a source stalled usually should fail.
+- When the read runs relative to each source's arrival. The lag budget must
+  cover the latest normal arrival: schedule the read after a batch-loaded
+  source is complete, or give that Parameter a larger budget.
+- Whether the read finishes within `deadline_seconds`. A long lookback over
+  several Parameters may need more than 30 seconds; measure it with
+  `fetch-inference-data` at a historical cutoff before choosing.
+
+Record the decision in the inference DatasetManifest. Declare every field the
+model relies on, even when it equals the current default: resolution is per
+field, and an omitted field takes the process environment value, which can
+differ between deployments. Note the reason for each value in the catalog
+review. Changing the policy needs a new inference manifest and package
+version, but not retraining. In each response, `policy.resolved_from` shows
+whether a field came from `manifest` or `environment`; check that nothing you
+meant to declare came from `environment`.
+
+The `fetch-inference-data` options `--max-source-lag-hours`,
+`--no-validate-freshness` and `--allow-missing` replace the manifest's whole
+`realtime_fetch` section for that request; fields you do not pass fall back to
+the environment, not to the manifest. Use them for diagnosis only.
+
+Ask the data owner, and do not guess:
+
+1. What each source timestamp means: interval start, interval end, or the
+   measurement instant, and in which timezone. This decides period boundaries
+   and the expected lag (an hourly record stamped at the hour start has a
+   watermark one hour before a midnight cutoff even when the day is complete).
+2. How data arrives: streaming or batch loads, when the previous period is
+   complete on a normal day, and how late it can be.
+3. Whether late corrections or backfills are written after first arrival;
+   these make a rerun at the same cutoff return a different row.
+4. Whether the table has a trustworthy ingestion-time column. Confirm the
+   column exists (for example with the `data-source-exploration` Skill) before
+   declaring `time_semantics.ingested_at_field` or an availability strategy
+   that reads it; never copy that field from a template.
+5. Which inputs may legitimately be absent (for example one faulty sensor) and
+   which must never be (for example a state indicator used by a business rule).
+
 `abnormal_windows` excludes target windows whose data must not be trusted. The
 policy and the lookback apply to the whole window set, not to individual
 windows: a dataset is either a filtered rowset or a marked full rowset.
@@ -494,9 +612,9 @@ windows: a dataset is either a filtered rowset or a marked full rowset.
 | `mark_only` | Drop nothing; record what is affected in lineage and validation. |
 
 How far past a window to drop is derived from the widest resolved Operator history
-requirement. Cutoff-relative history contributes `prediction.horizon + lookback`;
-target-time-relative history contributes its declared lookback directly. The
-maximum is used for source fetch and abnormal-window propagation. Do not restate
+requirement: the larger of the cutoff-anchored and target-time-anchored
+lookbacks (the horizon is not added, because the Operator cutoff is the row's
+`event_time`). The maximum is used for source fetch and abnormal-window propagation. Do not restate
 that number unless you want it asserted: an optional `input_lookback_hours` is
 checked against the derived value and a mismatch is rejected. Note that a window
 near the end of the target range can have its lookback clipped by the range end,

package/skills/feature-management/references/data-platform-mapping.md

@@ -0,0 +1,104 @@
+# Bind And Build From External Data Platform Mappings
+
+Use after the user has authorized publishing confirmed source contracts. Discovery remains in
+`data-source-exploration`; this workflow writes existing Parameter resources through the modeling API.
+Verify the installed CLI and configured API using the main Skill, then inspect command help.
+
+1. Discover a ready published Mapping with `list-data-platform-query-candidates`, then inspect
+   its pinned `active_revision.mapping_revision_id` with
+   `get-data-platform-mapping-revision ID REVISION`. Discovery returns the complete catalog
+   authorized for the integration account; it does not prove data coverage or reserve query access.
+   For older upstreams, follow the discovery reference's paged Mapping workflow.
+2. Confirm numeric value output keys, the unique event-time output, units, source timezone,
+   frequency, precision, entity scope and historical coverage. Do not treat Mapping readiness or a
+   complete snapshot as proof of causal correctness. SQL may already compute values or limit history.
+3. Write a binding JSON outside the platform repository:
+
+   ```json
+   {
+     "project": "default",
+     "mapping_id": "11111111-1111-4111-8111-111111111111",
+     "mapping_revision_id": "22222222-2222-4222-8222-222222222222",
+     "output_keys": ["out_value"],
+     "expected_frequency": "10min",
+     "event_time_precision": "s"
+   }
+   ```
+
+   Substitute confirmed IDs and settings. `output_keys` contains value outputs only. Source timezone
+   comes from the ReadSpec; if missing, supply `time_semantics`. Optional `time_semantics`,
+   `quality_rules`, `owner`, `partition_seconds`, `max_rows`, `timeout_seconds` follow the existing
+   contracts. Never infer availability from request time. Strict `as_of` requires the Mapping to
+   expose the trustworthy availability field declared in `time_semantics.availability`.
+4. Run `ml-platform bind-data-platform-parameters binding.json`. Preserve each returned `reference`
+   as a Feature input (`project`, `parameter`, `version`). The response also contains the full
+   Parameter. Do not manually recreate its source, SQL, credentials or version.
+5. Continue normal Operator → Feature → FeatureSet → DatasetManifest publication. Operator history
+   requirements govern source lookback; binding does not add a shared fixed window. Build and realtime
+   reads use `source_mode=direct`.
+6. For a DatasetManifest build, distinguish CLI submission from Worker execution. Inspect `get-job`
+   until a terminal status, then retrieve the DatasetArtifact. Confirm row and column counts against
+   the requested grid, Parquet column order, missing values, time coverage, and source-read evidence
+   (`mapping_revision_id`, `run_id`, `snapshot_id`, plan/window and actual read times) before handing
+   the dataset to model training. A submitted or running Kubernetes Job is not build evidence.
+
+Repeated identical binding requests reuse deterministic Parameter identities. Changed Mapping revision
+or effective configuration creates a new immutable version. Existing manually named Parameters are not
+searched or merged. Multiple outputs are validated before writing, but each Parameter persists
+independently; there is no cross-resource transaction. After an uncertain response, repeating the same
+binding safely converges to the same Parameter contents. This does not mean uncertain upstream query
+POSTs may be retried: inspect those runs separately.
+
+Supported sources are single structured, unaggregated series, or SQL ReadSpecs without composed
+inputs/expressions. SQL executes in the data platform. Ambiguous duplicate event times, partial snapshots and
+revision/window mismatches fail. The adapter preserves source warnings, run, snapshot, Mapping
+revision, plan, window and actual read times. Fixed definitions do not freeze mutable upstream rows
+or make expiring snapshots permanent.
+
+`read_started_at`/`read_finished_at` record when the adapter performed this read. They are audit
+metadata, not per-row source availability evidence. A Mapping without a trustworthy published
+availability output can be used for ordinary snapshot reads, but it cannot establish strict
+historical `as_of` visibility. Do not substitute request/read time, event time, or `ingested_at`
+for `available_at` unless the Parameter contract explicitly identifies that source field as the
+trusted availability boundary. A complete snapshot proves transfer completeness, not historical
+visibility or business coverage.
+
+Mapping reads bypass ITSMP source cache so each read retains its upstream snapshot lineage.
+`partition_seconds` defaults to 86400 (range 1–86400); each partition's `max_rows` defaults to and
+cannot exceed 10000. One source read accepts at most 1024 partitions. `timeout_seconds` defaults
+to 180 and accepts at most 600; it is checked between HTTP calls, whose own client timeout also
+applies. These are source-adapter budgets, separate from Dataset build execution partitions.
+If coverage exceeds a budget, choose a justified source partition or build window; never accept
+truncation or remove required Operator history to make a read succeed. The adapter converts
+upstream half-open windows into the closed source observation boundary, including its end instant;
+the Dataset target grid remains half-open.
+
+## Worker Runtime Requirements And Failures
+
+The business CLI does not configure Kubernetes, publish images or mount credentials. The platform
+operator must configure the modeling API with the upstream integration Secret and configure the
+dataset-build Job template to mount that Secret read-only and set
+`DATA_PLATFORM_UPSTREAM_CONFIG_FILE`. Build Workers must receive the updated dataset-build image
+containing the `data_platform_mapping` source adapter; changing only the API image is insufficient.
+The API and Worker must use compatible platform code, but training and serving images do not need
+this data-source adapter unless they themselves read Mapping data.
+
+After catalog publication and build submission:
+
+- On terminal success, inspect and download the artifact before reporting data success.
+- If the job is accepted but remains queued, check Worker scheduling, quotas and active Jobs with
+  the platform operator; do not resubmit while its durable state is uncertain.
+- If the Worker reports `unsupported source adapter` for `data_platform_mapping`, verify the
+  deployed dataset-build image includes the adapter. A healthy API or successful catalog `apply`
+  does not prove the Worker image was updated.
+- If the Worker cannot load `DATA_PLATFORM_UPSTREAM_CONFIG_FILE`, or upstream authentication/network
+  fails, have the platform operator verify the Secret key/path, read-only mount, Worker namespace,
+  egress and integration-account permissions. Never copy credentials into the Parameter, Manifest,
+  CLI arguments or logs.
+- For an uncertain query POST, inspect the upstream run before retrying. For a failed build, inspect
+  the durable Job and Worker logs, correct the identified cause, and submit a new attempt only after
+  confirming the previous attempt is terminal; do not blindly repeat the same request.
+
+The Skill can guide asset authoring and API/CLI evidence collection, but it cannot diagnose container
+exit causes from `get-job` alone or safely change the cluster. Keep Kubernetes events, pod reason,
+container exit code and log tail as separate operational evidence.