x4d-devkit 0.17.0__tar.gz → 0.22.0rc2__tar.gz

{x4d_devkit-0.17.0 → x4d_devkit-0.22.0rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: x4d-devkit
-Version: 0.17.0
+Version: 0.22.0rc2
 Summary: Official X-4D external SDK and CLI for datasets, platform APIs, training bridges, and automation
 Author: windzu
 License-Expression: Apache-2.0
@@ -23,7 +23,7 @@ Requires-Dist: numpy>=1.24
 Requires-Dist: httpx>=0.27
 Requires-Dist: rfc8785==0.1.4
 Provides-Extra: converters
-Requires-Dist: nuscenes-devkit>=1.1.0; extra == "converters"
+Requires-Dist: nuscenes-devkit>=1.2.0; extra == "converters"
 Provides-Extra: inference
 Requires-Dist: pydantic>=2.0; extra == "inference"
 Requires-Dist: fastapi>=0.110; extra == "inference"
@@ -31,6 +31,14 @@ Requires-Dist: uvicorn>=0.27; extra == "inference"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff>=0.9; extra == "dev"
+Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.27; extra == "docs"
+Provides-Extra: release
+Requires-Dist: build>=1.2; extra == "release"
+Requires-Dist: twine>=6.0; extra == "release"
 Dynamic: license-file
 
 # x4d-devkit
@@ -52,19 +60,20 @@ modules or mutate DB/MinIO internals.
 Design decisions for this boundary are tracked in GitHub issues, starting with:
 
 - [#42](https://github.com/windzu/x4d-devkit/issues/42) - official external SDK and CLI boundary
-- [#43](https://github.com/windzu/x4d-devkit/issues/43) - private source install in X-4D runtime
+- [#43](https://github.com/windzu/x4d-devkit/issues/43) - historical private-source runtime integration
 - [#44](https://github.com/windzu/x4d-devkit/issues/44) - capability registry and CLI sync
 
 ## Installation
 
 ```bash
-pip install x4d-devkit
+pip install x4d-devkit==0.22.0rc2
 ```
 
-X-4D platform runtime installs this package from the checked-out private source
-repo/submodule, not from an unconstrained package-index version. External
-environments may install from a pinned package release or directly from GitHub,
-depending on deployment policy.
+Devkit is released independently as an immutable wheel. X-4D, XPoints, and
+other consumers must install one exact package version during image builds.
+Runtime source copies, submodule installs, editable installs, VCS `main`
+installs, and fallback dependency paths are not supported deployment modes.
+Use a published prerelease for coordinated cross-repository development.
 
 Optional heavy dependencies:
 
@@ -77,6 +86,8 @@ pip install x4d-devkit[inference]
 ```
 
 The platform API client and `x4d-devkit` CLI are base package features.
+See [the release process](docs/release-process.md) for the immutable artifact
+and consumer cutover rules.
 
 ## Scope
 
@@ -88,24 +99,27 @@ platform-to-platform data flow. See
 [docs/current-state-identity.md](docs/current-state-identity.md) for the
 generic identity model, and
 [docs/training-current-state-data.md](docs/training-current-state-data.md) for
-diff/sync, work-session materialization, training manifests, and conditional
-annotation writeback flows built on that model.
+work-session materialization and conditional annotation writeback. Label-schema
+bodies are resolved from X-4D and passed explicitly at every clip loading and
+validation boundary.
 Shared validation rules for platform submit, X-Points preflight, local clip
 validation, and training consumers are documented in
 [docs/shared-contract-validation.md](docs/shared-contract-validation.md).
+Portable export producers and consumers share the exact `0.11` manifest
+parser documented in
+[docs/export-bundle-contract.md](docs/export-bundle-contract.md).
 `examples/current_state_identity_consumers.py` provides an offline downstream
-fixture for OpenPCDet-style, mmdetection3d-style, and annotation/review
-consumers.
+fixture for read-only, read-write, and materialization consumers.
 
 Included:
 
 - Local X-4D dataset loading, validation, transforms, annotations, calibration,
-  ego pose, manifests, converters, and evaluation helpers.
+  ego pose, converters, and evaluation helpers.
 - Platform API client and public `x4d-devkit` CLI commands for external
   workflows such as clip listing/download, screening preview, model/checkpoint
   registration, status queries, and capability discovery.
-- Training bridge workflows such as multi-project manifests, archive download,
-  split validation, and dataloader-friendly metadata.
+- Strict validation of `x4d-training-identity-v1` manifests.
+- Strict producer-option and manifest parsing for export bundle `0.11`.
 - External inference service wrappers for third-party model-serving projects.
 
 Excluded:
@@ -166,93 +180,27 @@ complete clip payload including `sweeps/`. If the requested profile has not been
 built, is stale, or failed to build, the devkit reports that profile-specific
 reason instead of falling back to another tar.
 
-### Create a multi-project training manifest
+### Validate a training identity manifest
 
-Training projects can combine multiple X-4D projects by downloading standard
-clip archives and generating one local manifest. The downstream training
-framework consumes only the manifest and local clip files; it does not need
-database, MinIO, or platform-private file paths.
+The `manifest` command has one contract and one operation. It validates an
+exact `x4d-training-identity-v1` record; it does not create datasets, download
+clips, or translate earlier manifest shapes.
 
 ```bash
-x4d-devkit manifest create \
-  --projects hg1.0 hg1.5 \
-  --data-root /data/x4d/cache/hg_mix \
-  --output /data/x4d/cache/hg_mix/manifest.json \
-  --download-missing \
-  --archive-profile keyframes_only \
-  --split-policy by_clip \
-  --val-ratio 0.2 \
-  --seed 42
-
 x4d-devkit manifest validate \
-  --manifest /data/x4d/cache/hg_mix/manifest.json \
-  --data-root /data/x4d/cache/hg_mix
-```
-
-Internal current-state consumers can snapshot remote clip identity before
-reviewing diffs or syncing local caches:
-
-```bash
-x4d-devkit dataset inspect \
-  --project-name hg1.5 \
-  --include-asset-revision \
-  --out remote_identity.json
-```
-
-Training runs can validate their consumed-data identity record:
-
-```bash
-x4d-devkit manifest identity-validate --manifest training_identity.json
-```
-
-Dataset update planning can also compare consumer-owned training identity
-fields, such as split membership, class-mapping fingerprint, data policy, or a
-generic consumer identity fingerprint:
-
-```bash
-x4d-devkit dataset diff \
-  --local-manifest current_training_identity.json \
-  --previous-training-identity previous_training_identity.json \
-  --out update_plan.json
-```
-
-`manifest validate` prints bounded warning output by default. It reports
-warning totals grouped by type, category, and project/category, then shows a
-small set of examples. Use `--max-warnings`, `--all-warnings`, or
-`--warnings-output warnings.jsonl` when full warning details are needed.
-
-Downloaded clips are stored under
-`projects/<project-name>/clips/<clip-id>/`. Manifest file paths are relative to
-`--data-root`, so the dataset can be moved as a directory.
-
-The platform is expected to assign globally unique `clip_id` values. The devkit
-validator checks duplicate clip identities and split leakage, but it does not
-perform expensive semantic near-duplicate detection across projects.
-
-```python
-from x4d_devkit import X4DClient
-
-client = X4DClient.from_config()
-manifest = client.create_training_manifest(
-    projects=["hg1.0", "hg1.5"],
-    data_root="/data/x4d/cache/hg_mix",
-    output="/data/x4d/cache/hg_mix/manifest.json",
-    download_missing=True,
-    split_policy="by_clip",
-    val_ratio=0.2,
-    seed=42,
-)
+  --manifest training_identity.json \
+  --format json
 ```
 
-The detection manifest keeps boxes in their native X-4D annotation frame
-(`meta.sensors[meta.annotation_source_channel].frame_id`). It does not align
-poses across clips; each clip's `clip_world` remains local to that clip.
+Unsupported versions, removed aliases, and malformed clip/schema identities
+fail validation. Producers own manifest construction and must emit the current
+contract directly.
 
 ### Consume the project label schema
 
-X-4D is the source of truth for label schemas. External training and inference
-projects should read the effective project schema from the platform instead of
-maintaining their own category list.
+X-4D is the only source of truth for label-schema bodies. Consumers resolve the
+effective project record from the platform instead of maintaining their own
+category list or copying the record into each clip.
 
 ```bash
 x4d-devkit label-schemas project-get --project-id 1 --format json
@@ -269,11 +217,13 @@ from x4d_devkit.label_schema import compare_project_label_schema_snapshot
 client = X4DClient.from_config()
 project = client.projects.resolve(project_name="nuscenes-mini")
 project_schema = client.label_schemas.resolve_project(project_name="nuscenes-mini")
+project_schema_view = client.label_schemas.get_project_view(project["id"])
 label_schema = project_schema.label_schema
 
 print(project["id"], project["name"])
 print(project_schema.binding.to_dict())
 print(sorted(label_schema.class_ids))
+print([category["id"] for category in project_schema_view.to_dict()["categories"]])
 
 label_schema.validate_category("tanker_truck")
 
@@ -283,40 +233,35 @@ comparison = compare_project_label_schema_snapshot(stored_snapshot, current)
 assert comparison.status == "same"
 ```
 
-The registry schema is the fine-grained annotation contract. Training code may
-still define experiment-specific class mappings, but those mappings should be
-validated against the project schema before a job starts.
-
-Training manifests keep the immutable project contract separate from the
-observed/configured class catalog:
+The native `0.11` clip stores only the immutable identity in
+`meta.label_schema`. It must not contain `label_schema.json`. The complete
+registry record is explicit runtime context for `ClipLoader`, `validate_clip`,
+conversion, and work-session parsing; its identity must exactly match
+`meta.label_schema`.
 
-```text
-label_schema.projects.<project>.contract       # binding + full registry record
-label_schema.projects.<project>.class_catalog  # training/observed class catalog
-```
+Training code may still define experiment-specific class mappings, but those
+mappings must be validated against the resolved project record before use.
 
-Consumers must parse `contract` with
-`parse_project_label_schema_snapshot()`. `class_catalog` is
-non-authoritative observation metadata; there are no per-project
-`classes`, `source`, or `schema_version` aliases.
+### Inspect or diff current state
 
-This strict parser/view contract is introduced in x4d-devkit `0.17.0`.
-Consumers tracking `main` should still record `get_build_info()["source"]` in
-training identity so an untagged development run remains reproducible by exact
-commit. Deployments that copy a source tree without its Git directory must set
-`X4D_DEVKIT_COMMIT` to the copied 40- or 64-hex commit; `get_build_info()` records
-that value with `commit_source=environment`.
+Dataset identity clients, local manifests, portable cache identities, remote
+identity manifests, and update plans use one schema field only:
+`label_schema`, containing the complete
+`x4d-label-schema-identity-v1` object. `label_schema_fingerprint` and
+`clip_label_schema` are not aliases and are rejected. A ready platform record
+without the complete identity is also rejected.
 
-Project resolution by name/key is exact and handles platform pagination
-internally; ambiguous or missing project references raise structured resolver
-errors instead of requiring callers to scan project lists themselves.
+`x4d-devkit dataset inspect` writes `x4d-remote-identity-v2`, and
+`x4d-devkit dataset diff` writes `x4d-update-plan-v2`. These artifacts contain
+normalized clip records once; they do not embed a duplicate raw platform
+response. V1 artifacts are rejected rather than migrated.
 
 ### Load a clip
 
 ```python
 from x4d_devkit import ClipLoader
 
-loader = ClipLoader("/path/to/clip")
+loader = ClipLoader("/path/to/clip", label_schema=label_schema)
 print(loader.meta)
 
 for sample in loader.samples:
@@ -340,7 +285,7 @@ materialize_work_session_clip(
     manifest,
     output_dir="/data/x4d_cache/clips/clip-id",
     client=client,
-    asset_policy="keyframes_only",
+    materialization_policy="keyframes_only",
 )
 
 loader = ClipLoader.from_work_session(manifest, api_url=client.api_url)
@@ -359,20 +304,21 @@ except WorkSessionConflictError as exc:
     raise
 ```
 
-Materialization preserves both schema facts atomically: the clip's recorded
-identity remains in `meta.label_schema`, while the effective full registry
-record is written to `label_schema.json`. Full and annotation-only refreshes
-write the same schema snapshot, so local validation does not require a live
-registry lookup.
+The work-session envelope may carry the complete registry record so
+`ClipLoader.from_work_session()` can validate without another request. That
+record is transient request context, not a second persisted schema source.
+Materialization writes the native tables with the identity in
+`meta.label_schema` and never writes `label_schema.json`. Loading or validating
+the materialized directory later therefore requires the caller to resolve and
+pass the complete registry record explicitly.
 
-A ready work session is accepted only when `label_schema_fingerprint`, the full
-registry record, `clip_label_schema`, and `meta.label_schema` declare the same
-identity. A not-ready session may expose a stale clip identity alongside the
-current project schema so migration tooling can report the mismatch, but it
-cannot be materialized.
+A ready work session is accepted only when its full registry record,
+`clip_label_schema`, and the resulting native `meta.label_schema` declare the
+same identity. A mismatch fails closed; there is no legacy work-session shape
+or schema-file fallback.
 
 Submit is conditional on the manifest's `source_revision`,
-`annotation_revision`, and `label_schema_fingerprint`. If the platform state
+`annotation_revision`, and `base_label_schema_fingerprint`. If the platform state
 changed after the session was opened, the client raises
 `WorkSessionConflictError` with machine-readable `conflicts`,
 `base_identities`, and `current_identities`.
@@ -405,7 +351,7 @@ from x4d_devkit import COORDINATE_CONTRACT_VERSION, get_training_coordinate_cont
 from x4d_devkit import ClipLoader
 from x4d_devkit.core.loader import CLIP_WORLD_FRAME_ID
 
-loader = ClipLoader("/path/to/clip")
+loader = ClipLoader("/path/to/clip", label_schema=label_schema)
 sd = loader.sample_data_for_channel("LIDAR_TOP")[0]
 
 # Load point cloud in different frames
@@ -427,16 +373,20 @@ pts_world = T.apply(pts_sensor[:, :3])  # or use T.as_matrix for 4x4
 ### Validate a clip
 
 ```bash
-x4d-devkit validate /path/to/clip
+x4d-devkit validate /path/to/clip --label-schema project_schema_record.json
 ```
 
 ```python
 from x4d_devkit import validate_clip
 
-report = validate_clip("/path/to/clip")
+report = validate_clip("/path/to/clip", label_schema=label_schema)
 print(report)
 ```
 
+For a remote clip, call `validate_clip_payload(..., asset_sizes=...)` with a
+mapping of clip-relative object paths to byte sizes. This applies the same
+asset existence and point-cloud layout checks without downloading the assets.
+
 ### Serve an external inference backend
 
 Third-party model projects should use `x4d-devkit[inference]` instead of
@@ -464,7 +414,11 @@ class MyDetector(DetectionModel):
         ]
 
 
-InferenceService(service_name="my-detector", models=[MyDetector()]).run(port=9000)
+InferenceService(
+    service_name="my-detector",
+    models=[MyDetector()],
+    clip_resolver=resolve_clip,
+).run(port=9000)
 ```
 
 See [docs/external-inference.md](docs/external-inference.md) and
@@ -519,6 +473,20 @@ converter = NuScenesConverter("/path/to/nuscenes")
 converter.convert_scene("scene-0001", output_dir="/path/to/output")
 ```
 
+## Development
+
+Development and verification run in the repository-owned Docker environment.
+A host Python installation is not part of the project workflow.
+
+```bash
+make verify
+```
+
+Use `make test`, `make lint`, `make docs`, `make package`, or `make shell` for
+individual tasks. Dependency changes are resolved and locked with `make lock`.
+See [the development environment guide](docs/development.md) for the ownership
+boundary and complete workflow.
+
 ## Modules
 
 | Module | Description |
@@ -526,7 +494,7 @@ converter.convert_scene("scene-0001", output_dir="/path/to/output")
 | `core` | Data models, token generation, coordinate transforms, clip loader |
 | `eval` | Detection evaluation (mAP, TP metrics, NDS) |
 | `converters` | Format converters (NuScenes → X4D) |
-| `manifest` | Training manifest creation, validation, and clip archive download |
+| `training_identity` | Strict `x4d-training-identity-v1` parsing and validation |
 | `validation` | Clip structure and data validation |
 | `client` | X-4D platform API client |