x4d-devkit 0.14.2__tar.gz → 0.16.0__tar.gz

x4d_devkit-0.16.0/PKG-INFO

@@ -0,0 +1,534 @@
+Metadata-Version: 2.4
+Name: x4d-devkit
+Version: 0.16.0
+Summary: Official X-4D external SDK and CLI for datasets, platform APIs, training bridges, and automation
+Author: windzu
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/windzu/x4d-devkit
+Project-URL: Repository, https://github.com/windzu/x4d-devkit
+Project-URL: Issues, https://github.com/windzu/x4d-devkit/issues
+Keywords: autonomous-driving,dataset,evaluation,annotation,lidar,camera,cli,sdk
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: httpx>=0.27
+Provides-Extra: converters
+Requires-Dist: nuscenes-devkit>=1.1.0; extra == "converters"
+Provides-Extra: inference
+Requires-Dist: pydantic>=2.0; extra == "inference"
+Requires-Dist: fastapi>=0.110; extra == "inference"
+Requires-Dist: uvicorn>=0.27; extra == "inference"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Dynamic: license-file
+
+# x4d-devkit
+
+Canonical contract library and `x4d-devkit` CLI for consuming X-4D data and
+public platform capabilities.
+
+`x4d-devkit` owns X-4D data semantics, identity, coordinate/schema contracts,
+platform IO adapters, and official evaluation primitives. It is the stable
+interpretation layer between X-4D platform state and consumers such as training
+platforms, X-Points, external inference services, evaluation scripts, and
+automation.
+
+It is not a downstream training, annotation-tool UI, experiment, deployment, or
+framework-specific cache-policy package. It may call public platform APIs or
+read standard X-4D files, but it must not import backend-private `app.*`
+modules or mutate DB/MinIO internals.
+
+Design decisions for this boundary are tracked in GitHub issues, starting with:
+
+- [#42](https://github.com/windzu/x4d-devkit/issues/42) - official external SDK and CLI boundary
+- [#43](https://github.com/windzu/x4d-devkit/issues/43) - private source install in X-4D runtime
+- [#44](https://github.com/windzu/x4d-devkit/issues/44) - capability registry and CLI sync
+
+## Installation
+
+```bash
+pip install x4d-devkit
+```
+
+X-4D platform runtime installs this package from the checked-out private source
+repo/submodule, not from an unconstrained package-index version. External
+environments may install from a pinned package release or directly from GitHub,
+depending on deployment policy.
+
+Optional heavy dependencies:
+
+```bash
+# NuScenes format converter
+pip install x4d-devkit[converters]
+
+# External inference service SDK
+pip install x4d-devkit[inference]
+```
+
+The platform API client and `x4d-devkit` CLI are base package features.
+
+## Scope
+
+See [docs/module-boundary.md](docs/module-boundary.md) for the module ownership
+table and command boundary.
+
+Internal consumers should use the current-state identity contract for
+platform-to-platform data flow. See
+[docs/current-state-identity.md](docs/current-state-identity.md) for the
+generic identity model, and
+[docs/training-current-state-data.md](docs/training-current-state-data.md) for
+diff/sync, work-session materialization, training manifests, and conditional
+annotation writeback flows built on that model.
+Shared validation rules for platform submit, X-Points preflight, local clip
+validation, and training consumers are documented in
+[docs/shared-contract-validation.md](docs/shared-contract-validation.md).
+`examples/current_state_identity_consumers.py` provides an offline downstream
+fixture for OpenPCDet-style, mmdetection3d-style, and annotation/review
+consumers.
+
+Included:
+
+- Local X-4D dataset loading, validation, transforms, annotations, calibration,
+  ego pose, manifests, converters, and evaluation helpers.
+- Platform API client and public `x4d-devkit` CLI commands for external
+  workflows such as clip listing/download, screening preview, model/checkpoint
+  registration, status queries, and capability discovery.
+- Training bridge workflows such as multi-project manifests, archive download,
+  split validation, and dataloader-friendly metadata.
+- External inference service wrappers for third-party model-serving projects.
+
+Excluded:
+
+- Backend migrations, DB/MinIO repair scripts, cache rebuild internals,
+  emergency operations, and one-off delivery importers.
+- Any code that imports backend-private `app.*`.
+- Downstream experiment taxonomy, `CLASS_NAMES`, model-class mappings, sampler
+  policy, model acceptance thresholds, deployment policy, and framework-specific
+  artifacts such as OpenPCDet infos/dbinfos or mmdetection3d pickle views.
+
+Promotion rule: an internal operation becomes an `x4d-devkit` command only
+after it is productized as an external SDK/API workflow with explicit
+permissions, side effects, idempotency, and failure modes. Data deletion, clip
+production, and internal service dispatch stay out of devkit.
+
+Training-specific convenience utilities must remain framework-neutral. Devkit
+may expose raw category stats, identity fingerprints, coordinate/schema
+validation, cache completeness checks, and official X-4D metrics; downstream
+training repos own taxonomy mapping, training-format generation, sampler
+configuration, and experiment lifecycle.
+
+## Quick Start
+
+### Connect to an X-4D platform
+
+The CLI can authenticate with a bearer token passed explicitly, through
+environment variables, or from the user config written by `x4d-devkit login`.
+
+```bash
+x4d-devkit --api-url http://host:8000 --token <token> projects list
+x4d-devkit --api-url http://host:8000 --token <token> clips list --project-id 1
+x4d-devkit --api-url http://host:8000 --token <token> clips download --clip-ids clip_a,clip_b --output /data/x4d/clips
+```
+
+```bash
+export X4D_API_URL=http://host:8000
+export X4D_TOKEN=<token>
+x4d-devkit projects list
+x4d-devkit clips list --project-name nuscenes-mini --has-archive true --format json
+x4d-devkit clips download --project-name nuscenes-mini --output /data/x4d/nuscenes-mini/clips --workers 8
+x4d-devkit clips download --project-name nuscenes-mini --archive-profile keyframes_only --output /data/x4d/nuscenes-mini/keyframes --workers 8
+```
+
+```bash
+x4d-devkit --api-url http://host:8000 login --username <user> --password <password>
+```
+
+`login` stores `api_url` and the returned access token as `token` in
+`~/.config/x4d/config.toml`. Archive downloads go through the platform file
+proxy and extract standard X4D clip directories; training containers do not
+need database or MinIO access.
+
+Clip archive downloads are profile-aware. `--archive-profile keyframes_only` is
+the default and requests the smaller archive with sweeps removed and metadata
+filtered accordingly. Use `--archive-profile full` when a consumer needs the
+complete clip payload including `sweeps/`. If the requested profile has not been
+built, is stale, or failed to build, the devkit reports that profile-specific
+reason instead of falling back to another tar.
+
+### Create a multi-project training manifest
+
+Training projects can combine multiple X-4D projects by downloading standard
+clip archives and generating one local manifest. The downstream training
+framework consumes only the manifest and local clip files; it does not need
+database, MinIO, or platform-private file paths.
+
+```bash
+x4d-devkit manifest create \
+  --projects hg1.0 hg1.5 \
+  --data-root /data/x4d/cache/hg_mix \
+  --output /data/x4d/cache/hg_mix/manifest.json \
+  --download-missing \
+  --archive-profile keyframes_only \
+  --split-policy by_clip \
+  --val-ratio 0.2 \
+  --seed 42
+
+x4d-devkit manifest validate \
+  --manifest /data/x4d/cache/hg_mix/manifest.json \
+  --data-root /data/x4d/cache/hg_mix
+```
+
+Internal current-state consumers can snapshot remote clip identity before
+reviewing diffs or syncing local caches:
+
+```bash
+x4d-devkit dataset inspect \
+  --project-name hg1.5 \
+  --include-asset-revision \
+  --out remote_identity.json
+```
+
+Training runs can validate their consumed-data identity record:
+
+```bash
+x4d-devkit manifest identity-validate --manifest training_identity.json
+```
+
+Dataset update planning can also compare consumer-owned training identity
+fields, such as split membership, class-mapping fingerprint, data policy, or a
+generic consumer identity fingerprint:
+
+```bash
+x4d-devkit dataset diff \
+  --local-manifest current_training_identity.json \
+  --previous-training-identity previous_training_identity.json \
+  --out update_plan.json
+```
+
+`manifest validate` prints bounded warning output by default. It reports
+warning totals grouped by type, category, and project/category, then shows a
+small set of examples. Use `--max-warnings`, `--all-warnings`, or
+`--warnings-output warnings.jsonl` when full warning details are needed.
+
+Downloaded clips are stored under
+`projects/<project-name>/clips/<clip-id>/`. Manifest file paths are relative to
+`--data-root`, so the dataset can be moved as a directory.
+
+The platform is expected to assign globally unique `clip_id` values. The devkit
+validator checks duplicate clip identities and split leakage, but it does not
+perform expensive semantic near-duplicate detection across projects.
+
+```python
+from x4d_devkit import X4DClient
+
+client = X4DClient.from_config()
+manifest = client.create_training_manifest(
+    projects=["hg1.0", "hg1.5"],
+    data_root="/data/x4d/cache/hg_mix",
+    output="/data/x4d/cache/hg_mix/manifest.json",
+    download_missing=True,
+    split_policy="by_clip",
+    val_ratio=0.2,
+    seed=42,
+)
+```
+
+The detection manifest keeps boxes in their native X-4D annotation frame
+(`meta.sensors[meta.annotation_source_channel].frame_id`). It does not align
+poses across clips; each clip's `clip_world` remains local to that clip.
+
+### Consume the project label schema
+
+X-4D is the source of truth for label schemas. External training and inference
+projects should read the effective project schema from the platform instead of
+maintaining their own category list.
+
+```bash
+x4d-devkit label-schemas project-get --project-id 1 --format json
+x4d-devkit label-schemas get \
+  --schema-id object_3d_low_speed_roadside \
+  --version 1.0.0 \
+  --format json
+```
+
+```python
+from x4d_devkit import X4DClient
+from x4d_devkit.label_schema import compare_project_label_schema_snapshot
+
+client = X4DClient.from_config()
+project = client.projects.resolve(project_name="nuscenes-mini")
+project_schema = client.label_schemas.resolve_project(project_name="nuscenes-mini")
+label_schema = project_schema.label_schema
+
+print(project["id"], project["name"])
+print(project_schema.binding.to_dict())
+print(sorted(label_schema.class_ids))
+
+label_schema.validate_category("tanker_truck")
+
+stored_snapshot = project_schema.to_dict()
+current = client.label_schemas.resolve_project(project_id=project["id"])
+comparison = compare_project_label_schema_snapshot(stored_snapshot, current)
+assert comparison.status == "same"
+```
+
+The registry schema is the fine-grained annotation contract. Training code may
+still define experiment-specific class mappings, but those mappings should be
+validated against the project schema before a job starts.
+
+Training manifests keep the immutable project contract separate from the
+observed/configured class catalog:
+
+```text
+label_schema.projects.<project>.contract       # binding + full registry record
+label_schema.projects.<project>.class_catalog  # training/observed class catalog
+```
+
+The deprecated per-project `classes`, `source`, and `schema_version` aliases
+remain in manifest v0.1 for existing consumers. New consumers must parse
+`contract` with `parse_project_label_schema_snapshot()` and treat
+`class_catalog` as non-authoritative observation metadata.
+
+This strict parser/view contract is introduced in x4d-devkit `0.16.0`.
+Consumers tracking `main` should still record `get_build_info()["source"]` in
+training identity so an untagged development run remains reproducible by exact
+commit. Deployments that copy a source tree without its Git directory must set
+`X4D_DEVKIT_COMMIT` to the copied 40- or 64-hex commit; `get_build_info()` records
+that value with `commit_source=environment`.
+
+Project resolution by name/key is exact and handles platform pagination
+internally; ambiguous or missing project references raise structured resolver
+errors instead of requiring callers to scan project lists themselves.
+
+### Load a clip
+
+```python
+from x4d_devkit import ClipLoader
+
+loader = ClipLoader("/path/to/clip")
+print(loader.meta)
+
+for sample in loader.samples:
+    for sd in loader.sample_data_for_sample(sample.token):
+        print(sd.channel, sd.file_path)
+```
+
+### Open a platform work session
+
+Internal tools such as X-Points can open the platform's current clip state
+without waiting for a prebuilt archive:
+
+```python
+from x4d_devkit import ClipLoader, X4DClient, materialize_work_session_clip
+from x4d_devkit.client import WorkSessionConflictError
+from x4d_devkit.core.loader import CLIP_WORLD_FRAME_ID
+
+client = X4DClient.from_config()
+manifest = client.clip_work_sessions.open(project_id=1812, clip_id="clip-id")
+materialize_work_session_clip(
+    manifest,
+    output_dir="/data/x4d_cache/clips/clip-id",
+    client=client,
+    asset_policy="keyframes_only",
+)
+
+loader = ClipLoader.from_work_session(manifest, api_url=client.api_url)
+anns_world = loader.annotations_for_sample("sample-token", frame=CLIP_WORLD_FRAME_ID)
+lidar_asset = loader.point_cloud_asset("sample-token", manifest.annotation_source_channel)
+
+try:
+    response = client.clip_work_sessions.submit_annotations(
+        manifest=manifest,
+        annotations=manifest.annotations,
+        instances=manifest.instances,
+    )
+    print(response["annotation_revision"])
+except WorkSessionConflictError as exc:
+    print(exc.current_identities)
+    raise
+```
+
+Materialization preserves both schema facts atomically: the clip's recorded
+identity remains in `meta.label_schema`, while the effective full registry
+record is written to `label_schema.json`. Full and annotation-only refreshes
+write the same schema snapshot, so local validation does not require a live
+registry lookup.
+
+A ready work session is accepted only when `label_schema_fingerprint`, the full
+registry record, `clip_label_schema`, and `meta.label_schema` declare the same
+identity. A not-ready session may expose a stale clip identity alongside the
+current project schema so migration tooling can report the mismatch, but it
+cannot be materialized.
+
+Submit is conditional on the manifest's `source_revision`,
+`annotation_revision`, and `label_schema_fingerprint`. If the platform state
+changed after the session was opened, the client raises
+`WorkSessionConflictError` with machine-readable `conflicts`,
+`base_identities`, and `current_identities`.
+
+### Coordinate frame transforms
+
+Frames are real frame_id strings: any node in the calibration tree (e.g.
+`"LIDAR_TOP"`, `"base_link"`, `"cam_front"`) plus the constant
+`"clip_world"` (the SLAM-anchored clip-local world). The legacy aliases
+`"sensor"`, `"ego"`, `"world"` are not accepted.
+
+Training converters should read the clip's self-described frame contract rather
+than assume fixed channel names. The native annotation frame is derived from
+`meta.sensors[meta.annotation_source_channel].frame_id`; `clip_world` is local
+to one clip and is not a global frame across clips.
+
+The installed package exposes the training coordinate contract for logs and
+debugging:
+
+```bash
+x4d-devkit dataset coordinate-contract
+x4d-devkit dataset coordinate-contract --format json
+```
+
+```python
+from x4d_devkit import COORDINATE_CONTRACT_VERSION, get_training_coordinate_contract
+```
+
+```python
+from x4d_devkit import ClipLoader
+from x4d_devkit.core.loader import CLIP_WORLD_FRAME_ID
+
+loader = ClipLoader("/path/to/clip")
+sd = loader.sample_data_for_channel("LIDAR_TOP")[0]
+
+# Load point cloud in different frames
+pts_sensor = loader.load_point_cloud(sd)                                 # raw sensor (default)
+pts_ego    = loader.load_point_cloud(sd, frame=loader.ego_pose_frame_id) # sensor → ego
+pts_world  = loader.load_point_cloud(sd, frame=CLIP_WORLD_FRAME_ID)      # sensor → clip_world
+
+# Get annotations transformed to clip-local world
+anns_world = loader.annotations_for_sample(sample.token, frame=CLIP_WORLD_FRAME_ID)
+
+# Or to a specific sensor's frame
+anns_lidar = loader.annotations_for_sample(sample.token, frame="LIDAR_TOP")
+
+# Get the transform matrix directly (sd is required when clip_world is involved)
+T = loader.get_transform(loader.sensor_frame_id(sd), CLIP_WORLD_FRAME_ID, sd=sd)
+pts_world = T.apply(pts_sensor[:, :3])  # or use T.as_matrix for 4x4
+```
+
+### Validate a clip
+
+```bash
+x4d-devkit validate /path/to/clip
+```
+
+```python
+from x4d_devkit import validate_clip
+
+report = validate_clip("/path/to/clip")
+print(report)
+```
+
+### Serve an external inference backend
+
+Third-party model projects should use `x4d-devkit[inference]` instead of
+hand-writing the X-4D inference HTTP protocol.
+
+```python
+from x4d_devkit.inference import DetectionModel, InferenceService, detection_3d_item
+
+
+class MyDetector(DetectionModel):
+    model_id = "centerpoint-v1"
+    display_name = "CenterPoint v1"
+    raw_classes = ["car", "truck", "pedestrian"]
+
+    def predict_clip(self, clip, config):
+        return [
+            detection_3d_item(
+                sample_token=clip.samples[0].token,
+                raw_class="car",
+                score=0.91,
+                translation=(1.0, 2.0, 0.5),
+                size=(4.5, 1.8, 1.6),
+                yaw=0.2,
+            )
+        ]
+
+
+InferenceService(service_name="my-detector", models=[MyDetector()]).run(port=9000)
+```
+
+See [docs/external-inference.md](docs/external-inference.md) and
+`examples/external_detection_service.py` for the standard integration flow.
+
+### Detection evaluation
+
+```python
+from x4d_devkit.eval import DetectionEval, DetectionConfig
+
+config = DetectionConfig(
+    class_names=["car", "pedestrian", "bicycle"],
+    dist_thresholds=[0.5, 1.0, 2.0, 4.0],
+)
+evaluator = DetectionEval(config, gt_clips=[...], pred_clips=[...])
+result = evaluator.evaluate()
+print(f"mAP: {result.mAP:.3f}, NDS: {result.NDS:.3f}")
+```
+
+For detectors that intentionally do not predict velocity, construct 7D boxes
+and evaluate with `with_velocity=False` so velocity error is excluded from the
+primary detection score:
+
+```python
+from x4d_devkit.eval import Box, DetectionConfig, evaluate
+
+pred = {
+    "sample_1": [
+        Box.from_xyzlwhyaw([0, 0, 0, 4, 2, 1.5, 0.0], category="car", score=0.9)
+    ]
+}
+config = DetectionConfig(
+    class_names=["car"],
+    dist_thresholds=[0.5, 1.0, 2.0, 4.0],
+    dist_th_tp=2.0,
+    min_recall=0.1,
+    min_precision=0.1,
+    max_boxes_per_sample=500,
+    class_range={"car": 50.0},
+    with_velocity=False,
+)
+result = evaluate(gt, pred, config)
+assert result.with_velocity is False
+```
+
+### Convert from NuScenes
+
+```python
+from x4d_devkit.converters import NuScenesConverter
+
+converter = NuScenesConverter("/path/to/nuscenes")
+converter.convert_scene("scene-0001", output_dir="/path/to/output")
+```
+
+## Modules
+
+| Module | Description |
+|--------|-------------|
+| `core` | Data models, token generation, coordinate transforms, clip loader |
+| `eval` | Detection evaluation (mAP, TP metrics, NDS) |
+| `converters` | Format converters (NuScenes → X4D) |
+| `manifest` | Training manifest creation, validation, and clip archive download |
+| `validation` | Clip structure and data validation |
+| `client` | X-4D platform API client |
+
+## License
+
+Apache License 2.0