@luxonis/depthai-viewer-common 3.7.4 → 3.7.6

package/REFACTOR_PLAN.md

@@ -0,0 +1,265 @@
+# REFACTOR_PLAN.md
+
+## Goal
+
+Refactor this repo-root package into a Luxonis-owned visualizer library that keeps message compatibility for currently supported inputs while removing Foxglove runtime/build coupling, improving throughput, and simplifying the codebase.
+
+## Non-Negotiable Constraints
+
+- Supported incoming message types must remain compatible.
+- Output semantics must stay the same unless a deliberate API break is documented and gated.
+- Performance work must be measured, not guessed.
+- Future agents must commit after each completed checkpoint and update `REFACTOR_HANDOFF.md`.
+- The real runtime target is browser-only. Validation strategy must respect that.
+
+## Target End State
+
+- One self-contained `src/` codebase with no dependency on the removed Foxglove workspace packages.
+- Luxonis-owned transport, decode, render, and panel state layers.
+- Bounded worker pools and explicit zero-copy or reduced-copy transfer paths.
+- Modernized build/package metadata and newer dependencies.
+- Compatibility fixtures and performance benchmarks that prevent regressions.
+
+Current progress snapshot:
+
+- `src/` no longer imports `@foxglove/studio-base`, `@foxglove/studio`, or `@foxglove/ws-protocol`.
+- `ImagePanel` and `PointCloudPanel` now use local renderers.
+- `tsconfig.json` and Rollup no longer depend on sibling-package source aliases or the old `packages/` monorepo layout.
+- Package metadata has been partially normalized (`main`, `types`, `exports`, and dependency cleanup), but publishability and runtime-config cleanup are still open.
+
+## Phase 0: Baseline And Guardrails
+
+### Objectives
+
+- Freeze current supported input/output behavior before refactoring internals.
+- Establish a measurable baseline for performance and correctness.
+
+### Tasks
+
+- Add representative browser-independent decoder fixtures for:
+  - raw color frames
+  - detections
+  - spatial detections
+  - IMU
+  - colored and uncolored point clouds
+- Add browser-runtime validation coverage for:
+  - encoded H.264
+  - MJPEG
+  - H.265
+  - raw depth preview
+  - RGBD
+- Add golden-output assertions for decoded event shape, schema name, timestamps, dimensions, and data-plane metadata.
+- Add a small benchmark harness for decode throughput and point-cloud generation latency.
+- Explicitly separate Node-safe checks from browser-only checks in docs and commands.
+- Record current baseline problems separately from new regressions.
+
+Current starting point:
+
+- Node-safe decoder fixtures already cover raw color, detections, spatial detections, IMU, and colored/uncolored point clouds.
+- Browser fixture scaffolding exists for worker-backed raw depth and RGBD validation via `scripts/browser-decoder-fixtures.html`.
+- Encoded video browser fixtures still need real sample coverage.
+- Real browser usage now shows periodic video hitching roughly every 5 seconds on encoded H.264/H.265 image streams. Treat this as a tracked regression candidate in the encoded-video decode path until disproven.
+
+### Exit Criteria
+
+- Agents can prove compatibility before and after a refactor.
+- Agents can identify which compatibility guarantees are covered in Node versus only in browser runtime.
+- Agents can measure decode/render improvements with repeatable commands.
+
+## Phase 1: Internal Contracts And Baseline Cleanup
+
+### Objectives
+
+- Define Luxonis-owned internal types and config flow.
+- Remove easy Foxglove coupling that blocks later work.
+
+### Tasks
+
+- Introduce internal equivalents for the minimal message-event and frame types used inside `lib/`.
+- Centralize runtime configuration instead of scattered `localStorage` reads.
+- Remove the legacy internal rate limiter if measurement continues to show it is inert or unused, instead of preserving dead throttling scaffolding in the runtime path.
+- Fix branch-baseline config WIP cleanly:
+  - either complete `runtime-config-storage.ts`
+  - or remove/replace it in favor of the chosen config system
+- Inline or replace small Foxglove helper dependencies where safe:
+  - time helpers
+  - point types
+  - RawImage extension types
+  - object-size policy, if retained at all
+- Ensure no new imports are added from `@foxglove/studio-base`.
+
+### Exit Criteria
+
+- `lib/` type-checks again.
+- New internal contracts exist and are used by at least one subsystem.
+- No change to decoded output behavior.
+
+## Phase 2: Decode And Worker Pipeline Refactor
+
+### Objectives
+
+- Reduce copies, thread explosion, and duplicated pipelines.
+- Make transport and decode architecture explicit and bounded.
+
+### Tasks
+
+- Replace per-topic deserialization workers with a bounded pool by workload class.
+- Prefer explicit `postMessage` transferables over opaque Comlink hops on hot paths.
+- Merge or simplify transport and decode stages where it removes a full clone/copy boundary.
+- Unify legacy depth+color point-cloud generation and RGBD point-cloud generation.
+- Replace internal misuse of Foxglove schema names for sync-only artifacts.
+- Fix compatibility helpers:
+  - WebCodecs detection
+  - encoded-stream capability gating
+  - supported-format declarations in one place
+- Remove hot-path `estimateObjectSize()` calls where exact or carried-forward sizes are available.
+- Preserve a browser-native validation path for any refactor touching `Worker`, `VideoDecoder`, or WebGPU-backed decoding.
+- Investigate and measure the periodic encoded-video lag observed in browser runtime:
+  - reproduce on H.264 and H.265 streams
+  - determine whether the hitch is caused by WebCodecs decode cadence, worker/message transfer cadence, frame queueing, or buffer repacking
+  - record timing evidence before attempting optimizations
+
+### Exit Criteria
+
+- Compatibility fixtures still pass.
+- Worker count is bounded and documented.
+- Decode latency and memory churn are measurably lower than baseline.
+- Encoded video no longer exhibits the observed periodic hitching in browser validation, or the remaining cause is explicitly documented with measured evidence.
+
+## Phase 3: Rendering Replacement
+
+### Objectives
+
+- Remove dependence on Foxglove `Image` and `ThreeDeeRender`.
+- Keep the public panel API usable while replacing internals.
+
+### Tasks
+
+- Build an internal image renderer first:
+  - stable mounted component
+  - image display
+  - annotation overlay
+  - click/cursor coordinate support
+  - render metrics hooks
+- Build an internal point-cloud renderer second:
+  - evaluate WebGPU first
+  - fallback to optimized Three.js if rewrite cost is too high for required UX
+  - support colored point clouds, camera control, measurement, and picking as needed
+- Remove remount-on-detection behavior.
+- Replace per-panel polling with shared metrics and better resize handling.
+
+### Exit Criteria
+
+- `src/panels/*` no longer lazy-load Foxglove renderer panels.
+- Rendering performance is equal or better on benchmark scenes.
+- Panel UX parity is documented.
+
+Status:
+
+- Largely complete on renderer ownership.
+- Remaining work in this phase is parity documentation, browser-runtime confidence, and any renderer-specific follow-up tuning rather than Foxglove panel replacement itself.
+
+## Phase 4: Context And Pipeline Simplification
+
+### Objectives
+
+- Remove Foxglove Studio runtime scaffolding that only exists to host reused panels.
+
+### Tasks
+
+- Replace `VisualizerContext` Foxglove providers with Luxonis-owned context/store layers.
+- Replace Foxglove `MessagePipeline` and `PanelAPI/useMessageReducer` usage with a simpler subscription model.
+- Remove stubbed/no-op Foxglove contexts.
+- Clarify supported connection model:
+  - either true multi-connection support
+  - or explicitly single-connection internal architecture
+
+### Exit Criteria
+
+- No runtime dependency on Foxglove Studio contexts/providers/hooks remains in `lib/src`.
+- Message subscription and render scheduling are simpler and documented.
+
+Status:
+
+- Mostly complete in `lib/src`.
+- Remaining work is documentation cleanup and any final simplification opportunities around config flow rather than direct Foxglove context removal.
+
+## Phase 5: Monorepo And Build Cleanup
+
+### Objectives
+
+- Make `lib/` self-contained and package-correct.
+
+### Tasks
+
+- Move or rewrite any remaining code still sourced from `../packages`.
+- Remove `tsconfig` and rollup aliases to `../packages/studio-base/src`.
+- Simplify build tooling while preserving worker bundling requirements.
+- Fix package metadata:
+  - exports
+  - `main`
+  - `types`
+  - worker assets
+- Remove obsolete or duplicate dependencies.
+- Bump dependency versions in a controlled pass after the architecture is stable.
+
+### Exit Criteria
+
+- `lib/` builds without `../packages/studio-base`.
+- Package metadata is correct and publishable.
+- Dependency versions are consistent and intentional.
+
+Status:
+
+- The `../packages/studio-base` / `../packages/theme` / `../packages` source-tree coupling is removed.
+- The remaining work is package-publish correctness, dependency/external reconciliation, and documentation alignment.
+
+## Phase 6: Release Prep
+
+### Objectives
+
+- Make the rewrite safe to adopt.
+
+### Tasks
+
+- Re-run compatibility fixtures and benchmarks.
+- Document API differences, if any.
+- Record migration notes for consuming apps.
+- Produce a final comparison of:
+  - supported inputs
+  - output compatibility
+  - performance deltas
+  - removed Foxglove dependencies
+
+### Exit Criteria
+
+- Refactor state is handoff-safe and release-ready.
+
+## Recommended Immediate Next Task
+
+Continue Phase 5 from the current branch state.
+
+Reason:
+
+- The large Foxglove source-boundary removals are already done in `lib/src`.
+- The highest-value remaining low-risk work is publishability/documentation correctness rather than another architectural rewrite by default.
+- Browser-runtime benchmarking and compatibility validation are still important, but they should build on the current cleaner package/runtime boundary.
+
+## Commit And Handoff Protocol
+
+For every completed checkpoint:
+
+1. Run the relevant validation commands.
+2. Update `REFACTOR_HANDOFF.md` with:
+   - what changed
+   - validation results
+   - open risks
+   - exact next task
+3. Commit the checkpoint.
+4. Record the commit hash in `REFACTOR_HANDOFF.md`.
+
+Preferred commit shape:
+
+- one logical checkpoint per commit
+- no mixed unrelated cleanup
+- no silent drive-by changes outside the current phase