@vincentt-xr/sdk 2.0.0-alpha.0 → 2.0.0-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/GROUNDING.md +1813 -27
- package/dist/{DebuggerUI-CEfdddkZ.cjs → DebuggerUI-B45_Gb9d.cjs} +1 -1
- package/dist/{DebuggerUI-Dgop6Q6o.js → DebuggerUI-Cq28c18s.js} +1 -1
- package/dist/FaceMeshRenderer-6t8OsQhK.js +1195 -0
- package/dist/FaceMeshRenderer-lQLhYIW-.cjs +155 -0
- package/dist/FaceRetouch-CDrL0pXg.js +551 -0
- package/dist/FaceRetouch-DtNl8-4e.cjs +134 -0
- package/dist/audio/AudioEngine.d.ts +3 -1
- package/dist/audio/AudioPlayer.d.ts +29 -0
- package/dist/audio/AudioPlayerLogic.d.ts +4 -0
- package/dist/audio/index.d.ts +2 -0
- package/dist/components/Axes/index.d.ts +6 -0
- package/dist/components/FaceMesh/FaceMeshRenderer.d.ts +17 -0
- package/dist/components/FaceMesh/canonical/CanonicalFaceMeshRenderer.d.ts +22 -1
- package/dist/components/FaceMesh/canonical/FaceMeshMaterial.d.ts +25 -0
- package/dist/components/FaceMesh/canonical/assets.d.ts +3 -0
- package/dist/components/FaceMesh/canonical/geometryGroups.d.ts +9 -0
- package/dist/components/FaceMesh/canonical/parseCanonicalFaceModel.d.ts +2 -0
- package/dist/components/FaceMesh/canonical/shaders.d.ts +2 -0
- package/dist/components/FaceMesh/deform.d.ts +3 -0
- package/dist/components/FaceMesh/index.d.ts +5 -0
- package/dist/components/FaceMesh/material/defaults.d.ts +74 -0
- package/dist/components/FaceMesh/material/index.d.ts +1 -0
- package/dist/components/FaceMesh/material/options.d.ts +5 -1
- package/dist/components/FaceMesh/material/support.d.ts +7 -2
- package/dist/components/FaceMesh/material/textures.d.ts +4 -0
- package/dist/components/FaceMesh/material/types.d.ts +4 -2
- package/dist/components/FaceRetouch/Background.d.ts +9 -5
- package/dist/components/FaceRetouch/FaceRetouch.d.ts +47 -0
- package/dist/components/FaceRetouch/index.d.ts +2 -6
- package/dist/components/FaceRetouch/landmarkMask.d.ts +4 -6
- package/dist/components/FaceRetouch/useFaceRetouchMask.d.ts +4 -7
- package/dist/components/FaceTracker/index.d.ts +5 -3
- package/dist/components/FaceWarp3D/FaceWarp3DHead.d.ts +15 -0
- package/dist/components/FaceWarp3D/defaults.d.ts +37 -0
- package/dist/components/FaceWarp3D/index.d.ts +3 -0
- package/dist/components/FaceWarp3D/warp.d.ts +13 -0
- package/dist/components/HandJoints2D/index.d.ts +6 -0
- package/dist/components/HandTracker/index.d.ts +2 -2
- package/dist/components/HandTracking2D/HandVisual.d.ts +12 -0
- package/dist/components/HandTracking2D/index.d.ts +5 -0
- package/dist/components/HandTracking2DController/index.d.ts +5 -0
- package/dist/components/HandTracking3D/index.d.ts +6 -0
- package/dist/components/LightObject/index.d.ts +20 -0
- package/dist/components/LookAt/index.d.ts +23 -0
- package/dist/components/MeshRenderer/index.d.ts +58 -0
- package/dist/components/PerspectiveCamera/index.d.ts +2 -8
- package/dist/components/PinToMesh/index.d.ts +47 -0
- package/dist/components/RenderGroupLayer/index.d.ts +6 -0
- package/dist/components/SceneObjectRenderer/index.d.ts +16 -0
- package/dist/components/Screen2D/ScreenTransformFrame.d.ts +11 -0
- package/dist/components/Screen2D/ScreenTransformHelpers.d.ts +19 -0
- package/dist/components/Screen2D/ScreenTransformInteraction.d.ts +18 -0
- package/dist/components/Screen2D/index.d.ts +5 -0
- package/dist/components/ScreenGlass/defaults.d.ts +3 -0
- package/dist/components/ScreenGlass/index.d.ts +5 -0
- package/dist/components/ScreenGlass/shaders.d.ts +2 -0
- package/dist/components/ScreenGlass/types.d.ts +10 -0
- package/dist/components/ScreenGradient/defaults.d.ts +5 -0
- package/dist/components/ScreenGradient/index.d.ts +5 -0
- package/dist/components/ScreenGradient/shaders.d.ts +2 -0
- package/dist/components/ScreenGradient/types.d.ts +19 -0
- package/dist/components/ScreenImage/ScreenImage.d.ts +4 -0
- package/dist/components/ScreenImage/components/DynamicTextureSync.d.ts +9 -0
- package/dist/components/ScreenImage/components/ScreenImageBody.d.ts +9 -0
- package/dist/components/ScreenImage/components/ScreenImageFrame.d.ts +9 -0
- package/dist/components/ScreenImage/components/ScreenImageTransformFrame.d.ts +9 -0
- package/dist/components/ScreenImage/fit.d.ts +52 -0
- package/dist/components/ScreenImage/index.d.ts +8 -0
- package/dist/components/ScreenImage/materials/ScreenImageBlendMaterial.d.ts +9 -0
- package/dist/components/ScreenImage/materials/ScreenImageMatcap.d.ts +3 -0
- package/dist/components/ScreenImage/source.d.ts +16 -0
- package/dist/components/ScreenImage/types.d.ts +37 -0
- package/dist/components/ScreenImage/utils/ScreenImageConstants.d.ts +9 -0
- package/dist/components/ScreenImage/utils/ScreenImageHelpers.d.ts +4 -0
- package/dist/components/ScreenShape/geometry.d.ts +5 -0
- package/dist/components/ScreenShape/index.d.ts +5 -0
- package/dist/components/ScreenShape/types.d.ts +41 -0
- package/dist/components/ScreenSpaceUI/ScreenTransformGuide.d.ts +15 -0
- package/dist/components/ScreenSpaceUI/ScreenTransformOverlayGuide.d.ts +9 -0
- package/dist/components/ScreenSpaceUI/index.d.ts +2 -0
- package/dist/components/ScreenText/components/ScreenTextBody.d.ts +2 -0
- package/dist/components/ScreenText/components/ScreenTextTransformFrame.d.ts +9 -0
- package/dist/components/ScreenText/constants.d.ts +5 -0
- package/dist/components/ScreenText/index.d.ts +6 -0
- package/dist/components/ScreenText/types.d.ts +31 -0
- package/dist/components/ScreenText/utils/ScreenTextHelpers.d.ts +14 -0
- package/dist/components/ScreenText/utils/ScreenTextMappings.d.ts +17 -0
- package/dist/components/SequentialRenderPasses/index.d.ts +6 -0
- package/dist/components/TextLabel/TextLabel.d.ts +5 -0
- package/dist/components/TextLabel/alignment.d.ts +4 -0
- package/dist/components/TextLabel/bend.d.ts +9 -0
- package/dist/components/TextLabel/canvas/buildTextLabelTexture.d.ts +4 -0
- package/dist/components/TextLabel/canvas/canvasTypography.d.ts +13 -0
- package/dist/components/TextLabel/canvas/drawBackground.d.ts +21 -0
- package/dist/components/TextLabel/canvas/drawEffects.d.ts +34 -0
- package/dist/components/TextLabel/canvas/drawText.d.ts +19 -0
- package/dist/components/TextLabel/components/TextLabelMesh.d.ts +13 -0
- package/dist/components/TextLabel/constants.d.ts +2 -0
- package/dist/components/TextLabel/hooks/useTextLabelTexture.d.ts +7 -0
- package/dist/components/TextLabel/index.d.ts +2 -62
- package/dist/components/TextLabel/layout/measurement.d.ts +8 -0
- package/dist/components/TextLabel/layout/resolveTextLayout.d.ts +25 -0
- package/dist/components/TextLabel/layout/wrapText.d.ts +6 -0
- package/dist/components/TextLabel/types.d.ts +227 -0
- package/dist/components/TrackingAnchor/index.d.ts +18 -2
- package/dist/components/Transform3D/index.d.ts +30 -0
- package/dist/components/XRMediaSurface/index.d.ts +6 -6
- package/dist/components/XRScene/index.d.ts +14 -3
- package/dist/components/XRViewport/XRViewport.d.ts +11 -0
- package/dist/components/XRViewport/components/HiddenMedia.d.ts +7 -0
- package/dist/components/XRViewport/components/MediaAssetPicker.d.ts +8 -0
- package/dist/components/XRViewport/components/MediaControlStack.d.ts +13 -0
- package/dist/components/XRViewport/components/XRViewportLayers.d.ts +30 -0
- package/dist/components/XRViewport/constants.d.ts +24 -0
- package/dist/components/XRViewport/contexts.d.ts +6 -0
- package/dist/components/XRViewport/hooks/useViewportExternalFaces.d.ts +13 -0
- package/dist/components/XRViewport/hooks/useViewportMediaSource.d.ts +16 -0
- package/dist/components/XRViewport/hooks/useViewportMediaTexture.d.ts +10 -0
- package/dist/components/XRViewport/hooks/useViewportPipeline.d.ts +41 -0
- package/dist/components/XRViewport/hooks/useViewportRenderState.d.ts +47 -0
- package/dist/components/XRViewport/index.d.ts +3 -101
- package/dist/components/XRViewport/layout.d.ts +16 -0
- package/dist/components/XRViewport/slots.d.ts +16 -0
- package/dist/components/XRViewport/types.d.ts +89 -0
- package/dist/components/XRViewport/useViewportSize.d.ts +3 -0
- package/dist/components/index.d.ts +19 -0
- package/dist/contexts-BKvTIayr.cjs +1 -0
- package/dist/contexts-Bk5zz_VR.js +214 -0
- package/dist/debug-ui/component-editor/index.d.ts +19 -0
- package/dist/debug-ui/default-setting/index.d.ts +37 -0
- package/dist/debug-ui/default-setting.cjs +21 -0
- package/dist/debug-ui/default-setting.js +777 -0
- package/dist/debug-ui/face-mesh-renderer-settings/index.d.ts +8 -0
- package/dist/debug-ui/face-retouch/index.d.ts +11 -0
- package/dist/debug-ui/face-warp-3d/index.d.ts +6 -0
- package/dist/debug-ui/hand-joints-2d/index.d.ts +6 -0
- package/dist/debug-ui/hand-tracking-2d/index.d.ts +6 -0
- package/dist/debug-ui/hand-tracking-2d-controller/index.d.ts +6 -0
- package/dist/debug-ui/hand-tracking-3d/index.d.ts +6 -0
- package/dist/debug-ui/hierarchy-panel/index.d.ts +29 -0
- package/dist/debug-ui/index.d.ts +46 -0
- package/dist/debug-ui/layer-panel/index.d.ts +18 -0
- package/dist/debug-ui/media-source/MediaSourceSwitcher.d.ts +10 -0
- package/dist/debug-ui/media-source/index.d.ts +6 -0
- package/dist/debug-ui/media-source/media-preset.d.ts +9 -0
- package/dist/debug-ui/media-source/session.d.ts +12 -0
- package/dist/debug-ui/media-source.cjs +1 -0
- package/dist/debug-ui/media-source.js +9 -0
- package/dist/debug-ui/mesh-renderer-settings/index.d.ts +6 -0
- package/dist/debug-ui/object-inspector-panel/index.d.ts +25 -0
- package/dist/debug-ui/pin-to-mesh-settings/index.d.ts +10 -0
- package/dist/debug-ui/scene-editor/context.d.ts +38 -0
- package/dist/debug-ui/scene-editor/hierarchy-panel.d.ts +9 -0
- package/dist/debug-ui/scene-editor/index.d.ts +11 -0
- package/dist/debug-ui/scene-editor/inspector-panel.d.ts +7 -0
- package/dist/debug-ui/scene-editor/store-utils.d.ts +2 -0
- package/dist/debug-ui/scene-editor/viewport-toolbar.d.ts +4 -0
- package/dist/debug-ui/scene-editor/viewport.d.ts +21 -0
- package/dist/debug-ui/screen-glass-settings/index.d.ts +6 -0
- package/dist/debug-ui/screen-gradient-settings/GradientStopsEditor.d.ts +5 -0
- package/dist/debug-ui/screen-gradient-settings/index.d.ts +6 -0
- package/dist/debug-ui/screen-gradient-settings/styles.d.ts +3 -0
- package/dist/debug-ui/screen-image-settings/index.d.ts +9 -0
- package/dist/debug-ui/screen-shape-settings/index.d.ts +6 -0
- package/dist/debug-ui/screen-text-settings/NumberField.d.ts +9 -0
- package/dist/debug-ui/screen-text-settings/ScreenTextSettingsPanel.d.ts +3 -0
- package/dist/debug-ui/screen-text-settings/controls/NestedSection.d.ts +7 -0
- package/dist/debug-ui/screen-text-settings/controls/Section.d.ts +8 -0
- package/dist/debug-ui/screen-text-settings/controls/icons.d.ts +4 -0
- package/dist/debug-ui/screen-text-settings/index.d.ts +2 -0
- package/dist/debug-ui/screen-text-settings/sections/BackgroundSection.d.ts +8 -0
- package/dist/debug-ui/screen-text-settings/sections/ContentSection.d.ts +5 -0
- package/dist/debug-ui/screen-text-settings/sections/EffectsSection.d.ts +25 -0
- package/dist/debug-ui/screen-text-settings/sections/StyleSection.d.ts +5 -0
- package/dist/debug-ui/screen-text-settings/sections/TransformSection.d.ts +5 -0
- package/dist/debug-ui/screen-text-settings/sections/background/BackgroundFillControls.d.ts +2 -0
- package/dist/debug-ui/screen-text-settings/sections/background/BackgroundSizeControls.d.ts +2 -0
- package/dist/debug-ui/screen-text-settings/sections/background/BackgroundStrokeControls.d.ts +2 -0
- package/dist/debug-ui/screen-text-settings/sections/background/types.d.ts +7 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/BendEffectSection.d.ts +4 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/EffectChrome.d.ts +21 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/GlowEffectSection.d.ts +4 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/InnerShadowEffectSection.d.ts +4 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/OutlineEffectSection.d.ts +4 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/ShadowEffectSection.d.ts +4 -0
- package/dist/debug-ui/screen-text-settings/sections/effects/types.d.ts +14 -0
- package/dist/debug-ui/screen-text-settings/settings.d.ts +90 -0
- package/dist/debug-ui/screen-text-settings/styles.d.ts +26 -0
- package/dist/debug-ui/screen-text-settings/transform.d.ts +6 -0
- package/dist/debug-ui/screen-text-settings/types.d.ts +9 -0
- package/dist/debug-ui/settings-panel/index.d.ts +38 -0
- package/dist/debug-ui/shared/MaterialSection.d.ts +13 -0
- package/dist/debug-ui/shared/ScreenTransformSection.d.ts +32 -0
- package/dist/debug-ui/transform-3d-settings/index.d.ts +11 -0
- package/dist/debug-ui.cjs +10 -0
- package/dist/debug-ui.js +8861 -0
- package/dist/drawNormalizedCrop-B-7zU-Xw.cjs +1 -0
- package/dist/drawNormalizedCrop-CRdIDyl1.js +44 -0
- package/dist/entries/all.d.ts +1 -0
- package/dist/entries/core.d.ts +55 -4
- package/dist/entries/face-effects.d.ts +12 -8
- package/dist/entries/internal.d.ts +1 -1
- package/dist/entries/low-level.d.ts +21 -4
- package/dist/entries/scene-object.d.ts +10 -0
- package/dist/entries/tracking.d.ts +25 -2
- package/dist/face-action/index.d.ts +34 -0
- package/dist/face-attribute-expression/index.d.ts +10 -0
- package/dist/face-effects.cjs +1 -252
- package/dist/face-effects.js +1404 -2613
- package/dist/face-info/blendshapes.d.ts +3 -0
- package/dist/face-info/derive.d.ts +8 -0
- package/dist/face-info/drawFaceCrop.d.ts +10 -0
- package/dist/face-info/getFaceInfo.d.ts +8 -0
- package/dist/face-info/index.d.ts +8 -0
- package/dist/face-info/landmarks.d.ts +21 -0
- package/dist/face-info/types.d.ts +52 -0
- package/dist/getFaceInfo-CyPAKh3h.cjs +1 -0
- package/dist/getFaceInfo-hUKMLFcr.js +118 -0
- package/dist/hand/handJoints2d.d.ts +49 -0
- package/dist/hand/handTracking2d.d.ts +33 -0
- package/dist/hand/handTracking2dController.d.ts +65 -0
- package/dist/hand/handTracking3d.d.ts +52 -0
- package/dist/hooks/faceResults.d.ts +1 -1
- package/dist/hooks/useCalibrationFaceTracker.d.ts +1 -1
- package/dist/hooks/useFaceAction.d.ts +3 -0
- package/dist/hooks/useFaceAttributeExpression.d.ts +16 -0
- package/dist/hooks/useFaceDetection.d.ts +18 -0
- package/dist/hooks/useFaceInfo.d.ts +7 -0
- package/dist/hooks/useFaceInfos.d.ts +15 -0
- package/dist/hooks/useFaceResultReader.d.ts +10 -0
- package/dist/hooks/useFrameCapture.d.ts +18 -0
- package/dist/hooks/useMediaRecorder.d.ts +14 -2
- package/dist/hooks/useOwnedTexture.d.ts +6 -0
- package/dist/hooks/useTrackerContext.d.ts +2 -2
- package/dist/hooks/useXRSession/facade.d.ts +3 -0
- package/dist/hooks/useXRSession/registry.d.ts +9 -0
- package/dist/index-82aENaRq.cjs +1 -0
- package/dist/{index-BD-JWMjE.js → index-BgQAX8rs.js} +198 -134
- package/dist/index-CEc1CAmB.cjs +1 -0
- package/dist/index-CTHeBQOs.js +178 -0
- package/dist/index-D4bLzBr6.cjs +1 -0
- package/dist/index-DSpzjuqd.cjs +1 -0
- package/dist/index-Di70KWuY.js +2049 -0
- package/dist/index-DzEzkBIH.cjs +17 -0
- package/dist/{index-CYMj26Ej.js → index-JygePOwo.js} +565 -498
- package/dist/index-UC0Ux38E.js +111 -0
- package/dist/{internal-KOKJDxCG.js → internal-DuMPpcbp.js} +86 -80
- package/dist/internal-i2JkM85N.cjs +1 -0
- package/dist/internal.cjs +1 -1
- package/dist/internal.js +2 -2
- package/dist/low-level.cjs +1 -1
- package/dist/low-level.js +959 -632
- package/dist/main.cjs +1 -1
- package/dist/main.js +556 -455
- package/dist/material/MaterialFill.d.ts +20 -0
- package/dist/material/MeshMaterial.d.ts +14 -0
- package/dist/material/index.d.ts +2 -0
- package/dist/models/Body/index.d.ts +1 -1
- package/dist/models/Face/index.d.ts +1 -0
- package/dist/models/Face/rotation.d.ts +3 -0
- package/dist/models/Hand/HandTracking.d.ts +2 -0
- package/dist/render/layerStack.d.ts +29 -0
- package/dist/render/renderGroups.d.ts +47 -0
- package/dist/render/renderPasses.d.ts +18 -0
- package/dist/renderGroups-C9QJrAXu.js +152 -0
- package/dist/renderGroups-Dvm1J4lW.cjs +1 -0
- package/dist/renderLayers-lDtL0_43.cjs +1 -0
- package/dist/renderLayers-zrK2pv5O.js +13 -0
- package/dist/renderPasses-CT5iBwtg.cjs +21 -0
- package/dist/renderPasses-CjgG_6kQ.js +1500 -0
- package/dist/scene-object/index.d.ts +146 -0
- package/dist/scene-object/runtimeRegistry.d.ts +14 -0
- package/dist/scene-object.cjs +1 -0
- package/dist/scene-object.js +40 -0
- package/dist/screen-image/screenImagePresets.d.ts +13 -0
- package/dist/screen-image/screenImageSettings.d.ts +57 -0
- package/dist/screen-text/screenTextPresets.d.ts +6 -0
- package/dist/screen-text/screenTextSettings.d.ts +147 -0
- package/dist/screen2d/coordinates.d.ts +87 -0
- package/dist/screen2d/index.d.ts +7 -0
- package/dist/screen2d/material-runtime.d.ts +32 -0
- package/dist/screen2d/screenTransform.d.ts +47 -0
- package/dist/screen2d/smoothing.d.ts +5 -0
- package/dist/screen2d/texture.d.ts +10 -0
- package/dist/screen2d/transform.d.ts +25 -0
- package/dist/screen2d/types.d.ts +90 -0
- package/dist/{selectors-Bub0o3qy.cjs → selectors-2QvO5RYY.cjs} +1 -1
- package/dist/{selectors-BHO6TnoC.js → selectors-Dw3C-gSk.js} +5 -5
- package/dist/services/MetricsCollector/index.d.ts +7 -1
- package/dist/services/MetricsCollector/utils.d.ts +1 -1
- package/dist/session-BChR4TgK.cjs +31 -0
- package/dist/session-C1StwN0f.js +363 -0
- package/dist/texture/animatedTexture.d.ts +61 -0
- package/dist/texture/animationMath.d.ts +8 -0
- package/dist/texture/sourceResolver.d.ts +30 -0
- package/dist/tracking.cjs +1 -1
- package/dist/tracking.js +582 -298
- package/dist/types/face.d.ts +1 -13
- package/dist/types/hand-tracking.d.ts +8 -0
- package/dist/types/index.d.ts +0 -1
- package/dist/types/nodes.d.ts +4 -1
- package/dist/types/session.d.ts +2 -0
- package/dist/types/tracking.d.ts +8 -2
- package/dist/types/util-types.d.ts +1 -1
- package/dist/useFaceInfo-C-WfMlYp.cjs +1 -0
- package/dist/useFaceInfo-CwsHZedC.js +62 -0
- package/dist/useFaceResults-CkFc1egZ.js +19 -0
- package/dist/useFaceResults-VcFdOUFV.cjs +1 -0
- package/dist/utils/drawNormalizedCrop.d.ts +22 -0
- package/dist/utils/integrity.d.ts +4 -25
- package/dist/utils/landmarkBounds.d.ts +1 -1
- package/dist/utils/logger.d.ts +2 -2
- package/dist/x-Co8jPhP0.js +127 -0
- package/dist/x-KC8E4Gaj.cjs +41 -0
- package/docs/MIGRATION_GUIDE.md +589 -0
- package/docs/SDK_DOCUMENTATION.md +1155 -0
- package/docs/api/audio.mdx +36 -0
- package/docs/api/behavior-components.mdx +164 -0
- package/docs/api/capture-recording.mdx +103 -0
- package/docs/api/debug-ui-scene-editor.mdx +249 -0
- package/docs/api/face-actions.mdx +104 -0
- package/docs/api/face-attribute-expression.mdx +87 -0
- package/docs/api/face-binding-anchors.mdx +53 -0
- package/docs/api/face-effects.mdx +19 -0
- package/docs/api/face-info.mdx +112 -0
- package/docs/api/face-mesh.mdx +98 -0
- package/docs/api/face-painting.mdx +33 -0
- package/docs/api/face-retouching.mdx +73 -0
- package/docs/api/material-render-state.mdx +132 -0
- package/docs/api/runtime-3d.mdx +311 -0
- package/docs/api/scene-object-store.mdx +213 -0
- package/docs/api/screen-image.mdx +41 -0
- package/docs/api/screen-shape.mdx +64 -0
- package/docs/api/screen-space-coordinates.mdx +41 -0
- package/docs/api/screen-text.mdx +39 -0
- package/docs/api/texture-sources.mdx +55 -0
- package/docs/api/trackers.mdx +46 -0
- package/docs/docs.json +70 -0
- package/docs/examples/scene-object-recipes.mdx +194 -0
- package/docs/guides/harness-debug-ui.mdx +139 -0
- package/docs/inspector/hierarchy-panel.mdx +51 -0
- package/docs/inspector/object-inspector-panel.mdx +51 -0
- package/docs/introduction.mdx +121 -0
- package/docs/scene-objects/overview.mdx +127 -0
- package/docs/scene-objects/render-groups.mdx +77 -0
- package/docs/scene-objects/runtime-rendering.mdx +75 -0
- package/docs/texture-api-consumer.md +58 -0
- package/package.json +55 -7
- package/dist/components/FaceRetouch/retouchRegions.d.ts +0 -23
- package/dist/components/FaceRetouch/useFaceRetouchRegions.d.ts +0 -12
- package/dist/index-B5J4gzn2.cjs +0 -1
- package/dist/index-BVBVg96G.cjs +0 -1
- package/dist/index-DtwHElmG.js +0 -157
- package/dist/index-PtFmD6B8.cjs +0 -1
- package/dist/internal-DH5d1Xnq.cjs +0 -1
- package/dist/modules/XRClient/index.d.ts +0 -9
- package/dist/modules/index.d.ts +0 -1
- package/dist/services/ApiService.d.ts +0 -12
- package/dist/types/client.d.ts +0 -3
- package/dist/useFaceResults-cpsSmKKc.js +0 -81
- package/dist/useFaceResults-hVUJYSEY.cjs +0 -1
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Face Attribute Expression"
|
|
3
|
+
description: "Image-based angry, disgust, fear, happy, neutral, sad, surprise, and unknown classification."
|
|
4
|
+
sidebarTitle: "Face Attribute Expression"
|
|
5
|
+
icon: "face-smile"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Face Attribute Expression classifies a 112×112 RGB face crop with the OpenCV
|
|
9
|
+
MobileFaceNet INT8 model. It is independent from `FaceInfo.blendshapes`:
|
|
10
|
+
blendshapes describe physical facial movement, while this API runs image
|
|
11
|
+
inference.
|
|
12
|
+
|
|
13
|
+
```tsx
|
|
14
|
+
import { useFaceAttributeExpression, useFaceInfo } from "@vincentt-xr/sdk/tracking";
|
|
15
|
+
|
|
16
|
+
function ExpressionLogic() {
|
|
17
|
+
const face = useFaceInfo();
|
|
18
|
+
const attribute = useFaceAttributeExpression(face, {
|
|
19
|
+
targetFps: 5,
|
|
20
|
+
minConfidence: 0.55,
|
|
21
|
+
});
|
|
22
|
+
|
|
23
|
+
if (attribute.status !== "ready") return null;
|
|
24
|
+
console.log(attribute.result.expression, attribute.result.confidence);
|
|
25
|
+
console.log(attribute.result.scores.happy);
|
|
26
|
+
return null;
|
|
27
|
+
}
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The hook must run inside `XRProvider` and `XRScene`. It reuses the session's
|
|
31
|
+
tracker-input frame and caller-owned `FaceInfo`; it starts neither another
|
|
32
|
+
camera nor another face detector. The model is lazy-loaded only after a face is
|
|
33
|
+
available.
|
|
34
|
+
|
|
35
|
+
## Result
|
|
36
|
+
|
|
37
|
+
`status` is `idle`, `loading`, `ready`, or `error`. `result` is non-null when
|
|
38
|
+
ready and contains:
|
|
39
|
+
|
|
40
|
+
- `expression`: `angry`, `disgust`, `fear`, `happy`, `neutral`, `sad`,
|
|
41
|
+
`surprise`, or `unknown`.
|
|
42
|
+
- `confidence`: the winning model probability in `[0, 1]`.
|
|
43
|
+
- `scores`: probabilities for the seven model classes. It intentionally has no
|
|
44
|
+
`unknown` score because `unknown` is a threshold fallback, not a model class.
|
|
45
|
+
|
|
46
|
+
A winning score below `minConfidence` returns `unknown`. It means the model ran
|
|
47
|
+
but did not produce a confident classification.
|
|
48
|
+
|
|
49
|
+
## Options
|
|
50
|
+
|
|
51
|
+
| Option | Default | Meaning |
|
|
52
|
+
| --------------- | ------------- | ------------------------------------------- |
|
|
53
|
+
| `active` | `true` | Pause expression inference when false |
|
|
54
|
+
| `targetFps` | `5` | Maximum classifier inference rate |
|
|
55
|
+
| `minConfidence` | `0.55` | Threshold below which the result is unknown |
|
|
56
|
+
| `modelUrl` | SDK model URL | Override with an exact-contract ONNX model |
|
|
57
|
+
|
|
58
|
+
A custom model must accept float32 NCHW input `[1, 3, 112, 112]`: planar RGB,
|
|
59
|
+
normalized to `[-1, 1]`. It must return seven logits in this exact order:
|
|
60
|
+
`angry`, `disgust`, `fear`, `happy`, `neutral`, `sad`, `surprise`. A generic
|
|
61
|
+
MobileFaceNet model is not necessarily compatible.
|
|
62
|
+
|
|
63
|
+
The default validated model is approximately 1.36 MB and is fetched separately
|
|
64
|
+
from the application bundle. Its bytes are SHA-384 verified before ONNX Runtime
|
|
65
|
+
loads them. ONNX Runtime's WASM worker files load from a version-pinned CDN
|
|
66
|
+
path matching the active `onnxruntime-web` runtime, so consumers do not need a
|
|
67
|
+
Vite `?url` asset pipeline for the runtime. Those runtime files are fetched
|
|
68
|
+
from jsDelivr without SDK-managed SRI because ONNX Runtime resolves its own WASM
|
|
69
|
+
assets.
|
|
70
|
+
|
|
71
|
+
The default ONNX Runtime backend uses threaded SIMD WASM when available. Deploy
|
|
72
|
+
pages that use this hook with cross-origin isolation headers for best threaded
|
|
73
|
+
performance:
|
|
74
|
+
|
|
75
|
+
```http
|
|
76
|
+
Cross-Origin-Opener-Policy: same-origin
|
|
77
|
+
Cross-Origin-Embedder-Policy: require-corp
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Without those headers, ONNX Runtime falls back to single-threaded WASM.
|
|
81
|
+
|
|
82
|
+
Treat classifications as model estimates of visible facial expression—not
|
|
83
|
+
ground-truth emotion, intent, identity, or mental state.
|
|
84
|
+
|
|
85
|
+
Use the verified `/facial-movement-expression` page to inspect the live winning
|
|
86
|
+
class and all seven probabilities. Its real-camera model flow, including every
|
|
87
|
+
expression class, was verified on 26 July 2026.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Face Binding Anchors"
|
|
3
|
+
description: "Verified semantic face presets, custom face-local anchors, smoothing, and roll following."
|
|
4
|
+
sidebarTitle: "Face Binding Anchors"
|
|
5
|
+
icon: "locate-fixed"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Use `TrackingAnchor` to attach content to semantic face locations. Build editor
|
|
9
|
+
dropdowns from `FACE_TARGETS`; do not copy landmark indices or invent target
|
|
10
|
+
strings. Left and right use the tracked person's perspective.
|
|
11
|
+
|
|
12
|
+
## Presets
|
|
13
|
+
|
|
14
|
+
`face.center`, `face.forehead`, `face.noseTip`, `face.chin`, `face.leftEye`,
|
|
15
|
+
`face.rightEye`, `face.leftEyeOuter`, `face.rightEyeOuter`, `face.leftCheek`,
|
|
16
|
+
`face.rightCheek`, `face.mouthCenter`, `face.leftEarlobe`,
|
|
17
|
+
`face.rightEarlobe`, and `face.custom` are supported.
|
|
18
|
+
|
|
19
|
+
```tsx
|
|
20
|
+
import { FaceTracker, TrackingAnchor } from "@vincentt-xr/sdk/tracking";
|
|
21
|
+
|
|
22
|
+
<FaceTracker followFaceRoll>
|
|
23
|
+
<TrackingAnchor target="face.rightEarlobe" smoothing={15}>
|
|
24
|
+
<mesh />
|
|
25
|
+
</TrackingAnchor>
|
|
26
|
+
</FaceTracker>;
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Eye presets resolve composite eye centers; `*EyeOuter` presets are exact outer
|
|
30
|
+
corners. Earlobes use the nearest stable face-oval landmarks because MediaPipe
|
|
31
|
+
does not provide anatomical earlobe points.
|
|
32
|
+
|
|
33
|
+
## Custom points
|
|
34
|
+
|
|
35
|
+
For `face.custom`, pass `customAnchorPoint`. X and Y are clamped to `[-1, 1]`;
|
|
36
|
+
the origin is face center, X runs from the person's left to right, Y runs from
|
|
37
|
+
chin to forehead, and positive Z points toward the camera.
|
|
38
|
+
|
|
39
|
+
```tsx
|
|
40
|
+
<TrackingAnchor target="face.custom" customAnchorPoint={{ x: 0.7, y: 0.2, z: 0 }} smoothing={15}>
|
|
41
|
+
<mesh />
|
|
42
|
+
</TrackingAnchor>
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
`smoothing > 0` uses the same One Euro filter for 2D and 3D bindings. Around
|
|
46
|
+
`15` is a useful default; `0` snaps to each result. The filter resets on face
|
|
47
|
+
loss and reacquisition. Rotation is not smoothed.
|
|
48
|
+
|
|
49
|
+
For serialized 2D scenes, combine `ScreenTransform` and `TrackingAnchor` on a
|
|
50
|
+
null parent, then place modular Screen Shape/Text/Image children beneath it.
|
|
51
|
+
`followFaceRoll` controls whether the group tilts with head roll. The verified
|
|
52
|
+
`/face-binding-anchors` example exposes every preset on the editable `rose null`
|
|
53
|
+
while its hierarchy shows only that combined parent.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Face Effects"
|
|
3
|
+
description: "Overview of the face mesh, face painting, and face retouching docs."
|
|
4
|
+
sidebarTitle: "Face Effects"
|
|
5
|
+
icon: "sparkles"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
This section is split into focused pages:
|
|
9
|
+
|
|
10
|
+
- [Face Mesh](/api/face-mesh) for the base 3D face mesh, canonical renderer,
|
|
11
|
+
and face mesh material settings.
|
|
12
|
+
- [Face Painting](/api/face-painting) for the composite shader variant used to
|
|
13
|
+
paint onto the live face mesh.
|
|
14
|
+
- [Face Retouching](/api/face-retouching) for the legacy skin-retouch shader
|
|
15
|
+
and its controls.
|
|
16
|
+
|
|
17
|
+
Use `@vincentt-xr/sdk/tracking` for trackers, `FaceInfo`, Face Actions, and the
|
|
18
|
+
bare deforming `FaceMesh`. Use `@vincentt-xr/sdk/face-effects` for the modular
|
|
19
|
+
face-effect surface.
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "FaceInfo"
|
|
3
|
+
description: "Verified face pose, landmarks, bounds, scale, and expression data for effect logic."
|
|
4
|
+
sidebarTitle: "FaceInfo"
|
|
5
|
+
icon: "scan-face"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Use `FaceInfo` instead of reading raw MediaPipe results or copying landmark
|
|
9
|
+
indices into effect logic.
|
|
10
|
+
|
|
11
|
+
```tsx
|
|
12
|
+
import { hasFaceExpression, useFaceInfo } from "@vincentt-xr/sdk/tracking";
|
|
13
|
+
|
|
14
|
+
function FaceLogic() {
|
|
15
|
+
const face = useFaceInfo({ active: true });
|
|
16
|
+
if (!face) return null;
|
|
17
|
+
|
|
18
|
+
const mouthOpen = hasFaceExpression(face, "jawOpen");
|
|
19
|
+
const lookingLeft = face.pose.rotation.y > 0.2;
|
|
20
|
+
return <group visible={mouthOpen} rotation-y={lookingLeft ? 0.2 : 0} />;
|
|
21
|
+
}
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Coordinate contract
|
|
25
|
+
|
|
26
|
+
| Field | Contract |
|
|
27
|
+
| --------------- | --------------------------------------------------------- |
|
|
28
|
+
| `index` | Current result-array index, not a persistent identity |
|
|
29
|
+
| `landmarks` | Normalized tracker-input coordinates with top-left origin |
|
|
30
|
+
| `bounds` | Bounds in the same normalized coordinate system |
|
|
31
|
+
| `pose.position` | Three.js world position |
|
|
32
|
+
| `pose.rotation` | Radians: pitch X (+up), yaw Y (+left), roll Z |
|
|
33
|
+
| `pose.scale` | Smoothed uniform face-sizing heuristic |
|
|
34
|
+
| `eyeDistance` | Outer-eye distance in normalized coordinates |
|
|
35
|
+
| `blendshapes` | Typed MediaPipe category scores in `[0, 1]` |
|
|
36
|
+
|
|
37
|
+
`useFaceInfo({ faceIndex })` reads one face. `useFaceInfos({ numFaces })` reads
|
|
38
|
+
all current faces. Both share the SDK face model and support `active`,
|
|
39
|
+
`numFaces`, `targetFps`, `throttleMs`, and `holdMs`.
|
|
40
|
+
|
|
41
|
+
## Face detection lifecycle
|
|
42
|
+
|
|
43
|
+
Use `useFaceDetection` when effect logic needs face presence, face count, or
|
|
44
|
+
appearance/disappearance edges. It wraps the same shared `useFaceInfos` reader;
|
|
45
|
+
it does not start another model or polling loop.
|
|
46
|
+
|
|
47
|
+
```tsx
|
|
48
|
+
const detection = useFaceDetection({ faceIndex: 0, numFaces: 5 });
|
|
49
|
+
|
|
50
|
+
if (detection.justDetected) showEffect();
|
|
51
|
+
if (detection.detected) updateEffect(detection.face);
|
|
52
|
+
if (detection.justLost) hideEffect();
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
| Lifecycle node output | SDK result |
|
|
56
|
+
| --------------------- | -------------------------------------- |
|
|
57
|
+
| Begin | `justDetected` or `phase === "begin"` |
|
|
58
|
+
| Stay | `detected` or `phase === "stay"` |
|
|
59
|
+
| End | `justLost` or `phase === "end"` |
|
|
60
|
+
| None | `!detected` or `phase === "none"` |
|
|
61
|
+
| Face Count | `faceCount` |
|
|
62
|
+
| Which Face | `faceIndex` option and selected `face` |
|
|
63
|
+
|
|
64
|
+
The result also includes `faces`. The default `holdMs: 250` suppresses brief
|
|
65
|
+
one-frame losses before `End`; pass `holdMs: 0` for immediate disappearance.
|
|
66
|
+
Use the verified `/face-detection` page to test five face slots,
|
|
67
|
+
Begin/Stay/End/None, selected-face presence, count, and edge history. Its
|
|
68
|
+
real-camera lifecycle flow was verified on 26 July 2026.
|
|
69
|
+
|
|
70
|
+
Use `FACE_BLENDSHAPE_NAMES`, `getFaceBlendshape`, and `hasFaceExpression` rather
|
|
71
|
+
than guessing names or thresholds. The typed list includes `jawOpen`, blink,
|
|
72
|
+
and eye-wide expressions.
|
|
73
|
+
|
|
74
|
+
For semantic blink, brow, mouth, or head gestures, use the Face Actions API.
|
|
75
|
+
Face Actions represent physical movement, not inferred expression. Use
|
|
76
|
+
`useFaceAttributeExpression` when an effect needs image-based angry, disgust,
|
|
77
|
+
fear, happy, neutral, sad, surprise, or unknown classification.
|
|
78
|
+
`getFaceActionValue`, `hasFaceAction`, and `useFaceAction` own the action names,
|
|
79
|
+
pose directions, thresholds, hysteresis, and hold timing. Keep raw blendshapes
|
|
80
|
+
for custom continuous expressions that Face Actions do not cover.
|
|
81
|
+
|
|
82
|
+
## Reusable face image crops
|
|
83
|
+
|
|
84
|
+
Use `drawFaceCrop` when an image model or preview needs pixels for one detected
|
|
85
|
+
face. It uses `FaceInfo.bounds`, expands by 20% by default, preserves the output
|
|
86
|
+
pixel aspect ratio, clamps at image edges, and writes into caller-owned canvas
|
|
87
|
+
storage.
|
|
88
|
+
|
|
89
|
+
```ts
|
|
90
|
+
import { drawFaceCrop } from "@vincentt-xr/sdk/tracking";
|
|
91
|
+
|
|
92
|
+
const cropCanvas = document.createElement("canvas");
|
|
93
|
+
drawFaceCrop({
|
|
94
|
+
source: video,
|
|
95
|
+
face,
|
|
96
|
+
canvas: cropCanvas,
|
|
97
|
+
outputSize: { width: 224, height: 224 },
|
|
98
|
+
});
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
For non-face regions, import `drawNormalizedCrop` from the core package and
|
|
102
|
+
pass any normalized `{ minX, minY, maxX, maxY }` bounds. Both functions return
|
|
103
|
+
`false` when the source, bounds, output size, or 2D context is unusable. They do
|
|
104
|
+
not allocate a canvas, start tracking, mirror pixels, or run inference.
|
|
105
|
+
|
|
106
|
+
For existing low-level frame loops, import `getFaceInfo` and
|
|
107
|
+
`createFaceInfoTarget` from `@vincentt-xr/sdk/low-level`, allocate the target
|
|
108
|
+
once, and reuse it. For overlays, convert landmarks with
|
|
109
|
+
`normalizedToScreenPixels`; do not mirror or cover-map tracker points again.
|
|
110
|
+
|
|
111
|
+
The verified `/face-info` example displays all landmarks, live bounds, pose,
|
|
112
|
+
eye distance, jaw-open state, and a 3D `Axes` gizmo using SDK components.
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Face Mesh"
|
|
3
|
+
description: "Base 3D face mesh rendering, canonical mesh support, and mesh deformation settings."
|
|
4
|
+
sidebarTitle: "Face Mesh"
|
|
5
|
+
icon: "scan-face"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Use `@vincentt-xr/sdk/tracking` for the bare deforming `FaceMesh`. Use
|
|
9
|
+
`@vincentt-xr/sdk/face-effects` for the scene-object renderer, shared material
|
|
10
|
+
settings, canonical renderer, and face warp utilities.
|
|
11
|
+
|
|
12
|
+
## Bare FaceMesh
|
|
13
|
+
|
|
14
|
+
```tsx
|
|
15
|
+
import { FaceMesh, FaceTracker } from "@vincentt-xr/sdk/tracking";
|
|
16
|
+
|
|
17
|
+
<FaceTracker>
|
|
18
|
+
<FaceMesh appearance={{ color: "#00e5ff", wireframe: true }} />
|
|
19
|
+
</FaceTracker>;
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Scene-object FaceMeshRenderer
|
|
23
|
+
|
|
24
|
+
Prefer `FaceMeshRenderer` for editor/runtime scene-object integrations. It owns
|
|
25
|
+
face tracking internally through `FaceTracker`, reads live landmarks, deforms
|
|
26
|
+
the canonical face mesh, and uses the shared FaceMesh material settings.
|
|
27
|
+
|
|
28
|
+
```tsx
|
|
29
|
+
import {
|
|
30
|
+
DEFAULT_FACE_MESH_RENDERER,
|
|
31
|
+
DEFAULT_FACE_MESH_TRANSFORM,
|
|
32
|
+
FACE_MESH_MATERIAL_TYPES,
|
|
33
|
+
FACE_MESH_SIDES,
|
|
34
|
+
type FaceMeshRendererSettings,
|
|
35
|
+
} from "@vincentt-xr/sdk/face-effects";
|
|
36
|
+
import {
|
|
37
|
+
createDefaultComponent,
|
|
38
|
+
createDefaultFaceMeshRendererSettings,
|
|
39
|
+
} from "@vincentt-xr/sdk/scene-object";
|
|
40
|
+
|
|
41
|
+
const objectId = "face-mesh";
|
|
42
|
+
const transform = createDefaultComponent("Transform", objectId);
|
|
43
|
+
transform.settings.position.y = DEFAULT_FACE_MESH_TRANSFORM.position.y;
|
|
44
|
+
|
|
45
|
+
const faceMesh = createDefaultComponent("FaceMeshRenderer", objectId);
|
|
46
|
+
faceMesh.settings = {
|
|
47
|
+
...createDefaultFaceMeshRendererSettings(),
|
|
48
|
+
materialType: FACE_MESH_MATERIAL_TYPES.wireframe,
|
|
49
|
+
side: FACE_MESH_SIDES.double,
|
|
50
|
+
baseColor: "#00e5ff",
|
|
51
|
+
opacity: 0.82,
|
|
52
|
+
} satisfies FaceMeshRendererSettings;
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
`DEFAULT_FACE_MESH_RENDERER` is the serialized renderer default.
|
|
56
|
+
`mapFaceMeshRendererSettingsToProps(settings)` bridges serialized settings to
|
|
57
|
+
`CanonicalFaceMeshRendererProps`. `resolveFaceMeshRendererSettings(current,
|
|
58
|
+
patch)` is the edit-time source of truth: use it whenever material, shader,
|
|
59
|
+
texture, blend, or mask state changes.
|
|
60
|
+
|
|
61
|
+
FaceMesh `Transform` is intentionally not normal world-space `Transform3D`.
|
|
62
|
+
`SceneObjectRenderer` passes an authored `Transform` into `FaceMeshRenderer`,
|
|
63
|
+
and FaceMesh applies supported values in MediaPipe landmark space before camera
|
|
64
|
+
projection. Supported values are position and uniform scale only. Rotation and
|
|
65
|
+
non-uniform scale are ignored until the SDK has a real object-space face solve.
|
|
66
|
+
Authored `position.y` is Y-up; MediaPipe image-space Y is down, so the renderer
|
|
67
|
+
subtracts Y while adding X/Z. `DEFAULT_FACE_MESH_TRANSFORM.position.y` is
|
|
68
|
+
`0.001` as a default authoring bias; set it to `0` to opt out.
|
|
69
|
+
|
|
70
|
+
## Canonical Renderer
|
|
71
|
+
|
|
72
|
+
`CanonicalFaceMeshRenderer` remains public for low-level experiments and
|
|
73
|
+
backwards compatibility. New scene-object work should use `FaceMeshRenderer`.
|
|
74
|
+
|
|
75
|
+
```tsx
|
|
76
|
+
import { CanonicalFaceMeshRenderer, FACE_WARP_3D_PRESET } from "@vincentt-xr/sdk/face-effects";
|
|
77
|
+
|
|
78
|
+
<CanonicalFaceMeshRenderer
|
|
79
|
+
materialType="shader"
|
|
80
|
+
shaderPreset="originalFace"
|
|
81
|
+
textureMap="none"
|
|
82
|
+
alphaMap="canonical"
|
|
83
|
+
faceWarp3D={FACE_WARP_3D_PRESET}
|
|
84
|
+
/>;
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`useCanonicalFaceMesh` loads and caches the canonical mesh.
|
|
88
|
+
`parseCanonicalFaceModel` parses canonical-compatible OBJ text.
|
|
89
|
+
`canonicalFaceTextureUrl`, `canonicalFaceTextureStyle2Url`, and
|
|
90
|
+
`canonicalFaceAlphaMaskUrl` are SDK-hosted runtime texture defaults.
|
|
91
|
+
|
|
92
|
+
## FaceWarp3D
|
|
93
|
+
|
|
94
|
+
`FACE_WARP_3D_PRESET`, `DEFAULT_FACE_WARP_3D_SETTINGS`, and
|
|
95
|
+
`FaceWarp3DSettings` cover face, forehead, cheek, eye, nose, mouth, jaw, and
|
|
96
|
+
chin proportions with normalized values around zero. Use `faceWarp3D` on
|
|
97
|
+
`CanonicalFaceMeshRenderer` or `FaceMeshRenderer` settings instead of writing a
|
|
98
|
+
second landmark deformation path.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Face Painting"
|
|
3
|
+
description: "Composite shader face painting on the live 3D face mesh."
|
|
4
|
+
sidebarTitle: "Face Painting"
|
|
5
|
+
icon: "paintbrush"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Use `FaceMeshRenderer` with the `composite` shader preset when you want the
|
|
9
|
+
live face mesh to act like a painted surface.
|
|
10
|
+
|
|
11
|
+
```tsx
|
|
12
|
+
import {
|
|
13
|
+
FACE_MESH_SHADER_PRESETS,
|
|
14
|
+
resolveFaceMeshRendererSettings,
|
|
15
|
+
} from "@vincentt-xr/sdk/face-effects";
|
|
16
|
+
|
|
17
|
+
const next = resolveFaceMeshRendererSettings(current, {
|
|
18
|
+
shaderPreset: FACE_MESH_SHADER_PRESETS.composite,
|
|
19
|
+
});
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Shader presets are `originalFace` and `composite`. Selecting `composite`
|
|
23
|
+
applies the SDK composite defaults: canonical texture, hard-light blend,
|
|
24
|
+
`blendOpacity: 0.85`, and `compositeShading: true`.
|
|
25
|
+
|
|
26
|
+
`maskTextureSource` is `solid`, `gradient`, `imageUrl`, or `mediaUrl`. Solid and
|
|
27
|
+
gradient masks use `maskColor`; URL sources use `maskSourceUrl`. `maskOpacity`
|
|
28
|
+
is `0..1`. Use `supportsTexturePanel` and `supportsBlendMode` to decide which
|
|
29
|
+
inspector controls to show for the active material state.
|
|
30
|
+
|
|
31
|
+
Geometry visibility is grouped by FaceMesh part: left eye, right eye, mouth,
|
|
32
|
+
and nose. Use `CANONICAL_FACE_MESH_GEOMETRY_SPECS` when an inspector needs the
|
|
33
|
+
runtime group order instead of hardcoding another list.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Face Retouching"
|
|
3
|
+
description: "Legacy skin-retouch shader, controls, and multi-face usage."
|
|
4
|
+
sidebarTitle: "Face Retouching"
|
|
5
|
+
icon: "sparkles"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Retouch exports the reusable legacy skin-retouch component and its SDK-owned
|
|
9
|
+
control schema: `FaceRetouch`, `FaceRetouchControls`,
|
|
10
|
+
`FACE_RETOUCH_DEFAULT_CONTROLS`, `FACE_RETOUCH_SETTINGS_FIELDS`,
|
|
11
|
+
`resolveFaceRetouchControls`, and `patchFaceRetouchControls`.
|
|
12
|
+
|
|
13
|
+
```tsx
|
|
14
|
+
import {
|
|
15
|
+
FaceRetouch,
|
|
16
|
+
resolveFaceRetouchControls,
|
|
17
|
+
type FaceRetouchControls,
|
|
18
|
+
} from "@vincentt-xr/sdk/face-effects";
|
|
19
|
+
|
|
20
|
+
const controls = resolveFaceRetouchControls({
|
|
21
|
+
skinSmoothing: 0.55,
|
|
22
|
+
skinRealism: 0.62,
|
|
23
|
+
blemishReduction: 0.35,
|
|
24
|
+
previewMask: false,
|
|
25
|
+
});
|
|
26
|
+
|
|
27
|
+
<FaceRetouch {...controls} />;
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Controls are intentionally minimal:
|
|
31
|
+
|
|
32
|
+
| Control | Range | Effect |
|
|
33
|
+
| ------------------ | ------- | ---------------------------------------- |
|
|
34
|
+
| `skinSmoothing` | `0..1` | Overall smoothing strength |
|
|
35
|
+
| `skinRealism` | `0..1` | Detail preservation after smoothing |
|
|
36
|
+
| `blemishReduction` | `0..1` | Local red/dark/high-contrast spot repair |
|
|
37
|
+
| `previewMask` | boolean | Shows the landmark retouch mask |
|
|
38
|
+
|
|
39
|
+
Use `FACE_RETOUCH_SETTINGS_FIELDS` with the generic debug `SettingsPanel`
|
|
40
|
+
instead of copying slider labels, ranges, or defaults into app pages. For
|
|
41
|
+
examples, prefer the debug `FaceRetouchSettingsPanel` wrapper:
|
|
42
|
+
|
|
43
|
+
```tsx
|
|
44
|
+
import { FaceRetouchSettingsPanel } from "@vincentt-xr/sdk/debug-ui";
|
|
45
|
+
|
|
46
|
+
<FaceRetouchSettingsPanel value={controls} onChange={setControls} />;
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`clipMinX` and `clipMaxX` are optional horizontal UV clipping props in `0..1`
|
|
50
|
+
space. They default to the full frame. The legacy compare demo uses
|
|
51
|
+
`clipMinX={0.5}` so retouch only draws on the right half.
|
|
52
|
+
|
|
53
|
+
For multiple people, render one `FaceRetouch` per person and pass `faceIndex`.
|
|
54
|
+
`numFaces` defaults to `faceIndex + 1`, so `<FaceRetouch faceIndex={1} />`
|
|
55
|
+
requests tracking for at least two faces. Multiple passes are transparent
|
|
56
|
+
outside their own face mask, so they can be stacked over the same camera
|
|
57
|
+
background.
|
|
58
|
+
|
|
59
|
+
```tsx
|
|
60
|
+
const faceCount = 2;
|
|
61
|
+
|
|
62
|
+
{
|
|
63
|
+
Array.from({ length: faceCount }, (_, faceIndex) => (
|
|
64
|
+
<FaceRetouch
|
|
65
|
+
key={faceIndex}
|
|
66
|
+
{...controls}
|
|
67
|
+
faceIndex={faceIndex}
|
|
68
|
+
numFaces={faceCount}
|
|
69
|
+
renderOrder={-1000 + faceIndex}
|
|
70
|
+
/>
|
|
71
|
+
));
|
|
72
|
+
}
|
|
73
|
+
```
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Material Render State API"
|
|
3
|
+
description: "MeshRenderer material draw-state controls: depth, color mask, culling, blending, and polygon offset."
|
|
4
|
+
sidebarTitle: "Material Render State"
|
|
5
|
+
icon: "sliders"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Material render state lives on `MeshRendererSettings.material.renderState`.
|
|
9
|
+
|
|
10
|
+
```ts
|
|
11
|
+
import {
|
|
12
|
+
DEFAULT_MESH_RENDERER_RENDER_STATE,
|
|
13
|
+
normalizeMeshRendererRenderState,
|
|
14
|
+
type MeshRendererRenderState,
|
|
15
|
+
} from "@vincentt-xr/sdk";
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Type definition
|
|
19
|
+
|
|
20
|
+
```ts
|
|
21
|
+
type MeshRendererRenderState = {
|
|
22
|
+
depthTest: boolean;
|
|
23
|
+
depthWrite: boolean;
|
|
24
|
+
depthFunc:
|
|
25
|
+
| "never"
|
|
26
|
+
| "less"
|
|
27
|
+
| "equal"
|
|
28
|
+
| "lessEqual"
|
|
29
|
+
| "greater"
|
|
30
|
+
| "notEqual"
|
|
31
|
+
| "greaterEqual"
|
|
32
|
+
| "always";
|
|
33
|
+
colorWrite: boolean;
|
|
34
|
+
colorMask: {
|
|
35
|
+
r: boolean;
|
|
36
|
+
g: boolean;
|
|
37
|
+
b: boolean;
|
|
38
|
+
a: boolean;
|
|
39
|
+
};
|
|
40
|
+
cullMode: "back" | "front" | "off";
|
|
41
|
+
blendMode: "normal" | "additive" | "subtractive" | "multiply" | "none";
|
|
42
|
+
polygonOffset: {
|
|
43
|
+
enabled: boolean;
|
|
44
|
+
factor: number;
|
|
45
|
+
units: number;
|
|
46
|
+
};
|
|
47
|
+
};
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Defaults
|
|
51
|
+
|
|
52
|
+
```ts
|
|
53
|
+
const DEFAULT_MESH_RENDERER_RENDER_STATE = {
|
|
54
|
+
depthTest: true,
|
|
55
|
+
depthWrite: true,
|
|
56
|
+
depthFunc: "lessEqual",
|
|
57
|
+
colorWrite: true,
|
|
58
|
+
colorMask: { r: true, g: true, b: true, a: true },
|
|
59
|
+
cullMode: "back",
|
|
60
|
+
blendMode: "normal",
|
|
61
|
+
polygonOffset: { enabled: false, factor: 0, units: 0 },
|
|
62
|
+
};
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Use `normalizeMeshRendererRenderState(partial)` when reading serialized scenes
|
|
66
|
+
that may omit nested defaults.
|
|
67
|
+
|
|
68
|
+
## Usage
|
|
69
|
+
|
|
70
|
+
```ts
|
|
71
|
+
const meshRenderer = createDefaultComponent("MeshRenderer", object.id);
|
|
72
|
+
|
|
73
|
+
meshRenderer.settings.material.renderState = {
|
|
74
|
+
depthTest: true,
|
|
75
|
+
depthWrite: false,
|
|
76
|
+
depthFunc: "lessEqual",
|
|
77
|
+
colorWrite: true,
|
|
78
|
+
colorMask: { r: true, g: false, b: false, a: true },
|
|
79
|
+
cullMode: "off",
|
|
80
|
+
blendMode: "additive",
|
|
81
|
+
polygonOffset: {
|
|
82
|
+
enabled: true,
|
|
83
|
+
factor: -1,
|
|
84
|
+
units: -1,
|
|
85
|
+
},
|
|
86
|
+
};
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
## Field behavior
|
|
90
|
+
|
|
91
|
+
| Field | Behavior |
|
|
92
|
+
| --------------- | --------------------------------------------------------------------------------------- |
|
|
93
|
+
| `depthTest` | Enables/disables depth testing before drawing the material. |
|
|
94
|
+
| `depthWrite` | Enables/disables writing this material into the depth buffer. |
|
|
95
|
+
| `depthFunc` | Selects the depth comparison function. |
|
|
96
|
+
| `colorWrite` | All-channel color write gate. |
|
|
97
|
+
| `colorMask` | Per-channel RGBA color write gates. |
|
|
98
|
+
| `cullMode` | `"back"` renders front faces, `"front"` renders back faces, `"off"` renders both sides. |
|
|
99
|
+
| `blendMode` | Maps to Three.js built-in blending modes. |
|
|
100
|
+
| `polygonOffset` | Offsets depth to reduce z-fighting or tune layering. |
|
|
101
|
+
|
|
102
|
+
`colorMask` is not a plain Three.js material field. The SDK applies it as scoped
|
|
103
|
+
WebGL draw state around each mesh render and restores the previous color mask
|
|
104
|
+
after the object renders.
|
|
105
|
+
|
|
106
|
+
## Asset-backed meshes
|
|
107
|
+
|
|
108
|
+
When `MeshRendererSettings.assetUrl` is set, the GLTF scene is cloned and mesh
|
|
109
|
+
materials are cloned before SDK material/render-state overrides are applied.
|
|
110
|
+
This prevents two objects using the same asset URL from mutating the shared Drei
|
|
111
|
+
GLTF cache.
|
|
112
|
+
|
|
113
|
+
## Debug UI
|
|
114
|
+
|
|
115
|
+
`MeshRendererSettingsPanel` exposes render-state controls:
|
|
116
|
+
|
|
117
|
+
- Depth Test
|
|
118
|
+
- Depth Write
|
|
119
|
+
- Depth Function
|
|
120
|
+
- Color Write
|
|
121
|
+
- Mask R/G/B/A
|
|
122
|
+
- Cull Mode
|
|
123
|
+
- Blend Mode
|
|
124
|
+
- Polygon Offset enabled/factor/units
|
|
125
|
+
|
|
126
|
+
All edits should route back to the selected object's `MeshRenderer` component
|
|
127
|
+
through `onComponentChange`.
|
|
128
|
+
|
|
129
|
+
## Explicit non-goal
|
|
130
|
+
|
|
131
|
+
`instanceCount` is not part of material render state. It belongs to mesh
|
|
132
|
+
instancing and should be added as a separate mesh/rendering API when needed.
|