neuromeka-vfm 0.1.4__tar.gz → 0.1.5__tar.gz

{neuromeka_vfm-0.1.4/src/neuromeka_vfm.egg-info → neuromeka_vfm-0.1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: neuromeka_vfm
-Version: 0.1.4
+Version: 0.1.5
 Summary: Client utilities for Neuromeka VFM FoundationPose RPC (upload meshes, call server)
 Author: Neuromeka
 License: MIT License
@@ -41,6 +41,8 @@ Requires-Dist: numpy
 Requires-Dist: pyzmq
 Requires-Dist: paramiko
 Requires-Dist: av
+Requires-Dist: trimesh
+Requires-Dist: tqdm
 Dynamic: license-file
 
 # neuromeka_vfm
@@ -89,12 +91,109 @@ seg.register_first_frame(frame=first_rgb,
                         use_image_prompt=False)
 
 # 등록된 mask에 대한 SAM2 tracking
-masks = seg.get_next(next_rgb)
+resp = seg.get_next(next_rgb)
+if isinstance(resp, dict) and resp.get("result") == "ERROR":
+    print(f"Tracking error: {resp.get('message')}")
+    seg.reset()
+else:
+    masks = resp
+
+# Segmentation 설정/모델 선택 (nrmk_realtime_segmentation v0.2+)
+caps = seg.get_capabilities()["data"]
+current = seg.get_config()["data"]
+seg.set_config(
+    {
+        "grounding_dino": {
+            "backbone": "Swin-B",        # Swin-T | Swin-B
+            "box_threshold": 0.35,
+            "text_threshold": 0.25,
+        },
+        "dino_detection": {
+            "threshold": 0.5,
+            "target_multiplier": 25,
+            "img_multiplier": 50,
+            "background_threshold": -1.0,
+            "final_erosion_count": 10,
+            "segment_min_size": 20,
+        },
+        "sam2": {
+            "model": "facebook/sam2.1-hiera-large",
+            "use_legacy": False,
+            "compile": False,
+            "offload_state_to_cpu": False,
+            "offload_video_to_cpu": False,
+        },
+    }
+)
+
+# SAM2 object 제거 (v0.2+, use_legacy=False에서만 지원)
+seg.remove_object("cup_0")
 
 
 seg.close()
 ```
 
+#### Segmentation v0.2 설정 요약 (defaults/choices)
+`seg.get_capabilities()` 결과는 서버 설정에 따라 달라질 수 있습니다. 아래는 v0.2 기본값입니다.
+```yaml
+grounding_dino:
+  backbone:
+    choices:
+      - Swin-B
+      - Swin-T
+    default: Swin-T
+  box_threshold:
+    default: 0.35
+    min: 0.0
+    max: 1.0
+  text_threshold:
+    default: 0.25
+    min: 0.0
+    max: 1.0
+
+dino_detection:
+  threshold:
+    default: 0.5
+  target_multiplier:
+    default: 25
+  img_multiplier:
+    default: 50
+  background_threshold:
+    default: -1.0
+  final_erosion_count:
+    default: 10
+  segment_min_size:
+    default: 20
+
+sam2:
+  model:
+    choices:
+      - facebook/sam2-hiera-base-plus
+      - facebook/sam2-hiera-large
+      - facebook/sam2-hiera-small
+      - facebook/sam2-hiera-tiny
+      - facebook/sam2.1-hiera-base-plus
+      - facebook/sam2.1-hiera-large
+      - facebook/sam2.1-hiera-small
+      - facebook/sam2.1-hiera-tiny
+    default: facebook/sam2.1-hiera-large
+  use_legacy:
+    default: false
+  compile:
+    default: false
+  offload_state_to_cpu:
+    default: false
+  offload_video_to_cpu:
+    default: false
+```
+
+#### Segmentation v0.2 주의사항/변경사항
+- SAM2 VRAM 추정 실패 시 `seg.get_next()`가 `{"result":"ERROR"}`로 반환될 수 있으니 에러 처리 후 `reset`/재등록을 권장합니다.
+- SAM2 `compile=True`는 첫 프레임 등록 및 `reset`이 느려질 수 있습니다.
+- SAM2 CPU offloading은 `offload_state_to_cpu=True`와 `offload_video_to_cpu=True`를 함께 설정할 때 효과가 큽니다(legacy 모드에서는 `offload_video_to_cpu` 미지원).
+- SAM2 `remove_object`는 `use_legacy=False`에서만 지원됩니다.
+- GroundingDINO는 Swin-B 백본이 추가되었고, 프롬프트 토큰 병합 이슈가 수정되었습니다.
+
 ### Pose Estimation
 
 **Mesh 파일 업로드**: 등록/인식하고자 하는 mesh 파일 (stl)을 호스트PC의 '/opt/meshes/' 경로에 업로드 (직접 SSH 통해 파일을 옮겨도 됨)

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5}/README.md

@@ -44,12 +44,109 @@ seg.register_first_frame(frame=first_rgb,
                         use_image_prompt=False)
 
 # 등록된 mask에 대한 SAM2 tracking
-masks = seg.get_next(next_rgb)
+resp = seg.get_next(next_rgb)
+if isinstance(resp, dict) and resp.get("result") == "ERROR":
+    print(f"Tracking error: {resp.get('message')}")
+    seg.reset()
+else:
+    masks = resp
+
+# Segmentation 설정/모델 선택 (nrmk_realtime_segmentation v0.2+)
+caps = seg.get_capabilities()["data"]
+current = seg.get_config()["data"]
+seg.set_config(
+    {
+        "grounding_dino": {
+            "backbone": "Swin-B",        # Swin-T | Swin-B
+            "box_threshold": 0.35,
+            "text_threshold": 0.25,
+        },
+        "dino_detection": {
+            "threshold": 0.5,
+            "target_multiplier": 25,
+            "img_multiplier": 50,
+            "background_threshold": -1.0,
+            "final_erosion_count": 10,
+            "segment_min_size": 20,
+        },
+        "sam2": {
+            "model": "facebook/sam2.1-hiera-large",
+            "use_legacy": False,
+            "compile": False,
+            "offload_state_to_cpu": False,
+            "offload_video_to_cpu": False,
+        },
+    }
+)
+
+# SAM2 object 제거 (v0.2+, use_legacy=False에서만 지원)
+seg.remove_object("cup_0")
 
 
 seg.close()
 ```
 
+#### Segmentation v0.2 설정 요약 (defaults/choices)
+`seg.get_capabilities()` 결과는 서버 설정에 따라 달라질 수 있습니다. 아래는 v0.2 기본값입니다.
+```yaml
+grounding_dino:
+  backbone:
+    choices:
+      - Swin-B
+      - Swin-T
+    default: Swin-T
+  box_threshold:
+    default: 0.35
+    min: 0.0
+    max: 1.0
+  text_threshold:
+    default: 0.25
+    min: 0.0
+    max: 1.0
+
+dino_detection:
+  threshold:
+    default: 0.5
+  target_multiplier:
+    default: 25
+  img_multiplier:
+    default: 50
+  background_threshold:
+    default: -1.0
+  final_erosion_count:
+    default: 10
+  segment_min_size:
+    default: 20
+
+sam2:
+  model:
+    choices:
+      - facebook/sam2-hiera-base-plus
+      - facebook/sam2-hiera-large
+      - facebook/sam2-hiera-small
+      - facebook/sam2-hiera-tiny
+      - facebook/sam2.1-hiera-base-plus
+      - facebook/sam2.1-hiera-large
+      - facebook/sam2.1-hiera-small
+      - facebook/sam2.1-hiera-tiny
+    default: facebook/sam2.1-hiera-large
+  use_legacy:
+    default: false
+  compile:
+    default: false
+  offload_state_to_cpu:
+    default: false
+  offload_video_to_cpu:
+    default: false
+```
+
+#### Segmentation v0.2 주의사항/변경사항
+- SAM2 VRAM 추정 실패 시 `seg.get_next()`가 `{"result":"ERROR"}`로 반환될 수 있으니 에러 처리 후 `reset`/재등록을 권장합니다.
+- SAM2 `compile=True`는 첫 프레임 등록 및 `reset`이 느려질 수 있습니다.
+- SAM2 CPU offloading은 `offload_state_to_cpu=True`와 `offload_video_to_cpu=True`를 함께 설정할 때 효과가 큽니다(legacy 모드에서는 `offload_video_to_cpu` 미지원).
+- SAM2 `remove_object`는 `use_legacy=False`에서만 지원됩니다.
+- GroundingDINO는 Swin-B 백본이 추가되었고, 프롬프트 토큰 병합 이슈가 수정되었습니다.
+
 ### Pose Estimation
 
 **Mesh 파일 업로드**: 등록/인식하고자 하는 mesh 파일 (stl)을 호스트PC의 '/opt/meshes/' 경로에 업로드 (직접 SSH 통해 파일을 옮겨도 됨)

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "neuromeka_vfm"
-version = "0.1.4"
+version = "0.1.5"
 description = "Client utilities for Neuromeka VFM FoundationPose RPC (upload meshes, call server)"
 readme = "README.md"
 requires-python = ">=3.8"
@@ -13,7 +13,9 @@ dependencies = [
     "numpy",
     "pyzmq",
     "paramiko",
-    "av",    
+    "av",
+    "trimesh",
+    "tqdm",
 ]
 authors = [{name = "Neuromeka"}]
 classifiers = [

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5}/src/neuromeka_vfm/__init__.py

@@ -2,6 +2,7 @@ from .pose_estimation import PoseEstimation, FoundationPoseClient
 from .upload_mesh import upload_mesh
 from .segmentation import Segmentation, NrmkRealtimeSegmentation
 from .compression import STRATEGIES as SEGMENTATION_COMPRESSION_STRATEGIES
+from .grasp_gen import GraspPoseGeneration
 
 __all__ = [
     "PoseEstimation",
@@ -10,4 +11,5 @@ __all__ = [
     "Segmentation",
     "NrmkRealtimeSegmentation",
     "SEGMENTATION_COMPRESSION_STRATEGIES",
+    "GraspPoseGeneration",
 ]

neuromeka_vfm-0.1.5/src/neuromeka_vfm/grasp_gen.py

@@ -0,0 +1,79 @@
+from typing import Tuple
+
+import numpy as np
+import trimesh
+
+from . import point_cloud_utils
+
+
+class GraspPoseGeneration:
+    """
+    Wrapper class for point cloud utilities used in grasp pose workflows.
+    """
+
+    def knn_points(self, X: np.ndarray, K: int, norm: int):
+        return point_cloud_utils.knn_points(X=X, K=K, norm=norm)
+
+    def point_cloud_outlier_removal(
+        self, obj_pc: np.ndarray, threshold: float = 0.014, K: int = 20
+    ) -> Tuple[np.ndarray, np.ndarray]:
+        return point_cloud_utils.point_cloud_outlier_removal(
+            obj_pc=obj_pc, threshold=threshold, K=K
+        )
+
+    def point_cloud_outlier_removal_with_color(
+        self,
+        obj_pc: np.ndarray,
+        obj_pc_color: np.ndarray,
+        threshold: float = 0.014,
+        K: int = 20,
+    ) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+        return point_cloud_utils.point_cloud_outlier_removal_with_color(
+            obj_pc=obj_pc,
+            obj_pc_color=obj_pc_color,
+            threshold=threshold,
+            K=K,
+        )
+
+    def depth_and_segmentation_to_point_clouds(
+        self,
+        depth_image: np.ndarray,
+        segmentation_mask: np.ndarray,
+        fx: float,
+        fy: float,
+        cx: float,
+        cy: float,
+        rgb_image: np.ndarray = None,
+        target_object_id: int = 1,
+        remove_object_from_scene: bool = False,
+    ) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+        return point_cloud_utils.depth_and_segmentation_to_point_clouds(
+            depth_image=depth_image,
+            segmentation_mask=segmentation_mask,
+            fx=fx,
+            fy=fy,
+            cx=cx,
+            cy=cy,
+            rgb_image=rgb_image,
+            target_object_id=target_object_id,
+            remove_object_from_scene=remove_object_from_scene,
+        )
+
+    def filter_colliding_grasps(
+        self,
+        scene_pc: np.ndarray,
+        grasp_poses: np.ndarray,
+        gripper_collision_mesh: trimesh.Trimesh,
+        collision_threshold: float = 0.002,
+        num_collision_samples: int = 2000,
+    ) -> np.ndarray:
+        return point_cloud_utils.filter_colliding_grasps(
+            scene_pc=scene_pc,
+            grasp_poses=grasp_poses,
+            gripper_collision_mesh=gripper_collision_mesh,
+            collision_threshold=collision_threshold,
+            num_collision_samples=num_collision_samples,
+        )
+
+
+__all__ = ["GraspPoseGeneration"]

neuromeka_vfm-0.1.5/src/neuromeka_vfm/point_cloud_utils.py

@@ -0,0 +1,377 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+import logging
+from typing import Tuple, Dict
+
+import numpy as np
+import trimesh
+import trimesh.transformations as tra
+from tqdm import tqdm
+
+logger = logging.getLogger(__name__)
+
+
+def _pairwise_distances(X: np.ndarray, Y: np.ndarray, norm: int) -> np.ndarray:
+    if norm == 1:
+        return np.sum(np.abs(X[:, None, :] - Y[None, :, :]), axis=2)
+    if norm == 2:
+        diff = X[:, None, :] - Y[None, :, :]
+        return np.sqrt(np.sum(diff * diff, axis=2))
+    diff = X[:, None, :] - Y[None, :, :]
+    return np.linalg.norm(diff, ord=norm, axis=2)
+
+
+def knn_points(X: np.ndarray, K: int, norm: int):
+    """
+    Computes the K-nearest neighbors for each point in the point cloud X.
+
+    Args:
+        X: (N, 3) array representing the point cloud.
+        K: Number of nearest neighbors.
+
+    Returns:
+        dists: (N, K) array containing distances to the K nearest neighbors.
+        idxs: (N, K) array containing indices of the K nearest neighbors.
+    """
+    X = np.asarray(X, dtype=np.float32)
+    if X.ndim != 2 or X.shape[1] != 3:
+        raise ValueError("X must be a (N, 3) array")
+    if K <= 0:
+        raise ValueError("K must be positive")
+    N, _ = X.shape
+    if K >= N:
+        raise ValueError("K must be smaller than number of points")
+
+    dists_out = np.empty((N, K), dtype=np.float32)
+    idxs_out = np.empty((N, K), dtype=np.int64)
+
+    max_bytes = 64 * 1024 * 1024
+    bytes_per_row = N * X.dtype.itemsize
+    chunk_size = max(1, min(N, max_bytes // max(bytes_per_row, 1)))
+
+    for start in range(0, N, chunk_size):
+        end = min(start + chunk_size, N)
+        chunk = X[start:end]
+        dist_matrix = _pairwise_distances(chunk, X, norm=norm)
+
+        row_idx = np.arange(end - start)
+        col_idx = row_idx + start
+        dist_matrix[row_idx, col_idx] = np.inf
+
+        idx_part = np.argpartition(dist_matrix, K, axis=1)[:, :K]
+        dist_part = np.take_along_axis(dist_matrix, idx_part, axis=1)
+        order = np.argsort(dist_part, axis=1)
+        idxs = np.take_along_axis(idx_part, order, axis=1)
+        dists = np.take_along_axis(dist_part, order, axis=1)
+
+        dists_out[start:end] = dists
+        idxs_out[start:end] = idxs
+
+    return dists_out, idxs_out
+
+
+def point_cloud_outlier_removal(
+    obj_pc: np.ndarray, threshold: float = 0.014, K: int = 20
+) -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Remove outliers from a point cloud. K-nearest neighbors is used to compute
+    the distance to the nearest neighbor for each point. If the distance is
+    greater than a threshold, the point is considered an outlier and removed.
+
+    Args:
+        obj_pc (np.ndarray): (N, 3) array representing the point cloud.
+        threshold (float): Distance threshold for outlier detection. Points with mean distance to
+            K nearest neighbors greater than this threshold are removed.
+        K (int): Number of nearest neighbors to consider for outlier detection.
+
+    Returns:
+        Tuple[np.ndarray, np.ndarray]: Tuple containing filtered and removed point clouds.
+    """
+    obj_pc = np.asarray(obj_pc, dtype=np.float32)
+    if obj_pc.ndim != 2 or obj_pc.shape[1] != 3:
+        raise ValueError("obj_pc must be a (N, 3) array")
+
+    nn_dists, _ = knn_points(obj_pc, K=K, norm=1)
+
+    mask = nn_dists.mean(axis=1) < threshold
+    filtered_pc = obj_pc[mask]
+    removed_pc = obj_pc[~mask]
+
+    logger.info(
+        "Removed %s points from point cloud",
+        obj_pc.shape[0] - filtered_pc.shape[0],
+    )
+    return filtered_pc, removed_pc
+
+
+def point_cloud_outlier_removal_with_color(
+    obj_pc: np.ndarray,
+    obj_pc_color: np.ndarray,
+    threshold: float = 0.014,
+    K: int = 20,
+) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+    """
+    Remove outliers from a point cloud with colors.
+
+    Args:
+        obj_pc (np.ndarray): (N, 3) array representing the point cloud.
+        obj_pc_color (np.ndarray): (N, 3) array representing the point cloud color.
+        threshold (float): Distance threshold for outlier detection. Points with mean distance to
+            K nearest neighbors greater than this threshold are removed.
+        K (int): Number of nearest neighbors to consider for outlier detection.
+
+    Returns:
+        Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: Tuple containing filtered and
+            removed point clouds and colors.
+    """
+    obj_pc = np.asarray(obj_pc, dtype=np.float32)
+    obj_pc_color = np.asarray(obj_pc_color, dtype=np.float32)
+    if obj_pc.ndim != 2 or obj_pc.shape[1] != 3:
+        raise ValueError("obj_pc must be a (N, 3) array")
+    if obj_pc_color.shape != obj_pc.shape:
+        raise ValueError("obj_pc_color must match obj_pc shape")
+
+    nn_dists, _ = knn_points(obj_pc, K=K, norm=1)
+
+    mask = nn_dists.mean(axis=1) < threshold
+    filtered_pc = obj_pc[mask]
+    removed_pc = obj_pc[~mask]
+
+    filtered_pc_color = obj_pc_color[mask]
+    removed_pc_color = obj_pc_color[~mask]
+
+    logger.info(
+        "Removed %s points from point cloud",
+        obj_pc.shape[0] - filtered_pc.shape[0],
+    )
+    return filtered_pc, removed_pc, filtered_pc_color, removed_pc_color
+
+
+def depth2points(
+    depth: np.array,
+    fx: int,
+    fy: int,
+    cx: int,
+    cy: int,
+    xmap: np.array = None,
+    ymap: np.array = None,
+    rgb: np.array = None,
+    seg: np.array = None,
+    mask: np.arange = None,
+) -> Dict:
+    """Compute point cloud from a depth image."""
+    if rgb is not None:
+        assert rgb.shape[0] == depth.shape[0] and rgb.shape[1] == depth.shape[1]
+    if xmap is not None:
+        assert xmap.shape[0] == depth.shape[0] and xmap.shape[1] == depth.shape[1]
+    if ymap is not None:
+        assert ymap.shape[0] == depth.shape[0] and ymap.shape[1] == depth.shape[1]
+
+    im_height, im_width = depth.shape[0], depth.shape[1]
+
+    if xmap is None or ymap is None:
+        ww = np.linspace(0, im_width - 1, im_width)
+        hh = np.linspace(0, im_height - 1, im_height)
+        xmap, ymap = np.meshgrid(ww, hh)
+
+    pt2 = depth
+    pt0 = (xmap - cx) * pt2 / fx
+    pt1 = (ymap - cy) * pt2 / fy
+
+    mask_depth = np.ma.getmaskarray(np.ma.masked_greater(pt2, 0))
+    if mask is None:
+        mask = mask_depth
+    else:
+        mask_semantic = np.ma.getmaskarray(np.ma.masked_equal(mask, 1))
+        mask = mask_depth * mask_semantic
+
+    index = mask.flatten().nonzero()[0]
+
+    pt2_valid = pt2.flatten()[:, np.newaxis].astype(np.float32)
+    pt0_valid = pt0.flatten()[:, np.newaxis].astype(np.float32)
+    pt1_valid = pt1.flatten()[:, np.newaxis].astype(np.float32)
+    pc_xyz = np.concatenate((pt0_valid, pt1_valid, pt2_valid), axis=1)
+    if rgb is not None:
+        r = rgb[:, :, 0].flatten()[:, np.newaxis]
+        g = rgb[:, :, 1].flatten()[:, np.newaxis]
+        b = rgb[:, :, 2].flatten()[:, np.newaxis]
+        pc_rgb = np.concatenate((r, g, b), axis=1)
+    else:
+        pc_rgb = None
+
+    if seg is not None:
+        pc_seg = seg.flatten()[:, np.newaxis]
+    else:
+        pc_seg = None
+
+    return {"xyz": pc_xyz, "rgb": pc_rgb, "seg": pc_seg, "index": index}
+
+
+def depth_and_segmentation_to_point_clouds(
+    depth_image: np.ndarray,
+    segmentation_mask: np.ndarray,
+    fx: float,
+    fy: float,
+    cx: float,
+    cy: float,
+    rgb_image: np.ndarray = None,
+    target_object_id: int = 1,
+    remove_object_from_scene: bool = False,
+) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+    """
+    Convert depth image and instance segmentation mask to scene and object point clouds.
+
+    Args:
+        depth_image: HxW depth image in meters
+        segmentation_mask: HxW instance segmentation mask with integer labels
+        fx, fy, cx, cy: Camera intrinsic parameters
+        rgb_image: HxWx3 RGB image (optional, for colored point clouds)
+        target_object_id: ID of the target object in the segmentation mask
+        remove_object_from_scene: If True, removes object points from scene point cloud
+
+    Returns:
+        scene_pc: Nx3 point cloud of the entire scene (excluding object if remove_object_from_scene=True)
+        object_pc: Mx3 point cloud of the target object only
+        scene_colors: Nx3 RGB colors for scene points (or None)
+        object_colors: Mx3 RGB colors for object points (or None)
+
+    Raises:
+        ValueError: If no target object found or multiple objects detected
+    """
+    unique_ids = np.unique(segmentation_mask)
+    if target_object_id not in unique_ids:
+        raise ValueError(
+            f"Target object ID {target_object_id} not found in segmentation mask. Available IDs: {unique_ids}"
+        )
+
+    non_background_ids = unique_ids[unique_ids != 0]
+    if len(non_background_ids) > 1:
+        raise ValueError(
+            "Multiple objects detected in segmentation mask: "
+            f"{non_background_ids}. Please ensure only one object is present."
+        )
+
+    pts_data = depth2points(
+        depth=depth_image,
+        fx=int(fx),
+        fy=int(fy),
+        cx=int(cx),
+        cy=int(cy),
+        rgb=rgb_image,
+        seg=segmentation_mask,
+    )
+
+    xyz = pts_data["xyz"]
+    rgb = pts_data["rgb"]
+    seg = pts_data["seg"]
+    index = pts_data["index"]
+
+    xyz_valid = xyz[index]
+    seg_valid = seg[index] if seg is not None else None
+    rgb_valid = rgb[index] if rgb is not None else None
+
+    scene_pc = xyz_valid
+    scene_colors = rgb_valid
+
+    if seg_valid is not None:
+        object_mask = seg_valid.flatten() == target_object_id
+        object_pc = xyz_valid[object_mask]
+        object_colors = rgb_valid[object_mask] if rgb_valid is not None else None
+
+        if remove_object_from_scene:
+            scene_mask = ~object_mask
+            scene_pc = xyz_valid[scene_mask]
+            scene_colors = rgb_valid[scene_mask] if rgb_valid is not None else None
+            logger.info(
+                "Removed %s object points from scene point cloud",
+                np.sum(object_mask),
+            )
+    else:
+        raise ValueError("Segmentation data not available from depth2points")
+
+    if len(object_pc) == 0:
+        raise ValueError(f"No points found for target object ID {target_object_id}")
+
+    logger.info("Scene point cloud: %s points", len(scene_pc))
+    logger.info("Object point cloud: %s points", len(object_pc))
+
+    return scene_pc, object_pc, scene_colors, object_colors
+
+
+def filter_colliding_grasps(
+    scene_pc: np.ndarray,
+    grasp_poses: np.ndarray,
+    gripper_collision_mesh: trimesh.Trimesh,
+    collision_threshold: float = 0.002,
+    num_collision_samples: int = 2000,
+) -> np.ndarray:
+    """
+    Filter grasps based on collision detection with scene point cloud.
+
+    Args:
+        scene_pc: Nx3 scene point cloud
+        grasp_poses: Kx4x4 array of grasp poses
+        gripper_collision_mesh: Trimesh of gripper collision geometry
+        collision_threshold: Distance threshold for collision detection (meters)
+        num_collision_samples: Number of points to sample from gripper mesh surface
+
+    Returns:
+        collision_mask: K-length boolean array, True if grasp is collision-free
+    """
+    gripper_surface_points, _ = trimesh.sample.sample_surface(
+        gripper_collision_mesh, num_collision_samples
+    )
+    gripper_surface_points = np.array(gripper_surface_points)
+
+    scene_pc = np.asarray(scene_pc, dtype=np.float32)
+    collision_free_mask = []
+
+    logger.info(
+        "Checking collision for %s grasps against %s scene points...",
+        len(grasp_poses),
+        len(scene_pc),
+    )
+
+    for _, grasp_pose in tqdm(
+        enumerate(grasp_poses), total=len(grasp_poses), desc="Collision checking"
+    ):
+        gripper_points_transformed = tra.transform_points(
+            gripper_surface_points, grasp_pose
+        ).astype(np.float32, copy=False)
+
+        min_distances_sq = []
+        batch_size = 100
+        for j in range(0, len(gripper_points_transformed), batch_size):
+            batch_gripper_points = gripper_points_transformed[j : j + batch_size]
+            diff = batch_gripper_points[:, None, :] - scene_pc[None, :, :]
+            dist_sq = np.einsum("ijk,ijk->ij", diff, diff)
+            batch_min_dist_sq = np.min(dist_sq, axis=1)
+            min_distances_sq.append(batch_min_dist_sq)
+
+        all_min_distances_sq = np.concatenate(min_distances_sq, axis=0)
+        collision_detected = np.any(
+            all_min_distances_sq < collision_threshold * collision_threshold
+        )
+        collision_free_mask.append(not bool(collision_detected))
+
+    collision_free_mask = np.array(collision_free_mask)
+    num_collision_free = np.sum(collision_free_mask)
+    logger.info("Found %s/%s collision-free grasps", num_collision_free, len(grasp_poses))
+
+    return collision_free_mask
+
+
+__all__ = [
+    "knn_points",
+    "point_cloud_outlier_removal",
+    "point_cloud_outlier_removal_with_color",
+    "depth2points",
+    "depth_and_segmentation_to_point_clouds",
+    "filter_colliding_grasps",
+]

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5}/src/neuromeka_vfm/segmentation.py

@@ -16,6 +16,7 @@ class Segmentation:
         self.client = PickleClient(hostname, port)
         self.tracking_object_ids = []
         self.current_frame_masks = {}
+        self.invisible_object_ids = []
         self.image_prompt_names = set()
         if compression_strategy in STRATEGIES:
             self.compression_strategy_name = compression_strategy
@@ -51,14 +52,28 @@ class Segmentation:
         else:
             raise ValueError(f"Only valid compression strategies are {list(STRATEGIES.keys())}")
 
+    def set_config(self, config):
+        data = {"operation": "set_config", "config": config}
+        return self.client.send_data(data)
+
+    def get_capabilities(self):
+        data = {"operation": "get_capabilities"}
+        return self.client.send_data(data)
+
+    def get_config(self):
+        data = {"operation": "get_config"}
+        return self.client.send_data(data)
+
     def reset(self):
         self.first_frame_registered = False
         self.tracking_object_ids = []
         self.current_frame_masks = {}
+        self.invisible_object_ids = []
         self.encoder = None
         if self.benchmark:
             self.call_time = {"add_image_prompt": 0, "register_first_frame": 0, "get_next": 0}
             self.call_count = {"add_image_prompt": 0, "register_first_frame": 0, "get_next": 0}
+        self.client.send_data({"operation": "reset"})
 
     def add_image_prompt(self, object_name, object_image):
         if self.benchmark:
@@ -100,6 +115,9 @@ class Segmentation:
                 if np.any(mask):
                     masks[obj_id] = mask
             self.current_frame_masks = masks
+            self.invisible_object_ids = [
+                obj_id for obj_id in self.tracking_object_ids if obj_id not in masks
+            ]
             if self.benchmark:
                 self.call_time["register_first_frame"] += time.time() - start
                 self.call_count["register_first_frame"] += 1
@@ -124,21 +142,57 @@ class Segmentation:
                 if np.any(mask):
                     masks[obj_id] = mask
             self.current_frame_masks = masks
+            self.invisible_object_ids = [
+                obj_id for obj_id in self.tracking_object_ids if obj_id not in masks
+            ]
             if self.benchmark:
                 self.call_time["get_next"] += time.time() - start
                 self.call_count["get_next"] += 1
             return masks
+        if isinstance(response, dict) and any(
+            key in response for key in ("result", "status", "success", "message")
+        ):
+            if self.benchmark:
+                self.call_time["get_next"] += time.time() - start
+                self.call_count["get_next"] += 1
+            return response
         if self.benchmark:
             self.call_time["get_next"] += time.time() - start
             self.call_count["get_next"] += 1
         return None
 
+    def remove_object(self, obj_id, strict=False, need_output=False):
+        if not self.first_frame_registered:
+            print("Segmentation: register_first_frame must be called first")
+            return None
+        data = {
+            "operation": "remove_object",
+            "obj_id": obj_id,
+            "strict": strict,
+            "need_output": need_output,
+        }
+        response = self.client.send_data(data)
+        if self._is_success(response):
+            obj_ids = response.get("data", {}).get("obj_ids")
+            if obj_ids is not None:
+                self.tracking_object_ids = obj_ids
+                self.current_frame_masks = {
+                    obj_id: mask
+                    for obj_id, mask in self.current_frame_masks.items()
+                    if obj_id in obj_ids
+                }
+                self.invisible_object_ids = [
+                    obj_id for obj_id in obj_ids if obj_id not in self.current_frame_masks
+                ]
+        return response
+
     def finish(self):
         if not self.first_frame_registered:
             print("Warning: Segmentation: register_first_frame must be called first")
         self.first_frame_registered = False
         self.tracking_object_ids = []
         self.current_frame_masks = {}
+        self.invisible_object_ids = []
 
     def close(self):
         """Close underlying ZeroMQ socket/context."""

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5/src/neuromeka_vfm.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: neuromeka_vfm
-Version: 0.1.4
+Version: 0.1.5
 Summary: Client utilities for Neuromeka VFM FoundationPose RPC (upload meshes, call server)
 Author: Neuromeka
 License: MIT License
@@ -41,6 +41,8 @@ Requires-Dist: numpy
 Requires-Dist: pyzmq
 Requires-Dist: paramiko
 Requires-Dist: av
+Requires-Dist: trimesh
+Requires-Dist: tqdm
 Dynamic: license-file
 
 # neuromeka_vfm
@@ -89,12 +91,109 @@ seg.register_first_frame(frame=first_rgb,
                         use_image_prompt=False)
 
 # 등록된 mask에 대한 SAM2 tracking
-masks = seg.get_next(next_rgb)
+resp = seg.get_next(next_rgb)
+if isinstance(resp, dict) and resp.get("result") == "ERROR":
+    print(f"Tracking error: {resp.get('message')}")
+    seg.reset()
+else:
+    masks = resp
+
+# Segmentation 설정/모델 선택 (nrmk_realtime_segmentation v0.2+)
+caps = seg.get_capabilities()["data"]
+current = seg.get_config()["data"]
+seg.set_config(
+    {
+        "grounding_dino": {
+            "backbone": "Swin-B",        # Swin-T | Swin-B
+            "box_threshold": 0.35,
+            "text_threshold": 0.25,
+        },
+        "dino_detection": {
+            "threshold": 0.5,
+            "target_multiplier": 25,
+            "img_multiplier": 50,
+            "background_threshold": -1.0,
+            "final_erosion_count": 10,
+            "segment_min_size": 20,
+        },
+        "sam2": {
+            "model": "facebook/sam2.1-hiera-large",
+            "use_legacy": False,
+            "compile": False,
+            "offload_state_to_cpu": False,
+            "offload_video_to_cpu": False,
+        },
+    }
+)
+
+# SAM2 object 제거 (v0.2+, use_legacy=False에서만 지원)
+seg.remove_object("cup_0")
 
 
 seg.close()
 ```
 
+#### Segmentation v0.2 설정 요약 (defaults/choices)
+`seg.get_capabilities()` 결과는 서버 설정에 따라 달라질 수 있습니다. 아래는 v0.2 기본값입니다.
+```yaml
+grounding_dino:
+  backbone:
+    choices:
+      - Swin-B
+      - Swin-T
+    default: Swin-T
+  box_threshold:
+    default: 0.35
+    min: 0.0
+    max: 1.0
+  text_threshold:
+    default: 0.25
+    min: 0.0
+    max: 1.0
+
+dino_detection:
+  threshold:
+    default: 0.5
+  target_multiplier:
+    default: 25
+  img_multiplier:
+    default: 50
+  background_threshold:
+    default: -1.0
+  final_erosion_count:
+    default: 10
+  segment_min_size:
+    default: 20
+
+sam2:
+  model:
+    choices:
+      - facebook/sam2-hiera-base-plus
+      - facebook/sam2-hiera-large
+      - facebook/sam2-hiera-small
+      - facebook/sam2-hiera-tiny
+      - facebook/sam2.1-hiera-base-plus
+      - facebook/sam2.1-hiera-large
+      - facebook/sam2.1-hiera-small
+      - facebook/sam2.1-hiera-tiny
+    default: facebook/sam2.1-hiera-large
+  use_legacy:
+    default: false
+  compile:
+    default: false
+  offload_state_to_cpu:
+    default: false
+  offload_video_to_cpu:
+    default: false
+```
+
+#### Segmentation v0.2 주의사항/변경사항
+- SAM2 VRAM 추정 실패 시 `seg.get_next()`가 `{"result":"ERROR"}`로 반환될 수 있으니 에러 처리 후 `reset`/재등록을 권장합니다.
+- SAM2 `compile=True`는 첫 프레임 등록 및 `reset`이 느려질 수 있습니다.
+- SAM2 CPU offloading은 `offload_state_to_cpu=True`와 `offload_video_to_cpu=True`를 함께 설정할 때 효과가 큽니다(legacy 모드에서는 `offload_video_to_cpu` 미지원).
+- SAM2 `remove_object`는 `use_legacy=False`에서만 지원됩니다.
+- GroundingDINO는 Swin-B 백본이 추가되었고, 프롬프트 토큰 병합 이슈가 수정되었습니다.
+
 ### Pose Estimation
 
 **Mesh 파일 업로드**: 등록/인식하고자 하는 mesh 파일 (stl)을 호스트PC의 '/opt/meshes/' 경로에 업로드 (직접 SSH 통해 파일을 옮겨도 됨)

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5}/src/neuromeka_vfm.egg-info/SOURCES.txt

@@ -3,7 +3,9 @@ README.md
 pyproject.toml
 src/neuromeka_vfm/__init__.py
 src/neuromeka_vfm/compression.py
+src/neuromeka_vfm/grasp_gen.py
 src/neuromeka_vfm/pickle_client.py
+src/neuromeka_vfm/point_cloud_utils.py
 src/neuromeka_vfm/pose_estimation.py
 src/neuromeka_vfm/segmentation.py
 src/neuromeka_vfm/upload_mesh.py

{neuromeka_vfm-0.1.4 → neuromeka_vfm-0.1.5}/src/neuromeka_vfm.egg-info/requires.txt

@@ -2,3 +2,5 @@ numpy
 pyzmq
 paramiko
 av
+trimesh
+tqdm