rtsm 0.1.0__tar.gz

rtsm-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

rtsm-0.1.0/PKG-INFO

@@ -0,0 +1,473 @@
+Metadata-Version: 2.4
+Name: rtsm
+Version: 0.1.0
+Summary: Real-Time Spatio-Semantic Memory for spatial AI and robotics
+Requires-Python: <3.14,>=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: Pillow>=7.1.2
+Requires-Dist: PyYAML>=5.3.1
+Requires-Dist: requests>=2.23.0
+Requires-Dist: scipy>=1.4.1
+Requires-Dist: tqdm>=4.64.0
+Requires-Dist: pandas>=1.1.4
+Requires-Dist: opencv-python-headless>=4.6.0
+Requires-Dist: psutil>=7.0.0
+Requires-Dist: setuptools>=64
+Requires-Dist: fastapi>=0.95.0
+Requires-Dist: uvicorn[standard]>=0.22.0
+Requires-Dist: prometheus-client>=0.16.0
+Requires-Dist: pyzmq>=25.0.0
+Provides-Extra: gpu
+Requires-Dist: torch>=2.1.0; extra == "gpu"
+Requires-Dist: torchvision>=0.16.0; extra == "gpu"
+Requires-Dist: open-clip-torch; extra == "gpu"
+Requires-Dist: sam2>=1.0; extra == "gpu"
+Requires-Dist: transformers>=4.44.0; extra == "gpu"
+Requires-Dist: faiss-cpu>=1.12.0; extra == "gpu"
+Provides-Extra: gpu-ultralytics
+Requires-Dist: rtsm[gpu]; extra == "gpu-ultralytics"
+Requires-Dist: ultralytics>=8.4.0; extra == "gpu-ultralytics"
+Provides-Extra: viz
+Requires-Dist: open3d>=0.18.0; extra == "viz"
+Requires-Dist: matplotlib>=3.2.2; extra == "viz"
+Requires-Dist: seaborn>=0.11.0; extra == "viz"
+Provides-Extra: mcp
+Requires-Dist: mcp[cli]>=1.0.0; extra == "mcp"
+Requires-Dist: httpx>=0.27.0; extra == "mcp"
+Provides-Extra: all
+Requires-Dist: rtsm[gpu,mcp,viz]; extra == "all"
+Dynamic: license-file
+
+![RTSM](repo_media/background.jpg)
+
+# RTSM — Real-Time Spatio-Semantic Memory
+
+RTSM is a real-time spatial memory system that turns RGB-D streams into a **persistent, queryable 3D object-centric world state**.
+
+Instead of treating perception as disposable frames, RTSM maintains **stable object identities over time**, enabling robots and embodied agents to answer questions like:
+- *What objects exist in this space?*
+- *Where are they right now?*
+- *What changed, and when?*
+
+**[Watch Demo Video](https://youtu.be/abhXsbvOLQg)** · **[Documentation](https://calabi-inc.github.io/rtsm)**
+
+---
+
+## Why RTSM
+
+Modern perception systems can detect objects, but they lack **memory**.
+SLAM systems build geometry, vision models detect semantics, and language models reason abstractly—but there is no shared layer that connects **space, objects, and history**.
+
+RTSM fills this gap by acting as an explicit **spatial memory layer**:
+- SLAM provides geometry and poses
+- Vision models provide object masks and semantics
+- RTSM fuses them into a persistent world representation
+
+This makes spatial state **inspectable, queryable, and reusable** across robots, agents, and applications.
+
+---
+
+## What RTSM Does
+
+- Builds a live 3D map from RGB-D + pose streams
+- Assigns **persistent IDs** to objects across viewpoints and time
+- Stores spatial, semantic, and temporal metadata per object
+- Supports semantic + spatial queries (e.g. *"red bin near dock 3"*)
+- Exposes a programmatic API and real-time 3D visualizer
+
+RTSM is **SLAM-agnostic** and designed to sit above existing perception stacks.
+
+---
+
+## Who This Is For
+
+- Robotics and embodied AI researchers
+- Developers building agentic or world-model-based systems
+- Teams exploring persistent perception, spatial reasoning, or digital twins
+
+---
+
+## Architecture
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│                 RTSM — Real-Time Spatio-Semantic Memory                  │
+└──────────────────────────────────────────────────────────────────────────┘
+
+  ┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐
+  │  Calabi Lens     │   │   D435i + SLAM   │   │  Recorded        │
+  │  (ARKit iOS)     │   │   (RTABMap)      │   │  Session         │
+  └────────┬─────────┘   └────────┬─────────┘   └────────┬─────────┘
+           │ WebSocket            │ ZeroMQ               │ --replay
+           ▼                      ▼                       ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│  I/O Layer                                                               │
+│                                                                          │
+│  ┌─────────────┐  ┌─────────────┐  ┌──────────────┐  ┌──────────────┐   │
+│  │  WebSocket  │  │  ZMQ Bridge │  │   Replay     │  │  Recorder    │   │
+│  │  Receiver   │  │  (sensors)  │  │  Receiver    │  │  (--record)  │   │
+│  └──────┬──────┘  └──────┬──────┘  └──────┬───────┘  └──────────────┘   │
+│         └────────────────┴────────────────┘                              │
+│                          │                                               │
+│                   ┌──────▼───────┐     ┌──────────────┐                  │
+│                   │ IngestQueue  │────>│ FramePacket  │                  │
+│                   │  (buffer)    │     │ (RGB,D,Pose) │                  │
+│                   └──────────────┘     └──────┬───────┘                  │
+│                                               │                          │
+└───────────────────────────────────────────────┼──────────────────────────┘
+                                                │
+                          ┌─────────────────────▼───────────────────────┐
+                          │              Ingest Gate                    │
+                          │   (keyframe priority, sweep-based skip)     │
+                          └─────────────────────┬───────────────────────┘
+                                                │
+┌───────────────────────────────────────────────▼──────────────────────────┐
+│  Perception Pipeline                                                     │
+│                                                                          │
+│  ┌────────────────┐  ┌────────────────┐                                   │
+│  │ Grounding DINO │  │     SAM2       │    Default: grounded_sam2          │
+│  │ (detection +   │─>│ (box-prompted  │    GDINO detects → SAM2 segments   │
+│  │  labels)       │  │  masks)        │    (Apache 2.0, no AGPL)           │
+│  └────────────────┘  └───────┬────────┘                                   │
+│                              │                                            │
+│                 ▼                                                        │
+│  ┌───────────────┐     ┌──────────────┐     ┌──────────────┐             │
+│  │ Mask Staging  │────>│ Top-K Select │────>│ CLIP Encode  │             │
+│  │ (heuristics)  │     │  (priority)  │     │(224x224 crop)│             │
+│  └───────────────┘     └──────────────┘     └──────┬───────┘             │
+│                                                    │                     │
+│                     ┌───────────────┐        ┌─────▼────────┐            │
+│                     │ Vocab Classify│<───────│  Embeddings  │            │
+│                     │ (label + conf)│        │  (512-D L2)  │            │
+│                     └───────┬───────┘        └──────────────┘            │
+│                             │                                            │
+└─────────────────────────────┼────────────────────────────────────────────┘
+                              │
+                              ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│  Association                                                             │
+│                                                                          │
+│  ┌─────────────┐     ┌─────────────┐     ┌───────────────┐               │
+│  │  Proximity  │────>│  Embedding  │────>│  Score Fusion │               │
+│  │   Query     │     │  Cosine Sim │     │ (match/create)│               │
+│  └─────────────┘     └─────────────┘     └───────────────┘               │
+│                                                                          │
+└───────────────┬──────────────────────────────────────────────────────────┘
+                │
+                ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│  Working Memory                                                          │
+│                                                                          │
+│  ObjectState:                                                            │
+│    - id, xyz_world (3D position)                                         │
+│    - emb_mean, emb_gallery (CLIP embeddings)                             │
+│    - view_bins (multi-view fusion)                                       │
+│    - label_scores (EWMA label confidence)                                │
+│    - stability, hits, confirmed                                          │
+│    - image_crops (JPEG snapshots)                                        │
+│                                                                          │
+│  Proto -> Confirmed (hits >= 2, stability >= 0.55, views >= 1)           │
+│                                                                          │
+└───────────────┬──────────────────────────────────────────────────────────┘
+                │
+                ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│  Long-Term Memory (FAISS / Milvus)                                       │
+│                                                                          │
+│  Semantic Search: query(text) -> CLIP -> top-k objects                   │
+│                                                                          │
+└───────────────┬──────────────────────────────────────────────────────────┘
+                │
+                ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│  API & Visualization                                                     │
+│                                                                          │
+│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐           │
+│  │    REST API     │  │    WebSocket    │  │     3D Demo     │           │
+│  │    /objects     │  │  point clouds   │  │    (Three.js)   │           │
+│  │    /search      │  │  objects_update │  │                 │           │
+│  └─────────────────┘  └─────────────────┘  └─────────────────┘           │
+│                                                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.12+
+- pip (or [uv](https://docs.astral.sh/uv/))
+- CUDA-capable GPU (tested on RTX 3080, RTX 5090)
+- **One of:**
+  - iPhone with [Calabi Lens](https://github.com/calabi-inc) app (ARKit, no external SLAM needed)
+  - RGB-D camera (Intel RealSense D435i) + SLAM system (RTAB-Map)
+  - A recorded session (for replay — no hardware needed)
+
+> **Tested on:** WSL2 Ubuntu 22.04 with RTAB-Map, and macOS/Windows with Calabi Lens.
+
+### Installation
+
+```bash
+git clone https://github.com/calabi-inc/rtsm.git
+cd rtsm
+
+# Core only (API server, I/O transports — no GPU needed)
+pip install .
+
+# With GPU — permissive license (SAM2 + Grounding DINO, Apache 2.0)
+pip install ".[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128
+
+# With GPU — ultralytics backends (FastSAM + YOLOE, AGPL-3.0)
+pip install ".[gpu-ultralytics]" --extra-index-url https://download.pytorch.org/whl/cu128
+
+# Everything (GPU + visualization)
+pip install ".[all]" --extra-index-url https://download.pytorch.org/whl/cu128
+```
+
+> **License note:** `rtsm[gpu]` uses only Apache 2.0 / MIT dependencies. `rtsm[gpu-ultralytics]` adds the `ultralytics` package (AGPL-3.0) for FastSAM and YOLOE backends.
+>
+> **CUDA version:** Use `cu128` for most GPUs (RTX 3080–5090). For Blackwell-only features use `cu130`. See [PyTorch install](https://pytorch.org/get-started/locally/) for other options.
+
+### Download Models
+
+```bash
+# Fetch default models (SAM2, Grounding DINO, CLIP)
+python scripts/fetch_models.py
+
+# Or fetch individually
+python scripts/fetch_models.py --only sam2
+python scripts/fetch_models.py --only gdino
+python scripts/fetch_models.py --only clip
+
+# Ultralytics models (only if you installed rtsm[gpu-ultralytics])
+python scripts/fetch_models.py --only fastsam
+python scripts/fetch_models.py --only yolo
+```
+
+### Run
+
+```bash
+# Live — Calabi Lens (ARKit over WebSocket)
+python -m rtsm
+
+# Live — D435i + RTAB-Map (ZeroMQ)
+# Set io.receiver: zeromq in config/rtsm.yaml first
+python -m rtsm
+
+# Replay a recorded session (no device needed)
+python -m rtsm --replay recordings/session1
+```
+
+RTSM will start:
+- **Perception pipeline** — processing frames
+- **REST API** — `http://localhost:8000`
+- **Visualization WebSocket** — `ws://localhost:8081`
+
+### Record & Replay
+
+Record a session for offline testing and reproducible iteration:
+
+```bash
+# Record only (no GPU needed, no pipeline — works with core-only install)
+python -m rtsm --record recordings/my_session --record-only
+
+# Record while running pipeline
+python -m rtsm --record recordings/my_session
+
+# Replay at original recording rate
+python -m rtsm --replay recordings/my_session
+```
+
+Recordings are self-contained directories with raw WebSocket data. Replay feeds the exact same bytes through the full decode + pipeline path, preserving all time-dependent behavior (TTL caches, throttles).
+
+### A/B Segmentation Debug
+
+Compare FastSAM vs YOLOE segmentation on a recorded session:
+
+```bash
+# Generate side-by-side overlays (cached — skips existing frames)
+python scripts/debug_segmentation.py --recording recordings/session1
+
+# Open the viewer
+# → debug/session1/compare.html (arrow keys to navigate)
+```
+
+### API Examples
+
+```bash
+# List all objects
+curl http://localhost:8000/objects
+
+# Semantic search
+curl "http://localhost:8000/search/semantic?query=red%20mug&top_k=5"
+
+# Get system stats
+curl http://localhost:8000/stats/detailed
+
+# Runtime analytics (latency, throughput, dual confirmation rates)
+curl http://localhost:8000/stats/analytics
+```
+
+---
+
+## Configuration
+
+See [`config/rtsm.yaml`](config/rtsm.yaml) for full configuration options:
+
+- **Camera intrinsics** — focal length, resolution
+- **I/O endpoints** — ZeroMQ addresses for camera and SLAM
+- **Pipeline tuning** — mask filtering, association thresholds
+- **Memory settings** — object promotion, expiry, vector store
+
+---
+
+## Segmentation Backends
+
+RTSM supports multiple segmentation backends via `segmentation.backend` in `config/rtsm.yaml`:
+
+| Backend | License | Description | Seg time* | Pipeline total* | Labels |
+|---------|---------|-------------|-----------|-----------------|--------|
+| `grounded_sam2` | Apache 2.0 | Grounding DINO detect + SAM2 segment | 222 ms | 510 ms | Open-vocab (text-prompted) |
+| `sam2` | Apache 2.0 | SAM2 auto-mask (segment everything) | ~860 ms | ~1000 ms | None (class-agnostic) |
+| `fastsam` | AGPL-3.0 | FastSAM (segment everything) | ~50 ms | ~200 ms | None (class-agnostic) |
+| `yoloe` | AGPL-3.0 | YOLOE detection + segmentation | ~60 ms | ~210 ms | Open-vocab / 1200+ built-in |
+| `dual` | AGPL-3.0 | FastSAM + YOLOE with IoU merge | 116 ms | 210 ms | Dual-confirmed labels |
+
+*Mean on RTX 5090, 640x480 input. `dual` and `grounded_sam2` measured via replay benchmark; others estimated.*
+
+**Default:** `grounded_sam2` — permissive license, open-vocabulary, no AGPL dependency.
+
+To switch backends, edit `config/rtsm.yaml`:
+```yaml
+segmentation:
+  backend: grounded_sam2    # or: sam2, fastsam, yoloe, dual
+```
+
+> `fastsam`, `yoloe`, and `dual` require `pip install "rtsm[gpu-ultralytics]"`.
+
+---
+
+## Project Structure
+
+```
+rtsm/
+├── core/           # Pipeline, association, ingest gate, data models
+├── models/         # SAM2, Grounding DINO, FastSAM, YOLOE, CLIP adapters
+├── stores/         # Working memory, proximity index, sweep cache, vector stores
+├── io/             # WebSocket + ZeroMQ receivers, recorder, replayer
+├── analytics/      # Runtime analytics (latency, segmentation, congestion buffers)
+├── api/            # REST API server (FastAPI)
+├── visualization/  # WebSocket server, TSDF fusion, 3D demo
+└── utils/          # Mask staging, transforms, helpers
+config/
+├── rtsm.yaml       # Main configuration (models, thresholds, I/O)
+└── clip/vocab.yaml  # CLIP vocabulary
+scripts/
+├── fetch_models.py          # Download all models (SAM2, GDINO, CLIP, FastSAM, YOLOE)
+├── debug_segmentation.py    # A/B segmentation viewer (FastSAM vs YOLOE)
+└── benchmark_backends.py    # Backend comparison benchmark (generates reports/)
+reports/                     # Benchmark results and comparison reports
+recordings/                  # Recorded sessions for replay testing (git-lfs)
+tests/                       # Unit + integration tests
+```
+
+---
+
+## Performance
+
+Benchmarked on RTX 5090 (32 GB), iPhone ARKit recording (162 frames, 458s indoor scene), 640x480 RGB input. Both backends run the same replay session through the identical 10-stage pipeline.
+
+### Backend Comparison
+
+| Metric | dual (FastSAM + YOLOE) | grounded_sam2 (GDINO + SAM2) |
+|--------|------------------------|------------------------------|
+| **Mean latency** | **210 ms** | **510 ms** |
+| P50 latency | 170 ms | 502 ms |
+| P95 latency | 509 ms | 721 ms |
+| Masks/frame | 28.8 | 13.4 |
+| Objects confirmed | 60 | 35 |
+| Confirmation rate | 52.2% | 45.5% |
+| License | AGPL-3.0 | Apache-2.0 |
+
+> Per-stage breakdown, dual confirmation analysis, and full methodology: **[Benchmarks](https://calabi-inc.github.io/rtsm/benchmarks/)** | [`reports/backend_comparison.md`](reports/backend_comparison.md)
+
+---
+
+## Roadmap
+
+- [x] Dual-confirmation segmentation (FastSAM + YOLOE)
+- [x] AGPL-clean default (SAM2 + Grounding DINO, Apache 2.0)
+- [x] YOLOE prompt-free (1200+ LVIS categories)
+- [x] WebSocket receiver for Calabi Lens (ARKit iOS)
+- [x] Record/replay system for offline testing
+- [x] A/B segmentation debug tooling
+- [x] Real-time analytics dashboard (Looker-style, per-stage latency, dual confirmation rates, congestion detection)
+- [ ] Evaluation framework (ArUco ground truth, precision/recall metrics)
+- [x] Agent architecture (MCP interface)
+- [ ] More communication protocols (ROS 2, gRPC)
+- [ ] LLM integration for high-level queries (agentic mode)
+- [ ] Dockerization
+
+---
+
+## Acknowledgments
+
+RTSM builds on excellent open-source work:
+
+- **SAM 2** — Ravi et al., *SAM 2: Segment Anything in Images and Videos*, 2024.
+  [arXiv:2408.00714](https://arxiv.org/abs/2408.00714) · [GitHub](https://github.com/facebookresearch/sam2)
+
+- **Grounding DINO** — Liu et al., *Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection*, 2023.
+  [arXiv:2303.05499](https://arxiv.org/abs/2303.05499) · [GitHub](https://github.com/IDEA-Research/GroundingDINO)
+
+- **FastSAM** — Zhao et al., *Fast Segment Anything*, 2023.
+  [arXiv:2306.12156](https://arxiv.org/abs/2306.12156) · [GitHub](https://github.com/CASIA-IVA-Lab/FastSAM)
+
+- **YOLOE** — THU-MIG, *YOLOE: Real-Time Seeing Anything*, ICCV 2025.
+  [GitHub](https://github.com/THU-MIG/yoloe) · [Ultralytics](https://docs.ultralytics.com/models/yoloe/)
+
+- **CLIP** — Radford et al., *Learning Transferable Visual Models From Natural Language Supervision*, 2021.
+  [arXiv:2103.00020](https://arxiv.org/abs/2103.00020) · [GitHub](https://github.com/openai/CLIP)
+
+- **RTAB-Map** — Labbé & Michaud, *RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operation*, Journal of Field Robotics, 2019.
+  [Paper](https://doi.org/10.1002/rob.21831) · [GitHub](https://github.com/introlab/rtabmap)
+
+---
+
+## Cite This
+
+If you use RTSM in your research, please cite:
+
+```bibtex
+@software{chang2025rtsm,
+  author       = {Chang, Chi Feng},
+  title        = {{RTSM}: Real-Time Spatio-Semantic Memory},
+  year         = {2025},
+  url          = {https://github.com/calabi-inc/rtsm},
+  note         = {Object-centric queryable memory for spatial AI and robotics}
+}
+```
+
+---
+
+## Community
+
+This project is under active development. If you have questions or run into issues, feel free to open an issue — I'm happy to help.
+
+If you find RTSM useful, please consider giving it a star! I'm also looking for design partners — reach out to [Calabi](https://github.com/calabi-team) if you're interested in collaborating.
+
+---
+
+## License
+
+Apache-2.0
+
+---
+
+## Author
+
+Built by [Chi Feng, Chang](https://github.com/vipipi)