@blueharford/scrypted-spatial-awareness 0.1.16 → 0.2.1

package/README.md

@@ -2,39 +2,107 @@
 
 Cross-camera object tracking for Scrypted NVR with spatial awareness capabilities.
 
+## Why This Plugin?
+
+**Traditional camera notifications** tell you *what* was detected on *which* camera:
+> "Person detected on Front Door camera"
+
+**Spatial Awareness** tells you *where they came from* and *where they're going*:
+> "Man in blue jacket walking from Garage towards Front Door"
+
+This plugin **tracks objects across your entire camera system**, understanding that the person who got out of a car in your driveway is the same person now walking to your front door. Instead of getting 5 separate "person detected" alerts from 5 cameras, you get one coherent narrative of movement across your property.
+
+### Key Differences from Normal Notifications
+
+| Feature | Normal Notifications | Spatial Awareness |
+|---------|---------------------|-------------------|
+| **Scope** | Single camera | Entire property |
+| **Identity** | New detection each camera | Same object tracked across cameras |
+| **Context** | "Person on Camera X" | "Person moving from X towards Y" |
+| **Alert Volume** | Alert per camera per detection | One alert per significant movement |
+| **Intelligence** | Basic detection | Movement patterns, unusual paths, dwell time |
+| **LLM Integration** | None | Rich descriptions like "Woman with dog" |
+
+## Use Cases
+
+### Home Security
+- **Delivery Tracking**: "Person arrived via Driveway, walked to Front Door, left package, exited via Driveway" (2 minutes on property)
+- **Suspicious Activity**: "Person entered via Back Fence, lingered 5 minutes near Garage, unusual path - did not use normal entry points"
+- **Family Awareness**: Know when family members arrive home and their path through the property
+
+### Vehicle Monitoring
+- **Guest Arrivals**: "Black SUV entered via Street, parked in Driveway"
+- **Unusual Vehicles**: "Unknown vehicle circling property - seen on Street Camera, Side Camera, Street Camera again"
+
+### Pet & Animal Tracking
+- **Pet Location**: Track your dog's movement through the yard
+- **Wildlife Alerts**: "Deer moving from Back Yard towards Garden"
+
+### Property Management
+- **Worker Tracking**: Know when contractors arrive, where they go, when they leave
+- **Occupancy Patterns**: Understand traffic flow through your property
+
 ## Features
 
 - **Cross-Camera Tracking**: Correlate objects (people, vehicles, animals) as they move between cameras
 - **Journey History**: Complete path history for each tracked object across your property
 - **Entry/Exit Detection**: Know when objects enter or leave your property
+- **Movement Alerts**: Get notified when objects move between camera zones
+- **LLM-Enhanced Descriptions**: Rich, contextual alerts like "Man in red shirt walking from garage towards front door" (requires LLM plugin)
 - **Visual Floor Plan Editor**: Configure camera topology with an intuitive visual editor
 - **MQTT Integration**: Export tracking data to Home Assistant for automations
 - **REST API**: Query tracked objects and journeys programmatically
-- **Smart Alerts**: Get notified about property entry/exit, unusual paths, dwell time, and restricted zones
+- **Smart Cooldowns**: Prevent alert spam with per-object cooldowns
+- **Loitering Threshold**: Only alert after objects are visible for a configurable duration
+- **Multiple Notifiers**: Send alerts to multiple notification services simultaneously
+
+### Advanced Spatial Awareness
+
+- **Landmarks & Static Objects**: Define landmarks like mailbox, shed, driveway, deck to give the system spatial context
+- **Camera Context**: Describe where each camera is mounted and what it can see for richer descriptions
+- **Field of View Configuration**: Define camera FOV (simple angle or polygon) to understand coverage overlap
+- **RAG-Powered Reasoning**: Uses Retrieval-Augmented Generation to understand property layout for intelligent descriptions
+- **AI Landmark Suggestions**: System learns to identify landmarks from camera footage over time
+- **Spatial Relationships**: Auto-inferred relationships between cameras and landmarks based on position
 
 ## Installation
 
+### From NPM (Recommended)
+```bash
+npm install @blueharford/scrypted-spatial-awareness
+```
+
 ### From Scrypted Plugin Repository
 1. Open Scrypted Management Console
 2. Go to Plugins
-3. Search for "Spatial Awareness"
+3. Search for "@blueharford/scrypted-spatial-awareness"
 4. Click Install
 
-### From NPM
-```bash
-npm install @blueharford/scrypted-spatial-awareness
-```
-
 ## Setup
 
-1. **Add Cameras**: In plugin settings, select cameras with object detection enabled
-2. **Configure Topology**:
-   - Access the visual editor at `/ui/editor` (via plugin's HTTP endpoint)
-   - Upload a floor plan image
+1. **Configure Topology**:
+   - Open the plugin settings
+   - Click "Open Topology Editor"
+   - Upload a floor plan image (or use the drawing tools to create one)
    - Place cameras on the floor plan
+   - Mark entry/exit points
    - Draw connections between cameras with expected transit times
-3. **Enable Integrations**: Optionally enable MQTT for Home Assistant
-4. **Create Zones**: Add tracking zones for specific area monitoring
+
+2. **Configure Alerts**:
+   - Select one or more notifiers (Pushover, email, Home Assistant, etc.)
+   - Adjust loitering threshold (how long before alerting)
+   - Adjust per-object cooldown (prevent duplicate alerts)
+   - Enable/disable specific alert types
+
+3. **Optional - Enable LLM Descriptions**:
+   - Install an LLM plugin (OpenAI, Ollama, etc.)
+   - Enable "Use LLM for Rich Descriptions" in settings
+   - Get alerts like "Woman with stroller" instead of just "Person"
+
+4. **Optional - Enable MQTT**:
+   - Enable MQTT integration
+   - Configure broker URL and credentials
+   - Use in Home Assistant automations
 
 ## How It Works
 
@@ -48,10 +116,56 @@ The plugin listens to object detection events from all configured cameras. When
    - **Class (10%)**: Is it the same type of object?
 3. **New Object**: If no correlation is found, a new tracked object is created
 
+### Loitering & Cooldown Logic
+
+To prevent alert spam and reduce noise:
+
+- **Loitering Threshold**: Object must be visible for X seconds before triggering any alerts (default: 3 seconds). This prevents alerts for someone briefly passing through frame.
+- **Per-Object Cooldown**: After alerting for a specific tracked object, won't alert again for that same object for Y seconds (default: 30 seconds). This prevents "Person moving from A to B", "Person moving from B to C", "Person moving from C to D" spam.
+
+### LLM Integration
+
+When an LLM plugin is installed and enabled, the plugin will:
+1. Capture a snapshot from the camera
+2. Send it to the LLM with context about the movement
+3. Get a rich description like "Man in blue jacket" or "Black pickup truck"
+4. Include this in the notification
+
+This transforms generic alerts into contextual, actionable information.
+
+## Configuration Options
+
+### Tracking Settings
+| Setting | Default | Description |
+|---------|---------|-------------|
+| Correlation Window | 30s | Maximum time to wait for cross-camera correlation |
+| Correlation Threshold | 0.6 | Minimum confidence (0-1) for automatic correlation |
+| Lost Timeout | 300s | Time before marking an object as lost |
+| Visual Matching | ON | Use visual embeddings for correlation |
+| Loitering Threshold | 3s | Object must be visible this long before alerting |
+| Per-Object Cooldown | 30s | Minimum time between alerts for same object |
+| LLM Descriptions | ON | Use LLM plugin for rich descriptions |
+
+### Alert Types
+| Alert | Description | Default |
+|-------|-------------|---------|
+| Property Entry | Object entered via an entry point | Enabled |
+| Property Exit | Object exited via an exit point | Enabled |
+| Movement | Object moved between cameras | Enabled |
+| Unusual Path | Object took an unexpected route | Enabled |
+| Dwell Time | Object lingered >5 minutes | Enabled |
+| Restricted Zone | Object entered a restricted zone | Enabled |
+| Lost Tracking | Object disappeared without exiting | Disabled |
+
+### Notification Settings
+- **Notifiers**: Select multiple notification services to receive alerts
+- **Thumbnails**: Automatically includes camera snapshot with notifications
+
 ## API Endpoints
 
 The plugin exposes a REST API via Scrypted's HTTP handler:
 
+### Core Endpoints
 | Endpoint | Method | Description |
 |----------|--------|-------------|
 | `/api/tracked-objects` | GET | List all tracked objects |
@@ -59,9 +173,23 @@ The plugin exposes a REST API via Scrypted's HTTP handler:
 | `/api/topology` | GET | Get camera topology configuration |
 | `/api/topology` | PUT | Update camera topology |
 | `/api/alerts` | GET | Get recent alerts |
+| `/api/alert-rules` | GET/PUT | Get or update alert rules |
+| `/api/cameras` | GET | List available cameras |
 | `/api/floor-plan` | GET/POST | Get or upload floor plan image |
 | `/ui/editor` | GET | Visual topology editor |
 
+### Landmark & Spatial Reasoning Endpoints
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/landmarks` | GET | List all configured landmarks |
+| `/api/landmarks` | POST | Add a new landmark |
+| `/api/landmarks/{id}` | GET/PUT/DELETE | Get, update, or delete a landmark |
+| `/api/landmark-suggestions` | GET | Get AI-suggested landmarks |
+| `/api/landmark-suggestions/{id}/accept` | POST | Accept an AI suggestion |
+| `/api/landmark-suggestions/{id}/reject` | POST | Reject an AI suggestion |
+| `/api/landmark-templates` | GET | Get landmark templates for quick setup |
+| `/api/infer-relationships` | GET | Get auto-inferred spatial relationships |
+
 ## MQTT Topics
 
 When MQTT is enabled, the plugin publishes to:
@@ -95,35 +223,24 @@ The plugin creates these virtual devices in Scrypted:
 - **Types**: Entry, Exit, Dwell, Restricted
 - **Use**: Create zone-specific automations and alerts
 
-## Alert Types
+## Example Alert Messages
 
-| Alert | Description |
-|-------|-------------|
-| Property Entry | Object entered the property via an entry point |
-| Property Exit | Object exited the property via an exit point |
-| Unusual Path | Object took an unexpected route between cameras |
-| Dwell Time | Object lingered too long in an area |
-| Restricted Zone | Object entered a restricted zone |
-| Lost Tracking | Object disappeared without exiting |
-
-## Configuration Options
-
-### Tracking Settings
-- **Correlation Window**: Maximum time (seconds) to wait for cross-camera correlation
-- **Correlation Threshold**: Minimum confidence (0-1) for automatic correlation
-- **Lost Timeout**: Time before marking an object as lost
-- **Visual Matching**: Enable/disable visual embedding matching
+With LLM enabled:
+- "Man in blue jacket walking from Garage towards Front Door (5s transit)"
+- "Black SUV driving from Street towards Driveway"
+- "Woman with dog walking from Back Yard towards Side Gate"
+- "Delivery person entered property via Driveway"
 
-### MQTT Settings
-- **Enable MQTT**: Toggle MQTT publishing
-- **Broker URL**: MQTT broker address (e.g., `mqtt://localhost:1883`)
-- **Username/Password**: MQTT authentication
-- **Base Topic**: Topic prefix for all messages
+Without LLM:
+- "Person moving from Garage towards Front Door (5s transit)"
+- "Car moving from Street towards Driveway"
+- "Dog moving from Back Yard towards Side Gate"
 
 ## Requirements
 
 - Scrypted with NVR plugin
 - Cameras with object detection enabled (via Scrypted NVR, OpenVINO, CoreML, ONNX, or TensorFlow Lite)
+- Optional: LLM plugin for rich descriptions (OpenAI, Ollama, etc.)
 - Optional: MQTT broker for Home Assistant integration
 
 ## Development