@blueharford/scrypted-spatial-awareness 0.1.0

package/.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+  "scrypted.debugHost": "127.0.0.1"
+}

package/CLAUDE.md

@@ -0,0 +1,168 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a Scrypted plugin for cross-camera object tracking ("Spatial Awareness"). It correlates detected objects as they move between cameras, maintains global tracking state, and provides alerts for property entry/exit and movement patterns.
+
+## Build & Development Commands
+
+```bash
+# Install dependencies
+npm install
+
+# Build the plugin
+npm run build
+
+# Deploy to local Scrypted server (127.0.0.1)
+npm run scrypted-deploy
+
+# Deploy to remote Scrypted server
+npm run scrypted-deploy <ip-address>
+
+# Debug in VS Code
+# 1. Edit .vscode/settings.json with Scrypted server IP
+# 2. Press F5 or use Run > Start Debugging
+
+# Login to Scrypted (required once)
+npx scrypted login
+```
+
+## Architecture
+
+### Plugin Entry Point
+- `src/main.ts` - Exports `SpatialAwarenessPlugin` class implementing `DeviceProvider`, `Settings`, `HttpRequestHandler`
+
+### Core Tracking Engine
+- `src/core/tracking-engine.ts` - Orchestrates detection events, correlation, and state updates
+- `src/core/object-correlator.ts` - Multi-factor correlation algorithm (timing, visual, spatial, class)
+- `src/core/transit-predictor.ts` - Predicts expected cameras and transit times
+
+### Data Models
+- `src/models/topology.ts` - Camera nodes, connections, zones
+- `src/models/tracked-object.ts` - Global tracked object with journey history
+- `src/models/alert.ts` - Alert types and rules
+
+### State Management
+- `src/state/tracking-state.ts` - In-memory store with persistence and change notifications
+
+### Virtual Devices
+- `src/devices/global-tracker-sensor.ts` - OccupancySensor for property-wide tracking
+- `src/devices/tracking-zone.ts` - Zone-specific motion/occupancy sensors
+
+## Key Scrypted Patterns
+
+### Listening to Object Detection Events
+```typescript
+import sdk, { ScryptedInterface, ObjectsDetected } from '@scrypted/sdk';
+
+const camera = sdk.systemManager.getDeviceById(cameraId);
+camera.listen(ScryptedInterface.ObjectDetector, async (source, details, data) => {
+  const results = data as ObjectsDetected;
+  // results.detections[] contains ObjectDetectionResult items
+});
+```
+
+### ObjectDetectionResult Structure
+```typescript
+{
+  boundingBox: [x, y, width, height],  // normalized coordinates
+  className: 'person' | 'car' | 'animal' | 'package',
+  score: 0.95,  // confidence
+  id: 'abc123',  // tracking ID (single camera)
+  history: { firstSeen: number, lastSeen: number },
+  movement: { moving: boolean },
+  zones: string[],
+  embedding?: string  // visual feature embedding (base64)
+}
+```
+
+### StorageSettings Pattern
+```typescript
+import { StorageSettings } from '@scrypted/sdk/storage-settings';
+
+storageSettings = new StorageSettings(this, {
+  settingKey: {
+    title: 'Setting Title',
+    type: 'number',
+    defaultValue: 30,
+    group: 'GroupName',
+  },
+});
+```
+
+### DeviceProvider Pattern
+```typescript
+async getDevice(nativeId: string): Promise<any> {
+  // Return or create device instance by nativeId
+}
+
+async releaseDevice(id: string, nativeId: string): Promise<void> {
+  // Cleanup device
+}
+```
+
+## Correlation Algorithm
+
+Objects are correlated across cameras using weighted factors:
+- **Timing (30%)**: Transit time within expected min/max range
+- **Visual (35%)**: Embedding similarity (cosine distance)
+- **Spatial (25%)**: Exit zone → Entry zone coherence
+- **Class (10%)**: Object class must match
+
+Threshold for automatic correlation: 0.6 (configurable)
+
+## Camera Topology Configuration
+
+Topology is stored as JSON with:
+- `cameras[]` - Camera nodes with entry/exit point flags
+- `connections[]` - Links between cameras with exit/entry zones and transit times
+- `globalZones[]` - Named zones spanning multiple cameras
+
+## API Endpoints
+
+- `GET /api/tracked-objects` - All tracked objects
+- `GET /api/journey/{globalId}` - Journey for specific object
+- `GET|PUT /api/topology` - Camera topology config
+- `GET /api/alerts` - Recent alerts
+- `GET|POST /api/floor-plan` - Floor plan image (base64)
+- `GET /ui/editor` - Visual topology editor
+
+## MQTT Integration
+
+When enabled, publishes to Home Assistant via MQTT:
+- `{baseTopic}/occupancy/state` - ON/OFF occupancy
+- `{baseTopic}/count/state` - Active object count
+- `{baseTopic}/person_count/state` - People on property
+- `{baseTopic}/vehicle_count/state` - Vehicles on property
+- `{baseTopic}/state` - Full JSON state
+- `{baseTopic}/alerts` - Alert events
+- `{baseTopic}/events/entry|exit|transition` - Movement events
+
+## Project Structure
+
+```
+src/
+├── main.ts                    # Plugin entry point
+├── core/
+│   ├── tracking-engine.ts     # Central orchestrator
+│   └── object-correlator.ts   # Cross-camera matching
+├── models/
+│   ├── topology.ts            # Camera topology types
+│   ├── tracked-object.ts      # Tracked object types
+│   └── alert.ts               # Alert types
+├── devices/
+│   ├── global-tracker-sensor.ts
+│   └── tracking-zone.ts
+├── state/
+│   └── tracking-state.ts      # State management
+├── alerts/
+│   └── alert-manager.ts       # Alert generation
+├── integrations/
+│   └── mqtt-publisher.ts      # MQTT for Home Assistant
+├── ui/
+│   └── editor.html            # Visual topology editor
+└── utils/
+    └── id-generator.ts
+```

package/README.md

@@ -0,0 +1,152 @@
+# Spatial Awareness - Scrypted Plugin
+
+Cross-camera object tracking for Scrypted NVR with spatial awareness capabilities.
+
+## Features
+
+- **Cross-Camera Tracking**: Correlate objects (people, vehicles, animals) as they move between cameras
+- **Journey History**: Complete path history for each tracked object across your property
+- **Entry/Exit Detection**: Know when objects enter or leave your property
+- **Visual Floor Plan Editor**: Configure camera topology with an intuitive visual editor
+- **MQTT Integration**: Export tracking data to Home Assistant for automations
+- **REST API**: Query tracked objects and journeys programmatically
+- **Smart Alerts**: Get notified about property entry/exit, unusual paths, dwell time, and restricted zones
+
+## Installation
+
+### From Scrypted Plugin Repository
+1. Open Scrypted Management Console
+2. Go to Plugins
+3. Search for "Spatial Awareness"
+4. Click Install
+
+### From NPM
+```bash
+npm install @blueharford/scrypted-spatial-awareness
+```
+
+## Setup
+
+1. **Add Cameras**: In plugin settings, select cameras with object detection enabled
+2. **Configure Topology**:
+   - Access the visual editor at `/ui/editor` (via plugin's HTTP endpoint)
+   - Upload a floor plan image
+   - Place cameras on the floor plan
+   - Draw connections between cameras with expected transit times
+3. **Enable Integrations**: Optionally enable MQTT for Home Assistant
+4. **Create Zones**: Add tracking zones for specific area monitoring
+
+## How It Works
+
+The plugin listens to object detection events from all configured cameras. When an object (person, car, animal, package) is detected:
+
+1. **Same Camera**: If the object is already being tracked on this camera, the sighting is added to its history
+2. **Cross-Camera Correlation**: If the object disappeared from another camera recently, the plugin attempts to correlate using:
+   - **Timing (30%)**: Does the transit time match the expected range?
+   - **Visual (35%)**: Do the visual embeddings match (if available)?
+   - **Spatial (25%)**: Was the object in the exit zone of the previous camera and entry zone of the new camera?
+   - **Class (10%)**: Is it the same type of object?
+3. **New Object**: If no correlation is found, a new tracked object is created
+
+## API Endpoints
+
+The plugin exposes a REST API via Scrypted's HTTP handler:
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/tracked-objects` | GET | List all tracked objects |
+| `/api/journey/{id}` | GET | Get journey for specific object |
+| `/api/topology` | GET | Get camera topology configuration |
+| `/api/topology` | PUT | Update camera topology |
+| `/api/alerts` | GET | Get recent alerts |
+| `/api/floor-plan` | GET/POST | Get or upload floor plan image |
+| `/ui/editor` | GET | Visual topology editor |
+
+## MQTT Topics
+
+When MQTT is enabled, the plugin publishes to:
+
+| Topic | Description |
+|-------|-------------|
+| `{baseTopic}/occupancy/state` | ON/OFF property occupancy |
+| `{baseTopic}/count/state` | Number of active tracked objects |
+| `{baseTopic}/person_count/state` | Number of people on property |
+| `{baseTopic}/vehicle_count/state` | Number of vehicles on property |
+| `{baseTopic}/state` | Full JSON state with all objects |
+| `{baseTopic}/alerts` | Alert events |
+| `{baseTopic}/events/entry` | Entry events |
+| `{baseTopic}/events/exit` | Exit events |
+| `{baseTopic}/events/transition` | Camera transition events |
+
+Default base topic: `scrypted/spatial-awareness`
+
+## Virtual Devices
+
+The plugin creates these virtual devices in Scrypted:
+
+### Global Object Tracker
+- **Type**: Occupancy Sensor
+- **Purpose**: Shows whether any objects are currently tracked on the property
+- **Use**: Trigger automations when property becomes occupied/unoccupied
+
+### Tracking Zones (User-Created)
+- **Type**: Motion + Occupancy Sensor
+- **Purpose**: Monitor specific areas across one or more cameras
+- **Types**: Entry, Exit, Dwell, Restricted
+- **Use**: Create zone-specific automations and alerts
+
+## Alert Types
+
+| Alert | Description |
+|-------|-------------|
+| Property Entry | Object entered the property via an entry point |
+| Property Exit | Object exited the property via an exit point |
+| Unusual Path | Object took an unexpected route between cameras |
+| Dwell Time | Object lingered too long in an area |
+| Restricted Zone | Object entered a restricted zone |
+| Lost Tracking | Object disappeared without exiting |
+
+## Configuration Options
+
+### Tracking Settings
+- **Correlation Window**: Maximum time (seconds) to wait for cross-camera correlation
+- **Correlation Threshold**: Minimum confidence (0-1) for automatic correlation
+- **Lost Timeout**: Time before marking an object as lost
+- **Visual Matching**: Enable/disable visual embedding matching
+
+### MQTT Settings
+- **Enable MQTT**: Toggle MQTT publishing
+- **Broker URL**: MQTT broker address (e.g., `mqtt://localhost:1883`)
+- **Username/Password**: MQTT authentication
+- **Base Topic**: Topic prefix for all messages
+
+## Requirements
+
+- Scrypted with NVR plugin
+- Cameras with object detection enabled (via Scrypted NVR, OpenVINO, CoreML, ONNX, or TensorFlow Lite)
+- Optional: MQTT broker for Home Assistant integration
+
+## Development
+
+```bash
+# Install dependencies
+npm install
+
+# Build
+npm run build
+
+# Deploy to local Scrypted
+npm run scrypted-deploy
+
+# Debug in VS Code
+# Edit .vscode/settings.json with your Scrypted server IP
+# Press F5 to start debugging
+```
+
+## License
+
+Apache-2.0
+
+## Author
+
+Joshua Seidel ([@blueharford](https://github.com/blueharford))