@blueharford/scrypted-spatial-awareness 0.4.6 → 0.4.8-beta.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +141 -355
- package/dist/main.nodejs.js +1 -1
- package/dist/main.nodejs.js.map +1 -1
- package/dist/plugin.zip +0 -0
- package/out/main.nodejs.js +554 -137
- package/out/main.nodejs.js.map +1 -1
- package/out/plugin.zip +0 -0
- package/package.json +1 -1
- package/src/alerts/alert-manager.ts +16 -9
- package/src/core/object-correlator.ts +41 -7
- package/src/core/spatial-reasoning.ts +315 -44
- package/src/core/tracking-engine.ts +57 -19
- package/src/main.ts +3 -3
- package/src/models/alert.ts +41 -14
package/README.md
CHANGED
|
@@ -1,428 +1,214 @@
|
|
|
1
1
|
# Spatial Awareness - Scrypted Plugin
|
|
2
2
|
|
|
3
|
-
Cross-camera object tracking for Scrypted NVR
|
|
3
|
+
Cross-camera object tracking for Scrypted NVR. Track people, vehicles, and animals as they move across your property.
|
|
4
4
|
|
|
5
|
-
##
|
|
5
|
+
## What It Does
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
> "Person detected on Front Door camera"
|
|
7
|
+
Instead of getting separate "person detected" alerts from each camera, get one coherent narrative:
|
|
9
8
|
|
|
10
|
-
|
|
11
|
-
> "Man in blue jacket walking from Garage towards Front Door"
|
|
9
|
+
> "Man in blue jacket entered via Driveway, walked to Front Door, left package, exited via Driveway" (2 min on property)
|
|
12
10
|
|
|
13
|
-
|
|
11
|
+
| Traditional Alerts | Spatial Awareness |
|
|
12
|
+
|-------------------|-------------------|
|
|
13
|
+
| Alert per camera | One alert per movement |
|
|
14
|
+
| "Person on Camera X" | "Person moving from X to Y" |
|
|
15
|
+
| No identity tracking | Same object tracked across cameras |
|
|
16
|
+
| Basic detection | Movement patterns, dwell time, unusual paths |
|
|
14
17
|
|
|
15
|
-
|
|
18
|
+
## Quick Start
|
|
16
19
|
|
|
17
|
-
|
|
18
|
-
|---------|---------------------|-------------------|
|
|
19
|
-
| **Scope** | Single camera | Entire property |
|
|
20
|
-
| **Identity** | New detection each camera | Same object tracked across cameras |
|
|
21
|
-
| **Context** | "Person on Camera X" | "Person moving from X towards Y" |
|
|
22
|
-
| **Alert Volume** | Alert per camera per detection | One alert per significant movement |
|
|
23
|
-
| **Intelligence** | Basic detection | Movement patterns, unusual paths, dwell time |
|
|
24
|
-
| **LLM Integration** | None | Rich descriptions like "Woman with dog" |
|
|
25
|
-
|
|
26
|
-
## Use Cases
|
|
27
|
-
|
|
28
|
-
### Home Security
|
|
29
|
-
- **Delivery Tracking**: "Person arrived via Driveway, walked to Front Door, left package, exited via Driveway" (2 minutes on property)
|
|
30
|
-
- **Suspicious Activity**: "Person entered via Back Fence, lingered 5 minutes near Garage, unusual path - did not use normal entry points"
|
|
31
|
-
- **Family Awareness**: Know when family members arrive home and their path through the property
|
|
32
|
-
|
|
33
|
-
### Vehicle Monitoring
|
|
34
|
-
- **Guest Arrivals**: "Black SUV entered via Street, parked in Driveway"
|
|
35
|
-
- **Unusual Vehicles**: "Unknown vehicle circling property - seen on Street Camera, Side Camera, Street Camera again"
|
|
36
|
-
|
|
37
|
-
### Pet & Animal Tracking
|
|
38
|
-
- **Pet Location**: Track your dog's movement through the yard
|
|
39
|
-
- **Wildlife Alerts**: "Deer moving from Back Yard towards Garden"
|
|
40
|
-
|
|
41
|
-
### Property Management
|
|
42
|
-
- **Worker Tracking**: Know when contractors arrive, where they go, when they leave
|
|
43
|
-
- **Occupancy Patterns**: Understand traffic flow through your property
|
|
44
|
-
|
|
45
|
-
## Features
|
|
46
|
-
|
|
47
|
-
### Core Tracking
|
|
48
|
-
- **Cross-Camera Tracking**: Correlate objects (people, vehicles, animals) as they move between cameras
|
|
49
|
-
- **Journey History**: Complete path history for each tracked object across your property
|
|
50
|
-
- **Entry/Exit Detection**: Know when objects enter or leave your property
|
|
51
|
-
- **Movement Alerts**: Get notified when objects move between camera zones
|
|
52
|
-
- **Smart Cooldowns**: Prevent alert spam with per-object cooldowns
|
|
53
|
-
- **Loitering Threshold**: Only alert after objects are visible for a configurable duration
|
|
54
|
-
- **Multiple Notifiers**: Send alerts to multiple notification services simultaneously
|
|
55
|
-
|
|
56
|
-
### LLM-Enhanced Descriptions
|
|
57
|
-
- **Rich Contextual Alerts**: Get alerts like "Man in red shirt walking from garage towards front door" (requires LLM plugin)
|
|
58
|
-
- **Configurable Rate Limiting**: Prevent LLM API overload with configurable debounce intervals
|
|
59
|
-
- **Automatic Fallback**: Falls back to basic notifications when LLM is slow or unavailable
|
|
60
|
-
- **Configurable Timeouts**: Set maximum wait time for LLM responses
|
|
61
|
-
|
|
62
|
-
### Visual Floor Plan Editor
|
|
63
|
-
- **Drag-and-Drop**: Place cameras, landmarks, and connections visually
|
|
64
|
-
- **Live Tracking Overlay**: See tracked objects move across your floor plan in real-time
|
|
65
|
-
- **Journey Visualization**: Click any tracked object to see their complete path drawn on the floor plan
|
|
66
|
-
- **Drawing Tools**: Add walls, rooms, and labels without needing an image
|
|
67
|
-
|
|
68
|
-
### Spatial Intelligence
|
|
69
|
-
- **Landmarks & Static Objects**: Define landmarks like mailbox, shed, driveway, deck to give the system spatial context
|
|
70
|
-
- **Camera Context**: Describe where each camera is mounted and what it can see for richer descriptions
|
|
71
|
-
- **Field of View Configuration**: Define camera FOV (simple angle or polygon) to understand coverage overlap
|
|
72
|
-
- **RAG-Powered Reasoning**: Uses Retrieval-Augmented Generation to understand property layout for intelligent descriptions
|
|
73
|
-
- **AI Landmark Suggestions**: System learns to identify landmarks from camera footage over time
|
|
74
|
-
- **Spatial Relationships**: Auto-inferred relationships between cameras and landmarks based on position
|
|
75
|
-
|
|
76
|
-
### Automatic Learning
|
|
77
|
-
- **Transit Time Learning**: Automatically adjusts connection transit times based on observed movement patterns
|
|
78
|
-
- **Connection Suggestions**: System suggests new camera connections based on observed object movements
|
|
79
|
-
- **Confidence Scoring**: Suggestions include confidence scores based on consistency of observations
|
|
80
|
-
- **One-Click Approval**: Accept or reject suggestions directly from the topology editor
|
|
81
|
-
|
|
82
|
-
### Training Mode (NEW in v0.4.0)
|
|
83
|
-
- **Guided Walkthrough**: Walk your property and let the system learn your camera layout
|
|
84
|
-
- **Mobile-Optimized UI**: Designed for phone use while walking around
|
|
85
|
-
- **Auto Camera Detection**: System detects you automatically as you walk
|
|
86
|
-
- **Transit Time Recording**: Learns actual transit times between cameras
|
|
87
|
-
- **Overlap Detection**: Identifies where camera coverage overlaps
|
|
88
|
-
- **Landmark Marking**: Mark landmarks (mailbox, gate, etc.) as you encounter them
|
|
89
|
-
- **One-Click Setup**: Apply training results to generate your complete topology
|
|
90
|
-
|
|
91
|
-
### Integrations
|
|
92
|
-
- **MQTT Integration**: Export tracking data to Home Assistant for automations
|
|
93
|
-
- **REST API**: Query tracked objects and journeys programmatically
|
|
94
|
-
|
|
95
|
-
## Installation
|
|
96
|
-
|
|
97
|
-
### From NPM (Recommended)
|
|
20
|
+
### 1. Install
|
|
98
21
|
```bash
|
|
99
|
-
|
|
22
|
+
npx scrypted install @blueharford/scrypted-spatial-awareness
|
|
100
23
|
```
|
|
101
24
|
|
|
102
|
-
###
|
|
103
|
-
1. Open Scrypted Management Console
|
|
104
|
-
2. Go to Plugins
|
|
105
|
-
3. Search for "@blueharford/scrypted-spatial-awareness"
|
|
106
|
-
4. Click Install
|
|
107
|
-
|
|
108
|
-
## Getting Started: Training Mode (NEW in v0.4.0)
|
|
109
|
-
|
|
110
|
-
The fastest way to set up Spatial Awareness is using **Training Mode** - a guided walkthrough where you physically walk around your property while the system learns your camera layout.
|
|
111
|
-
|
|
112
|
-
### Why Training Mode?
|
|
25
|
+
### 2. Train Your Property
|
|
113
26
|
|
|
114
|
-
|
|
115
|
-
1. Start training on your phone
|
|
116
|
-
2. Walk between cameras
|
|
117
|
-
3. The system automatically learns:
|
|
118
|
-
- Which cameras can see you
|
|
119
|
-
- How long it takes to walk between cameras
|
|
120
|
-
- Where cameras overlap
|
|
121
|
-
- Your property's layout
|
|
27
|
+
The fastest setup is **Training Mode** - walk your property while the system learns:
|
|
122
28
|
|
|
123
|
-
|
|
29
|
+
1. Open plugin settings → Click **Training Mode**
|
|
30
|
+
2. Tap **Start Training** on your phone
|
|
31
|
+
3. Walk between cameras naturally
|
|
32
|
+
4. System auto-detects you and records transit times
|
|
33
|
+
5. Tap **End Training** → **Apply Results**
|
|
124
34
|
|
|
125
|
-
|
|
126
|
-
- Navigate to: `/endpoint/@blueharford/scrypted-spatial-awareness/ui/training`
|
|
127
|
-
- Or scan the QR code in the plugin settings (mobile-optimized)
|
|
35
|
+
Done! Your camera topology is configured.
|
|
128
36
|
|
|
129
|
-
|
|
130
|
-
- Tap "Start Training"
|
|
131
|
-
- The system begins listening for person detections
|
|
37
|
+
### 3. Configure Alerts
|
|
132
38
|
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
- Camera positions
|
|
137
|
-
- Transit times between cameras
|
|
138
|
-
- Camera overlaps (when both cameras see you)
|
|
39
|
+
- Select notifiers (Pushover, email, etc.)
|
|
40
|
+
- Set loitering threshold (default: 3s)
|
|
41
|
+
- Set per-object cooldown (default: 30s)
|
|
139
42
|
|
|
140
|
-
|
|
141
|
-
- Tap the "Mark" tab to add landmarks as you encounter them
|
|
142
|
-
- Select type (mailbox, gate, shed, etc.) and name
|
|
143
|
-
- Landmarks are associated with the current camera
|
|
144
|
-
|
|
145
|
-
5. **End Training**
|
|
146
|
-
- When finished, tap "End Training"
|
|
147
|
-
- Review the statistics: cameras visited, transits recorded, landmarks marked
|
|
148
|
-
|
|
149
|
-
6. **Apply Results**
|
|
150
|
-
- Tap "Apply Results" to generate your topology
|
|
151
|
-
- The system creates camera connections with learned transit times
|
|
152
|
-
- Open the Topology Editor to fine-tune if needed
|
|
43
|
+
## Features
|
|
153
44
|
|
|
154
|
-
###
|
|
45
|
+
### Core
|
|
46
|
+
- **Cross-Camera Tracking** - Correlate objects across cameras using timing, visual similarity, and spatial position
|
|
47
|
+
- **Journey History** - Complete path for each tracked object
|
|
48
|
+
- **Entry/Exit Detection** - Know when objects enter or leave your property
|
|
49
|
+
- **Smart Alerts** - Loitering thresholds and per-object cooldowns prevent spam
|
|
155
50
|
|
|
156
|
-
|
|
157
|
-
- **
|
|
158
|
-
- **
|
|
159
|
-
- **
|
|
160
|
-
- **Re-train anytime** - Run training again to improve accuracy or add new cameras
|
|
51
|
+
### Visual Editor
|
|
52
|
+
- **Floor Plan** - Upload image or draw with built-in tools
|
|
53
|
+
- **Drag & Drop** - Place cameras, draw connections
|
|
54
|
+
- **Live Tracking** - Watch objects move in real-time
|
|
161
55
|
|
|
162
|
-
###
|
|
56
|
+
### AI Features (optional)
|
|
57
|
+
- **LLM Descriptions** - "Woman with stroller" instead of just "Person"
|
|
58
|
+
- **Auto-Learning** - Transit times adjust based on observations
|
|
59
|
+
- **Connection Suggestions** - System suggests new camera paths
|
|
60
|
+
- **Landmark Discovery** - AI identifies landmarks from footage
|
|
163
61
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
```
|
|
62
|
+
### Integrations
|
|
63
|
+
- **MQTT** - Home Assistant integration
|
|
64
|
+
- **REST API** - Query tracked objects programmatically
|
|
168
65
|
|
|
169
|
-
##
|
|
66
|
+
## Topology Configuration
|
|
170
67
|
|
|
171
|
-
|
|
172
|
-
- Open the plugin settings
|
|
173
|
-
- Click "Open Topology Editor"
|
|
174
|
-
- Upload a floor plan image (or use the drawing tools to create one)
|
|
175
|
-
- Place cameras on the floor plan
|
|
176
|
-
- Mark entry/exit points
|
|
177
|
-
- Draw connections between cameras with expected transit times
|
|
68
|
+
The plugin uses topology data (landmarks, zones, connections) to generate meaningful alerts. Camera names are **not** used for location descriptions - only topology landmarks matter.
|
|
178
69
|
|
|
179
|
-
|
|
180
|
-
- Select one or more notifiers (Pushover, email, Home Assistant, etc.)
|
|
181
|
-
- Adjust loitering threshold (how long before alerting)
|
|
182
|
-
- Adjust per-object cooldown (prevent duplicate alerts)
|
|
183
|
-
- Enable/disable specific alert types
|
|
70
|
+
### Setting Up Landmarks
|
|
184
71
|
|
|
185
|
-
|
|
186
|
-
- Install an LLM plugin (OpenAI, Ollama, etc.)
|
|
187
|
-
- Enable "Use LLM for Rich Descriptions" in settings
|
|
188
|
-
- Configure rate limiting and fallback options
|
|
189
|
-
- Get alerts like "Woman with stroller" instead of just "Person"
|
|
72
|
+
For best alert quality, configure landmarks in the topology editor:
|
|
190
73
|
|
|
191
|
-
|
|
192
|
-
-
|
|
193
|
-
-
|
|
194
|
-
- Use in Home Assistant automations
|
|
74
|
+
1. **Entry/Exit Points** - Mark where people enter/exit your property
|
|
75
|
+
- Examples: `Driveway`, `Front Gate`, `Side Gate`, `Street`
|
|
76
|
+
- Set `isEntryPoint: true` or `isExitPoint: true`
|
|
195
77
|
|
|
196
|
-
|
|
197
|
-
-
|
|
198
|
-
-
|
|
199
|
-
- Enable "Learn Landmarks from AI" for automatic landmark discovery
|
|
78
|
+
2. **Access Points** - Paths and walkways
|
|
79
|
+
- Examples: `Front Walkway`, `Back Path`, `Garage Door`
|
|
80
|
+
- Type: `access`
|
|
200
81
|
|
|
201
|
-
|
|
82
|
+
3. **Zones** - Areas of your property
|
|
83
|
+
- Examples: `Front Yard`, `Back Yard`, `Side Yard`, `Patio`
|
|
84
|
+
- Type: `zone`
|
|
202
85
|
|
|
203
|
-
|
|
86
|
+
4. **Structures** - Buildings and fixed features
|
|
87
|
+
- Examples: `Garage`, `Shed`, `Front Porch`, `Deck`
|
|
88
|
+
- Type: `structure`
|
|
204
89
|
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
-
|
|
208
|
-
- **Visual (35%)**: Do the visual embeddings match (if available)?
|
|
209
|
-
- **Spatial (25%)**: Was the object in the exit zone of the previous camera and entry zone of the new camera?
|
|
210
|
-
- **Class (10%)**: Is it the same type of object?
|
|
211
|
-
3. **New Object**: If no correlation is found, a new tracked object is created
|
|
90
|
+
5. **Features** - Other notable landmarks
|
|
91
|
+
- Examples: `Mailbox`, `Pool`, `Garden`, `Trash Cans`
|
|
92
|
+
- Type: `feature`
|
|
212
93
|
|
|
213
|
-
###
|
|
94
|
+
### Linking Landmarks to Cameras
|
|
214
95
|
|
|
215
|
-
|
|
96
|
+
Each camera should have `visibleLandmarks` configured - the landmarks visible in that camera's view:
|
|
216
97
|
|
|
217
|
-
|
|
218
|
-
|
|
98
|
+
```json
|
|
99
|
+
{
|
|
100
|
+
"cameras": [{
|
|
101
|
+
"deviceId": "abc123",
|
|
102
|
+
"name": "Front Camera",
|
|
103
|
+
"context": {
|
|
104
|
+
"visibleLandmarks": ["front-door", "driveway", "mailbox"]
|
|
105
|
+
}
|
|
106
|
+
}]
|
|
107
|
+
}
|
|
108
|
+
```
|
|
219
109
|
|
|
220
|
-
###
|
|
110
|
+
### Example Topology
|
|
111
|
+
|
|
112
|
+
```json
|
|
113
|
+
{
|
|
114
|
+
"landmarks": [
|
|
115
|
+
{ "id": "driveway", "name": "Driveway", "type": "access", "isEntryPoint": true },
|
|
116
|
+
{ "id": "front-door", "name": "Front Door", "type": "access" },
|
|
117
|
+
{ "id": "backyard", "name": "Back Yard", "type": "zone" },
|
|
118
|
+
{ "id": "garage", "name": "Garage", "type": "structure" }
|
|
119
|
+
]
|
|
120
|
+
}
|
|
121
|
+
```
|
|
221
122
|
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
4. Apply timeout (configurable, default: 3 seconds) with automatic fallback
|
|
227
|
-
5. Get a rich description like "Man in blue jacket" or "Black pickup truck"
|
|
228
|
-
6. Include this in the notification
|
|
123
|
+
With proper landmarks, alerts become rich and contextual:
|
|
124
|
+
- "Person arrived at the Driveway from Main Street"
|
|
125
|
+
- "Person moved from the Front Porch heading towards the Back Yard"
|
|
126
|
+
- "Person left the Garage towards Driveway after 2m on property - visited Driveway > Front Door > Garage"
|
|
229
127
|
|
|
230
|
-
|
|
128
|
+
## Configuration
|
|
231
129
|
|
|
232
|
-
|
|
130
|
+
### Settings
|
|
233
131
|
|
|
234
|
-
### Tracking Settings
|
|
235
|
-
| Setting | Default | Description |
|
|
236
|
-
|---------|---------|-------------|
|
|
237
|
-
| Correlation Window | 30s | Maximum time to wait for cross-camera correlation |
|
|
238
|
-
| Correlation Threshold | 0.6 | Minimum confidence (0-1) for automatic correlation |
|
|
239
|
-
| Lost Timeout | 300s | Time before marking an object as lost |
|
|
240
|
-
| Visual Matching | ON | Use visual embeddings for correlation |
|
|
241
|
-
| Loitering Threshold | 3s | Object must be visible this long before alerting |
|
|
242
|
-
| Per-Object Cooldown | 30s | Minimum time between alerts for same object |
|
|
243
|
-
|
|
244
|
-
### AI & Spatial Reasoning Settings (NEW in v0.3.0)
|
|
245
132
|
| Setting | Default | Description |
|
|
246
133
|
|---------|---------|-------------|
|
|
247
|
-
|
|
|
248
|
-
|
|
|
249
|
-
|
|
|
250
|
-
|
|
|
251
|
-
|
|
|
252
|
-
| Suggest Connections | ON | Suggest new camera connections |
|
|
253
|
-
| Learn Landmarks | ON | Allow AI to suggest landmarks |
|
|
254
|
-
| Landmark Confidence | 0.7 | Minimum confidence for landmark suggestions |
|
|
134
|
+
| Correlation Window | 30s | Max time for cross-camera matching |
|
|
135
|
+
| Correlation Threshold | 0.6 | Min confidence for auto-correlation |
|
|
136
|
+
| Loitering Threshold | 3s | Time before triggering alerts |
|
|
137
|
+
| Per-Object Cooldown | 30s | Min time between alerts for same object |
|
|
138
|
+
| LLM Rate Limit | 10s | Min time between LLM API calls |
|
|
255
139
|
|
|
256
140
|
### Alert Types
|
|
257
|
-
| Alert | Description | Default |
|
|
258
|
-
|-------|-------------|---------|
|
|
259
|
-
| Property Entry | Object entered via an entry point | Enabled |
|
|
260
|
-
| Property Exit | Object exited via an exit point | Enabled |
|
|
261
|
-
| Movement | Object moved between cameras | Enabled |
|
|
262
|
-
| Unusual Path | Object took an unexpected route | Enabled |
|
|
263
|
-
| Dwell Time | Object lingered >5 minutes | Enabled |
|
|
264
|
-
| Restricted Zone | Object entered a restricted zone | Enabled |
|
|
265
|
-
| Lost Tracking | Object disappeared without exiting | Disabled |
|
|
266
141
|
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
142
|
+
| Alert | Description |
|
|
143
|
+
|-------|-------------|
|
|
144
|
+
| Property Entry | Object entered via entry point |
|
|
145
|
+
| Property Exit | Object exited via exit point |
|
|
146
|
+
| Movement | Object moved between cameras |
|
|
147
|
+
| Unusual Path | Object took unexpected route |
|
|
148
|
+
| Dwell Time | Object lingered >5 minutes |
|
|
270
149
|
|
|
271
|
-
## API
|
|
150
|
+
## API
|
|
272
151
|
|
|
273
|
-
|
|
152
|
+
Base URL: `/endpoint/@blueharford/scrypted-spatial-awareness`
|
|
274
153
|
|
|
275
|
-
### Core Endpoints
|
|
276
154
|
| Endpoint | Method | Description |
|
|
277
155
|
|----------|--------|-------------|
|
|
278
|
-
| `/api/tracked-objects` | GET | List
|
|
279
|
-
| `/api/journey/{id}` | GET | Get journey
|
|
280
|
-
| `/api/
|
|
281
|
-
| `/api/
|
|
282
|
-
| `/api/
|
|
283
|
-
| `/api/alerts` | GET | Get recent alerts |
|
|
284
|
-
| `/api/alert-rules` | GET/PUT | Get or update alert rules |
|
|
285
|
-
| `/api/cameras` | GET | List available cameras |
|
|
286
|
-
| `/api/floor-plan` | GET/POST | Get or upload floor plan image |
|
|
156
|
+
| `/api/tracked-objects` | GET | List tracked objects |
|
|
157
|
+
| `/api/journey/{id}` | GET | Get object journey |
|
|
158
|
+
| `/api/topology` | GET/PUT | Camera topology |
|
|
159
|
+
| `/api/alerts` | GET | Recent alerts |
|
|
160
|
+
| `/api/live-tracking` | GET | Real-time object positions |
|
|
287
161
|
| `/ui/editor` | GET | Visual topology editor |
|
|
162
|
+
| `/ui/training` | GET | Training mode UI |
|
|
288
163
|
|
|
289
|
-
###
|
|
290
|
-
| Endpoint | Method | Description |
|
|
291
|
-
|----------|--------|-------------|
|
|
292
|
-
| `/api/live-tracking` | GET | Get current state of all tracked objects |
|
|
293
|
-
|
|
294
|
-
### Landmark & Spatial Reasoning Endpoints
|
|
295
|
-
| Endpoint | Method | Description |
|
|
296
|
-
|----------|--------|-------------|
|
|
297
|
-
| `/api/landmarks` | GET | List all configured landmarks |
|
|
298
|
-
| `/api/landmarks` | POST | Add a new landmark |
|
|
299
|
-
| `/api/landmarks/{id}` | GET/PUT/DELETE | Get, update, or delete a landmark |
|
|
300
|
-
| `/api/landmark-suggestions` | GET | Get AI-suggested landmarks |
|
|
301
|
-
| `/api/landmark-suggestions/{id}/accept` | POST | Accept an AI suggestion |
|
|
302
|
-
| `/api/landmark-suggestions/{id}/reject` | POST | Reject an AI suggestion |
|
|
303
|
-
| `/api/landmark-templates` | GET | Get landmark templates for quick setup |
|
|
304
|
-
| `/api/infer-relationships` | GET | Get auto-inferred spatial relationships |
|
|
305
|
-
|
|
306
|
-
### Connection Suggestion Endpoints
|
|
307
|
-
| Endpoint | Method | Description |
|
|
308
|
-
|----------|--------|-------------|
|
|
309
|
-
| `/api/connection-suggestions` | GET | Get suggested camera connections |
|
|
310
|
-
| `/api/connection-suggestions/{id}/accept` | POST | Accept a connection suggestion |
|
|
311
|
-
| `/api/connection-suggestions/{id}/reject` | POST | Reject a connection suggestion |
|
|
164
|
+
### Training API
|
|
312
165
|
|
|
313
|
-
### Training Mode Endpoints (NEW in v0.4.0)
|
|
314
166
|
| Endpoint | Method | Description |
|
|
315
167
|
|----------|--------|-------------|
|
|
316
|
-
| `/api/training/start` | POST | Start
|
|
317
|
-
| `/api/training/
|
|
318
|
-
| `/api/training/
|
|
319
|
-
| `/api/training/
|
|
320
|
-
| `/api/training/status` | GET | Get current training status and stats |
|
|
321
|
-
| `/api/training/landmark` | POST | Mark a landmark during training |
|
|
322
|
-
| `/api/training/apply` | POST | Apply training results to topology |
|
|
323
|
-
| `/ui/training` | GET | Mobile-optimized training UI |
|
|
168
|
+
| `/api/training/start` | POST | Start training session |
|
|
169
|
+
| `/api/training/end` | POST | End session, get results |
|
|
170
|
+
| `/api/training/apply` | POST | Apply results to topology |
|
|
171
|
+
| `/api/training/status` | GET | Current training status |
|
|
324
172
|
|
|
325
173
|
## MQTT Topics
|
|
326
174
|
|
|
327
|
-
|
|
175
|
+
Base: `scrypted/spatial-awareness`
|
|
328
176
|
|
|
329
177
|
| Topic | Description |
|
|
330
178
|
|-------|-------------|
|
|
331
|
-
|
|
|
332
|
-
|
|
|
333
|
-
|
|
|
334
|
-
| `
|
|
335
|
-
|
|
|
336
|
-
|
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
- **Use**: Trigger automations when property becomes occupied/unoccupied
|
|
351
|
-
|
|
352
|
-
### Tracking Zones (User-Created)
|
|
353
|
-
- **Type**: Motion + Occupancy Sensor
|
|
354
|
-
- **Purpose**: Monitor specific areas across one or more cameras
|
|
355
|
-
- **Types**: Entry, Exit, Dwell, Restricted
|
|
356
|
-
- **Use**: Create zone-specific automations and alerts
|
|
357
|
-
|
|
358
|
-
## Example Alert Messages
|
|
359
|
-
|
|
360
|
-
With LLM enabled:
|
|
361
|
-
- "Man in blue jacket walking from Garage towards Front Door (5s transit)"
|
|
362
|
-
- "Black SUV driving from Street towards Driveway"
|
|
363
|
-
- "Woman with dog walking from Back Yard towards Side Gate"
|
|
364
|
-
- "Delivery person entered property via Driveway"
|
|
365
|
-
|
|
366
|
-
Without LLM:
|
|
367
|
-
- "Person moving from Garage towards Front Door (5s transit)"
|
|
368
|
-
- "Car moving from Street towards Driveway"
|
|
369
|
-
- "Dog moving from Back Yard towards Side Gate"
|
|
370
|
-
|
|
371
|
-
## Changelog
|
|
372
|
-
|
|
373
|
-
### v0.4.0
|
|
374
|
-
- **Training Mode**: Guided walkthrough to train the system by walking your property
|
|
375
|
-
- **Mobile-Optimized Training UI**: Phone-friendly interface for training while walking
|
|
376
|
-
- **Auto Camera Detection**: System automatically detects you as you walk between cameras
|
|
377
|
-
- **Transit Time Learning**: Records actual transit times during training
|
|
378
|
-
- **Camera Overlap Detection**: Identifies where multiple cameras see the same area
|
|
379
|
-
- **Landmark Marking**: Mark landmarks (mailbox, gate, etc.) during training sessions
|
|
380
|
-
- **One-Click Topology Generation**: Apply training results to create complete topology
|
|
381
|
-
|
|
382
|
-
### v0.3.0
|
|
383
|
-
- **Live Tracking Overlay**: View tracked objects in real-time on the floor plan
|
|
384
|
-
- **Journey Visualization**: Click any tracked object to see their complete path
|
|
385
|
-
- **Transit Time Learning**: Automatically adjusts connection times based on observations
|
|
386
|
-
- **Connection Suggestions**: System suggests new camera connections
|
|
387
|
-
- **LLM Rate Limiting**: Configurable debounce intervals to prevent API overload
|
|
388
|
-
- **LLM Fallback**: Automatic fallback to basic notifications when LLM is slow
|
|
389
|
-
- **LLM Timeout**: Configurable timeout with automatic fallback
|
|
390
|
-
|
|
391
|
-
### v0.2.0
|
|
392
|
-
- **Landmark System**: Add landmarks for spatial context
|
|
393
|
-
- **RAG Reasoning**: Context-aware movement descriptions
|
|
394
|
-
- **AI Learning**: Automatic landmark suggestions
|
|
395
|
-
- **Camera Context**: Rich camera descriptions for better alerts
|
|
396
|
-
|
|
397
|
-
### v0.1.0
|
|
398
|
-
- Initial release with cross-camera tracking
|
|
399
|
-
- Entry/exit detection
|
|
400
|
-
- Movement alerts
|
|
401
|
-
- MQTT integration
|
|
402
|
-
- Visual topology editor
|
|
179
|
+
| `/occupancy/state` | ON/OFF property occupancy |
|
|
180
|
+
| `/count/state` | Active object count |
|
|
181
|
+
| `/person_count/state` | People on property |
|
|
182
|
+
| `/alerts` | Alert events |
|
|
183
|
+
| `/events/entry` | Entry events |
|
|
184
|
+
| `/events/exit` | Exit events |
|
|
185
|
+
|
|
186
|
+
## How Correlation Works
|
|
187
|
+
|
|
188
|
+
When an object is detected on a new camera, the system scores potential matches:
|
|
189
|
+
|
|
190
|
+
| Factor | Weight | Description |
|
|
191
|
+
|--------|--------|-------------|
|
|
192
|
+
| Timing | 30% | Transit time within expected range |
|
|
193
|
+
| Visual | 35% | Embedding similarity (if available) |
|
|
194
|
+
| Spatial | 25% | Exit zone → Entry zone coherence |
|
|
195
|
+
| Class | 10% | Object type match |
|
|
196
|
+
|
|
197
|
+
Objects are correlated if total score exceeds threshold (default: 0.6).
|
|
403
198
|
|
|
404
199
|
## Requirements
|
|
405
200
|
|
|
406
201
|
- Scrypted with NVR plugin
|
|
407
|
-
- Cameras with object detection
|
|
408
|
-
- Optional: LLM plugin for rich descriptions
|
|
409
|
-
- Optional: MQTT broker for Home Assistant
|
|
202
|
+
- Cameras with object detection (NVR, OpenVINO, CoreML, etc.)
|
|
203
|
+
- Optional: LLM plugin for rich descriptions
|
|
204
|
+
- Optional: MQTT broker for Home Assistant
|
|
410
205
|
|
|
411
206
|
## Development
|
|
412
207
|
|
|
413
208
|
```bash
|
|
414
|
-
# Install dependencies
|
|
415
209
|
npm install
|
|
416
|
-
|
|
417
|
-
# Build
|
|
418
210
|
npm run build
|
|
419
|
-
|
|
420
|
-
# Deploy to local Scrypted
|
|
421
211
|
npm run scrypted-deploy
|
|
422
|
-
|
|
423
|
-
# Debug in VS Code
|
|
424
|
-
# Edit .vscode/settings.json with your Scrypted server IP
|
|
425
|
-
# Press F5 to start debugging
|
|
426
212
|
```
|
|
427
213
|
|
|
428
214
|
## License
|