@camstack/addon-pipeline-orchestrator 1.2.50 → 1.2.51

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/index.mjs CHANGED
@@ -25119,13 +25119,24 @@ method(object({
25119
25119
  /** Playback-speed multiplier for the render (1 = realtime). */
25120
25120
  var ExportSpeedSchema = number().min(.25).max(32);
25121
25121
  /**
25122
- * One dense interval, in SECONDS FROM THE EXPORT'S OWN `fromMs`.
25122
+ * One dense interval, in WALL-CLOCK SECONDS FROM THE EXPORT'S OWN `fromMs`.
25123
25123
  *
25124
- * Relative and not absolute epoch on purpose: the renderer's frame-select
25125
- * expression sees ffmpeg's `t`, which starts at 0 for the export's source
25126
- * playlist. Handing it absolute epochs would make every call site responsible
25127
- * for the same subtraction, and the one that forgot would emit a filter that
25128
- * selects nothing silently, as a uniform timelapse.
25124
+ * **Wall clock, not ffmpeg's `t`** and the recorder translates. A caller
25125
+ * derives these bounds from things that happened at a TIME (a track's
25126
+ * `firstSeen`), while `t` runs over the source playlist: the concatenation of
25127
+ * every segment present for the range, with each recording GAP removed. The
25128
+ * two agree only on a window that recorded without one interruption, and only
25129
+ * the render side knows the segments, so the translation lives there
25130
+ * (`export-dense-map.ts`, addon-pipeline).
25131
+ *
25132
+ * It was not always so. These seconds were fed to `between(t,…)` verbatim, and
25133
+ * on a 10 h window holding 29,393 s of footage every range landed late by the
25134
+ * gap accumulated before it — up to 6,607 s, well past EOF. Nothing matched,
25135
+ * the video was a uniform timelapse, and the log line reported the five ranges
25136
+ * that had been ASKED for (2026-08-13, export `57d14363`, camera 615).
25137
+ *
25138
+ * Relative and not absolute epoch, because an absolute epoch would make every
25139
+ * call site responsible for the same subtraction.
25129
25140
  */
25130
25141
  var ExportDenseRangeSchema = object({
25131
25142
  fromSec: number().nonnegative(),
@@ -32367,8 +32378,8 @@ var DETAIL_CROP_PADDING_FIELD = {
32367
32378
  default: DEFAULT_DETAIL_CROP_CONVENTION.paddingRatio
32368
32379
  };
32369
32380
  /**
32370
- * THE native-frame **lease** knobs — TTL, RAM budget and demand window for the
32371
- * decode worker's native-resolution frame retention.
32381
+ * THE native-frame **lease** knobs — hold depth, RAM budget, demand window and
32382
+ * subject-tile budget for the decode worker's native-resolution retention.
32372
32383
  *
32373
32384
  * ## Why they live here and not in the addon that reads them
32374
32385
  *
@@ -32384,20 +32395,25 @@ var DETAIL_CROP_PADDING_FIELD = {
32384
32395
  * The lease is a per-decode-worker RAM window. Its purpose — the late
32385
32396
  * cross-process native crop landing on a full-resolution frame rather than the
32386
32397
  * ≤640 detection fallback — is a property of the PIPELINE, not of a node's
32387
- * hardware: a per-node TTL would mean the same camera produces different crop
32398
+ * hardware: a per-node window would mean the same camera produces different crop
32388
32399
  * quality depending on which node the balancer placed it on, and nobody could
32389
32400
  * tell that from the stored media. Node-level RAM pressure is already handled
32390
32401
  * by the per-session budget ceiling, which is itself one of these knobs.
32391
32402
  *
32392
32403
  * ## What each knob costs
32393
32404
  *
32394
- * A retained frame is a full NATIVE-resolution copy in system RAM. With the
32405
+ * A HELD frame is a full NATIVE-resolution copy in system RAM. With the
32395
32406
  * default pinned-RGB24 lease path (`CAMSTACK_SESSION_PINNED_RGB_CROP`, on):
32396
32407
  * 4K ≈ 24.9 MB/frame, 1080p ≈ 6.2 MB/frame. On the YUV420P path (flag off, and
32397
32408
  * for software-decoded sessions): 4K ≈ 12.4 MB, 1080p ≈ 3.1 MB. Worst-case
32398
- * resident RAM for ONE busy camera frameBytes × deliveredFps × ttlSeconds,
32399
- * clamped by the budget ceiling. See `docs/design/decode-path.md` "Lease
32400
- * admission" for what actually gets admitted.
32409
+ * resident RAM for ONE busy camera is now `frameBytes × holdFrames`, clamped by
32410
+ * the budget ceiling bounded by a COUNT because a held frame is waiting for
32411
+ * one specific event (its own detection result), not for a clock.
32412
+ *
32413
+ * A TILE is one subject at native resolution, JPEG-encoded: ~60-120 KB on 4K,
32414
+ * and nothing at all on a frame that detected nothing. That is the asymmetry
32415
+ * this whole shape exists for — see
32416
+ * `docs/design/2026-08-13-native-lease-two-tier-redesign.md`.
32401
32417
  */
32402
32418
  /**
32403
32419
  * Store identity of the lease knobs in `pipeline-orchestrator`'s GLOBAL
@@ -32405,10 +32421,11 @@ var DETAIL_CROP_PADDING_FIELD = {
32405
32421
  * the reader can walk every section instead of trusting the section id.
32406
32422
  */
32407
32423
  var NATIVE_LEASE_SECTION_ID = "native-lease";
32408
- var NATIVE_LEASE_TTL_KEY = "nativeLeaseTtlMs";
32424
+ var NATIVE_LEASE_HOLD_KEY = "nativeLeaseHoldFrames";
32409
32425
  var NATIVE_LEASE_BUDGET_KEY = "nativeLeaseBudgetMb";
32410
32426
  var NATIVE_LEASE_ACTIVITY_KEY = "nativeLeaseActivityMs";
32411
32427
  var NATIVE_LEASE_ADMISSION_KEY = "nativeLeaseAdmission";
32428
+ var NATIVE_LEASE_TILE_BUDGET_KEY = "nativeTileBudgetMb";
32412
32429
  /**
32413
32430
  * WHICH delivered frames the decode worker retains a native copy of.
32414
32431
  *
@@ -32430,25 +32447,32 @@ var NATIVE_LEASE_ADMISSION_KEY = "nativeLeaseAdmission";
32430
32447
  var NativeLeaseAdmissionSchema = _enum(["all", "inferred"]);
32431
32448
  object({
32432
32449
  /**
32433
- * How long a retained native frame is served before it counts as a miss.
32450
+ * How many delivered frames the worker HOLDS at once, waiting for each one's
32451
+ * detection result.
32452
+ *
32453
+ * This replaced a TTL on 2026-08-13, and the replacement is the whole point:
32454
+ * a time window was never related to the event the pixels were waiting for.
32455
+ * A held frame now lives from delivery until the runner has its `FrameResult`
32456
+ * — at which moment the runner cuts the subject tiles it actually wanted and
32457
+ * releases the frame. The bound exists only so a runner that stops answering
32458
+ * cannot pin RAM: above it the OLDEST held frame is dropped and counted.
32434
32459
  *
32435
- * Must cover the FULL late-crop horizon: detection inference + the
32436
- * cross-process inference-result hop to hub post-analysis + tracking + the
32437
- * tRPC crop round-trip back. Below ~500 ms the busiest cameras' subject crops
32438
- * outrun it and fall back to the ≤640 detection frame; above ~3 s the resident
32439
- * RAM per busy camera grows linearly with no measured hit-rate gain.
32460
+ * Sizing: the steady state is `inferenceLatency × deliveredFps`, measured at
32461
+ * 40-160 ms × ≤25 fps = 1-4 frames. The default leaves headroom for a hiccup
32462
+ * without ever approaching the old resident set (43 frames × 24.9 MB at 4K).
32463
+ * Raising it does not buy hit rate it buys tolerance for a slow runner, and
32464
+ * `holdOverflow` on the metrics line is what says you need it.
32440
32465
  */
32441
- ttlMs: number().int().min(250).max(1e4),
32466
+ holdFrames: number().int().min(1).max(64),
32442
32467
  /**
32443
32468
  * Hard per-decode-worker RAM ceiling for retained native frames, in MB.
32444
32469
  *
32445
- * Intended as a SAFETY ceiling with the TTL as the effective cap — but check
32446
- * which one is actually binding before reasoning from that. At the shipped
32447
- * 1024 MB and a 2 800 ms TTL, a 4K camera hits the CEILING first (~43 frames
32448
- * at ~24 MB each) and the TTL never gets to expire anything; `leaseMb` /
32449
- * `leaseFrames` on the metrics line say which. When the ceiling binds, a
32450
- * change that admits fewer frames buys retention WINDOW at constant RAM
32451
- * rather than giving RAM back — lower this knob if RAM is what you wanted.
32470
+ * Since 2026-08-13 this is a SAFETY ceiling and nothing else: `holdFrames`
32471
+ * is what decides how much is held, and the ceiling is the number above which
32472
+ * something is wrong. Before that it was the effective cap at 1024 MB with
32473
+ * a 2 800 ms TTL a 4K camera sat pinned at `leaseMb:1020, leaseFrames:43`
32474
+ * with the TTL expiring nothing, which is exactly the confusion the hold
32475
+ * removes. `leaseMb` / `leaseFrames` still say what is resident.
32452
32476
  * `0` DISABLES the lease entirely and falls the worker back to the tiny
32453
32477
  * leak-prone GPU surface ring (~85% crop miss; that is what the lease exists
32454
32478
  * to replace).
@@ -32474,25 +32498,47 @@ object({
32474
32498
  * there is the signal that some caller names frames outside the inference set
32475
32499
  * and that this must go back to `all`.
32476
32500
  */
32477
- admission: NativeLeaseAdmissionSchema
32501
+ admission: NativeLeaseAdmissionSchema,
32502
+ /**
32503
+ * RAM ceiling per decode worker, in MB, for the SUBJECT TILES — the
32504
+ * compressed native crops the worker cuts at the moment a frame's detection
32505
+ * result arrives, and keeps long after the frame itself is freed.
32506
+ *
32507
+ * This is the knob that replaced the old retention window, and it buys about
32508
+ * three orders of magnitude more of it: a tile is one subject at native
32509
+ * resolution, JPEG-encoded (~60-120 KB on a 4K person), against ~24.9 MB for
32510
+ * the frame it was cut from. A frame on which nothing was detected costs
32511
+ * nothing at all, which is the real change — the old lease paid per FRAME and
32512
+ * was interrogated per SUBJECT.
32513
+ *
32514
+ * `0` DISABLES tiles, leaving only the hold window and the ≤640 RAM
32515
+ * fallback — i.e. the pre-2026-08-13 miss profile. Set it there only to
32516
+ * reproduce that.
32517
+ */
32518
+ tileBudgetMb: number().int().min(0).max(1024)
32478
32519
  });
32479
32520
  /**
32480
- * The values in force when the operator has set nothing — byte-for-byte the
32481
- * constants the decode worker shipped with as env-var defaults, so making these
32482
- * settings changed no behaviour on the day it landed.
32521
+ * The values in force when the operator has set nothing.
32522
+ *
32523
+ * `budgetMb` stays at 1024 on the day the hold landed, deliberately: it stopped
32524
+ * being the retention window and became the OOM ceiling, and lowering a ceiling
32525
+ * in the same change that redefines it would make a regression and a retune
32526
+ * indistinguishable. Cut it once `tileHits` / `holdOverflow` have been read on
32527
+ * live traffic.
32483
32528
  */
32484
32529
  var DEFAULT_NATIVE_LEASE_SETTINGS = {
32485
- ttlMs: 1200,
32530
+ holdFrames: 8,
32486
32531
  budgetMb: 1024,
32487
32532
  activityMs: 15e3,
32533
+ tileBudgetMb: 64,
32488
32534
  admission: "inferred"
32489
32535
  };
32490
32536
  /** Slider bounds for the operator-facing knobs (orchestrator settings UI). */
32491
- var NATIVE_LEASE_TTL_FIELD = {
32492
- min: 250,
32493
- max: 1e4,
32494
- step: 50,
32495
- default: DEFAULT_NATIVE_LEASE_SETTINGS.ttlMs
32537
+ var NATIVE_LEASE_HOLD_FIELD = {
32538
+ min: 1,
32539
+ max: 64,
32540
+ step: 1,
32541
+ default: DEFAULT_NATIVE_LEASE_SETTINGS.holdFrames
32496
32542
  };
32497
32543
  var NATIVE_LEASE_BUDGET_FIELD = {
32498
32544
  min: 0,
@@ -32506,6 +32552,12 @@ var NATIVE_LEASE_ACTIVITY_FIELD = {
32506
32552
  step: 1e3,
32507
32553
  default: DEFAULT_NATIVE_LEASE_SETTINGS.activityMs
32508
32554
  };
32555
+ var NATIVE_LEASE_TILE_BUDGET_FIELD = {
32556
+ min: 0,
32557
+ max: 1024,
32558
+ step: 16,
32559
+ default: DEFAULT_NATIVE_LEASE_SETTINGS.tileBudgetMb
32560
+ };
32509
32561
  /** Select options for the admission knob (orchestrator settings UI). */
32510
32562
  var NATIVE_LEASE_ADMISSION_FIELD = {
32511
32563
  options: [{
@@ -32879,7 +32931,7 @@ var TRANSIENT_SLOT_READ = Symbol("transient-slot-read");
32879
32931
  * shape so video and audio plumbing self-heal identically.
32880
32932
  */
32881
32933
  /** Poll period — audio chunks arrive ~every 500ms; 200ms keeps latency low. */
32882
- var POLL_INTERVAL_MS = 200;
32934
+ var POLL_INTERVAL_MS$1 = 200;
32883
32935
  /** How many chunks to drain per poll — a small burst absorbs jitter. */
32884
32936
  var PULL_MAX_COUNT = 8;
32885
32937
  /**
@@ -33062,7 +33114,7 @@ function startPolling(options, lifecycle) {
33062
33114
  } });
33063
33115
  if (!lifecycle.stopped && consecutiveFailures >= RESUBSCRIBE_AFTER_FAILURES && (consecutiveFailures - RESUBSCRIBE_AFTER_FAILURES) % RESUBSCRIBE_THROTTLE_TICKS === 0) await resubscribe();
33064
33116
  }
33065
- if (!lifecycle.stopped) lifecycle.pollTimer = setTimeout(() => void tick(), POLL_INTERVAL_MS);
33117
+ if (!lifecycle.stopped) lifecycle.pollTimer = setTimeout(() => void tick(), POLL_INTERVAL_MS$1);
33066
33118
  };
33067
33119
  tick();
33068
33120
  }
@@ -41603,6 +41655,79 @@ async function migrateStepGating(deps) {
41603
41655
  } });
41604
41656
  }
41605
41657
  //#endregion
41658
+ //#region src/zone-mirror-hydration.ts
41659
+ /** How long to wait for a camera list: the hub wires `ctx.api` AFTER the addon
41660
+ * init chain resolves, and `device-manager` answers a moment later still, so a
41661
+ * task kicked off from `onInitialize` loses both races. Same shape as the
41662
+ * bindings migration's poll — one budget covering "no api yet" and "api, but
41663
+ * device-manager not answering yet", because to this sweep they are the same
41664
+ * thing: no fleet to hydrate. */
41665
+ var CAMERA_LIST_WAIT_MS = 6e3;
41666
+ var POLL_INTERVAL_MS = 200;
41667
+ /**
41668
+ * Hydrate the `zones` mirror for every camera once, at boot. Never throws: a
41669
+ * hydration sweep that cannot run must not take the orchestrator's boot with
41670
+ * it — but it is never silent either, because a skipped sweep is exactly the
41671
+ * failure this exists to end.
41672
+ */
41673
+ async function hydrateZoneMirrorsAtBoot(deps) {
41674
+ const cameraIds = await readCameraIds(deps);
41675
+ if (cameraIds === null) return;
41676
+ for (const hydrator of deps.hydrators) {
41677
+ let written = 0;
41678
+ let unchanged = 0;
41679
+ let failed = 0;
41680
+ for (const deviceId of cameraIds) try {
41681
+ const outcome = await hydrator.hydrate(deviceId);
41682
+ if (outcome === "written") written++;
41683
+ else if (outcome === "unchanged") unchanged++;
41684
+ else failed++;
41685
+ } catch (err) {
41686
+ failed++;
41687
+ deps.logger.warn("zone mirror boot hydration failed for this camera", {
41688
+ tags: { deviceId },
41689
+ meta: {
41690
+ mirror: hydrator.mirror,
41691
+ error: errMsg(err)
41692
+ }
41693
+ });
41694
+ }
41695
+ deps.logger.info("zone mirror boot hydration complete", { meta: {
41696
+ mirror: hydrator.mirror,
41697
+ cameras: cameraIds.length,
41698
+ written,
41699
+ unchanged,
41700
+ failed
41701
+ } });
41702
+ }
41703
+ }
41704
+ /**
41705
+ * The camera fleet, or `null` when it never became readable inside the budget
41706
+ * — in which case this has already said so. Retried inside one budget rather
41707
+ * than once, because at orchestrator boot "no api yet" and "device-manager not
41708
+ * answering yet" are both transient and indistinguishable from here.
41709
+ */
41710
+ async function readCameraIds(deps) {
41711
+ const deadline = Date.now() + (deps.apiWaitMs ?? CAMERA_LIST_WAIT_MS);
41712
+ let lastError = null;
41713
+ for (;;) {
41714
+ const api = deps.api();
41715
+ if (api) try {
41716
+ return (await api.deviceManager.listAll.query({
41717
+ isCamera: true,
41718
+ projection: "slim"
41719
+ })).map((camera) => camera.id);
41720
+ } catch (err) {
41721
+ lastError = err;
41722
+ }
41723
+ if (Date.now() >= deadline) {
41724
+ deps.logger.warn("zones mirror boot hydration SKIPPED — camera list never became readable; mirror-only consumers keep whatever the last write left", { meta: { error: lastError === null ? "ctx.api unavailable" : errMsg(lastError) } });
41725
+ return null;
41726
+ }
41727
+ await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL_MS));
41728
+ }
41729
+ }
41730
+ //#endregion
41606
41731
  //#region src/zone-rules-provider.ts
41607
41732
  /**
41608
41733
  * `zone-rules-provider.ts` — implements `zoneRulesCapability` for the
@@ -41619,6 +41744,20 @@ async function migrateStepGating(deps) {
41619
41744
  * before persisting — partial / corrupt writes are rejected outright
41620
41745
  * since rules drive runtime filtering and a bad payload would silently
41621
41746
  * widen the operator's intended scope.
41747
+ *
41748
+ * ── THE MIRROR IS HYDRATED AT BOOT, NOT ONLY ON MUTATION ──────────
41749
+ *
41750
+ * Same root cause as the `zones` slice (see `zones-provider.ts`): the
41751
+ * mirror used to be written only by `persist`, so a camera nobody had
41752
+ * mutated since its runtime-state row was last written had NO
41753
+ * `zone-rules` slice, and every mirror-only consumer — motion-wasm's
41754
+ * zone gate, the detection-pipeline zone gate, the admin rules editor
41755
+ * — read "no rules" until an operator happened to save one. Zones and
41756
+ * rules gate together, so hydrating one without the other still leaves
41757
+ * both runner-side gates inert. {@link ZoneRulesProvider.hydrateMirror}
41758
+ * is what the boot sweep (`zone-mirror-hydration.ts`) calls; it is an
41759
+ * RPC read, never an event replay (D8), and a stage that could not be
41760
+ * read leaves the mirror untouched (D49).
41622
41761
  */
41623
41762
  /**
41624
41763
  * Every zone-rule stage, in the declared enum order. The unified device-state
@@ -41662,6 +41801,11 @@ var ZoneRulesProvider = class {
41662
41801
  * write on one stage can never drop the other. Built lazily + memoised.
41663
41802
  */
41664
41803
  stateByDevice = /* @__PURE__ */ new Map();
41804
+ /** Per-device fingerprint of the slice this process last successfully wrote.
41805
+ * Absent ⇒ never mirrored here, so the next hydration writes. */
41806
+ mirroredFingerprint = /* @__PURE__ */ new Map();
41807
+ /** Devices already reported as unmirrorable — one warn per episode. */
41808
+ reportedMirrorFailure = /* @__PURE__ */ new Set();
41665
41809
  constructor(ctx) {
41666
41810
  this.ctx = ctx;
41667
41811
  }
@@ -41704,10 +41848,52 @@ var ZoneRulesProvider = class {
41704
41848
  if (!parsed.success) throw new Error(`zone-rules.setRules: invalid rule payload — ${parsed.error.issues.map((i) => i.message).join("; ")}`);
41705
41849
  await this.persist(deviceId, stage, parsed.data);
41706
41850
  }
41851
+ /**
41852
+ * Reconcile ONE device's `zone-rules` mirror against the durable block.
41853
+ * Called by the boot sweep for every camera. Never throws.
41854
+ *
41855
+ * A stage whose read FAILED aborts the whole hydration: the mirror is a
41856
+ * single slice carrying every stage, so writing a partially-read block would
41857
+ * publish "this stage has no rules" off a store blip — and an empty
41858
+ * `motion`/`detection` array is what makes a gate stop gating.
41859
+ */
41860
+ async hydrateMirror(deviceId) {
41861
+ const perDevice = this.stageCache(deviceId);
41862
+ for (const stage of ALL_STAGES) {
41863
+ const read = await this.readRules(deviceId, stage);
41864
+ if (!read.ok) return "unreadable";
41865
+ perDevice.set(stage, read.rules);
41866
+ }
41867
+ const slice = await this.buildSliceValue(deviceId, perDevice);
41868
+ const fingerprint = JSON.stringify(slice);
41869
+ if (this.mirroredFingerprint.get(deviceId) === fingerprint) return "unchanged";
41870
+ try {
41871
+ await (await this.ctx.fetchDevice(deviceId)).deviceState.setCapSlice({
41872
+ capName: ZONE_RULES_CAP_NAME,
41873
+ slice
41874
+ });
41875
+ } catch (err) {
41876
+ if (!this.reportedMirrorFailure.has(deviceId)) {
41877
+ this.reportedMirrorFailure.add(deviceId);
41878
+ this.ctx.logger.warn("zone-rules mirror write failed — mirror-only zone gates see NO rules for this camera until it lands", {
41879
+ tags: { deviceId },
41880
+ meta: { error: err instanceof Error ? err.message : String(err) }
41881
+ });
41882
+ }
41883
+ return "write-failed";
41884
+ }
41885
+ const first = !this.mirroredFingerprint.has(deviceId);
41886
+ this.mirroredFingerprint.set(deviceId, fingerprint);
41887
+ this.reportedMirrorFailure.delete(deviceId);
41888
+ if (first) this.ctx.logger.info("zone-rules mirror hydrated from the durable block", { tags: { deviceId } });
41889
+ return "written";
41890
+ }
41707
41891
  /** Drop a device's cache entries. Called when the device is removed. */
41708
41892
  forgetDevice(deviceId) {
41709
41893
  this.cache.delete(deviceId);
41710
41894
  this.stateByDevice.delete(deviceId);
41895
+ this.mirroredFingerprint.delete(deviceId);
41896
+ this.reportedMirrorFailure.delete(deviceId);
41711
41897
  }
41712
41898
  /** Cap-surface read: a failure folds to the empty list, as it always has. */
41713
41899
  async loadRules(deviceId, stage) {
@@ -41722,11 +41908,7 @@ var ZoneRulesProvider = class {
41722
41908
  * momentary store blip into a permanent one.
41723
41909
  */
41724
41910
  async readRules(deviceId, stage) {
41725
- let perDevice = this.cache.get(deviceId);
41726
- if (!perDevice) {
41727
- perDevice = /* @__PURE__ */ new Map();
41728
- this.cache.set(deviceId, perDevice);
41729
- }
41911
+ const perDevice = this.stageCache(deviceId);
41730
41912
  const cached = perDevice.get(stage);
41731
41913
  if (cached) return {
41732
41914
  ok: true,
@@ -41767,11 +41949,7 @@ var ZoneRulesProvider = class {
41767
41949
  };
41768
41950
  }
41769
41951
  async persist(deviceId, stage, rules) {
41770
- let perDevice = this.cache.get(deviceId);
41771
- if (!perDevice) {
41772
- perDevice = /* @__PURE__ */ new Map();
41773
- this.cache.set(deviceId, perDevice);
41774
- }
41952
+ const perDevice = this.stageCache(deviceId);
41775
41953
  perDevice.set(stage, rules);
41776
41954
  await this.rulesState(deviceId).update((prev) => ({
41777
41955
  ...prev,
@@ -41783,6 +41961,8 @@ var ZoneRulesProvider = class {
41783
41961
  capName: ZONE_RULES_CAP_NAME,
41784
41962
  slice: sliceValue
41785
41963
  });
41964
+ this.mirroredFingerprint.set(deviceId, JSON.stringify(sliceValue));
41965
+ this.reportedMirrorFailure.delete(deviceId);
41786
41966
  } catch (err) {
41787
41967
  this.ctx.logger.debug("zone-rules slice mirror failed", {
41788
41968
  tags: { deviceId },
@@ -41801,6 +41981,15 @@ var ZoneRulesProvider = class {
41801
41981
  * also warms the cache). Iterates {@link ALL_STAGES} so it stays exhaustive
41802
41982
  * over the cap's stage discriminator without a per-stage branch.
41803
41983
  */
41984
+ /** The per-stage cache map for a device, created on first use. */
41985
+ stageCache(deviceId) {
41986
+ let perDevice = this.cache.get(deviceId);
41987
+ if (!perDevice) {
41988
+ perDevice = /* @__PURE__ */ new Map();
41989
+ this.cache.set(deviceId, perDevice);
41990
+ }
41991
+ return perDevice;
41992
+ }
41804
41993
  async buildSliceValue(deviceId, perDevice) {
41805
41994
  const slice = {
41806
41995
  motion: [],
@@ -41818,11 +42007,38 @@ var ZoneRulesProvider = class {
41818
42007
  *
41819
42008
  * Per-camera CRUD over polygon detection zones. Persists to the
41820
42009
  * orchestrator's per-device settings store under the `zones` key and
41821
- * mirrors every change into the device-state `zones` slice via
42010
+ * mirrors the catalogue into the device-state `zones` slice via
41822
42011
  * `api.deviceState.setCapSlice`. Downstream consumers (motion-wasm,
41823
42012
  * pipeline-executor, analytics, admin UI) read the live state with
41824
42013
  * the canonical `dev.state.zones.onChanged` channel.
41825
42014
  *
42015
+ * ── THIS PROVIDER OWNS THE MIRROR FOR ITS WHOLE LIFETIME ──────────
42016
+ *
42017
+ * The mirror used to be written in exactly ONE place: `persist`, i.e.
42018
+ * only when an operator mutates a zone. Nothing seeded it at startup,
42019
+ * so a camera whose runtime-state row had never been written (or had
42020
+ * been reset) had no `zones` slice at all, and every mirror-only
42021
+ * consumer concluded "this camera has no zones" — forever, because no
42022
+ * mutation was coming. Live cost (camera 617 'Parcheggio',
42023
+ * 2026-08-12): `zones.listZones {617}` returned 'Parcheggio papà'
42024
+ * while `deviceState.getCapSlice {617,'zones'}` returned `null`;
42025
+ * occupancy dropped all three parked cars into `unzoned`, its zone
42026
+ * rule could never fire, and the admin Zones tab — which reads the
42027
+ * same mirror — showed "No zones yet".
42028
+ *
42029
+ * So the mirror is reconciled against the durable catalogue on the
42030
+ * READ path too ({@link ZonesProvider.hydrateMirror}), and the boot
42031
+ * sweep in `zone-mirror-hydration.ts` walks every camera once at
42032
+ * startup. That is deliberately an RPC read, never an event replay:
42033
+ * events are lossy telemetry and a slice change that was dropped is
42034
+ * never re-sent (D8).
42035
+ *
42036
+ * Two rules the hydration path must keep (D49):
42037
+ * - a durable read that FAILED changes nothing — it must never be
42038
+ * mirrored, and must not be cached as "this camera has no zones";
42039
+ * - hydration is a reconcile, not a mutation: `onZonesChanged` is
42040
+ * NOT fired, so nothing downstream re-dispatches on a boot read.
42041
+ *
41826
42042
  * Onboard / firmware-reported zones are out of scope for now — every
41827
42043
  * zone is operator-drawn. The provider keeps the surface symmetric:
41828
42044
  * `addZone` rejects id collisions, `updateZone` requires an existing
@@ -41831,6 +42047,11 @@ var ZoneRulesProvider = class {
41831
42047
  var ZONES_STORE_KEY = "zones";
41832
42048
  var ZONES_CAP_NAME = "zones";
41833
42049
  var ZonesArraySchema = array(ZoneSchema);
42050
+ /** Identity of a mirrored catalogue — cheap enough to compare on every read,
42051
+ * and it changes whenever anything an operator can see changes. */
42052
+ function fingerprintZones(zones) {
42053
+ return JSON.stringify(zones);
42054
+ }
41834
42055
  var ZonesProvider = class {
41835
42056
  ctx;
41836
42057
  /** Per-device cache. Hydrated lazily on first read for a device. */
@@ -41841,6 +42062,16 @@ var ZonesProvider = class {
41841
42062
  * be dropped on persist. Built lazily + memoised per device.
41842
42063
  */
41843
42064
  stateByDevice = /* @__PURE__ */ new Map();
42065
+ /**
42066
+ * Per-device fingerprint of the catalogue this process last successfully
42067
+ * wrote to the mirror. Absent ⇒ this process has never mirrored the device,
42068
+ * so the next read hydrates; a write that FAILED leaves it absent, which is
42069
+ * what makes the retry happen on the next read rather than never.
42070
+ */
42071
+ mirroredFingerprint = /* @__PURE__ */ new Map();
42072
+ /** Devices already reported as unmirrorable — keeps the warn to one per
42073
+ * episode instead of one per read. */
42074
+ reportedMirrorFailure = /* @__PURE__ */ new Set();
41844
42075
  constructor(ctx) {
41845
42076
  this.ctx = ctx;
41846
42077
  }
@@ -41866,55 +42097,141 @@ var ZonesProvider = class {
41866
42097
  }
41867
42098
  return handle;
41868
42099
  }
42100
+ /**
42101
+ * The device's catalogue — and, on the way past, the one place a mirror-only
42102
+ * consumer's boot blindness is cured: every authoritative read reconciles the
42103
+ * device-state slice against what it just read.
42104
+ */
41869
42105
  async listZones({ deviceId }) {
41870
- return this.loadZones(deviceId);
42106
+ const read = await this.readCatalogue(deviceId);
42107
+ if (read.ok) await this.ensureMirror(deviceId, read.zones);
42108
+ return read.zones;
41871
42109
  }
41872
42110
  async addZone({ deviceId, zone }) {
41873
- const existing = await this.loadZones(deviceId);
42111
+ const existing = await this.readForMutation(deviceId);
41874
42112
  if (existing.some((entry) => entry.id === zone.id)) throw new Error(`zones.addZone: id ${zone.id} already exists`);
41875
42113
  await this.persist(deviceId, [...existing, zone]);
41876
42114
  }
41877
42115
  async updateZone({ deviceId, zone }) {
41878
- const existing = await this.loadZones(deviceId);
42116
+ const existing = await this.readForMutation(deviceId);
41879
42117
  if (!existing.some((entry) => entry.id === zone.id)) throw new Error(`zones.updateZone: id ${zone.id} not found`);
41880
42118
  const next = existing.map((entry) => entry.id === zone.id ? zone : entry);
41881
42119
  await this.persist(deviceId, next);
41882
42120
  }
41883
42121
  async removeZone({ deviceId, zoneId }) {
41884
- const existing = await this.loadZones(deviceId);
42122
+ const existing = await this.readForMutation(deviceId);
41885
42123
  if (!existing.some((entry) => entry.id === zoneId)) return;
41886
42124
  await this.persist(deviceId, existing.filter((entry) => entry.id !== zoneId));
41887
42125
  }
41888
42126
  /**
42127
+ * Reconcile ONE device's mirror against the durable catalogue. The boot
42128
+ * sweep (`zone-mirror-hydration.ts`) calls this for every camera so a
42129
+ * mirror-only consumer never starts blind; `listZones` calls it too, so a
42130
+ * camera adopted after boot is covered by its first read.
42131
+ *
42132
+ * Never throws — a hydration that cannot happen is reported, and reported
42133
+ * once (see {@link ZoneMirrorHydration}).
42134
+ */
42135
+ async hydrateMirror(deviceId) {
42136
+ const read = await this.readCatalogue(deviceId);
42137
+ if (!read.ok) return "unreadable";
42138
+ return this.ensureMirror(deviceId, read.zones);
42139
+ }
42140
+ /**
41889
42141
  * Drop a device's cache entry. Called when the device is removed so
41890
42142
  * the next attach starts from a fresh disk read.
41891
42143
  */
41892
42144
  forgetDevice(deviceId) {
41893
42145
  this.cache.delete(deviceId);
41894
42146
  this.stateByDevice.delete(deviceId);
42147
+ this.mirroredFingerprint.delete(deviceId);
42148
+ this.reportedMirrorFailure.delete(deviceId);
41895
42149
  }
41896
- async loadZones(deviceId) {
42150
+ /**
42151
+ * The catalogue for a mutation. A mutation is read-modify-write over the
42152
+ * WHOLE array, so proceeding from a failed read would persist the operator's
42153
+ * zones away — refuse instead.
42154
+ */
42155
+ async readForMutation(deviceId) {
42156
+ const read = await this.readCatalogue(deviceId);
42157
+ if (!read.ok) throw new Error(`zones: catalogue unreadable for device ${deviceId} — refusing to write`);
42158
+ return read.zones;
42159
+ }
42160
+ /**
42161
+ * Read the durable catalogue, cached per device. A FAILED read is neither
42162
+ * cached nor reported as `[]` — the caller decides what an unknown answer
42163
+ * means for it.
42164
+ */
42165
+ async readCatalogue(deviceId) {
41897
42166
  const cached = this.cache.get(deviceId);
41898
- if (cached) return cached;
41899
- let zones = [];
42167
+ if (cached) return {
42168
+ ok: true,
42169
+ zones: cached
42170
+ };
42171
+ let zones;
41900
42172
  try {
41901
42173
  zones = await this.zonesState(deviceId).get();
41902
42174
  } catch (err) {
41903
- this.ctx.logger.warn("zones store read failed — using empty list", {
42175
+ this.ctx.logger.warn("zones store read failed — catalogue UNKNOWN for this device", {
41904
42176
  tags: { deviceId },
41905
42177
  meta: { error: err instanceof Error ? err.message : String(err) }
41906
42178
  });
42179
+ return {
42180
+ ok: false,
42181
+ zones: []
42182
+ };
41907
42183
  }
41908
42184
  this.cache.set(deviceId, zones);
41909
- return zones;
42185
+ return {
42186
+ ok: true,
42187
+ zones
42188
+ };
41910
42189
  }
41911
- async persist(deviceId, zones) {
41912
- this.cache.set(deviceId, zones);
41913
- await this.zonesState(deviceId).set(zones);
42190
+ /**
42191
+ * Make the device-state mirror agree with `zones`. Idempotent per process
42192
+ * via the fingerprint; the hub itself also no-ops an identical
42193
+ * `setCapSlice`, so this is belt-and-braces against needless RPCs, not
42194
+ * against needless writes.
42195
+ */
42196
+ async ensureMirror(deviceId, zones) {
42197
+ const fingerprint = fingerprintZones(zones);
42198
+ if (this.mirroredFingerprint.get(deviceId) === fingerprint) return "unchanged";
42199
+ try {
42200
+ await this.writeMirror(deviceId, zones);
42201
+ } catch (err) {
42202
+ if (!this.reportedMirrorFailure.has(deviceId)) {
42203
+ this.reportedMirrorFailure.add(deviceId);
42204
+ this.ctx.logger.warn("zones mirror write failed — mirror-only consumers see NO zones for this camera until it lands", {
42205
+ tags: { deviceId },
42206
+ meta: {
42207
+ zones: zones.length,
42208
+ error: err instanceof Error ? err.message : String(err)
42209
+ }
42210
+ });
42211
+ }
42212
+ return "write-failed";
42213
+ }
42214
+ const first = !this.mirroredFingerprint.has(deviceId);
42215
+ this.mirroredFingerprint.set(deviceId, fingerprint);
42216
+ this.reportedMirrorFailure.delete(deviceId);
42217
+ if (first) this.ctx.logger.info("zones mirror hydrated from the durable catalogue", {
42218
+ tags: { deviceId },
42219
+ meta: { zones: zones.length }
42220
+ });
42221
+ return "written";
42222
+ }
42223
+ async writeMirror(deviceId, zones) {
41914
42224
  await (await this.ctx.fetchDevice(deviceId)).deviceState.setCapSlice({
41915
42225
  capName: ZONES_CAP_NAME,
41916
42226
  slice: { zones }
41917
42227
  });
42228
+ }
42229
+ async persist(deviceId, zones) {
42230
+ this.cache.set(deviceId, zones);
42231
+ await this.zonesState(deviceId).set(zones);
42232
+ await this.writeMirror(deviceId, zones);
42233
+ this.mirroredFingerprint.set(deviceId, fingerprintZones(zones));
42234
+ this.reportedMirrorFailure.delete(deviceId);
41918
42235
  this.ctx.onZonesChanged?.(deviceId, zones);
41919
42236
  }
41920
42237
  };
@@ -42327,6 +42644,19 @@ async function buildOrchestratorControllers(deps) {
42327
42644
  });
42328
42645
  }
42329
42646
  });
42647
+ hydrateZoneMirrorsAtBoot({
42648
+ api: () => deps.ctx().api ?? null,
42649
+ hydrators: [{
42650
+ mirror: "zones",
42651
+ hydrate: (deviceId) => zonesProvider.hydrateMirror(deviceId)
42652
+ }, {
42653
+ mirror: "zone-rules",
42654
+ hydrate: (deviceId) => zoneRulesProvider.hydrateMirror(deviceId)
42655
+ }],
42656
+ logger: deps.ctx().logger.child("zones")
42657
+ }).catch((err) => {
42658
+ deps.ctxIfReady()?.logger.warn("zones mirror boot hydration failed", { meta: { error: errMsg(err) } });
42659
+ });
42330
42660
  const unsubOrchestratorSubscriptions = wireOrchestratorSubscriptions({
42331
42661
  eventBus: deps.ctx().eventBus,
42332
42662
  logger: deps.ctx().logger,
@@ -42630,25 +42960,37 @@ function buildGlobalSettingsSections(options) {
42630
42960
  id: NATIVE_LEASE_SECTION_ID,
42631
42961
  title: "Native frame lease",
42632
42962
  tab: "pipeline",
42633
- description: "How long each decode worker keeps a full-resolution copy of a delivered frame in RAM so a LATE native crop (post-analysis snapshot, face/plate detail) can still be cut from real pixels instead of the ≤640 detection frame. Cost is real: a retained 4K frame is ~24.9 MB on the default pinned-RGB24 path (~12.4 MB as YUV420P), a 1080p frame ~6.2 MB / ~3.1 MB. Worst case for ONE busy camera frameBytes × delivered fps × TTL seconds, capped by the budget below. Takes effect on the NEXT decode session for a camera, not on sessions already running.",
42963
+ description: "How a decode worker keeps native pixels available for a LATE crop (post-analysis snapshot, face/plate detail) instead of falling back to the ≤640 detection frame. Two things are kept and they cost very differently: a HELD FRAME is a full native raster (~24.9 MB at 4K on the default pinned-RGB24 path, ~6.2 MB at 1080p) and lives only until its own detection result arrives; a TILE is one subject cut from that frame at native resolution and JPEG-encoded (~60-120 KB at 4K), and lives for tens of seconds. A frame on which nothing was detected produces no tiles and costs nothing. Takes effect on the NEXT decode session for a camera, not on sessions already running.",
42634
42964
  fields: [
42635
42965
  {
42636
- key: NATIVE_LEASE_TTL_KEY,
42966
+ key: NATIVE_LEASE_HOLD_KEY,
42637
42967
  type: "slider",
42638
- label: "Lease TTL",
42639
- description: "How long a retained native frame stays claimable before it counts as a miss. It must cover the whole late-crop horizon detection inference, the cross-process hop to hub post-analysis, tracking, and the tRPC crop round-trip back. Below ~500 ms the busiest cameras outrun it and their crops silently fall back to the downscaled detection frame; every extra second multiplies resident RAM by roughly (frame bytes × delivered fps). 1200 ms is the shipped value.",
42640
- min: NATIVE_LEASE_TTL_FIELD.min,
42641
- max: NATIVE_LEASE_TTL_FIELD.max,
42642
- step: NATIVE_LEASE_TTL_FIELD.step,
42643
- default: NATIVE_LEASE_TTL_FIELD.default,
42968
+ label: "Frames held at once",
42969
+ description: "How many delivered frames a worker keeps alive while waiting for their detection results. A frame is freed as soon as its own result comes back and its subject tiles have been cut, so the steady state is inference latency × delivered fps 1 to 4 frames in practice. This number is only the bound above which the OLDEST held frame is dropped, which is what stops a runner that has stopped answering from pinning RAM. Raising it does not improve crop hit rate; it buys tolerance for a slow runner, and holdOverflow on the metrics line is what tells you that you need it.",
42970
+ min: NATIVE_LEASE_HOLD_FIELD.min,
42971
+ max: NATIVE_LEASE_HOLD_FIELD.max,
42972
+ step: NATIVE_LEASE_HOLD_FIELD.step,
42973
+ default: NATIVE_LEASE_HOLD_FIELD.default,
42644
42974
  showValue: true,
42645
- unit: "ms"
42975
+ unit: "frames"
42976
+ },
42977
+ {
42978
+ key: NATIVE_LEASE_TILE_BUDGET_KEY,
42979
+ type: "slider",
42980
+ label: "Subject tile RAM",
42981
+ description: "RAM per decode worker for the compressed SUBJECT TILES — the native-resolution crops taken at the moment a frame's detections are known, and kept long after the frame itself is gone. This is what serves a crop that arrives seconds late, which measurement says is the ordinary case (2 to 10 seconds on this cluster). At ~60-120 KB a tile, 64 MB is many hundreds of subjects. 0 turns tiles OFF and restores the old behaviour, where a late crop had nothing to fall back to but the ≤640 detection frame.",
42982
+ min: NATIVE_LEASE_TILE_BUDGET_FIELD.min,
42983
+ max: NATIVE_LEASE_TILE_BUDGET_FIELD.max,
42984
+ step: NATIVE_LEASE_TILE_BUDGET_FIELD.step,
42985
+ default: NATIVE_LEASE_TILE_BUDGET_FIELD.default,
42986
+ showValue: true,
42987
+ unit: "MB"
42646
42988
  },
42647
42989
  {
42648
42990
  key: NATIVE_LEASE_BUDGET_KEY,
42649
42991
  type: "slider",
42650
42992
  label: "Lease RAM ceiling",
42651
- description: "Hard RAM ceiling for retained frames, PER decode worker (one worker per camera per plane). This is a safety ceiling, not the working size — the TTL above is what normally reclaims frames, and at 1024 MB the ceiling is never the binding constraint on a single camera. Lower it on a small host to bound the worst case. 0 DISABLES the lease entirely and falls the worker back to the tiny GPU surface ring, which misses roughly 85% of late crops — that is the behaviour the lease exists to replace, so 0 is a diagnostic setting, not a tuning one.",
42993
+ description: "Hard RAM ceiling for held frames, PER decode worker (one worker per camera per plane). This is a safety ceiling, not the working size — the hold count above is what reclaims frames now, and the ceiling is the number above which something is wrong. Lower it on a small host to bound the worst case. 0 DISABLES the lease entirely and falls the worker back to the tiny GPU surface ring, which misses roughly 85% of late crops — that is the behaviour the lease exists to replace, so 0 is a diagnostic setting, not a tuning one.",
42652
42994
  min: NATIVE_LEASE_BUDGET_FIELD.min,
42653
42995
  max: NATIVE_LEASE_BUDGET_FIELD.max,
42654
42996
  step: NATIVE_LEASE_BUDGET_FIELD.step,