homebridge-hikvision-dvr 1.2.1 → 1.2.2

package/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 1.2.2
+
+Fixes video freezing and high load on a Raspberry Pi when several cameras are opened at once (for example the 4-camera grid on an Apple TV).
+
+- With *Video processing* on Automatic, the plugin now chooses how to serve each stream by how many are being viewed at the same time. One viewer gets the main stream at up to 1280x720; two get the main stream at up to 960x540; three or more get the DVR's sub stream scaled to 640x360 at 800 kbps (the Pi hardware encoder runs four of these at 25 fps; a fifth is passed through unchanged). The DVR can only deliver about 5.5 Mbit/s in total, so several main streams at once gave about 13 fps anyway.
+- The choice is made for each stream when it starts and the log says which path was used and why. When the grid is closed, the next single view is back on the best path.
+- A failing hardware encode no longer switches the whole plugin to libx264 until restart. Only that stream falls back (to libx264 at up to 640 px wide, or pass-through if two are already running). Only if nothing else is encoding in hardware is the encoder treated as wedged, and then it is avoided for 10 minutes and tried again, not for good.
+- Waits up to 8 s (was 6 s) for the hardware encoder's first output before falling back.
+- Live view starts much faster on a Pi: the first picture now arrives in about 0.5 s for a single view and 0.7-0.9 s per tile in the grid (was 2.5 s and 2.6-5 s). The hardware path was throwing away its first keyframe (about 2 s) and probing its input for longer than needed. When the DVR joins a sub stream mid-picture-group (which used to delay a tile by 2-3 s), the plugin now asks it for a keyframe.
+- Hardware *decoding* on the Pi was tested and is not used: it started slower, used more CPU and dropped frames.
+
 ## 1.2.1
 
 - The plugin's settings page in the Homebridge UI shows an **Open web console** button and the console's address, instead of trying to embed the console (the DVR's pages don't work well inside the Homebridge UI, which also blocks their dialogs).

package/README.md

@@ -31,7 +31,7 @@ Only the IP address, username and password are needed. Everything else is option
 | Motion sensors | on | A motion sensor per camera, from the DVR's own motion detection (see below) |
 | Motion reset | 30 s | How long motion stays on after the DVR's last motion event |
 | Verification code | read from DVR | Only needed if the plugin can't read it. Use the 6-letter code on the DVR's sticker, or the one set in Hik-Connect / HiLook |
-| Video processing | Automatic | Uses the Raspberry Pi hardware encoder when available (for views wider than 640 px; smaller views use libx264, which starts faster), otherwise libx264. *No transcoding* passes the DVR's stream through untouched (lowest CPU, experimental) |
+| Video processing | Automatic | Uses the Raspberry Pi hardware encoder when available (for views wider than 640 px; smaller views use libx264, which starts faster), otherwise libx264. With several viewers at once (e.g. an Apple TV camera grid) it automatically uses smaller output and the DVR's sub stream so a Pi keeps up (see below). *No transcoding* passes the DVR's stream through untouched (lowest CPU, experimental) |
 | Use sub stream up to width | 0 (never) | e.g. `320` serves Apple Watch views from the low-res sub stream |
 | Minimum bitrate | automatic | iOS often asks for only ~300 kbps; the plugin uses at least 1200 kbps for 720p+. `0` = exactly what iOS asks |
 | Max viewers per camera | 2 | |
@@ -79,6 +79,21 @@ which keeps the load on the DVR and the Homebridge machine down. Limitations:
 - It is plain HTTP, like the DVR itself, so **don't port-forward it** or expose it to the internet.
 - Tested only on a HiLook DVR-104G-F1.
 
+## Several cameras at once
+
+An Apple TV camera grid starts four streams at the same moment. On a Raspberry Pi (Automatic video processing) the plugin
+picks a path for every stream as it starts, depending on how many are being viewed, and logs the choice:
+
+| Viewers | Source | Output |
+|---|---|---|
+| 1 | main stream | up to 1280x720 (hardware encoder; libx264 if the view is 640 px or less) |
+| 2 | main stream | up to 960x540, hardware encoder |
+| 3 or more | sub stream | 640x360, 800 kbps, hardware encoder; a fifth and later stream is passed through unchanged |
+
+If the hardware encoder fails for one stream, only that stream falls back to libx264. If it is wedged (nothing is encoding in hardware),
+it is avoided for 10 minutes and then tried again. The DVR itself delivers only a few Mbit/s in total, so several main streams at once
+would be slow even with a faster computer. Hardware video *decoding* on the Pi was tried and made things slower.
+
 ## How it works
 
 - Discovers cameras via ISAPI: analog DVR inputs, NVR IP channels, or a single IP camera.

package/lib/camera.js

@@ -11,7 +11,26 @@ const START_CODE = Buffer.from([0, 0, 0, 1]);
 const PROFILES = ['baseline', 'main', 'high'];
 const LEVELS = ['3.1', '3.2', '4.0'];
 const SOFTWARE_MAX_WIDTH = 640; // auto mode: views up to this width use libx264 even on a Pi
-const HW_TIMEOUT = 6000; // ms to wait for the hardware encoder's first output
+const HW_TIMEOUT = 8000; // ms to wait for the hardware encoder's first output
+const HW_RETRY = 10 * 60 * 1000; // after a real hardware-encoder failure, use software only this long
+const PREPARE_TTL = 10000; // a prepared-but-not-started session counts towards "how many viewers are coming" this long
+// Concurrency tiers for auto mode on a Pi (measured on a Pi 3 A+, one stream = ffmpeg decode of the DVR's 960x1080 main stream,
+// scale, hardware encode: 1280x720 ~100% of a core, 960x540 ~87%; sub stream to 640x360 ~25%; the DVR itself delivers
+// only ~5.5 Mbit/s in total, so 3+ main streams drop to ~13 fps; the Pi codec fails to open an encoder for the 4th 720p session).
+const MAIN_MAX_VIEWERS = 2; // up to this many concurrent viewers get the main stream
+const SMALL = { width: 640, height: 360, bitrate: 800 }; // light path: sub stream scaled up, hardware encoded
+const MAX_HW_SESSIONS = 4; // hardware encodes at once; further viewers get the sub stream passed through
+const MAX_SW_SESSIONS = 2; // libx264 encodes at once while the hardware encoder is marked broken or one hardware encode failed
+// Hardware DECODE (-c:v h264_v4l2m2m before -i) was measured on the Pi 3 A+ and is deliberately not used: with the DVR's FLV pipe
+// input it was slower to start (2-7 s vs 1.5 s), used 2.5-3.3 cores instead of ~1 (ffmpeg copies every frame out of the codec's
+// buffers) and delivered only a fraction of the frames when combined with the hardware encoder, while software decoding of the
+// 960x1080 main stream only costs about half a core.
+
+/** Per-platform streaming state shared by all cameras (kept on the platform object). */
+function streamState(platform) {
+  if (!platform.streamState) platform.streamState = { active: new Set(), prepared: new Map(), hwBrokenUntil: 0 };
+  return platform.streamState;
+}
 
 /**
  * HomeKit camera streaming delegate for one DVR channel.
@@ -123,6 +142,7 @@ class HikCamera {
       });
       socket.on('error', (e) => this.log.debug(`[${this.name}] RTCP socket error: ${e.message}`));
       const ssrc = crypto.randomBytes(4).readUInt32BE(0) & 0x7fffffff || 1;
+      streamState(this.platform).prepared.set(request.sessionID, Date.now());
       this.pending.set(request.sessionID, {
         address: request.targetAddress,
         port: request.video.port,
@@ -160,35 +180,98 @@ class HikCamera {
     }
   }
 
+  /**
+   * Decides how to serve one viewer: which DVR stream, which encoder and what output size.
+   * On a Pi (auto mode) this depends on how many viewers there are in total, since 4 camera tiles on an Apple TV
+   * all start at once and the Pi cannot transcode four main streams.
+   */
+  plan(sessionID, video) {
+    const p = this.platform;
+    const st = streamState(p);
+    st.prepared.delete(sessionID);
+    const now = Date.now();
+    for (const [id, t] of st.prepared) if (now - t > PREPARE_TTL) st.prepared.delete(id);
+    const viewers = st.active.size + st.prepared.size + 1;
+    const userSub = p.subStreamMaxWidth > 0 && video.width <= p.subStreamMaxWidth;
+    let sub = userSub;
+    let mode = p.encoder;
+    let cap = null; // [maxWidth, maxHeight, maxBitrate] of the output
+    let why = '';
+    const count = (m) => [...st.active].filter((x) => x.mode === m).length;
+    if (mode === 'h264_v4l2m2m' && p.encoderAuto) {
+      const hwOk = now >= st.hwBrokenUntil;
+      if (!hwOk) {
+        sub = true;
+        mode = count('libx264') < MAX_SW_SESSIONS ? 'libx264' : 'copy';
+        cap = [SMALL.width, SMALL.height, SMALL.bitrate];
+        why = 'hardware encoder marked broken, sub stream';
+      } else if (viewers <= 1) {
+        if (video.width <= SOFTWARE_MAX_WIDTH) mode = 'libx264';
+        cap = [1280, 720, Infinity];
+        why = 'single viewer, main stream';
+      } else if (viewers <= MAIN_MAX_VIEWERS) {
+        cap = [960, 540, Infinity];
+        why = `${viewers} viewers, main stream at reduced size`;
+      } else {
+        sub = true;
+        cap = [SMALL.width, SMALL.height, SMALL.bitrate];
+        why = `${viewers} viewers, light sub-stream path`;
+        if (count('h264_v4l2m2m') >= MAX_HW_SESSIONS) {
+          mode = 'copy';
+          why = `${viewers} viewers, sub stream passed through`;
+        }
+      }
+    } else if (mode === 'libx264' && p.encoderAuto && viewers > MAIN_MAX_VIEWERS) {
+      // No hardware encoder at all (e.g. a Mac or a PC): keep CPU use bounded the same way.
+      sub = true;
+      cap = [SMALL.width, SMALL.height, SMALL.bitrate];
+      why = `${viewers} viewers, light sub-stream path`;
+    }
+    if (!why) why = 'configured';
+    let width = video.width;
+    let height = video.height;
+    if (cap && width > cap[0]) {
+      width = cap[0];
+      height = Math.round((video.height * width) / video.width / 2) * 2;
+    }
+    return { sub, mode, width, height, maxBitrate: cap ? cap[2] : Infinity, why, viewers };
+  }
+
   startStream(sessionID, info, video, callback) {
     const p = this.platform;
-    const useSub = p.subStreamMaxWidth > 0 && video.width <= p.subStreamMaxWidth;
+    const st = streamState(p);
+    const plan = this.plan(sessionID, video);
+    const useSub = plan.sub;
     const stream = useSub ? 2 : 1;
-    // 'h264_v4l2m2m' | 'libx264' | 'copy'. Small views are cheap in software and start faster, and the Pi's hardware
-    // encoder can wedge (no output, ever) after a kernel driver error, so only auto-pick it for larger views.
-    let mode = p.encoder;
-    if (mode === 'h264_v4l2m2m' && (p.hwBroken || (p.encoderAuto && video.width <= SOFTWARE_MAX_WIDTH))) mode = 'libx264';
+    // 'h264_v4l2m2m' | 'libx264' | 'copy'. A single small view is cheap in software and starts faster, and the Pi's
+    // hardware encoder can wedge (no output, ever) after a kernel driver error, so it is only auto-picked for larger views.
+    const mode = plan.mode;
+    let outW = plan.width;
+    let outH = plan.height;
     const fps = Math.min(video.fps, 25);
     // iOS often asks for ~300 kbps even on Wi-Fi, which looks poor; apply a resolution-based floor.
-    const floor = p.minBitrate !== undefined ? p.minBitrate : video.width >= 1280 ? 1200 : video.width >= 640 ? 600 : 300;
-    const bitrate = Math.max(video.max_bit_rate, floor, 100);
+    const floor = p.minBitrate !== undefined ? p.minBitrate : outW >= 1280 ? 1200 : outW >= 640 ? 600 : 300;
+    const bitrate = Math.min(Math.max(video.max_bit_rate, floor, 100), Math.max(plan.maxBitrate, floor));
     const log = this.log;
     const tag = `[${this.name}]`;
 
-    log.info(`${tag} Starting ${video.width}x${video.height}@${fps} ${bitrate}kbps from ${useSub ? 'sub' : 'main'} stream (${mode})`);
+    log.info(`${tag} Starting ${outW}x${outH}@${fps} ${bitrate}kbps from ${useSub ? 'sub' : 'main'} stream (${mode}): ${plan.why}${outW !== video.width ? `; viewer asked for ${video.width}x${video.height}` : ''}`);
 
-    const session = { procs: [], rtsp: null, socket: info.socket, timer: null, stopped: false, hwTimer: null, hwFallback: null, hwFirst: null, armHw: null };
+    const session = { mode, procs: [], rtsp: null, socket: info.socket, timer: null, stopped: false, hwTimer: null, hwFallback: null, hwFirst: null, armHw: null };
     this.sessions.set(sessionID, session);
+    st.active.add(session);
 
     const srtpOut = ['-payload_type', String(video.pt), '-ssrc', String(info.ssrc), '-f', 'rtp',
       '-srtp_out_suite', 'AES_CM_128_HMAC_SHA1_80', '-srtp_out_params', info.srtp,
       `srtp://${info.address}:${info.port}?rtcpport=${info.port}&pkt_size=${video.mtu}`];
     const logLevel = ['-hide_banner', '-loglevel', p.debug ? 'warning' : 'error'];
     // Decrypted camera video arrives as FLV carrying the DVR's own timestamps.
+    // probesize: the FLV already carries SPS/PPS, so only the first frame is needed; a larger value makes ffmpeg wait for that many
+    // bytes of (low bitrate) sub stream to arrive in real time, ~0.4 s.
     // showall: the DVR's keyframes carry no recovery point, so without it the decoder drops ~1 s of frames before the first output.
     const input = [...logLevel, '-fflags', '+nobuffer+discardcorrupt', '-flags', 'low_delay', '-flags2', '+showall',
-      '-probesize', '32768', '-analyzeduration', '0', '-f', 'flv', '-i', 'pipe:0', '-an', '-sn', '-dn'];
-    const scale = `scale=${video.width}:${video.height}:flags=fast_bilinear,format=yuv420p`;
+      '-probesize', '4096', '-analyzeduration', '0', '-f', 'flv', '-i', 'pipe:0', '-an', '-sn', '-dn'];
+    let scale = `scale=${outW}:${outH}:flags=fast_bilinear,format=yuv420p`;
 
     let first; // the ffmpeg process fed by the RTSP stream
     let flv;
@@ -206,8 +289,12 @@ class HikCamera {
         // Pi hardware encoder doesn't repeat SPS/PPS before keyframes, which HomeKit needs.
         // Encode to a pipe, re-insert SPS/PPS in Node, and let a second (copy-only) ffmpeg do SRTP.
         // The encoder output is constant frame rate, so the sender can number frames itself.
-        const sender = this.spawn(session, [...logLevel, '-fflags', '+nobuffer', '-probesize', '32768', '-analyzeduration', '0',
-          '-f', 'h264', '-framerate', String(fps), '-i', 'pipe:0', '-c:v', 'copy', ...srtpOut]);
+        // -copyinkf: ffmpeg's raw h264 demuxer does not flag the encoder's first IDR as a keyframe, so a plain stream copy discards
+        // everything up to the second IDR (~2 s); the stream always starts with an IDR here, so keep the first packet.
+        // Probing must stay minimal (one packet, 32 bytes): with larger values ffmpeg decodes more of the input first, and under
+        // startup load (several tiles at once) that sometimes took 0.4-1 s. SPS/PPS are in the stream and rtp output needs nothing else.
+        const sender = this.spawn(session, [...logLevel, '-fflags', '+nobuffer', '-probesize', '32', '-analyzeduration', '0', '-max_probe_packets', '1',
+          '-f', 'h264', '-framerate', String(fps), '-i', 'pipe:0', '-c:v', 'copy', '-copyinkf', ...srtpOut]);
         first = this.spawn(session, [...input, '-vf', scale, '-r', String(fps), '-c:v', 'h264_v4l2m2m', '-g', String(fps * 2),
           '-b:v', `${bitrate}k`, '-f', 'h264', 'pipe:1']);
         let sps = null;
@@ -227,21 +314,42 @@ class HikCamera {
           splitter.push(chunk);
           clearTimeout(session.hwTimer);
           session.hwFallback = null;
+          session.hwOutput = true;
         });
-        // No encoded output at all means the hardware encoder is wedged (or failed to open): switch this session to libx264 (and later ones).
+        // No encoded output at all: the hardware encoder failed to open, is overloaded or (rarely) wedged. Switch this session to
+        // libx264 at a small size. Only when no other hardware encode is working is the driver considered broken for everyone,
+        // and then only for HW_RETRY.
         session.hwFirst = first;
         session.hwFallback = (why) => {
           session.hwFallback = null;
           clearTimeout(session.hwTimer);
           if (session.stopped) return;
-          p.hwBroken = true;
-          log.warn(`${tag} Hardware encoder ${why} (the Pi's video driver may need a reboot), using libx264`);
+          const othersWorking = [...st.active].some((x) => x !== session && x.mode === 'h264_v4l2m2m' && x.hwOutput);
+          // Only a wedge (nothing else is encoding in hardware) disables the hardware encoder for everyone, and only for HW_RETRY.
+          // An ffmpeg that merely exited may have failed for other reasons (DVR stream gone), so it needs a V4L2 error to count.
+          const wedge = !othersWorking && (why.startsWith('produced') || /v4l2|video\d+|encoder|codec/i.test(session.lastErr || ''));
+          if (wedge) st.hwBrokenUntil = Date.now() + HW_RETRY;
+          const light = [...st.active].filter((x) => x !== session && x.mode === 'libx264').length < MAX_SW_SESSIONS;
+          log.warn(`${tag} Hardware encoder ${why}; this stream switches to ${light ? 'libx264' : 'pass-through'}${wedge ? ` and the hardware encoder is not used for ${HW_RETRY / 60000} min (the Pi's video driver may need a reboot)` : ' (other streams keep using the hardware encoder)'}`);
           for (const proc of session.procs.splice(0)) {
             proc.replaced = true;
             proc.kill('SIGTERM');
             setTimeout(() => proc.exitCode === null && proc.kill('SIGKILL'), 5000).unref();
           }
           started = false;
+          keyRequested = false;
+          requestKeyFrame();
+          if (!light) {
+            session.mode = 'copy';
+            launch('copy');
+            return;
+          }
+          session.mode = 'libx264';
+          if (outW > SOFTWARE_MAX_WIDTH) { // libx264 at 720p takes 2+ cores of a Pi
+            outH = Math.round((outH * SOFTWARE_MAX_WIDTH) / outW / 2) * 2;
+            outW = SOFTWARE_MAX_WIDTH;
+            scale = `scale=${outW}:${outH}:flags=fast_bilinear,format=yuv420p`;
+          }
           launch('libx264');
         };
         // The clock starts when the first keyframe is fed to the encoder, not while RTSP is still connecting.
@@ -261,8 +369,17 @@ class HikCamera {
     };
 
     let started = false;
+    let keyRequested = false;
     launch(mode);
 
+    // The DVR usually sends a keyframe right after PLAY, but sometimes (mostly sub streams, soon after another session on the
+    // same stream) it joins mid-GOP and the next keyframe is 2+ s away. Ask for one (ISAPI, harmless if unsupported), once.
+    const requestKeyFrame = () => {
+      if (keyRequested || !p.isapi || !p.isapi.requestKeyFrame) return;
+      keyRequested = true;
+      p.isapi.requestKeyFrame(this.channel, stream).then(() => log.debug(`${tag} Requested a keyframe`), (e) => log.debug(`${tag} Keyframe request failed: ${e.message}`));
+    };
+
     // RTSP -> decrypt -> ffmpeg
     const decryptor = new HikDecryptor(p.verificationCode);
     const rtsp = new RtspClient(this.rtspUrl(stream), { username: p.username, password: p.password });
@@ -280,7 +397,10 @@ class HikCamera {
       const plain = decryptor.process(nal, flagged);
       const type = plain[0] & 0x1f;
       if (!started) {
-        if (type !== 7 && type !== 8 && type !== 5) return; // start on a keyframe
+        if (type !== 7 && type !== 8 && type !== 5) { // start on a keyframe
+          requestKeyFrame();
+          return;
+        }
         started = true;
       }
       flv.push(plain, ts, marker);
@@ -293,6 +413,7 @@ class HikCamera {
 
     rtsp.start().then(() => {
       log.debug(`${tag} RTSP playing ${this.rtspUrl(stream)}`);
+      session.kfTimer = setTimeout(() => { if (!started) requestKeyFrame(); }, 400);
       callback();
     }).catch((e) => {
       log.error(`${tag} Could not open DVR stream: ${e.message}`);
@@ -320,6 +441,7 @@ class HikCamera {
     proc.stdin.on('error', () => { /* EPIPE when ffmpeg exits */ });
     proc.stderr.on('data', (d) => {
       const text = d.toString().trim();
+      if (text) session.lastErr = text.slice(-500);
       if (text) this.log[this.platform.debug ? 'info' : 'debug'](`${tag} ffmpeg: ${text}`);
     });
     proc.on('error', (e) => this.log.error(`${tag} Failed to start ffmpeg (${this.platform.ffmpeg}): ${e.message}`));
@@ -346,6 +468,7 @@ class HikCamera {
   }
 
   stopStream(sessionID) {
+    streamState(this.platform).prepared.delete(sessionID);
     const pending = this.pending.get(sessionID);
     if (pending) {
       this.pending.delete(sessionID);
@@ -354,8 +477,10 @@ class HikCamera {
     const session = this.sessions.get(sessionID);
     if (!session) return;
     this.sessions.delete(sessionID);
+    streamState(this.platform).active.delete(session);
     session.stopped = true;
     clearTimeout(session.timer);
+    clearTimeout(session.kfTimer);
     if (session.rtsp) session.rtsp.stop();
     for (const proc of session.procs) {
       try { proc.stdin.end(); } catch (e) { /* ignore */ }

package/lib/isapi.js

@@ -158,6 +158,12 @@ class Isapi {
     }
   }
 
+  /** Asks the DVR to send an IDR frame on a stream (Hikvision ISAPI; harmless on devices that lack it). */
+  async requestKeyFrame(channel, stream = 1) {
+    const res = await this.request('PUT', `/ISAPI/Streaming/channels/${channel * 100 + stream}/requestKeyFrame`);
+    if (res.status !== 200) throw new Error(`HTTP ${res.status}`);
+  }
+
   async snapshot(channel, stream = 1) {
     return this.get(`/ISAPI/Streaming/channels/${channel * 100 + stream}/picture`);
   }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "homebridge-hikvision-dvr",
   "displayName": "Hikvision DVR / NVR Cameras",
-  "version": "1.2.1",
+  "version": "1.2.2",
   "description": "Hikvision, HiLook and other Hik-based DVR/NVR cameras in HomeKit, with automatic camera discovery and support for encrypted (Hik-Connect / HiLook) streams.",
   "main": "index.js",
   "license": "MIT",