@linanw/homebridge-mijia-remote-camera 0.1.1 → 0.1.2

package/README.md

@@ -2,7 +2,7 @@
 
 插件：`@linanw/homebridge-mijia-remote-camera`，平台：`MijiaRemoteCamera`。
 
-登录、家庭选择、远程摄像头连接、快照和 HomeKit 视频由插件管理，无需独立的 viewer / relay 容器或另一个摄像头插件。摄像头可以位于其他网络。
+登录、家庭选择、远程摄像头连接、快照、HomeKit 视频、声音与双向对讲由插件管理，无需独立的 viewer / relay 容器或另一个摄像头插件。摄像头可以位于其他网络。
 
 目前后端仅实现 CS2/vendor 4。已在一台小米智能摄像机 3 Pro 云台版上验证跨网络远程取流，以及真实 Apple“家庭”的配对、预览和直播。具体测试环境见下文“验证范围”；其他型号、地区和网络环境尚未逐一验证。
 
@@ -54,9 +54,15 @@ mijia-camera-setup --from /homebridge/homebridge-mijia-remote-camera-0.1.0.tgz -
 
 摄像头作为独立配件发布。在 iPhone“家庭 → 添加配件 → 更多选项”中选择它，使用该摄像头启动日志中 `Please add [...] manually in Home app. Setup Code:` 后面的配对码。插件运行在子桥时，这个码可能与 Homebridge 首页的主桥配对码不同。
 
-默认最高 640×360、12 fps、700 kbps，最多一路实时视频。当前提供实时视频和快照，未接入声音、对讲、云台、运动事件、录像回放及 HomeKit Secure Video。
+默认最高 640×360、12 fps、700 kbps，最多一路实时视频。0.1.2 起接入摄像头实际返回的 PCMA 或 Opus 音频，经 AAC-ELD 16 kHz 发送到 Apple“家庭”；双向对讲会将“家庭”回传的声音转换为摄像头协议，并通过同一远程会话发送。没有收到真实音频时不会生成替代声音，视频可独立播放。尚未接入云台、运动事件、录像回放及 HomeKit Secure Video。
 
-为兼容 Apple“家庭”的媒体能力协商，插件默认公布 AAC-ELD 16 kHz 音频能力和静音 Microphone 服务；实际不采集或发送摄像头声音。家庭中可能显示麦克风控件，目前仍只提供视频。
+## 声音、对讲与夜视
+
+默认开启现场声音、双向对讲和夜视开关。升级后重启摄像头子桥即可，保留已有 HomeKit 配对。在“家庭”直播页面使用对讲按钮，并允许 iPhone 访问麦克风；松开或结束对讲后会停止向摄像头发送声音。声音静音和音量控制会作用于实际音频转发。插件设置页显示是否已收到真实摄像头声音，以及当前协议是否支持对讲。
+
+设置页可修改“收听摄像头现场声音”“允许双向对讲”“在家庭中显示夜视开关”；保存后需重启摄像头子桥。对应配置项为 `enableAudio`、`enableTalkback`、`enableNightVision`，默认均为 `true`。关闭现场声音也会禁用对讲。
+
+夜视目前仅支持已核对 MIOT 定义的 `chuangmi.camera.061a01`／`chuangmi.camera.061a03`。插件设置页先读取设备实际状态，再选择“自动／开启／关闭”并应用；写入后会回读确认。Apple“家庭”的“夜视”开关表示允许夜视：自动模式显示为开，重复开不会把自动改为强制开启；关掉后重新打开会设置为始终开启。需要恢复自动时，请在插件设置页选择。该开关表示夜视策略，不表示红外灯此刻是否点亮。启动和轮询只读状态，不更改策略。
 
 ## 数据和运行管理
 
@@ -70,21 +76,21 @@ mijia-camera-setup --from /homebridge/homebridge-mijia-remote-camera-0.1.0.tgz -
   runtime/
 ```
 
-私有目录权限为 `0700`，凭据和选择文件为 `0600`。小米 Token 不写入 Homebridge `config.json`。设置页通过 Homebridge 的受保护接口和本机 Unix socket 通信；原始视频仅监听临时的本机回环端口。Homebridge 启动后恢复所选摄像头，关闭时清理 Java、FFmpeg 和 UDP 会话，临时网络故障会自动重连。
+私有目录权限为 `0700`，凭据和选择文件为 `0600`。小米 Token 不写入 Homebridge `config.json`。设置页通过 Homebridge 的受保护接口和本机 Unix socket 通信；原始视频和音频仅监听临时的本机回环端口，不保存录音。对讲接口也仅允许本机访问。Homebridge 启动后恢复所选摄像头，关闭时清理 Java、FFmpeg 和 UDP 会话，临时网络故障会自动重连。
 
 插件按单个账号、单个摄像头设计，按所选家庭及实际所有者查询完整设备分页。迁移前请备份 Homebridge 存储目录。
 
 ## 验证范围
 
-实际设备验证使用一台小米智能摄像机 3 Pro 云台版，摄像头与 NAS 位于不同网络。Synology DS1618+ 上的 Homebridge Docker 使用 host 网络，Homebridge 2.0.2 / HAP 2.1.6；远程源为 HEVC，由 NAS 解码并转换为 HomeKit 视频。用户已确认 iOS 26.6.2 的 Apple“家庭”预览和直播正常。
+实际设备验证使用一台小米智能摄像机 3 Pro 云台版，摄像头与 NAS 位于不同网络。Synology DS1618+ 上的 Homebridge Docker 使用 host 网络，Homebridge 2.0.2 / HAP 2.1.6；远程源为 HEVC，由 NAS 解码并转换为 HomeKit 视频。用户已确认 iOS 26.6.2 的 Apple“家庭”预览和直播正常。0.1.2 在同一台摄像头上确认收到真实 Opus 48 kHz 双声道音频并成功独立解码；HomeKit 音频发送和手机回传的对讲音频也已在实机日志中确认，摄像头扬声器的可听效果仍需现场确认。夜视已在实机通过读取、关闭、回读确认、恢复原模式的往返验证，操作约需 0.6–2.2 秒。
 
-本地测试另覆盖 Homebridge 1／2 的原生 HAP CameraController、加密快照与 SRTP 发送／独立接收解码、进程清理、授权取消与原子保存、家庭作用域、真实 Homebridge 设置页 IPC 和本地 Java 启动。
+本地测试另覆盖 Homebridge 1／2 的原生 HAP CameraController、加密快照、视频与双向音频的 SRTP 发送／独立接收解码、夜视读取与写后回读、进程清理、授权取消与原子保存、家庭作用域、真实 Homebridge 设置页 IPC 和本地 Java 启动。
 
 远程中继仍可能间歇断开并自动重连，重连期间预览和直播可能暂不可用。目前尚未完成长时间稳定性验证。
 
 ## 连接诊断
 
-设置页可展开“连接诊断”，查看连接阶段、重试次数、上次失败原因及 HomeKit 快照／直播请求计数。“已收到摄像头视频”表示米家视频源已就绪；快照和直播计数则显示 HomeKit 是否已请求媒体。也可在 Homebridge 终端运行：
+设置页可展开“连接诊断”，查看连接阶段、重试次数、上次失败原因及 HomeKit 快照／直播／音频／对讲请求计数。“已收到摄像头视频”表示米家视频源已就绪；快照和直播计数则显示 HomeKit 是否已请求媒体。也可在 Homebridge 终端运行：
 
 ```sh
 mijia-camera-diagnose --storage /var/lib/homebridge

package/backend/AudioCodecs.java

@@ -0,0 +1,179 @@
+package mijia.bridge;
+
+import java.io.ByteArrayOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/** Owned, bounded G.711 A-law and Ogg/Opus framing. It never decodes or stores camera content on disk. */
+final class AudioCodecs {
+    static final int MAX_PACKET = 16384;
+    private AudioCodecs() { }
+
+    static short decodeAlaw(byte value) {
+        int a = (value & 255) ^ 0x55;
+        int t = (a & 15) << 4, segment = (a & 0x70) >>> 4;
+        if (segment == 0) t += 8;
+        else { t += 0x108; if (segment > 1) t <<= segment - 1; }
+        return (short) ((a & 0x80) != 0 ? t : -t);
+    }
+
+    static byte encodeAlaw(short value) {
+        int pcm = value, mask;
+        if (pcm >= 0) mask = 0xD5;
+        else { mask = 0x55; pcm = -pcm - 1; }
+        pcm = Math.min(pcm, 32767);
+        int segment = 0;
+        for (int end = 0xFF; segment < 8 && pcm > end; segment++, end = (end << 1) | 1) { }
+        if (segment >= 8) return (byte) (0x7F ^ mask);
+        int a = (segment << 4) | ((pcm >>> (segment < 2 ? 4 : segment + 3)) & 15);
+        return (byte) (a ^ mask);
+    }
+
+    static byte[] pcmFromAlaw(byte[] bytes) {
+        if (bytes.length > MAX_PACKET) throw new IllegalArgumentException("audio_packet_size");
+        ByteBuffer out = ByteBuffer.allocate(bytes.length * 2).order(ByteOrder.LITTLE_ENDIAN);
+        for (byte value : bytes) out.putShort(decodeAlaw(value));
+        return out.array();
+    }
+
+    static byte[] wavHeader(int sampleRate) {
+        if (sampleRate != 8000 && sampleRate != 16000) throw new IllegalArgumentException("audio_rate");
+        ByteBuffer out = ByteBuffer.allocate(44).order(ByteOrder.LITTLE_ENDIAN);
+        out.put("RIFF".getBytes(StandardCharsets.US_ASCII)).putInt(-1);
+        out.put("WAVEfmt ".getBytes(StandardCharsets.US_ASCII)).putInt(16).putShort((short) 1).putShort((short) 1);
+        out.putInt(sampleRate).putInt(sampleRate * 2).putShort((short) 2).putShort((short) 16);
+        out.put("data".getBytes(StandardCharsets.US_ASCII)).putInt(-1);
+        return out.array();
+    }
+
+    /** RFC 6716 section 3: samples represented by an Opus packet at the 48 kHz clock. */
+    static int opusSamples(byte[] packet) {
+        if (packet.length == 0 || packet.length > MAX_PACKET) throw new IllegalArgumentException("opus_packet_size");
+        int toc = packet[0] & 255, config = toc >>> 3, frame;
+        if (config >= 16) frame = 120 << (config & 3);
+        else if (config >= 12) frame = 480 << (config & 1);
+        else frame = switch (config & 3) { case 0 -> 480; case 1 -> 960; case 2 -> 1920; default -> 2880; };
+        int count = switch (toc & 3) {
+            case 0 -> 1;
+            case 1, 2 -> 2;
+            default -> packet.length >= 2 ? packet[1] & 63 : 0;
+        };
+        int samples = frame * count;
+        if (count == 0 || samples > 5760) throw new IllegalArgumentException("opus_duration");
+        return samples;
+    }
+
+    static int oggCrc(byte[] bytes) {
+        int crc = 0;
+        for (byte value : bytes) {
+            crc ^= (value & 255) << 24;
+            for (int bit = 0; bit < 8; bit++) crc = (crc & 0x80000000) != 0 ? (crc << 1) ^ 0x04C11DB7 : crc << 1;
+        }
+        return crc;
+    }
+
+    static final class OggWriter {
+        private final int serial;
+        private int sequence;
+        private long samples;
+        OggWriter(int serial) { this.serial = serial; }
+        byte[] headers() {
+            ByteBuffer head = ByteBuffer.allocate(19).order(ByteOrder.LITTLE_ENDIAN);
+            head.put("OpusHead".getBytes(StandardCharsets.US_ASCII)).put((byte) 1).put((byte) 2);
+            head.putShort((short) 0).putInt(48000).putShort((short) 0).put((byte) 0);
+            byte[] vendor = "mijia-homebridge".getBytes(StandardCharsets.US_ASCII);
+            ByteBuffer tags = ByteBuffer.allocate(16 + vendor.length).order(ByteOrder.LITTLE_ENDIAN);
+            tags.put("OpusTags".getBytes(StandardCharsets.US_ASCII)).putInt(vendor.length).put(vendor).putInt(0);
+            byte[] first = page(head.array(), 2, 0), second = page(tags.array(), 0, 0);
+            byte[] result = Arrays.copyOf(first, first.length + second.length);
+            System.arraycopy(second, 0, result, first.length, second.length);
+            return result;
+        }
+        byte[] packet(byte[] bytes) {
+            samples += opusSamples(bytes);
+            return page(bytes, 0, samples);
+        }
+        private byte[] page(byte[] packet, int flags, long granule) {
+            if (packet.length > MAX_PACKET) throw new IllegalArgumentException("ogg_packet_size");
+            int segments = packet.length / 255 + 1;
+            ByteBuffer page = ByteBuffer.allocate(27 + segments + packet.length).order(ByteOrder.LITTLE_ENDIAN);
+            page.put("OggS".getBytes(StandardCharsets.US_ASCII)).put((byte) 0).put((byte) flags).putLong(granule);
+            page.putInt(serial).putInt(sequence++).putInt(0).put((byte) segments);
+            for (int left = packet.length, i = 0; i < segments; i++) { int n = Math.min(255, left); page.put((byte) n); left -= n; }
+            page.put(packet);
+            byte[] result = page.array();
+            ByteBuffer.wrap(result).order(ByteOrder.LITTLE_ENDIAN).putInt(22, oggCrc(result));
+            return result;
+        }
+    }
+
+    /** A bounded Ogg demuxer for the local FFmpeg talkback POST, not arbitrary remote media. */
+    static final class OggReader {
+        private final InputStream input;
+        private final ByteArrayOutputStream partial = new ByteArrayOutputStream();
+        private final List<byte[]> ready = new ArrayList<>();
+        private Integer serial;
+        private int sequence, headers;
+        OggReader(InputStream input) { this.input = input; }
+        byte[] next() throws IOException {
+            for (;;) {
+                if (!ready.isEmpty()) return ready.remove(0);
+                byte[] header = input.readNBytes(27);
+                if (header.length == 0) {
+                    if (partial.size() != 0) throw new EOFException("ogg_truncated");
+                    return null;
+                }
+                if (header.length != 27 || !Arrays.equals(Arrays.copyOf(header, 4), "OggS".getBytes(StandardCharsets.US_ASCII)) || header[4] != 0)
+                    throw new IOException("ogg_header");
+                int flags = header[5] & 255;
+                if ((flags & ~7) != 0 || ((flags & 1) != 0) != (partial.size() != 0)) throw new IOException("ogg_continuation");
+                ByteBuffer fields = ByteBuffer.wrap(header).order(ByteOrder.LITTLE_ENDIAN);
+                int currentSerial = fields.getInt(14), currentSequence = fields.getInt(18), crc = fields.getInt(22);
+                if (serial == null) { if (currentSequence != 0 || (flags & 2) == 0) throw new IOException("ogg_begin"); serial = currentSerial; }
+                if (serial != currentSerial || currentSequence != sequence++) throw new IOException("ogg_sequence");
+                int count = header[26] & 255;
+                byte[] lacing = input.readNBytes(count);
+                if (lacing.length != count) throw new EOFException("ogg_truncated");
+                int size = 0; for (byte value : lacing) size += value & 255;
+                if (size > 65536) throw new IOException("ogg_page_size");
+                byte[] body = input.readNBytes(size);
+                if (body.length != size) throw new EOFException("ogg_truncated");
+                byte[] page = new byte[27 + count + size];
+                System.arraycopy(header, 0, page, 0, 27); Arrays.fill(page, 22, 26, (byte) 0);
+                System.arraycopy(lacing, 0, page, 27, count); System.arraycopy(body, 0, page, 27 + count, size);
+                if (oggCrc(page) != crc) throw new IOException("ogg_crc");
+                int offset = 0;
+                for (byte value : lacing) {
+                    int length = value & 255;
+                    if (partial.size() + length > MAX_PACKET) throw new IOException("ogg_packet_size");
+                    partial.write(body, offset, length); offset += length;
+                    if (length < 255) {
+                        byte[] packet = partial.toByteArray(); partial.reset();
+                        if (headers == 0) {
+                            if (packet.length < 19 || !starts(packet, "OpusHead") || packet[8] != 1 || packet[9] != 2 || packet[18] != 0)
+                                throw new IOException("opus_header");
+                            headers++;
+                        } else if (headers == 1) {
+                            if (!starts(packet, "OpusTags")) throw new IOException("opus_tags");
+                            headers++;
+                        } else {
+                            try { opusSamples(packet); } catch (IllegalArgumentException invalid) { throw new IOException("opus_packet"); }
+                            ready.add(packet);
+                        }
+                    }
+                }
+            }
+        }
+        private static boolean starts(byte[] packet, String value) {
+            byte[] prefix = value.getBytes(StandardCharsets.US_ASCII);
+            return packet.length >= prefix.length && Arrays.equals(Arrays.copyOf(packet, prefix.length), prefix);
+        }
+    }
+}

package/backend/AudioStreamServer.java

@@ -0,0 +1,182 @@
+package mijia.bridge;
+
+import com.sun.net.httpserver.HttpExchange;
+import com.sun.net.httpserver.HttpServer;
+import micam.miss.MediaPacket;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.security.SecureRandom;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CopyOnWriteArraySet;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/** Loopback-only live audio and one bounded talkback POST. Camera readout never waits for a listener. */
+final class AudioStreamServer implements Closeable {
+    record Format(String codec, int sampleRate, int channels) {
+        static Format of(MediaPacket packet) {
+            if (packet.payload().length == 0 || packet.payload().length > AudioCodecs.MAX_PACKET) return null;
+            if (packet.codecId() == MediaPacket.CODEC_PCMA)
+                return new Format("PCMA", ((packet.flags() >>> 3) & 15) == 0 ? 8000 : 16000, 1);
+            if (packet.codecId() == MediaPacket.CODEC_OPUS) {
+                try { AudioCodecs.opusSamples(packet.payload()); } catch (IllegalArgumentException invalid) { return null; }
+                return new Format("OPUS", 48000, 2);
+            }
+            return null;
+        }
+        String path() { return codec.equals("PCMA") ? "/audio.wav" : "/audio.ogg"; }
+        String talkPath() { return codec.equals("PCMA") ? "/talk.pcm" : "/talk.ogg"; }
+        String talkFormat() { return codec.equals("PCMA") ? "s16le" : "ogg"; }
+        int talkSampleRate() { return codec.equals("PCMA") ? 8000 : 48000; }
+        int talkChannels() { return codec.equals("PCMA") ? 1 : 2; }
+    }
+
+    private final HttpServer server;
+    private final ThreadPoolExecutor executor;
+    private final ScheduledExecutorService timer;
+    private final Format format;
+    private final Talkback talkback;
+    private final Set<Viewer> viewers = new CopyOnWriteArraySet<>();
+    private final Semaphore viewerSlots = new Semaphore(4);
+    private final AtomicBoolean talking = new AtomicBoolean();
+    private final AtomicReference<HttpExchange> talkExchange = new AtomicReference<>();
+    private volatile long lastTalkActivity, talkStarted;
+    private volatile boolean closed;
+
+    AudioStreamServer(Format format, Talkback talkback) throws IOException {
+        this.format = format; this.talkback = talkback;
+        server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 8);
+        executor = new ThreadPoolExecutor(6, 6, 30, TimeUnit.SECONDS, new ArrayBlockingQueue<>(16), runnable -> {
+            Thread thread = new Thread(runnable, "mijia-audio-http"); thread.setDaemon(true); return thread;
+        }, new ThreadPoolExecutor.AbortPolicy());
+        server.setExecutor(executor);
+        server.createContext("/", this::handle);
+        timer = Executors.newSingleThreadScheduledExecutor(runnable -> {
+            Thread thread = new Thread(runnable, "mijia-talk-timeout"); thread.setDaemon(true); return thread;
+        });
+        timer.scheduleAtFixedRate(() -> {
+            HttpExchange exchange = talkExchange.get();
+            long now = System.nanoTime();
+            if (exchange != null && (now - lastTalkActivity > TimeUnit.SECONDS.toNanos(5)
+                    || now - talkStarted > TimeUnit.MINUTES.toNanos(2))) exchange.close();
+        }, 1, 1, TimeUnit.SECONDS);
+    }
+
+    int port() { return server.getAddress().getPort(); }
+    Format format() { return format; }
+    void start() { server.start(); }
+
+    void publish(MediaPacket packet) {
+        if (closed || !format.equals(Format.of(packet))) return;
+        byte[] payload = packet.payload();
+        for (Viewer viewer : viewers) viewer.offer(payload);
+    }
+
+    private void handle(HttpExchange exchange) throws IOException {
+        String path = exchange.getRequestURI().getRawPath();
+        String host = exchange.getRequestHeaders().getFirst("Host");
+        if (closed || !exchange.getRemoteAddress().getAddress().isLoopbackAddress()
+                || !("127.0.0.1:" + port()).equals(host) || exchange.getRequestURI().getRawQuery() != null
+                || exchange.getRequestHeaders().containsKey("Origin")) { reject(exchange, 403); return; }
+        if (path.equals(format.path()) && exchange.getRequestMethod().equals("GET")) { listen(exchange); return; }
+        if (path.equals(format.talkPath()) && exchange.getRequestMethod().equals("POST") && talkback != null) { talk(exchange); return; }
+        reject(exchange, path.equals(format.path()) || path.equals(format.talkPath()) ? 405 : 404);
+    }
+
+    private void listen(HttpExchange exchange) throws IOException {
+        if (!viewerSlots.tryAcquire()) { reject(exchange, 503); return; }
+        Viewer viewer = new Viewer();
+        viewers.add(viewer);
+        try {
+        exchange.getResponseHeaders().set("Content-Type", format.codec.equals("PCMA") ? "audio/wav" : "audio/ogg");
+        exchange.getResponseHeaders().set("Cache-Control", "no-store");
+        exchange.sendResponseHeaders(200, 0);
+        try (OutputStream output = exchange.getResponseBody()) {
+            AudioCodecs.OggWriter ogg = format.codec.equals("OPUS") ? new AudioCodecs.OggWriter(new SecureRandom().nextInt()) : null;
+            output.write(ogg == null ? AudioCodecs.wavHeader(format.sampleRate) : ogg.headers()); output.flush();
+            long lastPacket = System.nanoTime();
+            while (!closed) {
+                byte[] packet;
+                try { packet = viewer.queue.poll(250, TimeUnit.MILLISECONDS); }
+                catch (InterruptedException interrupted) { Thread.currentThread().interrupt(); return; }
+                if (packet == null) {
+                    if (System.nanoTime() - lastPacket > TimeUnit.SECONDS.toNanos(10)) return;
+                    continue;
+                }
+                lastPacket = System.nanoTime();
+                try {
+                    byte[] encoded = ogg == null ? AudioCodecs.pcmFromAlaw(packet) : ogg.packet(packet);
+                    try { output.write(encoded); output.flush(); }
+                    finally { Arrays.fill(encoded, (byte) 0); }
+                } finally { Arrays.fill(packet, (byte) 0); }
+            }
+        }
+        } catch (IOException ignored) { }
+        finally { viewers.remove(viewer); viewer.clear(); viewerSlots.release(); exchange.close(); }
+    }
+
+    private void talk(HttpExchange exchange) throws IOException {
+        if (!talking.compareAndSet(false, true)) { reject(exchange, 409); return; }
+        talkStarted = lastTalkActivity = System.nanoTime(); talkExchange.set(exchange);
+        boolean started = false;
+        // The response acknowledges the local transport; speaker/codec correctness is verified separately on the real camera.
+        try {
+        exchange.getResponseHeaders().set("Cache-Control", "no-store"); exchange.sendResponseHeaders(200, 0);
+        try (InputStream input = exchange.getRequestBody(); OutputStream response = exchange.getResponseBody()) {
+            if (format.codec.equals("PCMA")) {
+                byte[] pcm;
+                while (!closed && (pcm = input.readNBytes(640)).length > 0) {
+                    lastTalkActivity = System.nanoTime();
+                    if (pcm.length != 640) { Arrays.fill(pcm, (byte) 0); break; }
+                    if (!started) { talkback.start(); started = true; }
+                    try { talkback.pcma(pcm); } finally { Arrays.fill(pcm, (byte) 0); }
+                }
+            } else {
+                AudioCodecs.OggReader ogg = new AudioCodecs.OggReader(input);
+                byte[] packet;
+                while (!closed && (packet = ogg.next()) != null) {
+                    lastTalkActivity = System.nanoTime();
+                    if (!started) { talkback.start(); started = true; }
+                    try { talkback.opus(packet); } finally { Arrays.fill(packet, (byte) 0); }
+                }
+            }
+        }
+        } catch (IOException | RuntimeException ignored) { }
+        finally {
+            if (started) talkback.stop();
+            talkExchange.compareAndSet(exchange, null); talking.set(false); exchange.close();
+        }
+    }
+
+    private static void reject(HttpExchange exchange, int status) throws IOException {
+        exchange.sendResponseHeaders(status, -1); exchange.close();
+    }
+
+    @Override public void close() {
+        if (closed) return; closed = true;
+        HttpExchange exchange = talkExchange.getAndSet(null); if (exchange != null) exchange.close();
+        server.stop(0); timer.shutdownNow(); executor.shutdownNow();
+        if (talkback != null) talkback.close();
+        for (Viewer viewer : viewers) viewer.clear(); viewers.clear();
+    }
+
+    private static final class Viewer {
+        final ArrayBlockingQueue<byte[]> queue = new ArrayBlockingQueue<>(64);
+        void offer(byte[] bytes) {
+            byte[] own = Arrays.copyOf(bytes, bytes.length);
+            if (!queue.offer(own)) { byte[] dropped = queue.poll(); if (dropped != null) Arrays.fill(dropped, (byte) 0); if (!queue.offer(own)) Arrays.fill(own, (byte) 0); }
+        }
+        void clear() { byte[] bytes; while ((bytes = queue.poll()) != null) Arrays.fill(bytes, (byte) 0); }
+    }
+}

package/backend/Bridge.java

@@ -106,9 +106,12 @@ public final class Bridge {
         if (!Set.of("homes", "devices", "stream").contains(action)) throw new Failure(7, "invalid_request");
         if (!(action.equals("homes") && args.length == 2)
                 && !(action.equals("devices") && args.length == 4)
-                && !(action.equals("stream") && args.length == 7)) throw new Failure(7, "invalid_request");
+                && !(action.equals("stream") && (args.length == 7 || args.length == 8))) throw new Failure(7, "invalid_request");
         if (args.length >= 4) { identifier(args[2]); identifier(args[3]); }
-        if (action.equals("stream")) { identifier(args[4]); number(args[5], 1024, 65535); number(args[6], 0, 3); }
+        if (action.equals("stream")) {
+            identifier(args[4]); number(args[5], 1024, 65535); number(args[6], 0, 3);
+            if (args.length == 8) number(args[7], 0, 1);
+        }
         try { credentials = store.load(); }
         catch (RuntimeException invalid) { throw new Failure(2, "authentication_required"); }
         credentials.region = args[1];
@@ -130,12 +133,13 @@ public final class Bridge {
                     "online", device.online())).toList()));
             return 0;
         }
-        if (!action.equals("stream") || args.length != 7) throw new Failure(7, "invalid_request");
+        if (!action.equals("stream") || (args.length != 7 && args.length != 8)) throw new Failure(7, "invalid_request");
         String did = identifier(args[4]);
         int port = number(args[5], 1024, 65535);
         int quality = number(args[6], 0, 3);
         Device device = cameras.stream().filter(camera -> did.equals(camera.did())).findFirst()
                 .orElseThrow(() -> new Failure(7, "camera_not_in_home"));
+        NightVisionControl.cacheVerifiedCapability(credentials, home, device);
         if (!device.online()) throw new Failure(4, "camera_offline");
         progress(Stage.PROVISIONING);
         Provisioning provision = request(() -> cloud.missGetVendor(did));
@@ -143,7 +147,7 @@ public final class Bridge {
             emit(Map.of("success", false, "reason", "unsupported_transport", "vendor", provision.vendorId()));
             return 1;
         }
-        return stream(provision, port, quality);
+        return stream(provision, port, quality, args.length == 8 && "1".equals(args[7]));
     }
 
     private <T> T request(Supplier<T> call) {
@@ -174,7 +178,7 @@ public final class Bridge {
         } catch (Exception failure) { throw new Failure(2, "authentication_required"); }
     }
 
-    private int stream(Provisioning provision, int port, int quality) {
+    private int stream(Provisioning provision, int port, int quality, boolean audioEnabled) {
         Resources resources = new Resources();
         AtomicBoolean running = new AtomicBoolean(true);
         Thread hook = new Thread(() -> { running.set(false); resources.close(); }, "mijia-bridge-stop");
@@ -188,13 +192,59 @@ public final class Bridge {
             resources.camera = new MissClient(resources.connection);
             progress(Stage.AUTHENTICATION, provision.vendorId());
             resources.camera.authenticate(provision.clientPublic(), provision.sign(), Duration.ofSeconds(15));
-            resources.camera.useSessionKey(MissCrypto.sessionKey(provision.clientPrivate(), provision.devicePublic()));
+            resources.sessionKey = MissCrypto.sessionKey(provision.clientPrivate(), provision.devicePublic());
+            resources.camera.useSessionKey(resources.sessionKey);
             progress(Stage.WAITING_VIDEO, provision.vendorId());
-            resources.camera.startVideo(quality, false);
+            resources.camera.startVideo(quality, audioEnabled);
             VideoWatchdog watchdog = new VideoWatchdog(System::nanoTime, Duration.ofSeconds(10), Duration.ofSeconds(15));
+            long lastAudio = 0, lastAudioStatus = 0;
+            boolean audioPresent = false, unsupportedReported = false;
             while (running.get()) {
                 MediaPacket frame = resources.camera.nextMedia(Duration.ofSeconds(1));
                 watchdog.observe(frame != null && frame.isVideo(), frame != null && frame.isVideo() && frame.isKeyframe());
+                long now = System.nanoTime();
+                if (audioEnabled && frame != null && !frame.isVideo()) {
+                    AudioStreamServer.Format format = AudioStreamServer.Format.of(frame);
+                    if (format != null) {
+                        if (resources.audio == null) {
+                            try {
+                                resources.talkback = new Talkback(resources.camera, resources.connection, resources.sessionKey);
+                                resources.audio = new AudioStreamServer(format, resources.talkback);
+                                resources.audio.start();
+                                emit(Map.ofEntries(Map.entry("success", true), Map.entry("event", "audio_ready"),
+                                        Map.entry("audioPort", resources.audio.port()), Map.entry("audioCodec", format.codec()),
+                                        Map.entry("format", format.codec().equals("PCMA") ? "wav" : "ogg"),
+                                        Map.entry("sampleRate", format.sampleRate()), Map.entry("channels", format.channels()),
+                                        Map.entry("talkFormat", format.talkFormat()), Map.entry("talkSampleRate", format.talkSampleRate()),
+                                        Map.entry("talkChannels", format.talkChannels()), Map.entry("talkPath", format.talkPath()),
+                                        Map.entry("flags", frame.flags()), Map.entry("payloadBytes", frame.payload().length)));
+                            } catch (IOException | RuntimeException failure) {
+                                if (resources.audio != null) resources.audio.close();
+                                resources.audio = null;
+                                if (resources.talkback != null) resources.talkback.close();
+                                resources.talkback = null;
+                                emit(Map.of("success", true, "event", "audio_status", "audioReceived", false));
+                            }
+                        }
+                        if (resources.audio != null && resources.audio.format().equals(format)) {
+                            resources.audio.publish(frame);
+                            lastAudio = now;
+                            if (!audioPresent || now - lastAudioStatus >= Duration.ofSeconds(5).toNanos()) {
+                                emit(Map.of("success", true, "event", "audio_status", "audioReceived", true));
+                                lastAudioStatus = now;
+                            }
+                            audioPresent = true;
+                        }
+                    } else if (!unsupportedReported) {
+                        emit(Map.of("success", true, "event", "audio_unsupported", "codecId", frame.codecId(),
+                                "flags", frame.flags(), "payloadBytes", frame.payload().length));
+                        unsupportedReported = true;
+                    }
+                }
+                if (audioPresent && now - lastAudio >= Duration.ofSeconds(10).toNanos()) {
+                    emit(Map.of("success", true, "event", "audio_status", "audioReceived", false));
+                    audioPresent = false;
+                }
                 if (frame == null || !frame.isVideo()) {
                     continue;
                 }
@@ -223,11 +273,17 @@ public final class Bridge {
         volatile PpppConnection connection;
         volatile MissClient camera;
         volatile HttpStreamServer server;
+        volatile AudioStreamServer audio;
+        volatile Talkback talkback;
+        volatile byte[] sessionKey;
         synchronized void close() {
+            if (audio != null) { try { audio.close(); } catch (Exception ignored) { } audio = null; }
+            if (talkback != null) { try { talkback.close(); } catch (Exception ignored) { } talkback = null; }
             if (camera != null) { try { camera.stopVideo(); } catch (Exception ignored) { } camera = null; }
             if (server != null) { try { server.close(); } catch (Exception ignored) { } server = null; }
             if (connection != null) { try { connection.close(); } catch (Exception ignored) { } connection = null; }
             if (relay != null) { try { relay.close(); } catch (Exception ignored) { } relay = null; }
+            if (sessionKey != null) { java.util.Arrays.fill(sessionKey, (byte) 0); sessionKey = null; }
         }
     }