scrypted-tuya 0.1.2-beta → 2.1.0

package/docs/tuya-webrtc-talkback.md

@@ -0,0 +1,215 @@
+# Tuya WebRTC talkback (Merkury CW048)
+
+This note is for the next engineer/model. It records what actually worked on a live camera, not Tuya’s marketing docs.
+
+**Status (2026-09-06):** live RTMP talkback uses the DAC-inverse μ-law encoder (`talkback-dac.ts`) plus a **telephone bandpass 300–3400 Hz**. Speech is usable at normal tempo; a 4 kHz whistle remains (firmware byte-play). Restricted 8-code book killed the whistle but was too quiet.
+
+## Camera
+
+|             |                                                                    |
+| ----------- | ------------------------------------------------------------------ |
+| Model       | Merkury CW048 Smart 1080p                                          |
+| DID         | `eba0193b64396cb4fbyqwf`                                           |
+| productId   | `hepu0fueo4rmwfac`                                                 |
+| UID         | `az1727388496362z0d7d`                                             |
+| Region      | US (`protect-us.ismartlife.me`, MQTT `m1.tuyaus.com`)              |
+| p2pType     | **4** (WebRTC). Not classic Tuya P2P command-channel.              |
+| Skill audio | `codecType: 101`, `dataBit: 16`, `sampleRate: 8000`, `channels: 1` |
+
+Tuya enum (after `TUYA_CODEC_VIDEO_MAX = 99`): ADPCM=100, **PCM=101**, AAC_RAW=102, …, **G711U=105**, G711A=106. So `codecType: 101` is **PCM**, not G.711. That describes capture, not necessarily the speaker path.
+
+## Native ThingP2P is a dead end on this device
+
+ThingCamera 7.10.2 / ThingP2P V3+V4 (PlayCover frameworks, macCatalyst worker in `/Users/resonaura/tuya-exp/p2p_camera_worker.mm`):
+
+- ICE + MQTT answer succeed (`RTC_CONNECTION_MODE_P2P`).
+- `connectWithP2pConnectParams` returns **-20002** if username/password are null (skips `SendAuthorizationInfo`).
+- Preview callback never fires (`errcode -40205`), no incoming audio.
+- MQTT `close_reason: 100` after ~15 s.
+- Wrong `SendAuthorizationInfo` (admin/MD5/configV4 passwd) → immediate `close_reason: 100`.
+- Protect `configV4` username/passwd stamped on the native token → ICE **-58**, no answer.
+
+Do **not** spend more time on channel-0 auth for this p2pType-4 camera. Talkback that the original Protect web app uses is **WebRTC + MQTT protocol 312 `speaker`**.
+
+## Working transport
+
+1. Protect cookies → `POST /api/jarvis/mqtt` and `/api/jarvis/config` or `configV4` (ICE + `auth`).
+2. MQTT WSS to `m1.tuyaus.com`. In: `/av/u/{msid}`. Out: `/av/moto/{motoId}/u/{did}`.
+3. Native `tuya-streamer` (`bridge/apps/native`, libdatachannel v0.22.3) builds a PeerConnection.
+4. MQTT protocol **302** `type=offer`, `msg.mode="webrtc"`, plus `auth` and ICE token. Do **not** stamp configV4 username/passwd onto this offer (ICE -58).
+5. Answer + candidates on MQTT → `setRemoteDescription` / `addIceCandidate`.
+6. MQTT protocol **312** `type=speaker`, `msg.mode="webrtc"`, `cmdValue: 1`. Camera replies `resCode: 0`.
+7. RTP talkback: UDP to `127.0.0.1:{talkback_port}` from Node; C++ rewrites SSRC/seq/timestamp and `Track::send()`s a **full RTP packet** (12-byte header + payload). No packetizer on the send track.
+8. `cmdValue: 0` to close speaker.
+
+Harness: `/Users/resonaura/tuya-exp/test_webrtc_talkback.js`  
+Encoder that sounds OK: `/Users/resonaura/tuya-exp/encode_talkback_dac.py`  
+Binary: `bridge/apps/native/build/tuya-streamer`
+
+### SDP that the camera accepts
+
+Offer we used (camera **rejects L16**):
+
+```
+m=audio 9 UDP/TLS/RTP/SAVPF 10 0 8
+a=sendrecv
+a=rtpmap:10 L16/8000/1
+a=rtpmap:0 PCMU/8000/1
+a=rtpmap:8 PCMA/8000/1
+```
+
+Answer:
+
+```
+m=audio 9 UDP/TLS/RTP/SAVPF 0 8
+a=sendrecv
+a=rtpmap:0 PCMU/8000/1
+a=rtpmap:8 PCMA/8000/1
+```
+
+Wire PT for talkback: **0** (PCMU). PT=8 (PCMA) was worse. L16 is never in the answer.
+
+## Asymmetric audio (the whole bug)
+
+|             | Camera → us (mic)                            | Us → camera (speaker)               |
+| ----------- | -------------------------------------------- | ----------------------------------- |
+| SDP         | PCMU/PCMA                                    | PCMU/PCMA                           |
+| RTP PT      | **0**                                        | **0**                               |
+| Payload     | **640 B raw s16le PCM**, 320 samples, ~40 ms | Must be **G.711 μ-law** (see below) |
+| Timestamp Δ | **+320**                                     | +160 per 160 B / 20 ms packet       |
+
+Dumping 640-byte RX payloads as `s16le` 8 kHz is real speech (RMS ~6500). Same bytes as μ-law are not speech.
+
+**Speaker path (proven by listening):**
+
+1. Firmware treats PT=0 as G.711 μ-law and runs `tuya_g711_decode`.
+2. Each μ-law byte becomes one int16 (2 bytes LE).
+3. The DAC / `tal_ao` plays **each of those bytes as an 8-bit sample at 8 kHz**.
+4. So 160 μ-law bytes → 160 int16 → 320 DAC ticks → **40 ms of playback for a 20 ms packet** → **exactly 2× slow, pitch −1 octave** (“vinyl”).
+
+That is why:
+
+- Naive PCM16 640 B / PT=0 → **noise** (linear bytes fed to μ-law decode). Big-endian too. Camera does **not** want L16 on ingress even though it **sends** L16-like PCM on egress.
+- Naive μ-law 320 B / 40 ms, `ts += 320` → **intelligible vinyl** (correct μ-law, then byte-play stretches 2×).
+- 2× speed / +1 octave precompensation (`asetrate=16000,aresample=8000` then μ-law) → **normal tempo**, still hiss + 4 kHz.
+
+## How hiss was removed
+
+Broadband hiss = μ-law bytes interpreted as linear PCM16 (wrong codec), or LSB of decoded int16 (quantization) mixed into the 8-bit DAC stream.
+
+Fix: do **not** send linear PCM on PT=0. Send μ-law, but **choose codes so the two decoded bytes are the DAC ticks we want**, not “encode 8 kHz PCM then hope”.
+
+Inverse model:
+
+- Target DAC is 8 kHz, 8-bit, **0 = silence** (μ-law `0xFF` decodes to `(0,0)`; confirmed **no whistle in silence**).
+- Each μ-law byte produces two DAC ticks (LSB, MSB of the decoded int16 LE).
+- Wire rate must be **half** of DAC rate: 8000 μ-law bytes/s on the wire → 16000 DAC ticks/s? Wait: 160 samples/20 ms = 8000 μ-law/s → 16000 bytes/s at DAC = **2× 8 kHz**. That is the stretch. So for 4.2 s of speaker audio we send **2.1 s** of μ-law (16800 bytes).
+
+Encoder: `/Users/resonaura/tuya-exp/encode_talkback_dac.py`.
+
+## How the 4 kHz whistle was removed
+
+User matched the tone on a generator: **exactly 4000 Hz**. That is Nyquist of 8 kHz byte-play: LSB and MSB of each decoded int16 alternating.
+
+- Silence `0xFF` → `(0,0)` → **no 4 kHz**.
+- Speech using the full μ-law table → 4 kHz **gated with voice** (like a noise gate): mid-level G.711 codes pin LSB at ~124/132 (bias `0x84`) and put the envelope in MSB.
+- `(s,s)` packing did **not** kill it: almost no μ-law code has LSB == MSB except silence. Nearest-neighbour still emits a 4 kHz square.
+
+**Fix that worked:** restrict the codebook to μ-law bytes whose decoded `|LSB − MSB| ≤ 3`.
+
+Working codes (8):
+
+`[0, 1, 103, 127, 128, 129, 231, 255]`
+
+Notable pairs:
+
+| μ-law    | decoded LE bytes | notes                                     |
+| -------- | ---------------- | ----------------------------------------- |
+| 255, 127 | 0, 0             | silence                                   |
+| 231      | 4, 1             | quiet positive                            |
+| 103      | 252, 254         | near-full as unsigned; noisy if DAC is U8 |
+| 128, 129 | ~124, ~121–125   | full-scale positive                       |
+| 0, 1     | ~132, ~130–134   | full-scale negative                       |
+
+Pipeline that killed the whistle:
+
+1. `lowpass=1600`, `aresample=4000`, `s8` (signed 8-bit so WAV silence → 0).
+2. One μ-law byte per 4 kHz sample, chosen from the 8-code set (both DAC ticks ≈ that sample).
+3. RTP: **PT=0, 160 bytes, 20 ms, timestamp += 160**, marker on **every** packet (`MARKER_ALL=1`).
+4. Silence padding with **0xFF**, never 0x00 (μ-law 0x00 is −32124, a rail slam).
+
+Side effect: voice is **bitcrushed / robotic**. Only ~3 usable amplitude steps. That is the remaining distortion.
+
+## Packet spec (current best)
+
+```
+PT            0
+payload       μ-law from encode_talkback_dac.py (restricted codebook)
+frame         160 bytes
+ptime         20 ms
+timestamp     +160 per packet   (peer.cpp: payload < 640 → += payload_bytes)
+marker        1 on every packet
+silence byte  0xFF
+SSRC          rewritten to audio send-track SSRC in peer.cpp
+```
+
+`peer.cpp` timestamp rule:
+
+```
+if payload_bytes >= 640:  ts += payload_bytes / 2   # PCM16 640 B (camera TX)
+else:                     ts += payload_bytes       # G.711 160/320 B
+```
+
+Do not send 640 B linear PCM on PT=0 for talkback.
+
+## Experiments (same 4.2 s 8 kHz s16le clip)
+
+|     | Payload                       | Result                                                                   |
+| --- | ----------------------------- | ------------------------------------------------------------------------ |
+| A   | PCMU 160 B / 20 ms            | Distorted + noise                                                        |
+| B   | s16le 640 B / 40 ms PT=0      | No words, whistle, holes                                                 |
+| C   | PCMA 320 B / 40 ms            | “Better”, 2× slow, noise                                                 |
+| D   | PCMA 160 B / 20 ms            | 2× slow, noise, whistle                                                  |
+| E   | PCMA 16 kHz 320 B / 20 ms     | Unintelligible                                                           |
+| F   | PCMU 320 B / 40 ms            | **Best naive intelligibility**, exactly 2× slow, −1 octave, hiss + 4 kHz |
+| G   | s16le 640 B / 40 ms, ts+=320  | Noise                                                                    |
+| H   | s16be 640 B / 40 ms           | Noise (endianness is **not** the fix)                                    |
+| I   | 2× precomp PCMU 160 B / 20 ms | Tempo OK, words under loud hiss                                          |
+| J   | Inverse DAC map, full table   | Hiss gone, speech OK-ish, **4 kHz whistle**                              |
+| K   | Codebook `\|LSB−MSB\|≤16`     | Whistle quieter, **gated with voice**                                    |
+| L   | Codebook `≤3` (8 codes)       | **Whistle gone.** Speech robotic.                                        |
+
+## Integration into the app (shipped)
+
+`rtmp.service.ts` now:
+
+1. Accepts any RTMP audio codec (AAC, MP3, Speex, Nellymoser, PCM, …) as FLV tags.
+2. ffmpeg: soxr → 8 kHz → **telephone 300–3400 Hz** → soxr 16 kHz s16le.
+3. `talkback-dac.ts`: 16 kHz u8 pairs → μ-law LUT (same inverse as the file tests).
+4. RTP 160 B / 20 ms, PT=0, marker every packet, `ts += 160`.
+5. MQTT 312 speaker on publish / off on unpublish (already wired).
+
+`peer.cpp` must keep `ts += payload_bytes` for 160 B frames (do not divide by 2).
+
+## Telephone filter
+
+**Before this integration: no.** The file harness only low-passed at ~3 kHz. Lows were not cut.
+
+**Now in the app:** ffmpeg `highpass=f=300:poles=2,lowpass=f=3400:poles=2` after resampling to 8 kHz (ITU G.711 / POTS band), then soxr upsample to 16 kHz so pairing emits 8000 μ-law bytes/s (same ratio as the working file tests). Any RTMP audio codec is decoded by ffmpeg first (`-f flv`).
+
+## Next work
+
+- 4 kHz whistle is a firmware DAC artifact; restricted codebook removes it but wrecks loudness.
+- Replay harness: `MARKER_ALL=1 AUDIO_FILE=... FRAME_SIZE=160 FRAME_MS=20 PAYLOAD_TYPE=0 node /Users/resonaura/tuya-exp/test_webrtc_talkback.js`
+
+## Files
+
+| Path                                                | Role                                        |
+| --------------------------------------------------- | ------------------------------------------- |
+| `docs/tuya-webrtc-talkback.md`                      | This note                                   |
+| `/Users/resonaura/tuya-exp/test_webrtc_talkback.js` | MQTT + RTP harness                          |
+| `/Users/resonaura/tuya-exp/encode_talkback_dac.py`  | Working encoder                             |
+| `bridge/apps/native/src/webrtc/peer.cpp`            | RTP rewrite + send                          |
+| `bridge/apps/server/src/streaming/talkback-dac.ts`  | LUT encoder (u8 pair → μ-law)               |
+| `bridge/apps/server/src/streaming/rtmp.service.ts`  | Live RTMP → telephone → DAC μ-law → UDP RTP |
+| `src/accessories/camera.ts`                         | Scrypted stream options (`pcm_alaw` today)  |

package/package.json

@@ -13,7 +13,8 @@
       "scrypted-deploy": "scrypted-deploy",
       "scrypted-changelog": "scrypted-changelog",
       "scrypted-package-json": "scrypted-package-json",
-      "scrypted-readme": "scrypted-readme"
+      "scrypted-readme": "scrypted-readme",
+      "audio:talkback": "npx tsx scripts/audio/talkback-sample.ts"
    },
    "keywords": [
       "scrypted",
@@ -41,9 +42,9 @@
       "@types/qrcode-svg": "^1.1.5",
       "@types/ws": "^8.18.0"
    },
-   "version": "0.1.2-beta",
+   "version": "2.1.0",
    "packageManager": "npm@10.9.2+sha512.8ab88f10f224a0c614cb717a7f7c30499014f77134120e9c1f0211ea3cf3397592cbe483feb38e0c4b3be1c54e347292c76a1b5edb94a3289d5448484ab8ac81",
-   "description": "Unofficial standalone Scrypted Tuya plugin with maximum advertised camera quality selection.",
+   "description": "Unofficial standalone Scrypted Tuya plugin with maximum advertised camera quality selection and Smart Life P2P bridge integration.",
    "repository": {
       "type": "git",
       "url": "git+https://github.com/resonaura/scrypted-tuya.git"
@@ -53,7 +54,6 @@
       "url": "https://github.com/resonaura/scrypted-tuya/issues"
    },
    "publishConfig": {
-      "access": "public",
-      "tag": "beta"
+      "access": "public"
    }
 }

package/scripts/audio/talkback-sample.ts

@@ -0,0 +1,120 @@
+import { execSync, spawn } from "child_process";
+import * as fs from "fs";
+import * as path from "path";
+import axios from "axios";
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+async function getCamerasFromBridge(apiBase: string): Promise<any[]> {
+  try {
+    const res = await axios.get(`${apiBase}/api/cameras`, { timeout: 3000 });
+    return res.data || [];
+  } catch {
+    return [];
+  }
+}
+
+async function main() {
+  console.log("🎙️ Tuya Talkback Sample (RTMP / Intercom wire format)\n");
+
+  const bridgePort = process.env.PORT || "6766";
+  const bridgeHost = process.env.BRIDGE_HOST || "localhost";
+  const rtmpPort = process.env.RTMP_PORT || "1935";
+  let apiBase = `http://${bridgeHost}:${bridgePort}`;
+
+  let cameras = await getCamerasFromBridge(apiBase);
+  if (cameras.length === 0 && !process.env.PORT) {
+    // Also try legacy port 8656
+    const altCameras = await getCamerasFromBridge(`http://${bridgeHost}:8656`);
+    if (altCameras.length > 0) {
+      cameras = altCameras;
+      apiBase = `http://${bridgeHost}:8656`;
+    }
+  }
+  const targetFilter = (
+    process.argv.slice(2).find((a) => !a.toLowerCase().endsWith(".wav")) ||
+    process.env.TARGET_CAM ||
+    ""
+  ).toLowerCase();
+
+  let targetCam = cameras.find(
+    (c: any) =>
+      c.name?.toLowerCase().includes(targetFilter) ||
+      c.did?.toLowerCase().includes(targetFilter),
+  );
+
+  if (!targetCam && cameras.length > 0) {
+    targetCam = cameras[0];
+  }
+
+  const slug = targetCam
+    ? targetCam.name
+        ?.toLowerCase()
+        .replace(/\bcamera\b/g, " ")
+        .replace(/[^a-z0-9]+/g, "-")
+        .replace(/^-+|-+$/g, "") || targetCam.did
+    : targetFilter || "tuya-cam";
+
+  console.log(`🎯 Targeting Camera: ${targetCam?.name || slug} (${targetCam?.did || slug})\n`);
+
+  const wavArgs = process.argv.slice(2).filter((a) => a.toLowerCase().endsWith(".wav"));
+  const defaultWav = path.resolve(__dirname, "../../audio/plop.wav");
+  const wavList = wavArgs.length
+    ? wavArgs.map((w) => {
+        const cands = [w, path.resolve(process.cwd(), w), path.resolve(__dirname, "../../audio", path.basename(w))];
+        const found = cands.find((c) => fs.existsSync(c));
+        return found || w;
+      })
+    : [defaultWav];
+
+  for (const wav of wavList) {
+    if (!fs.existsSync(wav)) {
+      console.error(`❌ Audio file not found: ${wav}`);
+      continue;
+    }
+
+    const rtmpUrl = `rtmp://${bridgeHost}:${rtmpPort}/talk/${slug}`;
+    console.log(`\n▶ Streaming ${path.basename(wav)} -> ${rtmpUrl}`);
+
+    await new Promise<void>((resolve, reject) => {
+      // Stream audio with low-delay flags matching talkback specification
+      const ffmpegArgs = [
+        "-hide_banner",
+        "-loglevel", "info",
+        "-re",
+        "-i", wav,
+        "-vn",
+        "-c:a", "aac",
+        "-b:a", "16k",
+        "-ar", "16000",
+        "-ac", "1",
+        "-f", "flv",
+        rtmpUrl,
+      ];
+
+      const proc = spawn("ffmpeg", ffmpegArgs, { stdio: "inherit" });
+      proc.on("close", (code) => {
+        if (code === 0) {
+          console.log(`✅ Finished ${path.basename(wav)} successfully!`);
+          resolve();
+        } else {
+          console.warn(`⚠️ FFmpeg exited with code ${code}`);
+          resolve();
+        }
+      });
+      proc.on("error", (err) => {
+        console.error(`❌ FFmpeg error: ${err.message}`);
+        reject(err);
+      });
+    });
+
+    await sleep(500);
+  }
+
+  console.log("\n🎉 Talkback playback complete.");
+}
+
+main().catch((err) => {
+  console.error("❌ Fatal error:", err);
+  process.exit(1);
+});