personal-vad 0.1.0__tar.gz

personal_vad-0.1.0/LICENSE

@@ -0,0 +1,176 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS

personal_vad-0.1.0/PKG-INFO

@@ -0,0 +1,478 @@
+Metadata-Version: 2.4
+Name: personal-vad
+Version: 0.1.0
+Summary: Personal VAD: Speaker-Conditioned Target Voice Activity Detection
+Home-page: https://github.com/wq2012/personal_vad
+Author: Quan Wang, Shaojin Ding, Ignacio Lopez Moreno
+Author-email: quanw@google.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: scipy>=1.7.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: matplotlib>=3.5.0
+Requires-Dist: tensorflow>=2.11.0
+Requires-Dist: safetensors>=0.4.0
+Requires-Dist: huggingface_hub>=0.20.0
+Requires-Dist: soundfile>=0.12.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Personal VAD (Speaker-Conditioned Voice Activity Detection)
+
+[![Python application](https://github.com/wq2012/personal_vad/actions/workflows/pythonapp.yml/badge.svg)](https://github.com/wq2012/personal_vad/actions/workflows/pythonapp.yml)
+[![HuggingFace Space](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space%20Demo-orange)](https://huggingface.co/spaces/wq2012/personal_vad)
+[![HuggingFace Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-13%20Pretrained%20Models-blue)](https://huggingface.co/wq2012/personal-vad-v2-conformer-film-dvector-cos)
+[![PyPI Version](https://img.shields.io/pypi/v/personal-vad.svg)](https://pypi.org/project/personal-vad/)
+[![Python Versions](https://img.shields.io/pypi/pyversions/personal-vad.svg)](https://pypi.org/project/personal-vad/)
+[![Downloads](https://static.pepy.tech/badge/personal-vad)](https://www.pepy.tech/projects/personal-vad)
+
+**This is not an officially supported Google product.**
+
+## Introduction
+
+This repository provides a standalone, open-source Python reproduction of
+**Personal VAD 1.0** and **Personal VAD 2.0** based on the Odyssey 2020 and
+Interspeech 2022 papers:
+
+> **Personal VAD: Speaker-Conditioned Voice Activity Detection**
+> *Shaojin Ding, Quan Wang, Shuo-yiin Chang, Li Wan, Ignacio Lopez Moreno*
+> Paper: [https://arxiv.org/pdf/1908.04284](https://arxiv.org/pdf/1908.04284) | Interactive Demo: [Hugging Face Space (`wq2012/personal_vad`)](https://huggingface.co/spaces/wq2012/personal_vad) | Blog Page: [https://google.github.io/speaker-id/publications/PersonalVAD/](https://google.github.io/speaker-id/publications/PersonalVAD/)
+
+> **Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition**
+> *Shaojin Ding, Rajeev Rikhye, Qiao Liang, Yanzhang He, Quan Wang, Arun Narayanan, Tom O'Malley, Ian McGraw*
+> Paper: [https://arxiv.org/pdf/2204.03793](https://arxiv.org/pdf/2204.03793) | Pretrained Models: [Hugging Face Hub (`wq2012`)](https://huggingface.co/wq2012)
+
+> [!NOTE]
+> **Open-Source Reproduction Notice**: This library is an independent
+> open-source reproduction of the published papers above. It does **not** use
+> the original internal codebase used in the papers, which relied on Google's
+> proprietary software infrastructure (including internal acoustic frontends,
+> proprietary room simulators, a proprietary 4.88M-parameter 3-layer GE2E LSTM
+> speaker verification d-vector model trained on vendor-collected datasets,
+> and internal streaming on-device ASR evaluation pipelines). Instead, all
+> modules, training scripts, evaluation suites, speaker embedding extractors,
+> and pretrained checkpoints in this repository are newly implemented from
+> scratch using open-source [`lingvo`](https://github.com/tensorflow/lingvo),
+> `tensorflow`, and `numpy`/`scipy`, and trained on the open-source 8-language
+> **Multilingual LibriSpeech (MLS)** corpus (`de`, `en`, `es`, `fr`, `it`,
+> `nl`, `pl`, `pt`).
+
+**Personal VAD** detects frame-level voice activity of a specific **target
+speaker** in multi-speaker and noisy acoustic environments, serving as a
+lightweight on-device gatekeeper before downstream speech recognition (ASR).
+Unlike standard VAD, which classifies each audio frame into two classes
+(`Speech` vs. `Silence`), Personal VAD classifies each frame into three classes:
+
+- **`0`: `SPEECH` (`tss`)** — Target Speaker Speech
+- **`1`: `SILENCE` (`ns`)** — Non-Speech / Silence
+- **`2`: `SPEECH_FROM_NON_TARGET_SPEAKER` (`ntss`)** — Non-Target Speaker Speech
+
+<p align="center">
+  <img src="resources/pvad_conventional.png" alt="Fig. 1: Conventional Personal VAD System Overview" width="440" />
+  &nbsp;&nbsp;&nbsp;
+  <img src="resources/speaker_verification_dvector.png" alt="Fig. 2: Utterance-level d-vector Speaker Verification System" width="440" />
+</p>
+
+---
+
+## Architecture & Features
+
+### 1. Personal VAD 1.0 Conditioning Architectures (`SC`, `ST`, `ET`, `SET`)
+
+<p align="center">
+  <img src="resources/pvad1_architectures.png" alt="Four Personal VAD 1.0 Architectures: SC, ST, ET, SET" width="760" />
+</p>
+
+- **`SC` (Score Combination Baseline)**: Runs a standard 2-class VAD and
+  frame-level speaker verification independently, rescaling the cosine
+  similarity score via online 5th/95th histogram percentiles
+  (`OnlinePercentileValue`) to split speech probability into `tss` and `ntss`.
+- **`ST` (Score-Conditioned Training)**: Concatenates 40-D log-Mel features and
+  frame-level speaker verification cosine similarity `[x_t, s_t]` (41-D input)
+  into a 2-layer 64-unit unidirectional LSTM.
+- **`ET` (Embedding-Conditioned Training)**: Concatenates 40-D log-Mel features
+  and the 256-D enrolled target speaker d-vector `[x_t, e_target]` (296-D input)
+  into a 2-layer 64-unit unidirectional LSTM (`129,795` parameters, `262 KB`
+  quantized TFLite), eliminating the runtime speaker verification network.
+- **`SET` (Score and Embedding Conditioned Training)**: Concatenates acoustic
+  features, target d-vector, and frame-level cosine score
+  `[x_t, e_target, s_t]` (297-D input).
+- **Weighted Pairwise Loss (`WPL`)**: Prioritizes `<tss, ns>` and `<tss, ntss>`
+  decision boundaries (`weight = 1.0`) over `<ns, ntss>` (`weight = 0.1`).
+
+<p align="center">
+  <img src="resources/wpl_weight_ablation.png" alt="Weighted Pairwise Loss Weight Ablation" width="440" />
+</p>
+
+### 2. Personal VAD 2.0 Conformer, FiLM Modulation & Speaker Pre-Net (`B1`, `B2`, `B0`, `E0`, `E1`, `E2`, `E3`, `E5`)
+
+<p align="center">
+  <img src="resources/pvad2_film_prenet.png" alt="Personal VAD 2.0 FiLM and Speaker Pre-Net Conditioning" width="680" />
+</p>
+
+- **Stacked Log-Mel Acoustic Frontend**: Extracts 128-dimensional log-Mel
+  filterbank energies (32ms window, 10ms step), stacks 4 contiguous frames
+  (3 left-context frames + current frame = 512-D), and subsamples by a factor
+  of 3 (30ms output frame step).
+- **Causal Streaming Conformer Backbone**: 4-layer causal Conformer
+  (`model_dim=64`, `num_heads=8`, `left_context=31`, `right_context=0`,
+  depthwise conv `kernel_size=7`, `ffn_multiplier=8`), supporting both
+  full-sequence training and stateful frame-by-frame streaming inference
+  (`stream_step`).
+- **Advanced Speaker Conditioning**:
+  - `FILM_DVECTOR` (**Architecture E1**): Feature-wise Linear Modulation (FiLM)
+    after the Conformer backbone using affine scale and shift parameters
+    generated from the target speaker d-vector `e_target`.
+  - `FILM_COS` (**Architecture E2**): A 2-layer Conformer **Speaker Pre-Net**
+    predicts frame-level speaker embeddings `e_prenet`, computes cosine
+    similarity `s_t = cos(e_prenet, e_target)`, and modulates the main
+    Conformer output via FiLM.
+  - `FILM_DVECTOR_COS` (**Architecture E3**): Concatenates both
+    `[e_target, s_t]` and feeds the combined vector into the FiLM generator.
+- **Unified Enrolled & Enrollment-Less Training (Algorithm 1, Architecture E5)**:
+  During training, with probability `p_0 = 0.2`, replaces `e_target` with a
+  zero vector `0` and maps `ntss` labels to `tss`, allowing a single on-device
+  model to operate as both a speaker-conditioned Personal VAD and an
+  enrollment-less Standard VAD.
+
+### 3. Speaker Embedding (`d-vector`) Extractor Design & Transparency (`speaker_subspace.npz`)
+
+In the original Personal VAD 1.0 and 2.0 papers, target speaker embeddings
+(`e_target`) and frame-level verification scores (`s_t`) were extracted using
+Google's internal 3-layer LSTM speaker verification model (`4.88M` parameters)
+trained with Generalized End-to-End (`GE2E`) loss ([Wan et al., ICASSP 2018](https://arxiv.org/pdf/1710.10467))
+on proprietary vendor-collected datasets across 8 languages.
+
+Because that internal speaker verification model and its training corpora are
+proprietary—and to keep this open-source package 100% self-contained in pure
+TensorFlow/NumPy with zero external PyTorch/SpeechBrain/JAX runtime
+dependencies—**we trained our own lightweight, open-set 256-D speaker embedding
+extractor (`OpenSetSpeakerSubspace`, serialized as `speaker_subspace.npz`, ~370 KB)
+from scratch on the 98 training speakers of the 8-language Multilingual
+LibriSpeech (MLS) dataset**:
+
+1. **Multi-Resolution Voiced Acoustic Summary (360-D)**:
+   - Over voiced speech frames (or over a 31-frame / ~310 ms causal sliding
+     window when computing frame-level embeddings `e_t` and cosine scores `s_t`
+     for Personal VAD 1.0 `SC`, `ST`, and `SET`), the extractor computes a
+     360-dimensional acoustic statistic vector combining 40-D log-Mel
+     `[mean, std, 10th-percentile, 90th-percentile, delta_std]` (200-D) with
+     high-resolution 80-D log-Mel `[mean, std]` (160-D).
+2. **Regularized LDA + Orthogonal PCA Subspace Projection (256-D)**:
+   - Fitted strictly on the **98 MLS training speakers** (`922` utterances
+     across `de`, `en`, `es`, `fr`, `it`, `nl`, `pl`, `pt`), it standardizes
+     the 360-D summary vector, solves the regularized **Linear Discriminant
+     Analysis (LDA)** generalized eigenvalue problem between inter-speaker and
+     intra-speaker scatter matrices, completes the orthogonal basis to 256
+     dimensions via PCA on the residual subspace, and L2-normalizes the
+     resulting 256-D speaker embedding (`||e||_2 = 1`).
+3. **Open-Set Unseen-Speaker Verification Accuracy**:
+   - Evaluated on the **35 held-out unseen test speakers** (`315` utterances,
+     zero speaker overlap with training), `speaker_subspace.npz` achieves an
+     open-set speaker verification **ROC-AUC of `0.9521`** (mean same-speaker
+     cosine similarity **`0.7130`** vs. different-speaker **`-0.0082`**).
+4. **Bring-Your-Own Neural Speaker Encoder**:
+   - Both `PersonalVadModel` and `PersonalVadInferenceEngine` accept any
+     external 256-D L2-normalized speaker embedding vector directly (e.g., from
+     [`FlaxSpeaker`](https://github.com/wq2012/FlaxSpeaker) GE2E/Conformer/ECAPA-TDNN
+     models) if you wish to train or run inference with a neural speaker encoder.
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip3 install personal-vad
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/wq2012/personal_vad.git
+cd personal_vad
+pip3 install -r requirements.txt
+pip3 install -e .
+```
+
+---
+
+## Quick Start: Pretrained Model Inference (`personal_vad.load_pretrained`)
+
+All 13 pretrained models are hosted on Hugging Face Hub under
+[`wq2012`](https://huggingface.co/wq2012) and can be loaded directly by repo ID
+or local directory path:
+
+```python
+import personal_vad
+
+# 1. Load a pretrained Personal VAD 2.0 model from Hugging Face Hub or local dir
+engine = personal_vad.load_pretrained(
+    "wq2012/personal-vad-v2-conformer-film-dvector-cos"
+)
+
+# 2. Run speaker-conditioned inference on 16kHz WAV files (or NumPy arrays)
+result = engine.infer_From_files(
+    audio_path="conversation_16k.wav",
+    enroll_path="target_speaker_enroll_16k.wav",
+    threshold=0.5,
+    output_filtered_wav="target_speaker_only.wav",
+)
+
+print("Frame timestamps (s):", result["frame_times_sec"].shape)
+print("Frame posteriors [tss, ns, ntss]:", result["probabilities"].shape)
+print("Target speech ratio:", result["target_mask"].mean())
+
+# 3. Enrollment-less Standard VAD fallback (Architecture E5 with zero d-vector)
+unified_engine = personal_vad.load_pretrained(
+    "wq2012/personal-vad-v2-conformer-unified"
+)
+std_vad_result = unified_engine.infer_From_files(
+    audio_path="conversation_16k.wav",
+    enroll_path=None,  # Uses zero vector e_target = 0 (Algorithm 1)
+)
+
+# 4. On-device 8-bit Quantized TFLite inference
+tflite_result = engine.infer_From_files(
+    audio_path="conversation_16k.wav",
+    enroll_path="target_speaker_enroll_16k.wav",
+    use_tflite=True,
+    quantized=True,
+)
+```
+
+Or run from the command line using `scripts/inference.py` (or `pvad-inference`):
+
+```bash
+python3 scripts/inference.py \
+  --model wq2012/personal-vad-v2-conformer-film-dvector-cos \
+  --audio_wav conversation_16k.wav \
+  --enroll_wav target_speaker_enroll_16k.wav \
+  --output_wav target_speaker_only.wav \
+  --output_json predictions.json
+```
+
+---
+
+## Pretrained Models on Hugging Face Hub (`wq2012`) & Benchmark Results
+
+### 1. Original Paper Reference Benchmarks
+
+#### Personal VAD 1.0 Reference Results ([arXiv:1908.04284](https://arxiv.org/pdf/1908.04284), Table 1 & Table 2)
+
+Average Precision (AP) and micro-averaged mean Average Precision (`mAP`) on
+concatenated LibriSpeech test utterances (clean vs. multistyle training `MTR`
+room reverberation + noise):
+
+| Method | Loss | Without MTR `tss` | Without MTR `ns` | Without MTR `ntss` | Without MTR `mAP` | With MTR `tss` | With MTR `ns` | With MTR `ntss` | With MTR `mAP` | Parameters |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **SC (baseline)** | N/A | 0.886 | 0.970 | 0.872 | 0.900 | 0.777 | 0.908 | 0.768 | 0.801 | 4.88M (SV) + 0.06M (VAD) |
+| **ST** | CE | 0.956 | 0.968 | 0.956 | 0.957 | 0.905 | 0.885 | 0.905 | 0.901 | 4.88M (SV) + 0.06M (PVAD) |
+| **ET** | CE | 0.932 | 0.962 | 0.946 | 0.946 | 0.878 | 0.873 | 0.890 | 0.883 | **0.13M (PVAD)** |
+| **SET** | CE | **0.970** | 0.969 | 0.972 | **0.969** | **0.938** | 0.888 | 0.938 | **0.928** | 4.88M (SV) + 0.13M (PVAD) |
+| **ET** | **WPL** | 0.955 | 0.965 | 0.961 | 0.959 | 0.916 | 0.883 | 0.920 | 0.912 | **0.13M (PVAD)** |
+
+Standard VAD task evaluation on non-concatenated LibriSpeech test utterances
+([arXiv:1908.04284](https://arxiv.org/pdf/1908.04284), Table 2):
+
+| Method | Loss | Without MTR Speech (`s`) AP | Without MTR Non-Speech (`ns`) AP | With MTR Speech (`s`) AP | With MTR Non-Speech (`ns`) AP |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| **Standard VAD** | CE | 0.992 | 0.975 | 0.975 | 0.918 |
+| **Personal VAD (ET)** | CE | 0.991 | 0.965 | 0.979 | 0.893 |
+| **Personal VAD (ET)** | WPL | 0.991 | 0.967 | 0.979 | 0.901 |
+
+#### Personal VAD 2.0 Reference Results ([arXiv:2204.03793](https://arxiv.org/pdf/2204.03793), Table 1 & Table 2)
+
+Downstream streaming ASR Word Error Rate (`WER %`), model size, and FLOPs on
+internal vendor-collected Speech Query (`Non-Concat` and `Concat`) and Voice
+Search (`VS`) test sets:
+
+| Exp | Model Architecture | Enrolled Non-Concat WER (%) | Enrolled Concat WER (%) | Enrollment-Less VS WER (%) | Enrollment-Less Non-Concat WER (%) | Size (MB) | FLOPs (M) |
+| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **B1** | LSTM Standard VAD | N/A | N/A | 7.2 | 11.5 | 1.5 | 3.41 |
+| **B2** | Conformer Standard VAD | N/A | N/A | 6.9 | 10.1 | 0.7 | 8.77 |
+| **B0** | Personal VAD 1.0 (3L LSTM Concat) | 17.9 | 41.0 | >= 100.0 | >= 100.0 | 5.8 | 3.54 |
+| **E0** | + 4L Causal Conformer Concat | 15.3 | 31.5 | — | — | 2.8 | 9.51 |
+| **E1** | + FiLM d-vector Modulation | 11.3 | 29.5 | — | — | 2.8 | 9.58 |
+| **E2** | + Speaker Pre-Net (`cos`) FiLM | 11.7 | 27.5 | — | — | 4.0 | 9.51 |
+| **E3** | + FiLM d-vector & Speaker Pre-Net `cos` | 11.5 | 27.6 | — | — | 4.0 | 9.58 |
+| **E4** | + 8-bit Quantization (w/o Alg. 1) | 12.2 | 27.2 | >= 100.0 | >= 100.0 | 1.0 | 9.58 |
+| **E5** | **Personal VAD 2.0 (w/ Alg. 1 + INT8)** | **12.4** | **32.7** | **7.0** | **10.1** | **1.0** | **9.58** |
+
+---
+
+### 2. Open-Source Reproduction Results on 8-Language Multilingual LibriSpeech (MLS)
+
+All 13 open-source pretrained models were trained and evaluated on the
+open-source 8-language **Multilingual LibriSpeech (MLS)** dataset (`de`, `en`,
+`es`, `fr`, `it`, `nl`, `pl`, `pt`; `1,323` utterances across `142` speakers in
+disjoint speaker splits: `98` train speakers / `315` unseen test utterances
+across `35` unseen test speakers). As detailed in
+[Section 3 above](#3-speaker-embedding-d-vector-extractor-design--transparency-speaker_subspacenpz),
+rather than using the original papers' proprietary 4.88M-parameter 3-layer GE2E
+LSTM speaker encoder, all models below use our self-contained open-set 256-D
+regularized LDA+PCA speaker subspace extractor (`speaker_subspace.npz`) trained
+from scratch on the 98 MLS training speakers, which achieves an unseen-speaker
+verification **ROC-AUC of `0.9521`** (mean positive cosine similarity `0.7130`
+vs. negative `-0.0082`).
+
+Every model repository on Hugging Face Hub includes:
+- `model.safetensors`: SafeTensors FP32 checkpoint
+- `model.weights.h5`: Keras 3 weights checkpoint
+- `model_fp32.tflite`: FP32 TensorFlow Lite FlatBuffer
+- `model_quantized.tflite`: 8-bit dynamic-range quantized TensorFlow Lite model
+- `model_config.json` & `speaker_subspace.npz`: Full model configuration and
+  trained 256-D open-set LDA+PCA speaker d-vector projection matrix
+- `evaluation_metrics.json`: Full evaluation metrics with 95% non-parametric
+  bootstrap confidence intervals
+
+#### A. Personal VAD 1.0 Open-Source Pretrained Models (MLS Concatenated Multi-Speaker Test Set)
+
+| Hugging Face Repository | Arch | Loss | Params | INT8 `.tflite` | Concat `tss` AP | Concat `ns` AP | Concat `ntss` AP | Concat `mAP` (95% Bootstrap CI) | INT8 `.tflite` `mAP` |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| [`wq2012/personal-vad-v1-sc`](https://huggingface.co/wq2012/personal-vad-v1-sc) | `SC` | CE | 64,194 | 196.4 KB | **0.9435** | **0.9604** | **0.7848** | **0.9161** (`[0.9056, 0.9277]`) | **0.9090** |
+| [`wq2012/personal-vad-v1-st-ce`](https://huggingface.co/wq2012/personal-vad-v1-st-ce) | `ST` | CE | 64,515 | 197.4 KB | 0.9394 | 0.9336 | 0.7550 | 0.8834 (`[0.8668, 0.9014]`) | 0.8804 |
+| [`wq2012/personal-vad-v1-et-ce`](https://huggingface.co/wq2012/personal-vad-v1-et-ce) | `ET` | CE | 129,795 | 262.4 KB | 0.9362 | 0.9536 | 0.7817 | 0.8665 (`[0.8460, 0.8897]`) | 0.8609 |
+| [`wq2012/personal-vad-v1-set-ce`](https://huggingface.co/wq2012/personal-vad-v1-set-ce) | `SET` | CE | 130,051 | 262.7 KB | 0.9385 | 0.9510 | 0.7725 | 0.8469 (`[0.8237, 0.8712]`) | 0.8521 |
+| [`wq2012/personal-vad-v1-et-wpl`](https://huggingface.co/wq2012/personal-vad-v1-et-wpl) | `ET` | **WPL** | 129,795 | 262.4 KB | **0.9432** | 0.9433 | 0.7695 | 0.8652 (`[0.8467, 0.8853]`) | **0.8691** |
+
+*Note on Personal VAD 1.0 Reproduction*: Consistent with Paper 1, training the
+lightweight `ET` model (`129.8K` parameters, `262 KB` quantized TFLite, no
+runtime speaker verification model) with **Weighted Pairwise Loss (`WPL`,
+`w_<ns,ntss> = 0.1`)** improves target speaker speech (`tss`) Average Precision
+from `0.9362` (`CE`) to **`0.9432`** (`WPL`) and improves quantized INT8 TFLite
+`mAP` from `0.8609` to **`0.8691`**.
+
+#### B. Personal VAD 2.0 Open-Source Pretrained Models (MLS Multi-Speaker & Single-Speaker Test Sets)
+
+| Hugging Face Repository | Exp | Backbone & Conditioning | Params | INT8 `.tflite` | Non-Concat `mAP` | Concat `tss` AP | Concat `ns` AP | Concat `ntss` AP | Concat `mAP` (95% Bootstrap CI) | INT8 `.tflite` Concat `mAP` |
+| :--- | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| [`wq2012/personal-vad-v2-lstm-concat`](https://huggingface.co/wq2012/personal-vad-v2-lstm-concat) | **B0** | 3L LSTM (256d) + Input Concat | 1,775,107 | 2007.9 KB | 0.7006 | 0.7464 | 0.9378 | 0.3172 | 0.8088 (`[0.7875, 0.8308]`) | 0.8092 |
+| [`wq2012/personal-vad-v2-conformer-concat`](https://huggingface.co/wq2012/personal-vad-v2-conformer-concat) | **E0** | 4L Conformer (64d) + Input Concat | 699,907 | 898.3 KB | 0.7408 | 0.8336 | 0.9678 | 0.5071 | 0.8502 (`[0.8323, 0.8688]`) | 0.8329 |
+| [`wq2012/personal-vad-v2-conformer-film-dvector`](https://huggingface.co/wq2012/personal-vad-v2-conformer-film-dvector) | **E1** | 4L Conformer + FiLM `d-vector` | 716,419 | 917.0 KB | 0.9025 | 0.9370 | 0.9710 | 0.7709 | 0.9060 (`[0.8933, 0.9196]`) | 0.9009 |
+| [`wq2012/personal-vad-v2-conformer-film-cos`](https://huggingface.co/wq2012/personal-vad-v2-conformer-film-cos) | **E2** | 4L Conformer + Speaker Pre-Net `cos` FiLM | 1,058,499 | 1364.0 KB | **0.9207** | **0.9492** | 0.9739 | **0.8172** | 0.9153 (`[0.9024, 0.9277]`) | 0.9086 |
+| [`wq2012/personal-vad-v2-conformer-film-dvector-cos`](https://huggingface.co/wq2012/personal-vad-v2-conformer-film-dvector-cos) | **E3** | 4L Conformer + Pre-Net & `d-vector` FiLM | 1,091,267 | 1397.8 KB | 0.9158 | 0.9476 | **0.9776** | 0.8169 | **0.9222** (`[0.9106, 0.9333]`) | **0.9127** |
+| [`wq2012/personal-vad-v2-conformer-unified`](https://huggingface.co/wq2012/personal-vad-v2-conformer-unified) | **E5** | 4L Conformer + Pre-Net & `d-vector` FiLM + Alg. 1 | 1,091,267 | 1397.8 KB | 0.8574 | 0.9250 | 0.9644 | 0.7558 | 0.9092 (`[0.8956, 0.9216]`) | 0.8939 |
+
+*Note on Personal VAD 2.0 Ablation Trajectory*: Our open-source reproduction on
+8-language MLS reproduces every architectural finding from Table 1 of
+Personal VAD 2.0 ([arXiv:2204.03793](https://arxiv.org/pdf/2204.03793)):
+1. **Conformer vs. LSTM (`B0 -> E0`)**: Replacing the 3-layer LSTM (`B0`,
+   `2007.9 KB`, Concat `mAP = 0.8088`) with the 4-layer causal Conformer (`E0`,
+   `898.3 KB`, Concat `mAP = 0.8502`) improves multi-speaker `mAP` by `+4.14`
+   points while cutting model size by more than 55%.
+2. **FiLM & Speaker Pre-Net Modulation (`E0 -> E1 -> E2 -> E3`)**: Moving from
+   input concatenation (`E0`, `mAP = 0.8502`) to FiLM d-vector modulation
+   (`E1`, `mAP = 0.9060`), Speaker Pre-Net cosine FiLM (`E2`, `mAP = 0.9153`),
+   and combined Speaker Pre-Net + d-vector FiLM (`E3`, **`mAP = 0.9222`**,
+   `tss AP = 0.9476`) yields monotonic improvements across all classes.
+3. **8-Bit Dynamic Range Quantization (`E4`)**: Quantizing `E3` to 8-bit
+   integer TFLite reduces model size from `4422.1 KB` to `1397.8 KB` (3.16x
+   compression) while retaining `0.9127` Concat `mAP` (`0.9444` `tss` AP).
+
+#### C. Standard VAD Baselines & Enrollment-Less Fallback Evaluation (`e_target = 0`, Algorithm 1)
+
+| Hugging Face Repository | Exp | Evaluation Mode | Params | INT8 `.tflite` | Speech AP | Silence (`ns`) AP | `mAP` (95% Bootstrap CI) | INT8 `.tflite` `mAP` |
+| :--- | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| [`wq2012/personal-vad-v2-lstm-std-vad`](https://huggingface.co/wq2012/personal-vad-v2-lstm-std-vad) | **B1** | Dedicated 3L LSTM Standard VAD | 1,709,314 | 1941.2 KB | 0.9965 | 0.9343 | 0.9927 (`[0.9912, 0.9942]`) | 0.9912 |
+| [`wq2012/personal-vad-v2-conformer-std-vad`](https://huggingface.co/wq2012/personal-vad-v2-conformer-std-vad) | **B2** | Dedicated 4L Conformer Standard VAD | 683,458 | 880.1 KB | **0.9968** | **0.9726** | **0.9937** (`[0.9923, 0.9949]`) | **0.9926** |
+| [`wq2012/personal-vad-v2-conformer-film-dvector-cos`](https://huggingface.co/wq2012/personal-vad-v2-conformer-film-dvector-cos) | **E3 / E4** | Enrollment-Less (`e_target = 0`, w/o Alg. 1) | 1,091,267 | 1397.8 KB | 0.9499 | 0.9760 | 0.8891 (`[0.8801, 0.8993]`) | 0.8884 |
+| [`wq2012/personal-vad-v2-conformer-unified`](https://huggingface.co/wq2012/personal-vad-v2-conformer-unified) | **E5** | **Enrollment-Less (`e_target = 0`, w/ Alg. 1)** | 1,091,267 | 1397.8 KB | **0.9957** | 0.9596 | **0.9522** (`[0.9399, 0.9624]`) | **0.9384** |
+
+*Note on Algorithm 1 (`E5` vs. `E3`)*: When no enrolled speaker is present
+(`e_target = 0`), `E5` (trained with Algorithm 1, `p_0 = 0.20`) achieves a
+**`0.9957` Speech AP** and **`0.9522` `mAP`** (`+6.31` points higher `mAP` and
+`+4.58` points higher Speech AP than `E3` trained without Algorithm 1), matching
+the dedicated Standard VAD Speech AP (`0.9968`) while using a single unified
+model for both enrolled and enrollment-less operation.
+
+---
+
+## Reproducing the Full Multilingual LibriSpeech Benchmark Suite
+
+### 1. Download 8-Language Multilingual LibriSpeech (MLS) Subset
+
+```bash
+python3 scripts/download_mls.py \
+  --output_dir ~/Data/multilingual_librispeech \
+  --speakers_per_lang 18 \
+  --utterances_per_speaker 10
+```
+
+### 2. Train, Export & Evaluate All 13 Personal VAD 1.0 & 2.0 Models
+
+```bash
+python3 scripts/run_benchmarks.py \
+  --mls_dir ~/Data/multilingual_librispeech \
+  --output_dir ~/Data/pvad_MLS_Watch_models \
+  --epochs_v1 30 \
+  --epochs_v2 35
+```
+
+### 3. Package Pretrained Checkpoints & Upload to Hugging Face Hub
+
+```bash
+python3 scripts/package_and_upload_hf.py \
+  --benchmark_dir ~/Data/pvad_MLS_Watch_models \
+  --mls_dir ~/Data/multilingual_librispeech \
+  --pretrained_dir pretrained_models \
+  --hf_space_dir hf_space \
+  --upload
+```
+
+---
+
+## Running Unit Tests, Linting & Type Checking
+
+To run the full test suite locally:
+
+```bash
+flake8 --indent-size 2 --max-line-length 80 .
+pytype personal_vad/
+bash run_tests.sh
+```
+
+---
+
+## Citations
+
+If you use this library or the pretrained models in your research, please cite
+the Personal VAD papers:
+
+- **Personal VAD: Speaker-Conditioned Voice Activity Detection** ([arXiv:1908.04284](https://arxiv.org/pdf/1908.04284))
+  ```bibtex
+  @inproceedings{ding2020personal,
+    title={Personal {VAD}: Speaker-Conditioned Voice Activity Detection},
+    author={Ding, Shaojin and Wang, Quan and Chang, Shuo-yiin and Wan, Li and Moreno, Ignacio Lopez},
+    booktitle={Proc. Odyssey The Speaker and Language Recognition Workshop},
+    pages={433--439},
+    year={2020}
+  }
+  ```
+
+- **Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition** ([arXiv:2204.03793](https://arxiv.org/pdf/2204.03793))
+  ```bibtex
+  @inproceedings{ding2022personal,
+    title={Personal {VAD} 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition},
+    author={Ding, Shaojin and Rikhye, Rajeev and Liang, Qiao and He, Yanzhang and Wang, Quan and Narayanan, Arun and O'Malley, Tom and McGraw, Ian},
+    booktitle={Proc. Interspeech},
+    pages={3744--3748},
+    year={2022}
+  }
+  ```