ssl-kaldi-feats 0.1.0__tar.gz

ssl_kaldi_feats-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright {yyyy} {name of copyright owner}
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

ssl_kaldi_feats-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+# The compatibility fixture must ship: it is what lets a user verify that their
+# transformers version still produces the same features. Without it,
+# `pytest -m needs_model` silently skips.
+include LICENSE
+recursive-include tests *.py *.npz *.npy *.json

ssl_kaldi_feats-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.1
+Name: ssl-kaldi-feats
+Version: 0.1.0
+Summary: Kaldi ark/scp features from any HuggingFace speech encoder, without a Kaldi install
+Author: Ibrahim Almajai
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/ialmajai/ssl-kaldi
+Project-URL: Source, https://github.com/ialmajai/ssl-kaldi/tree/main/packages/ssl-kaldi-feats
+Keywords: speech,kaldi,self-supervised,asr,features,wav2vec2,hubert,whisper
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy<2.0,>=1.22
+Requires-Dist: torch>=2.0
+Requires-Dist: transformers<5,>=4.46.0
+Requires-Dist: kaldiio>=2.18.0
+Provides-Extra: audio
+Requires-Dist: soundfile>=0.12; extra == "audio"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+
+# ssl-kaldi-feats
+
+Kaldi `ark`/`scp` features from any HuggingFace speech encoder. **No Kaldi
+installation required** - archives are written in pure Python, so the output is
+consumable by Kaldi, ESPnet, or anything else that reads the format, on a
+machine that has never compiled Kaldi.
+
+```bash
+pip install ssl-kaldi-feats
+```
+
+## Use
+
+```bash
+# single layer, from a Kaldi wav.scp (the "cmd |" form works too)
+ssl-kaldi-feats --model facebook/hubert-base-ls960 --layer 9 \
+    --wav-scp data/train/wav.scp --output-dir feats/train
+
+# or from a plain list of audio files
+ssl-kaldi-feats --model microsoft/wavlm-large --layer 16 \
+    --file-list files.txt --output-dir feats/train
+```
+
+Output is a directory Kaldi will accept as-is:
+
+```
+feats/train/feats.scp  feats.ark  utt2num_frames  utt2dur  frame_shift
+```
+
+As a library:
+
+```python
+from ssl_kaldi_feats import SSLExtractor
+
+ex = SSLExtractor("facebook/mms-300m", layers=14)
+feats = ex.extract_one(waveform)          # (frames, 1024) float32, 20 ms shift
+```
+
+## Sweeping layers costs one forward pass, not one per layer
+
+```bash
+ssl-kaldi-feats --model facebook/mms-300m --layers 10,12,14,16,18 \
+    --wav-scp data/train/wav.scp --output-dir feats/sweep
+# -> feats/sweep/layer10/ layer12/ layer14/ layer16/ layer18/
+```
+
+Probing those five layers separately costs 10+12+14+16+18 = 70
+layer-evaluations. Together it costs 18, plus one model load and one pass over
+the audio instead of five. Measured at **3.9x faster** on a 3696-utterance set.
+
+## The layer_norm gotcha
+
+Taking layer *k* by truncating the encoder and reading the last hidden state is
+**not** the same as indexing `hidden_states[k]` on the full model.
+`Wav2Vec2EncoderStableLayerNorm` (the large models: mms-300m, mms-1b,
+wavlm-large, wav2vec2-large, xls-r) and `WhisperEncoder` apply
+`encoder.layer_norm` *after* the layer loop, so truncation moves a norm trained
+for the top layer onto layer *k*. Base-sized models (hubert-base, wavlm-base,
+mHuBERT-147) norm before the loop and are unaffected.
+
+Verified exactly on mms-300m: cosine against `layer_norm(L_k)` is 1.000000 at
+layers 8, 14 and 20, while cosine against the raw layer is 0.67 to 0.87.
+
+```bash
+--final-layer-norm keep    # default: layer_norm(layer_k)
+--final-layer-norm strip   # hidden_states[k], comparable with the literature
+```
+
+`keep` is the default because it measures better as a front end: on TIMIT with
+mms-300m layer 14 it wins at every HMM-GMM stage by 0.67 to 2.21 PER. The norm
+acts as a free per-frame normalisation, which a per-speaker CMVN cannot
+substitute for. It also **changes which layer wins** (14 with the norm, 16
+without), so a layer study run through a misplaced norm is not measuring the
+layer alone. Use `strip` when you need numbers comparable with published layer
+studies.
+
+## Notes
+
+- **Frame rate is measured, not assumed.** A short probe pass at load time
+  derives seconds-per-frame and writes it to `frame_shift`. Every encoder
+  supported here is 20 ms, but hardcoding that would be silently wrong for
+  anything else, and a wrong `frame_shift` misaligns everything downstream
+  without erroring. Measured rates within 5% of 10, 20 or 40 ms snap to the
+  clean value (models typically emit one frame fewer than duration/shift, from
+  the convolutional receptive field); anything else is reported with a warning.
+  Override with `SSLExtractor(..., frame_shift=0.04)` if you disagree.
+- **Whisper** is handled: the decoder is dropped, and because its feature
+  extractor pads every utterance to 30 s the frames beyond the real audio are
+  discarded. Utterances over 30 s raise rather than being silently truncated.
+- **Multi-channel input is rejected**, not mangled. A stereo `(n, 2)` array
+  survives `np.squeeze` and the HF feature extractor then reads it as a *batch
+  of two*, returning wrong features with no error.
+- **Failures abort by default.** A tolerated failure silently shrinks the
+  output, which later looks like a smaller test set rather than an error. Raise
+  `--max-failures` deliberately.
+- **`soundfile`** is only needed for non-wav or non-PCM16 audio:
+  `pip install ssl-kaldi-feats[audio]`.
+
+## Where this comes from
+
+Extracted from [ssl-kaldi](https://github.com/ialmajai/ssl-kaldi), a set of
+Kaldi recipes that put frozen self-supervised features in front of a classic
+HMM-GMM and LF-MMI back end. Measured results, including which choices turned
+out not to matter, are in that repo's
+[FINDINGS.md](https://github.com/ialmajai/ssl-kaldi/blob/main/FINDINGS.md).
+
+Apache-2.0.

ssl_kaldi_feats-0.1.0/README.md

@@ -0,0 +1,107 @@
+# ssl-kaldi-feats
+
+Kaldi `ark`/`scp` features from any HuggingFace speech encoder. **No Kaldi
+installation required** - archives are written in pure Python, so the output is
+consumable by Kaldi, ESPnet, or anything else that reads the format, on a
+machine that has never compiled Kaldi.
+
+```bash
+pip install ssl-kaldi-feats
+```
+
+## Use
+
+```bash
+# single layer, from a Kaldi wav.scp (the "cmd |" form works too)
+ssl-kaldi-feats --model facebook/hubert-base-ls960 --layer 9 \
+    --wav-scp data/train/wav.scp --output-dir feats/train
+
+# or from a plain list of audio files
+ssl-kaldi-feats --model microsoft/wavlm-large --layer 16 \
+    --file-list files.txt --output-dir feats/train
+```
+
+Output is a directory Kaldi will accept as-is:
+
+```
+feats/train/feats.scp  feats.ark  utt2num_frames  utt2dur  frame_shift
+```
+
+As a library:
+
+```python
+from ssl_kaldi_feats import SSLExtractor
+
+ex = SSLExtractor("facebook/mms-300m", layers=14)
+feats = ex.extract_one(waveform)          # (frames, 1024) float32, 20 ms shift
+```
+
+## Sweeping layers costs one forward pass, not one per layer
+
+```bash
+ssl-kaldi-feats --model facebook/mms-300m --layers 10,12,14,16,18 \
+    --wav-scp data/train/wav.scp --output-dir feats/sweep
+# -> feats/sweep/layer10/ layer12/ layer14/ layer16/ layer18/
+```
+
+Probing those five layers separately costs 10+12+14+16+18 = 70
+layer-evaluations. Together it costs 18, plus one model load and one pass over
+the audio instead of five. Measured at **3.9x faster** on a 3696-utterance set.
+
+## The layer_norm gotcha
+
+Taking layer *k* by truncating the encoder and reading the last hidden state is
+**not** the same as indexing `hidden_states[k]` on the full model.
+`Wav2Vec2EncoderStableLayerNorm` (the large models: mms-300m, mms-1b,
+wavlm-large, wav2vec2-large, xls-r) and `WhisperEncoder` apply
+`encoder.layer_norm` *after* the layer loop, so truncation moves a norm trained
+for the top layer onto layer *k*. Base-sized models (hubert-base, wavlm-base,
+mHuBERT-147) norm before the loop and are unaffected.
+
+Verified exactly on mms-300m: cosine against `layer_norm(L_k)` is 1.000000 at
+layers 8, 14 and 20, while cosine against the raw layer is 0.67 to 0.87.
+
+```bash
+--final-layer-norm keep    # default: layer_norm(layer_k)
+--final-layer-norm strip   # hidden_states[k], comparable with the literature
+```
+
+`keep` is the default because it measures better as a front end: on TIMIT with
+mms-300m layer 14 it wins at every HMM-GMM stage by 0.67 to 2.21 PER. The norm
+acts as a free per-frame normalisation, which a per-speaker CMVN cannot
+substitute for. It also **changes which layer wins** (14 with the norm, 16
+without), so a layer study run through a misplaced norm is not measuring the
+layer alone. Use `strip` when you need numbers comparable with published layer
+studies.
+
+## Notes
+
+- **Frame rate is measured, not assumed.** A short probe pass at load time
+  derives seconds-per-frame and writes it to `frame_shift`. Every encoder
+  supported here is 20 ms, but hardcoding that would be silently wrong for
+  anything else, and a wrong `frame_shift` misaligns everything downstream
+  without erroring. Measured rates within 5% of 10, 20 or 40 ms snap to the
+  clean value (models typically emit one frame fewer than duration/shift, from
+  the convolutional receptive field); anything else is reported with a warning.
+  Override with `SSLExtractor(..., frame_shift=0.04)` if you disagree.
+- **Whisper** is handled: the decoder is dropped, and because its feature
+  extractor pads every utterance to 30 s the frames beyond the real audio are
+  discarded. Utterances over 30 s raise rather than being silently truncated.
+- **Multi-channel input is rejected**, not mangled. A stereo `(n, 2)` array
+  survives `np.squeeze` and the HF feature extractor then reads it as a *batch
+  of two*, returning wrong features with no error.
+- **Failures abort by default.** A tolerated failure silently shrinks the
+  output, which later looks like a smaller test set rather than an error. Raise
+  `--max-failures` deliberately.
+- **`soundfile`** is only needed for non-wav or non-PCM16 audio:
+  `pip install ssl-kaldi-feats[audio]`.
+
+## Where this comes from
+
+Extracted from [ssl-kaldi](https://github.com/ialmajai/ssl-kaldi), a set of
+Kaldi recipes that put frozen self-supervised features in front of a classic
+HMM-GMM and LF-MMI back end. Measured results, including which choices turned
+out not to matter, are in that repo's
+[FINDINGS.md](https://github.com/ialmajai/ssl-kaldi/blob/main/FINDINGS.md).
+
+Apache-2.0.

ssl_kaldi_feats-0.1.0/pyproject.toml

@@ -0,0 +1,61 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ssl-kaldi-feats"
+version = "0.1.0"
+description = "Kaldi ark/scp features from any HuggingFace speech encoder, without a Kaldi install"
+readme = "README.md"
+# 3.8 because that is the interpreter the ssl-kaldi recipes run on.
+# The code uses no 3.9+ syntax. Bump to >=3.9 (and switch the CLI
+# boolean flags to argparse.BooleanOptionalAction) when that env moves
+# to the 3.11 the top-level README already recommends.
+requires-python = ">=3.8"
+license = {text = "Apache-2.0"}
+authors = [{name = "Ibrahim Almajai"}]
+keywords = ["speech", "kaldi", "self-supervised", "asr", "features", "wav2vec2", "hubert", "whisper"]
+classifiers = [
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: Apache Software License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Scientific/Engineering",
+]
+dependencies = [
+  "numpy>=1.22,<2.0",
+  "torch>=2.0",
+  # Upper bound is deliberate. This package truncates model.encoder.layers,
+  # reads config.do_stable_layer_norm and swaps encoder.layer_norm for
+  # Identity: internals, not public API. A major release can change any of
+  # them, and the failure mode is silently different features rather than an
+  # exception. 5.x is untested, not known broken. To lift the bound, run
+  #   pytest -m needs_model tests/
+  # against 5.x and relax it if the fixture still reproduces.
+  "transformers>=4.46.0,<5",
+  "kaldiio>=2.18.0",
+]
+
+[project.optional-dependencies]
+audio = ["soundfile>=0.12"]
+dev = ["pytest>=7"]
+
+[project.urls]
+Homepage = "https://github.com/ialmajai/ssl-kaldi"
+Source = "https://github.com/ialmajai/ssl-kaldi/tree/main/packages/ssl-kaldi-feats"
+
+[project.scripts]
+ssl-kaldi-feats = "ssl_kaldi_feats.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+markers = [
+  "needs_model: downloads a model from HuggingFace; opt in with -m needs_model",
+]
+addopts = "-m 'not needs_model'"

ssl_kaldi_feats-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ssl_kaldi_feats-0.1.0/src/ssl_kaldi_feats/__init__.py

@@ -0,0 +1,18 @@
+"""Kaldi ark/scp features from any HuggingFace speech encoder."""
+from .extractor import (
+    EXPECTED_SAMPLE_RATE,
+    WHISPER_SAMPLES_PER_FRAME,
+    WHISPER_WINDOW_SECONDS,
+    SSLExtractor,
+    preprocess_waveform,
+)
+
+__version__ = "0.1.0"
+__all__ = [
+    "SSLExtractor",
+    "preprocess_waveform",
+    "EXPECTED_SAMPLE_RATE",
+    "WHISPER_WINDOW_SECONDS",
+    "WHISPER_SAMPLES_PER_FRAME",
+    "__version__",
+]

ssl_kaldi_feats-0.1.0/src/ssl_kaldi_feats/cli.py

@@ -0,0 +1,128 @@
+"""Command line entry point."""
+import argparse
+import logging
+import os
+import sys
+
+from kaldiio import WriteHelper
+
+from .extractor import SSLExtractor
+from .io import read_file_list, read_wav_scp, write_side_files
+
+logger = logging.getLogger("ssl_kaldi_feats")
+logger.setLevel(logging.INFO)
+
+
+def build_parser():
+    p = argparse.ArgumentParser(
+        prog="ssl-kaldi-feats",
+        description="Kaldi ark/scp features from any HuggingFace speech encoder. "
+                    "No Kaldi installation required.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    p.add_argument("--model", required=True,
+                   help="HuggingFace model id, e.g. facebook/hubert-base-ls960")
+    g = p.add_mutually_exclusive_group(required=True)
+    g.add_argument("--layer", type=int, help="single layer to extract, 1-indexed")
+    g.add_argument("--layers",
+                   help="comma-separated layers from ONE forward pass, e.g. "
+                        "'10,12,14'. Costs one pass to the deepest layer "
+                        "instead of one pass each")
+    src = p.add_mutually_exclusive_group(required=True)
+    src.add_argument("--wav-scp", help="Kaldi wav.scp, including the 'cmd |' form")
+    src.add_argument("--file-list", help="text file of audio paths, one per line")
+    p.add_argument("--output-dir", required=True,
+                   help="written as feats.scp/feats.ark plus utt2num_frames, "
+                        "utt2dur and frame_shift. With --layers, one "
+                        "subdirectory per layer")
+    p.add_argument("--final-layer-norm", choices=["keep", "strip"], default="keep",
+                   help="'keep' returns layer_norm(layer_k); 'strip' returns "
+                        "hidden_states[k], which is what published layer studies "
+                        "mean. Only affects models that norm after the layer "
+                        "loop (the large ones, and Whisper)")
+    p.add_argument("--device", default=None, help="torch device (default: cuda if available)")
+    # argparse.BooleanOptionalAction would be tidier but is 3.9+, and this
+    # package supports 3.8. Both halves carry help text so neither reads as an
+    # accident.
+    p.add_argument("--compress", dest="compress", action="store_true",
+                   default=True,
+                   help="write compressed archives, Kaldi kSpeechFeature, the "
+                        "same codec as copy-feats --compress=true (the default)")
+    p.add_argument("--no-compress", dest="compress", action="store_false",
+                   help="write uncompressed archives. Compression is lossy, so "
+                        "use this when comparing archives bit-for-bit")
+    p.add_argument("--max-failures", type=int, default=0,
+                   help="how many utterances may fail before aborting. A "
+                        "tolerated failure silently shrinks the output")
+    p.add_argument("--quiet", action="store_true")
+    return p
+
+
+def main(argv=None):
+    args = build_parser().parse_args(argv)
+    logging.basicConfig(
+        level=logging.WARNING if args.quiet else logging.INFO,
+        format="%(asctime)s %(levelname)s %(message)s",
+    )
+
+    layers = ([int(x) for x in args.layers.split(",")] if args.layers
+              else args.layer)
+    extractor = SSLExtractor(args.model, layers, args.final_layer_norm, args.device)
+    multi = len(extractor.layers) > 1
+
+    dirs = {}
+    for layer in extractor.layers:
+        d = os.path.join(args.output_dir, f"layer{layer}") if multi else args.output_dir
+        os.makedirs(d, exist_ok=True)
+        dirs[layer] = d
+
+    reader = (read_wav_scp(args.wav_scp) if args.wav_scp
+              else read_file_list(args.file_list))
+
+    comp = 2 if args.compress else None
+    writers = {}
+    for layer, d in dirs.items():
+        spec = f"ark,scp:{os.path.abspath(d)}/feats.ark,{os.path.abspath(d)}/feats.scp"
+        writers[layer] = WriteHelper(spec, compression_method=comp)
+
+    utt2num_frames, utt2dur, failed = {}, {}, []
+    processed = 0
+    try:
+        for utt_id, wav, sr in reader:
+            try:
+                feats = extractor.extract(wav, sr)
+                for layer, w in writers.items():
+                    w(utt_id, feats[layer])
+                first = feats[extractor.layers[0]]
+                utt2num_frames[utt_id] = len(first)
+                utt2dur[utt_id] = len(wav) / float(sr)
+                processed += 1
+                if processed % 100 == 0:
+                    logger.info("processed %d utterances", processed)
+            except Exception as e:  # noqa: BLE001
+                logger.error("failed on %s: %s", utt_id, e)
+                failed.append(utt_id)
+                if len(failed) > args.max_failures:
+                    raise SystemExit(
+                        f"aborting after {len(failed)} failure(s), "
+                        f"--max-failures is {args.max_failures}. "
+                        f"Failed: {' '.join(failed)}"
+                    )
+    finally:
+        for w in writers.values():
+            w.close()
+
+    if processed == 0:
+        raise SystemExit("no utterances were processed")
+
+    for layer, d in dirs.items():
+        write_side_files(d, utt2num_frames, utt2dur, extractor.frame_shift)
+
+    logger.info("wrote %d utterances to %s", processed, args.output_dir)
+    if failed:
+        logger.warning("tolerated %d failure(s): %s", len(failed), " ".join(failed))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())