audiosetfit 0.1.0__tar.gz

audiosetfit-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work, excluding
+          those notices that do not pertain to any part of the Derivative
+          Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and do
+          not modify the License. You may add Your own attribution notices
+          within Derivative Works that You distribute, alongside or as an
+          addendum to the NOTICE text from the Work, provided that such
+          additional attribution notices cannot be construed as modifying
+          the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 AudioSetFit contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

audiosetfit-0.1.0/PKG-INFO

@@ -0,0 +1,311 @@
+Metadata-Version: 2.4
+Name: audiosetfit
+Version: 0.1.0
+Summary: Efficient few-shot audio classification with contrastive fine-tuning (a SetFit for audio).
+Author: Ilja Baumann
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/iljab/audiosetfit
+Project-URL: Repository, https://github.com/iljab/audiosetfit
+Project-URL: Issues, https://github.com/iljab/audiosetfit/issues
+Keywords: audio,speech,few-shot,classification,contrastive-learning,setfit,clap,transformers
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Multimedia :: Sound/Audio
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.1
+Requires-Dist: transformers>=4.40
+Requires-Dist: datasets>=4.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: numpy>=1.24
+Requires-Dist: librosa>=0.10
+Requires-Dist: soundfile>=0.12
+Requires-Dist: torchcodec>=0.4
+Requires-Dist: tqdm>=4.65
+Requires-Dist: huggingface_hub>=0.23
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5; extra == "dev"
+Dynamic: license-file
+
+# AudioSetFit
+
+**Efficient few-shot audio classification with contrastive fine-tuning** - a [SetFit](https://github.com/huggingface/setfit) for audio.
+
+`audiosetfit` ports SetFit's prompt-free, few-shot recipe from text to audio. Instead of a
+`SentenceTransformer` body, it uses an **audio encoder** (CLAP by default) and trains in two
+phases:
+
+1. **Embedding fine-tuning (contrastive).** From a handful of labeled clips it builds
+  *positive* (same-class) and *negative* (different-class) pairs and fine-tunes the audio
+   body so same-class clips embed closer together. A few examples explode into hundreds of
+   informative pairs.
+2. **Classifier head.** A lightweight head (sklearn `LogisticRegression` by default, or a
+  differentiable torch head) is fit on the resulting embeddings.
+
+The contrastive trainer is **self-contained**, it does *not* depend on
+`sentence-transformers`. The pair-sampling and loss math are reimplemented to operate
+directly on audio embeddings, so any HF audio model can be plugged in as the body.
+
+The public API intentionally mirrors SetFit:
+
+```python
+from audiosetfit import AudioSetFitModel, Trainer, TrainingArguments, sample_dataset
+```
+
+## Installation
+
+```bash
+# from the repo root
+python3 -m venv .venv
+source .venv/bin/activate
+pip install --upgrade pip
+pip install -e .
+```
+
+This installs `torch`, `transformers`, `datasets`, `librosa`, `soundfile`, `torchcodec`,
+`scikit-learn`, etc. On Apple Silicon, PyTorch will use the **MPS** backend automatically;
+on NVIDIA GPUs it uses CUDA; otherwise CPU.
+
+> **FFmpeg required.** `datasets >= 4` decodes audio via `torchcodec`, which needs FFmpeg
+> (4–7) installed on your system. On macOS: `brew install ffmpeg`; on Debian/Ubuntu:
+> `sudo apt-get install ffmpeg`.
+
+## Examples
+
+The `examples/` scripts cover four small benchmarks across different audio domains. They share
+the same CLI flags (`--backbone`, `--classes`, `--num-samples`, `--epochs`, `--max-pairs`,
+`--no-embedding-finetuning`, `--differentiable-head`, `--num-workers`, ...).
+
+### ESC-50 (environmental sounds)
+
+[ESC-50](https://huggingface.co/datasets/ashraq/esc50) (2,000 clips, 50 classes) is the small
+default starting point. The example restricts to a few classes for a fast first run:
+
+```bash
+python examples/train_esc50.py                      # 5 classes, 8 shots, CLAP
+python examples/train_esc50.py --classes 10 --num-samples 16
+python examples/train_esc50.py --no-embedding-finetuning   # frozen-backbone baseline
+```
+
+### UrbanSound8K (urban sounds)
+
+[UrbanSound8K](https://huggingface.co/datasets/danavery/urbansound8K) (8,732 clips, 10 classes,
+10 folds) follows the dataset's fold protocol (folds 1-9 train, fold 10 test):
+
+```bash
+python examples/train_urbansound8k.py               # 5 classes, 8 shots, CLAP
+python examples/train_urbansound8k.py --classes 10 --num-samples 16
+```
+
+### CREMA-D (speech emotion)
+
+A **speech** task where self-supervised speech encoders shine:
+[CREMA-D](https://huggingface.co/datasets/confit/cremad-parquet) (7,442 clips, 91 actors,
+6 emotions). It defaults to `facebook/wav2vec2-base` and uses a *speaker-disjoint* train/test split:
+
+```bash
+python examples/train_cremad.py                         # wav2vec2-base
+python examples/train_cremad.py --backbone microsoft/wavlm-base
+python examples/train_cremad.py --backbone facebook/hubert-base-ls960
+python examples/train_cremad.py --backbone laion/clap-htsat-unfused   # compare vs CLAP
+```
+
+### MSWC (keyword spotting)
+
+A **keyword-spotting** example (SUPERB KS-style) on
+[MSWC](https://huggingface.co/datasets/confit/mswc-parquet) (Multilingual Spoken Words Corpus).
+Each clip is a single spoken word; this is a lexical/phonetic task, so it also defaults to a
+speech encoder and uses the dataset's predefined train/test splits:
+
+```bash
+python examples/train_mswc_keywords.py                  # 10 keywords, wav2vec2-base
+python examples/train_mswc_keywords.py --classes 5 --num-samples 16
+python examples/train_mswc_keywords.py --language spanish
+python examples/train_mswc_keywords.py --backbone laion/clap-htsat-unfused   # compare vs CLAP
+```
+
+### Benchmarking (multi-backbone / multi-seed)
+
+`examples/benchmark.py` drives the training scripts above across a grid of backbones x seeds and
+prints a mean +/- std table, so backbone comparisons are reproducible instead of single noisy runs.
+It reuses each dataset's own split logic. `Trainer.evaluate` reports both accuracy and macro-F1,
+and `Trainer.classification_report(...)` adds per-class accuracy and a confusion matrix.
+
+```bash
+# CLAP vs wav2vec2 on CREMA-D over 3 seeds
+python examples/benchmark.py --dataset cremad \
+    --backbones laion/clap-htsat-unfused facebook/wav2vec2-base --seeds 41 42 43
+
+# Keyword spotting with speech encoders, write a CSV of every run
+python examples/benchmark.py --dataset mswc \
+    --backbones facebook/wav2vec2-base microsoft/wavlm-base --seeds 42 43 --csv results.csv
+
+# Forward extra flags to the training script after a literal `--`
+python examples/benchmark.py --dataset esc50 --seeds 41 42 43 -- --no-embedding-finetuning
+```
+
+### Minimal end-to-end usage
+
+```python
+from datasets import Audio, load_dataset
+from audiosetfit import AudioSetFitModel, Trainer, TrainingArguments, sample_dataset
+
+ds = load_dataset("ashraq/esc50", split="train").cast_column("audio", Audio(sampling_rate=48000))
+labels = sorted(set(ds["category"]))
+train_ds = sample_dataset(ds, label_column="category", num_samples=8)
+
+model = AudioSetFitModel.from_pretrained("laion/clap-htsat-unfused", labels=labels)
+trainer = Trainer(
+    model=model,
+    args=TrainingArguments(embedding_num_epochs=1, max_pairs=256),
+    train_dataset=train_ds,
+    column_mapping={"category": "label"},  # the 'audio' column already matches
+)
+trainer.train()
+
+preds = model.predict(["dog_bark.wav", "rain.wav"])     # file paths, arrays, or Audio dicts
+model.save_pretrained("my-esc50-model")
+reloaded = AudioSetFitModel.from_pretrained("my-esc50-model")
+```
+
+### Inputs accepted everywhere
+
+`predict` / `encode` / datasets accept any mix of:
+
+- file paths (`"clip.wav"`),
+- raw waveforms (`np.ndarray`, assumed at the backbone's sample rate),
+- Hugging Face `datasets` Audio dicts (`{"array", "sampling_rate", "path"}`).
+
+Everything is resampled to the backbone's expected rate (CLAP = 48 kHz).
+
+
+## Project layout
+
+```
+src/audiosetfit/
+├── encoders.py      # AudioEncoder base + CLAP/AST/wav2vec2-family/Whisper + build_encoder()
+├── modeling.py      # AudioSetFitModel, AudioSetFitHead, save/from_pretrained
+├── sampler.py       # ContrastiveDataset (same/different-label pair generation)
+├── losses.py        # CosineSimilarityLoss, ContrastiveLoss (on embedding tensors)
+├── data.py          # load_audio (resampling), sample_dataset
+├── training_args.py # TrainingArguments (both phases)
+└── trainer.py       # self-contained two-phase Trainer
+examples/train_esc50.py
+examples/train_urbansound8k.py
+examples/train_cremad.py
+examples/train_mswc_keywords.py
+examples/benchmark.py            # multi-backbone / multi-seed harness
+```
+
+## Key training arguments
+
+
+| Argument                  | Default          | Purpose                                                  |
+| ------------------------- | ---------------- | -------------------------------------------------------- |
+| `train_embeddings`        | `True`           | Run phase 1. Set `False` for a frozen-backbone baseline. |
+| `embedding_num_epochs`    | `1`              | Epochs over contrastive pairs.                           |
+| `embedding_batch_size`    | `16`             | Pair batch size (lower it if you hit memory limits).     |
+| `body_learning_rate`      | `2e-5`           | LR for the audio body.                                   |
+| `loss`                    | `"cosine"`       | `"cosine"` or `"contrastive"` (or pass an `nn.Module`).  |
+| `sampling_strategy`       | `"oversampling"` | `"unique"` / `"oversampling"` / `"undersampling"`.       |
+| `max_steps` / `max_pairs` | `-1`             | Cap phase-1 work (handy on CPU/laptops).                 |
+| `classifier_num_epochs`   | `25`             | Torch-head epochs (ignored for sklearn head).            |
+
+
+## Backbones
+
+Pick a backbone by passing its Hugging Face id to `from_pretrained` (or `--backbone` in the
+example). The right `AudioEncoder` is selected automatically from the model's `model_type`.
+
+```python
+AudioSetFitModel.from_pretrained("laion/clap-htsat-unfused")                  # CLAP (default)
+AudioSetFitModel.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")   # AST
+AudioSetFitModel.from_pretrained("facebook/wav2vec2-base")                    # wav2vec2
+AudioSetFitModel.from_pretrained("facebook/hubert-base-ls960")                # HuBERT
+AudioSetFitModel.from_pretrained("microsoft/wavlm-base-plus")                 # WavLM
+AudioSetFitModel.from_pretrained("openai/whisper-base")                       # Whisper encoder
+```
+
+```bash
+python examples/train_esc50.py --backbone facebook/wav2vec2-base
+python examples/train_esc50.py --backbone MIT/ast-finetuned-audioset-10-10-0.4593
+```
+
+Audio is resampled to each backbone's expected rate automatically (CLAP 48 kHz, others
+16 kHz), so the same dataset works across all of them.
+
+
+| Backbone                      | `model_type` (built-in)                                                                                  | Embedding         | Pooling               | Best for                                  |
+| ----------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------- | --------------------- | ----------------------------------------- |
+| **CLAP** (default)            | `clap`                                                                                                   | 512-d, normalized | projection head       | General sound events, environmental audio |
+| **AST**                       | `audio-spectrogram-transformer`                                                                          | 768-d             | CLS+dist pooled       | AudioSet-style tagging                    |
+| **wav2vec2 / HuBERT / WavLM** | `wav2vec2` / `hubert` / `wavlm` (+ `unispeech`, `unispeech-sat`, `data2vec-audio`, `wav2vec2-conformer`) | hidden_size       | masked mean over time | Speech (commands, speaker, emotion)       |
+| **Whisper encoder**           | `whisper`                                                                                                | d_model           | mean over frames      | Robust speech in noise                    |
+
+
+### Adding another backbone
+
+The encoder is the only modality-specific piece. Subclass `AudioEncoder`, implement
+`prepare` (waveforms → model inputs) and `forward_features` (inputs → `[B, D]`), set
+`target_sr` / `embedding_dim`, then register it:
+
+```python
+from audiosetfit import encoders
+
+class MyEncoder(encoders.AudioEncoder):
+    def __init__(self, model_id, device=None):
+        super().__init__()
+        self.model_id = model_id
+        ...                       # load backbone + feature extractor
+        self.target_sr = 16000
+        self.embedding_dim = ...  # output dim
+        self.to(encoders._resolve_device(device))
+    def prepare(self, waveforms): ...
+    def forward_features(self, inputs): ...
+    def save(self, save_directory): ...
+
+encoders._ENCODER_REGISTRY["my_model_type"] = MyEncoder
+```
+
+## Roadmap / next steps
+
+**Benchmarking & evaluation**
+- [x] Reproducible multi-backbone / multi-seed benchmark harness (`examples/benchmark.py`) with mean ± std tables.
+- [x] Richer metrics in `Trainer.evaluate` (accuracy + macro-F1); per-class accuracy and confusion matrix via `Trainer.classification_report`.
+- [ ] Published results table (CLAP vs wav2vec2 vs WavLM across all example datasets).
+
+**Training method**
+- [ ] `SupConLoss` / InfoNCE with in-batch negatives + group-by-label batch sampler (so larger batches add real negatives, as in SetFit).
+- [ ] Audio augmentation for the few-shot regime (SpecAugment, additive noise, gain, time-shift, random crop).
+- [ ] Embedding cache for the frozen-backbone path (skip re-encoding clips across runs/sweeps).
+- [ ] Knowledge distillation from a large unlabeled audio pool (teacher → student).
+
+**Models & inputs**
+- [ ] Long-clip handling: windowing/chunking → encode → pool/vote.
+- [ ] Multilabel audio tagging end-to-end example (sampler already supports multilabel pairs).
+- [ ] (Optional) BEATs / OpenBEATs backbone (strongest general-purpose SSL embeddings).
+
+**Productionization**
+- [ ] ONNX / `torch.compile` export for fast CPU inference.
+- [ ] Hub `push_to_hub` with an auto-generated model card (incl. the eval table).
+- [ ] Smoke-test suite + CI using small real models (e.g. `openai/whisper-tiny`).
+
+## Acknowledgements
+
+Architecture and training recipe adapted from
+[Hugging Face SetFit](https://github.com/huggingface/setfit)
+(Tunstall et al., *Efficient Few-Shot Learning Without Prompts*, 2022).