sample-indexer 0.1.0__tar.gz

sample_indexer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Adrián Cancio and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sample_indexer-0.1.0/PKG-INFO

@@ -0,0 +1,233 @@
+Metadata-Version: 2.4
+Name: sample-indexer
+Version: 0.1.0
+Summary: SQLite index of a sample and MIDI library with incremental reconciliation
+Author: Adrián Cancio
+License-Expression: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Dynamic: license-file
+
+# sample-indexer
+
+A SQLite index and technical analysis engine for sample and MIDI libraries. Built on Python standard library, system `ffprobe`/`ffmpeg` binaries, and `numpy` for digital signal processing (DSP).
+
+The sample library is strictly **read-only**: files are opened `O_RDONLY` and never modified, moved, or deleted.
+
+---
+
+## Requirements
+
+- **Python**: `>= 3.12`
+- **System tools**: `ffmpeg` and `ffprobe` must be available in `PATH`.
+  - Fedora: `sudo dnf install ffmpeg-free` (or RPM Fusion `ffmpeg`)
+  - Ubuntu / Debian: `sudo apt install ffmpeg`
+  - Arch Linux: `sudo pacman -S ffmpeg`
+- **Python dependencies**: `numpy` (installed automatically via `pip`).
+
+---
+
+## Installation
+
+Install directly from the source repository:
+
+```bash
+pip install .
+```
+
+Or for development (editable mode):
+
+```bash
+pip install -e .
+```
+
+Once installed, the CLI tool is available as `sample-indexer` (or via `python3 -m sample_indexer`).
+
+---
+
+## Quick start
+
+```bash
+# Initialize or migrate database
+sample-indexer init
+
+# Scan library: compute SHA-256 identity and track paths
+sample-indexer scan --root /path/to/samples
+
+# Probe audio containers with ffprobe and extract name/folder labels
+sample-indexer metadata --workers 4
+
+# Run full decode & low-end DSP (sub-bass, pitch f0, decay) on kick/bass candidates
+sample-indexer lowend --workers 2
+
+# Run full decode & tempo/meter/key DSP on loop candidates
+sample-indexer loops --workers 2
+
+# General streaming timbre, true peak, and integrated loudness (LUFS)
+sample-indexer audio --workers 2
+
+# Parse Standard MIDI Files (SMF/RMID) into musical note metrics
+sample-indexer midi
+
+# Check coverage, health, and library statistics
+sample-indexer stats
+sample-indexer coverage
+sample-indexer midi-coverage
+
+# Query samples by technical criteria
+sample-indexer query --folder "Samples/Kicks" --ext wav --limit 20
+
+# Create a safe, online SQLite backup
+sample-indexer backup index.backup.sqlite
+```
+
+Global option `--db PATH` (placed before the subcommand) or the environment variable `SAMPLE_INDEXER_DB` selects the database location (default: `data/index.sqlite`).
+`SAMPLE_INDEXER_ROOT` or `--root PATH` configures the library root.
+
+Exit codes:
+- `0`: success
+- `1`: differences detected / file or probe errors
+- `2`: usage error or database error
+- `130`: interrupted (SIGINT / Ctrl-C)
+
+---
+
+## Analysis pipeline
+
+| Stage | Subcommand | Reads | Writes |
+|---|---|---|---|
+| **Identity** | `scan` | File content (SHA-256), `stat` | `resource` (sha256, size, ext), `path` (rel_path, mtime) |
+| **Probe** | `metadata` | Header via single `ffprobe` per audio file | duration, sample rate, channels, bit depth, codec |
+| **Labels** | `metadata` | File path and name only | kind, category, bpm_name, key_name, label evidence |
+| **Low-End** | `lowend` | Full decode via `ffmpeg -xerror` | `valid`/`valid_error`, peak, RMS, crest factor, bands, sub peak Hz, f0 pitch Hz, decay ms, stereo correlation |
+| **Loops** | `loops` | Full decode via `ffmpeg -xerror` | `bpm_est`, `beats`, `bars_est`, `key_est`, confidence metrics, chroma analysis |
+| **Audio** | `audio` | Complete streaming PCM | Integrated LUFS, true peak dBTP, spectral centroid, rolloff, flatness, structured descriptors |
+| **MIDI** | `midi` | SMF / RMID events (stdlib parser) | Note counts, pitch bounds, note density, tempo, bars |
+
+### Identity and reconciliation (`scan`)
+- Identity is determined by the content SHA-256 (`resource`), while file paths are recorded as attributes (`path`).
+- Paths with unchanged `(rel_path, size, mtime_ns)` are not re-hashed.
+- Relocated or renamed files are recognized by matching their SHA-256 with vanished records.
+- Deleted or missing files are marked with `missing=1`; database records are never hard-deleted.
+
+### Container probing and heuristics (`metadata`)
+- Fast, non-destructive header inspection using `ffprobe` captures duration, sample rate, bit depth, and codec.
+- Heuristic classification extracts provisional `kind` (`one_shot`, `loop`, `midi`, `unknown`), `category`, and filename BPM/key hints.
+- Audio validity is left `NULL` until a stage actually performs a conclusive full decode.
+
+### Low-End & Kick analysis (`lowend`)
+- Focuses on kick drums, 808s, and bass candidates.
+- Measures fundamental pitch ($f_0$) via time-domain autocorrelation, sub-bass peak frequency via high-resolution zero-padded FFT, decay time ($T_{60}$ equivalent), and low-frequency stereo correlation ($< 150$ Hz).
+
+### Loop & Tempo analysis (`loops`)
+- Targets loop candidates (folder conventions or loop labels).
+- Measures tempo via onset spectral flux autocorrelation across candidate BPM ranges.
+- Estimates musical key using 12-bin chroma profiling correlated against Krumhansl-Schmuckler key profiles.
+
+### Comprehensive timbre & loudness (`audio`)
+- Streams full audio signals through chunked spectral analysis.
+- Computes integrated loudness (LUFS) and true peak (dBTP) using EBU R128 metering standards.
+- Evaluates spectral centroid, 85% spectral rolloff, and spectral flatness.
+
+### MIDI parsing (`midi`)
+- Pure Python Standard MIDI File (SMF format 0/1/2) and RMID parser without external dependencies.
+- Calculates total notes, pitch range (`pitch_min`, `pitch_max`), note density (notes/beat), bar counts, and tempo events.
+
+---
+
+## Querying the index
+
+### CLI examples
+
+```bash
+# 1. Kick one-shots with sub frequency peak in 45-65 Hz range
+sample-indexer query --folder "Samples/Drums - One Shots/Kicks" --sub-peak-min 45 --sub-peak-max 65 --limit 10
+
+# 2. Fast loops (>= 140 BPM) with measured tempo confidence >= 0.20
+sample-indexer query --bpm-min 140 --bpm-confidence-min 0.20 --limit 10
+
+# 3. Stereo loops with high mono compatibility in the low end (stereo_corr_low >= 0.85)
+sample-indexer query --stereo-corr-low-min 0.85 --limit 10
+
+# 4. Short percussion/hi-hat one-shots (duration <= 0.15s, sr=44100)
+sample-indexer query --duration-max 0.15 --sr 44100 --limit 10
+
+# 5. Long atmospheric textures (duration >= 10.0s)
+sample-indexer query --duration-min 10.0 --limit 10
+
+# 6. Dense MIDI files (density >= 6.0 notes/beat)
+sample-indexer query --ext mid --midi-density-min 6.0 --limit 10
+
+# 7. Long MIDI arrangements with 100+ note events
+sample-indexer query --ext mid --midi-notes-min 100 --limit 10
+
+# 8. Samples matching musical key 'F min' (from filename hint or measured chroma)
+sample-indexer query --key "F min" --limit 10
+
+# 9. Basslines with fundamental pitch f0 between 40 and 90 Hz
+sample-indexer query --f0-min 40 --f0-max 90 --limit 10
+
+# 10. Locate known corrupted or unparseable files
+sample-indexer query --only-errors
+
+# 11. Output structured JSON including technical audio descriptors
+sample-indexer query --folder "Samples/Loops/Bass Loops" --bpm-min 130 --bpm-max 140 --json --limit 5
+```
+
+### Python API
+
+`sample-indexer` can be imported directly into Python scripts:
+
+```python
+import sqlite3
+from sample_indexer import query
+
+conn = sqlite3.connect("data/index.sqlite")
+conn.row_factory = sqlite3.Row
+
+# Query kicks tuned between 48 Hz and 55 Hz
+rows = query(
+    conn,
+    folder="Samples/Drums - One Shots/Kicks",
+    sub_peak_min=48.0,
+    sub_peak_max=55.0,
+    limit=25,
+)
+
+for row in rows:
+    print(f"{row['rel_path']} - {row['sub_peak_hz']:.1f} Hz")
+```
+
+---
+
+## Safety and execution semantics
+
+- **Resumable execution**: Each stage tracks progress in `resource_stage`. Interrupted runs (Ctrl-C / `--max-seconds`) commit finished checkpoints and exit cleanly.
+- **Error handling**: Corrupted files or failed decodes record the failure status in the index and count as processed to prevent retry storms. Use `--retry-errors` to retry them or `--force` to reprocess.
+- **Database backups**: Schema migrations automatically create timestamped SQLite backups before applying changes. Manual backups can be created at any time using `sample-indexer backup <dest>`.
+- **Subprocess security**: All external calls to `ffmpeg` and `ffprobe` use explicit argument lists (`file:<path>`) without shell interpolation.
+
+---
+
+## Running tests
+
+The test suite runs against temporary file trees and scratch SQLite databases:
+
+```bash
+python3 -m unittest discover
+```
+
+---
+
+## License
+
+This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.

sample_indexer-0.1.0/README.md

@@ -0,0 +1,214 @@
+# sample-indexer
+
+A SQLite index and technical analysis engine for sample and MIDI libraries. Built on Python standard library, system `ffprobe`/`ffmpeg` binaries, and `numpy` for digital signal processing (DSP).
+
+The sample library is strictly **read-only**: files are opened `O_RDONLY` and never modified, moved, or deleted.
+
+---
+
+## Requirements
+
+- **Python**: `>= 3.12`
+- **System tools**: `ffmpeg` and `ffprobe` must be available in `PATH`.
+  - Fedora: `sudo dnf install ffmpeg-free` (or RPM Fusion `ffmpeg`)
+  - Ubuntu / Debian: `sudo apt install ffmpeg`
+  - Arch Linux: `sudo pacman -S ffmpeg`
+- **Python dependencies**: `numpy` (installed automatically via `pip`).
+
+---
+
+## Installation
+
+Install directly from the source repository:
+
+```bash
+pip install .
+```
+
+Or for development (editable mode):
+
+```bash
+pip install -e .
+```
+
+Once installed, the CLI tool is available as `sample-indexer` (or via `python3 -m sample_indexer`).
+
+---
+
+## Quick start
+
+```bash
+# Initialize or migrate database
+sample-indexer init
+
+# Scan library: compute SHA-256 identity and track paths
+sample-indexer scan --root /path/to/samples
+
+# Probe audio containers with ffprobe and extract name/folder labels
+sample-indexer metadata --workers 4
+
+# Run full decode & low-end DSP (sub-bass, pitch f0, decay) on kick/bass candidates
+sample-indexer lowend --workers 2
+
+# Run full decode & tempo/meter/key DSP on loop candidates
+sample-indexer loops --workers 2
+
+# General streaming timbre, true peak, and integrated loudness (LUFS)
+sample-indexer audio --workers 2
+
+# Parse Standard MIDI Files (SMF/RMID) into musical note metrics
+sample-indexer midi
+
+# Check coverage, health, and library statistics
+sample-indexer stats
+sample-indexer coverage
+sample-indexer midi-coverage
+
+# Query samples by technical criteria
+sample-indexer query --folder "Samples/Kicks" --ext wav --limit 20
+
+# Create a safe, online SQLite backup
+sample-indexer backup index.backup.sqlite
+```
+
+Global option `--db PATH` (placed before the subcommand) or the environment variable `SAMPLE_INDEXER_DB` selects the database location (default: `data/index.sqlite`).
+`SAMPLE_INDEXER_ROOT` or `--root PATH` configures the library root.
+
+Exit codes:
+- `0`: success
+- `1`: differences detected / file or probe errors
+- `2`: usage error or database error
+- `130`: interrupted (SIGINT / Ctrl-C)
+
+---
+
+## Analysis pipeline
+
+| Stage | Subcommand | Reads | Writes |
+|---|---|---|---|
+| **Identity** | `scan` | File content (SHA-256), `stat` | `resource` (sha256, size, ext), `path` (rel_path, mtime) |
+| **Probe** | `metadata` | Header via single `ffprobe` per audio file | duration, sample rate, channels, bit depth, codec |
+| **Labels** | `metadata` | File path and name only | kind, category, bpm_name, key_name, label evidence |
+| **Low-End** | `lowend` | Full decode via `ffmpeg -xerror` | `valid`/`valid_error`, peak, RMS, crest factor, bands, sub peak Hz, f0 pitch Hz, decay ms, stereo correlation |
+| **Loops** | `loops` | Full decode via `ffmpeg -xerror` | `bpm_est`, `beats`, `bars_est`, `key_est`, confidence metrics, chroma analysis |
+| **Audio** | `audio` | Complete streaming PCM | Integrated LUFS, true peak dBTP, spectral centroid, rolloff, flatness, structured descriptors |
+| **MIDI** | `midi` | SMF / RMID events (stdlib parser) | Note counts, pitch bounds, note density, tempo, bars |
+
+### Identity and reconciliation (`scan`)
+- Identity is determined by the content SHA-256 (`resource`), while file paths are recorded as attributes (`path`).
+- Paths with unchanged `(rel_path, size, mtime_ns)` are not re-hashed.
+- Relocated or renamed files are recognized by matching their SHA-256 with vanished records.
+- Deleted or missing files are marked with `missing=1`; database records are never hard-deleted.
+
+### Container probing and heuristics (`metadata`)
+- Fast, non-destructive header inspection using `ffprobe` captures duration, sample rate, bit depth, and codec.
+- Heuristic classification extracts provisional `kind` (`one_shot`, `loop`, `midi`, `unknown`), `category`, and filename BPM/key hints.
+- Audio validity is left `NULL` until a stage actually performs a conclusive full decode.
+
+### Low-End & Kick analysis (`lowend`)
+- Focuses on kick drums, 808s, and bass candidates.
+- Measures fundamental pitch ($f_0$) via time-domain autocorrelation, sub-bass peak frequency via high-resolution zero-padded FFT, decay time ($T_{60}$ equivalent), and low-frequency stereo correlation ($< 150$ Hz).
+
+### Loop & Tempo analysis (`loops`)
+- Targets loop candidates (folder conventions or loop labels).
+- Measures tempo via onset spectral flux autocorrelation across candidate BPM ranges.
+- Estimates musical key using 12-bin chroma profiling correlated against Krumhansl-Schmuckler key profiles.
+
+### Comprehensive timbre & loudness (`audio`)
+- Streams full audio signals through chunked spectral analysis.
+- Computes integrated loudness (LUFS) and true peak (dBTP) using EBU R128 metering standards.
+- Evaluates spectral centroid, 85% spectral rolloff, and spectral flatness.
+
+### MIDI parsing (`midi`)
+- Pure Python Standard MIDI File (SMF format 0/1/2) and RMID parser without external dependencies.
+- Calculates total notes, pitch range (`pitch_min`, `pitch_max`), note density (notes/beat), bar counts, and tempo events.
+
+---
+
+## Querying the index
+
+### CLI examples
+
+```bash
+# 1. Kick one-shots with sub frequency peak in 45-65 Hz range
+sample-indexer query --folder "Samples/Drums - One Shots/Kicks" --sub-peak-min 45 --sub-peak-max 65 --limit 10
+
+# 2. Fast loops (>= 140 BPM) with measured tempo confidence >= 0.20
+sample-indexer query --bpm-min 140 --bpm-confidence-min 0.20 --limit 10
+
+# 3. Stereo loops with high mono compatibility in the low end (stereo_corr_low >= 0.85)
+sample-indexer query --stereo-corr-low-min 0.85 --limit 10
+
+# 4. Short percussion/hi-hat one-shots (duration <= 0.15s, sr=44100)
+sample-indexer query --duration-max 0.15 --sr 44100 --limit 10
+
+# 5. Long atmospheric textures (duration >= 10.0s)
+sample-indexer query --duration-min 10.0 --limit 10
+
+# 6. Dense MIDI files (density >= 6.0 notes/beat)
+sample-indexer query --ext mid --midi-density-min 6.0 --limit 10
+
+# 7. Long MIDI arrangements with 100+ note events
+sample-indexer query --ext mid --midi-notes-min 100 --limit 10
+
+# 8. Samples matching musical key 'F min' (from filename hint or measured chroma)
+sample-indexer query --key "F min" --limit 10
+
+# 9. Basslines with fundamental pitch f0 between 40 and 90 Hz
+sample-indexer query --f0-min 40 --f0-max 90 --limit 10
+
+# 10. Locate known corrupted or unparseable files
+sample-indexer query --only-errors
+
+# 11. Output structured JSON including technical audio descriptors
+sample-indexer query --folder "Samples/Loops/Bass Loops" --bpm-min 130 --bpm-max 140 --json --limit 5
+```
+
+### Python API
+
+`sample-indexer` can be imported directly into Python scripts:
+
+```python
+import sqlite3
+from sample_indexer import query
+
+conn = sqlite3.connect("data/index.sqlite")
+conn.row_factory = sqlite3.Row
+
+# Query kicks tuned between 48 Hz and 55 Hz
+rows = query(
+    conn,
+    folder="Samples/Drums - One Shots/Kicks",
+    sub_peak_min=48.0,
+    sub_peak_max=55.0,
+    limit=25,
+)
+
+for row in rows:
+    print(f"{row['rel_path']} - {row['sub_peak_hz']:.1f} Hz")
+```
+
+---
+
+## Safety and execution semantics
+
+- **Resumable execution**: Each stage tracks progress in `resource_stage`. Interrupted runs (Ctrl-C / `--max-seconds`) commit finished checkpoints and exit cleanly.
+- **Error handling**: Corrupted files or failed decodes record the failure status in the index and count as processed to prevent retry storms. Use `--retry-errors` to retry them or `--force` to reprocess.
+- **Database backups**: Schema migrations automatically create timestamped SQLite backups before applying changes. Manual backups can be created at any time using `sample-indexer backup <dest>`.
+- **Subprocess security**: All external calls to `ffmpeg` and `ffprobe` use explicit argument lists (`file:<path>`) without shell interpolation.
+
+---
+
+## Running tests
+
+The test suite runs against temporary file trees and scratch SQLite databases:
+
+```bash
+python3 -m unittest discover
+```
+
+---
+
+## License
+
+This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.

sample_indexer-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sample-indexer"
+version = "0.1.0"
+description = "SQLite index of a sample and MIDI library with incremental reconciliation"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.12"
+dependencies = [
+    "numpy",
+]
+authors = [
+    { name = "Adrián Cancio" },
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: End Users/Desktop",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Multimedia :: Sound/Audio :: Analysis",
+]
+
+[project.scripts]
+sample-indexer = "sample_indexer.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["sample_indexer*"]

sample_indexer-0.1.0/sample_indexer/__init__.py

@@ -0,0 +1,6 @@
+"""sample-indexer: SQLite index of a sample/MIDI library with incremental reconciliation."""
+
+from .reports import query
+
+__version__ = "0.1.0"
+__all__ = ["__version__", "query"]

sample_indexer-0.1.0/sample_indexer/__main__.py

@@ -0,0 +1,6 @@
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())