omniloader 1.0.0__tar.gz

omniloader-1.0.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,44 @@
+---
+name: Bug report
+about: Report something that is broken or not working as expected
+labels: bug
+assignees: fodorad
+---
+
+## Description
+
+A clear and concise description of the bug.
+
+## Steps to reproduce
+
+```python
+# Minimal reproducible example
+import torch
+from omniloader import DictTensorDataset, DatasetSchema, TensorSpec, OmniLoader
+
+# ...
+```
+
+## Expected behaviour
+
+What you expected to happen.
+
+## Actual behaviour
+
+What actually happened. Include the full traceback if applicable.
+
+```
+Traceback (most recent call last):
+  ...
+```
+
+## Environment
+
+- OmniLoader version: <!-- e.g. 1.0.0 — run `pip show omniloader` -->
+- PyTorch version: <!-- e.g. 2.11 -->
+- Python version: <!-- e.g. 3.12.3 -->
+- OS: <!-- e.g. Ubuntu 22.04 / macOS 15 / Windows 11 -->
+
+## Additional context
+
+Any other information that might be helpful (dataset shapes, schema, etc.).

omniloader-1.0.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,27 @@
+---
+name: Feature request
+about: Suggest a new capability or improvement
+labels: enhancement
+assignees: fodorad
+---
+
+## Problem
+
+What are you trying to do that OmniLoader does not currently support?
+
+## Proposed solution
+
+A clear description of the feature or API you would like. Sketch the call site if
+you can:
+
+```python
+# What you wish you could write
+```
+
+## Alternatives considered
+
+Any workarounds you have tried or other designs you weighed.
+
+## Additional context
+
+Links to papers, datasets, or related multi-task-learning techniques.

omniloader-1.0.0/.github/dependabot.yml

@@ -0,0 +1,13 @@
+version: 2
+updates:
+  - package-ecosystem: pip
+    directory: "/"
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 5
+
+  - package-ecosystem: github-actions
+    directory: "/"
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 5

omniloader-1.0.0/.github/pull_request_template.md

@@ -0,0 +1,22 @@
+## Summary
+
+<!-- One or two sentences describing what this PR does and why. -->
+
+## Type of change
+
+- [ ] Bug fix (`fix:` commit)
+- [ ] New feature (`feat:` commit)
+- [ ] Breaking change (`feat!:` or `BREAKING CHANGE:` commit)
+- [ ] Documentation update (`docs:` commit)
+- [ ] Refactor / test / chore (no behaviour change)
+
+## Checklist
+
+- [ ] `make fix` was run (ruff format + lint auto-fix)
+- [ ] `make check` passes locally (lint + type-check + tests + docs build)
+- [ ] New tests added for new or changed behaviour
+- [ ] Docs updated if user-facing behaviour changed
+
+## Related issues
+
+<!-- Closes #... -->

omniloader-1.0.0/.github/workflows/cd.yml

@@ -0,0 +1,43 @@
+name: CD
+
+on:
+  push:
+    tags:
+      - "v*"
+  workflow_dispatch:
+
+jobs:
+  publish:
+    name: Build, publish, and release
+    runs-on: ubuntu-latest
+    environment: cd
+    permissions:
+      id-token: write    # required for OIDC trusted publisher (PyPI)
+      contents: write    # required to create GitHub Release
+
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.12"
+
+    - name: Set up uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+
+    - name: Build package
+      run: uv build
+
+    - name: Publish to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+
+    - name: Create GitHub Release
+      uses: softprops/action-gh-release@v3
+      with:
+        generate_release_notes: true
+        files: dist/*

omniloader-1.0.0/.github/workflows/ci.yml

@@ -0,0 +1,71 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main, dev]
+
+jobs:
+  test:
+    name: test
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0  # required for hatch-vcs git describe
+
+    - name: Set up Python 3.12
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.12"
+
+    - name: Set up uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+
+    - name: Install dependencies
+      run: |
+        # On Linux CI, use CPU-only PyTorch wheels to avoid CUDA runtime
+        # dependencies that are absent on the runner. PEP 440 ignores local
+        # version segments, so torch>=2.0 matches both the CPU and CUDA builds.
+        uv pip install --system --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match -e ".[all,dev,docs]"
+
+    - name: Audit dependencies for vulnerabilities
+      run: uvx pip-audit
+      continue-on-error: true  # torch may have known CVEs; treat as warning
+
+    - name: Ruff Lint
+      run: ruff check .
+
+    - name: Ruff Format
+      run: ruff format --check .
+
+    - name: Type check (ty)
+      run: ty check omniloader
+
+    - name: Test & Coverage
+      run: |
+        coverage run -m unittest discover -s tests -v
+        coverage report
+        coverage html
+        coverage xml -o coverage.xml
+
+    - name: Upload coverage reports to Codecov
+      uses: codecov/codecov-action@v7
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        files: coverage.xml
+        fail_ci_if_error: true
+
+    - name: Upload coverage HTML
+      uses: actions/upload-artifact@v7
+      with:
+        name: coverage-report
+        path: coverage_html/
+      if: always()
+
+    - name: Build docs
+      run: sphinx-build -b html docs/ site/

omniloader-1.0.0/.github/workflows/docs.yml

@@ -0,0 +1,40 @@
+name: Docs
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.12"
+
+    - name: Set up uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+
+    - name: Install package and docs dependencies
+      run: uv pip install --system --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match -e ".[docs]"
+
+    - name: Build docs
+      run: sphinx-build -b html docs/ site/
+
+    - name: Deploy to GitHub Pages
+      uses: peaceiris/actions-gh-pages@v4
+      with:
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        publish_dir: ./site

omniloader-1.0.0/.github/workflows/release-please.yml

@@ -0,0 +1,18 @@
+name: Release Please
+
+on:
+  push:
+    branches: [main]
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  release-please:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: googleapis/release-please-action@v5
+      with:
+        release-type: python
+        token: ${{ secrets.RELEASE_PLEASE_TOKEN }}

omniloader-1.0.0/.gitignore

@@ -0,0 +1,221 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# MacOS
+.DS_Store
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#   you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml

omniloader-1.0.0/.pre-commit-config.yaml

@@ -0,0 +1,17 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.8.4
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: local
+    hooks:
+      - id: ty-check
+        name: ty type check
+        entry: ty check
+        args: [omniloader]
+        language: system
+        types: [python]
+        pass_filenames: false

omniloader-1.0.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "1.0.0"
+}

omniloader-1.0.0/CHANGELOG.md

@@ -0,0 +1,142 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+Releases are managed automatically with
+[release-please](https://github.com/googleapis/release-please) from
+[Conventional Commits](https://www.conventionalcommits.org/). Future entries will
+list the changes relative to the previous release.
+
+## 1.0.0 (2026-07-07)
+
+
+### Features
+
+* initial public release of OmniLoader ([5814077](https://github.com/fodorad/OmniLoader/commit/58140772992fb70a77865b4bacf5756cbd18da10))
+
+## 1.0.0 (2026-07-05)
+
+Initial public release. OmniLoader is a modality- and model-agnostic PyTorch meta
+data loader that unifies datasets with disjoint annotations into one masked sample
+scheme for joint multi-task training.
+
+### Unified sample scheme
+
+- **`OmniLoader`** concatenates any number of source datasets and presents them as
+  one dataset whose every sample follows a shared `UnifiedSchema`. Keys a source
+  lacks are filled with a placeholder tensor plus an all-`False` `<name>_mask`, so a
+  single batch may freely mix samples from different datasets.
+- **`SampleUnifier`** maps raw sample dicts onto the union schema, padding or cropping
+  sequence values to their declared length and ingesting any dataset-supplied
+  `<name>_mask` (aligned and AND-ed with the padding mask). Metadata (`dataset`,
+  `subset`, `key`) is preserved.
+- **`unified_collate`** stacks tensor entries and gathers string metadata into lists.
+
+### Structural schema
+
+- **`TensorSpec`** declares a single value — a feature or target purely by which list
+  it is placed in — as a **vector** (`()` or `(F,)`) or a **sequence** (`(T,)` or
+  `(T, F)`); it is a sequence exactly when `time_dim` is set. No modality or model
+  concept appears anywhere.
+- **`DatasetSchema`** / **`UnifiedSchema`** hold per-dataset specs and merge them into
+  the validated union (shared keys must agree on shape and dtype).
+
+### Dataset adapters
+
+- **`HDF5Dataset`** reads per-sample groups from HDF5 (worker-safe lazy handle,
+  byte-string decoding), with optional `cache_size` (LRU) and `preload`.
+- **`NpyFolderDataset`** reads memory-mapped `root/<sample>/<key>.npy` folders.
+- **`DictTensorDataset`** provides an in-memory, random-tensor dataset for tests and
+  quick experiments.
+- **`split_indices`** (+ `save_split_info` / `load_split_info`) produces reproducible,
+  optionally stratified train/val/test index splits.
+
+### Mixing strategies
+
+- **`ProportionalStrategy`**, **`TemperatureStrategy`** (`size ** (1 / T)`;
+  `temperature=2` recovers square-root scaling), **`AnnealedTemperatureStrategy`**
+  (per-epoch annealing), **`FixedWeightStrategy`** (explicit or uniform weights), and
+  **`RoundRobinStrategy`** (equal, interleaved draws) — each producing a flat list of
+  global indices per epoch.
+
+### Subsampling
+
+- **`SubsampleConfig`** / **`IndexPool`** support with/without-replacement draws, two
+  exhaustion policies (`FRESH` re-randomises every epoch, `EXHAUST` covers the whole
+  dataset before any reuse), per-dataset `effective_size`, and per-sample
+  `sample_weights` — all reproducible from a seed and epoch.
+- **`class_weights_for_sampler`** derives per-sample inverse-frequency `sample_weights`
+  from a target (for the sampler).
+
+### Class balance
+
+- **`class_weights_for_loss`** and **`class_histogram`** compute per-class **loss** weights and
+  exact counts over every valid labelled position of a categorical target (schemes:
+  inverse frequency or effective-number-of-samples, Cui et al. 2019). Persist and reuse
+  via ``omniloader class-weights-for-loss CONFIG --target KEY -o weights.json`` for
+  `CrossEntropyLoss(weight=...)` — the loss/model usage stays in your training code.
+
+### Variable-length batching
+
+- **`DynamicCollator`** pads each sequence key to the batch's longest sample instead of
+  a fixed `time_dim` (pair with `OmniLoader(pad_features=False)`).
+- **`LengthBucketBatchSampler`** (+ `OmniLoader.sequence_lengths`) groups similar-length
+  sequences to minimise padding waste in the native-length regime.
+
+### Normalization
+
+- **`Normalize`**, **`MinMaxNormalize`**, **`RobustNormalize`** (median/IQR) and
+  **`InstanceNormalize`** (per-sample, no external stats).
+- **`PerDatasetNormalize`** standardizes each sample by **its source dataset's** own
+  statistics (selected from the `dataset` metadata key) — the CMVN/AdaBN remedy for
+  cross-corpus domain shift — with a `fallback` (`instance` / `union` / `identity`) for
+  dataset ids unseen at training time, so inference on new sources is handled
+  deliberately.
+- **`compute_stats`** (pooled) and **`compute_dataset_stats`** (per source dataset)
+  return mean/std/min/max/median/iqr over valid positions; `save_stats` / `load_stats`
+  round-trip either the flat or nested per-dataset form through JSON. The streaming
+  `compute_feature_stats` remains for `(mean, std)` only. Config transforms may
+  reference a saved stats file via `stats_path` / `union_stats_path` (loaded with
+  `load_stats`) instead of inlining the numbers, so normalization is fully config-driven.
+
+### Augmentation
+
+- **`GaussianNoise`** (feature corruption), **`FeatureDropout`** (stream/modality
+  dropout, keeps at least one stream), **`SpanMasking`** (SpecAugment-style time spans),
+  **`FeatureMasking`** (feature bands), **`TimeWarp`** (random speed perturbation),
+  **`RandomCrop`** / **`CenterCrop`** (shared offset keeps features and targets aligned),
+  and **`MixupCollator`** (MixUp/CutMix at collate, emitting `mixup_lambda` and paired
+  targets for the model to combine — no loss logic in the library).
+- **`Compose`** chains transforms; every transform is mask-aware, seedable and
+  self-skips during evaluation.
+
+### Distributed & reproducibility
+
+- **`OmniSampler`** turns a strategy into a per-epoch index stream and shards it across
+  `rank` / `num_replicas` (inferred from `torch.distributed`) for multi-GPU runs.
+- Per-sample augmentation is seeded from `(seed, epoch, index)` via `set_epoch`, and
+  **`seed_worker`** seeds DataLoader workers — augmentation is deterministic and
+  independent of worker count.
+
+### Introspection
+
+- **`describe`** (coverage matrix, valid fractions, class distributions) and
+  **`validate`** (dry-run spec check), plus `OmniLoader.describe()` / `.validate()`.
+
+### Configuration, CLI & Lightning
+
+- **`OmniConfig`** loads from a dict, JSON or YAML and describes an entire run — seed,
+  DataLoader knobs, mixing strategy, per-dataset subsampling, transform pipeline,
+  collate function, length bucketing, distributed settings and the datasets themselves.
+  **`build_dataloader`** assembles a ready `DataLoader` end-to-end; `build_loader`,
+  `build_strategy`, `build_transform`, `build_collate`, `build_subsample` and
+  `build_datasets` expose the pieces. `seed_everything` seeds Python, NumPy and PyTorch.
+- **`omniloader` CLI**: `describe` / `validate` / `compute-stats` / `class-weights-for-loss`
+  over a config file.
+- **Annotated config template** shipped as package data (`config_template.yaml`),
+  documenting every option with defaults; locate it with `config_template_path`.
+- **`OmniDataModule`** (optional `omniloader[lightning]` extra) wires train/val/test
+  splits, the mixing strategy, distributed sharding and seeding; the core stays
+  pure-torch.