foley 0.0.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- foley-0.0.2/.claude/CLAUDE.md +55 -0
- foley-0.0.2/.claude/skills/foley-dev-add-source/SKILL.md +116 -0
- foley-0.0.2/.claude/skills/foley-dev-implement/SKILL.md +101 -0
- foley-0.0.2/.gitattributes +1 -0
- foley-0.0.2/.github/workflows/ci.yml +48 -0
- foley-0.0.2/.gitignore +121 -0
- foley-0.0.2/LICENSE +21 -0
- foley-0.0.2/PKG-INFO +119 -0
- foley-0.0.2/README.md +76 -0
- foley-0.0.2/foley/__init__.py +292 -0
- foley-0.0.2/foley/audio.py +424 -0
- foley-0.0.2/foley/base.py +452 -0
- foley-0.0.2/foley/index/__init__.py +78 -0
- foley-0.0.2/foley/index/embedders.py +173 -0
- foley-0.0.2/foley/index/indexes.py +595 -0
- foley-0.0.2/foley/index/library.py +476 -0
- foley-0.0.2/foley/index/protocols.py +134 -0
- foley-0.0.2/foley/index/search.py +174 -0
- foley-0.0.2/foley/index/taxonomy/__init__.py +48 -0
- foley-0.0.2/foley/index/taxonomy/audioset.py +71 -0
- foley-0.0.2/foley/index/taxonomy/model.py +103 -0
- foley-0.0.2/foley/index/taxonomy/resolver.py +236 -0
- foley-0.0.2/foley/index/taxonomy/seed.py +353 -0
- foley-0.0.2/foley/index/taxonomy/ucs.py +102 -0
- foley-0.0.2/foley/licensing.py +173 -0
- foley-0.0.2/foley/qc.py +584 -0
- foley-0.0.2/foley/stores.py +182 -0
- foley-0.0.2/misc/docs/design.md +209 -0
- foley-0.0.2/misc/docs/research/01-sfx-source-apis.md +344 -0
- foley-0.0.2/misc/docs/research/02-genai-sfx-generation.md +349 -0
- foley-0.0.2/misc/docs/research/03-sound-recognition-tagging.md +366 -0
- foley-0.0.2/misc/docs/research/04-embeddings-indexing-search.md +423 -0
- foley-0.0.2/misc/docs/research/05-context-retrieval-and-agent.md +384 -0
- foley-0.0.2/misc/docs/research/06-weaving-mixing-mastering.md +556 -0
- foley-0.0.2/misc/docs/research/07-licensing-provenance.md +346 -0
- foley-0.0.2/misc/docs/research/08-evaluation-quality.md +460 -0
- foley-0.0.2/misc/docs/research/09-audio-io-dsp-storage.md +513 -0
- foley-0.0.2/misc/docs/research/10-facade-architecture.md +744 -0
- foley-0.0.2/misc/docs/research/11-bootstrap-corpora-benchmarks.md +435 -0
- foley-0.0.2/misc/docs/research/12-additional-dimensions.md +477 -0
- foley-0.0.2/misc/docs/research/README.md +48 -0
- foley-0.0.2/misc/docs/research/deep-research-prompts.md +351 -0
- foley-0.0.2/misc/docs/research.md +3 -0
- foley-0.0.2/misc/docs/roadmap.md +92 -0
- foley-0.0.2/pyproject.toml +218 -0
- foley-0.0.2/tests/__init__.py +0 -0
- foley-0.0.2/tests/conftest.py +55 -0
- foley-0.0.2/tests/test_audio.py +289 -0
- foley-0.0.2/tests/test_base.py +421 -0
- foley-0.0.2/tests/test_embedders.py +134 -0
- foley-0.0.2/tests/test_indexes.py +187 -0
- foley-0.0.2/tests/test_library.py +236 -0
- foley-0.0.2/tests/test_qc.py +282 -0
- foley-0.0.2/tests/test_search.py +129 -0
- foley-0.0.2/tests/test_stores.py +220 -0
- foley-0.0.2/tests/test_taxonomy.py +207 -0
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# foley — agent guide
|
|
2
|
+
|
|
3
|
+
**foley** is a retrieval-first façade for sound effects: find (or generate) the right
|
|
4
|
+
sound for a moment of narration and weave it in. Four stages —
|
|
5
|
+
**SOURCE → INDEX → SELECT → WEAVE** — plus cross-cutting layers (licensing/provenance,
|
|
6
|
+
evaluation/QC, observability, `dol` storage). It is the SFX sibling of
|
|
7
|
+
[`arioso`](https://github.com/thorwhalen/arioso) (a unified façade over music-gen
|
|
8
|
+
backends) and shares its discipline.
|
|
9
|
+
|
|
10
|
+
**Status:** early implementation. The plan is **[Epic #13](https://github.com/thorwhalen/foley/issues/13)**
|
|
11
|
+
with 12 sub-issues (#1–#12). **#1 Foundation** (data models, `dol` stores, audio + Tier-0 QC)
|
|
12
|
+
and **#2 Index** (CLAP embedder, hybrid RRF search, `SoundLibrary` façade, UCS/AudioSet
|
|
13
|
+
taxonomy resolver) are built; next is **#3 Ingestion** → **#4 Bootstrap** (the `ingest()`→`search()` keystone).
|
|
14
|
+
|
|
15
|
+
## Read before doing foley work
|
|
16
|
+
|
|
17
|
+
- **Authoritative architecture** → [`misc/docs/research/10-facade-architecture.md`](../misc/docs/research/10-facade-architecture.md)
|
|
18
|
+
(component diagram, data models, façade API, protocols, module tree, build order).
|
|
19
|
+
- **Design map** → [`misc/docs/design.md`](../misc/docs/design.md) ·
|
|
20
|
+
**Roadmap** → [`misc/docs/roadmap.md`](../misc/docs/roadmap.md)
|
|
21
|
+
- **Research** → 12 cited reports in [`misc/docs/research/`](../misc/docs/research/)
|
|
22
|
+
(index + the 12-prompt library there).
|
|
23
|
+
|
|
24
|
+
## Dev skills (the agent toolkit for building foley)
|
|
25
|
+
|
|
26
|
+
| Skill | Use when |
|
|
27
|
+
|-------|----------|
|
|
28
|
+
| **`foley-dev-implement`** | implementing any epic subtask — the build loop, module layout, canonical data models, invariants, façade discipline, git flow |
|
|
29
|
+
| **`foley-dev-add-source`** | adding a sound source — a retrieve adapter (Freesound/BBC/…) or a generate adapter (Stable Audio Open/ElevenLabs/…) via the `SOURCE_CONFIG` pattern |
|
|
30
|
+
|
|
31
|
+
These are **dev** skills (for the agent building foley), not skills shipped to end
|
|
32
|
+
users. Real files live in `skills/foley-dev-*/`, symlinked into `.claude/skills/`.
|
|
33
|
+
|
|
34
|
+
## Behavioral rules (the non-negotiables)
|
|
35
|
+
|
|
36
|
+
- **Git flow:** work on a **feature branch → PR into `main`**. No direct-to-main commits.
|
|
37
|
+
- **Follow the architecture; don't fork it.** Build against the module layout,
|
|
38
|
+
protocols, and canonical data models in report 10 / `base.py`. Don't invent parallel
|
|
39
|
+
structures. See `foley-dev-implement`.
|
|
40
|
+
- **Licensing is load-bearing** (foley's output gets published): `LicenseRecord` is the
|
|
41
|
+
SSOT for *both* the candidate license-filter *and* each sound's storage mode;
|
|
42
|
+
`redistribute_standalone_ok` (copyright) ≠ `cache_bytes_ok` (TOS); the license gate is
|
|
43
|
+
**fail-closed**. See `foley-dev-add-source` and report 07.
|
|
44
|
+
- **Façade discipline:** mirror `arioso` (config-driven plugin adapters, unified
|
|
45
|
+
vocabulary → native params, zero-dep core + lazy optional-extras, `dol` storage) and
|
|
46
|
+
`accompy` (progressive disclosure, protocol-based extensibility, `check_requirements`
|
|
47
|
+
onboarding).
|
|
48
|
+
- **Every module needs a top-level docstring** (auto-extracted for docs). Prefer
|
|
49
|
+
functional style; keyword-only args past the 3rd position.
|
|
50
|
+
|
|
51
|
+
## Where things go (placement test)
|
|
52
|
+
|
|
53
|
+
Behavioral rules → this file / the skills. Content & context → the named files in
|
|
54
|
+
`misc/docs/` referenced above. *If deleting a sentence wouldn't change behavior, it
|
|
55
|
+
belongs in a file, not here.*
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: foley-dev-add-source
|
|
3
|
+
description: Use when adding a new sound SOURCE to foley — either a retrieve adapter (a sound-library/API such as Freesound, BBC Sound Effects, Epidemic, Storyblocks, Pond5) or a generate adapter (a text-to-SFX model/API such as Stable Audio Open, ElevenLabs Sound Effects, AudioGen). Covers the SOURCE_CONFIG plugin pattern (mirroring arioso's per-platform config), the SourceAdapter protocol, unified query/generation vocabulary → native-param translation, lazy optional-deps, and the mandatory per-source license policy (commercial_ok / redistribute_standalone_ok / cache_bytes_ok / ai_training_ok). Triggers on adding a source, adding an adapter, wiring a new SFX API or generator into foley.
|
|
4
|
+
metadata:
|
|
5
|
+
audience: developers
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Adding a source to foley
|
|
9
|
+
|
|
10
|
+
A **source** is anywhere foley gets a sound. Two kinds, one contract:
|
|
11
|
+
|
|
12
|
+
- **retrieve** — fetch existing sounds from a library/API. Anchor: **Freesound** (CC0
|
|
13
|
+
subset). Others (BBC, Epidemic, Storyblocks, Pond5, Pro Sound Effects) are mostly
|
|
14
|
+
partner-gated → ship as stubs. See `misc/docs/research/01-sfx-source-apis.md`.
|
|
15
|
+
- **generate** — synthesize a sound (the arioso analog). Defaults: **Stable Audio Open**
|
|
16
|
+
(local) + **ElevenLabs Sound Effects** (hosted). See
|
|
17
|
+
`misc/docs/research/02-genai-sfx-generation.md`.
|
|
18
|
+
|
|
19
|
+
The pattern mirrors [`arioso`](https://github.com/thorwhalen/arioso)'s
|
|
20
|
+
`arioso/platforms/<name>/` (a `config.py` declaring `PLATFORM_CONFIG` + an optional
|
|
21
|
+
`adapter.py`). Read an arioso platform before writing a foley source.
|
|
22
|
+
|
|
23
|
+
## Layout
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
foley/sources/<name>/
|
|
27
|
+
__init__.py
|
|
28
|
+
config.py # required: declares SOURCE_CONFIG
|
|
29
|
+
adapter.py # optional: custom logic (needed for non-trivial REST / local models)
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The source is auto-discovered by `registry.py`; third parties can also
|
|
33
|
+
`register_source(name, config, adapter)` at runtime.
|
|
34
|
+
|
|
35
|
+
## `SOURCE_CONFIG` (the declaration)
|
|
36
|
+
|
|
37
|
+
```python
|
|
38
|
+
SOURCE_CONFIG = {
|
|
39
|
+
"name": "<name>",
|
|
40
|
+
"kind": "retrieve", # or "generate"
|
|
41
|
+
"display_name": "...",
|
|
42
|
+
"website": "https://...",
|
|
43
|
+
|
|
44
|
+
"auth": {"type": "bearer_token", "env_var": "<NAME>_API_KEY"}, # or None
|
|
45
|
+
|
|
46
|
+
# unified foley vocabulary -> this source's native names
|
|
47
|
+
"param_map": {
|
|
48
|
+
"query": {"native_name": "text", "required": True}, # retrieve
|
|
49
|
+
"duration": {"native_name": "length_s"},
|
|
50
|
+
# generate uses GENERATION_AFFORDANCES (prompt, duration, prompt_influence, ...)
|
|
51
|
+
},
|
|
52
|
+
"supported_affordances": ["query", "duration", "license"],
|
|
53
|
+
"on_unsupported_param": "warn",
|
|
54
|
+
|
|
55
|
+
# REST sources: endpoints; local models: omit and implement adapter.generate()
|
|
56
|
+
"api": {"base_url": "https://...", "search_endpoint": {...}, "download_endpoint": {...}},
|
|
57
|
+
|
|
58
|
+
"output": {"default_format": "wav", "returns": "bytes"}, # or "url" / "array"
|
|
59
|
+
|
|
60
|
+
# ⚠️ MANDATORY license policy — see invariants below
|
|
61
|
+
"license": {
|
|
62
|
+
"default_license_id": "CC0-1.0", # per-item value can override
|
|
63
|
+
"commercial_ok": True,
|
|
64
|
+
"redistribute_standalone_ok": True, # copyright: may the raw file be re-exposed?
|
|
65
|
+
"cache_bytes_ok": False, # TOS: may we store the bytes? (Freesound: no)
|
|
66
|
+
"ai_training_ok": True,
|
|
67
|
+
"requires_attribution": False,
|
|
68
|
+
},
|
|
69
|
+
"commercial_ok": True, # generate guardrail (+ revenue_cap for Stable Audio Open)
|
|
70
|
+
"rate": {"per_min": 60, "per_day": 2000},
|
|
71
|
+
"cost": {...},
|
|
72
|
+
"data_egress": "external", # for offline/sensitive-narration mode
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## The `SourceAdapter` protocol
|
|
77
|
+
|
|
78
|
+
- **retrieve:** `search(query, **affordances) -> list[Candidate]` · `get(id) -> SoundRecord`
|
|
79
|
+
· `download(record) -> bytes` (honoring `cache_bytes_ok`).
|
|
80
|
+
- **generate:** `generate(prompt, **affordances) -> SoundRecord` (bytes/array in the blob
|
|
81
|
+
store; a `LicenseRecord` marking `is_ai_generated=True`, model/seed/prompt captured).
|
|
82
|
+
|
|
83
|
+
Translate unified affordances → native params via `param_map` (like arioso's translation
|
|
84
|
+
layer); warn-and-drop unsupported ones per `on_unsupported_param`.
|
|
85
|
+
|
|
86
|
+
## License invariants (do NOT skip — foley output gets published)
|
|
87
|
+
|
|
88
|
+
1. **Every fetched or generated sound gets a `LicenseRecord`.** Seed it from
|
|
89
|
+
`SOURCE_CONFIG["license"]`; let per-item API metadata override (e.g. a Freesound sound's
|
|
90
|
+
actual CC license).
|
|
91
|
+
2. **`redistribute_standalone_ok` (copyright) is distinct from `cache_bytes_ok` (TOS).**
|
|
92
|
+
Freesound CC0 → `redistribute_standalone_ok=True` but `cache_bytes_ok=False`: store
|
|
93
|
+
**by-reference** (keep the source URI/id, re-fetch on use), never a permanent local copy.
|
|
94
|
+
Own/generated/CC0-bulk audio → `cache_bytes_ok=True`: store **by-value** (content-hashed
|
|
95
|
+
bytes in the blob store).
|
|
96
|
+
3. **Generate adapters carry a `commercial_ok` guardrail.** Most open SFX weights are
|
|
97
|
+
CC-BY-NC → their outputs are non-commercial; mark them. Stable Audio Open is commercial
|
|
98
|
+
under a **$1M revenue cap** — encode `revenue_cap_usd`.
|
|
99
|
+
4. **Generated audio is watermarked/disclosed** (AudioSeal + optional C2PA) — handled by
|
|
100
|
+
`foley/provenance/`, but set `is_ai_generated` + generation fields on the record.
|
|
101
|
+
5. The agent's fail-closed `keep(record, IntendedUse)` gate depends on these flags being
|
|
102
|
+
correct. Wrong flags = legal/TOS violation slips through.
|
|
103
|
+
|
|
104
|
+
## Checklist for a new source
|
|
105
|
+
|
|
106
|
+
- [ ] `foley/sources/<name>/config.py` with a complete `SOURCE_CONFIG` (incl. the full
|
|
107
|
+
`license` block + `kind`).
|
|
108
|
+
- [ ] `adapter.py` if REST/local logic is non-trivial; else rely on the shared base adapter.
|
|
109
|
+
- [ ] `param_map` covers every affordance the source supports; unsupported → `warn`.
|
|
110
|
+
- [ ] License flags set correctly, especially `cache_bytes_ok` vs `redistribute_standalone_ok`.
|
|
111
|
+
- [ ] Optional dep in a `foley[<name>]` extra, **lazy-imported** (core stays zero-dep).
|
|
112
|
+
- [ ] Tests: adapter returns valid `Candidate`/`SoundRecord`s; license flags populate;
|
|
113
|
+
by-reference vs by-value storage honored.
|
|
114
|
+
- [ ] Auto-discovered by `registry.py` (or documented `register_source` usage).
|
|
115
|
+
|
|
116
|
+
See `foley-dev-implement` for the surrounding build loop, module layout, and git flow.
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: foley-dev-implement
|
|
3
|
+
description: Use when implementing (or planning to implement) any foley epic subtask — building the SOURCE / INDEX / SELECT / WEAVE stages or the cross-cutting layers of the foley SFX façade. Covers the build loop, the module layout, the canonical data models (SoundRecord, LicenseRecord, Candidate, SoundDesignTimeline), the non-negotiable invariants (license SSOT + fail-closed gate, cache_bytes_ok vs redistribute_standalone_ok, the eval PR gate, AI disclosure), the façade discipline (mirror arioso/accompy), and the branch→PR git flow. Triggers on foley implementation, foley subtask, foley adapter/index/agent/weave work, foley base.py / data models.
|
|
4
|
+
metadata:
|
|
5
|
+
audience: developers
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Building a foley subtask
|
|
9
|
+
|
|
10
|
+
foley is a retrieval-first SFX façade (SOURCE → INDEX → SELECT → WEAVE + cross-cutting
|
|
11
|
+
layers). The whole plan is **[Epic #13](https://github.com/thorwhalen/foley/issues/13)**,
|
|
12
|
+
subtasks **#1–#12**. Read this before implementing any of them.
|
|
13
|
+
|
|
14
|
+
## First: orient (don't invent)
|
|
15
|
+
|
|
16
|
+
1. Read **`misc/docs/research/10-facade-architecture.md`** — the authoritative synthesis
|
|
17
|
+
(component diagram, data models as code, façade API, protocols, module tree,
|
|
18
|
+
dependency graph). It reconciles all 12 research reports; treat it as the spec.
|
|
19
|
+
2. Skim **`misc/docs/design.md`** (the readable map) and the specific research report(s)
|
|
20
|
+
backing your subtask (each subtask issue lists them).
|
|
21
|
+
3. Look at how [`arioso`](https://github.com/thorwhalen/arioso) and
|
|
22
|
+
[`accompy`](https://github.com/thorwhalen/accompy) do the same shape — foley mirrors
|
|
23
|
+
their façade discipline.
|
|
24
|
+
|
|
25
|
+
## The build loop
|
|
26
|
+
|
|
27
|
+
1. **Pick a subtask** from Epic #13. Respect the dependency order — the keystone is
|
|
28
|
+
**#1 Foundation → #2 Index → #3 Ingestion → #4 Bootstrap** (gets `ingest()` → `search()`
|
|
29
|
+
working). Then a thin slice through Select (#7) → Weave (#8).
|
|
30
|
+
2. **Branch:** `git checkout -b <topic>` off `main`. Never commit foley code directly to
|
|
31
|
+
`main` — open a **PR** that references the subtask (`Closes #N`).
|
|
32
|
+
3. **Implement against the layout & protocols** below — extend the SSOT data models, don't
|
|
33
|
+
fork them. New optional dependency? Put it in an `optional-dependencies` extra
|
|
34
|
+
(`foley[<capability>]`) and **lazy-import** it; the core stays zero-dep.
|
|
35
|
+
4. **Test** (economist / testing-trophy: cheap, stable API-level checks first — see
|
|
36
|
+
`misc/docs/research/08-evaluation-quality.md`). Pure functions get unit tests; the
|
|
37
|
+
retrieval path gets Tier-1 metrics.
|
|
38
|
+
5. **Eval gate** (once the harness from #10 exists): a change to the index/embedder/
|
|
39
|
+
prompt must not drop **`nDCG@10` by > 0.02** on the frozen golden set.
|
|
40
|
+
6. **PR** into `main`; keep the subtask's task-list checkboxes updated.
|
|
41
|
+
|
|
42
|
+
## Module layout (from report 10 — build into this tree)
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
foley/
|
|
46
|
+
__init__.py # façade: find(), search(), generate(), ingest(), weave()
|
|
47
|
+
base.py # SoundRecord, LicenseRecord, Candidate, SoundEvent, Verdict,
|
|
48
|
+
# IntendedUse, affordance registries — the SSOT types
|
|
49
|
+
registry.py # adapter auto-discovery + lazy loading
|
|
50
|
+
audio.py # I/O + DSP primitives (soundfile/soxr/librosa/pyloudnorm)
|
|
51
|
+
sources/ # source adapters (config.py + adapter.py each) — see foley-dev-add-source
|
|
52
|
+
index/ # embedders, taggers, captioners, library (dol + LanceDB), search, taxonomy
|
|
53
|
+
agent/ # decompose, verify, policy(decide), tools, mcp
|
|
54
|
+
weave/ # align, anchor, mix, master, timeline, render
|
|
55
|
+
eval/ # qc (Tier-0), retrieval metrics (Tier-1), fit (Tier-2)
|
|
56
|
+
provenance/ # LicenseRecord wiring, credits, watermark/disclosure
|
|
57
|
+
obs/ # OpenTelemetry spans + run-manifest artifact
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Optional-extras: `foley[freesound]`, `foley[stable-audio]`, `foley[elevenlabs]`,
|
|
61
|
+
`foley[clap]`, `foley[tag]`, `foley[caption]`, `foley[index]`, `foley[weave]`,
|
|
62
|
+
`foley[agent]`, `foley[all]`. `ffmpeg`/`rubberband` are **system** deps surfaced via a
|
|
63
|
+
`check_requirements`/`verify_and_setup` helper (accompy-style), never inherited by `pip`.
|
|
64
|
+
|
|
65
|
+
## Canonical data models (SSOT — live in `base.py`, never fork)
|
|
66
|
+
|
|
67
|
+
- **`SoundRecord`** — the metadata SSOT for one sound (report 04 schema + report 09
|
|
68
|
+
storage fields + a nested `LicenseRecord` + a QC block + `named_cue` continuity). The
|
|
69
|
+
audio bytes live in the `dol` blob store; the record holds a content-hash `uri`, never
|
|
70
|
+
inline bytes.
|
|
71
|
+
- **`LicenseRecord`** — per-sound rights/provenance (report 07). Carries the derived
|
|
72
|
+
flags used everywhere: `commercial_ok`, `embed_in_derivative_ok`,
|
|
73
|
+
`redistribute_standalone_ok`, `modification_ok`, `ai_training_ok`, **`cache_bytes_ok`**,
|
|
74
|
+
plus attribution/watermark/generation fields.
|
|
75
|
+
- **`Candidate` / `SoundEvent` / `Verdict`** — the SELECT agent's working types (report 05).
|
|
76
|
+
- **`SoundDesignTimeline` / `TimelineItem`** — the WEAVE render model (report 06); it is a
|
|
77
|
+
**strict superset** of the SELECT plan. The agent emits the sparse plan; WEAVE resolves
|
|
78
|
+
anchors and fills processing defaults. It is JSON-serializable, diffable, and
|
|
79
|
+
re-renderable — the "editable sound-design timeline" (foley's snapshots/stories analog).
|
|
80
|
+
|
|
81
|
+
## Non-negotiable invariants
|
|
82
|
+
|
|
83
|
+
1. **`LicenseRecord` is the SSOT for two things:** the candidate license-filter *and*
|
|
84
|
+
each `SoundRecord`'s storage mode. Wire both from it.
|
|
85
|
+
2. **`redistribute_standalone_ok` (copyright) ≠ `cache_bytes_ok` (TOS/operational).** e.g.
|
|
86
|
+
Freesound CC0 is legally redistributable yet its API TOS forbids caching → store
|
|
87
|
+
**by-reference**. Get this wrong and you either break the law or break the TOS.
|
|
88
|
+
3. **The license gate is fail-closed.** `keep(record, IntendedUse)` runs as a hard gate
|
|
89
|
+
*before* verification/ranking in the agent's `decide()`; unknown rights → reject.
|
|
90
|
+
4. **AI-generated audio is disclosed & watermarked.** Watermark generations (AudioSeal),
|
|
91
|
+
optionally write C2PA, and honor the **EU AI Act Art. 50** disclosure deadline
|
|
92
|
+
(**2 Aug 2026**). Tracked in #9.
|
|
93
|
+
5. **The eval gate guards regressions** (once #10 lands): no silent `nDCG@10` drop.
|
|
94
|
+
|
|
95
|
+
## Reference
|
|
96
|
+
|
|
97
|
+
Depth for any subtask is in the matching research report in `misc/docs/research/`
|
|
98
|
+
(`01` sources · `02` generation · `03` recognition · `04` index · `05` agent · `06`
|
|
99
|
+
weave · `07` licensing · `08` eval · `09` audio/storage · `11` corpora · `12`
|
|
100
|
+
meta-scan · **`10` = authoritative architecture**). For adding a source, use the
|
|
101
|
+
**`foley-dev-add-source`** skill.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
*.ipynb linguist-documentation
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# wads CI — calls the reusable workflow hosted in i2mint/wads.
|
|
2
|
+
#
|
|
3
|
+
# All configuration comes from this repo's pyproject.toml [tool.wads.ci.*].
|
|
4
|
+
# To customize the workflow itself (rare), replace this file with the
|
|
5
|
+
# full inline template `wads/data/github_ci_uv.yml` from i2mint/wads.
|
|
6
|
+
#
|
|
7
|
+
# Pinning: `@master` floats with wads. If you need version stability for
|
|
8
|
+
# a release-sensitive repo, change `@master` to a wads tag (e.g. `@v0.1.81`).
|
|
9
|
+
# CI failure does not block a published release — it blocks the publish
|
|
10
|
+
# step itself — so floating master is generally safe.
|
|
11
|
+
#
|
|
12
|
+
# Permissions: GitHub validates that the caller grants AT LEAST the
|
|
13
|
+
# permissions any job in the called workflow requests — at workflow-parse
|
|
14
|
+
# time, not at run-time, even if the job would be skipped via `if:`.
|
|
15
|
+
# The reusable workflow needs:
|
|
16
|
+
# contents: write for the publish job's version-bump push-back
|
|
17
|
+
# and for the github-pages job's gh-pages branch push
|
|
18
|
+
# pages: write for the github-pages job's REST API Pages config
|
|
19
|
+
# Both default to `write` on org-account GITHUB_TOKEN and need to be
|
|
20
|
+
# granted explicitly on personal-account callers (where the default is
|
|
21
|
+
# read-only). No `id-token: write` needed — the publish-github-pages
|
|
22
|
+
# action uses peaceiris/actions-gh-pages (branch-based) + REST API,
|
|
23
|
+
# not the OIDC `actions/deploy-pages` flow.
|
|
24
|
+
name: Continuous Integration
|
|
25
|
+
on: [push, pull_request]
|
|
26
|
+
jobs:
|
|
27
|
+
ci:
|
|
28
|
+
uses: i2mint/wads/.github/workflows/uv-ci.yml@master
|
|
29
|
+
permissions:
|
|
30
|
+
contents: write
|
|
31
|
+
pages: write
|
|
32
|
+
# Explicit pass-through (not `secrets: inherit`) because `inherit` does
|
|
33
|
+
# not reliably propagate caller-repo secrets to a reusable workflow owned
|
|
34
|
+
# by a different account (verified empirically: personal-account caller +
|
|
35
|
+
# i2mint-org workflow → `${{ secrets.PYPI_PASSWORD }}` resolved to empty).
|
|
36
|
+
#
|
|
37
|
+
# This list is the per-repo *transport*: it should contain PYPI_PASSWORD
|
|
38
|
+
# (for publishing) plus every secret your tests/CI need. It is generated
|
|
39
|
+
# from [tool.wads.ci.env] in pyproject.toml. To add one, run
|
|
40
|
+
# wads-secrets add VAR_NAME # updates pyproject + this block
|
|
41
|
+
# or just append a line below. *Which* of these become job env vars (and
|
|
42
|
+
# which are required) is controlled by [tool.wads.ci.env] — passing a
|
|
43
|
+
# secret here does not by itself put it in the environment.
|
|
44
|
+
#
|
|
45
|
+
# A secret name must also be declared in the reusable workflow's superset
|
|
46
|
+
# (wads/ci_secrets.py). `wads-secrets add` warns if it is not.
|
|
47
|
+
secrets:
|
|
48
|
+
PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
|
foley-0.0.2/.gitignore
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
.claude/handoffs/
|
|
2
|
+
.claude/scratch/
|
|
3
|
+
|
|
4
|
+
# Byte-compiled / optimized / DLL files
|
|
5
|
+
__pycache__/
|
|
6
|
+
*.py[cod]
|
|
7
|
+
*$py.class
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
.DS_Store
|
|
11
|
+
# C extensions
|
|
12
|
+
*.so
|
|
13
|
+
|
|
14
|
+
# TLS certificates
|
|
15
|
+
## Ignore all PEM files anywhere
|
|
16
|
+
*.pem
|
|
17
|
+
## Also ignore any certs directory
|
|
18
|
+
certs/
|
|
19
|
+
|
|
20
|
+
# Distribution / packaging
|
|
21
|
+
.Python
|
|
22
|
+
build/
|
|
23
|
+
develop-eggs/
|
|
24
|
+
dist/
|
|
25
|
+
downloads/
|
|
26
|
+
eggs/
|
|
27
|
+
.eggs/
|
|
28
|
+
lib/
|
|
29
|
+
lib64/
|
|
30
|
+
parts/
|
|
31
|
+
sdist/
|
|
32
|
+
var/
|
|
33
|
+
wheels/
|
|
34
|
+
*.egg-info/
|
|
35
|
+
.installed.cfg
|
|
36
|
+
*.egg
|
|
37
|
+
MANIFEST
|
|
38
|
+
_build
|
|
39
|
+
|
|
40
|
+
# PyInstaller
|
|
41
|
+
# Usually these files are written by a python script from a template
|
|
42
|
+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
43
|
+
*.manifest
|
|
44
|
+
*.spec
|
|
45
|
+
|
|
46
|
+
# Installer logs
|
|
47
|
+
pip-log.txt
|
|
48
|
+
pip-delete-this-directory.txt
|
|
49
|
+
|
|
50
|
+
# Unit test / coverage reports
|
|
51
|
+
htmlcov/
|
|
52
|
+
.tox/
|
|
53
|
+
.coverage
|
|
54
|
+
.coverage.*
|
|
55
|
+
.cache
|
|
56
|
+
nosetests.xml
|
|
57
|
+
coverage.xml
|
|
58
|
+
*.cover
|
|
59
|
+
.hypothesis/
|
|
60
|
+
.pytest_cache/
|
|
61
|
+
|
|
62
|
+
# Translations
|
|
63
|
+
*.mo
|
|
64
|
+
*.pot
|
|
65
|
+
|
|
66
|
+
# Django stuff:
|
|
67
|
+
*.log
|
|
68
|
+
local_settings.py
|
|
69
|
+
db.sqlite3
|
|
70
|
+
|
|
71
|
+
# Flask stuff:
|
|
72
|
+
instance/
|
|
73
|
+
.webassets-cache
|
|
74
|
+
|
|
75
|
+
# Scrapy stuff:
|
|
76
|
+
.scrapy
|
|
77
|
+
|
|
78
|
+
# Sphinx documentation
|
|
79
|
+
docs/_build/
|
|
80
|
+
docs/*
|
|
81
|
+
|
|
82
|
+
# PyBuilder
|
|
83
|
+
target/
|
|
84
|
+
|
|
85
|
+
# Jupyter Notebook
|
|
86
|
+
.ipynb_checkpoints
|
|
87
|
+
|
|
88
|
+
# pyenv
|
|
89
|
+
.python-version
|
|
90
|
+
|
|
91
|
+
# celery beat schedule file
|
|
92
|
+
celerybeat-schedule
|
|
93
|
+
|
|
94
|
+
# SageMath parsed files
|
|
95
|
+
*.sage.py
|
|
96
|
+
|
|
97
|
+
# Environments
|
|
98
|
+
.env
|
|
99
|
+
.venv
|
|
100
|
+
env/
|
|
101
|
+
venv/
|
|
102
|
+
ENV/
|
|
103
|
+
env.bak/
|
|
104
|
+
venv.bak/
|
|
105
|
+
|
|
106
|
+
# Spyder project settings
|
|
107
|
+
.spyderproject
|
|
108
|
+
.spyproject
|
|
109
|
+
|
|
110
|
+
# Rope project settings
|
|
111
|
+
.ropeproject
|
|
112
|
+
|
|
113
|
+
# mkdocs documentation
|
|
114
|
+
/site
|
|
115
|
+
|
|
116
|
+
# mypy
|
|
117
|
+
.mypy_cache/
|
|
118
|
+
|
|
119
|
+
# PyCharm
|
|
120
|
+
.idea
|
|
121
|
+
.claude/skills/.system/
|
foley-0.0.2/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Thor Whalen
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
foley-0.0.2/PKG-INFO
ADDED
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: foley
|
|
3
|
+
Version: 0.0.2
|
|
4
|
+
Summary: Unified facade for sourcing, indexing, and searching sound effects to weave into narration
|
|
5
|
+
Project-URL: Homepage, https://github.com/thorwhalen/foley
|
|
6
|
+
Project-URL: Repository, https://github.com/thorwhalen/foley
|
|
7
|
+
Project-URL: Documentation, https://thorwhalen.github.io/foley
|
|
8
|
+
Author: Thor Whalen
|
|
9
|
+
License: mit
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: audio,audio-retrieval,clap,foley,narration,sfx,sound-design,sound-effects,text-to-audio
|
|
12
|
+
Requires-Python: >=3.10
|
|
13
|
+
Requires-Dist: config2py
|
|
14
|
+
Requires-Dist: dol
|
|
15
|
+
Provides-Extra: audio
|
|
16
|
+
Requires-Dist: librosa; extra == 'audio'
|
|
17
|
+
Requires-Dist: numpy; extra == 'audio'
|
|
18
|
+
Requires-Dist: pyloudnorm; extra == 'audio'
|
|
19
|
+
Requires-Dist: soundfile; extra == 'audio'
|
|
20
|
+
Requires-Dist: soxr; extra == 'audio'
|
|
21
|
+
Provides-Extra: clap
|
|
22
|
+
Requires-Dist: torch; extra == 'clap'
|
|
23
|
+
Requires-Dist: transformers>=4.40; extra == 'clap'
|
|
24
|
+
Provides-Extra: dev
|
|
25
|
+
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
|
|
26
|
+
Requires-Dist: pytest>=7.0; extra == 'dev'
|
|
27
|
+
Requires-Dist: ruff>=0.1.0; extra == 'dev'
|
|
28
|
+
Provides-Extra: docs
|
|
29
|
+
Requires-Dist: sphinx-rtd-theme>=1.0; extra == 'docs'
|
|
30
|
+
Requires-Dist: sphinx>=6.0; extra == 'docs'
|
|
31
|
+
Provides-Extra: index
|
|
32
|
+
Requires-Dist: lancedb>=0.13; extra == 'index'
|
|
33
|
+
Provides-Extra: index-sqlite
|
|
34
|
+
Requires-Dist: sqlite-vec>=0.1.0; extra == 'index-sqlite'
|
|
35
|
+
Provides-Extra: test
|
|
36
|
+
Requires-Dist: numpy; extra == 'test'
|
|
37
|
+
Requires-Dist: pyloudnorm; extra == 'test'
|
|
38
|
+
Requires-Dist: pytest-cov>=4.0; extra == 'test'
|
|
39
|
+
Requires-Dist: pytest>=7.0; extra == 'test'
|
|
40
|
+
Requires-Dist: soundfile; extra == 'test'
|
|
41
|
+
Requires-Dist: soxr; extra == 'test'
|
|
42
|
+
Description-Content-Type: text/markdown
|
|
43
|
+
|
|
44
|
+
# foley
|
|
45
|
+
|
|
46
|
+
Find (or generate) the **right sound effect for a moment of narration** — and weave it in.
|
|
47
|
+
|
|
48
|
+
foley is a **retrieval-first façade for sound effects**: one simple surface over many
|
|
49
|
+
sound *sources* (your own library, service APIs, and generative-AI models), a searchable
|
|
50
|
+
*index* of every sound (by keyword **and** meaning), an *agent* that picks the right
|
|
51
|
+
sound for a narrative context, and a *compositor* that places it under the voice.
|
|
52
|
+
|
|
53
|
+
It's the SFX sibling of [`arioso`](https://github.com/thorwhalen/arioso) (a unified
|
|
54
|
+
façade over AI music-generation backends): same discipline — one entry function,
|
|
55
|
+
config-driven plugin adapters, a unified vocabulary translated per-backend, zero
|
|
56
|
+
required core deps with lazy optional-deps — but centered on **search** rather than
|
|
57
|
+
generation, with generation as just one of several sources.
|
|
58
|
+
|
|
59
|
+
> **Status: design-stage.** The architecture is researched and specified; the code is
|
|
60
|
+
> being built. The API below is the *intended* surface (see the roadmap for what's
|
|
61
|
+
> live). Follow along in [`misc/docs/`](misc/docs/).
|
|
62
|
+
|
|
63
|
+
## The idea
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
import foley
|
|
67
|
+
|
|
68
|
+
# The headline — right sounds for a narrative context:
|
|
69
|
+
# decompose the passage into salient sound events → search → verify → decide
|
|
70
|
+
candidates = foley.find("She pushed open the heavy oak door; rain hammered outside.")
|
|
71
|
+
|
|
72
|
+
# Direct hybrid search of your library (text query or a reference clip)
|
|
73
|
+
hits = foley.search("distant thunder rumble", k=10, commercial_ok=True)
|
|
74
|
+
|
|
75
|
+
# Generate a sound when nothing fits (arioso-style; pluggable backends)
|
|
76
|
+
clip = foley.generate("a single wooden door creak", backend="stable_audio_open", duration=3)
|
|
77
|
+
|
|
78
|
+
# Grow the library — ingest auto-tags, captions, and embeds every file
|
|
79
|
+
foley.ingest("~/my_sounds/")
|
|
80
|
+
foley.add_from("freesound", query="ocean waves", license="cc0")
|
|
81
|
+
|
|
82
|
+
# Compose (future): render the sound design onto a narration
|
|
83
|
+
foley.weave(narration_audio, candidates)
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## How it works — four stages
|
|
87
|
+
|
|
88
|
+
| Stage | What it does | Built on |
|
|
89
|
+
|-------|--------------|----------|
|
|
90
|
+
| **Source** | your own files · Freesound (CC0) · generate (Stable Audio Open / ElevenLabs) | config-driven adapters, per-sound license tracking |
|
|
91
|
+
| **Index** | probe → tag → caption → embed every sound; hybrid keyword+semantic search | PANNs · CLAP · EnCLAP · LanceDB (local→cloud via `dol`) |
|
|
92
|
+
| **Select** | decompose a narrative context → search → verify → generate-or-retrieve | CLAP retrieval + LLM decomposition + a verification ladder |
|
|
93
|
+
| **Weave** | align to the voice, duck, place, master *(future)* | forced-alignment · LUFS/EBU-R128 |
|
|
94
|
+
|
|
95
|
+
The selection tools publish as an **MCP server** (via `py2mcp`) so the same capabilities
|
|
96
|
+
drive the agent, a CLI, and external hosts.
|
|
97
|
+
|
|
98
|
+
## Design & research
|
|
99
|
+
|
|
100
|
+
foley's design is grounded in five research reports (unified, cited):
|
|
101
|
+
|
|
102
|
+
- [Design](misc/docs/design.md) · [Roadmap](misc/docs/roadmap.md)
|
|
103
|
+
- [Research reports](misc/docs/research/) — sound sources, SFX generation, recognition/
|
|
104
|
+
tagging, embeddings/indexing, and the context-retrieval agent
|
|
105
|
+
- [Deep-research prompt library](misc/docs/research/deep-research-prompts.md) — 12
|
|
106
|
+
ready-to-run prompts covering every dimension of the project
|
|
107
|
+
|
|
108
|
+
## Install
|
|
109
|
+
|
|
110
|
+
```
|
|
111
|
+
pip install foley
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
(Optional per-capability extras — `foley[freesound]`, `foley[clap]`, `foley[stable-audio]`,
|
|
115
|
+
`foley[index]`, … — are added as each subsystem lands.)
|
|
116
|
+
|
|
117
|
+
## License
|
|
118
|
+
|
|
119
|
+
MIT
|
foley-0.0.2/README.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# foley
|
|
2
|
+
|
|
3
|
+
Find (or generate) the **right sound effect for a moment of narration** — and weave it in.
|
|
4
|
+
|
|
5
|
+
foley is a **retrieval-first façade for sound effects**: one simple surface over many
|
|
6
|
+
sound *sources* (your own library, service APIs, and generative-AI models), a searchable
|
|
7
|
+
*index* of every sound (by keyword **and** meaning), an *agent* that picks the right
|
|
8
|
+
sound for a narrative context, and a *compositor* that places it under the voice.
|
|
9
|
+
|
|
10
|
+
It's the SFX sibling of [`arioso`](https://github.com/thorwhalen/arioso) (a unified
|
|
11
|
+
façade over AI music-generation backends): same discipline — one entry function,
|
|
12
|
+
config-driven plugin adapters, a unified vocabulary translated per-backend, zero
|
|
13
|
+
required core deps with lazy optional-deps — but centered on **search** rather than
|
|
14
|
+
generation, with generation as just one of several sources.
|
|
15
|
+
|
|
16
|
+
> **Status: design-stage.** The architecture is researched and specified; the code is
|
|
17
|
+
> being built. The API below is the *intended* surface (see the roadmap for what's
|
|
18
|
+
> live). Follow along in [`misc/docs/`](misc/docs/).
|
|
19
|
+
|
|
20
|
+
## The idea
|
|
21
|
+
|
|
22
|
+
```python
|
|
23
|
+
import foley
|
|
24
|
+
|
|
25
|
+
# The headline — right sounds for a narrative context:
|
|
26
|
+
# decompose the passage into salient sound events → search → verify → decide
|
|
27
|
+
candidates = foley.find("She pushed open the heavy oak door; rain hammered outside.")
|
|
28
|
+
|
|
29
|
+
# Direct hybrid search of your library (text query or a reference clip)
|
|
30
|
+
hits = foley.search("distant thunder rumble", k=10, commercial_ok=True)
|
|
31
|
+
|
|
32
|
+
# Generate a sound when nothing fits (arioso-style; pluggable backends)
|
|
33
|
+
clip = foley.generate("a single wooden door creak", backend="stable_audio_open", duration=3)
|
|
34
|
+
|
|
35
|
+
# Grow the library — ingest auto-tags, captions, and embeds every file
|
|
36
|
+
foley.ingest("~/my_sounds/")
|
|
37
|
+
foley.add_from("freesound", query="ocean waves", license="cc0")
|
|
38
|
+
|
|
39
|
+
# Compose (future): render the sound design onto a narration
|
|
40
|
+
foley.weave(narration_audio, candidates)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## How it works — four stages
|
|
44
|
+
|
|
45
|
+
| Stage | What it does | Built on |
|
|
46
|
+
|-------|--------------|----------|
|
|
47
|
+
| **Source** | your own files · Freesound (CC0) · generate (Stable Audio Open / ElevenLabs) | config-driven adapters, per-sound license tracking |
|
|
48
|
+
| **Index** | probe → tag → caption → embed every sound; hybrid keyword+semantic search | PANNs · CLAP · EnCLAP · LanceDB (local→cloud via `dol`) |
|
|
49
|
+
| **Select** | decompose a narrative context → search → verify → generate-or-retrieve | CLAP retrieval + LLM decomposition + a verification ladder |
|
|
50
|
+
| **Weave** | align to the voice, duck, place, master *(future)* | forced-alignment · LUFS/EBU-R128 |
|
|
51
|
+
|
|
52
|
+
The selection tools publish as an **MCP server** (via `py2mcp`) so the same capabilities
|
|
53
|
+
drive the agent, a CLI, and external hosts.
|
|
54
|
+
|
|
55
|
+
## Design & research
|
|
56
|
+
|
|
57
|
+
foley's design is grounded in five research reports (unified, cited):
|
|
58
|
+
|
|
59
|
+
- [Design](misc/docs/design.md) · [Roadmap](misc/docs/roadmap.md)
|
|
60
|
+
- [Research reports](misc/docs/research/) — sound sources, SFX generation, recognition/
|
|
61
|
+
tagging, embeddings/indexing, and the context-retrieval agent
|
|
62
|
+
- [Deep-research prompt library](misc/docs/research/deep-research-prompts.md) — 12
|
|
63
|
+
ready-to-run prompts covering every dimension of the project
|
|
64
|
+
|
|
65
|
+
## Install
|
|
66
|
+
|
|
67
|
+
```
|
|
68
|
+
pip install foley
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
(Optional per-capability extras — `foley[freesound]`, `foley[clap]`, `foley[stable-audio]`,
|
|
72
|
+
`foley[index]`, … — are added as each subsystem lands.)
|
|
73
|
+
|
|
74
|
+
## License
|
|
75
|
+
|
|
76
|
+
MIT
|