paces 0.0.1__tar.gz → 0.0.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- paces-0.0.2/.gitignore +26 -0
- paces-0.0.2/LICENSE +21 -0
- paces-0.0.2/PKG-INFO +118 -0
- paces-0.0.2/README.md +92 -0
- paces-0.0.2/docs/README.md +143 -0
- paces-0.0.2/docs/alignment/README.md +47 -0
- paces-0.0.2/docs/poc-reference/README.md +46 -0
- paces-0.0.2/docs/poc-reference/artifacts/clips.json +197 -0
- paces-0.0.2/paces/__init__.py +92 -0
- paces-0.0.2/paces/__main__.py +31 -0
- paces-0.0.2/paces/model.py +353 -0
- paces-0.0.2/paces/projection.py +96 -0
- paces-0.0.2/paces/render.py +293 -0
- paces-0.0.2/paces/segmenters.py +443 -0
- paces-0.0.2/paces/tools.py +221 -0
- paces-0.0.2/pyproject.toml +188 -0
- paces-0.0.2/tests/data/routine.json +93 -0
- paces-0.0.2/tests/poc_fixture.py +252 -0
- paces-0.0.2/tests/test_roundtrip_poc.py +182 -0
- paces-0.0.2/tests/test_segment.py +147 -0
- paces-0.0.2/tests/test_smoke.py +105 -0
- paces-0.0.1/PKG-INFO +0 -19
- paces-0.0.1/README.md +0 -9
- paces-0.0.1/paces/__init__.py +0 -3
- paces-0.0.1/paces.egg-info/PKG-INFO +0 -19
- paces-0.0.1/paces.egg-info/SOURCES.txt +0 -7
- paces-0.0.1/paces.egg-info/dependency_links.txt +0 -1
- paces-0.0.1/paces.egg-info/top_level.txt +0 -1
- paces-0.0.1/pyproject.toml +0 -18
- paces-0.0.1/setup.cfg +0 -4
paces-0.0.2/.gitignore
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
dist/
|
|
6
|
+
build/
|
|
7
|
+
.venv/
|
|
8
|
+
.pytest_cache/
|
|
9
|
+
.ruff_cache/
|
|
10
|
+
.coverage
|
|
11
|
+
coverage.xml
|
|
12
|
+
|
|
13
|
+
# Editors / OS
|
|
14
|
+
.DS_Store
|
|
15
|
+
.idea/
|
|
16
|
+
.vscode/
|
|
17
|
+
|
|
18
|
+
# Claude Code session droppings (dev skills et al. stay tracked)
|
|
19
|
+
.claude/.system/
|
|
20
|
+
.claude/handoffs/
|
|
21
|
+
|
|
22
|
+
# Media and evidence stores never live in the repo
|
|
23
|
+
media/
|
|
24
|
+
*.annot
|
|
25
|
+
*.annot.sqlite
|
|
26
|
+
scrap/
|
paces-0.0.2/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Thor Whalen
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
paces-0.0.2/PKG-INFO
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: paces
|
|
3
|
+
Version: 0.0.2
|
|
4
|
+
Summary: Turn instructional media into structured, interactive learning material
|
|
5
|
+
Project-URL: Homepage, https://github.com/thorwhalen/paces
|
|
6
|
+
Project-URL: Repository, https://github.com/thorwhalen/paces
|
|
7
|
+
Project-URL: Documentation, https://thorwhalen.github.io/paces
|
|
8
|
+
Author: Thor Whalen
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: dance,instructional-video,learning-material,practice,segmentation,steps,tutorial
|
|
12
|
+
Requires-Python: >=3.10
|
|
13
|
+
Requires-Dist: pydantic>=2.6
|
|
14
|
+
Provides-Extra: cli
|
|
15
|
+
Requires-Dist: argcomplete>=3; extra == 'cli'
|
|
16
|
+
Requires-Dist: argh>=0.30; extra == 'cli'
|
|
17
|
+
Provides-Extra: dev
|
|
18
|
+
Requires-Dist: argh>=0.30; extra == 'dev'
|
|
19
|
+
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
|
|
20
|
+
Requires-Dist: pytest>=7.0; extra == 'dev'
|
|
21
|
+
Requires-Dist: ruff>=0.1.0; extra == 'dev'
|
|
22
|
+
Provides-Extra: docs
|
|
23
|
+
Requires-Dist: sphinx-rtd-theme>=1.0; extra == 'docs'
|
|
24
|
+
Requires-Dist: sphinx>=6.0; extra == 'docs'
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
|
|
27
|
+
# paces
|
|
28
|
+
|
|
29
|
+
Turn instructional media into structured, interactive learning material.
|
|
30
|
+
*Put it through its paces.*
|
|
31
|
+
|
|
32
|
+
Take a video of someone teaching something — a dance routine, a kata, a
|
|
33
|
+
recipe — plus, optionally, notes and a steering prompt. `paces` segments it
|
|
34
|
+
into named steps, builds a structured **step document** (an AST for
|
|
35
|
+
step-by-step instruction), and renders that into learning material: a
|
|
36
|
+
practice page with counts and deep links today, other guides later.
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
pip install paces
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Quick example
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
from paces import segment, to_document, render_html
|
|
46
|
+
|
|
47
|
+
seg = segment(
|
|
48
|
+
"https://youtu.be/q_TUyxUhoEw",
|
|
49
|
+
steps=[
|
|
50
|
+
("Mise en place", 2),
|
|
51
|
+
("Pas pieds pointe et ronde", 6),
|
|
52
|
+
("Soleil avec les bras", 4),
|
|
53
|
+
("Déhanchés", 8),
|
|
54
|
+
],
|
|
55
|
+
grid={"unit": "eight", "subdivisions": 8, "tempoBpm": "129.2", "origin": "51.2"},
|
|
56
|
+
)
|
|
57
|
+
doc = to_document(
|
|
58
|
+
seg,
|
|
59
|
+
doc_id="que-calor",
|
|
60
|
+
title="Chorégraphie Que Calor",
|
|
61
|
+
source="https://youtu.be/q_TUyxUhoEw",
|
|
62
|
+
)
|
|
63
|
+
open("page.html", "w").write(render_html(doc))
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The page lists every step with its counts, links each one back into the video
|
|
67
|
+
(both the at-tempo run-through and the slow breakdown, when both are known),
|
|
68
|
+
and — because the document carries a metric grid — includes a count-along
|
|
69
|
+
transport that paces you through the routine at the measured tempo.
|
|
70
|
+
|
|
71
|
+
Same thing from the shell:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
paces segment VIDEO_URL --steps steps.json --grid grid.json --output seg.json
|
|
75
|
+
paces to-document seg.json --source VIDEO_URL --title "My routine" --output document.json
|
|
76
|
+
paces render document.json --output page.html
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## How it thinks
|
|
80
|
+
|
|
81
|
+
**Analysis and rendering are separate phases** with a serialisable document
|
|
82
|
+
between them — like a parser emitting an AST and a backend interpreting it.
|
|
83
|
+
Renderers depend on the document, never on the analyser.
|
|
84
|
+
|
|
85
|
+
**Segmentation is a seam, not a stage.** `segment(media, segmenter=...)` —
|
|
86
|
+
segmenters are registered capabilities, the default follows from what is
|
|
87
|
+
present, and "the user typed the boundaries" is a first-class segmenter, not
|
|
88
|
+
a fallback. A segmenter that cannot *name* steps returns honest unnamed
|
|
89
|
+
boundaries (`flags: ['naming-abstained']`) rather than inventing names.
|
|
90
|
+
|
|
91
|
+
**The document keeps what the learner actually counts.** A dance step lasts
|
|
92
|
+
"4 eights", not "14.86 seconds" — seconds are derived from the metric grid
|
|
93
|
+
(tempo + origin), never stored. A step can have *several* source spans (the
|
|
94
|
+
run-through and the breakdown are the same step seen twice). Uncertainty is
|
|
95
|
+
content (`OpenQuestion`), and human edits are protected from regeneration
|
|
96
|
+
(`Lock`).
|
|
97
|
+
|
|
98
|
+
## The pieces
|
|
99
|
+
|
|
100
|
+
| you want | reach for |
|
|
101
|
+
|---|---|
|
|
102
|
+
| cut media into steps | `segment(media, steps=..., grid=...)` → `Segmentation` |
|
|
103
|
+
| explicit/human boundaries | `segment(media, boundaries=[...], steps=[names])` |
|
|
104
|
+
| the committed artifact | `to_document(seg, ...)` → `StepDocument` |
|
|
105
|
+
| a practice page | `render_html(doc)` |
|
|
106
|
+
| wall-clock times from counts | `resolve(doc)` |
|
|
107
|
+
| sanity checks | `validate_document(doc)` |
|
|
108
|
+
| what segmenters exist | `capabilities()` / `paces list-segmenters` |
|
|
109
|
+
| add a segmenter | `register(Capability(name=..., gives="segmentation", target="mymod:fn", needs={...}))` — a new file, nothing edited |
|
|
110
|
+
|
|
111
|
+
## Status
|
|
112
|
+
|
|
113
|
+
Young and moving. The document schema is validated by round-tripping a real
|
|
114
|
+
proof of concept ([an interactive dance-practice
|
|
115
|
+
page](https://thorwhalen.com/que_calor_dance/)) through it — see
|
|
116
|
+
`tests/test_roundtrip_poc.py`. Media derivation (auto-cropped looping clips),
|
|
117
|
+
intrinsic segmenters (scene/beat/speech detection), and the evidence layer
|
|
118
|
+
are designed (see `docs/`) and arrive next.
|
paces-0.0.2/README.md
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# paces
|
|
2
|
+
|
|
3
|
+
Turn instructional media into structured, interactive learning material.
|
|
4
|
+
*Put it through its paces.*
|
|
5
|
+
|
|
6
|
+
Take a video of someone teaching something — a dance routine, a kata, a
|
|
7
|
+
recipe — plus, optionally, notes and a steering prompt. `paces` segments it
|
|
8
|
+
into named steps, builds a structured **step document** (an AST for
|
|
9
|
+
step-by-step instruction), and renders that into learning material: a
|
|
10
|
+
practice page with counts and deep links today, other guides later.
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
pip install paces
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## Quick example
|
|
17
|
+
|
|
18
|
+
```python
|
|
19
|
+
from paces import segment, to_document, render_html
|
|
20
|
+
|
|
21
|
+
seg = segment(
|
|
22
|
+
"https://youtu.be/q_TUyxUhoEw",
|
|
23
|
+
steps=[
|
|
24
|
+
("Mise en place", 2),
|
|
25
|
+
("Pas pieds pointe et ronde", 6),
|
|
26
|
+
("Soleil avec les bras", 4),
|
|
27
|
+
("Déhanchés", 8),
|
|
28
|
+
],
|
|
29
|
+
grid={"unit": "eight", "subdivisions": 8, "tempoBpm": "129.2", "origin": "51.2"},
|
|
30
|
+
)
|
|
31
|
+
doc = to_document(
|
|
32
|
+
seg,
|
|
33
|
+
doc_id="que-calor",
|
|
34
|
+
title="Chorégraphie Que Calor",
|
|
35
|
+
source="https://youtu.be/q_TUyxUhoEw",
|
|
36
|
+
)
|
|
37
|
+
open("page.html", "w").write(render_html(doc))
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The page lists every step with its counts, links each one back into the video
|
|
41
|
+
(both the at-tempo run-through and the slow breakdown, when both are known),
|
|
42
|
+
and — because the document carries a metric grid — includes a count-along
|
|
43
|
+
transport that paces you through the routine at the measured tempo.
|
|
44
|
+
|
|
45
|
+
Same thing from the shell:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
paces segment VIDEO_URL --steps steps.json --grid grid.json --output seg.json
|
|
49
|
+
paces to-document seg.json --source VIDEO_URL --title "My routine" --output document.json
|
|
50
|
+
paces render document.json --output page.html
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## How it thinks
|
|
54
|
+
|
|
55
|
+
**Analysis and rendering are separate phases** with a serialisable document
|
|
56
|
+
between them — like a parser emitting an AST and a backend interpreting it.
|
|
57
|
+
Renderers depend on the document, never on the analyser.
|
|
58
|
+
|
|
59
|
+
**Segmentation is a seam, not a stage.** `segment(media, segmenter=...)` —
|
|
60
|
+
segmenters are registered capabilities, the default follows from what is
|
|
61
|
+
present, and "the user typed the boundaries" is a first-class segmenter, not
|
|
62
|
+
a fallback. A segmenter that cannot *name* steps returns honest unnamed
|
|
63
|
+
boundaries (`flags: ['naming-abstained']`) rather than inventing names.
|
|
64
|
+
|
|
65
|
+
**The document keeps what the learner actually counts.** A dance step lasts
|
|
66
|
+
"4 eights", not "14.86 seconds" — seconds are derived from the metric grid
|
|
67
|
+
(tempo + origin), never stored. A step can have *several* source spans (the
|
|
68
|
+
run-through and the breakdown are the same step seen twice). Uncertainty is
|
|
69
|
+
content (`OpenQuestion`), and human edits are protected from regeneration
|
|
70
|
+
(`Lock`).
|
|
71
|
+
|
|
72
|
+
## The pieces
|
|
73
|
+
|
|
74
|
+
| you want | reach for |
|
|
75
|
+
|---|---|
|
|
76
|
+
| cut media into steps | `segment(media, steps=..., grid=...)` → `Segmentation` |
|
|
77
|
+
| explicit/human boundaries | `segment(media, boundaries=[...], steps=[names])` |
|
|
78
|
+
| the committed artifact | `to_document(seg, ...)` → `StepDocument` |
|
|
79
|
+
| a practice page | `render_html(doc)` |
|
|
80
|
+
| wall-clock times from counts | `resolve(doc)` |
|
|
81
|
+
| sanity checks | `validate_document(doc)` |
|
|
82
|
+
| what segmenters exist | `capabilities()` / `paces list-segmenters` |
|
|
83
|
+
| add a segmenter | `register(Capability(name=..., gives="segmentation", target="mymod:fn", needs={...}))` — a new file, nothing edited |
|
|
84
|
+
|
|
85
|
+
## Status
|
|
86
|
+
|
|
87
|
+
Young and moving. The document schema is validated by round-tripping a real
|
|
88
|
+
proof of concept ([an interactive dance-practice
|
|
89
|
+
page](https://thorwhalen.com/que_calor_dance/)) through it — see
|
|
90
|
+
`tests/test_roundtrip_poc.py`. Media derivation (auto-cropped looping clips),
|
|
91
|
+
intrinsic segmenters (scene/beat/speech detection), and the evidence layer
|
|
92
|
+
are designed (see `docs/`) and arrive next.
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
# `paces` — a library for turning instructional video into learning material
|
|
2
|
+
|
|
3
|
+
*Put it through its paces.* The name is settled (`adr/0002`); so is the core abstraction
|
|
4
|
+
(`adr/0003`). These docs were written while the package was still called `stepped` — where the
|
|
5
|
+
prose says "stepped", read `paces`.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## What this is
|
|
10
|
+
|
|
11
|
+
A working proof-of-concept exists. In one session, a YouTube video of a choreographer teaching
|
|
12
|
+
a dance, plus a hand-written HTML aide-mémoire, plus a paragraph of steering prompt, became a
|
|
13
|
+
deployed interactive practice page: <https://thorwhalen.com/que_calor_dance/>.
|
|
14
|
+
|
|
15
|
+
The user now wants that generalised into a library, integrated with the **reelee** /
|
|
16
|
+
`video_gen` fleet. Your job is to research, design and build it. **This folder exists so you
|
|
17
|
+
do not start from scratch.** It records what was built, the parameters that were expensive to
|
|
18
|
+
find, the failures and what each one teaches, the user's own framing of the generalisation,
|
|
19
|
+
and an inventory of what already exists in the fleet.
|
|
20
|
+
|
|
21
|
+
Nothing here is a design you must follow. It is evidence and framing. Argue with it.
|
|
22
|
+
|
|
23
|
+
## Read in this order
|
|
24
|
+
|
|
25
|
+
| | file | why |
|
|
26
|
+
|---|---|---|
|
|
27
|
+
| 1 | **`01-what-was-built.md`** | The POC, factually. Includes the incident list — every failure is a requirement in disguise. |
|
|
28
|
+
| 2 | **`03-design-brief.md`** | The user's own framing: the parse→AST→render metaphor they explicitly asked to have recorded, the three generalisation axes, and the constraints the POC discovered. |
|
|
29
|
+
| 3 | **`04-reelee-core.md`** | What reelee is, what a reelee "genre" is, and what to reuse. **This determines the package boundary**, so read it before deciding anything structural. |
|
|
30
|
+
| 4 | **`07-annotation-model.md`** | The proposed shape of the AST — the contract between analysis and rendering. |
|
|
31
|
+
| 5 | **`05-fleet-inventory.md`** | What already exists across `video_gen`, and honestly which parts are stubs. |
|
|
32
|
+
| 6 | **`02-technical-recipes.md`** | Every technique with working parameters. Reference, not narrative — come back to it when implementing. |
|
|
33
|
+
| 7 | **`06-surfaces-and-conventions.md`** | House style: architecture-first seams, qh, py2mcp, storage, frontend, deploy. |
|
|
34
|
+
| 8 | **`08-naming-candidates.md`** | PyPI-verified name options, and a better word for the "subject" axis. |
|
|
35
|
+
| 9 | **`09-subgenre-candidates.md`** | What to build after dance, and what each choice would force the core to get right. |
|
|
36
|
+
| 10 | **`10-session-archaeology.md`** | The 27 MB session transcript, and how to query it when these docs fall short. |
|
|
37
|
+
| — | **`adr/`** | The decisions. **`0003` `video + segmenter`** — read it before designing the analysis phase. `0002` the name. `0001` the alignment engine (intent) — note that much of what you'd otherwise build already exists in `muvid`, `mixing` and `kodokan`. |
|
|
38
|
+
| — | **`alignment/`** | The research behind that ADR: one file per method family, prepared so you don't start from a literature search. |
|
|
39
|
+
| — | **`KICKOFF.md`** | A paste-ready prompt to start a fresh session on this. |
|
|
40
|
+
| — | **`REGISTRATION.md`** | **One command still pending** to register `paces` in the `video_gen` group, and why `priv pkg add-package` is deliberately deferred. |
|
|
41
|
+
| — | **`poc-reference/`** | The actual scripts and data. Read `poc-reference/README.md` first — several of those files are recorded dead ends. |
|
|
42
|
+
|
|
43
|
+
## The one-paragraph version
|
|
44
|
+
|
|
45
|
+
Take instructional media (a video of someone teaching something) plus optional notes and a
|
|
46
|
+
steering prompt. **Analyse**: extract signals (audio structure, transcript, beat grid, subject
|
|
47
|
+
tracking), segment into named steps, align against the notes, and emit a structured step
|
|
48
|
+
document — an *AST*. **Render**: consume that AST to produce learning material — an
|
|
49
|
+
interactive practice page today, other guides later. The AST is the contract; the analyser and
|
|
50
|
+
the renderers depend on it and not on each other. A shared core handles step-by-step
|
|
51
|
+
instructional content in general (reelee's word: **genre**); *subgenres* — dance, kata,
|
|
52
|
+
recipe, repair — specialise the segmentation signals, the duration unit, and the rendering.
|
|
53
|
+
|
|
54
|
+
## Four findings from the research that change the starting position
|
|
55
|
+
|
|
56
|
+
These came out of `04`–`07` and are worth knowing before you read anything else, because each
|
|
57
|
+
one removes work you might otherwise plan for.
|
|
58
|
+
|
|
59
|
+
1. **"Genre" is an `nw` concept, not a reelee one.** `nw` (`$PP/t/nw`) owns `Project`,
|
|
60
|
+
`Transform`, `Genre`, freshness and jobs, with a real registry
|
|
61
|
+
(`register_genre`, `register_genre_resolver`, `register_genre_initializer`,
|
|
62
|
+
`register_genre_project_factory`). If you want a "step-by-step" genre with a "dance"
|
|
63
|
+
subgenre, that machinery already exists and reelee is a *consumer* of it. `04-reelee-core.md`.
|
|
64
|
+
2. **reelee is deliberately small.** Its own `__init__` says the substance lives in the focused
|
|
65
|
+
packages below it — `lacing`, `falaw`, `nw`, `artful` — and reelee's surface is
|
|
66
|
+
orchestration. That strongly suggests this library is **another focused package below
|
|
67
|
+
reelee**, not a fork of it and not a plugin inside it. Confirm with the user, but start from
|
|
68
|
+
that hypothesis.
|
|
69
|
+
3. **`lacing` is already the annotation substrate**, and reelee already ships
|
|
70
|
+
regenerate-without-losing-human-edits machinery on top of it. That is constraint §5 of
|
|
71
|
+
`03-design-brief.md` — the one that looked hardest — already solved. `07-annotation-model.md`
|
|
72
|
+
recommends owning only a small *document* type and delegating everything below it.
|
|
73
|
+
4. **The step structure was never in `clips.json`.** It was in the page's own
|
|
74
|
+
`const ROUTINE = [...]`; `clips.json` is the *span* table. Reading the POC as a two-layer
|
|
75
|
+
model (steps ↕ spans) rather than one flat list is the single most useful reframe in these
|
|
76
|
+
docs.
|
|
77
|
+
5. **Most of the analysis phase already exists in the fleet, unwired.** `muvid/align.py` is an
|
|
78
|
+
aligner registry with dispatch and a `lacing` writeback; `muvid.footage.select_score` is a
|
|
79
|
+
constrained sequence solver; `mixing.audio` has beat grids, speech/music segmentation and
|
|
80
|
+
cross-correlation offset alignment; `kodokan` has a complete, tested, Apple-Silicon-native
|
|
81
|
+
**pose front-end that has never been pointed at an alignment problem**. `adr/0001` and
|
|
82
|
+
`alignment/00-existing-in-fleet.md` have the map. Do not write a second one of any of these.
|
|
83
|
+
|
|
84
|
+
## What the POC actually proves
|
|
85
|
+
|
|
86
|
+
- Separating analysis from rendering is real, not architectural decoration: the media was
|
|
87
|
+
re-rendered three times and the page a dozen times from unchanged manifests.
|
|
88
|
+
- A step legitimately has **several source spans** (at-tempo run-through *and* slow
|
|
89
|
+
explanation). Six of nine blocks shipped both. A one-span model is wrong on day one.
|
|
90
|
+
- Cheap signals go a long way: a sub-bass energy ratio separates speech from music in five
|
|
91
|
+
lines and did more work than anything else in the session.
|
|
92
|
+
- An LLM looking at timestamped contact sheets makes genuinely good editorial calls about
|
|
93
|
+
which few seconds show a move — and is slow and expensive doing it.
|
|
94
|
+
|
|
95
|
+
## What the POC does *not* prove
|
|
96
|
+
|
|
97
|
+
- **Nothing here segmented a video cold.** The nine steps came from the input document; the
|
|
98
|
+
analysis confirmed, corrected and timed them. Cold segmentation is a different, harder
|
|
99
|
+
problem and the POC has no evidence about it.
|
|
100
|
+
- Everything assumed **one subject**. Two people breaks the crop envelope, the face-slot logic
|
|
101
|
+
and the privacy mask simultaneously.
|
|
102
|
+
- The renderer was a patched copy of the input document. There is no template contract yet.
|
|
103
|
+
|
|
104
|
+
## Suggested first moves
|
|
105
|
+
|
|
106
|
+
1. Read `04-reelee-core.md`, then settle the **integration shape** with the user. The research
|
|
107
|
+
points at "a focused package below reelee, registering an `nw` genre" — test that
|
|
108
|
+
hypothesis rather than assuming it. Boundaries are the expensive thing to move later.
|
|
109
|
+
2. ~~Settle the name~~ — **done, `paces`** (`adr/0002`). Still open: the better word for the
|
|
110
|
+
"subject" axis. Leading candidates in `08-naming-candidates.md §6`: `discipline`, `craft`,
|
|
111
|
+
`grammar`, `idiom`. **Do not run another availability sweep** — 684 names are already
|
|
112
|
+
checked, and that file is now also the name pool for sub-packages this work spawns.
|
|
113
|
+
3. ~~The v1 scope fork~~ — **dissolved by `adr/0003`.** Segmentation is a seam, not a stage:
|
|
114
|
+
`video + segmenter=`, kept open-closed by a strategy pattern. A segmenter's information can
|
|
115
|
+
be **intrinsic** (features of the media), **external-explicit** (coordinates handed over),
|
|
116
|
+
**external-derived** (computed from the surrounding annotations), or **mixed** — and mixed is
|
|
117
|
+
the normal case. Many *intrinsic* segmenters share one shape — featurize → reduce → threshold
|
|
118
|
+
→ regularize — which the package should ship as composable stages so common cases are
|
|
119
|
+
assembled rather than authored. "Ask the user" is first-class. Read that ADR, then
|
|
120
|
+
`alignment/07-segmenter-strategies.md` **with its correction preamble**, then decide which
|
|
121
|
+
two segmenters v1 ships.
|
|
122
|
+
4. Write the AST schema and validate it by **re-expressing the POC's `clips.json` in it**
|
|
123
|
+
(`poc-reference/artifacts/clips.json`). If the dance case does not round-trip, the schema is
|
|
124
|
+
wrong. That is a cheap, real test available on day one.
|
|
125
|
+
5. Only then pick seams and build (`03-design-brief.md §6`, `06-surfaces-and-conventions.md`).
|
|
126
|
+
|
|
127
|
+
## Open questions for the user
|
|
128
|
+
|
|
129
|
+
Collected from across these docs; raise them early rather than guessing.
|
|
130
|
+
|
|
131
|
+
1. ~~Notes required in v1, or cold segmentation?~~ Answered by `adr/0003`.
|
|
132
|
+
2. Integration shape with reelee? (decides the package boundary — the live one)
|
|
133
|
+
3. Is the renderer part of this library or a sibling package?
|
|
134
|
+
4. How much of the analysis may be LLM-driven — what is the acceptable cost per project?
|
|
135
|
+
5. What is the editing story? Human edits must survive re-analysis.
|
|
136
|
+
6. ~~The name~~ (`paces`, `adr/0002`) — but still the word for the "subject/genre" axis.
|
|
137
|
+
|
|
138
|
+
## Provenance
|
|
139
|
+
|
|
140
|
+
Built from session `0f75703c-6761-4aa0-b796-aafe02c94155` (2026-08-27/28). The full transcript
|
|
141
|
+
is on disk and queryable — `10-session-archaeology.md`. Docs `01`, `02`, `03`, `10` and
|
|
142
|
+
`poc-reference/README.md` were written from direct session context; `04`–`09` were researched
|
|
143
|
+
by subagents and each carries its own "verified vs inferred" notes.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Alignment research
|
|
2
|
+
|
|
3
|
+
*What this folder is for: preparation for a possible dedicated **alignment tool** in the
|
|
4
|
+
`video_gen` / reelee fleet — the thing that answers "given some artifacts and some media,
|
|
5
|
+
which span of the media does each artifact correspond to?". The decision to build it is
|
|
6
|
+
recorded as an intent in `../adr/0001-alignment-engine-as-a-fleet-package.md`; **read that
|
|
7
|
+
first**, then come here for the method-by-method detail.*
|
|
8
|
+
|
|
9
|
+
The user's framing of the problem and of the top surface:
|
|
10
|
+
|
|
11
|
+
> *"This is a very common task in our reelee work: being able to match and align things. More
|
|
12
|
+
> specifically … being able to align annotations/artifacts to segments of audio and/or video."*
|
|
13
|
+
>
|
|
14
|
+
> *"I'm imagining that the highest surface would be an agent that can study the context, what
|
|
15
|
+
> is available, and have a list of possible methods/algorithms/tools it could use. It could use
|
|
16
|
+
> beats. It could use the transcribed words (and transcribe if they're not available). It could
|
|
17
|
+
> use some knowledge of the order of the artifacts to be matched. It could use gestures."*
|
|
18
|
+
|
|
19
|
+
## The files
|
|
20
|
+
|
|
21
|
+
| file | what it covers |
|
|
22
|
+
|---|---|
|
|
23
|
+
| **`00-existing-in-fleet.md`** | **Start here.** What the fleet already has. Verdict: the capability doesn't exist as a capability, but nearly every part does — `muvid/align.py` is the v0 registry, `muvid.footage.select_score` is the order-prior solver, `mixing.audio` is the signal layer, `kodokan` is a complete but dormant pose front-end, `lacing` is the settled output substrate. Also argues *separate package vs a module in `lacing`* (§4.3) and concludes separate. |
|
|
24
|
+
| `01-text-to-audio-alignment.md` | ASR with word timings, true forced alignment (CTC, MFA, aeneas), **fuzzy paraphrase→transcript matching** (the common reelee case, and the hard one), VAD, diarization. |
|
|
25
|
+
| `02-music-rhythm-and-structure.md` | Beats, downbeats, tempo and its octave errors, **phase/offset** (tempo gives spacing, not where bar 1 starts), music structure segmentation, music-vs-speech discrimination, audio-to-audio alignment and fingerprinting. |
|
|
26
|
+
| `03-visual-signals.md` | Scene cuts, motion energy and optical flow, **pose estimation and pose→segmentation**, repetition/periodicity detection, gesture and action recognition, hand/object interaction, OCR. This is the file that closes the "we never used gestures" gap. |
|
|
27
|
+
| `04-semantic-and-llm-matching.md` | CLIP/SigLIP frame–text scoring, CLAP for audio, video-text retrieval, the **LLM-over-timestamped-contact-sheets** pattern written up as a reusable technique, transcript structure extraction, and confidence calibration. |
|
|
28
|
+
| `05-sequence-alignment-algorithms.md` | The algorithm layer under everything else: DTW and subsequence DTW, gapped Needleman–Wunsch, CTC segmentation, Viterbi, **change-point detection with a known segment count**, grid fitting for `(offset, period)`, evidence fusion, and how to evaluate an aligner at all. |
|
|
29
|
+
| `06-the-planner-surface.md` | The top surface. Proposes collapsing all five sibling method Protocols into one `Capability(needs, gives, …)` record, a measured ~1.4 s/min context probe, and a deterministic ranked graph walk — with the LLM deliberately outside the control loop. |
|
|
30
|
+
| **`07-segmenter-strategies.md`** | **The `video + segmenter` seam** the user settled on. A catalogue of ~16 video-only segmentation families plus six richer input tiers (step list, structured document, steering prompt, chapters/subtitles/**re-watch heatmap**, ask-the-user, escalation), organised by *what input is present* because that is what picks the default. Then the `Segmenter` protocol — reusing `06`'s `Capability` with **two** new products and **seven** new facts and no new fields — the generated default-selection table, and the recommendation: three segmenters in v1, `novelty-k` as the video-only default, **and it deliberately does not name the steps**. |
|
|
31
|
+
| `04-evidence/` | Real experiment scripts the semantic agent wrote and ran (SigLIP SO400M, CLAP, VLM scoring, calibration, ASR cue extraction). Kept because the numbers in `04-…md` came from them. Scratch quality — read them as evidence, not as library code. |
|
|
32
|
+
|
|
33
|
+
## How to use this
|
|
34
|
+
|
|
35
|
+
These files are a **menu with prices**, not a plan. Each method entry states what signal it
|
|
36
|
+
needs, what it produces, its cost per minute of media, its licence, and the failure mode that
|
|
37
|
+
will bite. The planner file is what turns the menu into a selection procedure.
|
|
38
|
+
|
|
39
|
+
Two things worth internalising before reading in depth, both from the POC that motivated all of
|
|
40
|
+
this (`../01-what-was-built.md`):
|
|
41
|
+
|
|
42
|
+
- **The cheapest signal did the most work.** A five-line sub-bass energy ratio separated
|
|
43
|
+
talking-head from music-with-dancing and made everything downstream tractable. Any design that
|
|
44
|
+
reaches for pose or a VLM first will be slower *and* worse.
|
|
45
|
+
- **No single method was sufficient.** The answer came from combining a cheap audio feature, a
|
|
46
|
+
beat model, ASR, a vision-language judgement, and the order prior. Fusion and confidence are
|
|
47
|
+
not optional extras — they're the point.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# POC reference code and artifacts
|
|
2
|
+
|
|
3
|
+
*What this is: the actual scripts and data from the session that produced
|
|
4
|
+
<https://thorwhalen.com/que_calor_dance/>. **This is not a library and must not be treated as
|
|
5
|
+
one** — it is a pile of session tooling, written to be thrown away, kept because the
|
|
6
|
+
parameters in it were expensive to find. Read it for the recipes, not the structure.
|
|
7
|
+
`../02-technical-recipes.md` explains every one of them.*
|
|
8
|
+
|
|
9
|
+
Everything ran under `~/.pyenv/versions/3.12.12/envs/p12/bin/python`, from a working directory
|
|
10
|
+
containing `source.mp4`.
|
|
11
|
+
|
|
12
|
+
## `tools/` — the pipeline
|
|
13
|
+
|
|
14
|
+
| file | what it does | worth keeping? |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| `sheet.py` | Labelled contact sheet over a time window — `sheet.py START END STEP OUT.jpg [COLS]`. Every tile stamped with its absolute timestamp. | **Yes.** This is how an LLM looks at video. The timestamps are the whole point. |
|
|
17
|
+
| `zsheet.py`, `sheet_crop.py` | Same, cropped to the subject. Written by subagents mid-session to judge a wide shot. | Merge into one tool. |
|
|
18
|
+
| `mkclip.py` | Cut + auto-crop + encode one clip as gif/mp4/webp. Holds `crop_box()`: YOLO person boxes → robust percentile envelope → forced aspect → clamp. | **Yes** — `crop_box()` especially. |
|
|
19
|
+
| `track.py` | Whole-video person tracking at 5 fps into `boxes.npz`. | **No.** Abandoned: far too slow (>45 min on CPU). Kept as a record of the dead end. |
|
|
20
|
+
| `bg.py` | Empty-room plate as an 88th-percentile-per-pixel composite. | Only for *compositing*. Do not use it to *find* the subject — see `../02-technical-recipes.md §7`. |
|
|
21
|
+
| `stylize.py` | The anonymisation pipeline, adapted from kodokan and made streaming. Contains the narrowed head-band fix. | **Yes**, with the licensing caveat. |
|
|
22
|
+
| `build_media.py` / `build_media_styl.py` / `restyle_clips.py` | Batch drivers: manifest → mp4 + gif + poster for every clip. Three generations of the same script. | As a spec for what the render stage does. |
|
|
23
|
+
| `build_page.py` | Renderer. Splices a `MEDIA` map into the source document's own JS and patches its HTML by string surgery. | **Read it, then do the opposite.** It is the clearest possible argument for a real template contract. |
|
|
24
|
+
| `shot.py` | Playwright screenshot + console/HTTP error capture. | **Yes.** Verification, and image generation from the page's own CSS. |
|
|
25
|
+
| `ship.sh` | Rebuild page → refresh the deploy app dir. | Trivial, illustrative. |
|
|
26
|
+
|
|
27
|
+
## `artifacts/` — the intermediate representation
|
|
28
|
+
|
|
29
|
+
| file | what it is |
|
|
30
|
+
|---|---|
|
|
31
|
+
| `clips.json` | **The POC's entire AST.** 15 entries. `src: "RT" \| "BD"` is run-through vs breakdown — the two-passes-over-the-same-material idea in its crudest form. Analysed in `../07-annotation-model.md`. |
|
|
32
|
+
| `crops.json` | `{clip_id: [x, y, w, h]}` — a derived cache, correctly kept out of the semantic model. |
|
|
33
|
+
| `transcript.json` | mlx-whisper output for the whole 651 s. Note how it degrades over the music section (hundreds of near-empty segments, invented numbers) and is excellent over the spoken breakdown. |
|
|
34
|
+
| `source-video-metadata.json` | Trimmed `yt-dlp --write-info-json`. `availability: "unlisted"` drove a real publishing decision. |
|
|
35
|
+
|
|
36
|
+
## `render/` — the output side
|
|
37
|
+
|
|
38
|
+
| file | what it is |
|
|
39
|
+
|---|---|
|
|
40
|
+
| `rendered-page.html` | The deployed page, self-contained apart from `media/`. The transport/metronome, the per-card clip tabs, and the IntersectionObserver playback are all in here. |
|
|
41
|
+
| `og.html`, `icon.html` | Social image and favicon, as HTML screenshotted at DSF 2 by Playwright — so the type and gradient match the page exactly, for free. |
|
|
42
|
+
| `howto.html` | The annotated-card infographic. Generated: it takes a screenshot of a real card, reads the real bounding boxes of the annotated elements out of the DOM, and lays the connectors onto those measured anchors. |
|
|
43
|
+
| `annotated-card.jpg` | The rendered result of `howto.html`. |
|
|
44
|
+
|
|
45
|
+
The media itself (mp4/gif/jpg, ~17 MB) is not copied here; it lives in
|
|
46
|
+
`~/Dropbox/py/proj/tt/tw_platform/apps/que_calor_dance/frontend/media/`.
|