paces 0.0.1__tar.gz → 0.0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
paces-0.0.2/.gitignore ADDED
@@ -0,0 +1,26 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.egg-info/
5
+ dist/
6
+ build/
7
+ .venv/
8
+ .pytest_cache/
9
+ .ruff_cache/
10
+ .coverage
11
+ coverage.xml
12
+
13
+ # Editors / OS
14
+ .DS_Store
15
+ .idea/
16
+ .vscode/
17
+
18
+ # Claude Code session droppings (dev skills et al. stay tracked)
19
+ .claude/.system/
20
+ .claude/handoffs/
21
+
22
+ # Media and evidence stores never live in the repo
23
+ media/
24
+ *.annot
25
+ *.annot.sqlite
26
+ scrap/
paces-0.0.2/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Thor Whalen
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
paces-0.0.2/PKG-INFO ADDED
@@ -0,0 +1,118 @@
1
+ Metadata-Version: 2.5
2
+ Name: paces
3
+ Version: 0.0.2
4
+ Summary: Turn instructional media into structured, interactive learning material
5
+ Project-URL: Homepage, https://github.com/thorwhalen/paces
6
+ Project-URL: Repository, https://github.com/thorwhalen/paces
7
+ Project-URL: Documentation, https://thorwhalen.github.io/paces
8
+ Author: Thor Whalen
9
+ License-Expression: MIT
10
+ License-File: LICENSE
11
+ Keywords: dance,instructional-video,learning-material,practice,segmentation,steps,tutorial
12
+ Requires-Python: >=3.10
13
+ Requires-Dist: pydantic>=2.6
14
+ Provides-Extra: cli
15
+ Requires-Dist: argcomplete>=3; extra == 'cli'
16
+ Requires-Dist: argh>=0.30; extra == 'cli'
17
+ Provides-Extra: dev
18
+ Requires-Dist: argh>=0.30; extra == 'dev'
19
+ Requires-Dist: pytest-cov>=4.0; extra == 'dev'
20
+ Requires-Dist: pytest>=7.0; extra == 'dev'
21
+ Requires-Dist: ruff>=0.1.0; extra == 'dev'
22
+ Provides-Extra: docs
23
+ Requires-Dist: sphinx-rtd-theme>=1.0; extra == 'docs'
24
+ Requires-Dist: sphinx>=6.0; extra == 'docs'
25
+ Description-Content-Type: text/markdown
26
+
27
+ # paces
28
+
29
+ Turn instructional media into structured, interactive learning material.
30
+ *Put it through its paces.*
31
+
32
+ Take a video of someone teaching something — a dance routine, a kata, a
33
+ recipe — plus, optionally, notes and a steering prompt. `paces` segments it
34
+ into named steps, builds a structured **step document** (an AST for
35
+ step-by-step instruction), and renders that into learning material: a
36
+ practice page with counts and deep links today, other guides later.
37
+
38
+ ```bash
39
+ pip install paces
40
+ ```
41
+
42
+ ## Quick example
43
+
44
+ ```python
45
+ from paces import segment, to_document, render_html
46
+
47
+ seg = segment(
48
+ "https://youtu.be/q_TUyxUhoEw",
49
+ steps=[
50
+ ("Mise en place", 2),
51
+ ("Pas pieds pointe et ronde", 6),
52
+ ("Soleil avec les bras", 4),
53
+ ("Déhanchés", 8),
54
+ ],
55
+ grid={"unit": "eight", "subdivisions": 8, "tempoBpm": "129.2", "origin": "51.2"},
56
+ )
57
+ doc = to_document(
58
+ seg,
59
+ doc_id="que-calor",
60
+ title="Chorégraphie Que Calor",
61
+ source="https://youtu.be/q_TUyxUhoEw",
62
+ )
63
+ open("page.html", "w").write(render_html(doc))
64
+ ```
65
+
66
+ The page lists every step with its counts, links each one back into the video
67
+ (both the at-tempo run-through and the slow breakdown, when both are known),
68
+ and — because the document carries a metric grid — includes a count-along
69
+ transport that paces you through the routine at the measured tempo.
70
+
71
+ Same thing from the shell:
72
+
73
+ ```bash
74
+ paces segment VIDEO_URL --steps steps.json --grid grid.json --output seg.json
75
+ paces to-document seg.json --source VIDEO_URL --title "My routine" --output document.json
76
+ paces render document.json --output page.html
77
+ ```
78
+
79
+ ## How it thinks
80
+
81
+ **Analysis and rendering are separate phases** with a serialisable document
82
+ between them — like a parser emitting an AST and a backend interpreting it.
83
+ Renderers depend on the document, never on the analyser.
84
+
85
+ **Segmentation is a seam, not a stage.** `segment(media, segmenter=...)` —
86
+ segmenters are registered capabilities, the default follows from what is
87
+ present, and "the user typed the boundaries" is a first-class segmenter, not
88
+ a fallback. A segmenter that cannot *name* steps returns honest unnamed
89
+ boundaries (`flags: ['naming-abstained']`) rather than inventing names.
90
+
91
+ **The document keeps what the learner actually counts.** A dance step lasts
92
+ "4 eights", not "14.86 seconds" — seconds are derived from the metric grid
93
+ (tempo + origin), never stored. A step can have *several* source spans (the
94
+ run-through and the breakdown are the same step seen twice). Uncertainty is
95
+ content (`OpenQuestion`), and human edits are protected from regeneration
96
+ (`Lock`).
97
+
98
+ ## The pieces
99
+
100
+ | you want | reach for |
101
+ |---|---|
102
+ | cut media into steps | `segment(media, steps=..., grid=...)` → `Segmentation` |
103
+ | explicit/human boundaries | `segment(media, boundaries=[...], steps=[names])` |
104
+ | the committed artifact | `to_document(seg, ...)` → `StepDocument` |
105
+ | a practice page | `render_html(doc)` |
106
+ | wall-clock times from counts | `resolve(doc)` |
107
+ | sanity checks | `validate_document(doc)` |
108
+ | what segmenters exist | `capabilities()` / `paces list-segmenters` |
109
+ | add a segmenter | `register(Capability(name=..., gives="segmentation", target="mymod:fn", needs={...}))` — a new file, nothing edited |
110
+
111
+ ## Status
112
+
113
+ Young and moving. The document schema is validated by round-tripping a real
114
+ proof of concept ([an interactive dance-practice
115
+ page](https://thorwhalen.com/que_calor_dance/)) through it — see
116
+ `tests/test_roundtrip_poc.py`. Media derivation (auto-cropped looping clips),
117
+ intrinsic segmenters (scene/beat/speech detection), and the evidence layer
118
+ are designed (see `docs/`) and arrive next.
paces-0.0.2/README.md ADDED
@@ -0,0 +1,92 @@
1
+ # paces
2
+
3
+ Turn instructional media into structured, interactive learning material.
4
+ *Put it through its paces.*
5
+
6
+ Take a video of someone teaching something — a dance routine, a kata, a
7
+ recipe — plus, optionally, notes and a steering prompt. `paces` segments it
8
+ into named steps, builds a structured **step document** (an AST for
9
+ step-by-step instruction), and renders that into learning material: a
10
+ practice page with counts and deep links today, other guides later.
11
+
12
+ ```bash
13
+ pip install paces
14
+ ```
15
+
16
+ ## Quick example
17
+
18
+ ```python
19
+ from paces import segment, to_document, render_html
20
+
21
+ seg = segment(
22
+ "https://youtu.be/q_TUyxUhoEw",
23
+ steps=[
24
+ ("Mise en place", 2),
25
+ ("Pas pieds pointe et ronde", 6),
26
+ ("Soleil avec les bras", 4),
27
+ ("Déhanchés", 8),
28
+ ],
29
+ grid={"unit": "eight", "subdivisions": 8, "tempoBpm": "129.2", "origin": "51.2"},
30
+ )
31
+ doc = to_document(
32
+ seg,
33
+ doc_id="que-calor",
34
+ title="Chorégraphie Que Calor",
35
+ source="https://youtu.be/q_TUyxUhoEw",
36
+ )
37
+ open("page.html", "w").write(render_html(doc))
38
+ ```
39
+
40
+ The page lists every step with its counts, links each one back into the video
41
+ (both the at-tempo run-through and the slow breakdown, when both are known),
42
+ and — because the document carries a metric grid — includes a count-along
43
+ transport that paces you through the routine at the measured tempo.
44
+
45
+ Same thing from the shell:
46
+
47
+ ```bash
48
+ paces segment VIDEO_URL --steps steps.json --grid grid.json --output seg.json
49
+ paces to-document seg.json --source VIDEO_URL --title "My routine" --output document.json
50
+ paces render document.json --output page.html
51
+ ```
52
+
53
+ ## How it thinks
54
+
55
+ **Analysis and rendering are separate phases** with a serialisable document
56
+ between them — like a parser emitting an AST and a backend interpreting it.
57
+ Renderers depend on the document, never on the analyser.
58
+
59
+ **Segmentation is a seam, not a stage.** `segment(media, segmenter=...)` —
60
+ segmenters are registered capabilities, the default follows from what is
61
+ present, and "the user typed the boundaries" is a first-class segmenter, not
62
+ a fallback. A segmenter that cannot *name* steps returns honest unnamed
63
+ boundaries (`flags: ['naming-abstained']`) rather than inventing names.
64
+
65
+ **The document keeps what the learner actually counts.** A dance step lasts
66
+ "4 eights", not "14.86 seconds" — seconds are derived from the metric grid
67
+ (tempo + origin), never stored. A step can have *several* source spans (the
68
+ run-through and the breakdown are the same step seen twice). Uncertainty is
69
+ content (`OpenQuestion`), and human edits are protected from regeneration
70
+ (`Lock`).
71
+
72
+ ## The pieces
73
+
74
+ | you want | reach for |
75
+ |---|---|
76
+ | cut media into steps | `segment(media, steps=..., grid=...)` → `Segmentation` |
77
+ | explicit/human boundaries | `segment(media, boundaries=[...], steps=[names])` |
78
+ | the committed artifact | `to_document(seg, ...)` → `StepDocument` |
79
+ | a practice page | `render_html(doc)` |
80
+ | wall-clock times from counts | `resolve(doc)` |
81
+ | sanity checks | `validate_document(doc)` |
82
+ | what segmenters exist | `capabilities()` / `paces list-segmenters` |
83
+ | add a segmenter | `register(Capability(name=..., gives="segmentation", target="mymod:fn", needs={...}))` — a new file, nothing edited |
84
+
85
+ ## Status
86
+
87
+ Young and moving. The document schema is validated by round-tripping a real
88
+ proof of concept ([an interactive dance-practice
89
+ page](https://thorwhalen.com/que_calor_dance/)) through it — see
90
+ `tests/test_roundtrip_poc.py`. Media derivation (auto-cropped looping clips),
91
+ intrinsic segmenters (scene/beat/speech detection), and the evidence layer
92
+ are designed (see `docs/`) and arrive next.
@@ -0,0 +1,143 @@
1
+ # `paces` — a library for turning instructional video into learning material
2
+
3
+ *Put it through its paces.* The name is settled (`adr/0002`); so is the core abstraction
4
+ (`adr/0003`). These docs were written while the package was still called `stepped` — where the
5
+ prose says "stepped", read `paces`.
6
+
7
+ ---
8
+
9
+ ## What this is
10
+
11
+ A working proof-of-concept exists. In one session, a YouTube video of a choreographer teaching
12
+ a dance, plus a hand-written HTML aide-mémoire, plus a paragraph of steering prompt, became a
13
+ deployed interactive practice page: <https://thorwhalen.com/que_calor_dance/>.
14
+
15
+ The user now wants that generalised into a library, integrated with the **reelee** /
16
+ `video_gen` fleet. Your job is to research, design and build it. **This folder exists so you
17
+ do not start from scratch.** It records what was built, the parameters that were expensive to
18
+ find, the failures and what each one teaches, the user's own framing of the generalisation,
19
+ and an inventory of what already exists in the fleet.
20
+
21
+ Nothing here is a design you must follow. It is evidence and framing. Argue with it.
22
+
23
+ ## Read in this order
24
+
25
+ | | file | why |
26
+ |---|---|---|
27
+ | 1 | **`01-what-was-built.md`** | The POC, factually. Includes the incident list — every failure is a requirement in disguise. |
28
+ | 2 | **`03-design-brief.md`** | The user's own framing: the parse→AST→render metaphor they explicitly asked to have recorded, the three generalisation axes, and the constraints the POC discovered. |
29
+ | 3 | **`04-reelee-core.md`** | What reelee is, what a reelee "genre" is, and what to reuse. **This determines the package boundary**, so read it before deciding anything structural. |
30
+ | 4 | **`07-annotation-model.md`** | The proposed shape of the AST — the contract between analysis and rendering. |
31
+ | 5 | **`05-fleet-inventory.md`** | What already exists across `video_gen`, and honestly which parts are stubs. |
32
+ | 6 | **`02-technical-recipes.md`** | Every technique with working parameters. Reference, not narrative — come back to it when implementing. |
33
+ | 7 | **`06-surfaces-and-conventions.md`** | House style: architecture-first seams, qh, py2mcp, storage, frontend, deploy. |
34
+ | 8 | **`08-naming-candidates.md`** | PyPI-verified name options, and a better word for the "subject" axis. |
35
+ | 9 | **`09-subgenre-candidates.md`** | What to build after dance, and what each choice would force the core to get right. |
36
+ | 10 | **`10-session-archaeology.md`** | The 27 MB session transcript, and how to query it when these docs fall short. |
37
+ | — | **`adr/`** | The decisions. **`0003` `video + segmenter`** — read it before designing the analysis phase. `0002` the name. `0001` the alignment engine (intent) — note that much of what you'd otherwise build already exists in `muvid`, `mixing` and `kodokan`. |
38
+ | — | **`alignment/`** | The research behind that ADR: one file per method family, prepared so you don't start from a literature search. |
39
+ | — | **`KICKOFF.md`** | A paste-ready prompt to start a fresh session on this. |
40
+ | — | **`REGISTRATION.md`** | **One command still pending** to register `paces` in the `video_gen` group, and why `priv pkg add-package` is deliberately deferred. |
41
+ | — | **`poc-reference/`** | The actual scripts and data. Read `poc-reference/README.md` first — several of those files are recorded dead ends. |
42
+
43
+ ## The one-paragraph version
44
+
45
+ Take instructional media (a video of someone teaching something) plus optional notes and a
46
+ steering prompt. **Analyse**: extract signals (audio structure, transcript, beat grid, subject
47
+ tracking), segment into named steps, align against the notes, and emit a structured step
48
+ document — an *AST*. **Render**: consume that AST to produce learning material — an
49
+ interactive practice page today, other guides later. The AST is the contract; the analyser and
50
+ the renderers depend on it and not on each other. A shared core handles step-by-step
51
+ instructional content in general (reelee's word: **genre**); *subgenres* — dance, kata,
52
+ recipe, repair — specialise the segmentation signals, the duration unit, and the rendering.
53
+
54
+ ## Four findings from the research that change the starting position
55
+
56
+ These came out of `04`–`07` and are worth knowing before you read anything else, because each
57
+ one removes work you might otherwise plan for.
58
+
59
+ 1. **"Genre" is an `nw` concept, not a reelee one.** `nw` (`$PP/t/nw`) owns `Project`,
60
+ `Transform`, `Genre`, freshness and jobs, with a real registry
61
+ (`register_genre`, `register_genre_resolver`, `register_genre_initializer`,
62
+ `register_genre_project_factory`). If you want a "step-by-step" genre with a "dance"
63
+ subgenre, that machinery already exists and reelee is a *consumer* of it. `04-reelee-core.md`.
64
+ 2. **reelee is deliberately small.** Its own `__init__` says the substance lives in the focused
65
+ packages below it — `lacing`, `falaw`, `nw`, `artful` — and reelee's surface is
66
+ orchestration. That strongly suggests this library is **another focused package below
67
+ reelee**, not a fork of it and not a plugin inside it. Confirm with the user, but start from
68
+ that hypothesis.
69
+ 3. **`lacing` is already the annotation substrate**, and reelee already ships
70
+ regenerate-without-losing-human-edits machinery on top of it. That is constraint §5 of
71
+ `03-design-brief.md` — the one that looked hardest — already solved. `07-annotation-model.md`
72
+ recommends owning only a small *document* type and delegating everything below it.
73
+ 4. **The step structure was never in `clips.json`.** It was in the page's own
74
+ `const ROUTINE = [...]`; `clips.json` is the *span* table. Reading the POC as a two-layer
75
+ model (steps ↕ spans) rather than one flat list is the single most useful reframe in these
76
+ docs.
77
+ 5. **Most of the analysis phase already exists in the fleet, unwired.** `muvid/align.py` is an
78
+ aligner registry with dispatch and a `lacing` writeback; `muvid.footage.select_score` is a
79
+ constrained sequence solver; `mixing.audio` has beat grids, speech/music segmentation and
80
+ cross-correlation offset alignment; `kodokan` has a complete, tested, Apple-Silicon-native
81
+ **pose front-end that has never been pointed at an alignment problem**. `adr/0001` and
82
+ `alignment/00-existing-in-fleet.md` have the map. Do not write a second one of any of these.
83
+
84
+ ## What the POC actually proves
85
+
86
+ - Separating analysis from rendering is real, not architectural decoration: the media was
87
+ re-rendered three times and the page a dozen times from unchanged manifests.
88
+ - A step legitimately has **several source spans** (at-tempo run-through *and* slow
89
+ explanation). Six of nine blocks shipped both. A one-span model is wrong on day one.
90
+ - Cheap signals go a long way: a sub-bass energy ratio separates speech from music in five
91
+ lines and did more work than anything else in the session.
92
+ - An LLM looking at timestamped contact sheets makes genuinely good editorial calls about
93
+ which few seconds show a move — and is slow and expensive doing it.
94
+
95
+ ## What the POC does *not* prove
96
+
97
+ - **Nothing here segmented a video cold.** The nine steps came from the input document; the
98
+ analysis confirmed, corrected and timed them. Cold segmentation is a different, harder
99
+ problem and the POC has no evidence about it.
100
+ - Everything assumed **one subject**. Two people breaks the crop envelope, the face-slot logic
101
+ and the privacy mask simultaneously.
102
+ - The renderer was a patched copy of the input document. There is no template contract yet.
103
+
104
+ ## Suggested first moves
105
+
106
+ 1. Read `04-reelee-core.md`, then settle the **integration shape** with the user. The research
107
+ points at "a focused package below reelee, registering an `nw` genre" — test that
108
+ hypothesis rather than assuming it. Boundaries are the expensive thing to move later.
109
+ 2. ~~Settle the name~~ — **done, `paces`** (`adr/0002`). Still open: the better word for the
110
+ "subject" axis. Leading candidates in `08-naming-candidates.md §6`: `discipline`, `craft`,
111
+ `grammar`, `idiom`. **Do not run another availability sweep** — 684 names are already
112
+ checked, and that file is now also the name pool for sub-packages this work spawns.
113
+ 3. ~~The v1 scope fork~~ — **dissolved by `adr/0003`.** Segmentation is a seam, not a stage:
114
+ `video + segmenter=`, kept open-closed by a strategy pattern. A segmenter's information can
115
+ be **intrinsic** (features of the media), **external-explicit** (coordinates handed over),
116
+ **external-derived** (computed from the surrounding annotations), or **mixed** — and mixed is
117
+ the normal case. Many *intrinsic* segmenters share one shape — featurize → reduce → threshold
118
+ → regularize — which the package should ship as composable stages so common cases are
119
+ assembled rather than authored. "Ask the user" is first-class. Read that ADR, then
120
+ `alignment/07-segmenter-strategies.md` **with its correction preamble**, then decide which
121
+ two segmenters v1 ships.
122
+ 4. Write the AST schema and validate it by **re-expressing the POC's `clips.json` in it**
123
+ (`poc-reference/artifacts/clips.json`). If the dance case does not round-trip, the schema is
124
+ wrong. That is a cheap, real test available on day one.
125
+ 5. Only then pick seams and build (`03-design-brief.md §6`, `06-surfaces-and-conventions.md`).
126
+
127
+ ## Open questions for the user
128
+
129
+ Collected from across these docs; raise them early rather than guessing.
130
+
131
+ 1. ~~Notes required in v1, or cold segmentation?~~ Answered by `adr/0003`.
132
+ 2. Integration shape with reelee? (decides the package boundary — the live one)
133
+ 3. Is the renderer part of this library or a sibling package?
134
+ 4. How much of the analysis may be LLM-driven — what is the acceptable cost per project?
135
+ 5. What is the editing story? Human edits must survive re-analysis.
136
+ 6. ~~The name~~ (`paces`, `adr/0002`) — but still the word for the "subject/genre" axis.
137
+
138
+ ## Provenance
139
+
140
+ Built from session `0f75703c-6761-4aa0-b796-aafe02c94155` (2026-08-27/28). The full transcript
141
+ is on disk and queryable — `10-session-archaeology.md`. Docs `01`, `02`, `03`, `10` and
142
+ `poc-reference/README.md` were written from direct session context; `04`–`09` were researched
143
+ by subagents and each carries its own "verified vs inferred" notes.
@@ -0,0 +1,47 @@
1
+ # Alignment research
2
+
3
+ *What this folder is for: preparation for a possible dedicated **alignment tool** in the
4
+ `video_gen` / reelee fleet — the thing that answers "given some artifacts and some media,
5
+ which span of the media does each artifact correspond to?". The decision to build it is
6
+ recorded as an intent in `../adr/0001-alignment-engine-as-a-fleet-package.md`; **read that
7
+ first**, then come here for the method-by-method detail.*
8
+
9
+ The user's framing of the problem and of the top surface:
10
+
11
+ > *"This is a very common task in our reelee work: being able to match and align things. More
12
+ > specifically … being able to align annotations/artifacts to segments of audio and/or video."*
13
+ >
14
+ > *"I'm imagining that the highest surface would be an agent that can study the context, what
15
+ > is available, and have a list of possible methods/algorithms/tools it could use. It could use
16
+ > beats. It could use the transcribed words (and transcribe if they're not available). It could
17
+ > use some knowledge of the order of the artifacts to be matched. It could use gestures."*
18
+
19
+ ## The files
20
+
21
+ | file | what it covers |
22
+ |---|---|
23
+ | **`00-existing-in-fleet.md`** | **Start here.** What the fleet already has. Verdict: the capability doesn't exist as a capability, but nearly every part does — `muvid/align.py` is the v0 registry, `muvid.footage.select_score` is the order-prior solver, `mixing.audio` is the signal layer, `kodokan` is a complete but dormant pose front-end, `lacing` is the settled output substrate. Also argues *separate package vs a module in `lacing`* (§4.3) and concludes separate. |
24
+ | `01-text-to-audio-alignment.md` | ASR with word timings, true forced alignment (CTC, MFA, aeneas), **fuzzy paraphrase→transcript matching** (the common reelee case, and the hard one), VAD, diarization. |
25
+ | `02-music-rhythm-and-structure.md` | Beats, downbeats, tempo and its octave errors, **phase/offset** (tempo gives spacing, not where bar 1 starts), music structure segmentation, music-vs-speech discrimination, audio-to-audio alignment and fingerprinting. |
26
+ | `03-visual-signals.md` | Scene cuts, motion energy and optical flow, **pose estimation and pose→segmentation**, repetition/periodicity detection, gesture and action recognition, hand/object interaction, OCR. This is the file that closes the "we never used gestures" gap. |
27
+ | `04-semantic-and-llm-matching.md` | CLIP/SigLIP frame–text scoring, CLAP for audio, video-text retrieval, the **LLM-over-timestamped-contact-sheets** pattern written up as a reusable technique, transcript structure extraction, and confidence calibration. |
28
+ | `05-sequence-alignment-algorithms.md` | The algorithm layer under everything else: DTW and subsequence DTW, gapped Needleman–Wunsch, CTC segmentation, Viterbi, **change-point detection with a known segment count**, grid fitting for `(offset, period)`, evidence fusion, and how to evaluate an aligner at all. |
29
+ | `06-the-planner-surface.md` | The top surface. Proposes collapsing all five sibling method Protocols into one `Capability(needs, gives, …)` record, a measured ~1.4 s/min context probe, and a deterministic ranked graph walk — with the LLM deliberately outside the control loop. |
30
+ | **`07-segmenter-strategies.md`** | **The `video + segmenter` seam** the user settled on. A catalogue of ~16 video-only segmentation families plus six richer input tiers (step list, structured document, steering prompt, chapters/subtitles/**re-watch heatmap**, ask-the-user, escalation), organised by *what input is present* because that is what picks the default. Then the `Segmenter` protocol — reusing `06`'s `Capability` with **two** new products and **seven** new facts and no new fields — the generated default-selection table, and the recommendation: three segmenters in v1, `novelty-k` as the video-only default, **and it deliberately does not name the steps**. |
31
+ | `04-evidence/` | Real experiment scripts the semantic agent wrote and ran (SigLIP SO400M, CLAP, VLM scoring, calibration, ASR cue extraction). Kept because the numbers in `04-…md` came from them. Scratch quality — read them as evidence, not as library code. |
32
+
33
+ ## How to use this
34
+
35
+ These files are a **menu with prices**, not a plan. Each method entry states what signal it
36
+ needs, what it produces, its cost per minute of media, its licence, and the failure mode that
37
+ will bite. The planner file is what turns the menu into a selection procedure.
38
+
39
+ Two things worth internalising before reading in depth, both from the POC that motivated all of
40
+ this (`../01-what-was-built.md`):
41
+
42
+ - **The cheapest signal did the most work.** A five-line sub-bass energy ratio separated
43
+ talking-head from music-with-dancing and made everything downstream tractable. Any design that
44
+ reaches for pose or a VLM first will be slower *and* worse.
45
+ - **No single method was sufficient.** The answer came from combining a cheap audio feature, a
46
+ beat model, ASR, a vision-language judgement, and the order prior. Fusion and confidence are
47
+ not optional extras — they're the point.
@@ -0,0 +1,46 @@
1
+ # POC reference code and artifacts
2
+
3
+ *What this is: the actual scripts and data from the session that produced
4
+ <https://thorwhalen.com/que_calor_dance/>. **This is not a library and must not be treated as
5
+ one** — it is a pile of session tooling, written to be thrown away, kept because the
6
+ parameters in it were expensive to find. Read it for the recipes, not the structure.
7
+ `../02-technical-recipes.md` explains every one of them.*
8
+
9
+ Everything ran under `~/.pyenv/versions/3.12.12/envs/p12/bin/python`, from a working directory
10
+ containing `source.mp4`.
11
+
12
+ ## `tools/` — the pipeline
13
+
14
+ | file | what it does | worth keeping? |
15
+ |---|---|---|
16
+ | `sheet.py` | Labelled contact sheet over a time window — `sheet.py START END STEP OUT.jpg [COLS]`. Every tile stamped with its absolute timestamp. | **Yes.** This is how an LLM looks at video. The timestamps are the whole point. |
17
+ | `zsheet.py`, `sheet_crop.py` | Same, cropped to the subject. Written by subagents mid-session to judge a wide shot. | Merge into one tool. |
18
+ | `mkclip.py` | Cut + auto-crop + encode one clip as gif/mp4/webp. Holds `crop_box()`: YOLO person boxes → robust percentile envelope → forced aspect → clamp. | **Yes** — `crop_box()` especially. |
19
+ | `track.py` | Whole-video person tracking at 5 fps into `boxes.npz`. | **No.** Abandoned: far too slow (>45 min on CPU). Kept as a record of the dead end. |
20
+ | `bg.py` | Empty-room plate as an 88th-percentile-per-pixel composite. | Only for *compositing*. Do not use it to *find* the subject — see `../02-technical-recipes.md §7`. |
21
+ | `stylize.py` | The anonymisation pipeline, adapted from kodokan and made streaming. Contains the narrowed head-band fix. | **Yes**, with the licensing caveat. |
22
+ | `build_media.py` / `build_media_styl.py` / `restyle_clips.py` | Batch drivers: manifest → mp4 + gif + poster for every clip. Three generations of the same script. | As a spec for what the render stage does. |
23
+ | `build_page.py` | Renderer. Splices a `MEDIA` map into the source document's own JS and patches its HTML by string surgery. | **Read it, then do the opposite.** It is the clearest possible argument for a real template contract. |
24
+ | `shot.py` | Playwright screenshot + console/HTTP error capture. | **Yes.** Verification, and image generation from the page's own CSS. |
25
+ | `ship.sh` | Rebuild page → refresh the deploy app dir. | Trivial, illustrative. |
26
+
27
+ ## `artifacts/` — the intermediate representation
28
+
29
+ | file | what it is |
30
+ |---|---|
31
+ | `clips.json` | **The POC's entire AST.** 15 entries. `src: "RT" \| "BD"` is run-through vs breakdown — the two-passes-over-the-same-material idea in its crudest form. Analysed in `../07-annotation-model.md`. |
32
+ | `crops.json` | `{clip_id: [x, y, w, h]}` — a derived cache, correctly kept out of the semantic model. |
33
+ | `transcript.json` | mlx-whisper output for the whole 651 s. Note how it degrades over the music section (hundreds of near-empty segments, invented numbers) and is excellent over the spoken breakdown. |
34
+ | `source-video-metadata.json` | Trimmed `yt-dlp --write-info-json`. `availability: "unlisted"` drove a real publishing decision. |
35
+
36
+ ## `render/` — the output side
37
+
38
+ | file | what it is |
39
+ |---|---|
40
+ | `rendered-page.html` | The deployed page, self-contained apart from `media/`. The transport/metronome, the per-card clip tabs, and the IntersectionObserver playback are all in here. |
41
+ | `og.html`, `icon.html` | Social image and favicon, as HTML screenshotted at DSF 2 by Playwright — so the type and gradient match the page exactly, for free. |
42
+ | `howto.html` | The annotated-card infographic. Generated: it takes a screenshot of a real card, reads the real bounding boxes of the annotated elements out of the DOM, and lays the connectors onto those measured anchors. |
43
+ | `annotated-card.jpg` | The rendered result of `howto.html`. |
44
+
45
+ The media itself (mp4/gif/jpg, ~17 MB) is not copied here; it lives in
46
+ `~/Dropbox/py/proj/tt/tw_platform/apps/que_calor_dance/frontend/media/`.