@allyhub/sdk 0.1.0-alpha.8 → 0.2.0-next.1

package/docs/docs.md

@@ -0,0 +1,148 @@
+# Docs
+
+A doc is prose a person reads and edits as a page and an agent writes as text: one file, `doc.mdx`. It is not a piece: it has no format, languages or combos. `allyhub check` checks it, `allyhub dev` shows it in its editor, `allyhub render` prints it.
+
+## A doc item
+
+A doc is a folder whose entry file is `doc.mdx`:
+
+```
+launch/plan/
+  doc.mdx          the doc
+  comments.json    optional: what was said about it
+  shot.png         anything else it shows, kept beside it
+```
+
+- The folder may be anywhere in the project but its root, a folder whose name starts with a dot, or `node_modules`. Its place is its path from the project (`launch/plan`), which commands take: `allyhub render launch/plan`.
+- Everything in its folder is the doc's own. A doc is never inside another item's folder (AK-M008), and no item is inside a doc's folder (AK-M003).
+
+## Title
+
+The title is the first-level heading the file opens with, when nothing folds under it:
+
+```mdx
+# Launch plan
+
+What we ship, and when.
+```
+
+It is the doc's name in the project's lists and the PDF's title. A file that opens with anything else has no title, and its PDF is titled by its place.
+
+## What a doc reads
+
+The file is MDX: Markdown with GFM tables, strike-through and task lists, plus tags and `import` lines. Bare addresses are not links (no autolinks): write `[text](https://…)`.
+
+| Written | Block |
+| --- | --- |
+| A paragraph | paragraph |
+| `#` … `######` | heading, levels 1 to 6 |
+| `- item` | bullet item |
+| `1. item` | numbered item; a list that does not start at 1 keeps its start, and a numbered list right after another is a list of its own |
+| `- [ ] item`, `- [x] item` | task, unchecked or checked |
+| `> line` | quote |
+| A fenced code block | code, with its language |
+| A GFM table | table, its first row the head |
+| `![alt](path "caption")` alone on a line | picture |
+| `---` | divider |
+
+Items and quotes hold blocks under them by indentation. Inside a line: `**bold**`, `*italic*`, `~~struck~~`, `` `code` ``, `[link](href)`, and a hard line break.
+
+`{` and `<` are MDX syntax: write `\{` and `\<` for the characters.
+
+### What Markdown cannot say
+
+Tags with plain string attributes (`name="value"`, or the bare name for `toggle`). An attribute that is not listed, or a value in `{…}`, is a problem (AK-M008).
+
+| Written | Means |
+| --- | --- |
+| `<u>…</u>` | Underlined |
+| `<span color="red" bg="yellow">…</span>` | Text in a colour, on a colour |
+| `<strong>` `<b>`, `<em>` `<i>`, `<del>` `<s>`, `<code>` | The marks `**`, `*`, `~~` and a backtick make, where Markdown's own would not read back (`**“引号”**` between Chinese characters is not bold; `<strong>“引号”</strong>` is) |
+| `<br />` | Inside a line, a line break; on a line of its own, an empty paragraph |
+| `<Block align color bg toggle>` | One block's look. Inside it: the block, then the blocks under it. `align` is `center`, `right` or `justify`; `toggle` makes a heading or a line fold the blocks after it |
+| `<Callout icon color bg>` | A line set apart in a box with a sign before it (💡 when `icon` is absent). Inside it: the line, then the blocks under it |
+| `<img src alt title width align />` | A picture with a width in px or a place (`align`: `center` or `right`); `title` is its caption. Without those, write `![alt](src)` |
+| `<Table widths head>` around one GFM table | Column widths in px, comma-separated, an empty entry for a column that keeps its own (`widths="120,,200"`); `head` is `row` (the default), `column`, `both` or `none` |
+
+A colour is one of `gray`, `brown`, `red`, `orange`, `yellow`, `green`, `blue`, `purple`, `pink`, or a CSS colour (`#0b6e99`, `rgb(…)`, `hsl(…)`).
+
+`<Block>`, `<Callout>` and `<Table>` go on lines of their own, with what they hold between them:
+
+```mdx
+<Block align="center" color="blue">
+
+A centred line, in blue.
+
+</Block>
+
+<Block toggle>
+
+## Details
+
+Folded under the heading.
+
+</Block>
+
+<Callout icon="⚠️" bg="yellow">
+
+Check the dates.
+
+</Callout>
+
+<Table widths="120,,200" head="none">
+
+| a   | b   | c   |
+| --- | --- | --- |
+| 1   | 2   | 3   |
+
+</Table>
+```
+
+### Kept as written
+
+- **Embeds**: any other tag on lines of its own (`<Chart kind="bar" />`) is kept as written and not drawn: on paper it is a box with the tag's name.
+- **Imports, exports and link definitions** are kept as written and not shown.
+
+An edit to some blocks in the editor rewrites only those blocks' lines: the rest of the file keeps its bytes (how a paragraph was wrapped, which marker a list uses).
+
+## Pictures and links
+
+- A picture's path is relative to the doc's folder (`./shot.png`, `../../assets/media/logo.svg`) and names a file of the project; one that is not there is AK-M008. An address (`https://…`) is not checked.
+- A link to another item is a relative path to its entry file or its folder: `[the film](../film/video.ts)`, `[the notes](../notes/doc.mdx)`. Moving an item in `allyhub dev` or Studio writes the relative links and pictures that point into it, or out of it, again.
+
+## comments.json
+
+Threads about a doc are kept beside it in `comments.json` (JSON Schema: `https://allyhub.com/schemas/comments.schema.json`). An agent answers a thread by writing its reply into the file.
+
+```json
+{
+  "threads": [
+    {
+      "id": "c1",
+      "quote": "Forms that fill themselves in",
+      "text": "Shorter?",
+      "at": "2026-10-14T09:30:00Z",
+      "replies": [
+        { "by": "agent", "text": "Cut to five words.", "at": "2026-10-14T09:41:00Z" }
+      ],
+      "resolved": true
+    }
+  ]
+}
+```
+
+| Field | Type | |
+| --- | --- | --- |
+| `threads[].id` | string, required | `c1`, `c2`…: how a reply names its thread |
+| `threads[].text` | string, required | What was said |
+| `threads[].quote` | string | The doc's words it is about; empty for a place or the whole doc |
+| `threads[].at` | string | When, ISO 8601 |
+| `threads[].replies[]` | `{ by, text, at }` | `by` is `person` or `agent` (anything but `person` reads as `agent`); `text` is required |
+| `threads[].resolved` | boolean | Done; absent is false |
+
+A thread without an `id` or a `text` is left out when the file is read.
+
+## Check and render
+
+- `allyhub check` reports AK-M008 for a doc whose file is not MDX, that holds what a doc does not draw (`{…}`, a tag inside a line other than the marks above, a wrong attribute or value), that shows a picture which is not in the project, or that is inside another item's folder; each problem has its file and line.
+- `allyhub render` prints each doc to `.allyhub/out/<place>/doc.pdf`; `--type png` writes a picture of the whole page (`doc.png`), `--type md` plain Markdown (`doc.md`: what the doc says without what only a doc can say), and `--type pdf,png,md` all three. A doc with a problem is not rendered: the command fails and says why. `allyhub help render` has every option.

package/docs/footage.md

@@ -0,0 +1,249 @@
+# Footage
+
+`footage()` is a layer that shows a video in a part: `footage()` alone is the part's own take in sync with its speech, `footage(file, opts)` another file, `footage(videoItem, opts)` another video item of the project. It stacks with the part's other layers in the order they are listed ([video.md](video.md)). A take, its cut list and its captions are in [speech.md](speech.md).
+
+```ts file=talk/short/video.ts
+import { footage, part, take, video } from "@allyhub/sdk";
+
+import broll from "./broll/inbox.mp4";
+
+export default video({
+  format: "9x16",
+  parts: () => [
+    part({
+      key: "intro",
+      speech: take("./host.mp4").range("So here is", "every time"),
+      layers: [
+        footage({
+          reframe: "face",
+          punchIn: ["free", "insane@1.15"],
+          pullback: true,
+        }),
+        footage(broll, {
+          at: "inbox",
+          dur: 1.5,
+          box: [0.55, 0.1, 0.4, 0.3],
+          radius: 24,
+        }),
+        footage("./screen.mp4", {
+          from: 3,
+          crop: [0, 0, 0.5, 1],
+          speed: 2,
+          fit: "contain",
+          audio: true,
+          gainDb: -6,
+        }),
+      ],
+    }),
+    part({
+      key: "inset",
+      speech: take("./host.mp4").range("Now look"),
+      layers: [
+        footage({
+          box: {
+            portrait: [0.1, 0.6, 0.8, 0.3],
+            default: [
+              { t: 0, box: "full" },
+              { t: "look", box: [0.6, 0.6, 0.35, 0.35], ease: "in-out" },
+            ],
+          },
+        }),
+      ],
+    }),
+  ],
+});
+```
+
+## The file
+
+- **A file**: an import (`import clip from "./clip.mp4"`) or a path. A path is relative to the folder of the module that wrote the `footage()` call, also once its part is spread into another piece ([project.md](project.md)).
+- **No file**: the part's take, in sync with its speech, following its cut list. A part with no take is AK-A001; a take that is not a video is AK-A003. A take played at another speed (`take(…).speed(1.2)`) cannot show its own picture: `part()` throws.
+- **A video item**: an imported `video()` of the project ([below](#a-video-item-as-footage)).
+- A file that is not a video (mp4, webm or mov) is AK-A003; a file that is not there AK-A001.
+- When the layer lasts longer than the file, its last frame holds; before a take's file starts (speech in the lead), its first frame holds. A video with alpha (ProRes 4444, VP9 with alpha) shows what is under its transparent pixels.
+
+## Options
+
+Besides `at`, `until` and `dur` (word references or seconds; default the whole part):
+
+| Option | Type | Default | What it is |
+| --- | --- | --- | --- |
+| `from` | seconds | 0 | Seconds into the file at the layer's start. A take's own footage follows its speech instead |
+| `speed` | over 0, up to 16 | 1 | Playback speed; its sound follows, pitch too |
+| `audio` | boolean | false | Mix the file's own sound in. A take's own footage sounds as the part's speech |
+| `gainDb` | dB | 0 | Its sound's gain |
+| `box` | `Rect`, `"full"`, `BoxKey[]`, or per frame shape | `"full"` | Where it shows, `[x, y, w, h]` as fractions of the frame, inside it |
+| `fit` | `"cover"`, `"contain"` | `"cover"` | How the picture fills its box |
+| `crop` | `Rect`, or per frame shape | none | A part of the source (fractions of it), then fitted to the box |
+| `shape` | `"rect"`, `"circle"` | `"rect"` | `"circle"`: the ellipse inscribed in the box |
+| `radius` | px at 1080p | 0 | Rounds the box's corners, at most half its shorter side; what is outside the corners shows what is under the layer. Not with `"circle"` |
+| `reframe` | `"face"`, `{ face: true, faceY? }`, `"center"`, `{ x, y }` | none | Crops the source to the box's shape round the speaker's face (a smoothed path; the face's centre at `faceY` of the crop's height, default 0.4), its centre, or a point of the source |
+| `speaker` | a face id (whole number from 0) | the track's default speaker | The face the layer follows: for a face reframe, punch-ins, the pullback and `ctx.face` |
+| `punchIn` | `(WordRef \| PunchIn)[]` | none | Zooms in on words ([below](#punch-ins-and-the-pullback)) |
+| `pullback` | `true` or `Pullback` | none | Opens zoomed in and eases back to the whole picture |
+| `sharpen` | 0 to 1 | none | Sharpens after scaling, for footage drawn larger than its pixels (the `footage-scale` fact). Renders and stills only |
+| `background` | `FootageBackground` | none | The picture behind the person treated apart from them ([below](#the-background-behind-the-person)) |
+
+- **Lengths** in px at 1080p (`radius`, a background's `blur`, `feather`, `shrink`) scale with the format's shorter side, and `radius` with a keyframed box.
+- **A reframe** with no face (no face track yet, or none seen then) is centred on `{ x: 0.5, y: 0.4 }` of the source.
+- **Per frame shape**: `box` and `crop` take a `PerOrient`, `{ landscape?, portrait?, square?, default? }`. Each piece takes the entry for its `ctx.dims.orient`, else `default`, else none (the full frame, no crop). So pieces of one film in two formats can share their parts.
+
+### Box keyframes
+
+`box` as `BoxKey[]`: `[{ t, box, ease? }]`, `t` a word reference or seconds, `box` a rectangle or `"full"`, `ease` the move into that key (`"linear"`, `"in"`, `"out"`, `"in-out"`; default `"linear"`).
+
+- The picture keeps its shape in every box, fitted as `fit` says: a box that changes shape shows more or less of it, never a stretched one.
+- Every key lies inside the frame; `footage()` throws for one that reaches past it. A push-in larger than the frame is a punch-in.
+- A keyframed box takes no `reframe`, `punchIn` or `pullback` (`footage()` throws).
+
+### Punch-ins and the pullback
+
+- **A punch-in** is a word reference, `"free"` (scale 1.12), `"free@1.15"` (a decimal after `@` is the scale; `"free@2"` is still the second "free"), or `{ at, until?, scale?, in?, out?, ease?, center? }`. It eases in over `in` seconds (default 0.2) centred on `at`, holds for the word (or until `until`), and eases out over `out` (default 0.25) with `ease` (default `"in-out"`). `scale` is over 1, up to 3; `in` and `out` are over 0, up to 30.
+- **A pullback** starts at `scale` (default 1.12, up to 3) and eases (`ease`, default `"out"`) back to the whole picture over `dur` seconds (default 0.6, up to 10).
+- Both zoom about `center`, a point `{ x, y }` as fractions of the box; default the speaker's face, else `{ x: 0.5, y: 0.42 }`.
+
+## Faces and mattes
+
+A face track and a person matte are what AllyHub measures once from a video and keeps in the project's media index (`.allyhub/cache/index/<sha256 of the file>/`).
+
+- The **face track** is read by `reframe: "face"`, the punch-in and pullback centres, `speaker`, and in a layer by `ctx.face(t, id?)` (one face's box `[x, y, w, h]` in the frame's pixels at part time `t`, by default the layer's speaker; `undefined` when none) and `ctx.faces(t)` (`[{ id, box }]`), from the topmost footage showing at `t` that has one. Every face has a stable id; `allyhub frames --overlay` draws them with their ids.
+- The **person matte** is read by graphic layers drawn `z: "behind-person"` and by a footage layer's `background`.
+- Every command that draws a piece (`check`, `render`, `dev`, `frames`, …) makes what the media index lacks first; the first time downloads the runtime and models into `~/.allyhub/`. `ALLYHUB_NO_INDEX=1` skips it.
+- Until one is made, a face reframe is centred, layers behind the person are drawn in front, and a background is left as it is; a `speaker` that is not a face of the track is centred too. Each is a `fallback` fact, which `allyhub inspect` prints.
+
+## Stacking with graphic layers
+
+A part's `layers` are listed back to front: each is drawn over those before it, footage and graphics alike. A graphic layer listed before a footage layer is under it (the `graphics-under-footage` fact lists such layers).
+
+A graphic layer with `z: "behind-person"` is drawn with the part's own footage (`footage()` without a file) wherever that is listed: right over its picture and under its speaker, who is cut out with the take's person matte. Without the part's own footage it is drawn in front (a `fallback` fact). `z: "overlay"` is over the finished picture ([video.md](video.md)).
+
+## The background behind the person
+
+`background` treats the picture behind the person apart from the person, through the file's person matte.
+
+```ts
+import { footage } from "@allyhub/sdk";
+
+export const backgrounds = [
+  footage({ background: { blur: 24 } }),
+  footage({ background: { dim: 0.4, desaturate: 1 } }),
+  footage({ background: { color: "var(--ak-color-bg)" } }),
+  footage({ background: { image: "./studio.jpg", blur: 6 } }),
+  footage({ background: { color: "transparent" } }),
+];
+```
+
+| Field | Range | What it is |
+| --- | --- | --- |
+| `blur` | 0 to 200 | The background's Gaussian blur, its standard deviation as CSS `blur()`, px at 1080p |
+| `dim` | 0 to 1 | Darker, to black |
+| `desaturate` | 0 to 1 | Greyer, to gray |
+| `color` | a CSS colour | A flat colour instead; `"transparent"` draws the person alone over what is under the layer. Not with `blur`, `dim`, `desaturate` or `image` |
+| `image` | PNG, JPEG or WebP | An image instead (an import or a path from the module), filling the box (cover); `blur`, `dim` and `desaturate` then treat it |
+| `feather` | 0 to 40, default 2 | The softness of the person's edge, px at 1080p (`MATTE_FEATHER`) |
+| `shrink` | −40 to 40, default 4 | How far the person's edge is pulled in, px at 1080p; negative lets it out (`MATTE_SHRINK`) |
+
+- The order is: the treated background, graphic layers drawn `z: "behind-person"`, the person, the layers in front.
+- The matte is made firm, then softened by `feather` and moved by `shrink`.
+- An image that is not a picture is AK-A003. Until the file has a matte the footage shows as it is (a `fallback` fact).
+
+## The preview and the render
+
+The render and stills draw footage with ffmpeg, the exact frame of the file for every frame. The live preview plays each footage layer as a muted `<video>` kept to the clock, placed by the same plan. Where they differ: the preview plays `sharpen` unsharpened, and approximates a `background` (its blur is of the whole picture, its matte edge coarser). `allyhub frames` draws as the render does.
+
+## A video item as footage
+
+A video item of the project, imported, is drawn by the stage at its own time, never rendered first.
+
+```ts file=demo/video.ts
+import { custom, part, video } from "@allyhub/sdk";
+
+export default video({
+  format: "16x9",
+  deliverable: false,
+  parts: () => [
+    part({
+      key: "open",
+      dur: 3,
+      anchors: { turn: 1.5 },
+      layers: [custom({ render() {} })],
+    }),
+  ],
+});
+```
+
+```ts file=film/video.ts
+import { footage, part, video } from "@allyhub/sdk";
+
+import demo from "../demo/video.ts";
+
+export default video({
+  format: "9x16",
+  parts: () => [
+    part({
+      key: "inset",
+      dur: 6,
+      layers: [footage(demo, { box: [0.55, 0.1, 0.4, 0.3], from: "@turn" })],
+    }),
+  ],
+});
+```
+
+- `footage(videoItem, opts)` takes `at`, `until`, `dur`, `from` (its own moment at the layer's start: seconds, an anchor of one of its parts such as `"@turn"`, or a part's key), `speed` (over 0, up to 16), `box` and `fit`, `audio` and `gainDb` (its speech, room and sound effects; at speed 1 only), `camera` and `cameras` (the camera its scenery layers are drawn from, on its clock: one for all, or by layer name), and `quality: { scale }` (how finely its scenery layers are drawn here, over 0 up to 4, as `scenery()`'s; [scenery.md](scenery.md)). `crop`, `reframe`, `speaker`, `punchIn`, `pullback`, `sharpen`, `background`, `shape` and `radius` throw.
+- It draws its graphic layers only: a video item with footage of its own cannot be shown inside another yet (the load fails and says so).
+- What it reads counts as the outer video's, so the part cache and the check follow it. A video that reaches itself through the videos it shows is AK-M010.
+- `deliverable: false` marks a video only shown inside others: it is not rendered as a film of its own unless named.
+- `take(videoItem).range(from, to)` makes a part that plays a stretch of it, hearing its sound and saying its words; `footage()` without a file in that part draws its picture in sync ([speech.md](speech.md)).
+- `ctx.video(videoItem, el)` draws it in an element of a layer and resolves to an `EmbeddedVideo` (`{ id, duration, width, height, seek(t, layout?), at(ref) }`); seek it from `render` through `ctx.wait`:
+
+```ts file=film/screen.ts
+import { custom, part, type EmbeddedVideo } from "@allyhub/sdk";
+
+import demo from "../demo/video.ts";
+
+const shown = new WeakMap<HTMLElement, EmbeddedVideo>();
+
+export const screen = part({
+  key: "screen",
+  dur: 6,
+  layers: [
+    custom({
+      mount(el, ctx) {
+        if (!ctx.video) return;
+        ctx.wait(ctx.video(demo, el).then((v) => shown.set(el, v)));
+      },
+      render(el, t, ctx) {
+        const v = shown.get(el);
+        if (!v) return;
+        const at = Math.min(Math.max(t, 0), v.duration - 1e-3);
+        ctx.wait(v.seek(at, ctx.layoutOnly === true));
+      },
+    }),
+  ],
+});
+```
+
+## Two recordings of one moment
+
+A separate microphone and a camera start at different moments. `allyhub align <reference> <other>` measures the offset between them from their sound (other = reference + offset), how sharply they match, and how it drifts. Writing it in is the piece's: the microphone is the part's speech, the camera its picture.
+
+```ts file=talk/two/video.ts
+import { footage, part, take, video } from "@allyhub/sdk";
+
+const OFFSET = 1.5; // from allyhub align mic.wav cam.mp4
+
+export default video({
+  format: "16x9",
+  parts: () => [
+    part({
+      key: "intro",
+      speech: take("./mic.wav").range("So here is", "every time"),
+      layers: [footage("./cam.mp4", { from: 12.84 + OFFSET })],
+    }),
+  ],
+});
+```
+
+- `align` also prints the drift as a line, offset(t) = at0 + slope × t, and the speed (1 + slope) that keeps the camera in step; and, in its JSON, `pictureOffset`: the offset for the camera's picture when its file's sound starts later on its own clock.
+- Only a take's own footage follows its cut list: to cut both, make the two one file first.
+- `allyhub frames cuts` draws a review sheet per cut (a cut take's jumps, part joins) to look at the result.