xbintsc 0.3.69 → 0.3.81

package/doc/gui/media.md

@@ -0,0 +1,87 @@
+# Media: images, video and audio
+
+> Part of the [GUI extension](../gui.md); the engine overview, locked
+> decisions and roadmap live there.
+
+## Images (M6a)
+
+`runtime/ext_gui/image.{h,cpp}` wraps the vendored **stb_image** header
+decoding PNG/JPEG/BMP/GIF/TGA to RGBA8, with a decode cache and a header-only
+size cache (`stbi_info`). Paths accept a `file://` prefix and percent-encoding.
+`runtime/ext_gui/svg.{h,cpp}` adds SVG on top: it wraps the vendored **NanoSVG**
+library (parse + anti-aliased scanline rasteriser), detects an SVG by extension
+or by its contents, and produces the same RGBA8 `Image` the pipeline already
+consumes. `stb_image` cannot decode SVG, so this is what makes
+`<img src="logo.svg">` work.
+
+- `<img>` is a replaced element (`display: inline-block`): layout gives it the
+  CSS size when set, otherwise the intrinsic pixel size, and preserves the
+  aspect ratio when only one axis is constrained. The intrinsic size is read
+  from the file header (no full decode) while building the box tree, and for an
+  SVG it comes from the parsed `width`/`height` or the `viewBox` (falling back
+  to 300x150 when neither is present).
+- Paint emits one `PaintImage` per `<img>`; the renderer uploads each unique
+  `src` to an RGBA texture and draws textured quads, batching consecutive quads
+  that share a texture. A **bitmap** decodes at its intrinsic resolution and is
+  cached by path; a **vector** (SVG) is rasterised for the drawn size in device
+  pixels (logical size × the window's pixel density), preserving the viewBox
+  aspect ratio, and cached by that size, so scaling or a HiDPI display produces
+  a crisp render instead of a magnified bitmap.
+- The parse is cached per path; the parse and rasteriser state are released in
+  `xt_svg_shutdown` (`xt_image_shutdown`).
+
+Still to do for images: CSS `background-image: url(...)`, `data:` URIs,
+`object-fit`, 9-slice borders and inline `<svg>` elements (only SVG *files* are
+supported; see [Open questions](./known-issues.md#open-questions)).
+
+## Video & audio (M13)
+
+`runtime/ext_gui/media.{h,cpp}` wraps a statically vendored **FFmpeg** build and
+`runtime/ext_gui/gui_media.cpp` connects the DOM tree to it. `<video>` and
+`<audio>` really decode and play; there is no WebView and no platform media
+framework in the path.
+
+- **Decoding.** `avformat_open_input` opens the `src` (a path — `file://` is
+  accepted, but media paths are *not* percent-decoded, unlike images), the
+  best stream is selected for each kind, and a software decoder yields
+  `yuv/rgb` frames which `sws_scale` converts to RGBA8 at the coded size. The
+  supported containers/codecs are the minimal FFmpeg configuration from
+  `scripts/build-gui.ts`: MP4/MOV/MKV/AVI/FLV/MPEG-TS/M4V/ASF, MP3/OGG/WAV/FLAC/
+  AAC; H.264, HEVC, VP8, VP9, AV1, MPEG-4, MJPEG video; AAC, MP3, Opus, Vorbis,
+  FLAC, ALAC, AC3/E-AC3, WMAv2 and the raw PCM formats audio.
+- **Video.** `<video>` is a replaced element like `<img>` (UA default
+  `300x150`, overridden by the decoded size via `setMediaSize`). Paint emits the
+  same `PaintImage` as an image; the renderer uploads each new decoded frame to
+  an RGBA texture (`ensureMediaTexture`) and draws a textured quad, so the GPU
+  scales it to the CSS box instead of the CPU resizing. Playback timing is the
+  wall clock (`SDL_GetTicks`), not the frame delta.
+- **Audio.** There is no audio callback: the decoder is pushed on the main
+  thread as it is ticked and the PCM is queued with
+  `SDL_OpenAudioDeviceStream` in push mode (`NULL` callback), buffering about
+  0.25 s ahead at 48 kHz stereo `SDL_AUDIO_F32`. `volume` maps to
+  `SDL_SetAudioStreamGain`; audio is best-effort and is skipped when no device
+  is available (headless CI).
+- **Threading.** The engine is single-threaded and not thread-aware, so all
+  decoding happens on the main thread inside `xt_gui_media_tick` — no decode
+  thread, no cross-thread GC exposure.
+- **DOM API.** Element handles expose the media surface directly (see
+  `runtime/ext_gui/dom_api_media.cpp` and the prototype in
+  `dom_api_proto.cpp`): `play()` / `pause()` / `load()`, and the
+  `paused` / `ended` / `duration` / `currentTime` / `volume` / `muted` / `loop`
+  / `autoplay` / `videoWidth` / `videoHeight` / `readyState` / `src`
+  properties. Media elements also emit `loadedmetadata`, `play`, `pause`,
+  `ended`, `timeupdate` (every 250 ms) and `error`, so the usual
+  `addEventListener` pattern works.
+- **No built-in controls.** `<video controls>` is deliberately *not*
+  implemented: the element always renders the raw frame, and the program draws
+  its own chrome (a progress bar, buttons) with ordinary elements and CSS. See
+  `examples/gui/media.html` (`examples/gui/media.ts`) for the pattern.
+- **Lifecycle.** Players are owned per window (`XtGuiMediaEntry`, keyed by
+  node). `xt_gui_media_sync` drops entries whose node is gone, reopens on a
+  `src` change, and opens on first use; `xt_gui_media_shutdown` runs on window
+  close/quit/`loadHTML`.
+
+Still to do for media: `<video controls>`, separate audio/video clock drift
+correction, hardware decode, streaming protocols (only `file` is enabled),
+subtitles, `playbackRate` and per-frame `requestVideoFrameCallback`.
+

package/doc/gui/rendering.md

@@ -0,0 +1,113 @@
+# Paint, GPU and text
+
+> Part of the [GUI extension](../gui.md); the engine overview, locked
+> decisions and roadmap live there.
+
+## Paint and display list (M4a/M4b)
+
+`runtime/ext_gui/paint.{h,cpp}` walks the layout tree in painter's order and
+emits a backend-agnostic `DisplayList`: a painter-order `ops` list of
+**rectangles** (backgrounds and four solid border edges, with `border-radius`),
+**text runs** (each carrying its text, colour and resolved `FontSpec`) and
+**images**, backed by parallel payload arrays. Each op also carries the active
+clip rectangle (the padding box of the nearest `overflow: hidden` ancestor), so
+`overflow` clipping and `z-index` ordering survive into the renderer.
+`DisplayList::dump()` feeds `paintList()`/`paintCount()`.
+
+`opacity` is honoured only at its extremes: a box whose computed `opacity` is
+`0` (and its whole subtree) paints nothing, and anything above `0` paints fully
+opaque. A shape carries its colour's alpha only, so there is no compositing of a
+partly transparent box — the `opacity` transitions the CSS parser accepts are
+therefore not visible either. Hiding and revealing a layer with `opacity: 0` on
+the hidden state is what `examples/gui/pelican-bike`'s game-over overlay does.
+
+`runtime/ext_gui/renderer.{h,cpp}` turns that list into batched vertex buffers
+per window (shapes, glyph quads and images) and merges consecutive ops that
+share a pipeline, clip and texture into **draw runs**. Each run is drawn with
+its own GPU scissor rectangle (scaled to device pixels), so painter order and
+`overflow` clipping are applied per op through the SDL_GPU graphics pipelines:
+
+- The shared pipelines are created lazily from the device's supported shader
+  format. Each of SDL_GPU's three backends consumes a different binary and none
+  of them compiles GLSL/HLSL at runtime, so the engine ships all three and
+  `selectShader` (`renderer_shaders.h`) picks the one the device accepts:
+  **MSL** for Metal (compiled by SDL from the embedded source),
+  **SPIR-V** for Vulkan (`runtime/ext_gui/spirv/*.{vert,frag}`, compiled by
+  `glslc`), and **DXIL** for Direct3D 12 (`shaders.hlsl`, compiled by `dxc`).
+  `scripts/build-gui-shaders.mjs` regenerates the embedded blobs and the result
+  is committed, so nothing extra is needed to build the engine.
+  The descriptor bindings differ per format — SPIR-V follows what
+  `SDL_gpu_vulkan.c` builds (uniforms `set=1, binding=0`, samplers
+  `set=2, binding=0`, textures `set=2, binding=1`) — and glslc names every entry
+  point `main` where MSL/DXIL keep the descriptive names.
+  Note that SDL 3.2.10's Direct3D 12 backend cannot create a graphics pipeline
+  whose shaders declare a uniform buffer (it fails with `E_INVALIDARG`); the
+  engine passes the viewport that way, so the DXIL path is built but not usable
+  until that is resolved upstream.
+- A **rounded-rectangle distance field** in the shape fragment shader gives
+  antialiased fills; the vertex carries `position`, `local`, `half extents`,
+  `radius` and colour, and a viewport-size push constant does the projection.
+  Alpha blending is enabled.
+- The text pipeline samples a **single shared grayscale glyph atlas**
+  (`R8_UNORM`, 2048², shelf-packed, LINEAR filtering) and draws each glyph as a
+  textured quad (`position`, `uv`, colour), modulating alpha by the coverage.
+- Geometry is uploaded only when the document or viewport changes
+  (`geometry.dirty`), so steady-state frames are bind-and-draw.
+
+Each op also carries the `Matrix2D` of the box it came from
+(`runtime/ext_gui/transform.h`, `PaintRect::transform` and friends). It is
+applied while the vertex buffer is built, not in the shader: `pushRect` maps its
+four corners through the matrix (the distance-field `local`/half-extent/radius
+fields stay in the *box* frame, so a rounded corner stays round after a
+rotation), and `pushQuad`/`pushGlyph` map the glyph or image quad the same way.
+That kept the MSL/SPIR-V/DXIL shaders and the pipelines untouched — a transform
+costs a few multiplies per vertex and nothing per pixel. A clip rectangle of a
+transformed box is the bounding box of the transformed padding box, because the
+GPU scissor is axis aligned. Text is the one case that needs more than
+multiplying corners: a scaling transform would magnify an atlas raster, so the
+atlas is asked for glyphs at the device scale **times the matrix's
+magnification** (clamped to 0.5–4x, since the atlas is finite) and the quad is
+divided back into logical units, exactly the way HiDPI scaling already worked.
+
+The window background (`setBackground`) is the render-pass clear colour.
+
+Still to do in M4: gradients.
+
+## Text stack (M4b)
+
+`runtime/ext_gui/text.{h,cpp}` wraps **HarfBuzz** (shaping) and **FreeType**
+(metrics + eventual rasterisation). Both are built statically and linked into
+`gui.a` by `scripts/build-gui.ts`.
+
+- Fonts are resolved from well-known system paths (Helvetica/Arial on macOS,
+  DejaVu/Liberation on Linux, Segoe UI/Arial on Windows), overridable with
+  `XT_GUI_FONT` (and `XT_GUI_FONT_MONO`), and cached per `(family class, size)`.
+  Only regular upright faces are used for now; weight/italic selection is a
+  later refinement.
+- The default candidates prefer a **CJK-capable** face (Microsoft YaHei on
+  Windows, PingFang/STHeiti on macOS, Noto Sans CJK/WenQuanYi on Linux) before
+  the Latin-only ones. Setting `XT_GUI_FONT` to a Latin-only face gives its
+  Latin design instead (at the cost of CJK coverage).
+- **Codepoints the primary face lacks fall back to another face.** The engine
+  keeps a small fallback chain — the platform's symbol face (Segoe UI Symbol,
+  Apple Symbols, Noto Sans Symbols, DejaVu Sans) first, then the other family
+  class — and resolves each codepoint to the first face in it that has the
+  glyph. The run is shaped in segments, one per face, so a mixed run (Latin
+  plus an arrow, box-drawing or geometric shape, or Latin inside CJK) draws
+  every glyph instead of dropping the missing ones as `.notdef`. This is what
+  makes a character such as `▮`/`▯` (U+25AE/U+25AF, used by the pelican-bike
+  speed bar) visible even when the CJK primary has no geometric shapes. Each
+  shaped glyph carries its face (`ShapedGlyph::font`) so the renderer rasterises
+  it from the right atlas entry; `xt_text_scale` maps a face to the display's
+  scale factor.
+- `xt_text_measure_width` shapes the run with HarfBuzz (so kerning and
+  ligatures are honoured), `xt_text_metrics` returns FreeType's ascent /
+  descent / normal line height. When no font file can be found the module falls
+  back to a deterministic per-byte approximation, so layout still works.
+- `xt_text_shape_run` returns positioned glyphs and `xt_text_rasterize` renders
+  an 8-bit bitmap; the renderer packs those into the atlas. On HiDPI displays
+  glyphs are rasterised at `font_size * SDL_GetWindowPixelDensity` while quads
+  are positioned in logical pixels, so text stays crisp.
+- Layout uses these real metrics for text widths, line breaking and
+  `line-height: normal`; `measureText`/`fontMetrics` expose them to tests.
+

package/doc/gui/runtime.md

@@ -0,0 +1,88 @@
+# Runtime integration and memory
+
+> Part of the [GUI extension](../gui.md); the engine overview, locked
+> decisions and roadmap live there.
+
+## Event loop integration
+
+The runtime change (already landed):
+
+- `int xt_loop_poll(int timeout_ms)` — one reactor iteration; `0` polls without
+  blocking, `< 0` blocks.
+- `void xt_loop_set_main(xt_main_loop_fn fn)` — a host may take over the main
+  loop. `xt_run_event_loop()` delegates to it; the generated `main` is unchanged.
+- `xt_loop_set_main(NULL)` restores the default `select(2)` loop.
+
+The GUI engine can either register its own loop through `xt_loop_set_main`, or
+expose an explicit `run()`; **M2 uses the explicit `run()`** so the window is a
+plain native call the TypeScript program controls. Each tick it:
+
+1. pumps SDL window/input events for every window,
+2. builds/uploads the display list when it changed and renders every open
+   window (clear pass + shape geometry + text geometry),
+3. calls `xt_loop_poll(0)` and `xt_drain_microtasks()` so sockets/timers and
+   `await` continuations keep making progress,
+4. repeats until all windows close or `quit()` is called.
+
+The `xt_loop_set_main` hook remains available for hosts that want to own the
+loop themselves.
+
+This keeps network I/O, timers and `await` working inside a GUI program.
+
+## Native ↔ TS bridge
+
+- **TS → engine**: direct `xt_gui_*` calls / window methods.
+- **Engine → TS**: `xt_call_with_this(fn, thisValue, argc, argv)` with function
+  values captured from TS (e.g. event handlers registered with `win.on(...)`).
+- **Future page→native RPC**: not needed while there is no page JS; native TS is
+  the controller. If a declarative layer is added later, it will use the same
+  `xt_call_*` entry points.
+
+## Memory & GC
+
+The engine owns most of a GUI program's memory outside the runtime heap: the DOM
+tree, the detached-node pool, the element-handle tables, the layout box tree and
+the display list are C++, so the mark-sweep collector cannot see them. `stats()`
+(see [Diagnostics](./api.md#diagnostics)) reports the parts that can grow:
+
+| Counter | Expected in a steady state |
+| --- | --- |
+| `treeNodes` | constant — the authored document |
+| `detachedRoots`, `detachedNodes` | 0, or constant for nodes the script removed itself and still holds |
+| `handles`, `handleSlots` | constant — one per element the script still refers to |
+| `listeners` | constant — one entry per element with listeners |
+| `heapBytes`, `heapAllocations` | sawtooth around the collector's threshold |
+
+Everything else is per-frame scratch (boxes, display list, vertex staging) that is
+freed before the next frame. A counter that climbs with the frame count is the
+leak signature to look for first; `tests/e2e/gui-retention.test.ts` asserts that
+the usual mutation patterns leave all of them unchanged.
+
+Two engine details matter when reading those numbers:
+
+- **What releases nodes.** `innerHTML`/`inner`/`textContent`/`replaceChild`
+  release the subtrees they drop; `removeChild`/`remove` keep them (see [Node
+  lifetime](./dom.md#node-lifetime)). A game that rebuilds a sprite with
+  `innerHTML` therefore runs in constant memory.
+- **What can still accumulate.** A node created with `createElement` and never
+  attached stays in the pool, because the collector has no weak references and
+  the engine cannot tell that the script dropped its handle. The same applies to
+  a node removed and then forgotten by the script. Both are bounded by the number
+  of *live* handles, but a program that creates elements per frame without
+  attaching them grows the pool with the frame count.
+- **Vector images are cached by pixel size.** A bitmap `<img>` decodes once and
+  is cached by path, but an SVG is rasterised at the drawn device size, so the
+  decoded-pixel and GPU-texture caches hold one entry per distinct size. That is
+  constant for a static layout (the usual case) and small for a handful of
+  sizes, but a program that reflows one SVG to a new integer size every frame
+  accumulates entries; prefer animating a bitmap, or keep the vector to a fixed
+  size.
+
+The collector behind `heapBytes` is a non-moving mark-sweep (`runtime/xt_alloc.c`).
+Its trigger follows the live set: after every collection the threshold becomes
+twice the surviving heap (floor 1 MiB), so a small live set means small, frequent
+collections and a bounded resident set. `XT_GC_THRESHOLD=<bytes>` pins the
+threshold instead, and `XT_GC_TRACE=1` prints one line per collection (heap and
+live-allocation counts before and after, the next trigger, and the live set by
+object kind) to stderr — that is how the DOM leak was traced to the node pool.
+

package/doc/gui-scripts.md

@@ -241,6 +241,9 @@ frees, so cycles are harmless).
 - mutation: `appendChild` / `insertBefore` / `removeChild` / `replaceChild` /
   `remove` / `cloneNode(deep?)`.
 - geometry/focus: `getBoundingClientRect`, `focus`, `blur`, `click`.
+- media (`<video>` / `<audio>`): `play()` / `pause()` / `load()` and the
+  `src`, `paused`, `ended`, `duration`, `currentTime`, `volume`, `muted`,
+  `loop`, `autoplay`, `videoWidth`, `videoHeight`, `readyState` properties.
 - events: `addEventListener(type, fn, optionsOrCapture?)`,
   `removeEventListener`, `dispatchEvent`.
 
@@ -342,9 +345,10 @@ legacy `target` field over the handle.
 
 ## Docs to update
 
-- `doc/gui.md` — Non-goals ("Executing page `<script>`" → "Executing *runtime*
-  page scripts; compile-time scripts are AOT-compiled"), locked decision #1,
-  TS-facing API, milestones, progress log.
+- `doc/gui.md` (overview) and `doc/gui/dom.md` — Non-goals ("Executing page
+  `<script>`" → "Executing *runtime* page scripts; compile-time scripts are
+  AOT-compiled"), locked decision #1, the AOT script section, milestones and
+  the progress log. The TS-facing API is in `doc/gui/api.md`.
 - `doc/implemented.md` / `doc/unimplemented.md` — move the new DOM/script
   capabilities across.
 - `README.md` — mention the DOM API + AOT scripts.