lecodes-assets 0.1.0

package/README.md

@@ -0,0 +1,226 @@
+# lecodes-assets — the LeCodes asset pipeline
+
+FBX → GLB converter + GLB doctor, one package on one glTF-Transform `Document`. Reachable as
+`lecodes assets …` (optional peer of the `lecodes` CLI) or `bun src/cli.ts …` in the monorepo.
+Plan/rationale: [docs/animation-plan.md §2.9](../../docs/animation-plan.md).
+
+```
+lecodes assets convert "Erika Archer.fbx" --clips Idle.fbx Walking.fbx "Fast Run.fbx" --drop-model-clips -o hero.glb
+lecodes assets convert hero.glb --clips Jump.fbx                 # add clips to an existing GLB
+lecodes assets convert hero.fbx --fix                            # + doctor passes (bone cap, texture normalize)
+lecodes assets convert hero.fbx --ktx2                           # + textures → KTX2 (needs KTX-Software toktx)
+lecodes assets probe hero.fbx | hero.glb                         # skeleton / clips / meshes / textures / bind sanity / world size
+lecodes assets doctor hero.glb [--fix] [--ktx2] [--dump]         # the old tools/glb-reduce
+lecodes assets doctor hero.glb --fix-scale                       # bake a unit fix (see Scale below)
+lecodes assets retarget fox.glb --from wolf.fbx --map auto       # clips from a DIFFERENT skeleton (map from skeleton shape)
+```
+
+## Retarget — clips from a different skeleton
+
+`convert --clips` binds tracks by BONE NAME and copies them verbatim; that only works when both
+files came from the same rig (the Mixamo case). `retarget` drives one rig with a completely
+different rig's clips — different bone names, different chain lengths, different proportions,
+different local axis conventions, even different facing:
+
+```
+lecodes assets retarget fox.glb --from Wolf_Animation.fbx --map auto --clips wolf_walk wolf_run -o fox-anim.glb
+lecodes assets retarget giraffe.glb --from Deer_Animation.fbx --auto-map      # writes giraffe-from-Deer_Animation.map.json to edit
+lecodes assets retarget giraffe.glb --from Deer_Animation.fbx --map giraffe-from-Deer_Animation.map.json -o giraffe-anim.glb
+lecodes assets retarget fox-anim.glb --from Wolf_Howl.fbx --map … --keep-clips --clip-names howl
+lecodes assets retarget fox.glb --from Wolf_Animation.fbx --skeletons        # both bone lists, to write a map by hand
+lecodes assets retarget fox.glb --from Cat.fbx --map auto --anim Cat_Sleep.anim   # donor clips that live outside the FBX
+```
+
+**The map** is target-bone → donor-bone; `root` names the bone that receives root motion. `--map auto`
+/ `--auto-map` derive it from the two skeletons' SHAPE — root, spine, neck, head, jaw, ears, tail and
+the four legs (side + front/back) are read off the bind pose, then paired chain by chain with bones
+matched by arc length (names only break ties when they agree: `backAnkle`↔`Ankle_L`). It works on
+`joint1…joint42` rigs and prints exactly what it paired — read it before trusting it. Optional keys:
+
+```json
+{ "root": "Hips",
+  "bones": { "Hips": "Root_M", "chest": "Chest_M", "backleg": "Hip_L", "backleg0": "Knee_L", "…": "…" },
+  "align": { "chest": "keep", "tail": "direction", "head": [ "headend" ] },   // per-bone reference policy (see below)
+  "legs": [ "backleg2", "R_backleg2", "frontleg2", "R_frontleg2" ],        // feet, when auto-detection picks wrong
+  "donorRef": "auto" | "bind" | { "clip": "wolf_idle", "time": 0 } }
+```
+
+**How it works.** Everything happens in WORLD space, where a rotation means the same thing in both
+rigs: `ΔR(t) = Rdonor(t)·Rdonor_ref⁻¹` is applied to the target's own reference orientation and
+converted back to local against the already-retargeted parent. That transfer is exact — the thing
+that decides quality is what the two REFERENCE poses are and what happens where proportions differ:
+
+- **Donor reference** (`donorRef`, default `auto`). An exporter's bind is often a posture the clips
+  never visit — the polyperfect wolf binds upright on vertical thighs and walks 10% lower with its
+  thighs 30° forward; measured from that bind every clip reads as "crouch and lean". When the two
+  binds put the legs differently (>12°: different conventions), the donor is referenced to the MEAN
+  pose of its walk (else idle, else the longest clip) — its actual neutral posture. When the binds
+  agree, bind-to-bind is the faithful transfer (an identity retarget reproduces the donor exactly).
+  `--donor-ref bind|auto|<clip>` overrides.
+- **Target reference**, per bone: `keep` its bind orientation, or `direction` — turn it so the bone
+  points the way the donor's does (an auto-T-pose step, measured to the nearest mapped descendant).
+  A target that binds in a STANCE (feet level, legs hanging) keeps everything: that is its natural
+  posture and the donor's deltas ride on it — the horse stays tall under wolf clips, the giraffe's
+  neck stays up under deer clips. A target that binds posed (T-pose, folded legs) keeps its core
+  (root, pelvis, chest, head — artists put those joints anywhere, so their joint lines are not
+  comparable) and direction-aligns its chains. `align` overrides per bone.
+- **Feet** (`--ik legs`, the default). Directions transfer, positions do not: with different
+  front/back leg ratios the feet float or sink, which the eye catches first. Legs are detected from
+  the map (lateral chains reaching the ground; a foot is what stands on the donor's ground, a horse's
+  wrist-ended front leg is not), the reference pose is planted (root lowered/raised, every foot bent
+  onto the ground), then every frame each foot gets a goal built as THE TARGET'S OWN STANCE plus the
+  donor's departure from ITS stance, scaled by that leg's length ratio, on all three axes — height
+  above each rig's own stance level, and the horizontal offset each rig plants its foot at under its
+  own hip — and a FABRIK pass from the FK pose bends the chain onto it. Both builds are therefore
+  preserved: a target that tucks its feet inboard of its hips keeps doing so against a donor that
+  stands on vertical legs. (Copying the donor's absolute foot-under-hip offset instead planted this
+  repo's fox — which tucks each foot ~8% of body height inboard, while every polyperfect animal
+  stands vertical — bow-legged in every clip.) When the legs are scaled copies the goal IS the FK
+  pose, so IK only ever spends the proportion residual; goals out of reach are reported per leg.
+- **Root motion** is scaled by the mean leg-length ratio (a stride is leg length, not body height),
+  so stance feet that were still on the donor stay still on the target.
+- **Chain lengths.** A longer donor chain collapses for free (the target's `chest` mapped to
+  `Chest_M` picks up Spine1+Spine2, because a world rotation already contains everything above it).
+  A longer target chain — unmapped bones between two mapped ones — interpolates the delta along the
+  chain (a 6-bone spine against a 2-bone one bends everywhere, not at one joint).
+- **Spaces.** The anchor is the BIND pose (inverse of the skin's inverse-bind matrices), never the TRS
+  on the nodes — the wolf's node pose is 0.37 m from its bind. A skin whose bind space is scaled
+  against node space (Maya "actor" exports, ×100) is rescaled; a donor facing the other way is turned
+  (facing is read from the skeleton shape, so `joint1…` rigs work too); non-uniform scales are refused.
+
+What it does not do yet: foot locking on the target beyond what the donor's feet do (an in-place
+donor still treadmills), stride warping when a leg is far too short for the donor's reach (reported
+as "out of reach"), torso pitch to compensate for mismatched front/hind leg ratios, bipeds/arms (the
+auto-map is quadruped-shaped; a humanoid map is best written by hand or by name).
+
+### Donors whose clips are not in the FBX
+
+Asset packs routinely ship the rig as FBX and the animations as Unity `.anim` assets — the
+polyperfect cat and all three dogs have EMPTY FBX files and eight `.anim` clips each. `--anim` reads
+those directly (`src/retarget/unityAnim.ts`, YAML only — a project saved with binary serialisation
+has to be re-exported with *Asset Serialization = Force Text*) and attaches them to the donor rig as
+clips, bound by the LAST segment of each curve's transform path:
+
+```
+lecodes assets retarget fox.glb --from Cat.fbx --map fox-from-cat.map.json   --anim Cat_Walk.anim Cat_Idle.anim Cat_Sleep.anim Cat_StandUp.anim   --clips Cat_Sleep Cat_StandUp --clip-names sleep stand_up -o fox-cat.glb
+```
+
+Pass the donor's locomotion clips even when you do not want them in the output: `--donor-ref auto`
+takes the donor's neutral posture from a walk/idle, and without one it falls back to the bind pose.
+
+### What a short spine cannot borrow
+
+Rotations transfer; positions follow from the TARGET's own bones. Leg IK covers that for feet, but
+nothing covers the spine, so a clip whose whole point is a spine CURL does not survive a target with
+fewer spine bones. The cat sits by curving a 7-bone spine to hold its chest up over a dropping
+pelvis; this fox has two spine bones and no way to curve, so its chest is simply carried down by the
+pelvis and the head ends up on the floor (4% of body height, against the cat's 79%). Hanging the
+pelvis on a mid-spine bone instead recovers part of it (27%) but then breaks lying-down clips, whose
+heads go through the floor. Clips that END lying (`Sleep`) or start there (`StandUp`) transfer fine —
+they need no upward curl.
+
+### Seeing the result
+
+A bone map is judged by eye. `tools/filmstrip.ts` skins a clip on the CPU and writes a PNG strip —
+no browser, no GPU, no engine, so it runs on every iteration:
+
+```
+bun tools/filmstrip.ts fox-anim.glb strip.png 6 wolf_walk threequarter
+BONES=1 CELL=360 bun tools/filmstrip.ts fox-anim.glb strip.png 5 wolf_walk side   # skeleton overlay
+BIND=1 BONES=1 bun tools/filmstrip.ts fox.glb bind.png 1 - side                     # the bind pose itself
+CELL=440 bun tools/filmstrip.ts fox-anim.glb strip.png 4 wolf_walk side "fox.glb|-|0"   # rest pose first
+```
+
+For the real thing on the real engine, `packages/viewer` → `node anim-check/model-check.mjs`.
+
+## Scale
+
+`probe` and `doctor` always **report** the model's rendered world size; nothing is changed without a
+flag. A skinned primitive ignores its node's transform (glTF spec), so the size is measured through
+the skin's bind space (`world(joint₀)·IBM₀`) — measuring via the mesh node reports garbage on exactly
+the assets that are mis-scaled.
+
+| flag | what it does |
+|---|---|
+| `--fix-scale` | guess the unit and bake it: cm / mm / feet / inches / the Blender-FBX `0.01` armature quirk. Picks the **smallest** correction that lands the largest dimension in 0.1–100 m |
+| `--scale N` | bake an explicit factor (works on clip-only files too) |
+| `--scale-to M` | bake whatever makes the largest dimension `M` metres |
+
+The factor is **baked into the data** — positions, node translations, IBM offsets and translation
+tracks all × s — not parked on a wrapper node. No node is added and no local scale is rewritten, so
+every node path an app addresses stays what it was. Rotation and scale tracks are untouched by
+construction: scaling only translations gives `world' = S·world·S⁻¹`, and the skinning identity then
+needs `IBM' = S·IBM·S⁻¹`, which is just the IBM's translation column × s.
+
+Quantized (`KHR_mesh_quantization`) positions are refused rather than silently corrupted.
+
+## Shading — why a downloaded model renders too bright
+
+`doctor` always reports two material quirks that make an asset look washed out, and neither one is
+the usual suspect (metallic, or the texture):
+
+- **self-lit** — the base colour texture is *also* wired into `emissiveTexture` with
+  `emissiveFactor = [1,1,1]`, so the shader draws `lit(albedo) + albedo`: roughly twice as bright as
+  intended, with the shading flattened out. Common from Sketchfab downloads and several FBX
+  exporters. `--fix-emissive` drops it and leaves the material otherwise PBR.
+- **specular boost** — `KHR_materials_specular` with a `specularColorFactor` above 1.
+
+`--unlit` goes further and converts the material to **`KHR_materials_unlit`**: flat
+`baseColorFactor × baseColorTexture`, no lights, no IBL, no exposure — the shading model hand-painted
+stylised assets are usually authored for. It also clears the emissive and drops specular/ior/normal/
+occlusion/MR inputs, because gltfio *refuses* to combine specular with unlit ("Cannot use specular
+extension with unlit") and silently falls back to lit if it is left in place.
+
+Both engines implement it: Filament through gltfio's `unlit_opaque` ubershader, web-lite through the
+`UNLIT` branch of its glTF shader (`viewer-lite/src/gl/glb.ts`, covered by `bun run render:check`).
+
+## What `convert` does
+
+- **Reads FBX with [ufbx](https://github.com/ufbx/ufbx)** compiled to wasm (`native/fbx_reader.c` +
+  `vendor/ufbx`, built by `build-wasm.sh` into `src/wasm/fbxReader.js` — a single-file module, no
+  native builds, runs in Bun/Node/browsers). Everything is converted to glTF conventions in C:
+  meters, right-handed Y-up (`space_conversion = MODIFY_GEOMETRY`), geometry transforms / inherit
+  modes as helper nodes, triangulated + welded primitives, top-4 skin weights renormalized, UV v
+  flipped, animations **baked** (30 fps, key-reduced, trimmed to start at 0). One JSON description +
+  one binary arena cross the wasm boundary; `src/fbx/toDocument.ts` builds the Document.
+- **Textures** embedded: PNG/JPEG pass through, TGA/BMP decoded here (→ PNG), everything else via
+  `sharp` when installed; duplicates de-duplicated by content hash; external references resolved
+  next to the FBX / `textures/`.
+- **Materials**: ufbx's PBR view of FBX Phong/Lambert → metallic-roughness (opacity map → MASK,
+  opacity factor < 1 → BLEND, double-sided honored). Separate metallic/roughness maps are not
+  packed (factors used).
+- **`--clips`**: each clip file's animations are re-targeted onto the model **by bone name** (exact,
+  then the part after the last `:`/`|` — the same rule the SDK `Animator` uses); clip meshes/skins
+  dropped; unbound tracks counted + warned. Naming: a single-clip file → its stack name unless generic
+  (`mixamo.com`, `Take 001`) → the file's basename; `--clip-names` overrides.
+- Output goes through the same validation as the doctor (`validateGlb`) and a `probe` summary.
+
+Skipped in v1: cameras/lights, morph targets, NURBS.
+
+## Verified with
+
+`packages/viewer` → `node anim-check/model-check.mjs anim-check/fixtures/<file>.glb --clip N`: loads the
+GLB on the Filament wasm engine, binds every clip through the native Animator, screenshots the rest
+pose and a playing clip (the mesh must draw and deform). `bun test` here covers the reader/converter
+against the Mixamo sample set when `LECODES_FBX_SAMPLES` (or the default sample folder) exists.
+
+Retargeting was tuned on the polyperfect *Low Poly Animated Animals* zoo (different authoring
+conventions within one pack: AdvancedSkeleton `Root_M/Hip_L`, Maya `actor:skin_*` with ×100 bind
+space and −Z facing, unnamed `joint1…joint42`): wolf → Meshy fox, lion → cat, wolf → horse / golden
+retriever, deer → giraffe, lion → elephant, all through `--map auto`, judged on `BONES=1` filmstrips
+next to the donor's own walk. `tests/retarget.test.ts` pins the invariants on toy rigs (exact
+direction transfer, keep/direction policies, bridges, leg detection + IK goals, donor reference).
+
+## Layout
+
+```
+native/fbx_reader.c      ufbx → JSON + arena (compiled to wasm)
+vendor/ufbx/             ufbx.h / ufbx.c (MIT)
+src/wasm/fbxReader.js    generated single-file Emscripten module (tracked; rebuild with build-wasm.sh)
+src/fbx/                 read.ts (wasm boundary + IR) · toDocument.ts · convert.ts (merge, write)
+src/textures/            resolveImage.ts (sniff, TGA/BMP decode, PNG encode, external lookup)
+src/doctor/              doctor.ts (analyze/apply/report) · reduceGlbBones · transcodeTextures · transcodeKtx2 · normalizeScale · validateGlb (mirror of viewer's)
+src/retarget/            rig.ts (hierarchy + bind pose + clip sampling) · retarget.ts (reference poses, world-delta transfer, leg IK, bridges) · automap.ts (skeleton classification + chain pairing) · unityAnim.ts (Unity .anim YAML reader)
+tools/filmstrip.ts       CPU-skinned PNG filmstrip of a GLB clip (browser-free visual check)
+src/probe.ts · src/cli.ts · src/index.ts
+```