@carbonenginejs/runtime-resource 0.11.1 → 0.12.0

package/docs/formats/webgpu/guides/effect-packaging.md

@@ -0,0 +1,199 @@
+# Build a CEWGPU package from compiled effect bytes
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu`
+Audience: Shader-tool authors and engine integrators
+Summary: Shows how to package the complete supported pass scope of one selected compiled-effect body as CEWGPU data.
+
+## Purpose
+
+Use `buildEffect` when an application or build tool already has compiled
+effect bytes and needs one structurally valid CEWGPU package for a selected
+effect body. The operation performs effect analysis, exact selection,
+DXBC-to-IR lowering, pass-global binding allocation, WGSL emission, package
+assembly, and structural qualification.
+
+## Prerequisites
+
+- Caller-supplied compiled `.sm_*` effect bytes.
+- An exact technique, pass index, and complete stage list when deterministic
+  selection matters.
+- Permutation assertions for every axis whose value must not depend on effect
+  defaults.
+- A complete, internally consistent source permutation header: every ordered
+  axis must have valid names/options/defaults, and every Cartesian permutation
+  must have one correctly indexed, in-bounds body record. PGRF construction
+  validates the whole header even though only one body is translated. The
+  synchronous builder accepts at most 65,536 Cartesian permutations.
+
+## Build one pass
+
+```js
+import { CjsWebgpuFormat } from "@carbonenginejs/runtime-resource/formats/webgpu";
+
+const result = CjsWebgpuFormat.buildEffect(effectBytes, {
+    source: "res:/graphics/effect.dx11/example.sm_hi",
+    mode: "selected",
+    permutation: [
+        { name: "QUALITY", value: "HIGH" }
+    ],
+    selection: {
+        techniqueName: "Main",
+        passIndex: 0,
+        stageNames: [ "vertex", "pixel" ]
+    }
+});
+
+const packageBytes = result.bytes;
+const emittedShaders = result.wgsl;
+```
+
+The `source` value is diagnostic provenance only. The method does not open
+that path. Callers may separately provide `sourceIdentity` with a canonical
+`logicalPath` plus optional `game`, `client`, `build`, `md5`, and `sha256`
+fields. Its logical path need not equal the diagnostic label. `BuildEffect`
+always computes the lower-case SHA-256 digest over the exact input byte view;
+when a caller supplies `sha256`, the build fails if it does not match.
+
+Version-15 packages use INFO schema version 3 while the binary CEWGPU
+container remains version 1. INFO v3 identifies the `webgpu` target, the
+producing `@carbonenginejs/runtime-resource/formats/webgpu` package version, and the
+`dxbc-js-wgsl` translator version. Versions 8-14 use INFO v2 without
+reflection. The reader continues to accept legacy INFO v1 packages, INFO v2
+packages without PGRF, and INFO v2 packages with selected-body RFLX v1.
+
+Every new selected-effect package also includes a complete source permutation
+graph in `PGRF`. This preserves every axis, Cartesian permutation index,
+option-index tuple, compiler alias, and unique raw body identity even though
+only one selected body's WGSL is currently packaged. INFO v3 binds the exact
+PGRF bytes with SHA-256.
+
+For version-15 input, the package also includes complete portable reflection
+for every unique source body in RFLX v2 and its exact immutable byte payloads
+in one shared `RBLB`. These include authored parameter/resource metadata,
+constant defaults, stage/library source programs, signatures, static samplers,
+annotations, and the opaque native source hash. Earlier source versions keep
+the legacy package surface because portable reflection version 1 is
+intentionally version-15-only.
+
+## Result
+
+The returned record contains:
+
+| Field | Purpose |
+| --- | --- |
+| `bytes` | Encoded CEWGPU package bytes. |
+| `info` | Translator and package information. |
+| `metadata` | Selection and caller provenance. |
+| `permutationGraph` | Complete source permutation topology and identity-only body table. |
+| `reflection` | Complete all-unique portable source reflection for version-15 input, otherwise `null`. |
+| `reflectionBlobs` | Exact RBLB bytes for reflected programs/defaults/native hash, otherwise `null`. |
+| `analysis` | Compact selected-body diagnostic binding/stage data; not lossless effect reflection. |
+| `wgsl` | Portable shader set and pass layouts. |
+| `inspection` | Summary produced by reading the built package. |
+| `qualification` | Structural conversion outcome. |
+
+`mode: "selected"` is the default and currently the only supported backend
+body mode. The package retains normalized analysis for that resolved body while
+emitting WGSL for the selected complete passes. PGRF retains the complete
+source permutation graph and RFLX/RBLB retain every unique version-15 body's
+portable source reflection. `META.bodyIndex` selects a PGRF variant; its
+`bodyKey` selects the matching RFLX body. A raw reader exposes that join as
+`GetPortableEffectReflection(permutationIndex)`, returning a freshly owned,
+format-hlsl-validated single-body document with every byte reference expanded
+to `Uint8Array`. `runtime-resource` `Tr2EffectRes` consumes that document,
+hydrates a canonical device-free `Tr2Shader`, and caches it by selected body
+index. Engines still own prepared pipelines and GPU realization.
+
+`mode: "all"` additionally packages translated programs, layouts, and resource
+transforms for every unique body in a `WGSB` chunk, and
+`GetBackendBodyPrograms(permutationIndex)` resolves any permutation to them.
+Because one pass of one body is the translation unit, bodies whose pass is
+byte-identical share a single stored unit rather than duplicating its WGSL.
+Bodies the compiler cannot lower stay in the package as explicitly unsupported
+records with a reason, and never remove that body's source reflection.
+
+JSON `Read` output exposes `reflection` with byte references and
+`reflectionBlobByteLength`. Consumers that need exact defaults or source
+program bytes use `Read(bytes, { emit: "raw" })`, then call
+`GetPortableEffectReflection(permutationIndex)`. Omit the index to use
+`META.bodyIndex`. For individual payload access,
+`GetReflectionBlob(referenceOrKey)` returns an owned `Uint8Array`; an object
+reference must exactly match its RFLX inventory entry.
+
+Raw stage bytecode, decoded DXBC instruction trees, and compiler IR are
+transient build inputs and are not embedded in `ANLS`. Use `AnalyzeEffect`
+when return-only DXBC/IR diagnostics are required.
+
+The qualification record distinguishes structural package validity from
+broader completeness. `packageValid` means only that the selected CEWGPU
+container passed required-chunk, schema, cross-document, key, layout, and
+selection reconciliation. Version-15 INFO v3 reports `sourceComplete: true`
+because PGRF plus RFLX/RBLB cover every unique body's portable reflection and
+immutable exact defaults for that exact input file. This does not embed raw
+body records or make CEWGPU an archive of the original `.sm_*` bytes.
+`backendComplete` would additionally require every required translated
+program, layout, and transform. `runtimeComplete` would require
+complete-resource hydration and selection. None of these fields is
+prepared-pipeline or rendered evidence. The same four booleans are retained
+under `INFO.completeness` in the package.
+
+Shader-tier and permutation evidence remain orthogonal to source reflection.
+High is `.sm_depth`; Medium is `.sm_hi`; Low is `.sm_lo`. Source completeness
+applies only to the exact input tier. For unpacked Quad ship gates, explicitly
+select `SPACE_OBJECT_PPT_ENABLED=SOPPT_ENABLED`: an all-unique RFLX does not
+turn a PPT-disabled selected WGSL body into PPT-on backend evidence.
+
+When exact semantic metadata and shader use prove an allowed physical resource
+coalescing, the returned WGSL document is a `CJS_WGSL_SET` version 3 record.
+Its `resourceTransforms` recipes are required runtime work, not optional
+diagnostics: the consumer must build the described resource and bind it through
+the matching transformed layout entry. See the package-format contract before
+passing version 3 output to an engine. Packages without transforms remain WGSL
+set version 2.
+
+## Binding scope
+
+A D3D resource tuple is stage-local unless the caller has authoritative
+metadata proving that the vertex and fragment declarations name one compatible
+resource. Build one binding plan from the complete stage set; do not build
+independent stage plans and combine them afterward.
+
+When compatible sharing is proven, pass the base binding identity through
+`sharedIdentities`:
+
+```js
+const plan = CjsWebgpuFormat.buildWgslBindingPlan(
+    [ vertexIr, fragmentIr ],
+    { sharedIdentities: [ "uniform-buffer:0:0" ] }
+);
+```
+
+Unshared identities receive distinct `@vertex` or `@fragment` scopes and
+numeric binding slots.
+
+## Errors
+
+Conversion fails explicitly when:
+
+- a permutation assertion is unknown or unresolved;
+- any axis/default/option or positional body record in the complete source
+  permutation header is malformed, out of bounds, partially overlapping, or
+  inconsistent, even when the selected body itself is translatable;
+- the Cartesian permutation product exceeds the 65,536-entry synchronous-build
+  limit;
+- the technique, pass, or requested stage does not exist;
+- the requested stage list is incomplete or duplicated;
+- the selected shader uses unsupported semantics;
+- resource declarations cannot form one unambiguous pass layout; or
+- emitted package records fail structural validation.
+
+An unsupported requested shader aborts selected-pass packaging; no partially
+translated pass is emitted. This fail-closed behavior does not imply
+whole-effect completeness.
+
+## Related documentation
+
+- [Public API reference](../reference/api.md)
+- [CEWGPU package format](../formats/cewgpu.md)
+- [WGSL compatibility](../reference/wgsl-compatibility.md)

package/docs/formats/webgpu/reference/api.md

@@ -0,0 +1,197 @@
+# Public API reference
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu`
+Audience: Shader-tool authors and engine integrators
+Summary: Lists the public `CjsWebgpuFormat` profile, one-shot helpers, options, and output contracts.
+
+## Export
+
+The package root exports `CjsWebgpuFormat` as both a named and default export:
+
+```js
+import CjsWebgpuFormat, {
+    CjsWebgpuFormat as WebgpuFormat
+} from "@carbonenginejs/runtime-resource/formats/webgpu";
+```
+
+## Reusable profile
+
+Construct a profile when several operations share output, source, permutation,
+schema, or class-registration options:
+
+```js
+const reader = new WebgpuFormat({
+    emit: "json",
+    source: "example.cewgpu",
+    decodeInstructions: true,
+    permutation: null
+});
+```
+
+| Instance method | Purpose |
+| --- | --- |
+| `SetValues(options)` | Merges reusable profile defaults. |
+| `GetValues(options?)` | Returns effective values with optional per-call overrides. |
+| `SetClasses(classes)` | Registers several package-shape constructors. |
+| `SetClass(type, Class)` | Registers or removes one constructor. |
+| `GetClass(type)` | Returns one registered constructor. |
+| `HasClass(type)` | Reports whether a constructor is registered. |
+| `Read(bytes, options?)` | Reads CEWGPU bytes as JSON or a raw package. |
+| `Inspect(bytes, options?)` | Returns a package summary. |
+| `Build(chunks)` | Builds CEWGPU bytes from ordered chunks. |
+| `AnalyzeEffect(bytes, options?)` | Builds normalized analysis from compiled effect bytes. |
+| `BuildEffect(bytes, options?)` | Converts one selected effect body/pass scope into CEWGPU data. |
+| `BuildShaderIr(input, options?)` | Builds validated shader IR from DXBC bytes or decoded input. |
+| `BuildWgsl(input, options?)` | Emits a supported typed shader as WGSL. |
+| `BuildWgslBindingPlan(programs, options?)` | Allocates one binding layout across a complete pass. |
+| `BuildWgslSet(entries)` | Assembles emitted shaders and pass layouts. |
+| `ToJSON(value)` | Converts format output to JSON-compatible data. |
+
+`Read` currently returns plain data. Class registrations are validated and
+stored for forward compatibility but do not hydrate the returned package.
+Raw output is an internal zero-copy package view; callers must treat its chunk
+bytes as immutable and reread/rebuild after any byte change.
+
+## One-shot static helpers
+
+The static helpers use lower camel case and share the implementation of the
+instance methods:
+
+| Static helper | Purpose |
+| --- | --- |
+| `isCewgpu(bytes)` | Checks the `CWGP` package magic. |
+| `read(bytes, options?)` | Reads one package. |
+| `inspect(bytes, options?)` | Inspects one package. |
+| `build(chunks)` | Builds one package. |
+| `analyzeEffect(bytes, options?)` | Analyzes one compiled effect. |
+| `buildEffect(bytes, options?)` | Builds one selected effect body/pass scope. |
+| `buildShaderIr(input, options?)` | Builds shader IR. |
+| `buildWgsl(input, options?)` | Emits WGSL. |
+| `buildWgslBindingPlan(programs, options?)` | Allocates a pass binding plan. |
+| `buildWgslSet(entries)` | Builds a portable shader set. |
+| `toJSON(value)` | Converts output to JSON-compatible data. |
+
+## Profile options
+
+| Option | Meaning |
+| --- | --- |
+| `emit` | `"json"` by default or `"raw"` for the internal package object. |
+| `source` | Caller-owned diagnostic label; it is never opened. |
+| `decodeInstructions` | Includes decoded instruction and shader IR detail during analysis. |
+| `permutation` | Exact NAME=VALUE assertions as an array or `Map`. |
+| `schema` | Optional caller schema record retained by the profile. |
+| `classes` | Optional constructor registrations keyed by `CLASS_KEYS`. |
+
+`AnalyzeEffect` decodes real selected-body stage bytes for return-only
+diagnostics. `decodeInstructions: false` retains compact DXBC program metadata
+without instruction or IR trees. `BuildEffect` keeps those bytes transient for
+WGSL compilation and writes compact selected-body `ANLS` diagnostics instead.
+Both `AnalyzeEffect` and `BuildEffect` reject malformed, duplicate, unknown, or
+unresolved permutation assertions rather than silently selecting a default.
+
+## Effect-package options
+
+`BuildEffect` and `buildEffect` accept `mode: "selected"`, which remains the
+default. They resolve one permutation body and emit complete passes within the
+requested stage selection.
+
+`mode: "all"` additionally translates every unique source body into a `WGSB`
+chunk and reports `backendBodyCoverage: "all-unique"`, or `"partial"` when some
+body could not be lowered. It requires complete version-15 source reflection and
+fails closed on versions 8-14, which carry no validated body inventory. The
+orchestration compatibility option `allPermutations: true` selects the same
+mode, and `allPermutations: false` means selected mode.
+
+`CewgpuPackage.GetBackendBodyPrograms(permutationIndex)` resolves any
+permutation to its translated passes, defaulting to `META.bodyIndex`. It returns
+null when the package carries no all-body graph, and an explicitly unsupported
+record when that body could not be lowered.
+
+Both `GetBackendBodyPrograms` and `GetPortableEffectReflection` return null
+until the package has passed canonical envelope validation. Every documented
+read entry point validates, so this is transparent to normal consumers; it
+prevents a hand-assembled or tampered container from being hydrated as though
+it had been checked.
+
+`source` remains a caller-owned diagnostic label. An optional
+`sourceIdentity.logicalPath` records the canonical resource identity
+independently and may differ from that label. The builder records the exact
+source byte length and computes a lower-case SHA-256 digest over the active
+input byte view. A caller-supplied `sourceIdentity.sha256` is accepted only
+when it matches that digest.
+
+For version-15 input, `BuildEffect` emits INFO schema version 3 with explicit
+WebGPU target, backend-package name/version, translator provenance, and
+source/backend body coverage. Versions 8-14 emit INFO v2 without reflection.
+The CEWGPU binary container remains version 1. The reader retains legacy INFO
+v1, pre-PGRF INFO v2, and selected-body INFO v2/RFLX v1 support.
+
+New packages also emit a `PGRF` permutation graph and expose it as
+`result.permutationGraph`, JSON-read `permutationGraph`, and raw
+`CewgpuPackage.permutationGraph`. The graph contains every ordered axis,
+Cartesian permutation index, option-index tuple, source record, and
+package-local unique-body key/digest. `Inspect` reports `permutationCount` and
+`uniqueBodyCount`. This is complete source topology with identity-only bodies;
+it does not provide backend translation by itself.
+
+For version-15 input, new packages emit complete all-unique source reflection
+in RFLX v2 and exact referenced byte payloads in one shared `RBLB`. Build
+results expose
+these as `result.reflection` and `result.reflectionBlobs`. JSON reads expose
+`reflection` plus `reflectionBlobByteLength`; raw reads expose
+`CewgpuPackage.reflection`, `reflectionBlobBytes`, and
+`GetReflectionBlob(referenceOrKey)`. Raw reads also expose
+`GetPortableEffectReflection(permutationIndex)`: it performs the PGRF/RFLX
+join, expands every referenced payload to fresh owned `Uint8Array` values, and
+reruns the format-hlsl portable validator. Its optional index defaults to
+`META.bodyIndex`; legacy selected-body RFLX v1 accepts only its selected
+permutation. `GetReflectionBlob` remains the lower-level accessor and requires
+an object reference to match its stored key, offset, byte length, and digest
+exactly. `Inspect` reports reflection body/source-program/blob counts and blob
+byte length. The body count covers every PGRF unique body. The selector joins
+`META.bodyIndex -> PGRF.variants[index].bodyKey -> RFLX.bodies[].bodyKey`.
+Earlier source versions omit both chunks.
+
+The returned structural qualification separates preservation from execution.
+`packageValid` reports successful container construction. Version-15 INFO v3
+reports `sourceComplete: true` for the exact input's portable semantic graph
+while `backendComplete` and `runtimeComplete` remain false. Versions 8-14
+report all three completeness flags false. Source completeness does not embed
+raw body records, translate every body, construct a live `Tr2EffectRes`, or
+prove prepared pipelines/rendering. `GetPortableEffectReflection` reconstructs
+and validates fresh owned plain portable data; `runtime-resource`
+`Tr2EffectRes` performs canonical runtime-class hydration and selection. The
+accessor does not construct renderer-owned handles, layouts, resource sets, or
+stage programs.
+
+## Static metadata
+
+The class exposes output-mode constants, accepted class keys, media and input
+type metadata, implementation status, the CEWGPU format label, analysis format,
+and package version.
+
+## Errors
+
+Malformed package input and unsafe analysis paths throw or report a
+`CjsWebgpuReadError` internally. Unsupported WGSL semantics fail closed with
+the operation, stage, and source context needed to identify the boundary.
+Duplicate/non-ASCII chunk tags are rejected. A declared
+`tr2-effect-webgpu` package also fails closed on missing or malformed JSON
+chunks, unsupported document versions, or inconsistent INFO/META/ANLS/WGSL
+identity, counts, keys, layouts, selection, and completeness fields. Declared
+PGRF pointers, exact INFO-v3 chunk digests, schemas, counts, variant tuples,
+body references, and the selected index/options are reconciled as part of the
+same gate. Optional
+INFO/RFLX/RBLB reflection units additionally reconcile the exact RFLX digest,
+every PGRF body and representative, portable closed schemas, exact blob
+references/digests, and the selected body's ANLS pass/stage source identities.
+Strict effect validation is activated by the `INFO.packageKind` marker;
+effect-only consumers must require that marker because unmarked CEWGPU
+containers intentionally remain generic.
+
+## Related documentation
+
+- [Effect packaging guide](../guides/effect-packaging.md)
+- [CEWGPU package format](../formats/cewgpu.md)
+- [Class-purpose catalog](classes/README.md)

package/docs/formats/webgpu/reference/classes/README.md

@@ -0,0 +1,66 @@
+# Class catalog
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu` maintained classes
+Audience: Users, maintainers, and automated readers
+Summary: Provides one-sentence purpose descriptors for every maintained class in the WebGPU format package.
+
+<!-- class:CjsWebgpuFormat -->
+## `CjsWebgpuFormat`
+
+CarbonEngineJS-facing format surface for `.cewgpu` WebGPU packages, plus an offline effect-analysis helper built on `format-hlsl` and `format-dxbc`.
+
+- Export: `@carbonenginejs/runtime-resource/formats/webgpu`
+- Source: `src/CjsWebgpuFormat.js`
+- Visibility: Public
+- Kind: CarbonEngineJS
+
+<!-- class:CjsWebgpuReadError -->
+## `CjsWebgpuReadError`
+
+Error raised when a CEWGPU package or WebGPU analysis pass cannot be completed safely.
+
+- Export: None
+- Source: `src/core/errors.js`
+- Visibility: Internal
+- Kind: Internal implementation
+
+<!-- class:CjsBinaryReader -->
+## `CjsBinaryReader`
+
+Minimal little-endian binary reader for the flat CEWGPU chunk container.
+
+- Export: None
+- Source: `src/core/cewgpu/binary.js`
+- Visibility: Internal
+- Kind: Internal implementation
+
+<!-- class:CewgpuPackage -->
+## `CewgpuPackage`
+
+Reader for CarbonEngineJS CEWGPU shader packages.
+
+- Export: None
+- Source: `src/core/cewgpu/CewgpuPackage.js`
+- Visibility: Internal
+- Kind: Internal implementation
+
+<!-- class:CewgpuPackageBuilder -->
+## `CewgpuPackageBuilder`
+
+Builds CarbonEngineJS CEWGPU package bytes.
+
+- Export: None
+- Source: `src/core/cewgpu/CewgpuPackageBuilder.js`
+- Visibility: Internal
+- Kind: Internal implementation
+
+<!-- class:ReflectionBlobStore -->
+## `ReflectionBlobStore`
+
+Builds the deterministic deduplicated byte arena for complete source reflection.
+
+- Export: None
+- Source: `src/core/effectReflectionPackage.js`
+- Visibility: Internal
+- Kind: Internal implementation