@carbonenginejs/runtime-resource 0.11.1 → 0.12.1

package/docs/formats/webgpu/README.md

@@ -0,0 +1,84 @@
+# WebGPU format documentation
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu`
+Audience: Shader-tool authors, engine integrators, and maintainers
+Summary: Explains the Carbon-record CEWGPU format, compiled-effect conversion API, and bounded DXBC-to-WGSL compiler.
+
+## Purpose
+
+`@carbonenginejs/runtime-resource/formats/webgpu` reads and builds WebGPU
+effect containers and converts supported compiled-effect stages into WGSL. It
+owns effect analysis, DXBC-to-intermediate-representation lowering, WGSL
+emission, pass-global binding planning, and the WebGPU backend block carried by
+Carbon version-15 records.
+
+Unsupported requested shader semantics fail explicitly instead of producing a
+partially translated selected pass. `BuildEffect` preserves every permutation
+row and representable non-program description/reflection fields in the Carbon
+container. Non-dynamic sampler names are unrecoverable and stage order is
+canonicalized. Source-stage DXBC is replaced by WGSL or an empty program slot.
+Selected mode writes WGSL only for the resolved body's requested passes; all
+mode attempts every distinct body after the resolved body passes the initial
+translation gate. Backend and runtime completeness remain broader gates.
+
+## Use this package when
+
+Use the WebGPU format subpath when you need to:
+
+- inspect or build a `.cewgpu` package;
+- analyze caller-supplied compiled effect bytes;
+- lower supported DXBC vertex and fragment programs to WGSL;
+- build one collision-free WebGPU binding layout across a complete pass; or
+- translate selected or all distinct version-15 effect bodies while preserving
+  permutation topology and representable non-program reflection fields.
+
+Use `@carbonenginejs/runtime-resource/formats/hlsl` directly for effect metadata without WGSL
+conversion, and `@carbonenginejs/runtime-resource/formats/dxbc` directly for standalone DXBC
+inspection. GPU device, shader-module, bind-group, and pipeline realization
+belong in `@carbonenginejs/engine-webgpu`.
+
+## Where it fits
+
+```text
+compiled effect bytes
+        |
+        +---- formats/hlsl ---- effect and binding metadata
+        |
+        +---- formats/dxbc ---- decoded shader programs
+        |                              |
+        +------------------------------+
+                       |
+                       v
+               formats/webgpu
+       analysis + WGSL + Carbon records
+                       |
+                       v
+                 engine-webgpu
+```
+
+The package is browser-safe at its public source boundary. Repository-only
+commands may adapt filesystem input for development, but the core conversion
+path accepts bytes and does not depend on Node filesystem APIs or native
+executables.
+
+## Start here
+
+```js
+import { CjsWebgpuFormat } from "@carbonenginejs/runtime-resource/formats/webgpu";
+
+const summary = CjsWebgpuFormat.inspect(packageBytes);
+const packageData = CjsWebgpuFormat.read(packageBytes);
+```
+
+For compiled-effect conversion, continue with the
+[effect packaging guide](guides/effect-packaging.md).
+
+## Documentation map
+
+- [Architecture and boundaries](architecture.md)
+- [Effect packaging guide](guides/effect-packaging.md)
+- [Public API reference](reference/api.md)
+- [CEWGPU package format](formats/cewgpu.md)
+- [WGSL compatibility](reference/wgsl-compatibility.md)
+- [Class-purpose catalog](reference/classes/README.md)

package/docs/formats/webgpu/architecture.md

@@ -0,0 +1,96 @@
+# Architecture and boundaries
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu`
+Audience: Shader-tool authors, engine integrators, and maintainers
+Summary: Defines the package's compiler, container, dependency, and engine boundaries.
+
+## Purpose
+
+The WebGPU format subpath is the transformation and package layer between
+compiled Carbon effects and WebGPU engine realization. It converts supported
+shader programs into WGSL and writes Carbon version-15 effect records without
+creating live GPU objects.
+
+## Dependency direction
+
+```text
+@carbonenginejs/runtime-resource/formats/hlsl
+          |
+          +---- effect selection and binding metadata
+          |
+          v
+@carbonenginejs/runtime-resource/formats/webgpu <---- @carbonenginejs/runtime-resource/formats/dxbc
+          |                              decoded DXBC
+          |
+          +---- CEWGPU bytes, analysis, WGSL, canonical layouts
+          |
+          v
+@carbonenginejs/engine-webgpu
+```
+
+Node build tools may call the public byte-oriented API, but the format package
+does not import the toolchain. Dependency direction remains tools to formats,
+then package data to the engine.
+
+## Owned responsibilities
+
+- Reading, inspecting, and building Carbon-record CEWGPU containers.
+- Resolving one compiled-effect permutation and complete pass.
+- Preserving every source permutation row and representable non-program
+  description field; non-dynamic sampler names are unrecoverable and stage
+  order is canonicalized.
+  Stored-body identity is exact emitted-description-byte identity; it does not
+  promise to retain the source alias partition after programs are replaced.
+- Writing WGSL to program slots and bind-group/transform data to the optional
+  per-pass backend block.
+- Normalizing effect, stage, binding, and decoded DXBC analysis.
+- Building validated shader intermediate representation and structured control
+  flow.
+- Lowering the supported vertex and fragment profiles to WGSL.
+- Allocating a pass-global binding plan with explicit stage scope.
+- Assembling emitted shaders and layouts into a portable WGSL set.
+- Rejecting unsupported or ambiguous semantics with explicit diagnostics.
+
+## Ownership elsewhere
+
+- `@carbonenginejs/runtime-resource/formats/hlsl` owns compiled-effect parsing, permutation resolution,
+  unique-body enumeration, binding-manifest interpretation, and the shared
+  body-local portable reflection schema/validator.
+- `@carbonenginejs/runtime-resource/formats/dxbc` owns DXBC container and instruction decoding.
+- `@carbonenginejs/engine-webgpu` owns `GPUDevice`, shader-module compilation, bind groups,
+  pipelines, resource realization, device loss, and draw execution.
+- Node tooling owns indexed input acquisition, filesystem adapters, caching,
+  build reports, and optional native comparison.
+- `runtime-resource` owns effect-resource lifecycle, permutation selection,
+  canonical device-free `Tr2Shader` hydration, and shader caching.
+- `runtime-trinity` owns authored effect/material facades, parameters, options,
+  sampler overrides, and scene references.
+
+## Browser and Node boundary
+
+The public class accepts `ArrayBuffer`, typed-array, and compatible byte views.
+Its source does not read files, inspect processes, or invoke native programs.
+This allows applications to fetch or select bytes and build CEWGPU data in a
+browser.
+
+The repository's command-line scripts are development adapters over the same
+public operations. They are not a second compiler contract.
+
+## Shader target
+
+The general translation target is DX11 SM5.0 vertex and fragment bytecode.
+The compiler also admits a finite, frozen set of exact bounded compute
+profiles. DX12 SM5.1 input is useful for differential analysis where
+supported, but DX12-only bindless resource ranges are not part of the current
+translation target.
+
+Geometry, hull, and domain stages, plus compute programs outside the admitted
+profiles, are not emitted. The exact supported and adapted boundaries are
+listed in [WGSL compatibility](reference/wgsl-compatibility.md).
+
+## Related documentation
+
+- [Package documentation](README.md)
+- [CEWGPU package format](formats/cewgpu.md)
+- [Public API reference](reference/api.md)

package/docs/formats/webgpu/formats/cewgpu.md

@@ -0,0 +1,216 @@
+# CEWGPU effect container
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu`
+Audience: Shader-tool authors and engine integrators
+Summary: Defines the Carbon v15 record container used for WebGPU effects, its backend block, and its derived compatibility views.
+
+## Purpose
+
+A `.cewgpu` file is a stock Carbon version-15 compiled-effect container whose
+program slots carry WGSL instead of DXBC. It preserves every permutation row
+and representable non-program description/reflection fields. Non-dynamic
+sampler names are unrecoverable and stage order is canonicalized. Source-stage
+programs are not stored: a translated slot contains WGSL and an untranslated slot is empty.
+Each translated pass may also carry one WebGPU backend block containing
+bind-group layouts and resource transforms.
+
+There is no CEWGPU-specific magic, envelope, payload tag, or container version.
+Backend identity comes from the resource path, such as `effect.webgpu/`, just
+as Carbon selects `effect.dx11/`, `effect.dx12/`, or `effect.metal/`.
+`isCewgpu(bytes)` is therefore only a Carbon-v15 shape check. It cannot prove
+that arbitrary version-15 bytes contain WGSL.
+
+## Wire layout
+
+The shared Carbon container contract is documented in
+[Carbon compiled-effect container](../../carbon-effect-container.md). In
+outline, the file contains:
+
+- Carbon's version-15 header, compiler version, source-hash slot (zero-filled
+  by the current builder), string table, and permutation axes;
+- one dense offset-table row for every permutation;
+- one stored description tree for every distinct emitted body, with exact
+  emitted-description-byte aliases sharing that body; and
+- one optional backend block after each pass's render states.
+
+The Carbon region is backend-invariant. CEWGPU substitutes:
+
+- UTF-8 WGSL in each translated stage's `shaderData`;
+- the fixed entry point `main`, which is omitted from the wire because every
+  current lowerer must emit it; and
+- a versioned backend block for bind-group layouts and resource transforms
+  that Carbon reflection cannot derive.
+
+A stage with zero program bytes is reflection-only. This is how selected-mode
+packages retain untranslated bodies and how unsupported Carbon stage types
+remain represented without pretending WebGPU can execute them.
+
+## No stored chunks
+
+The former flat CEWGPU format stored `INFO`, `META`, `PGRF`, `RFLX`, `RBLB`,
+`ANLS`, `WGSL`, and `WGSB` chunks. The current wire stores none of them.
+
+Equivalent read surfaces are derived from the one Carbon record tree:
+
+| Former document | Current source of truth |
+| --- | --- |
+| `INFO` | Carbon version, compiler version, zero-filled rebuilt source hash, and counted records |
+| `META` | The resolved permutation and the passes/stages that carry programs |
+| `PGRF` | Carbon's permutation axes, dense offset table, and emitted-body aliases |
+| `RFLX` / `RBLB` | Representable non-program description fields and exact arena-backed values |
+| `ANLS` | A normalized analysis view rebuilt from one description |
+| `WGSL` | Program text and layouts read from stage records and backend blocks |
+| `WGSB` | A body-set view derived across distinct stored bodies |
+
+`Read(..., { emit: "json" })` returns these compatibility views as plain data.
+They are not independent stored documents and carry no cross-document digests.
+`Read(..., { emit: "raw" })` returns the internal `CewgpuContainer` reader.
+There is no `chunks` array and no generic `Build(chunks)` API.
+
+## Building
+
+`BuildEffect` accepts version-15 compiled-effect bytes only. It parses the
+complete input, resolves the requested permutation, lowers selected programs,
+and writes a new Carbon v15 container. Full portable source reflection exists
+in the in-memory build result; source-stage program bytes and the caller's
+source hash are not retained in the emitted wire.
+
+The returned build record is richer than the bytes. Its `info`, `metadata`,
+`permutationGraph`, `reflection`, `analysis`, `wgsl`, `backendBodySet`, and
+`qualification` fields are build-time evidence for callers. They must not be
+interpreted as separate records stored in the container.
+
+### Selected mode
+
+`mode: "selected"` is the default. The container still carries every
+permutation row and representable non-program description fields, but only the
+resolved body's requested complete passes carry translated WGSL. Other bodies
+have zero-length program slots.
+
+Selected mode narrows backend translation without reducing permutation
+topology.
+
+### All mode
+
+`mode: "all"` first lowers the resolved selection through the same initial gate
+as selected mode; an unsupported resolved body therefore aborts the build.
+After that precondition succeeds, the builder attempts every distinct body. A
+later body that cannot be lowered remains in the container with its non-program
+description fields and zero-length programs. The in-memory body-set view
+records its specific failure; rereading the wire can report only that the body
+carries no translated programs.
+
+Translation units are pass-scoped because binding plans and resource
+transforms are pass contracts. Arena deduplication shares identical emitted
+program and backend blobs; the wire does not contain a separate unit table.
+
+## Reading and validation
+
+The shared Carbon reader validates version, count caps, dense positional
+offsets, arena bounds, and exact record ends. The WebGPU layer then checks each
+distinct body and rejects any program-bearing stage outside vertex, pixel, and
+compute. Geometry, hull, and domain reflection may remain only when the
+corresponding program slot is empty.
+
+Inspection reports:
+
+- Carbon version and compiler-version bytes;
+- permutation and distinct-body counts; and
+- stage, shader, and layout counts for the resolved translation.
+
+The JSON read shape contains `info`, `metadata`, `permutationGraph`,
+`analysis`, `wgsl`, `backendBodySet`, and convenience `stages`, `shaders`, and
+`layouts` arrays. `analysis`, `wgsl`, and `backendBodySet` are derived views.
+
+## Backend block
+
+Each translated pass may reference one version-1 backend block from the Carbon
+arena. The block carries:
+
+- physical bind-group and binding slots;
+- resource kind, stage visibility, register identity, generated symbol, and
+  optional structured-buffer or array-layer metadata; and
+- resource-transform family, identifier, and ordered source inputs.
+
+Strings inside the block are inline and length-prefixed. The block contains no
+arena offsets, so its bytes remain independent of the arena's content sort and
+can deduplicate safely. An unknown backend-block version is skipped instead of
+being guessed.
+
+Fields such as `identity`, `layoutKey`, transform output, and fixed layer
+numbers are reconstructed from the block plus record position. Backend-block
+version 1 stores visibility but not the original `scopeIdentity`; the reader
+reconstructs `${identity}@${visibility[0]}`. A multi-stage shared binding
+therefore rereads as stage-qualified rather than recovering its original bare
+scope. Callers must not use the wire view to infer that original sharing
+decision.
+
+## Structured WGSL set
+
+The derived `CJS_WGSL_SET` version-2 view contains shader descriptors and
+pass-level layouts. A layout records the numeric bind group and binding slots
+already present in the WGSL source.
+
+Each binding keeps:
+
+- a D3D-derived base `identity`;
+- a resource-resolution `scopeIdentity`;
+- stage visibility;
+- the buffer, texture, or sampler layout; and
+- its numeric group and binding.
+
+Resource tuples are stage-scoped unless the caller explicitly proves one
+compatible shared identity. The builder rejects duplicate scopes, duplicate
+numeric slots, mixed shared and stage-scoped forms, incomplete visibility,
+and stage/layout conflicts. It never renumbers slots during WGSL-set assembly.
+
+Version-1 binding plans remain accepted as legacy input. Ordinary new plans
+and WGSL sets use version 2.
+
+### Version 3 resource transforms
+
+A derived set becomes version 3 when the compiler proves that several logical
+source resources can be represented by one physical WebGPU resource. Its
+`resourceTransforms` array describes the realization recipe; the matching
+physical layout binding carries `transformId` and `arrayLayerCount`.
+
+The currently supported version-1 recipe is `kind: "texture-2d-array"`.
+Inputs are ordered by fixed array layer. The output reuses layer zero's D3D
+identity, and later logical inputs do not remain as physical bindings.
+
+`representation: "native-or-rgba8"` requires the consumer to assemble one
+compatible `texture_2d_array` from the named inputs, either in a shared native
+format or after decoding every layer to RGBA8. Dimensions, mip coverage,
+sample type, and texture format must agree. `missingLayer: "reject"` forbids a
+fallback layer.
+
+The set builder fails closed unless every recipe:
+
+- targets an emitted fragment stage in its own pass;
+- links exactly one `texture_2d_array<f32>` physical binding;
+- numbers distinct inputs contiguously from layer zero;
+- matches the binding's identity, view dimension, and layer count; and
+- removes only the later input scopes from that recipe's owning pass.
+
+`engine-webgpu` accepts WGSL-set versions 1, 2, and 3 and realizes the
+version-1 `texture-2d-array` recipe. Unsupported recipe kinds or versions fail
+closed.
+
+## Compatibility boundary
+
+The JSON compatibility views exist so current consumers can cross the
+container switchover without a second artifact. New code should treat the
+Carbon record tree as the wire authority and should not rebuild assumptions
+around the retired chunk names.
+
+The raw container's current reflection-to-`Tr2Shader` adapter and the engine's
+body-program view remain integration boundaries under active repair. Their
+existence does not change the wire contract described here.
+
+## Related documentation
+
+- [Carbon compiled-effect container](../../carbon-effect-container.md)
+- [Effect packaging guide](../guides/effect-packaging.md)
+- [Public API reference](../reference/api.md)
+- [WGSL compatibility](../reference/wgsl-compatibility.md)

package/docs/formats/webgpu/guides/effect-packaging.md

@@ -0,0 +1,191 @@
+# Build a CEWGPU package from compiled effect bytes
+
+Status: Evolving
+Scope: `@carbonenginejs/runtime-resource/formats/webgpu`
+Audience: Shader-tool authors and engine integrators
+Summary: Shows how to translate version-15 compiled effects into Carbon-record CEWGPU bytes.
+
+## Purpose
+
+Use `buildEffect` when a caller already has version-15 compiled-effect bytes
+and needs a WebGPU-targeted effect container. The operation parses the complete
+input, resolves a permutation, lowers supported DXBC, allocates pass-global
+bindings, emits WGSL, writes a Carbon v15 container, and validates the result.
+
+The method is byte-oriented and does not open files or resource paths.
+
+## Build selected passes
+
+```js
+import { CjsWebgpuFormat } from "@carbonenginejs/runtime-resource/formats/webgpu";
+
+const result = CjsWebgpuFormat.buildEffect(effectBytes, {
+    source: "res:/graphics/effect.dx11/example.sm_hi",
+    mode: "selected",
+    permutation: [
+        { name: "QUALITY", value: "HIGH" }
+    ],
+    selection: {
+        techniqueName: "Main",
+        passIndex: 0,
+        stageNames: [ "vertex", "pixel" ]
+    }
+});
+
+const packageBytes = result.bytes;
+const emittedShaders = result.wgsl;
+```
+
+`source` is a caller-owned diagnostic label. An optional `sourceIdentity`
+records build provenance in the returned result; it does not change the wire
+identity, which belongs to the resource path used to load the emitted bytes.
+When `sourceIdentity.sha256` is supplied, the build checks it against the exact
+input bytes.
+
+## What the bytes contain
+
+The emitted bytes are a stock Carbon v15 effect container:
+
+- every source permutation remains in the dense offset table;
+- each distinct emitted description body is stored once, based on exact emitted
+  bytes rather than the source alias partition;
+- representable non-program description/reflection fields remain in the Carbon
+  tree; non-dynamic sampler names are unrecoverable and stage order is
+  canonicalized;
+- translated stage program slots contain UTF-8 WGSL;
+- untranslated or unsupported stage program slots have zero length; and
+- translated passes may carry a WebGPU backend block with bind-group layouts
+  and resource transforms.
+
+Source-stage DXBC and the original source hash are not stored in CEWGPU bytes.
+Full portable source reflection remains available only in the in-memory
+`BuildEffect` result.
+
+There are no stored `INFO`, `META`, `PGRF`, `RFLX`, `ANLS`, `WGSL`, or `WGSB`
+chunks. The read API derives compatible JSON views from the Carbon records.
+
+## Translation modes
+
+### Selected
+
+`mode: "selected"` is the default. It translates the resolved body's requested
+complete passes. Every permutation row and representable non-program
+description fields remain in the container, but untranslated program slots are
+empty.
+
+Selected mode does not discard permutations. It narrows backend translation.
+
+### All
+
+`mode: "all"` first lowers the resolved selection; an unsupported resolved body
+aborts the build. Once that precondition succeeds, later bodies that lower
+successfully carry WGSL and backend blocks. A later body outside the compiler's
+current boundary remains present with non-program description fields and empty
+program slots. The in-memory body-set view records its reason; a reread can
+report only that it carries no translated programs.
+
+Passes, rather than individual stages, are the translation unit because a pass
+owns one binding plan and resource-transform plan.
+
+The compatibility option `allPermutations: true` selects all mode.
+`allPermutations: false` selects the requested `mode` or the selected default.
+
+## Build result
+
+The returned record contains build-time evidence in addition to `bytes`:
+
+| Field | Purpose |
+| --- | --- |
+| `bytes` | Carbon v15 CEWGPU bytes. |
+| `info` | Producer, source, translation-scope, and completeness evidence. |
+| `metadata` | Resolved selection and caller provenance. |
+| `permutationGraph` | Complete source permutation and body-alias view. |
+| `reflection` | Complete portable source reflection used while building. |
+| `reflectionBlobs` | Exact portable-reflection payload bytes. |
+| `analysis` | Selected-body diagnostic analysis. |
+| `wgsl` | Emitted shaders, layouts, and transforms. |
+| `backendBodySet` | All-body translation result, or `null` in selected mode. |
+| `inspection` | Summary obtained by rereading the emitted bytes. |
+| `qualification` | Structural build outcome and translation counts. |
+
+These fields are returned data. They are not separate documents stored beside
+the Carbon records.
+
+`qualification.packageValid` means the emitted container passed structural
+validation. It is not prepared-pipeline or rendered evidence.
+`backendComplete` and `runtimeComplete` remain false until the broader compiler,
+resource-hydration, selection, and execution gates are satisfied.
+
+## Read the result
+
+```js
+const summary = CjsWebgpuFormat.inspect(packageBytes, {
+    source: "res:/graphics/effect.webgpu/example.sm_hi"
+});
+
+const data = CjsWebgpuFormat.read(packageBytes, {
+    source: "res:/graphics/effect.webgpu/example.sm_hi"
+});
+```
+
+The JSON read derives `info`, `metadata`, `permutationGraph`, `analysis`,
+`wgsl`, and `backendBodySet` views from the one record tree. It also exposes
+convenience `stages`, `shaders`, and `layouts` arrays.
+
+Raw reads return the internal container reader. That surface is useful for
+current package integration but is not a second artifact and should not be
+persisted as a replacement wire format.
+
+## Binding scope
+
+A D3D resource tuple is stage-local unless authoritative metadata proves that
+the vertex and fragment declarations name one compatible resource. Build one
+binding plan from the complete stage set.
+
+```js
+const plan = CjsWebgpuFormat.buildWgslBindingPlan(
+    [ vertexIr, fragmentIr ],
+    { sharedIdentities: [ "uniform-buffer:0:0" ] }
+);
+```
+
+Unshared identities receive separate `@vertex` or `@fragment`
+`scopeIdentity` values and numeric slots. Shared identities retain one bare
+scope with combined visibility.
+
+## Resource transforms
+
+When semantic metadata proves that several logical textures may be represented
+by one physical array texture, the derived WGSL set uses version 3 and carries
+a `texture-2d-array` transform recipe.
+
+The consumer must assemble the named layers, match size/mips/sample
+type/format, and bind the resulting array through the transformed layout.
+Missing layers fail closed. A version-3 document is not executable evidence by
+itself.
+
+## Errors
+
+Conversion fails explicitly when:
+
+- the source is not a version-15 compiled effect;
+- a permutation assertion is unknown or unresolved;
+- the permutation table is sparse, misordered, out of bounds, or malformed;
+- the technique, pass, or requested stage does not exist;
+- the requested stage list is duplicated or incomplete;
+- the selected shader uses unsupported semantics;
+- resource declarations cannot form one unambiguous pass layout;
+- a lowerer emits an entry point other than `main`; or
+- the emitted Carbon records or backend block fail structural validation.
+
+Unsupported selected programs abort selected-mode packaging and also abort the
+initial selection gate in all mode. After that gate succeeds, an unsupported
+later body remains represented with empty program slots and an explicit
+in-memory derived status.
+
+## Related documentation
+
+- [CEWGPU effect container](../formats/cewgpu.md)
+- [Public API reference](../reference/api.md)
+- [WGSL compatibility](../reference/wgsl-compatibility.md)
+- [Carbon compiled-effect container](../../carbon-effect-container.md)