@carbonenginejs/runtime-resource 0.18.0 → 0.19.0

package/docs/formats/static.md

@@ -0,0 +1,166 @@
+# Client `.static` container identification
+
+Status: Experimental
+Scope: `@carbonenginejs/runtime-resource/formats/static`
+Audience: Resource integrators reading client static data
+Summary: Identifies which of three unrelated containers a `.static` file holds, so a caller can route it to the format that decodes it.
+
+## Why this exists
+
+`.static` names a role, not a format. Three unrelated containers ship under the
+single extension, and each fails differently when guessed at. Measured across
+the 45 `.static` files in one build:
+
+| Family | Count | Signature |
+|---|---:|---|
+| SQLite 3 | 14 | `SQLite format 3\0` |
+| Embedded schema | 25 | four-byte schema LENGTH, then `(d` or `(l` |
+| Sibling schema | 6 | no signature; has a `.schema` companion |
+
+The six unidentified files are exactly the six with a `.schema` companion —
+`constellations`, `dialogs`, `factionsowningsolarsystems`, `jumps`, `regions`
+and `systems` — so the detector's "unknown" set is not a gap in coverage but the
+family that cannot be read without its companion.
+
+## Boundary
+
+`CjsStaticFormat` identifies. It decodes nothing.
+
+- **SQLite** containers hold `cache(key, value, time)` and
+  `indexes(key, value)`, with a JSON document per record.
+- **Embedded-schema** containers put a schema, not a record, behind that prefix.
+  The prefix is the schema's LENGTH: read `[4, 4 + length)` with
+  `CjsPickleFormat` and hand the rest to `CjsSchemaBoundFormat`.
+- **Sibling-schema** containers report `unknown` with `requires: "schema"`, and
+  are decoded by `CjsSchemaBoundFormat` once the caller has that companion.
+
+Detection is signature-based. It never trusts a file name and never executes
+anything.
+
+## This format identifies; it does not decode
+
+It reports the family, where the payload starts, and what is still missing. The
+caller takes that to the format that owns the family:
+
+```js
+const probe = await CjsStaticFormat.resolveType(bytes);
+
+if (probe.preferred === CJS_STATIC_FAMILIES.SQLITE)
+{
+  return CjsSqliteFormat.readJSON(bytes);
+}
+
+if (probe.preferred === CJS_STATIC_FAMILIES.PICKLE)
+{
+  // The prefix is the SCHEMA's length, not a wrapper to skip. Handing the whole
+  // remainder to a pickle reader throws CJS_PICKLE_FORMAT_TRAILING_DATA, on the binary
+  // payload, long after the schema has parsed.
+  const length = new DataView(bytes.buffer, bytes.byteOffset).getUint32(0, true);
+
+  return CjsSchemaBoundFormat.read(bytes.subarray(4 + length), {
+    schema: CjsPickleFormat.read(bytes.subarray(4, 4 + length))
+  });
+}
+```
+
+**This changed on 2026-08-15.** A `read()` here dispatched to those two formats
+itself, and the SQLite family additionally required a driver injected through
+`options.sqlite`. Both are gone. An identification format should not be the
+routing table for two others, and deciding what to decode belongs to whoever
+asked. Nothing outside this format's own tests ever called `read()`.
+
+
+## Both of the remaining families are the same container
+
+The schema is encoded differently; the payload behind it is identical.
+
+**The pickle behind that prefix is a SCHEMA, not a record.** Protocol 0's `c`
+(`GLOBAL`) names a module and an attribute for the unpickler to import, and
+`R` then calls it — the remote-execution vector — so `CjsPickleFormat` refuses
+globals by design. These files need exactly one: `collections.OrderedDict`,
+because a schema's attribute order is its field order. That one name is rebuilt
+as a plain object and every other global is still refused. See
+[the pickle format](pickle.md).
+
+**The sibling-schema family states its layout in YAML.** The `.schema` file is
+YAML and states the whole binary layout — sizes, types, optional flags, list item
+sizes, vector precision and a key-to-offset footer — so **nothing needs
+deriving**, unlike a container whose layout is defined outside the file and has
+to be worked out and pinned. `CjsSchemaBoundFormat` reads it:
+[schema-bound containers](schemabound.md). All six datasets decode — the map
+skeleton of regions, constellations and systems. The celestial detail (moons,
+planets, belts, stars, gates) is in the embedded-schema family, not this one.
+
+## Use
+
+Identification goes through the shared type-resolution seam rather than a
+private entry point — see [format type resolution](../concepts/format-type-resolution.md).
+
+```js
+import { CjsStaticFormat, CJS_STATIC_FAMILIES } from
+    "@carbonenginejs/runtime-resource/formats/static";
+
+const probe = await CjsStaticFormat.resolveType(bytes);
+
+if (probe.preferred === CJS_STATIC_FAMILIES.PICKLE)
+{
+    // See the routing example above: the prefix is a schema length.
+}
+```
+
+`isSupported()` and its `inspect()` alias report on the declaration seam;
+`resolveType()` is the content-verified one. This format is an unusual case for
+that contract: `.static` carries no in-band declaration at all, so there is
+nothing for the content to disagree with. The signature is both claim and
+evidence, `resolveType()` is therefore always `verified`, and `metadata.declared`
+is `null` with `mismatch` always false.
+
+`describe()` returns the underlying
+`{ family, byteLength, payloadOffset, prefix, decodable, requires, reason }` without
+building a probe. `payload()` returns the bytes past any wrapper, which is what a
+caller hands to the format that owns the family.
+
+## Reading a container, rather than identifying one
+
+Identifying a family and then routing it to the format that owns it is the same
+twenty lines in every caller, so they are written once here and exported from the
+same subpath:
+
+```js
+import {
+    ReadStaticContainer,          // SQLite family
+    ReadEmbeddedSchemaContainer,  // schema length, pickled schema, payload
+    ReadSchemaBoundContainer,     // payload plus its .schema sibling
+} from "@carbonenginejs/runtime-resource/formats/static";
+
+const skins = await ReadStaticContainer(bytes, "res:/staticdata/skins.static");
+```
+
+The `path` argument only ever names the file in an error.
+
+These import the pickle, schema-bound and SQLite formats, because reading a
+`.static` genuinely needs them - a container format that wraps other containers
+has to reach the formats it wraps. What a format must not do is pull the rest of
+the library in behind it, and nothing here reaches outside `formats/`.
+
+`CjsStaticFormat` itself still imports nothing and decodes nothing. Routing is a
+sibling module so that identification is not also the routing table, which is the
+arrangement `read()` was cut back to on 2026-08-15.
+
+### Errors
+
+| Code | When |
+| --- | --- |
+| `CJS_STATIC_FORMAT_FAMILY_UNSUPPORTED` | the bytes are a `.static` of a family this reader does not read; carries `family` |
+| `CJS_STATIC_FORMAT_SHAPE_INVALID` | right family, wrong container - no `cache` table, or a schema length running past the end |
+| `CJS_STATIC_FORMAT_RECORD_INVALID` | one stored value is not JSON; carries `key` |
+
+The family check is kept even on `ReadSchemaBoundContainer`, where the caller has
+already supplied a schema: those bytes carry no signature at all, so given the
+wrong schema they decode into plausible nonsense rather than failing.
+
+## Related documentation
+
+- [Formats](README.md)
+- [Data-only pickle protocol 0](pickle.md)
+- [Schema-bound containers](schemabound.md)

package/docs/formats/webgl/README.md

@@ -39,9 +39,11 @@ compiled effect bytes
 The emitted artifact is a shared Carbon v15 container, not a tagged-chunk
 package; the GLSL replaces the DXBC in each stage's program slot. Backend body
 keys deliberately retain the historical `body_<offset>_<size>` form.
-`EFFECT_PERMUTATION_GRAPH_CHUNK` (`PGRF`) still records every Cartesian
-permutation, its option indices, its exact source record, and a
-content-deduplicated body identity.
+The permutation graph still records every Cartesian permutation, its option
+indices, its exact source record, and a content-deduplicated body identity — but
+it is a **derived view** (`CJS_EFFECT_PERMUTATION_GRAPH`, built by
+`buildEffectPermutationGraph`), not a stored `PGRF` chunk. That chunk and its
+`EFFECT_PERMUTATION_GRAPH_CHUNK` export are retired; nothing emits or reads one.
 
 ## Completeness
 

package/docs/formats/webgpu/README.md

@@ -16,8 +16,8 @@ Carbon version-15 records.
 Unsupported requested shader semantics fail explicitly instead of producing a
 partially translated selected pass. `BuildEffect` preserves every permutation
 row and representable non-program description/reflection fields in the Carbon
-container. Non-dynamic sampler names are unrecoverable and stage order is
-canonicalized. Source-stage DXBC is replaced by WGSL or an empty program slot.
+container, including non-dynamic sampler names and the authored stage order.
+Source-stage DXBC is replaced by WGSL or an empty program slot.
 Selected mode writes WGSL only for the resolved body's requested passes; all
 mode attempts every distinct body after the resolved body passes the initial
 translation gate. Backend and runtime completeness remain broader gates.

package/docs/formats/webgpu/architecture.md

@@ -38,8 +38,8 @@ then package data to the engine.
 - Reading, inspecting, and building Carbon-record Carbon WebGPU containers.
 - Resolving one compiled-effect permutation and complete pass.
 - Preserving every source permutation row and representable non-program
-  description field; non-dynamic sampler names are unrecoverable and stage
-  order is canonicalized.
+  description field, including non-dynamic sampler names and the authored stage
+  order — both are preserved, whatever Carbon's runtime does with them.
   Stored-body identity is exact emitted-description-byte identity; it does not
   promise to retain the source alias partition after programs are replaced.
 - Writing WGSL to program slots and bind-group/transform data to the optional

package/docs/formats/webgpu/formats/carbon-webgpu.md

@@ -9,12 +9,23 @@ Summary: Defines the Carbon v15 record container used for WebGPU effects, its ba
 
 A `.carbonwebgpu` file is a stock Carbon version-15 compiled-effect container whose
 program slots carry WGSL instead of DXBC. It preserves every permutation row
-and representable non-program description/reflection fields. Non-dynamic
-sampler names are unrecoverable and stage order is canonicalized. Source-stage
+and representable non-program description/reflection fields, including
+non-dynamic sampler names and the authored stage order. Source-stage
 programs are not stored: a translated slot contains WGSL and an untranslated slot is empty.
 Each translated pass may also carry one WebGPU backend block containing
 bind-group layouts and resource transforms.
 
+That preservation is **inherited from the shared codec, and now pinned on this
+path.** It holds because the emit round-trips the description through the same
+Carbon classes the plain container uses and mutates only `stage.sourceProgram`
+and `pass.backendBlock`.
+
+`carbon-webgpu-emit-fidelity.test.mjs` compares both fields source-to-emitted
+across the corpus rather than trusting that restraint: 2,960 emitted files,
+39,252 sampler names and 84,912 stage orders, of which 102 are non-canonical —
+the case that distinguishes a preserved order from a re-derived one. Widening
+what the emit touches now fails there instead of surfacing in a consumer later.
+
 There is no Carbon WebGPU-specific magic, envelope, payload tag, or container version.
 Backend identity comes from the resource path, such as `effect.webgpu/`, just
 as Carbon selects `effect.dx11/`, `effect.dx12/`, or `effect.metal/`.

package/docs/formats/webgpu/guides/effect-packaging.md

@@ -50,8 +50,7 @@ The emitted bytes are a stock Carbon v15 effect container:
 - each distinct emitted description body is stored once, based on exact emitted
   bytes rather than the source alias partition;
 - representable non-program description/reflection fields remain in the Carbon
-  tree; non-dynamic sampler names are unrecoverable and stage order is
-  canonicalized;
+  tree, non-dynamic sampler names and the authored stage order included;
 - translated stage program slots contain UTF-8 WGSL;
 - untranslated or unsupported stage program slots have zero length; and
 - translated passes may carry a WebGPU backend block with bind-group layouts

package/docs/formats/webgpu/reference/wgsl-compatibility.md

@@ -1403,8 +1403,8 @@ program slots and coverage degrades to `partial`. The in-memory build result
 retains its specific reason; the wire does not. A reread can say only that the
 body carries no translated programs. Permutation topology and representable
 non-program description fields remain present, but source-stage programs are
-not stored. Non-dynamic sampler names are unrecoverable and stage order is
-canonicalized.
+not stored. Non-dynamic sampler names and the authored stage order are both
+preserved.
 
 Translating every body is deliberately **not** treated as backend completeness.
 `backendComplete` and `runtimeComplete` remain false, matching the sibling

package/docs/reference/classes/formats.md

@@ -689,6 +689,39 @@ Data-only Python pickle format facade that currently decodes protocol 0 into JSO
 - Visibility: Public
 - Kind: Original CarbonEngineJS class
 
+<!-- class:CjsSchemaBoundFormat -->
+## `CjsSchemaBoundFormat`
+
+Reads a binary record container against the separate schema document that describes its layout.
+
+- Export: `@carbonenginejs/runtime-resource/formats/schemabound`
+- Source: `src/formats/schemabound/CjsSchemaBoundFormat.js`
+- Visibility: Public
+- Kind: Original CarbonEngineJS class
+- Notes: The schema is required and there is no default. These bytes carry no signature, so the wrong schema decodes into plausible nonsense rather than failing.
+
+<!-- class:CjsSqliteFormat -->
+## `CjsSqliteFormat`
+
+Reads a SQLite 3 container as data: the tables it holds and every row of them, with no SQL and no query engine.
+
+- Export: `@carbonenginejs/runtime-resource/formats/sqlite`
+- Source: `src/formats/sqlite/CjsSqliteFormat.js`
+- Visibility: Public
+- Kind: Original CarbonEngineJS class
+- Notes: Read-only and full-scan. Index b-trees are never walked, and nothing here needs a driver or a filesystem.
+
+<!-- class:CjsStaticFormat -->
+## `CjsStaticFormat`
+
+Identifies which of three unrelated containers a client `.static` file holds, so a caller can route it to the format that decodes it.
+
+- Export: `@carbonenginejs/runtime-resource/formats/static`
+- Source: `src/formats/static/CjsStaticFormat.js`
+- Visibility: Public
+- Kind: Original CarbonEngineJS class
+- Notes: Signature-based, and it decodes nothing at all since 2026-08-15. It reports the family and the payload offset; `CjsPickleFormat`, `CjsSqliteFormat` and `CjsSchemaBoundFormat` decode.
+
 <!-- class:CjsPickleProtocol0Reader -->
 ## `CjsPickleProtocol0Reader`
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@carbonenginejs/runtime-resource",
-  "version": "0.18.0",
+  "version": "0.19.0",
   "description": "CarbonEngineJS resource lifecycle, cache, source, and object loading contracts.",
   "type": "module",
   "exports": {
@@ -39,6 +39,9 @@
     "./formats/png": "./dist/formats/png/index.js",
     "./formats/red": "./dist/formats/red/index.js",
     "./formats/red/schema": "./dist/formats/red/core/blackDefinitions.js",
+    "./formats/schemabound": "./dist/formats/schemabound/index.js",
+    "./formats/sqlite": "./dist/formats/sqlite/index.js",
+    "./formats/static": "./dist/formats/static/index.js",
     "./formats/stl": "./dist/formats/stl/index.js",
     "./formats/tga": "./dist/formats/tga/index.js",
     "./formats/wav": "./dist/formats/wav/index.js",