@carbonenginejs/runtime-resource 0.17.0 → 0.18.1

package/docs/formats/schemabound.md

@@ -0,0 +1,112 @@
+# Schema-bound containers
+
+Status: Experimental
+Scope: `@carbonenginejs/runtime-resource/formats/schemabound`
+Audience: Anyone reading client static data whose layout ships beside it
+Summary: Reads a binary record container against the separate schema document that describes its layout, including the variable-length record section that makes a fixed-stride reader silently wrong.
+
+## Why this exists
+
+Some client containers carry no layout at all. They pair a binary payload with a
+sibling schema document — YAML — that states every attribute's offset, width and
+type, which fields are optional, how lists are strided, and how the container
+indexes its own records.
+
+That makes them the opposite of a hash-identified container, where the header
+names a layout it does not describe and the layout has to be derived and pinned
+per dataset. **Nothing needs deriving here.** Supply the schema and the payload
+decodes.
+
+It also makes them unusually easy to read wrongly. The bytes carry no signature,
+no version and no field names, so given the wrong schema they decode into
+plausible nonsense rather than failing — there is nothing in them to disagree
+with. Pair each payload with the schema that shipped beside it.
+
+## Use
+
+```js
+import { CjsSchemaBoundFormat } from
+    "@carbonenginejs/runtime-resource/formats/schemabound";
+
+const records = CjsSchemaBoundFormat.read(payloadBytes, { schema: schemaBytes });
+```
+
+`schema` accepts the schema document in any form it arrives in: YAML bytes, YAML
+text, or an already-parsed object. YAML is parsed with `CjsYamlFormat`, and
+anchors and aliases are rejoined — these schemas share repeated declarations that
+way, and left unresolved an anchor reads as one more field.
+
+- `read` / `readJSON` — plain JSON-compatible values; a wide integer becomes a
+  decimal string.
+- `readPayload` — the same, with wide integers left as `BigInt`.
+- `is` / `isSupported` — **these ask about the schema, not the payload.** These
+  containers have no signature, so claiming to recognize the bytes would be a
+  claim this format cannot support. Which family a `.static` file belongs to is
+  [`CjsStaticFormat`](static.md)'s question.
+
+The root is whatever the schema declares: a keyed map decodes to an object, a
+list decodes to an array.
+
+## The format
+
+Four properties are worth knowing before writing anything that consumes the
+result. Each of them is a way to get a plausible wrong answer.
+
+**A keyed container's index is at the end, and nothing points at it.** The last
+four bytes of the block are the index's own size; the index starts that far back,
+and begins with its entry count. Reading forward from the header will not find
+it.
+
+**Record offsets are relative to the four-byte length header, not to the file.**
+Decoding from the file start reads one field early, and produces zeroes that look
+like a wrong layout rather than a wrong base.
+
+**The per-record offset table varies in length.** After the fixed attributes come
+a presence bitfield and then one offset per attribute *actually present* — an
+optional attribute whose bit is clear takes no slot. Consecutive records
+therefore hold their variable data at different distances from their own start,
+so a reader written around a constant stride decodes the first record correctly
+and then drifts.
+
+**A list has two framings and the schema says which.** `fixedItemSize` means the
+items are packed at that stride. Its absence means they are variable, so the
+count is followed by one offset per item, measured from the start of the list.
+
+Two smaller rules:
+
+- The presence bitfield is written whenever a record has a variable section at
+  all, including where nothing in it is optional and the field is always zero.
+  It is the offset table that shrinks, never the bitfield.
+- An absent optional takes the default its schema declares. Where no default is
+  declared the attribute is left off the record entirely.
+
+## Types
+
+| Schema type | Decoded as |
+|---|---|
+| `int` | number at the declared width; signed unless `min` is zero or more; `BigInt` beyond the safe range |
+| `float` | number, single or double by declared size |
+| `bool` | boolean |
+| `enum` | the member's name, or its number when `readEnumValue` is set |
+| `vector3` | an object keyed by the schema's own component aliases |
+| `string`, `resPath` | length-prefixed UTF-8 |
+| `list` | array, strided or offset-indexed as above |
+| `dict` | object, framed exactly as the file's own root |
+| `object` | record, as above |
+
+## Evidence
+
+The reader was verified field for field against the published static data export
+for every container of this family in one build, and cross-checked between
+containers that describe the same relationships from different sides. The
+detailed measurements are recorded in the organization documentation rather than
+here.
+
+The tests in this package hold the structure instead, on containers laid out byte
+by byte: the shrinking offset table, both list framings, declared defaults,
+nested maps, anchors in the schema, and the widths ordinary data never reaches.
+
+## Related documentation
+
+- [Client `.static` container identification](static.md) — which family a file holds
+- [Formats](README.md)

package/docs/formats/static.md

@@ -0,0 +1,119 @@
+# Client `.static` container identification
+
+Status: Experimental
+Scope: `@carbonenginejs/runtime-resource/formats/static`
+Audience: Resource integrators reading client static data
+Summary: Identifies which of three unrelated containers a `.static` file holds, so a caller can route it to the format that decodes it.
+
+## Why this exists
+
+`.static` names a role, not a format. Three unrelated containers ship under the
+single extension, and each fails differently when guessed at. Measured across
+the 45 `.static` files in one build:
+
+| Family | Count | Signature |
+|---|---:|---|
+| SQLite 3 | 14 | `SQLite format 3\0` |
+| Prefixed pickle | 25 | four-byte little-endian prefix, then `(d` or `(l` |
+| Schema-bound | 6 | no signature; has a `.schema` companion |
+
+The six unidentified files are exactly the six with a `.schema` companion —
+`constellations`, `dialogs`, `factionsowningsolarsystems`, `jumps`, `regions`
+and `systems` — so the detector's "unknown" set is not a gap in coverage but the
+family that cannot be read without its companion.
+
+## Boundary
+
+`CjsStaticFormat` identifies. It decodes nothing.
+
+- **SQLite** containers hold `cache(key, value, time)` and
+  `indexes(key, value)`, with a JSON document per record.
+- **Prefixed pickle** containers are decoded through `CjsPickleFormat` after
+  the four-byte prefix.
+- **Schema-bound** containers report `unknown` with `requires: "schema"`, and
+  are decoded by `CjsSchemaBoundFormat` once the caller has that companion.
+
+Detection is signature-based. It never trusts a file name and never executes
+anything.
+
+## This format identifies; it does not decode
+
+It reports the family, where the payload starts, and what is still missing. The
+caller takes that to the format that owns the family:
+
+```js
+const probe = await CjsStaticFormat.resolveType(bytes);
+
+if (probe.preferred === CJS_STATIC_FAMILIES.SQLITE)
+{
+  return CjsSqliteFormat.readJSON(bytes);
+}
+
+if (probe.preferred === CJS_STATIC_FAMILIES.PICKLE)
+{
+  return CjsPickleFormat.read(CjsStaticFormat.payload(bytes));
+}
+```
+
+**This changed on 2026-08-15.** A `read()` here dispatched to those two formats
+itself, and the SQLite family additionally required a driver injected through
+`options.sqlite`. Both are gone. An identification format should not be the
+routing table for two others, and deciding what to decode belongs to whoever
+asked. Nothing outside this format's own tests ever called `read()`.
+
+
+## The pickles name classes, and the schemas describe layouts
+
+Two notes that decide how the remaining families get decoded.
+
+**The pickle family carries class-construction opcodes.** Protocol 0's `c`
+(`GLOBAL`) names a module and an attribute for the unpickler to import, and
+`R`/`i`/`o`/`b` then call it. That is the pickle remote-execution vector, so
+`CjsPickleFormat` rejects those opcodes by design. Client `.static` pickles use
+them legitimately, to name the classes their records are constructed from, so
+decoding this family fully means mapping each named global to an inert
+descriptor and never invoking it. **Not implemented**; the rejection is
+surfaced rather than worked around, and widening it is a deliberate decision
+rather than a bug fix.
+
+**The schema-bound family is self-describing.** Its `.schema` companion is
+YAML and states the whole binary layout — sizes, types, optional flags, list item
+sizes, vector precision and a key-to-offset footer — so **nothing needs
+deriving**, unlike an FSD container. `CjsSchemaBoundFormat` reads it:
+[schema-bound containers](schemabound.md). All six datasets decode, the celestial
+tables among them.
+
+## Use
+
+Identification goes through the shared type-resolution seam rather than a
+private entry point — see [format type resolution](../concepts/format-type-resolution.md).
+
+```js
+import { CjsStaticFormat, CJS_STATIC_FAMILIES } from
+    "@carbonenginejs/runtime-resource/formats/static";
+
+const probe = await CjsStaticFormat.resolveType(bytes);
+
+if (probe.preferred === CJS_STATIC_FAMILIES.PICKLE)
+{
+    const value = CjsPickleFormat.read(CjsStaticFormat.payload(bytes));
+}
+```
+
+`isSupported()` and its `inspect()` alias report on the declaration seam;
+`resolveType()` is the content-verified one. This format is an unusual case for
+that contract: `.static` carries no in-band declaration at all, so there is
+nothing for the content to disagree with. The signature is both claim and
+evidence, `resolveType()` is therefore always `verified`, and `metadata.declared`
+is `null` with `mismatch` always false.
+
+`describe()` returns the underlying
+`{ family, byteLength, payloadOffset, prefix, decodable, requires, reason }` without
+building a probe. `payload()` returns the bytes past any wrapper, which is what a
+caller hands to the format that owns the family.
+
+## Related documentation
+
+- [Formats](README.md)
+- [Data-only pickle protocol 0](pickle.md)
+- [Schema-bound containers](schemabound.md)

package/docs/formats/webgl/README.md

@@ -39,9 +39,11 @@ compiled effect bytes
 The emitted artifact is a shared Carbon v15 container, not a tagged-chunk
 package; the GLSL replaces the DXBC in each stage's program slot. Backend body
 keys deliberately retain the historical `body_<offset>_<size>` form.
-`EFFECT_PERMUTATION_GRAPH_CHUNK` (`PGRF`) still records every Cartesian
-permutation, its option indices, its exact source record, and a
-content-deduplicated body identity.
+The permutation graph still records every Cartesian permutation, its option
+indices, its exact source record, and a content-deduplicated body identity — but
+it is a **derived view** (`CJS_EFFECT_PERMUTATION_GRAPH`, built by
+`buildEffectPermutationGraph`), not a stored `PGRF` chunk. That chunk and its
+`EFFECT_PERMUTATION_GRAPH_CHUNK` export are retired; nothing emits or reads one.
 
 ## Completeness
 

package/docs/formats/webgpu/README.md

@@ -16,8 +16,8 @@ Carbon version-15 records.
 Unsupported requested shader semantics fail explicitly instead of producing a
 partially translated selected pass. `BuildEffect` preserves every permutation
 row and representable non-program description/reflection fields in the Carbon
-container. Non-dynamic sampler names are unrecoverable and stage order is
-canonicalized. Source-stage DXBC is replaced by WGSL or an empty program slot.
+container, including non-dynamic sampler names and the authored stage order.
+Source-stage DXBC is replaced by WGSL or an empty program slot.
 Selected mode writes WGSL only for the resolved body's requested passes; all
 mode attempts every distinct body after the resolved body passes the initial
 translation gate. Backend and runtime completeness remain broader gates.

package/docs/formats/webgpu/architecture.md

@@ -38,8 +38,8 @@ then package data to the engine.
 - Reading, inspecting, and building Carbon-record Carbon WebGPU containers.
 - Resolving one compiled-effect permutation and complete pass.
 - Preserving every source permutation row and representable non-program
-  description field; non-dynamic sampler names are unrecoverable and stage
-  order is canonicalized.
+  description field, including non-dynamic sampler names and the authored stage
+  order — both are preserved, whatever Carbon's runtime does with them.
   Stored-body identity is exact emitted-description-byte identity; it does not
   promise to retain the source alias partition after programs are replaced.
 - Writing WGSL to program slots and bind-group/transform data to the optional

package/docs/formats/webgpu/formats/carbon-webgpu.md

@@ -9,12 +9,23 @@ Summary: Defines the Carbon v15 record container used for WebGPU effects, its ba
 
 A `.carbonwebgpu` file is a stock Carbon version-15 compiled-effect container whose
 program slots carry WGSL instead of DXBC. It preserves every permutation row
-and representable non-program description/reflection fields. Non-dynamic
-sampler names are unrecoverable and stage order is canonicalized. Source-stage
+and representable non-program description/reflection fields, including
+non-dynamic sampler names and the authored stage order. Source-stage
 programs are not stored: a translated slot contains WGSL and an untranslated slot is empty.
 Each translated pass may also carry one WebGPU backend block containing
 bind-group layouts and resource transforms.
 
+That preservation is **inherited from the shared codec, and now pinned on this
+path.** It holds because the emit round-trips the description through the same
+Carbon classes the plain container uses and mutates only `stage.sourceProgram`
+and `pass.backendBlock`.
+
+`carbon-webgpu-emit-fidelity.test.mjs` compares both fields source-to-emitted
+across the corpus rather than trusting that restraint: 2,960 emitted files,
+39,252 sampler names and 84,912 stage orders, of which 102 are non-canonical —
+the case that distinguishes a preserved order from a re-derived one. Widening
+what the emit touches now fails there instead of surfacing in a consumer later.
+
 There is no Carbon WebGPU-specific magic, envelope, payload tag, or container version.
 Backend identity comes from the resource path, such as `effect.webgpu/`, just
 as Carbon selects `effect.dx11/`, `effect.dx12/`, or `effect.metal/`.
@@ -63,10 +74,17 @@ Equivalent read surfaces are derived from the one Carbon record tree:
 | `WGSL` | Program text and layouts read from stage records and backend blocks |
 | `WGSB` | A body-set view derived across distinct stored bodies |
 
-`Read(..., { emit: "json" })` returns these compatibility views as plain data.
-They are not independent stored documents and carry no cross-document digests.
-`Read(..., { emit: "raw" })` returns the internal `CarbonWebgpuContainer` reader.
-There is no `chunks` array and no generic `Build(chunks)` API.
+`Read` returns these derived views as plain data. They are not independent stored
+documents and carry no cross-document digests. There is no `chunks` array and no
+generic `Build(chunks)` API.
+
+There is exactly one emit, as there is for WebGL, and the document it returns is
+complete: alongside the views above it carries `permutationGraph` and
+`backendBodySet`, the latter being every translated body joined to its shared
+translation units. A second `raw` emit used to hand back the internal
+`CarbonWebgpuContainer` because the former chunk package could not express the
+body set in JSON. It is removed. The container is internal, and consumers read
+the document rather than binding to a reader object.
 
 ## Building
 

package/docs/formats/webgpu/guides/effect-packaging.md

@@ -50,8 +50,7 @@ The emitted bytes are a stock Carbon v15 effect container:
 - each distinct emitted description body is stored once, based on exact emitted
   bytes rather than the source alias partition;
 - representable non-program description/reflection fields remain in the Carbon
-  tree; non-dynamic sampler names are unrecoverable and stage order is
-  canonicalized;
+  tree, non-dynamic sampler names and the authored stage order included;
 - translated stage program slots contain UTF-8 WGSL;
 - untranslated or unsupported stage program slots have zero length; and
 - translated passes may carry a WebGPU backend block with bind-group layouts

package/docs/formats/webgpu/reference/api.md

@@ -75,7 +75,7 @@ Callers establish identity through the resource path that supplied the bytes.
 
 | Option | Meaning |
 | --- | --- |
-| `emit` | `"json"` by default or `"raw"` for the internal container reader. |
+| `emit` | `"json"`, the only accepted value. Any other value is a `TypeError`. |
 | `source` | Caller-owned diagnostic label; it is never opened. |
 | `decodeInstructions` | Includes decoded instruction and IR detail during analysis. |
 | `permutation` | Exact `NAME=VALUE` assertions for `AnalyzeEffect` and `BuildEffect`; ignored by current `Read` and `Inspect`. |
@@ -85,8 +85,9 @@ Callers establish identity through the resource path that supplied the bytes.
 Class registrations are validated and stored for forward compatibility.
 Current JSON reads return plain data rather than hydrated package classes.
 
-Raw output is an internal reader over the same bytes. It is not a second wire
-format and should not be persisted.
+There is one emit. The document it returns carries every view a consumer needs,
+including `permutationGraph` and the complete `backendBodySet`, so there is no
+second read mode to choose and no reader object to hold.
 
 ## Read result
 
@@ -174,9 +175,9 @@ coalescing uses version 3 and carries an explicit transform recipe.
 
 ## Static metadata
 
-The class exposes `OUTPUT_JSON`, `OUTPUT_RAW`, `CLASS_KEYS`, `type`,
-`mediaTypes`, `inputTypes`, `outputTypes`, `debugOutputTypes`,
-`implementationStatus`, `format`, `analysisFormat`, and `packageVersion`.
+The class exposes `OUTPUT_JSON`, `CLASS_KEYS`, `type`, `mediaTypes`,
+`inputTypes`, `outputTypes`, `implementationStatus`, `format`, `analysisFormat`,
+and `packageVersion`. There is no `OUTPUT_RAW` and no `debugOutputTypes`.
 
 ## Errors
 

package/docs/formats/webgpu/reference/wgsl-compatibility.md

@@ -1403,8 +1403,8 @@ program slots and coverage degrades to `partial`. The in-memory build result
 retains its specific reason; the wire does not. A reread can say only that the
 body carries no translated programs. Permutation topology and representable
 non-program description fields remain present, but source-stage programs are
-not stored. Non-dynamic sampler names are unrecoverable and stage order is
-canonicalized.
+not stored. Non-dynamic sampler names and the authored stage order are both
+preserved.
 
 Translating every body is deliberately **not** treated as backend completeness.
 `backendComplete` and `runtimeComplete` remain false, matching the sibling

package/docs/reference/classes/formats.md

@@ -689,6 +689,39 @@ Data-only Python pickle format facade that currently decodes protocol 0 into JSO
 - Visibility: Public
 - Kind: Original CarbonEngineJS class
 
+<!-- class:CjsSchemaBoundFormat -->
+## `CjsSchemaBoundFormat`
+
+Reads a binary record container against the separate schema document that describes its layout.
+
+- Export: `@carbonenginejs/runtime-resource/formats/schemabound`
+- Source: `src/formats/schemabound/CjsSchemaBoundFormat.js`
+- Visibility: Public
+- Kind: Original CarbonEngineJS class
+- Notes: The schema is required and there is no default. These bytes carry no signature, so the wrong schema decodes into plausible nonsense rather than failing.
+
+<!-- class:CjsSqliteFormat -->
+## `CjsSqliteFormat`
+
+Reads a SQLite 3 container as data: the tables it holds and every row of them, with no SQL and no query engine.
+
+- Export: `@carbonenginejs/runtime-resource/formats/sqlite`
+- Source: `src/formats/sqlite/CjsSqliteFormat.js`
+- Visibility: Public
+- Kind: Original CarbonEngineJS class
+- Notes: Read-only and full-scan. Index b-trees are never walked, and nothing here needs a driver or a filesystem.
+
+<!-- class:CjsStaticFormat -->
+## `CjsStaticFormat`
+
+Identifies which of three unrelated containers a client `.static` file holds, so a caller can route it to the format that decodes it.
+
+- Export: `@carbonenginejs/runtime-resource/formats/static`
+- Source: `src/formats/static/CjsStaticFormat.js`
+- Visibility: Public
+- Kind: Original CarbonEngineJS class
+- Notes: Signature-based, and it decodes nothing at all since 2026-08-15. It reports the family and the payload offset; `CjsPickleFormat`, `CjsSqliteFormat` and `CjsSchemaBoundFormat` decode.
+
 <!-- class:CjsPickleProtocol0Reader -->
 ## `CjsPickleProtocol0Reader`
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@carbonenginejs/runtime-resource",
-  "version": "0.17.0",
+  "version": "0.18.1",
   "description": "CarbonEngineJS resource lifecycle, cache, source, and object loading contracts.",
   "type": "module",
   "exports": {
@@ -39,6 +39,9 @@
     "./formats/png": "./dist/formats/png/index.js",
     "./formats/red": "./dist/formats/red/index.js",
     "./formats/red/schema": "./dist/formats/red/core/blackDefinitions.js",
+    "./formats/schemabound": "./dist/formats/schemabound/index.js",
+    "./formats/sqlite": "./dist/formats/sqlite/index.js",
+    "./formats/static": "./dist/formats/static/index.js",
     "./formats/stl": "./dist/formats/stl/index.js",
     "./formats/tga": "./dist/formats/tga/index.js",
     "./formats/wav": "./dist/formats/wav/index.js",
@@ -52,7 +55,7 @@
     "node": ">=18"
   },
   "dependencies": {
-    "@carbonenginejs/runtime-utils": "^0.1.5",
+    "@carbonenginejs/runtime-utils": "^0.1.6",
     "meshoptimizer": "^1.2.0",
     "yaml": "^2.4.0"
   },