@carbonenginejs/runtime 0.1.0-alpha.2 → 0.1.0-alpha.4

package/docs/resource/formats/pickle.md

@@ -1,20 +1,23 @@
-# Data-only pickle protocol 0
-
-Status: Experimental
-Scope: `@carbonenginejs/runtime/resource/formats/pickle`
-Audience: Resource integrators handling inert legacy data caches
-Summary: Defines safe protocol-0 pickle decoding without Python object reconstruction.
-
-## Boundary
-
-`CjsPickleFormat` reads `.pickle` bytes through an internal protocol-0 reader.
-It is a data decoder, not a general Python unpickler. It never imports a
-module, resolves a global, calls a reducer, follows a persistent ID, or
-constructs a Python object. Every executable, object-bearing, newer-protocol,
-or unknown opcode fails at its exact byte offset.
-
-## One global is rebuilt, and it is a closed set
-
+# Data-only pickle
+
+Status: Experimental
+Scope: `@carbonenginejs/runtime/resource/formats/pickle`
+Audience: Resource integrators handling inert legacy data caches
+Summary: Defines safe pickle decoding, protocols 0 through 4, without Python object reconstruction.
+
+## Boundary
+
+`CjsPickleFormat` reads `.pickle` bytes through one of two internal readers,
+chosen from the bytes themselves. Protocol 0 is printable text with no header;
+every binary protocol opens with `PROTO`, which the text protocol cannot begin
+with, so the first byte decides and the caller passes nothing.
+It is a data decoder, not a general Python unpickler. It never imports a
+module, resolves a global, calls a reducer, follows a persistent ID, or
+constructs a Python object. Every executable, object-bearing, newer-protocol,
+or unknown opcode fails at its exact byte offset.
+
+## One global is rebuilt, and it is a closed set
+
 `GLOBAL` names a module and attribute to import; `REDUCE` calls it. This
 remote-execution path is refused: `os.system` fails at `GLOBAL`, before its
 argument is read.
@@ -23,91 +26,114 @@ argument is read.
 builds a plain JavaScript object preserving insertion order. Nothing is imported,
 resolved or invoked. `REDUCE` independently refuses every other target, so it
 cannot bypass the `GLOBAL` check.
-
+
 Integer-like keys are rejected because JavaScript objects sort them ahead of
 other keys, breaking insertion order.
 
 Any additional allowed name must represent pure data without behavior and be
 rebuilt directly, never through a callable.
-
-Why it matters: every one of the 25 embedded-schema static-data containers in one
-client build was scanned for the `GLOBAL` opcode's module and attribute lines,
-and this is the **only** name any of them uses — once per file, 25 occurrences,
-no second name. They use it because a schema's attribute order is its field order, which
-an ordinary dictionary would lose. Refusing it left 25 containers unreadable,
-including one of 88 MB holding roughly 477,000 records.
-
-Two limits exist because of this opcode and are worth knowing before raising
-either: `REDUCE` is the only path that builds many properties for a constant
-number of opcodes, so rebuilt properties are budgeted **across the whole decode**
-rather than per container, and a global may only ever be consumed by a `REDUCE` —
-appending one to a list or leaving it as the result is refused, because it would
-reach the caller as an empty object indistinguishable from an empty dictionary.
-
-The initial reader accepts the protocol-0 scalar, string, list, tuple,
-dictionary, memo, append, and set-item operations required by inert data
-graphs. Lists and tuples become JavaScript arrays. Integers outside the safe
-JavaScript range become lossless decimal strings. Dictionary keys must be
-strings or safe integers; collisions introduced by JSON key normalization are
-rejected.
-
-## Outputs
-
-The default `json` output returns a plain JSON-compatible graph and rejects
-cycles. `payload` and debug `raw` output preserve memo aliases and may contain
-cycles:
-
-```js
-import { CjsPickleFormat } from
-  "@carbonenginejs/runtime/resource/formats/pickle";
-
-const values = CjsPickleFormat.read(bytes);
-const exactGraph = CjsPickleFormat.read(bytes, { emit: "payload" });
-```
-
-`CjsPickleFormat.supportedProtocols` currently contains only protocol `0`.
-The public format name follows the `.pickle` extension so future protocol
-dispatch can be added without changing resource registration. Unsupported
-protocols continue to fail closed until their data-only semantics are
-implemented and tested.
-
-## Resource registration
-
-The format contains no domain target policy. A caller registers the extension
-and decides whether an exact resource path returns raw decoded data or hydrates
-a target:
-
-```js
-import {
-  CjsLoadingObject,
-  CjsResMan
-} from "@carbonenginejs/runtime/resource";
-import { CjsPickleFormat } from
-  "@carbonenginejs/runtime/resource/formats/pickle";
-
-const resMan = new CjsResMan({ source });
-
-resMan.RegisterExtension("pickle", CjsLoadingObject, {
-  Format: CjsPickleFormat,
-  Identify(_values, context) {
-    return context.resFilePath === "res:/data/profile.pickle";
-  }
-});
-```
-
-Resource paths, extensions, and filenames in the context are normalized to
-lowercase. URL-backed sources additionally expose the exact translated URL.
-
-## Limits
-
-Every read is bounded by configurable positive safe-integer limits for input
-bytes, operations, stack depth, memo size and IDs, container items, and string
-bytes. Defaults are conservative for ordinary resource files. Callers may
-lower limits per profile or read; raising them remains an explicit trust and
-memory decision.
-
-## Related documentation
-
-- [Format subpaths](README.md)
-- [Queues, publication, and registration](../reference/queues.md)
-- [Browser worker execution](../reference/workers.md)
+
+Why it matters: every one of the 25 embedded-schema static-data containers in one
+client build was scanned for the `GLOBAL` opcode's module and attribute lines,
+and this is the **only** name any of them uses — once per file, 25 occurrences,
+no second name. They use it because a schema's attribute order is its field order, which
+an ordinary dictionary would lose. Refusing it left 25 containers unreadable,
+including one of 88 MB holding roughly 477,000 records.
+
+Two limits exist because of this opcode and are worth knowing before raising
+either: `REDUCE` is the only path that builds many properties for a constant
+number of opcodes, so rebuilt properties are budgeted **across the whole decode**
+rather than per container, and a global may only ever be consumed by a `REDUCE` —
+appending one to a list or leaving it as the result is refused, because it would
+reach the caller as an empty object indistinguishable from an empty dictionary.
+
+Both readers accept the scalar, string, list, tuple, dictionary, memo, append,
+and set-item operations required by inert data graphs. Lists and tuples become
+JavaScript arrays. Integers outside the safe JavaScript range become lossless
+decimal strings. Dictionary keys must be strings or safe integers; collisions
+introduced by JSON key normalization are rejected.
+
+## Two readers, one set of rules
+
+The binary protocols differ from the text one in how a value is written down,
+not in what a decoded value is allowed to be. So the limits, the memo, the
+container bookkeeping, the dictionary-key rules, the JSON boundary and the
+closed set above live in a shared module and the readers own only their opcode
+spellings. Duplicating the refusal rules per protocol was the alternative, and a
+second reader that refuses slightly less is indistinguishable from one that
+refuses correctly until the day it matters.
+
+Two decoded shapes exist only in the binary protocols, because the text protocol
+has no way to write them:
+
+- `BINBYTES` produces a `Uint8Array`. Bytes have no JSON spelling, and inventing
+  one - an array of numbers, a base64 string - would be indistinguishable in the
+  output from data that really was that. The `payload` emit carries it; the
+  `json` emit refuses it.
+- `BINSTRING` produces latin-1 text, matching protocol 0's `STRING`. It is
+  Python 2's `str`, a byte string with no declared encoding, and latin-1 is the
+  mapping that loses nothing.
+
+## Outputs
+
+The default `json` output returns a plain JSON-compatible graph and rejects
+cycles. `payload` and debug `raw` output preserve memo aliases and may contain
+cycles:
+
+```js
+import { CjsPickleFormat } from
+  "@carbonenginejs/runtime/resource/formats/pickle";
+
+const values = CjsPickleFormat.read(bytes);
+const exactGraph = CjsPickleFormat.read(bytes, { emit: "payload" });
+```
+
+`CjsPickleFormat.supportedProtocols` contains `0` through `4`. The public format
+name follows the `.pickle` extension, and dispatch happens inside the format, so
+a caller registering it never chooses a protocol. A `PROTO` opcode declaring a
+protocol above 4 fails closed, as does any opcode outside the data subset,
+at its own byte offset.
+
+Protocol 4 is not academic: EVE Frontier's localisation containers ship in it,
+and no type in that client could be named without it.
+
+## Resource registration
+
+The format contains no domain target policy. A caller registers the extension
+and decides whether an exact resource path returns raw decoded data or hydrates
+a target:
+
+```js
+import {
+  CjsLoadingObject,
+  CjsResMan
+} from "@carbonenginejs/runtime/resource";
+import { CjsPickleFormat } from
+  "@carbonenginejs/runtime/resource/formats/pickle";
+
+const resMan = new CjsResMan({ source });
+
+resMan.RegisterExtension("pickle", CjsLoadingObject, {
+  Format: CjsPickleFormat,
+  Identify(_values, context) {
+    return context.resFilePath === "res:/data/profile.pickle";
+  }
+});
+```
+
+Resource paths, extensions, and filenames in the context are normalized to
+lowercase. URL-backed sources additionally expose the exact translated URL.
+
+## Limits
+
+Every read is bounded by configurable positive safe-integer limits for input
+bytes, operations, stack depth, memo size and IDs, container items, and string
+bytes. Defaults are conservative for ordinary resource files. Callers may
+lower limits per profile or read; raising them remains an explicit trust and
+memory decision.
+
+## Related documentation
+
+- [Format subpaths](README.md)
+- [Queues, publication, and registration](../reference/queues.md)
+- [Browser worker execution](../reference/workers.md)

package/docs/resource/formats/provenance.md

@@ -1,7 +1,7 @@
 # Format ownership and fork provenance
 
 Status: Stable  
-Scope: `@carbonenginejs/runtime/resource/formats`
+Scope: `@carbonenginejs/runtime/resource/formats`
 Audience: Users, integrators, and maintainers  
 Summary: Records where each format implementation came from, retained snapshots and digests, and what was deliberately not copied.
 
@@ -80,14 +80,20 @@ formats, not aliases. Donor MIT license and notice files are retained under
 
 ## Black definition snapshot
 
-The Black reader uses the package-owned generated definition snapshot at
-`src/resource/formats/black/core/black-schema-v1-2026-07-23.json`. The Red format exposes
-the same catalog for discovery, but its YAML reader currently accepts named
-fields without registry enforcement. The refreshed consumer snapshot landed
-in runtime-resource revision `6beda70` with SHA-256
-`9234DD86DBB5EB3676EF060E383ADBCE8F073941E76A32865D3528DD5DA7B737`.
-Its exact tools-core generator revision was not recorded and remains unknown;
-this page does not infer one from the consumer commit.
+The Black reader uses the package-owned generated definition snapshot that
+`src/resource/formats/black/core/blackDefinitions.js` imports, currently
+`black-schema-v1-2026-09-19.json`. The Red format exposes the same catalog for
+discovery, but its YAML reader currently accepts named fields without registry
+enforcement. That snapshot was generated by tools-core revision `d8c20bb` from
+the 2026-09-19 Carbon scan and landed in runtime revision `2c2b8c9f`, SHA-256
+`0A4EBE699C9055C6132A41F6F709C2CC4BF80E928DC958DE13C1AF1F45F19A19`.
+
+Dated snapshots are never edited in place; a new one is generated, and
+tools-core's `--black-module` points `blackDefinitions.js` at the newest. The
+previous snapshot, `black-schema-v1-2026-07-23.json` (runtime-resource revision
+`6beda70`, SHA-256
+`9234DD86DBB5EB3676EF060E383ADBCE8F073941E76A32865D3528DD5DA7B737`, generator
+revision unrecorded), stays in the package and is no longer imported.
 
 This retained generated artifact keeps the published readers deterministic and
 browser-safe without a runtime dependency on a sibling checkout or an