wscodec 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 auroris
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,211 @@
+# wscodec
+
+Pure-JS codec for Soulmask `actor_data` property streams — the binary
+payload UE 4.27 emits for every actor's serialized state, with the
+Soulmask-specific quirks layered on top.
+
+Zero runtime dependencies. Accepts uncompressed bytes, returns
+JavaScript objects, and vice versa. Round-trip is byte-identical
+against every actor in a tested `world.db` (174.6 MB across 11,667
+rows; `npm test`).
+
+## Scope
+
+The library covers the property-stream wire format only. Soulmask's
+column wire format adds an outer LZ4 envelope on top:
+
+```
+actor_data column bytes
+├── [0..3]  u32 LE     outer version tag = 0x00000002
+└── [4..]   LZ4 block  size-prefixed; decompresses to:
+            ├── [0..3]  u32 LE  inner version tag = 0x00000002
+            └── [4..]   FPropertyTag stream + "None" terminator
+```
+
+wscodec handles the bottom half (the bytes that come out of LZ4
+decompression). The caller handles LZ4 — see "LZ4 integration" below
+for a copy-paste recipe with `lz4-wasm-nodejs`.
+
+The SQLite `actor_table.data_version` column stores the NEGATIVE of
+the on-wire DataVersion. A healthy blob with wire `DataVersion=2`
+lives in a row whose `data_version` column reads `-2`. The wire
+bytes themselves are always the unsigned `0x00000002`.
+
+## Install
+
+```sh
+npm install wscodec
+```
+
+## API
+
+### Top-level
+
+```js
+import { UnrealBlob } from 'wscodec';
+
+const blob = UnrealBlob.decode(uncompressedBytes); // Uint8Array → blob
+const bytes = blob.serialize();                    // blob → Uint8Array
+UnrealBlob.detect(u8);                              // sniff version tag → boolean
+```
+
+`UnrealBlob.decode(u8)` parses the version tag + property stream and
+returns an `UnrealBlob` with:
+
+| field | type | meaning |
+|---|---|---|
+| `versionTag` | `number` | the wire version tag (always 2 in the wild) |
+| `properties` | `Property[]` | top-level property list |
+| `terminated` | `boolean` | whether the stream ended on a `None` terminator |
+| `bodyTrailing` | `Uint8Array \| null` | any bytes past the terminator (rare) |
+| `error` | `string \| null` | populated when structural decode failed |
+| `_raw` | `Uint8Array` | the input bytes, retained for pass-through serialize |
+| `_dirty` | `boolean` | set by mutating callers to force re-encode |
+
+`blob.serialize()` returns a `Uint8Array`. When `_dirty` is false it
+returns `_raw` verbatim (byte-identical pass-through). When `_dirty`
+is true it re-emits the property stream from `properties` via
+`writePropertyStream`.
+
+`blob.findProperty(name)` returns the first top-level property whose
+tag name matches, or `null`.
+
+### Property tree
+
+`blob.properties` is an array of `Property` instances. Each carries
+a `PropertyTag` (`name`, `type`, `size`, …) and a `value` whose
+JavaScript shape depends on the tag's type:
+
+| tag type | value shape |
+|---|---|
+| `IntProperty`, `FloatProperty`, `BoolProperty`, … | plain JS primitive |
+| `StrProperty`, `NameProperty` | string / `FName` |
+| `StructProperty` | `StructValue` — `.value` is either a plain object (known binary structs like `Vector`, `Quat`, `Transform`, `Guid`, …) or a nested property array |
+| `ArrayProperty`, `SetProperty` | `ArrayValue` / `SetValue` with `.elements` |
+| `MapProperty` | `MapValue` with `.entries: [[key, value], …]` |
+| `ObjectProperty`, `ClassProperty`, `Weak*`, `Lazy*`, `WSObjectProperty` | `ObjectRef` (kind + optional path/classPath/embedded stream) |
+| `SoftObjectProperty`, `SoftClassProperty` | `SoftObjectRef` (`assetPath`, `subPath`) |
+| `TextProperty` | `FTextValue` (handles UE4 FText history types -1, 0, 2, 4) |
+| anything wscodec couldn't structurally decode | `OpaqueValue` — bytes retained verbatim |
+
+Submodule re-exports make the value classes importable directly:
+
+```js
+import { ObjectRef, SoftObjectRef, FTextValue, OpaqueValue, StructValue } from 'wscodec';
+import { PropertyTag, ArrayValue, SetValue, MapValue } from 'wscodec';
+import { FName, FGuid } from 'wscodec';
+```
+
+Lower-level helpers (`Cursor`, `Writer`, `readPropertyStream`,
+`writePropertyStream`, `readValue`, `writeValue`, `STRUCT_HANDLERS`)
+are also exported for callers building custom workflows on top.
+
+### Editing
+
+```js
+import { UnrealBlob } from 'wscodec';
+
+const blob = UnrealBlob.decode(inner);
+
+// Mutate the tree directly. The library does not provide typed
+// mutators; callers manipulate `properties` and set `_dirty` to
+// force re-encode.
+blob.findProperty('JianZhuHP').value = 100;
+blob._dirty = true;
+
+const updatedBytes = blob.serialize(); // re-emits from properties
+```
+
+`serialize()` for a dirty blob is byte-identical to a fresh
+`decode + serialize` cycle on its output — verified on every row
+of the tested `world.db`.
+
+## LZ4 integration
+
+`actor_data` column bytes come out of LZ4 compression. wscodec
+doesn't bundle an LZ4 implementation — that's a caller concern. A
+working recipe using `lz4-wasm-nodejs`:
+
+```js
+import Database from 'better-sqlite3';
+import * as lz4 from 'lz4-wasm-nodejs';
+import { UnrealBlob } from 'wscodec';
+
+const OUTER_VERSION_TAG = 0x00000002;
+const OUTER_HEADER_SIZE = 4;
+
+function decodeColumnBytes(u8) {
+  // Sniff the outer version tag.
+  const dv = new DataView(u8.buffer, u8.byteOffset, u8.byteLength);
+  if (u8.length < OUTER_HEADER_SIZE + 4 || dv.getUint32(0, true) !== OUTER_VERSION_TAG) {
+    throw new Error('Not a Soulmask actor_data blob');
+  }
+  // LZ4 block starts right after the version tag.
+  const inner = lz4.decompress(u8.subarray(OUTER_HEADER_SIZE));
+  return UnrealBlob.decode(inner);
+}
+
+function encodeBlob(blob) {
+  const inner = blob.serialize();
+  const compressed = lz4.compress(inner);
+  const out = new Uint8Array(OUTER_HEADER_SIZE + compressed.length);
+  new DataView(out.buffer).setUint32(0, OUTER_VERSION_TAG, true);
+  out.set(compressed, OUTER_HEADER_SIZE);
+  return out;
+}
+
+const db = new Database('./world.db', { readonly: true });
+for (const row of db.prepare('SELECT actor_serial, actor_data FROM actor_table').all()) {
+  const blob = decodeColumnBytes(new Uint8Array(row.actor_data));
+  console.log(row.actor_serial, blob.properties.length, 'properties');
+}
+```
+
+Note: LZ4 compression is not deterministic — two compressors will
+produce different bytes for the same input. wscodec's
+byte-identical guarantee covers the inner property-stream bytes;
+the outer column bytes round-trip only for unmodified blobs (cache
+the input column bytes if you need that).
+
+## Round-trip guarantees
+
+For every row in the tested `world.db`:
+
+- `UnrealBlob.decode(inner)` succeeds without `error` set.
+- `blob.serialize()` with `_dirty = false` returns the input bytes byte-identical.
+- `blob.serialize()` with `_dirty = true` re-emits from `properties` and is byte-identical to the input.
+
+Coverage: 174,610,207 bytes across 11,667 actors, including every
+known Soulmask wire-format quirk:
+
+- `kind=0x01` ObjectProperty with the 4-byte actor-ref prefix.
+- Embedded ObjectProperty streams with the 4-byte FName.Number trailer
+  (the Soulmask `JianZhuInstGLQComponent`-style nested format).
+- ArrayProperty<ObjectProperty> with the JianZhuInstYuanXings
+  per-element placement-binary blocks (8-byte header + three
+  stride/count sections per yuan-xing prototype).
+- ArrayProperty<TextProperty> elements with FText history types
+  -1, 0, 2, and 4 (including the legacy UE3-style uint32 booleans in
+  FNumberFormattingOptions).
+- SetProperty<StructProperty> with implicit FGuid struct keys.
+- Custom Soulmask Map<Struct,Struct> framing.
+
+## Running the test
+
+```sh
+git clone …                # repo with world.db at the root
+cd repo/wscodec
+npm install
+npm test                    # looks for world.db two dirs up by default
+# or
+node test/test-roundtrip.mjs /path/to/world.db
+```
+
+Test deps: `lz4-wasm-nodejs` (LZ4 inside the test), `better-sqlite3`
+(reads the `world.db` SQLite file). Both are picked up via npm
+module resolution; if `better-sqlite3` isn't installed at the package
+root the test will surface that with a clear error.
+
+## License
+
+MIT.

package/io.mjs

@@ -0,0 +1,150 @@
+/**
+ * Cursor + Writer — byte-level read/write primitives over a Uint8Array.
+ *
+ * No Unreal semantics here. FString lives here too because it's a stateful
+ * read/write on the same DataView; everything else (FName, FGuid, structs,
+ * properties) builds on top of these.
+ */
+
+export class Cursor {
+  constructor(bytes, offset = 0) {
+    this.bytes = bytes;
+    this.dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+    this.offset = offset;
+  }
+  pos()       { return this.offset; }
+  eof()       { return this.offset >= this.bytes.length; }
+  remaining() { return this.bytes.length - this.offset; }
+  skip(n)     { this.offset += n; }
+  seek(n)     { this.offset = n; }
+
+  readUint8()   { const v = this.dv.getUint8(this.offset);            this.offset += 1; return v; }
+  readInt8()    { const v = this.dv.getInt8(this.offset);             this.offset += 1; return v; }
+  readUint16()  { const v = this.dv.getUint16(this.offset, true);     this.offset += 2; return v; }
+  readInt16()   { const v = this.dv.getInt16(this.offset, true);      this.offset += 2; return v; }
+  readUint32()  { const v = this.dv.getUint32(this.offset, true);     this.offset += 4; return v; }
+  readInt32()   { const v = this.dv.getInt32(this.offset, true);      this.offset += 4; return v; }
+  readUint64()  { const v = this.dv.getBigUint64(this.offset, true);  this.offset += 8; return v; }
+  readInt64()   { const v = this.dv.getBigInt64(this.offset, true);   this.offset += 8; return v; }
+  readFloat32() { const v = this.dv.getFloat32(this.offset, true);    this.offset += 4; return v; }
+  readFloat64() { const v = this.dv.getFloat64(this.offset, true);    this.offset += 8; return v; }
+  readBytes(n)  { const out = this.bytes.subarray(this.offset, this.offset + n); this.offset += n; return out; }
+
+  /**
+   * FString:  int32 SaveNum  (length in code units INCLUDING null terminator)
+   *           SaveNum > 0 → ANSI;  SaveNum < 0 → UTF-16 LE;  SaveNum == 0 → empty.
+   *
+   * The wire format distinguishes two flavors of empty:
+   *   SaveNum =  0          → "null" form. No further bytes.
+   *   SaveNum =  1 (or -1)  → "empty-with-terminator". 1 ANSI byte (or 1
+   *                           UTF-16 unit) follows; both are the NUL
+   *                           terminator. The decoded string is still "".
+   *
+   * Both produce the same JS value (""), but to round-trip byte-identical
+   * we need to know which one was on the wire. That distinction lives in
+   * `isNull` on the return value.
+   */
+  readFString() {
+    const saveNum = this.readInt32();
+    if (saveNum === 0) return { value: '', isUnicode: false, isNull: true };
+    const isUnicode = saveNum < 0;
+    const codeUnits = isUnicode ? -saveNum : saveNum;
+    const byteLen = isUnicode ? codeUnits * 2 : codeUnits;
+    const slice = this.readBytes(byteLen);
+    const data = isUnicode ? slice.subarray(0, byteLen - 2) : slice.subarray(0, byteLen - 1);
+    let value;
+    if (isUnicode) {
+      const codes = [];
+      const dv = new DataView(data.buffer, data.byteOffset, data.byteLength);
+      for (let i = 0; i + 1 < data.length; i += 2) codes.push(dv.getUint16(i, true));
+      value = String.fromCharCode(...codes);
+    } else {
+      value = '';
+      for (let i = 0; i < data.length; i++) value += String.fromCharCode(data[i]);
+    }
+    return { value, isUnicode, isNull: false };
+  }
+}
+
+export class Writer {
+  constructor(initialCapacity = 256) {
+    this.buffer = new ArrayBuffer(initialCapacity);
+    this.bytes = new Uint8Array(this.buffer);
+    this.dv = new DataView(this.buffer);
+    this.offset = 0;
+  }
+  pos() { return this.offset; }
+  finalize() { return this.bytes.slice(0, this.offset); }
+
+  _ensure(n) {
+    if (this.offset + n <= this.buffer.byteLength) return;
+    let cap = this.buffer.byteLength;
+    while (cap < this.offset + n) cap *= 2;
+    const newBuf = new ArrayBuffer(cap);
+    const newU8 = new Uint8Array(newBuf);
+    newU8.set(this.bytes.subarray(0, this.offset));
+    this.buffer = newBuf;
+    this.bytes = newU8;
+    this.dv = new DataView(newBuf);
+  }
+
+  writeUint8(v)   { this._ensure(1); this.dv.setUint8(this.offset, v);              this.offset += 1; }
+  writeInt8(v)    { this._ensure(1); this.dv.setInt8(this.offset, v);               this.offset += 1; }
+  writeUint16(v)  { this._ensure(2); this.dv.setUint16(this.offset, v, true);       this.offset += 2; }
+  writeInt16(v)   { this._ensure(2); this.dv.setInt16(this.offset, v, true);        this.offset += 2; }
+  writeUint32(v)  { this._ensure(4); this.dv.setUint32(this.offset, v >>> 0, true); this.offset += 4; }
+  writeInt32(v)   { this._ensure(4); this.dv.setInt32(this.offset, v | 0, true);    this.offset += 4; }
+  writeUint64(v)  { this._ensure(8); this.dv.setBigUint64(this.offset, BigInt(v), true); this.offset += 8; }
+  writeInt64(v)   { this._ensure(8); this.dv.setBigInt64(this.offset, BigInt(v), true);  this.offset += 8; }
+  writeFloat32(v) { this._ensure(4); this.dv.setFloat32(this.offset, v, true);      this.offset += 4; }
+  writeFloat64(v) { this._ensure(8); this.dv.setFloat64(this.offset, v, true);      this.offset += 8; }
+  writeBytes(u8)  { this._ensure(u8.length); this.bytes.set(u8, this.offset);       this.offset += u8.length; }
+
+  /**
+   * Write an FString. The `isNull` parameter only matters when `value` is
+   * the empty string (or `null`/`undefined`); for non-empty strings the
+   * wire form is unambiguous.
+   *
+   *   value = null/undefined         → SaveNum = 0 (null form)
+   *   value = ''  and isNull truthy  → SaveNum = 0 (null form)
+   *   value = ''  and isNull false   → SaveNum = 1/-1 (empty-with-terminator)
+   *   value = ''  and isNull null    → SaveNum = 0 (default; matches prior behavior)
+   *   value = 'x' (any non-empty)    → SaveNum encodes content
+   *
+   * `isUnicode` is auto-detected from the content when not supplied. For
+   * an empty-with-terminator string the caller picks the encoding via
+   * `isUnicode` (defaults to ANSI).
+   */
+  writeFString(value, isUnicode = null, isNull = null) {
+    // Null form: no payload bytes.
+    if (value == null || (value === '' && isNull !== false)) {
+      this.writeInt32(0);
+      return;
+    }
+    // Auto-detect encoding for non-empty content. Empty-with-terminator
+    // keeps whatever the caller passed (default ANSI).
+    if (isUnicode === null) {
+      isUnicode = false;
+      for (let i = 0; i < value.length; i++) {
+        if (value.charCodeAt(i) >= 0x80) { isUnicode = true; break; }
+      }
+    }
+    if (isUnicode) {
+      const len = value.length + 1;
+      this.writeInt32(-len);
+      this._ensure(len * 2);
+      for (let i = 0; i < value.length; i++) {
+        this.dv.setUint16(this.offset + i * 2, value.charCodeAt(i), true);
+      }
+      this.dv.setUint16(this.offset + value.length * 2, 0, true);
+      this.offset += len * 2;
+    } else {
+      const len = value.length + 1;
+      this.writeInt32(len);
+      this._ensure(len);
+      for (let i = 0; i < value.length; i++) this.bytes[this.offset + i] = value.charCodeAt(i);
+      this.bytes[this.offset + value.length] = 0;
+      this.offset += len;
+    }
+  }
+}

package/package.json

@@ -0,0 +1,39 @@
+{
+  "name": "wscodec",
+  "version": "0.1.0",
+  "description": "Pure-JS codec for Soulmask actor_data property streams (UE4.27 FPropertyTag wire format). Zero runtime dependencies — accepts uncompressed bytes, returns JS objects, and vice versa. Round-trip byte-identical against every actor.",
+  "type": "module",
+  "main": "./wscodec.mjs",
+  "exports": {
+    ".": "./wscodec.mjs",
+    "./io": "./io.mjs",
+    "./primitives": "./primitives.mjs",
+    "./structs": "./structs.mjs",
+    "./values": "./values.mjs",
+    "./properties": "./properties.mjs"
+  },
+  "files": [
+    "wscodec.mjs",
+    "io.mjs",
+    "primitives.mjs",
+    "structs.mjs",
+    "values.mjs",
+    "properties.mjs",
+    "README.md"
+  ],
+  "scripts": {
+    "test": "node test/test-roundtrip.mjs"
+  },
+  "keywords": [
+    "soulmask",
+    "unreal",
+    "ue4",
+    "fpropertytag",
+    "codec"
+  ],
+  "license": "MIT",
+  "devDependencies": {
+    "better-sqlite3": "^12.10.0",
+    "lz4-wasm-nodejs": "^0.9.2"
+  }
+}

package/primitives.mjs

@@ -0,0 +1,70 @@
+/**
+ * FName and FGuid.
+ *
+ * In this Soulmask format FName is serialized as a plain FString (no
+ * trailing FName.Number int32). The `number` field stays 0 and exists
+ * only for symmetry with full UE FNames.
+ */
+
+export class FName {
+  constructor(value, { isUnicode = false, number = 0, isNull = false } = {}) {
+    this.value = value;
+    this.isUnicode = isUnicode;
+    this.number = number;
+    // Tracks the wire-form distinction between an FString with SaveNum=0
+    // (the "null" form) and SaveNum=1 (empty-with-terminator). Only ever
+    // meaningful when `value === ''`; for non-empty FNames this stays
+    // false and is ignored on write.
+    this.isNull = isNull;
+  }
+  toString() { return this.value; }
+
+  static read(cursor) {
+    const s = cursor.readFString();
+    return new FName(s.value, { isUnicode: s.isUnicode, isNull: !!s.isNull });
+  }
+
+  write(writer) { writer.writeFString(this.value, this.isUnicode, this.isNull); }
+
+  /** Accepts an FName, a bare string, or a plain {value,isUnicode,isNull} record. */
+  static from(x) {
+    if (x instanceof FName) return x;
+    if (typeof x === 'string') return new FName(x);
+    if (x && typeof x === 'object') {
+      return new FName(x.value, {
+        isUnicode: !!x.isUnicode,
+        number: x.number || 0,
+        isNull: !!x.isNull,
+      });
+    }
+    throw new Error('FName.from: unsupported value ' + typeof x);
+  }
+}
+
+export class FGuid {
+  /** Stored as the canonical 8-4-4-4-12 hex string (uppercase). */
+  constructor(value) { this.value = value; }
+  toString() { return this.value; }
+
+  static read(cursor) {
+    const A = cursor.readUint32(), B = cursor.readUint32(), C = cursor.readUint32(), D = cursor.readUint32();
+    const h = (n, w) => n.toString(16).padStart(w, '0').toUpperCase();
+    return new FGuid(`${h(A, 8)}-${h(B >>> 16, 4)}-${h(B & 0xFFFF, 4)}-${h(C >>> 16, 4)}-${h(C & 0xFFFF, 4)}${h(D, 8)}`);
+  }
+
+  write(writer) {
+    const m = String(this.value).match(/^([0-9A-Fa-f]{8})-([0-9A-Fa-f]{4})-([0-9A-Fa-f]{4})-([0-9A-Fa-f]{4})-([0-9A-Fa-f]{4})([0-9A-Fa-f]{8})$/);
+    if (!m) throw new Error(`FGuid.write: invalid FGuid string '${this.value}'`);
+    const A = parseInt(m[1], 16);
+    const B = (parseInt(m[2], 16) << 16) | parseInt(m[3], 16);
+    const C = (parseInt(m[4], 16) << 16) | parseInt(m[5], 16);
+    const D = parseInt(m[6], 16);
+    writer.writeUint32(A); writer.writeUint32(B); writer.writeUint32(C); writer.writeUint32(D);
+  }
+
+  static from(x) {
+    if (x instanceof FGuid) return x;
+    if (typeof x === 'string') return new FGuid(x);
+    throw new Error('FGuid.from: unsupported value ' + typeof x);
+  }
+}