@sofa-buffers/corelib 0.8.0

package/CHANGELOG.md

@@ -0,0 +1,155 @@
+# Changelog
+
+All notable changes to `@sofa-buffers/corelib` are documented here.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+While the version is below `1.0.0`, breaking changes bump the **minor** version.
+
+## [Unreleased]
+
+### Changed
+
+- **Strict UTF-8 for `string` fields (corelib-ts#85, MESSAGE_SPEC §8,
+  CORELIB_PLAN §6.4).** JavaScript strings are a Unicode string type, so the
+  corelib transcodes `string` payloads at the boundary and is now **always
+  strict** — there is no lossy mode and the `SOFAB_STRICT_UTF8` option is a no-op
+  that is omitted. Silent `U+FFFD` substitution, previously produced by both the
+  decoder and the encoder, is removed in **both** directions:
+  - *Decode:* the corelib builds the string with a **fatal** `TextDecoder`
+    (`new TextDecoder("utf-8", { fatal: true })`). An invalid-UTF-8 payload that
+    is materialized (`Cursor.readString`) is now the `INVALID` outcome —
+    `SofabError` with `SofabErrorCode.InvalidMsg` (`"INVALID_MSG"`) — instead of
+    decoding to a string full of replacement characters. Skipped fields are never
+    validated; embedded `U+0000` round-trips.
+  - *Encode:* `writeString` (both the in-memory fast path and the streaming
+    `TextEncoder` path) now **rejects** an **unpaired surrogate** with
+    `SofabError` / `SofabErrorCode.Argument` (`"ARGUMENT"`) rather than emitting
+    `EF BF BD`. Every valid string — ASCII, multibyte BMP, correctly paired
+    astral code points, embedded `U+0000` — still encodes byte-for-byte as
+    before.
+
+  The shared `assets/test_vectors.json` gains the top-level `invalid_utf8`
+  negative-vector array (tracked by corelib-c-cpp#97); the conformance suite
+  exercises it under the strict decode and encode paths.
+
+### Added
+
+- **`Cursor.fixSub` — the delivered fixlen subtype (corelib-ts#58).** A new
+  public accessor on `Cursor`, the companion to `wire`, that reports the fixlen
+  subtype of the header `readHeader` just accepted — one of `FixlenSubtype`
+  (`Fp32`/`Fp64`/`String`/`Blob`) when `wire` is `Fixlen` or `ArrayFixlen`, and
+  `-1` otherwise. The four fixlen subtypes all share one wire type, so `wire`
+  alone cannot separate them; `fixSub` lets a generated guard skip a fixlen
+  field whose subtype contradicts the schema (MESSAGE_SPEC §7.3) — exactly as it
+  already does on `wire` for the other kinds — instead of passing the wire-type
+  guard and then throwing from the wrong-typed reader. It is *peeked* (the
+  subtype word is not consumed), so the matching typed reader / `skip()` still
+  reads and validates the word and a malformed or truncated one still surfaces
+  `INVALID` / `INCOMPLETE`. Completes §7.3 for the TypeScript target, matching
+  corelib-py's `Field.subtype` and corelib-cpp's `fixType()`.
+
+- **Opt-in decode limits (corelib-ts#38).** A new optional `DecodeLimits`
+  options object — `{ maxArrayCount?, maxStringLen?, maxBlobLen? }` — is accepted
+  by every decode entry point: `decode(bytes, visitor, limits?)`, the `IStream`
+  constructor, and the `Cursor` constructor. When set, an array count or string /
+  blob byte length that exceeds the cap is rejected at the field's header —
+  before the array is sized or any payload is decoded / streamed to the visitor —
+  with the new `SofabErrorCode.LimitExceeded` (`"LIMIT_EXCEEDED"`). The decoder
+  never clamps or truncates. `LimitExceeded` is deliberately distinct from
+  `InvalidMsg`: exceeding a receiver-configured limit is *policy*, not wire
+  malformation — the identical bytes decode fine under a looser limit. **Default:
+  no limits (today's behavior); the corelib invents no default cap** — the values
+  come from the sofabgen config, baked into generated code (generator#102). Also
+  hardens `Cursor` so a wire array `count` larger than the bytes remaining is
+  rejected as `Incomplete` before `new Array(count)` is sized, so a hostile count
+  can never drive an allocation larger than the input.
+- **Finish-less three-valued decode outcome (MESSAGE_SPEC §7).** Truncation — a
+  decode that ends *inside* a field — is now a distinct outcome from a malformed
+  message. New `SofabErrorCode.Incomplete` (`"INCOMPLETE"`) and a `DecodeStatus`
+  enum (`Complete` / `Incomplete` / `Invalid`) are exported. Every one-shot
+  truncation site (`decode()`, `Cursor`) that used to throw `INVALID_MSG` — an
+  unterminated varint, a payload / array shorter than its declared length, or a
+  nested sequence left open at end-of-buffer — now throws `INCOMPLETE` instead;
+  genuinely malformed input (varint over 64 bits, bad subtype/length/count, id
+  over max, dangling sequence-end, over-`MAX_DEPTH` nesting) still throws
+  `INVALID_MSG`. Mirrors corelib-go#42.
+
+### Changed
+
+- **BREAKING (decode API):** there is no finish/finalize step. `IStream.end()`
+  no longer throws to promote an incomplete stream to an error; it is now a pure
+  accessor returning `DecodeStatus.Complete` when the stream ended on a field
+  boundary or `DecodeStatus.Incomplete` when it ended inside one. A malformed
+  message still throws from `IStream.feed()`. Callers that relied on `end()`
+  throwing on truncation must check its return value instead.
+- **BREAKING (wire format):** a fixlen array (`fp32`/`fp64`) now always carries
+  its `fixlen_word` — even when empty (`element_count == 0`). Previously an empty
+  fixlen array was `[header][count=0]` with no `fixlen_word`, making an empty
+  `fp32` array byte-identical to an empty `fp64` one (`05 00`); a decoder could
+  not tell them apart. An empty fixlen array is now
+  `[header][count=0][fixlen_word]` with no payload (`05 00 20` for `fp32`,
+  `05 00 41` for `fp64`), so the element subtype stays recoverable. Integer
+  arrays (`u8`…`u64`, `i8`…`i64`) are unchanged — they never carry a
+  `fixlen_word` — so an empty integer array stays `[header][count=0]`. Mirrors
+  CORELIB_PLAN §4.8 / MESSAGE_SPEC §3 and corelib-c-cpp#45.
+
+## [0.2.0] - 2026-06-29
+
+A performance release: the encode and decode hot paths no longer churn
+short-lived `BigInt` objects, which V8 profiling identified as the dominant
+cost. The wire format is unchanged and all shared conformance vectors still
+pass. One source-level breaking change to the decode `Visitor` enables the
+decode-side win.
+
+### Changed
+
+- **BREAKING:** `Visitor.unsigned`, `Visitor.signed`, `Visitor.arrayUnsigned`,
+  and `Visitor.arraySigned` now receive `value: number | bigint` instead of
+  `bigint`. Integer values are delivered **number-first** — a `number` when the
+  value fits exactly (`≤ 2^53 − 1`, covering field ids, `u8`…`u32` and small
+  `u64`/`i64` values) and a `bigint` only beyond that. This avoids a per-value
+  `bigint` allocation on the common path.
+
+  **Migration:** a handler that did `bigint`-only arithmetic on a decoded value
+  must coerce the argument, e.g. `const n = typeof v === "bigint" ? v : BigInt(v)`
+  (to keep working in `bigint`) or `Number(v)` (to work in `number`, safe for
+  values `≤ 2^53`). The encoder is unaffected — it already accepted
+  `number | bigint` — so re-encoding a decoded value is byte-identical.
+
+### Added
+
+- `decode()` now runs a dedicated **contiguous fast-path decoder** that advances
+  a single cursor over the whole buffer (the technique Protocol Buffers uses),
+  instead of driving the resumable per-byte state machine. Same API and
+  validation; markedly faster when the whole message is in hand. The streaming
+  `IStream` remains for chunked input.
+- Expanded the shared conformance suite to the 67-vector `test_vectors.json`,
+  including the new `skip-ids` decode scenario (auto-skipping fields by id at any
+  nesting depth, including whole nested sequences) and `requires`/`skip_ids`
+  metadata.
+
+### Performance
+
+- **Decode:** number-first values + the contiguous fast path cut BigInt-builtin
+  time from ~35% to ~4% and GC from ~10% to ~1% on small-value workloads. A
+  `u32` array decodes ~2.2× faster streaming and ~2.6× faster contiguous
+  (≈165 / ≈270 MB/s) for a number-consuming visitor. (#6)
+- **Decode (streaming):** the resumable varint reader accumulates into two 32-bit
+  number halves instead of doing a per-byte `bigint` shift, with no loss of
+  64-bit fidelity.
+- **Encode:** `encodeVarint` / `varintSize` split the 64-bit value into two
+  32-bit number halves once and emit LEB128 with number-only arithmetic,
+  dropping per-value `bigint` allocations from ~20 to 2. A full-range `u64`
+  array encodes ~4.4× faster (≈14.5 → ≈64 MB/s, isolated); ids, lengths, counts
+  and small scalars/arrays take a number fast path. (#5)
+
+## [0.1.0]
+
+- Initial release: streaming, dependency-free TypeScript implementation of the
+  SofaBuffers binary serialization format — `OStream` to encode and `IStream`
+  (driving a `Visitor`) to decode, both chunkable, with a swappable acceleration
+  `Kernel` seam.
+
+[0.2.0]: https://github.com/sofa-buffers/corelib-ts/releases/tag/v0.2.0
+[0.1.0]: https://github.com/sofa-buffers/corelib-ts/releases/tag/v0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SofaBuffers
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,274 @@
+<p align="center"><img src="assets/sofabuffers_logo.png" alt="SofaBuffers" height="140"></p>
+
+# SofaBuffers
+
+<b>Structured Objects For Anyone</b><br>
+<i>... so optimized, feels amazing.</i>
+
+[Would you like to know more?](https://github.com/sofa-buffers)
+
+## SofaBuffers TypeScript library
+
+[![CI](https://github.com/sofa-buffers/corelib-ts/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/sofa-buffers/corelib-ts/actions/workflows/ci.yml)
+[![Coverage](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fsofa-buffers%2Fcorelib-ts%2Fbadges%2Fcoverage.json)](https://github.com/sofa-buffers/corelib-ts/actions/workflows/ci.yml)
+[![Branches](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fsofa-buffers%2Fcorelib-ts%2Fbadges%2Fbranches.json)](https://github.com/sofa-buffers/corelib-ts/actions/workflows/ci.yml)
+[![Docs](https://img.shields.io/badge/docs-API-blue)](https://sofa-buffers.github.io/corelib-ts/)
+
+[GitHub repository](https://github.com/sofa-buffers/corelib-ts)
+
+A dependency-free, streaming TypeScript implementation of the SofaBuffers
+(*Sofab*) serialization format — the runtime stream core that runs anywhere
+JavaScript does (Node.js, browsers, Electron, Deno, Bun, a `<script>` tag).
+
+Like protobuf's `CodedInputStream` / `CodedOutputStream`, it is meant to be
+driven by generated code: the `sofabgen` generator emits one class per message
+with marshal / unmarshal methods that call these primitives. Two decode models
+are offered — a resumable push / visitor decoder for streaming, and a
+monomorphic pull cursor (`Cursor`) driven by a single `switch` over the field id.
+
+### Requirements
+
+Node.js 20+ (CI runs 20 / 24), or any modern browser / Electron / Deno /
+Bun. Built with TypeScript 6.x; targets ES2020 (`bigint` required).
+
+### Dependencies
+
+None. Zero runtime dependencies; uses only standard JS / Web APIs
+(`Uint8Array`, `DataView`, `TextEncoder` / `TextDecoder`).
+
+### Packaging
+
+Published as `@sofa-buffers/corelib`:
+
+```bash
+npm install @sofa-buffers/corelib
+```
+
+Ships ESM (`.js`), CommonJS (`.cjs`), a browser IIFE global (`SofaBuffers`) and
+full type declarations.
+
+## Why this design
+
+| Goal | How |
+|------|-----|
+| Runs everywhere | Pure TypeScript over `Uint8Array` / `DataView` / `TextEncoder`, no Node built-ins on the hot path. |
+| Streaming **out** | `OStream` writes into a small caller buffer and calls a `FlushSink` when it fills, so a message can exceed the buffer. |
+| Streaming **in** | `IStream` is a resumable state machine fed arbitrary chunks; large string / blob payloads arrive in pieces. |
+| Fast whole-buffer decode | With the whole message in one buffer, `decode()` (push) and `Cursor` (pull) advance a single cursor. |
+| Full 64-bit fidelity | Scalars round-trip the entire `uint64` / `int64` range: `number` when exact, `bigint` beyond `2^53-1` (`Long` offers a `bigint`-free array path). |
+| Generated-code friendly | The pull `Cursor` gives a monomorphic `readHeader()` + typed `read*` loop; the push `Visitor` has all-optional methods. |
+| Reserve-offset | `new OStream(buf, offset)` leaves room at the front for a lower-layer header, saving a copy. |
+| Explicit endianness | IEEE-754 values are read / written little-endian via `DataView`, identical on every engine. |
+| Pluggable acceleration | The encoder's bulk array paths run through a swappable `Kernel`; the default is pure TypeScript. |
+
+## Usage
+
+The codec has four use cases — serialize a message that fits in one buffer,
+serialize one too large for the buffer (streamed out in chunks), deserialize a
+whole message, and deserialize one arriving in chunks — plus the generated-code
+path that wraps them. Problems are reported by throwing `SofabError`; the cause is
+on `SofabError.code` (`ARGUMENT`, `USAGE`, `BUFFER_FULL`, `INVALID_MSG`,
+`INCOMPLETE`). The decoder splits its two failure kinds (MESSAGE_SPEC §7):
+`INVALID_MSG` is a message malformed regardless of what follows, while
+`INCOMPLETE` means the bytes merely ended *inside* a field — a truncation more
+bytes could complete, which is not an error the caller must treat as one. There
+is no finish/finalize step: a streaming decode reports `INCOMPLETE` from `end()`
+(see below), never by promoting it to a throw.
+
+### Serialize
+
+Write fields into an in-memory `OStream` and take a view of the finished bytes:
+
+```ts
+import { OStream } from "@sofa-buffers/corelib";
+
+const os = new OStream();          // in-memory, auto-growing buffer
+os.writeUnsigned(1, 42);
+os.writeSigned(2, -7);
+os.writeString(3, "hi");
+const bytes = os.bytes();          // Uint8Array view of the finished message
+```
+
+### Serialize stream
+
+Constructed over a caller-owned buffer with a `FlushSink`, `OStream` drains that
+small buffer whenever it fills, so the buffer never has to be message-sized:
+
+```ts
+import { OStream, type FlushSink } from "@sofa-buffers/corelib";
+
+const out: number[] = [];
+const sink: FlushSink = (chunk) => out.push(...chunk); // or socket / file / stream
+const os = new OStream(new Uint8Array(16), 0, sink);   // tiny 16-byte buffer
+for (let i = 0; i < 1000; i++) os.writeUnsigned(i, BigInt(i));
+os.flush();                                            // push the tail
+```
+
+### Deserialize
+
+`decode()` walks a whole buffer and calls one optional `Visitor` method per field;
+unhandled fields are silently skipped:
+
+```ts
+import { decode, type Visitor } from "@sofa-buffers/corelib";
+
+class My implements Visitor {
+  a = 0;
+  b = 0;
+  unsigned(id: number, v: number | bigint) { if (id === 1) this.a = Number(v); }
+  signed(id: number, v: number | bigint)   { if (id === 2) this.b = Number(v); }
+  // fp32(), fp64(), string(), blob(), arrayBegin(), sequenceBegin(), ... as needed
+}
+
+decode(bytes, new My());
+```
+
+### Deserialize stream
+
+`IStream` resumes across chunk boundaries, so feed it whatever the transport hands
+you — from any source — and read the outcome from `end()`. String / blob payloads
+arrive as one or more chunks tagged with the field's `total` length and byte
+`offset`:
+
+```ts
+import { IStream, DecodeStatus, type Visitor } from "@sofa-buffers/corelib";
+
+const visitor: Visitor = {
+  blob(id, total, offset, chunk) {
+    /* append `chunk` at `offset`; the field is `total` bytes */
+  },
+};
+
+const is = new IStream();
+for await (const chunk of source) is.feed(chunk, visitor); // any async byte source
+// end() is a pure accessor — it never throws and never promotes an incomplete
+// decode to an error (MESSAGE_SPEC §7). The caller owns end-of-input.
+if (is.end() !== DecodeStatus.Complete) {
+  // stream ended inside a field (INCOMPLETE) — wait for more bytes, or treat
+  // the truncation as an error if this really was the end of input.
+}
+```
+
+### Code generator
+
+`sofabgen` compiles a schema to one class per message with a `marshal` (chaining
+`OStream` writes) and a `static decode` driven by a monomorphic pull `Cursor` —
+one `switch` over `c.id`. A hand-written stand-in, encoded then decoded:
+
+```ts
+import { OStream, Cursor } from "@sofa-buffers/corelib";
+
+// generated by: sofabgen --lang typescript
+class Point {
+  x = 0;
+  y = 0;
+
+  marshal(os: OStream): void {
+    os.writeSigned(1, this.x);
+    os.writeSigned(2, this.y);
+  }
+
+  static decode(bytes: Uint8Array): Point {
+    return Point.decodeFrom(new Cursor(bytes));
+  }
+
+  static decodeFrom(c: Cursor): Point {
+    const p = new Point();
+    while (c.readHeader()) {
+      switch (c.id) {
+        case 1: p.x = Number(c.readSigned()); break;
+        case 2: p.y = Number(c.readSigned()); break;
+        // case 3: p.child = Child.decodeFrom(c); break;  // nested sequence
+        default: c.skip(c.wire); break;                   // forward-compatible
+      }
+    }
+    return p;
+  }
+}
+
+const p = new Point(); p.x = 3; p.y = 4;
+const os = new OStream(); p.marshal(os);
+const wire = os.bytes();
+const got = Point.decode(wire);   // got.x === 3, got.y === 4
+```
+
+## Memory handling
+
+Who owns the bytes:
+
+- **Encode (`OStream`).** In-memory `new OStream()` — the library allocates and
+  auto-grows an internal buffer (never throws `BUFFER_FULL`); `bytes()` returns a
+  **view** of the finished message, so `.slice()` it if it must outlive the next
+  write or grow. Streaming `new OStream(buf, offset?, flush?)` — writes into the
+  caller-owned buffer and never grows; when it fills it drains a view to the
+  `flush` sink (valid only during that callback) and, with no sink, throws
+  `BUFFER_FULL`.
+- **Decode (`decode()` / `Cursor` / `IStream`).** Input payload bytes are
+  zero-copy: string / blob chunks and `Cursor.readBlob` are `subarray` **views**
+  aliasing the input (or, for `IStream`, the chunk you fed). A visitor chunk is
+  valid **only during that callback**; a `Cursor` view lasts as long as the source
+  buffer lives. Scalars are delivered by value. Copy (`.slice()`) or decode
+  (`Cursor.readString` decodes for you) to retain a payload.
+
+### Decode limits
+
+For a schema whose `count` / `maxlen` bounds are omitted, the decoder otherwise
+accepts whatever count / length the received message claims. Pass an optional
+`DecodeLimits` object to cap that and protect a receiver from a hostile oversized
+field:
+
+```ts
+const limits = { maxArrayCount: 65536, maxStringLen: 1 << 20, maxBlobLen: 1 << 20 };
+decode(bytes, visitor, limits);       // one-shot push
+new Cursor(bytes, limits);            // pull
+new IStream(limits);                  // streaming
+```
+
+An over-limit array count or string / blob length is rejected at the field's
+header — **before** the array is sized or any payload is decoded or streamed to
+the visitor — by throwing `SofabError` with code
+`SofabErrorCode.LimitExceeded`. The decoder never clamps or truncates. Each limit
+is independent, and an omitted one means **no cap** (the default is today's
+unlimited behavior — the corelib invents no default). `LimitExceeded` is distinct
+from `INVALID_MSG`: exceeding a receiver-configured limit is policy, not a
+malformed message. Generated code supplies these values from the sofabgen config.
+
+## Feature flags
+
+None — the build always ships every wire type.
+
+## Build & test
+
+```bash
+npm ci
+npm run typecheck      # tsc --noEmit (strict)
+npm test               # vitest run: vectors, chunked feeding, cursor, errors, round-trips
+npm run coverage       # vitest run --coverage (v8)
+npm run build          # tsup -> ESM + CJS + IIFE + .d.ts in dist/
+npm run smoke          # cross-runtime smoke test of the built bundle
+```
+
+Tests live in `test/` as focused suites, including `vectors.test.ts` (encode +
+decode every shared conformance vector), `istream.chunked.test.ts` (every vector
+fed one byte at a time), `cursor.test.ts`, `errors.test.ts`, `ostream.test.ts`,
+`roundtrip.test.ts` and more. CI type-checks, tests and builds on Node 20 /
+24, smoke-tests the bundle on Node, Deno and Bun, and publishes coverage badges;
+a separate `docs.yml` deploys the TypeDoc API reference to GitHub Pages.
+
+## Benchmarks
+
+Two standalone tools mirror the other-language ports so implementations can be
+compared directly:
+
+```bash
+npm run perf              # per-op cost: code-cost figure plus throughput MB/s
+npm run bench             # throughput table (MB/s) for a u64 array and a mixed message
+npm run bench:callgrind   # machine-independent instructions/op under Valgrind
+```
+
+`perf` and `bench` encode the identical message as their counterparts in the
+other ports and print the same report layout. Since JS engines expose no portable
+cycle counter, `perf` uses CPU time/op as the code-cost proxy; `bench:callgrind`
+counts instructions/op under Valgrind for a fully machine-independent figure.
+Running the same tools under Node (V8) and Bun (JavaScriptCore) gives directly
+comparable numbers.