@pbwire/pbwire 0.0.0-stage → 0.1.0

package/README.md

@@ -1,3 +1,205 @@
-# Temporary Holding Version
+# pbwire Node.js binding
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A `FileDescriptorSet` (`protoc --descriptor_set_out`) becomes a schema at
+runtime; no code generation, no `protoc` plugin, no npm dependencies. The
+runtime is `runtime/addon.c`, an N-API addon in C over `libpbwire.a`; the
+`Schema`/`Message` surface is `lib/pbwire.js`.
+
+```js
+const { Schema } = require('./lib/pbwire');
+
+const schema = Schema.fromFile('tests/data/demo.desc');
+const person = schema.message('demo.Person');
+
+const wire = person.encode({name: 'alice', id: 7, inner: {x: 42}});
+// or, by hand:
+const msg = person.newMessage();
+msg.set('name', 'alice').set('inner', {x: 42});
+msg.encode();          // Buffer
+msg.free();
+
+const back = person.decode(wire);
+back.get('name');      // 'alice'
+back.get('inner').x;   // 42
+back.encode().equals(wire);   // true: byte-exact re-encode
+back.free();
+```
+
+## How it works
+
+A message is one of two things, and both live behind the same API:
+
+- **A built message** (`newMessage()`/`set()`) is a `pbw_message` value tree in
+  C. Field writes go straight into it, and `encode()` runs the core encoder
+  into a per-type reusable buffer, so re-encoding a message never crosses the
+  boundary again.
+- **A decoded message** (`decode()`) is the bytes it came from plus one 12-byte
+  slice per wire field: decode validates the payload once while recording those
+  slices, `get()` materializes a single field on demand, and `encode()` hands
+  the same bytes back. A round trip is therefore byte-exact — unknown fields,
+  field order and non-canonical encodings included — and nothing is built for
+  fields that are never read.
+
+The wire form holds no native state: the bytes are a JS `Buffer` and the slices
+a `Uint32Array`, both owned by the message object. That is deliberate. Attaching
+a finalizer to native memory costs N-API ~250 ns per object, which is more than
+the entire decode of a small message, so only the built form (which has no
+JS-visible storage) pays for one.
+
+Writing to a decoded message promotes it to a tree (one `pbw_decode` of the
+retained bytes), so `set()` works on either form and only callers that actually
+edit a parsed message pay for the tree.
+
+Field facts (number, type, wire type, label, oneof, closed enum, sub-message)
+are resolved once per descriptor into a cached table with a name hash and a
+number index, so `get()`/`set()` and the wire scan never walk the descriptor.
+
+## Build and run
+
+Needs Node (headers ship with it) and a C compiler. No `npm install`, no
+`node-gyp`.
+
+```sh
+bindings/node/build.sh          # build the addon, then run the tests
+bindings/node/build.sh build    # build only
+node bindings/node/test/test_pbwire.js    # the tests alone (node:test)
+node bindings/node/examples/roundtrip.js  # the example
+```
+
+`build.sh` builds `build/libpbwire.a` first when it is missing (the root `make`
+default goal is the shared library, so it asks for the archive by name), then
+compiles the addon with clang against `$(node -p ...)/include/node` and links
+the static core. Set `NODE_INCLUDE` if a packaging splits the Node headers away
+from the binary. Output: `bindings/node/build/pbwire_node.node`.
+
+## API
+
+`Schema`
+| member | meaning |
+|---|---|
+| `Schema.fromBytes(buffer)` | load a `FileDescriptorSet`; throws on malformed input |
+| `Schema.fromFile(path)` | same, from a file |
+| `schema.message(name)` | the message type `name` (`"pkg.Msg"`); throws when it is not a message |
+
+`MessageType`
+| member | meaning |
+|---|---|
+| `type.name` / `type.schema` | dotted full name and owning schema |
+| `type.newMessage()` | empty `Message` (the mutable tree form) |
+| `type.encode(object \| Message)` | encode a plain object or an existing `Message` |
+| `type.decode(buffer)` | decode wire bytes into a `Message`; throws on malformed input |
+
+`Message`
+| member | meaning |
+|---|---|
+| `msg.get(name)` | field value, `undefined` when the field is absent; throws for an unknown name |
+| `msg.set(name, value)` | store a value, replacing what was there; `undefined`/`null` clears the field |
+| `msg.encode()` | wire bytes as a `Buffer` |
+| `msg.free()` | release early; a second call is a no-op, any other call afterwards throws |
+
+Values map per field type:
+
+| field type | JS value |
+|---|---|
+| `int32`, `uint32`, `sint32`, `fixed32`, `sfixed32`, `enum` | `number` |
+| `int64`, `uint64`, `sint64`, `fixed64`, `sfixed64` | `BigInt` (a `number` is accepted on `set`) |
+| `float`, `double` | `number` |
+| `bool` | `boolean` |
+| `string` | `string` |
+| `bytes` | `Buffer` |
+| message | plain `object` (nested `get`/`set` by field name) |
+| repeated | `array` |
+
+Presence is explicit, as in the C core: a field set to its zero value is
+written to the wire instead of being skipped the way a generated proto3
+serializer would skip it. A message field is replaced on `set`, not merged, and
+a repeated write replaces the whole list. Reading follows the wire: a repeated
+field that is not on the wire reads `undefined`, a oneof reads only its last
+member, and a singular sub-message that appears twice merges (the payloads
+concatenate, which is what parsing them as one message means).
+
+Two proto2 behaviours are worth naming:
+
+- **Closed enums.** A number the enum declaration does not list is not a value
+  of the field (the reference parsers move it to the unknown fields), so `get()`
+  skips it, and `encode()` passes the original bytes on untouched.
+- **`required` fields.** Whether a required field is present is a property of
+  the *merged* message, which is what the tree decoder checks after parsing. A
+  type that reaches a `required` field therefore decodes through the tree
+  (`decodeEager`) instead of the wire view; everything else decodes lazily.
+
+## Tests
+
+`test/test_pbwire.js` covers descriptor-set loading, every field kind, oneof,
+packed and unpacked repetition, groups, closed enums, required fields,
+byte-exact re-encode (including unknown field pass-through), editing a decoded
+message, absent fields, schema/wire errors, nesting caps, and double-free
+safety. The last two tests are differential: one decodes a corpus of truncated
+and bit-flipped payloads, asserts that a decoded message is always its own
+bytes, and compares every field against the tree decoder's answer on the same
+bytes (so the two materializers cannot drift apart); the other feeds the addon a
+slice array it did not produce and expects an error rather than a read past the
+buffer.
+
+The fixtures are `tests/data/{demo,shapes,legacy}.desc`, generated from the
+`.proto` files next to them:
+
+```sh
+protoc -Ibindings/node/tests/data \
+    --descriptor_set_out=bindings/node/tests/data/demo.desc \
+    bindings/node/tests/data/demo.proto
+```
+
+## Benchmark
+
+`benchmarks/bench.sh` generates `matrix3_pb.js` with `protoc-gen-js`, installs
+`google-protobuf` (the jspb runtime), `protoc-gen-js` and [protobufjs](https://www.npmjs.com/package/protobufjs) (the
+fastest widely used JS runtime) into `benchmarks/node_modules` on first run,
+and prints the four standard tiers from [`BENCHMARKS.md`](../../BENCHMARKS.md)
+for `pbwire-node`, jspb and protobufjs side by side:
+
+```sh
+./benchmarks/bench.sh      # or: make node-bench
+```
+
+| payload | op | pbwire-node (MB/s, x jspb) | jspb (MB/s) | protobufjs (MB/s) |
+|---|---|---|---|---|
+| 136 B | encode | 608.5 (10.9x) | 56.0 | 243.3 |
+| 136 B | decode | 375.3 (3.1x) | 119.5 | 395.7 |
+| 174 B | encode | 568.6 (9.0x) | 62.9 | 300.8 |
+| 174 B | decode | 431.9 (3.8x) | 112.7 | 391.1 |
+| 2469 B | encode | 1201.6 (20.8x) | 57.9 | 445.6 |
+| 2469 B | decode | 1122.9 (14.0x) | 80.3 | 375.4 |
+| 74913 B | encode | 1350.2 (22.3x) | 60.6 | 496.5 |
+| 74913 B | decode | 1655.0 (20.0x) | 82.9 | 402.0 |
+
+jspb's throughput does not move with payload size: its writer resolves a
+code/type table per field and hands every repeated element to a closure, where
+the C core writes one tag byte and one inline value, or memcpys an already
+validated slice. Two boundary costs decide the small tiers: N-API's own
+`Buffer` creation costs ~200 ns, so `encode()` encodes into the type's reusable
+buffer and copies into a JS-allocated (pooled) `Buffer` of the exact size, and
+decode copies the payload once with `Buffer.from` instead of paying for a
+per-message finalizer.
+
+## Deliberate skips
+
+- **No protoc plugin / code generation.** The schema is always loaded at run
+  time, like the Python binding: a `FileDescriptorSet` is the only schema
+  source, and there is no generated layout to keep in sync.
+- **No conformance testee.** The official suite is covered through the C core
+  and `conformance/`, not through this binding. The differential test above is
+  what keeps the addon's wire reader honest against the core's.
+
+Also out of scope here: JSON/text format (`pbw_json_print`/`pbw_text_print`),
+field presence introspection (`pbw_message_field_count`), and delimited framing.
+Each is one addon export away when a consumer needs it.
+
+## Typed models
+
+`protoc-gen-pbwire-typed` (in `../typed`) generates `matrix3.d.ts` interfaces
+(for TypeScript) and a JSDoc/`@ts-check` `.js` with the same shapes plus the enum
+constants, describing exactly the plain objects `MessageType.encode()` accepts
+and `Message.get()` returns. The committed output is `tests/gen/` and
+`examples/gen/`; the typed test type-checks with `tsc --strict --checkJs`.
+See `../typed/README.md`.

package/build.sh

@@ -0,0 +1,49 @@
+#!/bin/sh
+# Build and check the Node.js binding: compile the N-API addon against
+# build/libpbwire.a (building it first when missing), then run the self-check.
+# clang on purpose: no node-gyp, no npm, no dependencies.
+#
+#   bindings/node/build.sh          build the addon, then run the tests
+#   bindings/node/build.sh build    build only
+set -eu
+
+here=$(cd "$(dirname "$0")" && pwd)
+root=$(cd "$here/../.." && pwd)
+cc=${CC:-clang}
+
+if [ ! -f "$root/build/libpbwire.a" ]; then
+    # The default make goal is the shared library; the addon links the static
+    # archive, so name it explicitly.
+    make -C "$root" build/libpbwire.a
+fi
+
+# Node ships its headers next to the binary: <prefix>/bin/node and
+# <prefix>/include/node. Override with NODE_INCLUDE if a packaging split them.
+: "${NODE_INCLUDE:=$(node -p 'require("path").join(require("path").dirname(require("path").dirname(process.execPath)), "include", "node")')}"
+if [ ! -f "$NODE_INCLUDE/node_api.h" ]; then
+    echo "build.sh: node_api.h not found in $NODE_INCLUDE (set NODE_INCLUDE)" >&2
+    exit 1
+fi
+
+# A .node module is a loadable module, not a library: resolve the node symbols
+# at load time on macOS, at link time from libc elsewhere.
+case $(uname -s) in
+Darwin) link="-dynamiclib -undefined dynamic_lookup" ;;
+*) link="-shared" ;;
+esac
+
+mkdir -p "$here/build"
+# shellcheck disable=SC2086 # $link is a flag list
+$cc -std=c99 -O2 -fPIC -Wall -Wextra \
+    -I"$NODE_INCLUDE" -I"$root/include" \
+    $link -o "$here/build/pbwire_node.node" \
+    "$here/runtime/addon.c" "$root/build/libpbwire.a"
+
+echo "built $here/build/pbwire_node.node"
+
+if [ "${1:-all}" = build ]; then
+    exit 0
+fi
+
+# --expose-gc so the finalizer test can force the collections it checks.
+exec node --expose-gc --test "$here/test/test_pbwire.js" "$here/tests/typed_test.js"

package/lib/pbwire.js

@@ -0,0 +1,194 @@
+'use strict';
+
+/* Node.js binding: a FileDescriptorSet becomes a schema at runtime (no code
+ * generation, no protoc plugin), and messages are read and written by name.
+ *
+ *   const { Schema } = require('./lib/pbwire');
+ *   const schema = Schema.fromFile('demo.desc');
+ *   const person = schema.message('demo.Person');
+ *
+ *   const msg = person.newMessage();
+ *   msg.set('name', 'alice');
+ *   msg.set('inner', { x: 1, tag: 'kid' });
+ *   const wire = msg.encode();          // Buffer
+ *   msg.free();
+ *
+ *   const back = person.decode(wire);   // Message
+ *   back.get('inner').x;                // 1
+ *   back.encode().equals(wire);         // true
+ *
+ * A message is one of two things:
+ *
+ *   A built message (newMessage()/set()) is a value tree in C. Writing to it
+ *   is cheap after the first field, and encode() runs the core encoder into a
+ *   reusable buffer.
+ *
+ *   A decoded message is the bytes plus one slice per wire field: decode()
+ *   validates the payload once, a value is materialized only when it is read,
+ *   and encode() hands the bytes back, so a round trip is byte-exact (unknown
+ *   fields, field order and non-canonical encodings included).
+ *
+ * Writing to a decoded message turns it into a tree (one decode of the retained
+ * bytes), so the API is the same either way.
+ *
+ * Values map per field type:
+ *
+ *   int32/uint32/sint32/fixed32/sfixed32/enum   number
+ *   int64/uint64/sint64/fixed64/sfixed64        BigInt
+ *   float/double                                number
+ *   bool                                        boolean
+ *   string                                      string
+ *   bytes                                       Buffer
+ *   message / repeated                          object / array
+ *
+ * There is no int64 precision loss: 64-bit fields are BigInt, and a JS number
+ * is accepted when setting (rounded the way Number->int64 rounds).
+ */
+
+const fs = require('node:fs');
+const path = require('node:path');
+
+const addon = require(path.join(__dirname, '..', 'build', 'pbwire_node.node'));
+
+class Schema {
+  constructor(handle) {
+    this._handle = handle;
+  }
+
+  /** Load a FileDescriptorSet (protoc --descriptor_set_out, buf, ...). */
+  static fromBytes(bytes) {
+    return new Schema(addon.loadDescriptorSet(bytes));
+  }
+
+  static fromFile(file) {
+    return Schema.fromBytes(fs.readFileSync(file));
+  }
+
+  /** A message type of this schema; throws when the name is not a message. */
+  message(name) {
+    const handle = addon.descriptor(this._handle, name);
+    if (handle === null) {
+      throw new Error(`pbwire: ${name} is not a message in this schema`);
+    }
+    return new MessageType(this, name, handle);
+  }
+}
+
+class MessageType {
+  constructor(schema, name, handle) {
+    this.schema = schema;
+    this.name = name;
+    this._handle = handle;
+    // A proto2 `required` field has to hold in the merged message, which is
+    // what the tree decoder checks; such a type decodes eagerly.
+    this._eager = addon.isEager(handle);
+  }
+
+  /** An empty message of this type. */
+  newMessage() {
+    const message = new Message(this);
+    message._handle = addon.messageNew(this._handle);
+    return message;
+  }
+
+  /** Encode a plain object or a Message built by this type. */
+  encode(value) {
+    if (value instanceof Message) {
+      if (value.type !== this) {
+        throw new TypeError(`pbwire: ${value.type.name} is not ${this.name}`);
+      }
+      return value.encode();
+    }
+    if (value === null || typeof value !== 'object') {
+      throw new TypeError('pbwire: encode expects an object or a Message');
+    }
+    // The addon builds the tree, encodes it into the type's scratch buffer and
+    // frees the tree; only the bytes cross back.
+    return this._copy(addon.encodeObject(this._handle, value));
+  }
+
+  /** Decode wire bytes into a Message; throws on malformed input. */
+  decode(bytes) {
+    if (!Buffer.isBuffer(bytes)) {
+      throw new TypeError('pbwire: decode expects a Buffer');
+    }
+    // A private copy: the caller may reuse its Buffer, and a decoded message
+    // keeps the bytes it came from.
+    const wire = Buffer.from(bytes);
+    const message = new Message(this);
+    if (this._eager) {
+      message._handle = addon.decodeEager(this._handle, wire);
+    } else {
+      message._wire = wire;
+      message._slices = addon.decode(this._handle, wire);
+    }
+    return message;
+  }
+
+  /** Copies the scratch an encode just filled into a pooled Buffer. */
+  _copy(len) {
+    const out = Buffer.allocUnsafe(len);
+    addon.copyEncoded(this._handle, out);
+    return out;
+  }
+}
+
+class Message {
+  constructor(type) {
+    this.type = type;
+    this._handle = null; // tree form (newMessage/set)
+    this._wire = null; // wire form (decode): the bytes
+    this._slices = null; // wire form: (field index, offset, length) triples
+  }
+
+  /** Field value by name, or undefined when the field is absent. */
+  get(name) {
+    this._live();
+    if (this._wire !== null) {
+      return addon.wireGet(this.type._handle, this._wire, this._slices, name);
+    }
+    return addon.get(this._handle, name);
+  }
+
+  /** Store a field value by name; `undefined`/`null` clears the field. */
+  set(name, value) {
+    this._live();
+    if (this._wire !== null) {
+      // Writing needs a mutable tree: decode the retained bytes once.
+      this._handle = addon.promote(this.type._handle, this._wire);
+      this._wire = null;
+      this._slices = null;
+    }
+    addon.set(this._handle, name, value);
+    return this;
+  }
+
+  /** Wire bytes of this message. */
+  encode() {
+    this._live();
+    if (this._wire !== null) {
+      // Untouched since decode: the bytes are the encoding.
+      return Buffer.from(this._wire);
+    }
+    return this.type._copy(addon.encode(this._handle));
+  }
+
+  /** Release the tree early; calling it twice is a no-op. A decoded message
+   * only drops its bytes, which the garbage collector would do anyway. */
+  free() {
+    if (this._handle !== null) {
+      addon.free(this._handle);
+      this._handle = null;
+    }
+    this._wire = null;
+    this._slices = null;
+  }
+
+  _live() {
+    if (this._handle === null && this._wire === null) {
+      throw new TypeError('pbwire: message was freed');
+    }
+  }
+}
+
+module.exports = { Schema, MessageType, Message, abiVersion: addon.abiVersion };

package/package.json

@@ -1,6 +1,21 @@
 {
   "name": "@pbwire/pbwire",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Dynamic protobuf over the pbwire C ABI, schema at runtime, no generated code",
+  "license": "Apache-2.0",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/jinganix/pbwire.git",
+    "directory": "bindings/node"
+  },
+  "author": "jinganix",
+  "main": "lib/pbwire.js",
+  "files": ["lib/", "runtime/", "build.sh", "README.md"],
+  "scripts": {
+    "install": "./build.sh build",
+    "test": "./build.sh"
+  },
+  "engines": {
+    "node": ">=20"
+  }
+}