sonic-ws 2.2.0 → 2.4.0

package/README.md

@@ -15,6 +15,8 @@ It is designed for real-time applications such as games, dashboards, collaborati
 - Batch, validate, compress, and rate-limit packets with minimal boilerplate.
 - Reconnect with bounded state recovery and opt-in missed-packet replay.
 - Use validated RPC, server-side rooms, and pluggable scaling adapters.
+- Inspect, encode, decode, size, validate, and generate types from packet manifests with the CLI.
+- Drop replaceable updates under backpressure without silently dropping reliable messages.
 - Code is much more readable than every other socket library*
 
 <details>
@@ -72,7 +74,7 @@ Whether you're making a real-time game, a dashboard, a distributed system, or an
 
 You define the packets each side is allowed to send. SonicWS exchanges that schema during the connection handshake, assigns compact numeric packet IDs, and validates incoming data before calling your listeners.
 
-The TypeScript/python layer handles connections, packet definitions, middleware, and the public API. The Rust core handles the wire format:
+The TypeScript, Python, and native Rust packages handle connections, packet definitions, and their language-specific public APIs. The shared Rust core handles the wire format:
 
 - Primitive and object packet encoding/decoding
 - Packet validation and range checks
@@ -80,7 +82,7 @@ The TypeScript/python layer handles connections, packet definitions, middleware,
 - Object framing, batching, and raw DEFLATE compression
 - TypeScript/Python-side JSON conversion transported through reserved wire type 16 as raw bytes
 
-The same Rust implementation runs through WASM in Node, browsers, and Python. Python loads its packaged WASM core through `wasmtime`, while Node uses the packaged Node-target module. Protocol behavior therefore does not depend on the operating system or connected runtime.
+Node and browsers run the core through WebAssembly, and Python loads its packaged WASM core through `wasmtime`. The native Rust package links the core directly and adds a Tokio WebSocket client/server runtime. Protocol behavior therefore comes from the same codec on every supported runtime.
 
 Automatic browser-file serving at `/SonicWS/bundle.js` and
 `/SonicWS/bundle.wasm` is supported only by the Node.js server, where it can be
@@ -107,6 +109,10 @@ Python* (*unpublished currently):
 ```sh
 pip install sonic-ws
 ```
+Rust:
+```toml
+sonic-ws = { path = "projects/rust" }
+```
 
 ## BUILDING AND TESTING
 
@@ -116,7 +122,8 @@ From the repository root, use the build dispatcher:
 
 ```sh
 ./build.sh all       # Build every project
-./build.sh rust      # Rust core only
+./build.sh rust      # Shared core and native Rust runtime
+./build.sh core      # Shared codec core only
 ./build.sh ts        # Complete Node/browser package
 ./build.sh py        # Python wheel and sdist
 ./build.sh test      # Run all test suites
@@ -135,7 +142,7 @@ npm run test_node   # Node end-to-end packet tests
 npm run test_web    # Headless browser/WASM end-to-end tests
 ```
 
-The workspace is split into `projects/core`, `projects/ts`, and `projects/py`.
+The workspace is split into `projects/core`, `projects/rust`, `projects/ts`, and `projects/py`.
 Each project owns its source, tests, and packaging configuration. Shared browser
 artifacts remain at `bundled/bundle.js` and `bundled/bundle.wasm`.
 
@@ -149,6 +156,13 @@ Full API documentation:
 
 - [TypeScript / Node / browser](docs/ts/README.md)
 - [Python](docs/py/README.md)
+- [Native Rust](projects/rust/README.md)
+- [Protocol version 25](docs/protocol.md)
+- [Production defaults](docs/defaults.md)
+- [Authentication and recovery](docs/authentication.md)
+- [Inspector, generated types, conformance, and benchmarks](docs/tooling.md)
+- [Backpressure and delivery](docs/backpressure.md)
+- [Runnable examples](examples/README.md)
 
 Packet schemas can now map the existing single-type wire format directly to application objects:
 
@@ -167,6 +181,12 @@ This remains a homogeneous `VARINT` packet. Schema mapping, row-major `autoFlatt
 
 Clients can opt into capped exponential-backoff reconnect. Packets marked `replay: true` are retained in a bounded per-session buffer; successful recovery also restores server-side `state` and room membership. RPC request payloads use ordinary packet definitions, so validation and compact encoding still apply. Room broadcasts work locally and can be forwarded across processes through an adapter. Long-polling fallback is intentionally outside SonicWS's scope.
 
+## WHERE SONICWS FITS
+
+SonicWS is strongest for schema-known realtime traffic: numeric movement updates, repeated entity snapshots, compact enums, bounded RPC payloads, and other messages where validation and wire size matter. Positional sends are available as the lowest-allocation hot path, while schema mapping trades some CPU for cleaner application objects.
+
+Arbitrary chat strings and already-compressed or highly dynamic JSON may see little or no size improvement over ordinary JSON WebSockets. SonicWS should be selected for its explicit packet contract, validation, lifecycle behavior, and compact structured traffic—not because every possible payload encodes faster or smaller than JSON.
+
 ## KNOWN ISSUES
 
 
@@ -175,6 +195,10 @@ Clients can opt into capped exponential-backoff reconnect. Packets marked `repla
 - Publish the Rust core as a standalone, documented crate
 - Add first-class Go bindings
 
+## API AND PROTOCOL STABILITY
+
+SonicWS uses semantic package versions and an explicit wire-protocol version. Incompatible wire changes require a protocol-version change and fail during handshake instead of silently decoding with different semantics. Public APIs may still evolve between major package versions; pin the package version for production deployments and run the shared conformance corpus when upgrading.
+
 ## LICENSE
 This project is source-available.
 

package/bin/sonicws.mjs

@@ -0,0 +1,159 @@
+#!/usr/bin/env node
+
+import {
+	readFile,
+	writeFile
+} from "node:fs/promises";
+import {
+	deflateRawSync
+} from "node:zlib";
+import {
+	LoadPacketManifest,
+	ValidatePacketSchema
+} from "../dist/index.js";
+import {
+	PacketHolder
+} from "../dist/ws/util/packets/PacketHolder.js";
+import {
+	processPacket
+} from "../dist/ws/util/packets/PacketUtils.js";
+
+function usage(message) {
+	if (message) console.error(message);
+	console.error(`Usage:
+  sonicws inspect <manifest.swsm>
+  sonicws validate <manifest.swsm>
+  sonicws types <manifest.swsm> [output.d.ts]
+  sonicws encode <manifest.swsm> <client|server> <tag> <json>
+  sonicws decode <manifest.swsm> <client|server> <tag> <hex>
+  sonicws size   <manifest.swsm> <client|server> <tag> <json>`);
+	process.exit(message ? 2 : 0);
+}
+
+const [, , command, filename, direction, tag, input] = process.argv;
+if (!command || command === "help" || command === "--help") usage();
+if (!filename) usage("A packet manifest is required");
+const manifest = LoadPacketManifest(new Uint8Array(await readFile(filename)));
+const select = name => name === "client" ? manifest.clientPackets : name === "server" ? manifest.serverPackets : usage("Direction must be client or server");
+
+if (command === "inspect") {
+	const summarize = packets => packets.map((packet, index) => ({
+		key: index + 1,
+		tag: packet.tag,
+		type: packet.type,
+		schema: packet.fields,
+		dataMin: packet.dataMin,
+		dataMax: packet.dataMax,
+		batchingMs: packet.dataBatching,
+		compressed: packet.gzipCompression,
+		replay: packet.replay,
+	}));
+	console.log(JSON.stringify({
+		protocol: 24,
+		clientPackets: summarize(manifest.clientPackets),
+		serverPackets: summarize(manifest.serverPackets)
+	}, null, 2));
+	process.exit(0);
+}
+
+if (command === "validate") {
+	const client = ValidatePacketSchema(manifest.clientPackets, {
+		direction: "client",
+		warnUnbounded: true
+	});
+	const server = ValidatePacketSchema(manifest.serverPackets, {
+		direction: "server"
+	});
+	console.log(JSON.stringify({
+		client,
+		server,
+		valid: client.errors.length + server.errors.length === 0
+	}, null, 2));
+	process.exit(client.errors.length + server.errors.length ? 1 : 0);
+}
+
+if (command === "types") {
+	const scalar = packetType => ({
+		0: "undefined",
+		1: "Uint8Array",
+		2: "string",
+		3: "string",
+		4: "unknown",
+		5: "number",
+		6: "number",
+		7: "number",
+		8: "number",
+		9: "number",
+		10: "number",
+		11: "number",
+		12: "number",
+		13: "number",
+		14: "boolean",
+		16: "unknown",
+		17: "string",
+	})[packetType] ?? "unknown";
+	const payload = (packet, packets, group = true) => {
+		if (group && packet.isParent) {
+			const variants = packets.filter(child => child.parent === packet.tag && child.variant)
+				.map(child => `{ variant: ${JSON.stringify(child.variant)}; payload: ${payload(child, packets, false)} }`);
+			return [`{ variant: ""; payload: undefined }`, ...variants].join(" | ");
+		}
+		const types = Array.isArray(packet.type) ? packet.type : [packet.type];
+		if (packet.fields) {
+			const record = `{ ${packet.fields.map((field, index) => `${JSON.stringify(field)}: ${scalar(types[Math.min(index, types.length - 1)])}`).join("; ")} }`;
+			return packet.autoFlatten ? `${record}[]` : record;
+		}
+		if (packet.type === 0) return "undefined";
+		const value = scalar(types[0]);
+		return packet.dataMax === 1 ? value : `${value}[]`;
+	};
+	const table = (name, packets) => [
+		`export interface ${name} {`,
+		...packets.map(packet => `  ${JSON.stringify(packet.tag)}: ${payload(packet, packets)};`),
+		"}",
+	].join("\n");
+	const generated = `// Generated by sonicws types; protocol 25.\n${table("ClientPackets", manifest.clientPackets)}\n\n${table("ServerPackets", manifest.serverPackets)}\n\ntype PacketArguments<T> = T extends undefined ? [] : [value: T];\nexport interface TypedSonicWS {\n  send<K extends keyof ClientPackets>(tag: K, ...value: PacketArguments<ClientPackets[K]>): Promise<void>;\n  sendSafe<K extends keyof ClientPackets>(tag: K, ...value: PacketArguments<ClientPackets[K]>): Promise<boolean>;\n  on<K extends keyof ServerPackets>(tag: K, listener: (value: ServerPackets[K]) => void): void;\n}\n`;
+	if (direction) await writeFile(direction, generated);
+	else process.stdout.write(generated);
+	process.exit(0);
+}
+
+if (!direction || !tag || input === undefined) usage(`Missing arguments for ${command}`);
+const packets = select(direction);
+const holder = new PacketHolder(packets);
+const packet = holder.getPacket(tag);
+
+if (command === "decode") {
+	if (!/^(?:[0-9a-fA-F]{2})*$/.test(input)) usage("Encoded data must be even-length hexadecimal");
+	let bytes = Uint8Array.from(Buffer.from(input, "hex"));
+	if (bytes[0] === holder.getKey(tag)) bytes = bytes.slice(1);
+	const decoded = await packet.listen(bytes, null);
+	if (typeof decoded === "string") throw new Error(decoded);
+	console.log(JSON.stringify(decoded[0], null, 2));
+	process.exit(0);
+}
+
+let value;
+try {
+	value = JSON.parse(input);
+} catch {
+	usage("Value must be valid JSON");
+}
+const values = packet.fields || packet.autoFlatten ? [value] : Array.isArray(value) ? value : [value];
+const [key, encoded] = await processPacket(holder, tag, values, [false, [], undefined], 0);
+const frame = Buffer.from([key, ...encoded]);
+if (command === "encode") {
+	console.log(frame.toString("hex"));
+} else if (command === "size") {
+	const json = Buffer.from(JSON.stringify({
+		event: tag,
+		data: value
+	}));
+	console.log(JSON.stringify({
+		tag,
+		sonicBytes: frame.length,
+		rawDeflateBytes: 1 + deflateRawSync(encoded).length,
+		jsonBytes: json.length,
+		reductionPercent: 100 * (1 - frame.length / json.length)
+	}, null, 2));
+} else usage(`Unknown command: ${command}`);