@thi.ng/wasm-api 0.5.0 → 0.8.0

package/CHANGELOG.md

@@ -1,6 +1,6 @@
 # Change Log
 
-- **Last updated**: 2022-08-08T22:36:17Z
+- **Last updated**: 2022-08-16T16:05:07Z
 - **Generator**: [thi.ng/monopub](https://thi.ng/monopub)
 
 All notable changes to this project will be documented in this file.
@@ -9,6 +9,65 @@ See [Conventional Commits](https://conventionalcommits.org/) for commit guidelin
 **Note:** Unlisted _patch_ versions only involve non-code or otherwise excluded changes
 and/or version bumps of transitive dependencies.
 
+## [0.8.0](https://github.com/thi-ng/umbrella/tree/@thi.ng/wasm-api@0.8.0) (2022-08-16)
+
+#### 🚀 Features
+
+- add preliminary string handling support ([3da4efe](https://github.com/thi-ng/umbrella/commit/3da4efe))
+  - update/rename IWasmMemoryAccess (add string getter/setter)
+  - update StructField.type (add `string`)
+  - add CodeGenOpts.stringType option
+  - update codegen fns
+  - update TS & Zig codegen impls
+
+## [0.7.0](https://github.com/thi-ng/umbrella/tree/@thi.ng/wasm-api@0.7.0) (2022-08-15)
+
+#### 🚀 Features
+
+- add CLI wrapper for codegens ([683a560](https://github.com/thi-ng/umbrella/commit/683a560))
+- update TSOpts & TS codegen ([4f6bbbf](https://github.com/thi-ng/umbrella/commit/4f6bbbf))
+  - add `uppercaseEnum` option to force UC enum IDs
+- update helper predicates ([65b23d4](https://github.com/thi-ng/umbrella/commit/65b23d4))
+
+#### 🩹 Bug fixes
+
+- allow signed ints for enum tags ([78d0822](https://github.com/thi-ng/umbrella/commit/78d0822))
+- correct TS __mapArray codegen ([289b137](https://github.com/thi-ng/umbrella/commit/289b137))
+
+## [0.6.0](https://github.com/thi-ng/umbrella/tree/@thi.ng/wasm-api@0.6.0) (2022-08-15)
+
+#### 🚀 Features
+
+- add C11 header/include file, update WasmBridge ([a67dc00](https://github.com/thi-ng/umbrella/commit/a67dc00))
+  - migrate headers/includes to /include
+  - rename "core" import section => "wasmapi"
+  - rename WasmBridge.core => WasmBridge.api
+  - update pkg file
+- update codegens, add opts, fix alignments ([5c1fec5](https://github.com/thi-ng/umbrella/commit/5c1fec5))
+  - add global CodeGenOpts
+  - update generateTypes() to consider new opts
+  - add global USIZE type (for pointer sizes & codegens)
+  - add options for Zig codegen (extra debug helpers)
+  - simplify TS codegen
+  - fix sizeOf() for struct fields
+  - make prepareType() idempotent
+- add bindings code generator framework ([17ee06f](https://github.com/thi-ng/umbrella/commit/17ee06f))
+  - add/update deps
+  - add preliminary codegens for Zig & TS
+  - add supporting types & utils
+  - add generateTypes() codegen facade fn
+- update allocate/free() fns, update Zig core API ([8a55989](https://github.com/thi-ng/umbrella/commit/8a55989))
+  - add _wasm_free() Zig impl
+  - add printFmt() Zig fn
+  - update WasmBridge.allocate() (add clear option)
+  - update WasmBridge.free()
+  - ensure memory in WasmBridge.getString()
+  - add/update docstrings
+
+#### ♻️ Refactoring
+
+- extract WasmMemViews interface, update test WASM ([4c73e65](https://github.com/thi-ng/umbrella/commit/4c73e65))
+
 ## [0.5.0](https://github.com/thi-ng/umbrella/tree/@thi.ng/wasm-api@0.5.0) (2022-08-08)
 
 #### 🚀 Features

package/README.md

@@ -12,30 +12,53 @@ This project is part of the
 - [About](#about)
   - [Custom API modules](#custom-api-modules)
   - [Object indices & handles](#object-indices--handles)
+  - [Data bindings & code generators](#data-bindings--code-generators)
+    - [CLI generator](#cli-generator)
+    - [Data type definitions](#data-type-definitions)
+    - [Example usage](#example-usage)
+  - [String handling](#string-handling)
   - [Status](#status)
 - [Installation](#installation)
 - [Dependencies](#dependencies)
 - [API](#api)
+- [Basic usage example](#basic-usage-example)
+  - [Zig version](#zig-version)
+  - [C11 version](#c11-version)
 - [Authors](#authors)
 - [License](#license)
 
 ## About
 
-Modular, extensible API bridge and generic glue code between JS & WebAssembly.
+Generic, modular, extensible API bridge, glue code and bindings code generator for hybrid JS & WebAssembly projects.
 
-This package provides a small, generic and modular
-[`WasmBridge`](https://docs.thi.ng/umbrella/wasm-api/classes/WasmBridge.html)
-class as interop basis and a much reduced boilerplate for hybrid JS/WebAssembly
-applications. At the moment only a minimal core API is provided (i.e. for debug
-output, string, pointer, typed array accessors [8/16/32/64 bit (u)ints, 32/64
-bit floats]), but in the future we aim to also supply support modules for DOM
-manipulation, WebGL, WebGPU, WebAudio etc.
+This package provides a the following:
 
-In general, all languages with a WebAssembly target are supported, however
-currently only bindings for [Zig](https://ziglang.org) are included.
+1. A small, generic and modular
+[`WasmBridge`](https://docs.thi.ng/umbrella/wasm-api/classes/WasmBridge.html)
+class as interop basis and much reduced boilerplate for hybrid JS/WebAssembly
+applications.
+2. A minimal core API for debug output, string, pointer, typed array accessors
+for 8/16/32/64 bit (u)ints and 32/64 bit floats. In the future we aim to also
+supply support modules for DOM manipulation, WebGL, WebGPU, WebAudio etc.
+3. [Include files for C11/C++ and
+   Zig](https://github.com/thi-ng/umbrella/tree/develop/packages/wasm-api/include),
+   defining WASM imports of the JS [core
+   API](https://docs.thi.ng/umbrella/wasm-api/interfaces/CoreAPI.html) defined
+   by this package
+4. Extensible shared datatype code generators for (currently)
+   [Zig](https://ziglang.org) & TypeScript. The latter also generates fully type
+   checked memory-mapped accessors of WASM-side data. In general, all languages
+with a WebAssembly target are supported, however currently only bindings for
+these few langs are included.
+5. [CLI frontend/utility](#cli-generator) to invoke the code generator(s)
 
 ### Custom API modules
 
+The [`WasmBridge`](https://docs.thi.ng/umbrella/wasm-api/classes/WasmBridge.html) is extensible via custom defined API modules. Such API extensions
+will consist of a collection of JS/TS functions & variables, their related
+counterparts (import definitions) for the WASM target and (optionally) some
+shared data types ([bindings for which _can_ be generated by this package too](#data-bindings--code-generators)).
+
 On the JS side, custom API modules can be easily integrated via the [`IWasmAPI`
 interface](https://docs.thi.ng/umbrella/wasm-api/interfaces/IWasmAPI.html). The
 following example provides a brief overview:
@@ -113,12 +136,12 @@ export fn test_randomVec2() void {
 
 Since only numeric values can be exchanged between the WASM module and the JS
 host, any JS native objects the WASM side might want to be working with must be
-managed in JS. For this purpose the [`ObjectIndex`
+managed manually in JS. For this purpose the [`ObjectIndex`
 class](https://docs.thi.ng/umbrella/wasm-api/classes/ObjectIndex.html) can be
 used by API modules to handle ID generation (incl. recycling, using
 [@thi.ng/idgen](https://github.com/thi-ng/umbrella/tree/develop/packages/idgen))
-& indexing of different types of JS objects/values. Only the numeric IDs will
-then need to be exchanged with the WASM module...
+and the indexing of different types of JS objects/values. Only the numeric IDs
+(handles) will then need to be exchanged with the WASM module...
 
 ```ts
 import { ObjectIndex } from "@thi.ng/wasm-api";
@@ -156,6 +179,278 @@ canvases.delete(0);
 // true
 ```
 
+### Data bindings & code generators
+
+The package provides an extensible codegeneration framework to simplify the
+bilateral design & exchange of data structures shared between the WASM & JS host
+env. Currently, code generators for TypeScript & Zig are supplied (more are
+planned). A CLI wrapper is worked on too.
+
+#### CLI generator
+
+The package includes a [small CLI
+wrapper](https://github.com/thi-ng/umbrella/blob/develop/packages/wasm-api/src/cli.ts)
+to invoke the codegenerator(s) from JSON type definitions and to write the
+generated source code(s) to different files:
+
+```text
+$ npx @thi.ng/wasm-api
+
+ █ █   █           │
+██ █               │
+ █ █ █ █   █ █ █ █ │ @thi.ng/wasm-api 0.6.0
+ █ █ █ █ █ █ █ █ █ │ Multi-language data bindings code generator
+                 █ │
+               █ █ │
+
+usage: wasm-api [OPTS] JSON-INPUT-FILE(S) ...
+       wasm-api --help
+
+Flags:
+
+-d, --debug                     enable debug output
+--dry-run                       enable dry run (don't overwrite files)
+
+Main:
+
+-c FILE, --config FILE          JSON config file with codegen options
+-l ID[,..], --lang ID[,..]      [multiple] target language: "ts", "zig" (default: ["ts","zig"])
+-o FILE, --out FILE             [multiple] output file path
+```
+
+By default, the CLI generates sources for both TypeScript and Zig (in this
+order!). Order is important, since the output file paths must be given in the
+same order as the target languages. It's recommended to be more explicit. An
+example invocation looks like:
+
+```bash
+wasm-api --config codegen-opts.json \
+  --lang ts -o src/generated.ts \
+  --lang zig -o src.zig/generated.zig \
+  typedefs.json
+```
+
+The structure of the config file is as follows (all optional):
+
+```text
+{
+	global: { ... },
+	ts: { ... },
+	zig: { ... },
+}
+```
+
+More details about possible
+[`global`](https://docs.thi.ng/umbrella/wasm-api/interfaces/CodeGenOpts.html),
+[`ts`](https://docs.thi.ng/umbrella/wasm-api/interfaces/TSOpts.html) and
+[`zig`](https://docs.thi.ng/umbrella/wasm-api/interfaces/ZigOpts.html) config
+options & values.
+
+#### Data type definitions
+
+Currently, the code generator supports structs and enums. See API docs for
+further details:
+
+- [`Enum`](https://docs.thi.ng/umbrella/wasm-api/interfaces/Enum.html)
+- [`EnumValue`](https://docs.thi.ng/umbrella/wasm-api/interfaces/EnumValue.html)
+- [`Struct`](https://docs.thi.ng/umbrella/wasm-api/interfaces/Struct.html)
+- [`StructField`](https://docs.thi.ng/umbrella/wasm-api/interfaces/StructField.html)
+- [`TopLevelType`](https://docs.thi.ng/umbrella/wasm-api/interfaces/TopLevelType.html)
+
+#### Example usage
+
+Below is an example file with JSON type definitions and the resulting source
+codes:
+
+**⬇︎ CLICK TO EXPAND EACH CODE BLOCK ⬇︎**
+
+<details><summary>types.json (Type definitions)</summary>
+
+```json
+[
+	{
+		"name": "Foo",
+		"type": "struct",
+		"doc": "Example struct",
+		"auto": true,
+		"fields": [
+			{ "name": "id", "type": "u8", "doc": "Unique ID" },
+			{ "name": "bars", "type": "Bar", "tag": "array", "len": 3 },
+			{ "name": "color", "type": "f32", "tag": "vec", "len": 4 }
+		]
+	},
+	{
+		"name": "Bar",
+		"type": "struct",
+		"fields": [
+			{ "name": "kind", "type": "Kind" },
+			{ "name": "flags", "type": "u32" }
+		]
+	},
+	{
+		"name": "Kind",
+		"type": "enum",
+		"tag": "u16",
+		"values": [
+			"unknown",
+			{ "name": "good", "value": 100 },
+			{ "name": "best", "value": 1000 }
+		]
+	}
+]
+```
+</details>
+
+<details><summary>generated.ts (generated TypeScript source)</summary>
+
+```ts
+/** Generated by @thi.ng/wasm-api at 2022-08-15T22:32:21.189Z - DO NOT EDIT! */
+
+import type { WasmTypeBase, WasmTypeConstructor } from "@thi.ng/wasm-api";
+
+/** Example struct */
+export interface Foo extends WasmTypeBase {
+	color: Float32Array;
+	bars: Bar[];
+	/** Unique ID */
+	id: number;
+}
+
+export const $Foo: WasmTypeConstructor<Foo> = (mem) => ({
+	get align() { return 16; },
+	get size() { return 48; },
+	instance: (base) => ({
+		get __base() { return base; },
+		get __bytes() { return mem.u8.subarray(base, base + 48); },
+		get color(): Float32Array {
+			const addr = base >>> 2;
+			return mem.f32.subarray(addr, addr + 4);
+		},
+		get bars(): Bar[] {
+			const addr = (base + 16);
+			const inst = $Bar(mem);
+			const slice: Bar[] = [];
+			for(let i = 0; i < 3; i++) slice.push(inst.instance(addr + i * 24));
+			return slice;
+		},
+		get id(): number {
+			return mem.u8[(base + 40)];
+		},
+		set id(x: number) {
+			mem.u8[(base + 40)] = x;
+		},
+	})
+});
+
+export interface Bar extends WasmTypeBase {
+	kind: Kind;
+	flags: number;
+}
+
+export const $Bar: WasmTypeConstructor<Bar> = (mem) => ({
+	get align() { return 4; },
+	get size() { return 8; },
+	instance: (base) => ({
+		get __base() { return base; },
+		get __bytes() { return mem.u8.subarray(base, base + 8); },
+		get kind(): Kind {
+			return mem.u16[base >>> 1];
+		},
+		set kind(x: Kind) {
+			mem.u16[base >>> 1] = x;
+		},
+		get flags(): number {
+			return mem.u32[(base + 4) >>> 2];
+		},
+		set flags(x: number) {
+			mem.u32[(base + 4) >>> 2] = x;
+		},
+	})
+});
+
+export enum Kind {
+	UNKNOWN,
+	GOOD = 100,
+	BEST = 1000,
+}
+```
+</details>
+
+<details><summary>generated.zig (generated Zig source)</summary>
+
+```zig
+//! Generated by @thi.ng/wasm-api at 2022-08-15T22:32:21.191Z - DO NOT EDIT!
+
+/// Example struct
+pub const Foo = struct {
+    color: @Vector(4, f32),
+    bars: [3]Bar,
+    /// Unique ID
+    id: u8,
+};
+
+pub const Bar = struct {
+    kind: Kind,
+    flags: u32,
+};
+
+pub const Kind = enum(u16) {
+    unknown,
+    good = 100,
+    best = 1000,
+};
+```
+</details>
+
+On the TypeScript/JS side, the memory-mapped wrappers (e.g. `$Foo` and `$Bar`)
+can be used in combination with the `WasmBridge` to obtain fully typed views
+(according to the generated types) of the underlying WASM memory. Basic usage is
+like:
+
+```ts
+import { WasmBridge } from "@thi.ng/wasm-api";
+import { $Foo, Kind } from "./generated.ts";
+
+const bridge = new WasmBridge();
+// bridge initialization omitted here (see other examples below)
+// ...
+
+// Create an instance using the bridge's memory views
+// and mapping a `Foo` struct from given address
+// (e.g. obtained from an exported WASM function/value)
+const foo = $Foo(bridge).instance(0x10000);
+
+// then use like normal JS object
+foo.color
+// Float32Array(4) [0.1, 0.2, 0.3, 0.4]
+
+// this even applies to arrays using other types
+foo.bars[2].kind = Kind.BEST;
+
+// IMPORTANT: any modifications like this are directly
+// applied to the underlying WASM memory...
+```
+
+**IMPORTANT:** Struct field setters are currently only supported for single
+values, incl. enums, strings, structs. The latter 2 will always be copied by
+value (mem copy). Arrays or slices of strings do not currently provide write
+access...
+
+### String handling
+
+Most low-level languages deal with strings very differently and alas there's no
+general standard. Some have UTF-8/16 support, others don't. In some languages
+(incl. C & Zig), strings are stored as zero terminated, in others they aren't...
+It's outside the scope of this package to provide an allround out-of-the-box
+solution. However, the code generators provide the global `stringType` option to
+interpret the `string` type of a struct field in different ways:
+
+- `slice` (default): Considers strings as Zig-style slices (i.e. pointer + length)
+- `ptr`: Considers strings as C-style raw `*char` pointer (without any length)
+
+Note: If setting this global option to `ptr`, it also has to be stated for the
+TypeScript code generator explicitly.
+
 ### Status
 
 **ALPHA** - bleeding edge / work-in-progress
@@ -185,12 +480,22 @@ node --experimental-repl-await
 > const wasmApi = await import("@thi.ng/wasm-api");
 ```
 
-Package sizes (gzipped, pre-treeshake): ESM: 1.61 KB
+Package sizes (gzipped, pre-treeshake): ESM: 4.32 KB
+
+**IMPORTANT:** The package includes various code generators and supporting
+functions which are NOT required during runtime. Hence the actual package size
+in production will be MUCH smaller!
 
 ## Dependencies
 
 - [@thi.ng/api](https://github.com/thi-ng/umbrella/tree/develop/packages/api)
+- [@thi.ng/args](https://github.com/thi-ng/umbrella/tree/develop/packages/args)
+- [@thi.ng/binary](https://github.com/thi-ng/umbrella/tree/develop/packages/binary)
+- [@thi.ng/checks](https://github.com/thi-ng/umbrella/tree/develop/packages/checks)
+- [@thi.ng/compare](https://github.com/thi-ng/umbrella/tree/develop/packages/compare)
+- [@thi.ng/defmulti](https://github.com/thi-ng/umbrella/tree/develop/packages/defmulti)
 - [@thi.ng/errors](https://github.com/thi-ng/umbrella/tree/develop/packages/errors)
+- [@thi.ng/file-io](https://github.com/thi-ng/umbrella/tree/develop/packages/file-io)
 - [@thi.ng/hex](https://github.com/thi-ng/umbrella/tree/develop/packages/hex)
 - [@thi.ng/idgen](https://github.com/thi-ng/umbrella/tree/develop/packages/idgen)
 - [@thi.ng/logger](https://github.com/thi-ng/umbrella/tree/develop/packages/logger)
@@ -199,6 +504,8 @@ Package sizes (gzipped, pre-treeshake): ESM: 1.61 KB
 
 [Generated API docs](https://docs.thi.ng/umbrella/wasm-api/)
 
+## Basic usage example
+
 ```ts
 import { WasmBridge, WasmExports } from "@thi.ng/wasm-api";
 import { readFileSync } from "fs";
@@ -223,12 +530,20 @@ interface App extends WasmExports {
 })();
 ```
 
+### Zig version
+
+Requires [Zig](https://ziglang.org) to be installed:
+
 ```zig
 //! Example Zig application (hello.zig)
 
 /// import externals
 /// see build command for configuration
 const js = @import("wasmapi");
+const std = @import("std");
+
+// set custom memory allocator (here to disable)
+pub const WASM_ALLOCATOR: ?std.mem.Allocator = null;
 
 export fn start() void {
 	js.printStr("hello world!");
@@ -242,7 +557,7 @@ folder):
 ```bash
 # compile WASM binary
 zig build-lib \
-	--pkg-begin wasmapi node_modules/@thi.ng/wasm-api/zig/core.zig --pkg-end \
+	--pkg-begin wasmapi node_modules/@thi.ng/wasm-api/include/wasmapi.zig --pkg-end \
 	-target wasm32-freestanding \
 	-O ReleaseSmall -dynamic --strip \
 	hello.zig
@@ -257,18 +572,112 @@ The resulting WASM:
 (module
  (type $i32_i32_=>_none (func (param i32 i32)))
  (type $none_=>_none (func))
- (import "core" "_printStr" (func $fimport$0 (param i32 i32)))
+ (type $i32_=>_i32 (func (param i32) (result i32)))
+ (import "wasmapi" "_printStr" (func $fimport$0 (param i32 i32)))
  (global $global$0 (mut i32) (i32.const 65536))
  (memory $0 2)
  (data (i32.const 65536) "hello world!\00")
  (export "memory" (memory $0))
  (export "start" (func $0))
+ (export "_wasm_allocate" (func $1))
+ (export "_wasm_free" (func $2))
  (func $0
   (call $fimport$0
    (i32.const 65536)
    (i32.const 12)
   )
  )
+ (func $1 (param $0 i32) (result i32)
+  (i32.const 0)
+ )
+ (func $2 (param $0 i32) (param $1 i32)
+ )
+)
+```
+
+### C11 version
+
+Requires [Emscripten](https://emscripten.org/) to be installed:
+
+```c
+#include <wasmapi.h>
+
+void WASM_KEEP start() {
+	wasm_printStr0("hello world!");
+}
+```
+
+Building the WASM module:
+
+```bash
+emcc -Os -Inode_modules/@thi.ng/wasm-api/include -DWASMAPI_NO_MALLOC \
+  -sERROR_ON_UNDEFINED_SYMBOLS=0 --no-entry \
+  -o hello.wasm hello.c
+```
+
+Resulting WASM:
+
+```wasm
+(module
+ (type $i32_=>_none (func (param i32)))
+ (type $none_=>_none (func))
+ (type $i32_=>_i32 (func (param i32) (result i32)))
+ (type $none_=>_i32 (func (result i32)))
+ (import "wasmapi" "_printStr0" (func $fimport$0 (param i32)))
+ (global $global$0 (mut i32) (i32.const 5243936))
+ (memory $0 256 256)
+ (data (i32.const 1024) "hello world!")
+ (table $0 2 2 funcref)
+ (elem (i32.const 1) $0)
+ (export "memory" (memory $0))
+ (export "_wasm_allocate" (func $1))
+ (export "_wasm_free" (func $2))
+ (export "start" (func $3))
+ (export "__indirect_function_table" (table $0))
+ (export "_initialize" (func $0))
+ (export "__errno_location" (func $7))
+ (export "stackSave" (func $4))
+ (export "stackRestore" (func $5))
+ (export "stackAlloc" (func $6))
+ (func $0
+  (nop)
+ )
+ (func $1 (param $0 i32) (result i32)
+  (i32.const 0)
+ )
+ (func $2 (param $0 i32)
+  (nop)
+ )
+ (func $3
+  (call $fimport$0
+   (i32.const 1024)
+  )
+ )
+ (func $4 (result i32)
+  (global.get $global$0)
+ )
+ (func $5 (param $0 i32)
+  (global.set $global$0
+   (local.get $0)
+  )
+ )
+ (func $6 (param $0 i32) (result i32)
+  (global.set $global$0
+   (local.tee $0
+    (i32.and
+     (i32.sub
+      (global.get $global$0)
+      (local.get $0)
+     )
+     (i32.const -16)
+    )
+   )
+  )
+  (local.get $0)
+ )
+ (func $7 (result i32)
+  (i32.const 1040)
+ )
 )
 ```