@fraczak/k 6.3.3 → 6.5.0

package/backends/wasm/README.md

@@ -0,0 +1,97 @@
+# k WebAssembly Backend
+
+`@fraczak/k-wasm` is the experimental WebAssembly backend for
+[`@fraczak/k`](https://github.com/fraczak/k). It lowers typed k programs through
+the kVM representation, emits standard `.wasm` modules with WABT, and provides
+a Node.js runner for the binary `pattern + value` streams used by the k codec
+toolchain.
+
+## Install
+
+Install dependencies from the monorepo root:
+
+```bash
+npm install
+```
+
+Optionally expose the three backend commands on your path:
+
+```bash
+npm link
+```
+
+## Quick Start
+
+Compile an expression into a standalone WebAssembly artifact:
+
+```bash
+node ./bin/k-wasm-compile.mjs '|ok' /tmp/ok.wasm
+```
+
+Run it over the unit value and print the decoded result:
+
+```bash
+node ../../codecs/unit.mjs --parse |
+  node ./bin/k-wasm-run.mjs /tmp/ok.wasm |
+  node ../../codecs/k-print.mjs
+```
+
+Expected output:
+
+```json
+"ok"
+```
+
+With the backend and k codec commands on your path, the equivalent shorter
+form is:
+
+```bash
+k-wasm-compile '|ok' /tmp/ok.wasm
+k-unit --parse | k-wasm-run /tmp/ok.wasm | k-print
+```
+
+## Commands
+
+| Command | Purpose |
+| --- | --- |
+| `k-wasm` | Compile k source, `.ko`, or `.kvm` input in memory and run it immediately |
+| `k-wasm-compile` | Compile k source, `.ko`, or `.kvm` input into a `.wasm` artifact |
+| `k-wasm-run` | Run an existing `.wasm` artifact over a binary input stream |
+
+All commands accept `--help`. The compile commands follow the `k-compile`
+input convention: pass inline source, an existing source path, a `.ko` object,
+or a `.kvm` program. They also accept one `--lib` option for a `.klib`
+dependency and repeated `--export` options to bring library aliases into the
+source scope. Existing paths are treated as files; non-existing names with
+`.k`, `.ko`, `.kvm`, `.klib`, or `.wasm` extensions are treated as missing
+files; other arguments are treated as inline source. The runners read a binary
+input stream from standard input unless an input file is given.
+
+## Artifacts
+
+The compiler produces a standard WebAssembly binary module containing the
+reachable relations, the arena runtime, and a `k.metadata` custom section. The
+metadata stores the entry point and the pattern information needed to encode
+and decode values. A compiled artifact can be run later without its original k
+source, object, kVM, or `.klib` files.
+
+The current host runner targets Node.js. It is still needed to bridge between
+WebAssembly linear memory and k binary streams.
+
+## Development
+
+Run the test suite:
+
+```bash
+npm test
+```
+
+Run the comparative IEEE benchmark:
+
+```bash
+npm run perf:ieee
+```
+
+See [DOCS/USE-WEB-ASSEMBLY.md](DOCS/USE-WEB-ASSEMBLY.md) for the artifact
+format, source-file and library examples, profiling options, and current
+limitations.

package/backends/wasm/TECHNICAL-DEBT.md

@@ -0,0 +1,58 @@
+# Technical Debt
+
+## Polymorphic Input Retyping for Precompiled Wasm
+
+`k-wasm` can retype a polymorphic program when it compiles and runs in one step:
+the CLI decodes the input stream first, intersects the program input pattern with
+the input value envelope, and compiles a specialized WebAssembly module for that
+run.
+
+`k-wasm-run` cannot currently do the same for an already compiled `.wasm`
+artifact. By the time the runner sees the input envelope, the k program has
+already been lowered to WebAssembly and the artifact only carries metadata such
+as the generic input/output patterns. The original source/object relation graph
+needed to re-run type derivation is not available.
+
+This means precompiled artifacts can execute polymorphic programs, but their
+output envelopes are limited to the generic artifact metadata rather than being
+retyped against each input envelope.
+
+One possible fix is to delay compilation for programs with polymorphic input
+until the input envelope is available. That could mean storing enough source,
+object, or typed relation data in the artifact to specialize on first run, or
+using a separate artifact form for deferred compilation.
+
+## Wasm Arena Memory Management
+
+The WebAssembly backend currently uses a bump arena for all product and variant
+values created during execution. This keeps allocation cheap and simple, and
+failed union branches can roll the arena back to a checkpoint, but successful
+paths do not reclaim temporary values until the whole run ends.
+
+Tail-call lowering now avoids allocating replacement input products for some
+self-tail-recursive loops by keeping top-level product fields in Wasm locals.
+Furthermore, self-tail-recursive loops now incorporate threshold-based arena
+compaction: when arena growth within a tail loop exceeds 1 MB, live accumulator
+and argument graphs are compacted down to the loop entry mark using an iterative
+staging-and-copy mechanism in `runtime.wat`, freeing dead scratch data while
+preserving performance for hot short-lived loops.
+
+Additionally, linear memory exhaustion and 32-bit unsigned offset overflow are
+now tracked explicitly via exported flags (`oom_flag`, `oom_size`) in `runtime.wat`,
+which the JS runtime intercepts to throw a descriptive `OutOfMemoryError` instead
+of an untyped `unreachable` trap.
+
+Remaining memory-management targets:
+
+- Add allocation profiling that attributes arena growth to generated relations
+  and static allocation sites. Large regressions should be visible without
+  manually instrumenting `factorial:o` or similar workloads.
+- Extend local-loop lowering beyond top-level product fields. Hot tail loops
+  should keep common accumulator state in locals where possible and materialize
+  arena values only at function boundaries or observable return points.
+- Introduce safe lifetime boundaries for successful non-tail calls. The runtime
+  needs a way to reclaim scratch values after a callee result has been consumed
+  without copying large live values on every call.
+- Keep unsigned pointer handling explicit. Wasm arena addresses are `i32`
+  offsets, and JavaScript sees exported `i32` values as signed numbers, so all
+  JS-side arena offsets must be normalized before `DataView` access.

package/backends/wasm/bin/k-wasm-compile.mjs

@@ -0,0 +1,50 @@
+#!/usr/bin/env node
+
+import fs from "node:fs";
+import { argv, exit, stdin, stdout } from "node:process";
+
+import {
+  compileProgramInput,
+  parseCompileOptions,
+  resolveProgramInput
+} from "../src/cli.mjs";
+
+function usage(stream = console.error) {
+  const prog = argv[1] || "k-wasm-compile.mjs";
+  stream(`Usage: node ${prog} [options] [source-snippet | input-file [wasm-file]]`);
+  stream("Compile k source, .ko, or .kvm input into a standalone WebAssembly artifact.");
+  stream("");
+  stream("Arguments:");
+  stream("  source-snippet  Inline k source, in the same style as k.mjs.");
+  stream("  input-file      Source .k, .ko, or .kvm file. Reads UTF-8 source from stdin when omitted.");
+  stream("  wasm-file       Output .wasm path. Writes the binary artifact to stdout when omitted.");
+  stream("");
+  stream("Options:");
+  stream("  --lib file       Load one .klib dependency before compiling.");
+  stream("  --export spec    Export a library alias into source scope. May be repeated.");
+  stream("                   spec is 'name' or 'libname:localname'.");
+  stream("  -h, --help       Show this help.");
+}
+
+try {
+  const args = argv.slice(2);
+  if (args.includes("-h") || args.includes("--help")) {
+    usage(console.log);
+    exit(0);
+  }
+
+  const { libraries, exportSpecs } = parseCompileOptions(args);
+  const input = resolveProgramInput(args, { allowStdinSource: true });
+  const outputPath = args.shift();
+  if (args.length > 0) throw new Error("Too many arguments");
+  const artifact = await compileProgramInput(input, { libraries, exportSpecs, stdin });
+  if (outputPath == null) {
+    stdout.write(artifact);
+  } else {
+    fs.writeFileSync(outputPath, artifact);
+  }
+} catch (error) {
+  console.error(error.stack || error.message || String(error));
+  usage();
+  exit(1);
+}

package/backends/wasm/bin/k-wasm-run.mjs

@@ -0,0 +1,140 @@
+#!/usr/bin/env node
+
+import fs from "node:fs";
+import { argv, exit, stdin, stdout, stderr } from "node:process";
+
+import { instantiateWasmArtifact } from "../src/wasm.mjs";
+
+function usage(stream = console.error) {
+  const prog = argv[1] || "k-wasm-run.mjs";
+  stream(`Usage: node ${prog} [ --server ] wasm-file [ input-file ]`);
+  stream("Run a standalone k WebAssembly artifact over a binary pattern+value stream.");
+  stream("");
+  stream("Arguments:");
+  stream("  wasm-file   Input .wasm artifact produced by k-wasm-compile.");
+  stream("  input-file  Optional binary input stream. Reads stdin when omitted.");
+  stream("");
+  stream("Options:");
+  stream("  --server    Run as a persistent server reading framed requests from stdin.");
+  stream("  -h, --help  Show this help.");
+}
+
+function readAll(stream) {
+  return new Promise((resolve, reject) => {
+    const chunks = [];
+    stream.on("data", (chunk) => chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)));
+    stream.on("end", () => resolve(Buffer.concat(chunks)));
+    stream.on("error", reject);
+  });
+}
+
+async function runServer(runner) {
+  const tracing = process.env.K_TRACE === "1" || process.env.TRACE === "1";
+  let inBuffer = Buffer.alloc(0);
+  let readStartNs = 0n;
+
+  stdin.on("data", (chunk) => {
+    if (inBuffer.length === 0) {
+      readStartNs = process.hrtime.bigint();
+    }
+    inBuffer = Buffer.concat([inBuffer, chunk]);
+    processBuffer();
+  });
+
+  stdin.on("end", () => {
+    exit(0);
+  });
+
+  function processBuffer() {
+    while (inBuffer.length >= 4) {
+      const len = inBuffer.readUInt32BE(0);
+      if (inBuffer.length < 4 + len) {
+        break;
+      }
+      const t1 = process.hrtime.bigint();
+      const payload = inBuffer.subarray(4, 4 + len);
+      inBuffer = inBuffer.subarray(4 + len);
+
+      try {
+        let outWire;
+        let trace = null;
+        if (tracing) {
+          const res = runner.execute(payload, { trace: true });
+          outWire = res.outputWire;
+          trace = res.trace;
+        } else {
+          outWire = runner.execute(payload);
+        }
+
+        const writeStart = process.hrtime.bigint();
+        const outHdr = Buffer.allocUnsafe(4);
+        outHdr.writeUInt32BE(outWire.length, 0);
+        stdout.write(outHdr);
+        stdout.write(outWire);
+        const tEnd = process.hrtime.bigint();
+
+        if (tracing && trace) {
+          const ipcReadNs = readStartNs > 0n ? Number(t1 - readStartNs) : 0;
+          const decodeNs = trace.decodeNs;
+          const flatInNs = trace.flatInNs;
+          const evalNs = trace.evalNs;
+          const flatOutNs = trace.flatOutNs;
+          const encodeNs = trace.encodeNs;
+          const ipcWriteNs = Number(tEnd - writeStart);
+          const totalNs = Number(tEnd - (readStartNs > 0n ? readStartNs : t1));
+          stderr.write(`K_TRACE_PHASES backend=wasm ipc_read_ns=${ipcReadNs} decode_ns=${decodeNs} flat_in_ns=${flatInNs} eval_ns=${evalNs} flat_out_ns=${flatOutNs} encode_ns=${encodeNs} ipc_write_ns=${ipcWriteNs} total_ns=${totalNs}\n`);
+        }
+      } catch (err) {
+        stderr.write(`Wasm server error: ${err.stack || err.message || String(err)}\n`);
+        exit(1);
+      }
+
+      readStartNs = process.hrtime.bigint();
+    }
+  }
+}
+
+async function main() {
+  const rawArgs = argv.slice(2);
+  if (rawArgs.includes("-h") || rawArgs.includes("--help")) {
+    usage(console.log);
+    return;
+  }
+
+  const isServer = rawArgs.includes("--server");
+  const args = rawArgs.filter(a => a !== "--server");
+
+  const wasmPath = args.shift();
+  if (!wasmPath) throw new Error("Missing .wasm artifact path");
+
+  const artifact = fs.readFileSync(wasmPath);
+  const runner = await instantiateWasmArtifact(artifact);
+
+  if (isServer) {
+    if (args.length > 0) throw new Error("Too many arguments for --server mode");
+    await runServer(runner);
+    return;
+  }
+
+  const inputPath = args.shift();
+  if (args.length > 0) throw new Error("Too many arguments");
+
+  const input = await readAll(inputPath == null ? stdin : fs.createReadStream(inputPath));
+  const tracing = process.env.K_TRACE === "1" || process.env.TRACE === "1";
+  if (tracing) {
+    const res = runner.execute(input, { trace: true });
+    stdout.write(res.outputWire);
+    if (res.trace) {
+      const trace = res.trace;
+      stderr.write(`K_TRACE_PHASES backend=wasm ipc_read_ns=0 decode_ns=${trace.decodeNs} flat_in_ns=${trace.flatInNs} eval_ns=${trace.evalNs} flat_out_ns=${trace.flatOutNs} encode_ns=${trace.encodeNs} ipc_write_ns=0 total_ns=${trace.evalNs}\n`);
+    }
+  } else {
+    stdout.write(runner.execute(input));
+  }
+}
+
+main().catch((error) => {
+  console.error(error.stack || error.message || String(error));
+  usage();
+  exit(1);
+});

package/backends/wasm/bin/k-wasm.mjs

@@ -0,0 +1,49 @@
+#!/usr/bin/env node
+
+import fs from "node:fs";
+import { argv, exit, stdin, stdout } from "node:process";
+import { decodeWire } from "@fraczak/k/backend-api.mjs";
+
+import {
+  compileProgramInput,
+  parseCompileOptions,
+  readAll,
+  resolveProgramInput
+} from "../src/cli.mjs";
+import { runWasmArtifact } from "../src/wasm.mjs";
+
+function usage(stream = console.error) {
+  const prog = argv[1] || "k-wasm.mjs";
+  stream(`Usage: node ${prog} [options] ( source-snippet | input-file ) [ input-file ]`);
+  stream("Compile k source, .ko, or .kvm input to WebAssembly in memory and run it over a binary pattern+value stream.");
+  stream("");
+  stream("Options:");
+  stream("  --lib file       Load one .klib dependency before compiling.");
+  stream("  --export spec    Export a library alias into source scope. May be repeated.");
+  stream("                   spec is 'name' or 'libname:localname'.");
+  stream("  -h, --help       Show this help.");
+}
+
+async function main() {
+  const args = argv.slice(2);
+  if (args.includes("-h") || args.includes("--help")) {
+    usage(console.log);
+    return;
+  }
+
+  const { libraries, exportSpecs } = parseCompileOptions(args);
+  const programInput = resolveProgramInput(args);
+  const inputPath = args.shift();
+  if (args.length > 0) throw new Error("Too many arguments");
+
+  const inputBuffer = await readAll(inputPath == null ? stdin : fs.createReadStream(inputPath));
+  const { pattern: inputEnvelopePattern } = decodeWire(inputBuffer);
+  const artifact = await compileProgramInput(programInput, { libraries, exportSpecs, inputEnvelopePattern });
+  stdout.write(await runWasmArtifact(artifact, inputBuffer));
+}
+
+main().catch((error) => {
+  console.error(error.stack || error.message || String(error));
+  usage();
+  exit(1);
+});