porffor 0.0.0-425ea20 → 0.0.0-48403fd

package/README.md

@@ -1,11 +1,10 @@
-# porffor
-a basic experimental wip *aot* optimizing js -> wasm engine/compiler/runtime in js. not serious/intended for (real) use. (this is a straight forward, honest readme)<br>
-age: ~1 month
+# porffor &nbsp;<sup><sub>/ˈpɔrfɔr/ &nbsp;*(poor-for)*</sup></sub>
+a basic experimental wip *aot* optimizing js -> wasm/c engine/compiler/runtime in js. not serious/intended for (real) use. (this is a straight forward, honest readme)<br>
+age: ~2 months
 
 ## design
 porffor is a very unique js engine, due a very different approach. it is seriously limited, but what it can do, it does pretty well. key differences:
 - 100% aot compiled *(not jit)*
-- everything is a number
 - no constant runtime/preluded code
 - least Wasm imports possible (only stdio)
 
@@ -18,6 +17,12 @@ porffor is mostly built from scratch, the only thing that is not is the parser (
 - no variables between scopes (except args and globals)
 - literal callees only in calls (eg `print()` works, `a = print; a()` does not)
 
+## rhemyn
+rhemyn is porffor's own regex engine; it compiles literal regex to wasm bytecode aot (remind you of anything?). it is quite basic and wip. see [its readme](rhemyn/README.md) for more details.
+
+## 2c
+2c is porffor's own wasm -> c compiler, using generated wasm bytecode and internal info to generate specific and efficient/fast c code. no boilerplate/preluded code or required external files, just for cli binaries (not like wasm2c very much at all).
+
 ## supported
 see [optimizations](#optimizations) for opts implemented/supported.
 
@@ -78,6 +83,9 @@ these include some early (stage 1/0) and/or dead (last commit years ago) proposa
 - truthy/falsy (eg `!'' == true`)
 - string comparison (eg `'a' == 'a'`, `'a' != 'b'`)
 - nullish coalescing operator (`??`)
+- `for...of` (arrays and strings)
+- array member setting (`arr[0] = 2`, `arr[0] += 2`, etc)
+- array constructor (`Array(5)`, `new Array(1, 2, 3)`)
 
 ### built-ins
 
@@ -90,7 +98,7 @@ these include some early (stage 1/0) and/or dead (last commit years ago) proposa
 - basic `eval` (literals only)
 - `Math.random()` using self-made xorshift128+ PRNG
 - some of `performance` (`now()`)
-- some of `Array.prototype` (`at`, `push`, `pop`, `shift`)
+- some of `Array.prototype` (`at`, `push`, `pop`, `shift`, `fill`)
 - some of `String.prototype` (`at`, `charAt`, `charCodeAt`)
 
 ### custom
@@ -105,19 +113,15 @@ these include some early (stage 1/0) and/or dead (last commit years ago) proposa
 no particular order and no guarentees, just what could happen soon™
 
 - arrays
-  - member setting (`arr[0] = 2`)
   - more of `Array` prototype
   - arrays/strings inside arrays
   - destructuring
-  - for .. of
 - strings
   - member setting
 - objects
   - basic object expressions (eg `{}`, `{ a: 0 }`)
 - wasm
   - *basic* wasm engine (interpreter) in js
-- regex
-  - *basic* regex engine (in wasm compiled aot or js interpreter?)
 - more math operators (`**`, etc)
 - `do { ... } while (...)`
 - rewrite `console.log` to work with strings/arrays
@@ -129,10 +133,11 @@ no particular order and no guarentees, just what could happen soon™
   - rewrite local indexes per func for smallest local header and remove unused idxs
   - smarter inline selection (snapshots?)
   - remove const ifs (`if (true)`, etc)
-  - use data segments for initing arrays
 
-## test262
-porffor can run test262 via some hacks/transforms which remove unsupported features whilst still doing the same asserts (eg simpler error messages using literals only). it currently passes >10% (see latest commit desc for latest and details). use `node test262` to test, it will also show a difference of overall results between the last commit and current results.
+## porfformance
+*for the things it supports most of the time*, porffor is blazingly fast compared to most interpreters, and common engines running without JIT. for those with JIT, it is not that much slower like a traditional interpreter would be; mostly the same or a bit faster/slower depending on what.
+
+![Screenshot of comparison chart](https://github.com/CanadaHonk/porffor/assets/19228318/76c75264-cc68-4be1-8891-c06dc389d97a)
 
 ## optimizations
 mostly for reducing size. do not really care about compiler perf/time as long as it is reasonable. we do not use/rely on external opt tools (`wasm-opt`, etc), instead doing optimization inside the compiler itself creating even smaller code sizes than `wasm-opt` itself can produce as we have more internal information. (this also enables fast + small runtime use as a potential cursed jit in frontend).
@@ -153,11 +158,15 @@ mostly for reducing size. do not really care about compiler perf/time as long as
 - remove unneeded single just used vars
 - remove unneeded blocks (no `br`s inside)
 - remove unused imports
+- use data segments for initing arrays/strings
 
 ### wasm module
 - type cache/index (no repeated types)
 - no main func if empty (and other exports)
 
+## test262
+porffor can run test262 via some hacks/transforms which remove unsupported features whilst still doing the same asserts (eg simpler error messages using literals only). it currently passes >10% (see latest commit desc for latest and details). use `node test262` to test, it will also show a difference of overall results between the last commit and current results.
+
 ## codebase
 - `compiler`: contains the compiler itself
   - `builtins.js`: all built-ins of the engine (spec, custom. vars, funcs)
@@ -178,6 +187,10 @@ mostly for reducing size. do not really care about compiler perf/time as long as
   - `info.js`: runs with extra info printed
   - `repl.js`: basic repl (uses `node:repl`)
 
+- `rhemyn`: contains [rhemyn](#rhemyn) - the regex engine used by porffor
+  - `compile.js`: compiles regex ast into wasm bytecode
+  - `parse.js`: own regex parser
+
 - `test`: contains many test files for majority of supported features
 - `test262`: test262 runner and utils
 
@@ -195,8 +208,13 @@ basically nothing will work :). see files in `test` for examples.
 you can also use deno (`deno run -A ...` instead of `node ...`), or bun (`bun ...` instead of `node ...`)
 
 ### flags
-- `-raw` for no info logs (just raw js output)
-- `-valtype=i32|i64|f64` to set valtype, f64 by default
+- `-target=wasm|c|native` (default: `wasm`) to set target output (native compiles c output to binary, see args below)
+- `-target=c|native` only:
+  - `-o=out.c|out.exe|out` to set file to output c or binary
+- `-target=native` only:
+  - `-compiler=clang` to set compiler binary (path/name) to use to compile
+  - `-cO=O3` to set compiler opt argument
+- `-valtype=i32|i64|f64` (default: `f64`) to set valtype
 - `-O0` to disable opt
 - `-O1` (default) to enable basic opt (simplify insts, treeshake wasm imports)
 - `-O2` to enable advanced opt (inlining)
@@ -204,11 +222,14 @@ you can also use deno (`deno run -A ...` instead of `node ...`), or bun (`bun ..
 - `-no-run` to not run wasm output, just compile
 - `-opt-log` to log some opts
 - `-code-log` to log some codegen (you probably want `-funcs`)
-- `-funcs` to log funcs (internal representations)
+- `-regex-log` to log some regex
+- `-funcs` to log funcs
+- `-ast-log` to log AST
 - `-opt-funcs` to log funcs after opt
 - `-sections` to log sections as hex
 - `-opt-no-inline` to not inline any funcs
-- `-tail-call` to enable tail calls (not widely implemented)
+- `-tail-call` to enable tail calls (experimental + not widely implemented)
+- `-compile-hints` to enable V8 compilation hints (experimental + doesn't seem to do much?)
 
 ## vscode extension
 there is a vscode extension in `porffor-for-vscode` which tweaks js syntax highlighting to be nicer with porffor features (eg highlighting wasm inside of inline asm).

package/compiler/2c.js

@@ -0,0 +1,354 @@
+import { read_ieee754_binary64, read_signedLEB128 } from './encoding.js';
+import { Blocktype, Opcodes, Valtype } from './wasmSpec.js';
+import { operatorOpcode } from './expression.js';
+
+const CValtype = {
+  i8: 'char',
+  i16: 'unsigned short', // presume all i16 stuff is unsigned
+  i32: 'long',
+  i32_u: 'unsigned long',
+  i64: 'long long',
+  i64_u: 'unsigned long long',
+
+  f32: 'float',
+  f64: 'double',
+
+  undefined: 'void'
+};
+
+const inv = (obj, keyMap = x => x) => Object.keys(obj).reduce((acc, x) => { acc[keyMap(obj[x])] = x; return acc; }, {});
+const invOpcodes = inv(Opcodes);
+
+for (const x in CValtype) {
+  if (Valtype[x]) CValtype[Valtype[x]] = CValtype[x];
+}
+
+const todo = msg => {
+  class TodoError extends Error {
+    constructor(message) {
+      super(message);
+      this.name = 'TodoError';
+    }
+  }
+
+  throw new TodoError(`todo: ${msg}`);
+};
+
+const removeBrackets = str => str.startsWith('(') && str.endsWith(')') ? str.slice(1, -1) : str;
+
+export default ({ funcs, globals, tags, exceptions, pages }) => {
+  const invOperatorOpcode = Object.values(operatorOpcode).reduce((acc, x) => {
+    for (const k in x) {
+      acc[x[k]] = k;
+    }
+    return acc;
+  }, {});
+  const invGlobals = inv(globals, x => x.idx);
+
+  const includes = new Map(), unixIncludes = new Map(), winIncludes = new Map();
+  let out = '';
+
+  for (const x in globals) {
+    const g = globals[x];
+
+    out += `${CValtype[g.type]} ${x} = ${g.init ?? 0}`;
+    out += ';\n';
+  }
+
+  for (const [ x, p ] of pages) {
+    out += `${CValtype[p.type]} ${x.replace(': ', '_').replace(/[^0-9a-zA-Z_]/g, '')}[100]`;
+    out += ';\n';
+  }
+
+  if (out) out += '\n';
+
+  let depth = 1;
+  const line = (str, semi = true) => out += `${' '.repeat(depth * 2)}${str}${semi ? ';' : ''}\n`;
+  const lines = lines => {
+    for (const x of lines) {
+      out += `${' '.repeat(depth * 2)}${x}\n`;
+    }
+  };
+
+  const platformSpecific = (win, unix, add = true) => {
+    let tmp = '';
+
+    if (win) {
+      if (add) out += '#ifdef _WIN32\n';
+        else tmp += '#ifdef _WIN32\n';
+
+      if (add) lines(win.split('\n'));
+        else tmp += win + (win.endsWith('\n') ? '' : '\n');
+    }
+
+    if (unix) {
+      if (add) out += (win ? '#else' : '#ifndef _WIN32') + '\n';
+        else tmp += (win ? '#else' : '#ifndef _WIN32') + '\n';
+
+      if (add) lines(unix.split('\n'));
+        else tmp += unix + (unix.endsWith('\n') ? '' : '\n');
+    }
+
+    if (win || unix)
+      if (add) out += '#endif\n';
+        else tmp += '#endif\n';
+
+    return tmp;
+  };
+
+  for (const f of funcs) {
+    depth = 1;
+
+    const invLocals = inv(f.locals, x => x.idx);
+    if (f.returns.length > 1) todo('funcs returning >1 value unsupported');
+
+    const sanitize = str => str.replace(/[^0-9a-zA-Z_]/g, _ => String.fromCharCode(97 + _.charCodeAt(0) % 32));
+
+    const returns = f.returns.length === 1;
+
+    const shouldInline = f.internal;
+    out += `${f.name === 'main' ? 'int' : CValtype[f.returns[0]]} ${shouldInline ? 'inline ' : ''}${sanitize(f.name)}(${f.params.map((x, i) => `${CValtype[x]} ${invLocals[i]}`).join(', ')}) {\n`;
+
+    const localKeys = Object.keys(f.locals).sort((a, b) => f.locals[a].idx - f.locals[b].idx).slice(f.params.length).sort((a, b) => f.locals[a].idx - f.locals[b].idx);
+    for (const x of localKeys) {
+      const l = f.locals[x];
+      line(`${CValtype[l.type]} ${x} = 0`);
+    }
+
+    if (localKeys.length !== 0) out += '\n';
+
+    let vals = [];
+    const endNeedsCurly = [], ignoreEnd = [];
+    let beginLoop = false, lastCond = false, ifTernary = false;
+    for (let _ = 0; _ < f.wasm.length; _++) {
+      const i = f.wasm[_];
+
+      if (invOperatorOpcode[i[0]]) {
+        const b = vals.pop();
+        const a = vals.pop();
+
+        let op = invOperatorOpcode[i[0]];
+        if (op.length === 3) op = op.slice(0, 2);
+
+        if (['==', '!=', '>', '>=', '<', '<='].includes(op)) lastCond = true;
+          else lastCond = false;
+
+        // vals.push(`(${removeBrackets(a)} ${op} ${b})`);
+        vals.push(`(${a} ${op} ${b})`);
+        continue;
+      }
+
+      // misc insts
+      if (i[0] === 0xfc) {
+        switch (i[1]) {
+          // i32_trunc_sat_f64_s
+          case 0x02:
+            vals.push(`(${CValtype.i32})${vals.pop()}`);
+            break;
+
+          // i32_trunc_sat_f64_u
+          case 0x03:
+            vals.push(`(${CValtype.i32})(${CValtype.i32_u})${vals.pop()}`);
+            break;
+        }
+
+        lastCond = false;
+        continue;
+      }
+
+      switch (i[0]) {
+        case Opcodes.i32_const:
+        case Opcodes.i64_const:
+          vals.push(read_signedLEB128(i.slice(1)).toString());
+          break;
+
+        case Opcodes.f64_const:
+          vals.push(read_ieee754_binary64(i.slice(1)).toExponential());
+          break;
+
+        case Opcodes.local_get:
+          vals.push(`${invLocals[i[1]]}`);
+          break;
+
+        case Opcodes.local_set:
+          line(`${invLocals[i[1]]} = ${removeBrackets(vals.pop())}`);
+          break;
+
+        case Opcodes.local_tee:
+          // line(`${invLocals[i[1]]} = ${removeBrackets(vals.pop())}`);
+          // vals.push(`${invLocals[i[1]]}`);
+          vals.push(`((${invLocals[i[1]]} = ${vals.pop()}))`);
+          break;
+
+        case Opcodes.global_get:
+          vals.push(`${invGlobals[i[1]]}`);
+          break;
+
+        case Opcodes.global_set:
+          line(`${invGlobals[i[1]]} = ${removeBrackets(vals.pop())}`);
+          break;
+
+        case Opcodes.f64_trunc:
+          // vals.push(`trunc(${vals.pop()})`);
+          vals.push(`(int)(${removeBrackets(vals.pop())})`); // this is ~10x faster with clang??
+          break;
+
+        case Opcodes.f64_convert_i32_u:
+        case Opcodes.f64_convert_i32_s:
+        case Opcodes.f64_convert_i64_u:
+        case Opcodes.f64_convert_i64_s:
+          // int to double
+          vals.push(`(double)${vals.pop()}`);
+          break;
+
+        case Opcodes.return:
+          line(`return${returns ? ` ${removeBrackets(vals.pop())}` : ''}`);
+          break;
+
+        case Opcodes.if:
+          let cond = removeBrackets(vals.pop());
+          if (!lastCond) {
+            if (cond.startsWith('(long)')) cond = `${cond.slice(6)} == 1e+0`;
+              else cond += ' == 1';
+          }
+
+          ifTernary = i[1] !== Blocktype.void;
+          if (ifTernary) {
+            ifTernary = cond;
+            break;
+          }
+
+          if (beginLoop) {
+            beginLoop = false;
+            line(`while (${cond}) {`, false);
+
+            depth++;
+            endNeedsCurly.push(true);
+            ignoreEnd.push(false, true);
+            break;
+          }
+
+          line(`if (${cond}) {`, false);
+
+          depth++;
+          endNeedsCurly.push(true);
+          ignoreEnd.push(false);
+          break;
+
+        case Opcodes.else:
+          if (ifTernary) break;
+
+          depth--;
+          line(`} else {`, false);
+          depth++;
+          break;
+
+        case Opcodes.loop:
+          // not doing properly, fake a while loop
+          beginLoop = true;
+          break;
+
+        case Opcodes.end:
+          if (ignoreEnd.pop()) break;
+
+          if (ifTernary) {
+            const b = vals.pop();
+            const a = vals.pop();
+            vals.push(`${ifTernary} ? ${a} : ${b}`);
+            break;
+          }
+
+          depth--;
+          if (endNeedsCurly.pop() === true) line('}', false);
+          break;
+
+        case Opcodes.call:
+          let func = funcs.find(x => x.index === i[1]);
+          if (!func) {
+            const importFunc = importFuncs[i[1]];
+            switch (importFunc.name) {
+              case 'print':
+                line(`printf("%f\\n", ${vals.pop()})`);
+                includes.set('stdio.h', true);
+                break;
+              case 'printChar':
+                line(`printf("%c", (int)(${vals.pop()}))`);
+                includes.set('stdio.h', true);
+                break;
+
+              case 'time':
+                line(`double _time_out`);
+                /* platformSpecific(
+`FILETIME _time_filetime;
+GetSystemTimeAsFileTime(&_time_filetime);
+
+ULARGE_INTEGER _time_ularge;
+_time_ularge.LowPart = _time_filetime.dwLowDateTime;
+_time_ularge.HighPart = _time_filetime.dwHighDateTime;
+_time_out = (_time_ularge.QuadPart - 116444736000000000i64) / 10000.;`,
+`struct timespec _time;
+clock_gettime(CLOCK_MONOTONIC, &_time);
+_time_out = _time.tv_nsec / 1000000.;`); */
+                platformSpecific(
+`LARGE_INTEGER _time_freq, _time_t;
+QueryPerformanceFrequency(&_time_freq);
+QueryPerformanceCounter(&_time_t);
+_time_out = ((double)_time_t.QuadPart / _time_freq.QuadPart) * 1000.;`,
+`struct timespec _time;
+clock_gettime(CLOCK_MONOTONIC, &_time);
+_time_out = _time.tv_nsec / 1000000. + _time.tv_sec * 1000.;`);
+                vals.push(`_time_out`);
+
+                unixIncludes.set('time.h', true);
+                winIncludes.set('windows.h', true);
+                break;
+
+              default:
+                log('2c', `unimplemented import: ${importFunc.name}`);
+                break;
+            }
+
+            break;
+          }
+
+          let args = [];
+          for (let j = 0; j < func.params.length; j++) args.unshift(removeBrackets(vals.pop()));
+
+          if (func.returns.length === 1) vals.push(`${sanitize(func.name)}(${args.join(', ')})`)
+            else line(`${sanitize(func.name)}(${args.join(', ')})`);
+
+          break;
+
+        case Opcodes.drop:
+          line(vals.pop());
+          break;
+
+        case Opcodes.br:
+          // ignore
+          // reset "stack"
+          vals = [];
+          break;
+
+        default:
+          log('2c', `unimplemented op: ${invOpcodes[i[0]]}`);
+          // todo(`unimplemented op: ${invOpcodes[i[0]]}`);
+      }
+
+      lastCond = false;
+    }
+
+    if (vals.length === 1 && returns) {
+      line(`return ${vals.pop()}`);
+    }
+
+    out += '}\n\n';
+  }
+
+  depth = 0;
+
+  const makeIncludes = includes => [...includes.keys()].map(x => `#include <${x}>\n`).join('');
+
+  out = platformSpecific(makeIncludes(winIncludes), makeIncludes(unixIncludes), false) + '\n' + makeIncludes(includes) + '\n' + out;
+
+  return out;
+};

package/compiler/builtins.js

@@ -1,6 +1,5 @@
 import { Blocktype, Opcodes, Valtype } from "./wasmSpec.js";
 import { number, i32x4 } from "./embedding.js";
-import { signedLEB128 } from "./encoding.js";
 
 export const importedFuncs = [
   {
@@ -416,13 +415,13 @@ export const BuiltinFuncs = function() {
     wasm: [
       // setup: s1 = state0, s0 = state1, state0 = s0
       [ Opcodes.global_get, 0 ], // state0
-      [ Opcodes.local_set, 0 ], // s1
+      [ Opcodes.local_tee, 0 ], // s1
       [ Opcodes.global_get, 1 ], // state1
       [ Opcodes.local_tee, 1, ], // s0
       [ Opcodes.global_set, 0 ], // state0
 
       // s1 ^= s1 << 23
-      [ Opcodes.local_get, 0 ], // s1
+      // [ Opcodes.local_get, 0 ], // s1
       [ Opcodes.local_get, 0 ], // s1
       [ Opcodes.i64_const, 23 ],
       [ Opcodes.i64_shl ], // <<
@@ -452,20 +451,24 @@ export const BuiltinFuncs = function() {
 
       // you thought it was over? now we need the result as a f64 between 0-1 :)
 
-      // mantissa = (state1 + s0) & ((1 << 53) - 1)
+      // state1 + s0
       [ Opcodes.global_get, 1 ], // state1
       [ Opcodes.local_get, 1 ], // s0
       [ Opcodes.i64_add ],
 
-      [ Opcodes.i64_const, ...signedLEB128((1 << 53) - 1) ],
-      [ Opcodes.i64_and ],
+      // should we >> 12 here?
+      // it feels like it but it breaks values
+
+      // | 0x3FF0000000000000
+      [ Opcodes.i64_const, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0xf8, 0x3f ],
+      [ Opcodes.i64_or ],
 
-      // double(mantissa)
-      [ Opcodes.f64_convert_i64_u ],
+      // bit cast as f64
+      [ Opcodes.f64_reinterpret_i64 ],
 
-      // / (1 << 53)
-      ...number(1 << 53),
-      [ Opcodes.f64_div ]
+      // - 1
+      ...number(1),
+      [ Opcodes.f64_sub ],
     ]
   };
 
@@ -568,7 +571,6 @@ export const BuiltinFuncs = function() {
     params: [ Valtype.i32 ],
     locals: [],
     returns: [ Valtype.v128 ],
-    memory: true,
     wasm: [
       [ Opcodes.local_get, 0 ],
       [ ...Opcodes.v128_load, 0, 0 ]