@suchipi/quickjs 0.5.0 → 0.7.0

package/README.md

@@ -1,8 +1,21 @@
 # suchipi/quickjs
 
-Fork of the fantastic QuickJS engine by Fabrice Bellard, with the following changes:
+Fork of the fantastic QuickJS engine by Fabrice Bellard, with many changes.
 
-## Changes to `quickjs`:
+## High-level List of Most Notable Changes
+
+- APIs from 'std' and 'os' modules were changed to be more idiomatic-to-JS; notably, they throw Errors on failure instead of returning null or errno numbers.
+- TypeScript-format type interface files (`.d.ts` files) were added for everything.
+- Hooks have been added to the module loader that make its functionality more customizable.
+- Some ES2022 features were added, and some non-standard ECMAScript proposals and extensions were added.
+- The way the project is organized and built was changed dramatically.
+- Several new JS bindings for C APIs have been added, such as `strftime`, `access`, `fsync`, `setvbuf`, `getuid`...
+- Scripts were added that cross-compile binaries for several operating systems and architectures
+- FreeBSD support added
+
+## Detailed List of Changes
+
+### Changes to `quickjs`:
 
 - A TypeScript `.d.ts` file is provided for all QuickJS-specific APIs (operator overloading APIs, BigInt extensions, BigFloat, BigDecimal, etc).
 - Non-standard `Object.toPrimitive` added (static method that invokes ToPrimitive on the given value, using the optionally-provided hint).
@@ -23,8 +36,9 @@ Fork of the fantastic QuickJS engine by Fabrice Bellard, with the following chan
   - `JS_FreezeObjectValue` (performs Object.freeze)
   - `JS_IsPrimitive`
 - ModuleDefs now have an optional "user_data" property (pointer to void) which can be accessed during module initialization (via `JS_GetModuleUserData` and `JS_SetModuleUserData`)
+- Added `JS_SetContextOpaqueValue` and `JS_GetContextOpaqueValue`, which let you associate a JSValue with a JSContext, which will be garbage-collected when that JSContext is garbage-collected.
 
-## Changes to `quickjs-libc`:
+### Changes to `quickjs-libc`:
 
 - `std` and `os` builtin modules are now namespaced under "quickjs:". In other words, you have to import them as "quickjs:std" and "quickjs:os".
 - APIs in `std` and `os` no longer return errno anywhere; instead, Error objects are thrown. `errno` is available as a property on the thrown Error objects.
@@ -35,6 +49,7 @@ Fork of the fantastic QuickJS engine by Fabrice Bellard, with the following chan
 - `os.access` function added (wrapper for libc `access`).
 - `FILE.prototype.sync` method added (wrapper for `fsync`).
 - `FILE.prototype.setvbuf` method added (wrapper for `setvbuf`).
+- `FILE.prototype.writeTo` method added (pipe data from one FILE to another, given a buffer size and limit).
 - `std.isFILE` function added (returns whether the provided object is a `FILE` (via `js_std_file_class_id`)).
 - `os.{WUNTRACED,WEXITSTATUS,WTERMSIG,WSTOPSIG,WIFEXITED,WIFSIGNALED,WIFSTOPPED,WIFCONTINUED}` added, for working with `os.waitpid`.
 - `os.{S_IRWXU,S_IRUSR,S_IWUSR,S_IXUSR,S_IRWXG,S_IRGRP,S_IWGRP,S_IXGRP,S_IRWXO,S_IROTH,S_IWOTH,S_IXOTH}` added, for working with file modes.
@@ -42,24 +57,25 @@ Fork of the fantastic QuickJS engine by Fabrice Bellard, with the following chan
 - `std.strftime` added (wrapper for libc `strftime`).
 - Added `std.getuid`, `std.geteuid`, `std.getgid`, and `std.getegid` (wrappers for the libc functions of the same names).
 - Most module-loading-related code was moved into `quickjs-modulesys`.
-- Synchronous import function added (`std.importModule`), which provides the same module record object you would get via dynamic (async) import.
-- JS api for using the engine's configured module name normalization function was added (`std.resolveModule`).
 - `setTimeout` and `clearTimeout` are now available as globals (previously they were only available as exports).
 - `setInterval` and `clearInterval` are added, available as globals.
 - `String.dedent` added (template tag function, like the proposed [String.dedent](https://github.com/tc39/proposal-string-dedent)).
 - Several C-side helper functions were moved out of quickjs-libc and into quickjs-utils.
 - Most module-related code (setting import.meta, etc) was moved into quickjs-modulesys.
 - Added `std.setExitCode`, `std.getExitCode`, and made `std.exit`'s parameter optional. The value passed to `std.setExitCode` will be used when the process exits normally, or when `std.exit` is called without any arguments. `std.setExitCode`, `std.getExitCode`, and `std.exit` throw if called from a thread other than the main thread (ie. a Worker).
+- The manual garbage collection function `std.gc()` was moved to `"quickjs:engine"`.
 
-## Changes to the `qjs` repl:
+### Changes to the `qjs` binary:
 
-- The new `inspect` global is used to print results
-- The last error is accessible via the `_error` global
-- If a thrown error has additional properties added onto it, those are printed along with the thrown error
-- Ctrl+W deletes words backwards from the cursor position
 - Blocking the main thread is allowed
+- `-m` (module mode) now affects eval strings passed to `-e`
+- Changes to the repl:
+  - Ctrl+W deletes words backwards from the cursor position
+  - The last error is accessible via the `_error` global
+  - If a thrown error has additional properties added onto it, those are printed along with the thrown error
+  - The new `inspect` global is used to print results
 
-## New binary: `qjsbootstrap`:
+### New binary: `qjsbootstrap`:
 
 Appending some JS code to the end of this binary changes it into a binary that executes that JS code:
 
@@ -74,23 +90,23 @@ You can use this to create distributable binaries that run JS code without needi
 
 > Note: On FreeBSD, `qjsbootstrap` requires procfs. You can mount it with `mount -t procfs proc /proc`. I started some work to use libprocstat instead, but haven't completed it yet.
 
-## New binary: `quickjs-run`:
+### New binary: `quickjs-run`:
 
 Barebones binary for running files, without any of the arg parsing logic from qjs. Good for testing some unusual cases, or writing programs with custom argv parsing logic.
 
-## New module: "quickjs:bytecode"
+### New module: "quickjs:bytecode"
 
 A Module that exposes QuickJS's value <-> bytecode (de)serialization APIs to JavaScript code. Generated bytecode can be combined with `qjsbootstrap-bytecode` in the same way that source code strings can be combined with `qjsbootstrap`.
 
-## New module: "quickjs:context"
+### New module: "quickjs:context"
 
 A Module that allows JS code to create new JS Contexts (Realms). You can create new Contexts and run code inside them. Contexts can have certain features disabled (like eval) for security purposes. You can share values between Contexts. Contexts are destroyed when they get garbage-collected.
 
-## New module: "quickjs:pointer"
+### New module: "quickjs:pointer"
 
 A barebones Module that exports a JS class which can be used to represent an opaque pointer. C modules can use the `js_new_pointer` function provided by this module to pass opaque pointer handles to users without needing to make their own wrapper class for stuff. This is mostly just useful in order to have a codified convention for how FFI libraries and such should represent foreign pointers.
 
-## New library: `quickjs-utils`
+### New library: `quickjs-utils`
 
 Helper structs, functions, and macros that make it easier to work with QuickJS in C code.
 
@@ -98,55 +114,63 @@ Helper structs, functions, and macros that make it easier to work with QuickJS i
 - Helper function for loading a file from disk into char a buffer
 - Helper functions for printing JS errors to stderr
 
-## New library: `quickjs-modulesys`
+### New module: "quickjs:engine"
 
-- Module-loading code in `quickjs-libc` was moved into `quickjs-modulesys`, with the following changes:
-  - `.js` extensions can now be omitted from import specifiers; they're optional.
-  - If your import specifier points to a folder, it will attempt to load `index.js` from that folder.
+This module contains APIs related to engine internals like script execution, module loading, code eval, filename reflection, and garbage collection. Several parts of quickjs-libc were moved here so that quickjs-libc could be focused on "C standard library" bindings.
+
+### Changes to the module loader
+
+- `.js` extensions can now be omitted from import specifiers; they're optional.
+- If your import specifier points to a folder, it will attempt to load `index.js` from that folder.
 - Adds the global `require`, a CommonJS-like synchronous module loading function.
   - The `require` function is not fully CommonJS-compliant; for instance, `require.main` is not present. `require.resolve` is, though.
 - Adds `import.meta.require`
   - It's the same as the global `require`; it's just added to import.meta for compatibility with bundlers that output `import.meta.require`, like `bun`.
 - Adds `import.meta.resolve`
   - Similar to [the one in the browser](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve#specifications), but it's actually just `require.resolve` exposed via `import.meta`.
+- Module and eval helpers have been moved from "quickjs:std" to the new module "quickjs:engine".
+- Makes the module loader's resolution and loading behavior configurable
+  - The module "quickjs:engine" exports an object called `ModuleDelegate`.
+  - You can specify additional implicit import specifier extensions by adding to the `ModuleDelegate.searchExtensions` array.
+  - You can transform any file prior to evaluating it as a module by adding a function to the `ModuleDelegate.compilers` object. Useful for compile-to-ts languages like TypeScript, Coffeescript, etc.
+  - You can override module name normalization (aka module resolution) by replacing the `ModuleDelegate.resolve` function.
+    - Note that you must handle `ModuleDelegate.searchExtensions` yourself in your replacement implementation.
+  - You can override the method used to load modules by replacing the `ModuleDelegate.read` function.
+    - Note that you must handle `ModuleDelegate.compilers` yourself in your replacement implementation.
 - Makes `import.meta.main` configurable
-  - You can use `std.setMainModule` to make `import.meta.main` true within that module's code. Note, however, that it does not work retroactively; only modules loaded after the `setMainModule` call will be affected. To defer module load, use `import()`, `std.importModule`, or `require`.
-  - You can use `std.isMainModule` to check if a given module would be the main module without loading it.
-- Adds the global object `Module`.
-  - `instanceof Module` can be used to identify module namespace objects.
-  - You can specify additional implicit import specifier extensions by adding to the `Module.searchExtensions` array.
-  - You can transform any file prior to evaluating it as a module by adding a function to the `Module.compilers` object. Useful for compile-to-ts languages like TypeScript, Coffeescript, etc.
-  - You can define custom builtin modules using the `Module.define` function.
-  - You can override module name normalization (aka module resolution) by replacing the `Module.resolve` function.
-    - Note that you must handle `Module.searchExtensions` yourself in your replacement implementation.
-  - You can override the method used to load modules by replacing the `Module.read` function.
-    - Note that you must handle `Module.compilers` yourself in your replacement implementation.
-- The `eval_*` functions that were duplicated in each of the programs (`eval_buf`, `eval_file`, and `eval_binary`) were moved here
-- The default module loader function from quickjs-libc was moved here and augmented to support the features related to the `Module` global as mentioned above
-- When using `require` to load a module which contains an export named `__cjsExports`, the value of the `__cjsExports` property will be returned from `require` instead of the usual module namespace object. This can be leveraged by user-defined properties on `Module` to add some CommonJS <-> ESM interop. Note, however, that dynamic import and `std.importModule` always receive the usual module namespace object.
-
-## Changes to project organization
+  - The module "quickjs:engine" exports two functions named `setMainModule` and `isMainModule`.
+  - You can use `setMainModule` to make `import.meta.main` true within that module's code. Note, however, that it does not work retroactively; only modules loaded after the `setMainModule` call will be affected. To defer module load, use `import()`, `importModule` from "quickjs:engine", or `require`.
+  - You can use `isMainModule` to check if a given module would be the main module without loading it.
+- New `isModuleNamespace` function lets users identify module namespace objects
+- New `defineBuiltinModule` function lets users add their own builtin modules
+- When using `require` to load a module which contains an export named `__cjsExports`, the value of the `__cjsExports` property will be returned from `require` instead of the usual module namespace object. This can be leveraged by users configuring the module loader to add some CommonJS <-> ESM interop. Note, however, that dynamic import and `"quickjs:engine"`'s `importModule` always receive the usual module namespace object.
+- Synchronous import function added (`importModule`), which provides the same module record object you would get via dynamic (async) import.
+- JS api for using the engine's configured module name normalization function was added (`resolveModule`).
+
+### Changes to project organization
 
 - Stuff is reorganized into separate folders under `src`.
 - Ninja is used instead of make. Ninja build config is generated via `.ninja.js` files which get loaded into [@suchipi/shinobi](https://github.com/suchipi/shinobi).
 - Line endings have been made consistent and trailing whitespace has been removed
 - The tests are authored in a new format which leverages jest snapshot testing.
+- Some parts of `quickjs-libc` were moved into `quickjs-modulesys` and `quickjs-libengine`.
+- The `eval_*` functions that were duplicated in each of the programs (`eval_buf`, `eval_file`, and `eval_binary`) were deduplicated and moved into `quickjs-modulesys`.
 
-## More target OSes/runtimes
+### More target OSes/runtimes
 
 We now include support for more platforms, and cross-compilation scripts to build most of those platforms from any platform where Docker is available.
 
 See the `meta/ninja/envs` folder to see all the supported platforms. The `host` folder represents the machine where compilation is taking place, and the `target` folder represents the target platform for the output binaries.
 
-## Library Archives
+### Library Archives
 
 We create `.a` files containing all of quickjs as part of the build.
 
-## Other changes
+### Other changes
 
 There are also probably some other miscellaneous changes I forgot to write down in the README.
 
-# Compiling
+## Compiling
 
 The repo has stuff set up to compile quickjs binaries for:
 
@@ -164,9 +188,9 @@ QuickJS itself has no external dependencies outside this repo except pthreads, a
 
 Linux, macOS, iOS, and Windows binaries can be compiled using Docker. Or, you can compile binaries for just your own unix system, without using Docker.
 
-If you're not gonna use Docker, you'll need to install [Ninja](https://ninja-build.org/) and [Node.js](https://nodejs.org/) in order to compile. I use Ninja 1.10.1 and Node.js 18.12.1, but it should work with most versions of both of those.
+If you're not gonna use Docker, you'll need to install [Ninja](https://ninja-build.org/) and [Node.js](https://nodejs.org/) in order to compile. I use Ninja 1.10.1 and Node.js 18.18.0, but it should work with most versions of both of those.
 
-## Compilation Instructions
+### Compilation Instructions
 
 To compile binaries for Linux, macOS, iOS, and Windows (using Docker):
 
@@ -177,7 +201,7 @@ To compile binaries for Linux, macOS, iOS, and Windows (using Docker):
 
 Or, to compile binaries for just your own unix system:
 
-- Make sure you have both [Ninja](https://ninja-build.org/) and [Node.js](https://nodejs.org/) installed. I use Ninja 1.10.1 and Node.js 18.12.1, but it should work with most versions of both of those.
+- Make sure you have both [Ninja](https://ninja-build.org/) and [Node.js](https://nodejs.org/) installed. I use Ninja 1.10.1 and Node.js 18.18.0, but it should work with most versions of both of those.
 - Clone the repo and cd to its folder
 - Run `meta/build.sh`
 - Build artifacts will be placed in the `build` folder. You're probably most interested in stuff in the `build/bin` and `build/lib` folders.

package/build/aarch64-apple-darwin/bin/file-to-bytecode.js

@@ -11,7 +11,8 @@ import * as std from "quickjs:std";
 import * as Bytecode from "quickjs:bytecode";
 
 function main() {
-  const [_quickjsRun, _thisScript, inputFile, outputFile] = scriptArgs;
+  let [_quickjsRun, _thisScript, inputFile, outputFile, encodedFileName] =
+    scriptArgs;
 
   if (!(inputFile && outputFile)) {
     throw new Error(
@@ -19,7 +20,11 @@ function main() {
     );
   }
 
-  const bytecode = Bytecode.fromFile(inputFile);
+  if (!encodedFileName) {
+    encodedFileName = inputFile;
+  }
+
+  const bytecode = Bytecode.fromFile(inputFile, { encodedFileName });
   const out = std.open(outputFile, "wb");
   out.write(bytecode, 0, bytecode.byteLength);
 }

package/build/aarch64-unknown-linux-gnu/bin/file-to-bytecode.js

@@ -11,7 +11,8 @@ import * as std from "quickjs:std";
 import * as Bytecode from "quickjs:bytecode";
 
 function main() {
-  const [_quickjsRun, _thisScript, inputFile, outputFile] = scriptArgs;
+  let [_quickjsRun, _thisScript, inputFile, outputFile, encodedFileName] =
+    scriptArgs;
 
   if (!(inputFile && outputFile)) {
     throw new Error(
@@ -19,7 +20,11 @@ function main() {
     );
   }
 
-  const bytecode = Bytecode.fromFile(inputFile);
+  if (!encodedFileName) {
+    encodedFileName = inputFile;
+  }
+
+  const bytecode = Bytecode.fromFile(inputFile, { encodedFileName });
   const out = std.open(outputFile, "wb");
   out.write(bytecode, 0, bytecode.byteLength);
 }

package/build/aarch64-unknown-linux-musl/bin/file-to-bytecode.js

@@ -11,7 +11,8 @@ import * as std from "quickjs:std";
 import * as Bytecode from "quickjs:bytecode";
 
 function main() {
-  const [_quickjsRun, _thisScript, inputFile, outputFile] = scriptArgs;
+  let [_quickjsRun, _thisScript, inputFile, outputFile, encodedFileName] =
+    scriptArgs;
 
   if (!(inputFile && outputFile)) {
     throw new Error(
@@ -19,7 +20,11 @@ function main() {
     );
   }
 
-  const bytecode = Bytecode.fromFile(inputFile);
+  if (!encodedFileName) {
+    encodedFileName = inputFile;
+  }
+
+  const bytecode = Bytecode.fromFile(inputFile, { encodedFileName });
   const out = std.open(outputFile, "wb");
   out.write(bytecode, 0, bytecode.byteLength);
 }

package/build/aarch64-unknown-linux-static/bin/file-to-bytecode.js

@@ -11,7 +11,8 @@ import * as std from "quickjs:std";
 import * as Bytecode from "quickjs:bytecode";
 
 function main() {
-  const [_quickjsRun, _thisScript, inputFile, outputFile] = scriptArgs;
+  let [_quickjsRun, _thisScript, inputFile, outputFile, encodedFileName] =
+    scriptArgs;
 
   if (!(inputFile && outputFile)) {
     throw new Error(
@@ -19,7 +20,11 @@ function main() {
     );
   }
 
-  const bytecode = Bytecode.fromFile(inputFile);
+  if (!encodedFileName) {
+    encodedFileName = inputFile;
+  }
+
+  const bytecode = Bytecode.fromFile(inputFile, { encodedFileName });
   const out = std.open(outputFile, "wb");
   out.write(bytecode, 0, bytecode.byteLength);
 }

package/build/dts/quickjs-libbytecode.d.ts

@@ -6,7 +6,11 @@ declare module "quickjs:bytecode" {
    */
   export function fromFile(
     path: string,
-    options?: { byteSwap?: boolean; sourceType?: "module" | "script" }
+    options?: {
+      byteSwap?: boolean;
+      sourceType?: "module" | "script";
+      encodedFileName?: string;
+    }
   ): ArrayBuffer;
 
   /**

package/build/dts/quickjs-libc.d.ts

@@ -98,6 +98,20 @@ declare interface FILE {
   /** Write `length` bytes from the ArrayBuffer `buffer` at byte position `position` into the file (wrapper to the libc `fwrite`). Returns the number of bytes written. */
   write(buffer: ArrayBuffer, position: number, length: number): number;
 
+  /**
+   * Write this file into `target`, using a memory buffer of size `bufferSize`.
+   *
+   * If `limit` is specified, only that amount of bytes will be read and
+   * written. Otherwise, data is read and written until this file reaches EOF.
+   *
+   * A `limit` of 0 is treated the same as not specifying a limit.
+   *
+   * Internally, this function uses libc `fread` and `fwrite` in a loop.
+   *
+   * Returns the number of bytes read and written.
+   */
+  writeTo(target: FILE, bufferSize: number, limit?: number): number;
+
   /**
    * Return the next line from the file, assuming UTF-8 encoding, excluding the trailing line feed or EOF.
    *
@@ -257,9 +271,6 @@ declare module "quickjs:std" {
   /** Constant for {@link FILE.setvbuf}. Declares that the buffer mode should be 'no buffering'. */
   export var _IONBF: number;
 
-  /** Manually invoke the cycle removal algorithm (garbage collector). The cycle removal algorithm is automatically started when needed, so this function is useful in case of specific memory constraints or for testing. */
-  export function gc(): void;
-
   /** Return the value of the environment variable `name` or `undefined` if it is not defined. */
   export function getenv(name: string): string | undefined;
 
@@ -910,7 +921,7 @@ declare module "quickjs:os" {
   export function dup2(oldfd: number, newfd: number): number;
 
   /** `pipe` Unix system call. Return two handles as `[read_fd, write_fd]`. */
-  export function pipe(): null | [number, number];
+  export function pipe(): [number, number];
 
   /** Sleep for `delay_ms` milliseconds. */
   export function sleep(delay_ms: number): void;

package/build/dts/quickjs-libcontext.d.ts

@@ -137,7 +137,7 @@ declare module "quickjs:context" {
       console?: boolean;
       /** Enables `print`. Defaults to `true`. */
       print?: boolean;
-      /** Enables `require` and `Module`. Defaults to `true`. */
+      /** Enables `require`. Defaults to `true`. */
       moduleGlobals?: boolean;
       /**
        * Enables `setTimeout`, `clearTimeout`, `setInterval`, and
@@ -155,8 +155,8 @@ declare module "quickjs:context" {
         "quickjs:bytecode"?: boolean;
         /** Enables the "quickjs:context" module. Defaults to `true`. */
         "quickjs:context"?: boolean;
-        /** Enables the "quickjs:module" module. Defaults to `true`. */
-        "quickjs:module"?: boolean;
+        /** Enables the "quickjs:engine" module. Defaults to `true`. */
+        "quickjs:engine"?: boolean;
       };
     });
 

package/build/dts/{quickjs-libmodule.d.ts → quickjs-libengine.d.ts}

@@ -1,4 +1,4 @@
-declare module "quickjs:module" {
+declare module "quickjs:engine" {
   /**
    * Return whether the provided resolved module path is set as the main module.
    *
@@ -69,4 +69,32 @@ declare module "quickjs:module" {
    * @param stackLevels - How many levels up the stack to search for a filename. Defaults to 0, which uses the current stack frame.
    */
   export function getFileNameFromStack(stackLevels?: number): string;
+
+  /**
+   * Returns true if `target` is a module namespace object.
+   */
+  export function isModuleNamespace(target: any): boolean;
+
+  /**
+   * Create a virtual built-in module whose exports consist of the own
+   * enumerable properties of `obj`.
+   */
+  export function defineBuiltinModule(
+    name: string,
+    obj: { [key: string]: any }
+  ): void;
+
+  /**
+   * An object which lets you configure the module loader (import/export/require).
+   * You can change these properties to add support for importing new filetypes.
+   */
+  export const ModuleDelegate: ModuleDelegate;
+
+  /**
+   * Manually invoke the cycle removal algorithm (garbage collector).
+   *
+   * The cycle removal algorithm is automatically started when needed, so this
+   * function is useful in case of specific memory constraints or for testing.
+   */
+  export function gc(): void;
 }