koffi 1.2.4 → 1.3.0-rc.1

package/doc/_static/custom.css

@@ -0,0 +1,22 @@
+/* This program is free software: you can redistribute it and/or modify
+   it under the terms of the GNU Affero General Public License as published by
+   the Free Software Foundation, either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU Affero General Public License for more details.
+
+   You should have received a copy of the GNU Affero General Public License
+   along with this program. If not, see https://www.gnu.org/licenses/. */
+
+aside.footnote {
+    display: flex;
+    margin-top: 6px;
+}
+
+aside.footnote > p {
+    margin: 0;
+    margin-left: 1rem;
+}

package/doc/benchmarks.md

@@ -0,0 +1,113 @@
+# Benchmarks
+
+Here is a quick overview of the execution time of Koffi calls on three test cases (one based around rand, one based on atoi and one based on Raylib) compared to theoretical ideal implementations.
+
+<table style="margin: 0 auto;">
+    <tr>
+        <td><img src="_static/bench_linux.png" alt="Linux performance" style="width: 350px;"/></td>
+        <td><img src="_static/bench_windows.png" alt="Windows performance" style="width: 350px;"/></td>
+    </tr>
+</table>
+
+These results are detailed and explained below, and compared to node-ffi/node-ffi-napi.
+
+## rand results
+
+This test is based around repeated calls to a simple standard C function atoi, and has three implementations:
+
+- the first one is the reference, it calls atoi through an N-API module, and is close to the theoretical limit of a perfect (no overhead) Node.js > C FFI implementation.
+- the second one calls atoi through Koffi
+- the third one uses the official Node.js FFI implementation, node-ffi-napi
+
+Because rand is a pretty small function, the FFI overhead is clearly visible.
+
+### Linux x86_64
+
+The results below were measured on my x86_64 Linux machine (AMD® Ryzen™ 7 4700U):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+rand_napi     | 20000000   | 1.44s       | (baseline)
+rand_koffi    | 20000000   | 2.60s       | x1.81
+rand_node_ffi | 20000000   | 107.58s     | x75
+
+### Windows x86_64
+
+The results below were measured on my x86_64 Windows machine (Intel® Core™ i5-4460):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+rand_napi     | 20000000   | 2.10s       | (baseline)
+rand_koffi    | 20000000   | 3.87s       | x1.84
+rand_node_ffi | 20000000   | 87.84s      | x42
+
+## atoi results
+
+This test is similar to the rand one, but it is based on atoi, which takes a string parameter. Javascript (V8) to C string conversion is relatively slow and heavy.
+
+Because rand is a pretty small function, the FFI overhead is clearly visible.
+
+### Linux x86_64
+
+The results below were measured on my x86_64 Linux machine (AMD® Ryzen™ 7 4700U):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+atoi_napi     | 20000000   | 2.97s       | (baseline)
+atoi_koffi    | 20000000   | 5.07s       | x1.71
+atoi_node_ffi | 20000000   | 693.16s     | x233
+
+### Windows x86_64
+
+The results below were measured on my x86_64 Windows machine (Intel® Core™ i5-4460):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+atoi_napi     | 20000000   | 2.97s       | (baseline)
+atoi_koffi    | 20000000   | 5.91s       | x1.99
+atoi_node_ffi | 20000000   | 479.34s     | x161
+
+## Raylib results
+
+This benchmark uses the CPU-based image drawing functions in Raylib. The calls are much heavier than in the atoi benchmark, thus the FFI overhead is reduced. In this implementation, Koffi is compared to:
+
+- Baseline: Full C++ version of the code (no JS)
+- [node-raylib](https://github.com/RobLoach/node-raylib): This is a native wrapper implemented with N-API
+
+### Linux x86_64
+
+The results below were measured on my x86_64 Linux machine (AMD® Ryzen™ 7 4700U):
+
+Benchmark          | Iterations | Total time  | Overhead
+---------------    | ---------- | ----------- | ----------
+raylib_cc          | 100        | 9.31s       | (baseline)
+raylib_node_raylib | 100        | 10.90s      | x1.17
+raylib_koffi       | 100        | 12.86s      | x1.38
+raylib_node_ffi    | 100        | 35.76s      | x3.84
+
+### Windows x86_64
+
+The results below were measured on my x86_64 Windows machine (Intel® Core™ i5-4460):
+
+Benchmark          | Iterations | Total time  | Overhead
+---------------    | ---------- | ----------- | ----------
+raylib_cc          | 100        | 10.67s      | (baseline)
+raylib_node_raylib | 100        | 12.05s      | x1.13
+raylib_koffi       | 100        | 14.84s      | x1.39
+raylib_node_ffi    | 100        | 44.63s      | x4.18
+
+## Running benchmarks
+
+Open a console, go to `koffi/benchmark` and run `../../cnoke/cnoke.js` (or `node ..\..\cnoke\cnoke.js` on Windows) before doing anything else.
+
+```sh
+cd koffi/benchmark
+node ../../cnoke/cnoke.js
+```
+
+Once this is done, you can execute each implementation, e.g. `build/raylib_cc` or `node ./atoi_koffi.js`. You can optionally define a custom number of iterations, e.g. `node ./atoi_koffi.js 10000000`.
+
+```sh
+node ./atoi_napi.js
+node ./atoi_koffi.js
+```

package/doc/conf.py

@@ -0,0 +1,54 @@
+# -- Project information -----------------------------------------------------
+
+project = 'Koffi'
+copyright = '2022, Niels Martignène'
+author = 'Niels Martignène'
+version = '1.2.4'
+revision = '1.2.4'
+
+# -- General configuration ---------------------------------------------------
+
+extensions = [
+    'myst_parser',
+    'sphinx.ext.autosectionlabel'
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+exclude_patterns = []
+
+# -- Options for HTML output -------------------------------------------------
+
+html_title = 'Koffi'
+
+html_theme = 'furo'
+
+html_static_path = ['_static']
+
+html_theme_options = {
+    'light_css_variables': {
+        'color-brand-primary': '#FF6600',
+        'color-brand-content': '#FF6600'
+    },
+    'dark_css_variables': {
+        'color-brand-primary': '#FF6600',
+        'color-brand-content': '#FF6600'
+    }
+}
+
+html_link_suffix = ''
+
+html_css_files = ['custom.css']
+
+# -- MyST parser options -------------------------------------------------
+
+myst_enable_extensions = [
+    'linkify'
+]
+
+myst_heading_anchors = 3
+
+myst_linkify_fuzzy_links = False
+
+myst_number_code_blocks = ['c', 'js']

package/doc/contribute.md

@@ -0,0 +1,115 @@
+# Contributing
+
+## Bugs and feature requests
+
+Use the official repository (named Luigi, because this is a monorepo containing multiple projects) for bugs, ideas and features requests.
+
+Go here: https://github.com/Koromix/luigi/issues
+
+## Build from source
+
+We provide prebuilt binaries, packaged in the NPM archive, so in most cases it should be as simple as `npm install koffi`. If you want to hack Koffi or use a specific platform, follow the instructions below.
+
+### Windows
+
+First, make sure the following dependencies are met:
+
+- The "Desktop development with C++" workload from [Visual Studio 2022 or 2019](https://visualstudio.microsoft.com/downloads/) or the "C++ build tools" workload from the [Build Tools](https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022), with the default optional components.
+- [CMake meta build system](https://cmake.org/)
+- [Node.js](https://nodejs.org/) 12 or later
+
+Once this is done, run this command _from the test or the benchmark directory_ (depending on what you want to build):
+
+```sh
+cd koffi/test # or cd koffi/benchmark
+node ../../cnoke/cnoke.js
+```
+
+### Other platforms
+
+Make sure the following dependencies are met:
+
+- `gcc` and `g++` >= 8.3 or newer
+- GNU Make 3.81 or newer
+- [CMake meta build system](https://cmake.org/)
+- [Node.js](https://nodejs.org/) 12 or later
+
+Once this is done, run this command _from the test or the benchmark directory_ (depending on what you want to build):
+
+```sh
+cd koffi/test # or cd koffi/benchmark
+node ../../cnoke/cnoke.js
+```
+
+## Running tests
+
+Koffi is tested on multiple architectures using emulated (accelerated when possible) QEMU machines. First, you need to install qemu packages, such as `qemu-system` (or even `qemu-system-gui`) on Ubuntu.
+
+These machines are not included directly in this repository (for license and size reasons), but they are available here: https://koromix.dev/files/machines/
+
+For example, if you want to run the tests on Debian ARM64, run the following commands:
+
+```sh
+cd luigi/koffi/qemu/
+wget -q -O- https://koromix.dev/files/machines/qemu_debian_arm64.tar.zst | zstd -d | tar xv
+sha256sum -c --ignore-missing registry/sha256sum.txt
+```
+
+Note that the machine disk content may change each time the machine runs, so the checksum test will fail once a machine has been used at least once.
+
+And now you can run the tests with:
+
+```sh
+node qemu.js # Several options are available, use --help
+```
+
+And be patient, this can be pretty slow for emulated machines. The Linux machines have and use ccache to build Koffi, so subsequent build steps will get much more tolerable.
+
+By default, machines are started and stopped for each test. But you can start the machines ahead of time and run the tests multiple times instead:
+
+```sh
+node qemu.js start # Start the machines
+node qemu.js # Test (without shutting down)
+node qemu.js # Test again
+node qemu.js stop # Stop everything
+```
+
+You can also restrict the test to a subset of machines:
+
+```sh
+# Full test cycle
+node qemu.js test debian_x64 debian_i386
+
+# Separate start, test, shutdown
+node qemu.js start debian_x64 debian_i386
+node qemu.js test debian_x64 debian_i386
+node qemu.js stop
+```
+
+Finally, you can join a running machine with SSH with the following shortcut, if you need to do some debugging or any other manual procedure:
+
+```sh
+node qemu.js ssh debian_i386
+```
+
+Each machine is configured to run a VNC server available locally, which you can use to access the display, using KRDC or any other compatible viewer. Use the `info` command to get the VNC port.
+
+```sh
+node qemu.js info debian_x64
+```
+
+## Todo list
+
+After the release of version 1.3.0, the current priorities for the next major release are:
+
+- Automate Windows/AArch64 (qemu) and macOS/AArch64 (how?) tests and builds
+- Create a real-world example, using several libraries (Raylib, SQLite, libsodium) to illustrate how to work with various C API styles
+
+The following features are also planned eventually, not necessarily in that order:
+
+- Optimize passing of structs and arrays, with separate HFA-specific helper functions
+- Add simple struct type parser
+- Add more ways to manually encode and decode various types to and from byte arrays
+- Add support for unions
+- Provide better ways to automatically deal with caller/heap-allocated memory (strings, etc.)
+- Port Koffi to PowerPC (POWER9+) ABI

package/doc/dist/html/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 1029792c9b37058f12795e72139e7221
+tags: 645f666f9bcd5a90fca523b33c5a78b7

package/doc/dist/html/_sources/benchmarks.md.txt

@@ -0,0 +1,113 @@
+# Benchmarks
+
+Here is a quick overview of the execution time of Koffi calls on three test cases (one based around rand, one based on atoi and one based on Raylib) compared to theoretical ideal implementations.
+
+<table style="margin: 0 auto;">
+    <tr>
+        <td><img src="_static/bench_linux.png" alt="Linux performance" style="width: 350px;"/></td>
+        <td><img src="_static/bench_windows.png" alt="Windows performance" style="width: 350px;"/></td>
+    </tr>
+</table>
+
+These results are detailed and explained below, and compared to node-ffi/node-ffi-napi.
+
+## rand results
+
+This test is based around repeated calls to a simple standard C function atoi, and has three implementations:
+
+- the first one is the reference, it calls atoi through an N-API module, and is close to the theoretical limit of a perfect (no overhead) Node.js > C FFI implementation.
+- the second one calls atoi through Koffi
+- the third one uses the official Node.js FFI implementation, node-ffi-napi
+
+Because rand is a pretty small function, the FFI overhead is clearly visible.
+
+### Linux x86_64
+
+The results below were measured on my x86_64 Linux machine (AMD® Ryzen™ 7 4700U):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+rand_napi     | 20000000   | 1.44s       | (baseline)
+rand_koffi    | 20000000   | 2.60s       | x1.81
+rand_node_ffi | 20000000   | 107.58s     | x75
+
+### Windows x86_64
+
+The results below were measured on my x86_64 Windows machine (Intel® Core™ i5-4460):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+rand_napi     | 20000000   | 2.10s       | (baseline)
+rand_koffi    | 20000000   | 3.87s       | x1.84
+rand_node_ffi | 20000000   | 87.84s      | x42
+
+## atoi results
+
+This test is similar to the rand one, but it is based on atoi, which takes a string parameter. Javascript (V8) to C string conversion is relatively slow and heavy.
+
+Because rand is a pretty small function, the FFI overhead is clearly visible.
+
+### Linux x86_64
+
+The results below were measured on my x86_64 Linux machine (AMD® Ryzen™ 7 4700U):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+atoi_napi     | 20000000   | 2.97s       | (baseline)
+atoi_koffi    | 20000000   | 5.07s       | x1.71
+atoi_node_ffi | 20000000   | 693.16s     | x233
+
+### Windows x86_64
+
+The results below were measured on my x86_64 Windows machine (Intel® Core™ i5-4460):
+
+Benchmark     | Iterations | Total time  | Overhead
+------------- | ---------- | ----------- | ----------
+atoi_napi     | 20000000   | 2.97s       | (baseline)
+atoi_koffi    | 20000000   | 5.91s       | x1.99
+atoi_node_ffi | 20000000   | 479.34s     | x161
+
+## Raylib results
+
+This benchmark uses the CPU-based image drawing functions in Raylib. The calls are much heavier than in the atoi benchmark, thus the FFI overhead is reduced. In this implementation, Koffi is compared to:
+
+- Baseline: Full C++ version of the code (no JS)
+- [node-raylib](https://github.com/RobLoach/node-raylib): This is a native wrapper implemented with N-API
+
+### Linux x86_64
+
+The results below were measured on my x86_64 Linux machine (AMD® Ryzen™ 7 4700U):
+
+Benchmark          | Iterations | Total time  | Overhead
+---------------    | ---------- | ----------- | ----------
+raylib_cc          | 100        | 9.31s       | (baseline)
+raylib_node_raylib | 100        | 10.90s      | x1.17
+raylib_koffi       | 100        | 12.86s      | x1.38
+raylib_node_ffi    | 100        | 35.76s      | x3.84
+
+### Windows x86_64
+
+The results below were measured on my x86_64 Windows machine (Intel® Core™ i5-4460):
+
+Benchmark          | Iterations | Total time  | Overhead
+---------------    | ---------- | ----------- | ----------
+raylib_cc          | 100        | 10.67s      | (baseline)
+raylib_node_raylib | 100        | 12.05s      | x1.13
+raylib_koffi       | 100        | 14.84s      | x1.39
+raylib_node_ffi    | 100        | 44.63s      | x4.18
+
+## Running benchmarks
+
+Open a console, go to `koffi/benchmark` and run `../../cnoke/cnoke.js` (or `node ..\..\cnoke\cnoke.js` on Windows) before doing anything else.
+
+```sh
+cd koffi/benchmark
+node ../../cnoke/cnoke.js
+```
+
+Once this is done, you can execute each implementation, e.g. `build/raylib_cc` or `node ./atoi_koffi.js`. You can optionally define a custom number of iterations, e.g. `node ./atoi_koffi.js 10000000`.
+
+```sh
+node ./atoi_napi.js
+node ./atoi_koffi.js
+```

package/doc/dist/html/_sources/contribute.md.txt

@@ -0,0 +1,115 @@
+# Contributing
+
+## Bugs and feature requests
+
+Use the official repository (named Luigi, because this is a monorepo containing multiple projects) for bugs, ideas and features requests.
+
+Go here: https://github.com/Koromix/luigi/issues
+
+## Build from source
+
+We provide prebuilt binaries, packaged in the NPM archive, so in most cases it should be as simple as `npm install koffi`. If you want to hack Koffi or use a specific platform, follow the instructions below.
+
+### Windows
+
+First, make sure the following dependencies are met:
+
+- The "Desktop development with C++" workload from [Visual Studio 2022 or 2019](https://visualstudio.microsoft.com/downloads/) or the "C++ build tools" workload from the [Build Tools](https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022), with the default optional components.
+- [CMake meta build system](https://cmake.org/)
+- [Node.js](https://nodejs.org/) 12 or later
+
+Once this is done, run this command _from the test or the benchmark directory_ (depending on what you want to build):
+
+```sh
+cd koffi/test # or cd koffi/benchmark
+node ../../cnoke/cnoke.js
+```
+
+### Other platforms
+
+Make sure the following dependencies are met:
+
+- `gcc` and `g++` >= 8.3 or newer
+- GNU Make 3.81 or newer
+- [CMake meta build system](https://cmake.org/)
+- [Node.js](https://nodejs.org/) 12 or later
+
+Once this is done, run this command _from the test or the benchmark directory_ (depending on what you want to build):
+
+```sh
+cd koffi/test # or cd koffi/benchmark
+node ../../cnoke/cnoke.js
+```
+
+## Running tests
+
+Koffi is tested on multiple architectures using emulated (accelerated when possible) QEMU machines. First, you need to install qemu packages, such as `qemu-system` (or even `qemu-system-gui`) on Ubuntu.
+
+These machines are not included directly in this repository (for license and size reasons), but they are available here: https://koromix.dev/files/machines/
+
+For example, if you want to run the tests on Debian ARM64, run the following commands:
+
+```sh
+cd luigi/koffi/qemu/
+wget -q -O- https://koromix.dev/files/machines/qemu_debian_arm64.tar.zst | zstd -d | tar xv
+sha256sum -c --ignore-missing registry/sha256sum.txt
+```
+
+Note that the machine disk content may change each time the machine runs, so the checksum test will fail once a machine has been used at least once.
+
+And now you can run the tests with:
+
+```sh
+node qemu.js # Several options are available, use --help
+```
+
+And be patient, this can be pretty slow for emulated machines. The Linux machines have and use ccache to build Koffi, so subsequent build steps will get much more tolerable.
+
+By default, machines are started and stopped for each test. But you can start the machines ahead of time and run the tests multiple times instead:
+
+```sh
+node qemu.js start # Start the machines
+node qemu.js # Test (without shutting down)
+node qemu.js # Test again
+node qemu.js stop # Stop everything
+```
+
+You can also restrict the test to a subset of machines:
+
+```sh
+# Full test cycle
+node qemu.js test debian_x64 debian_i386
+
+# Separate start, test, shutdown
+node qemu.js start debian_x64 debian_i386
+node qemu.js test debian_x64 debian_i386
+node qemu.js stop
+```
+
+Finally, you can join a running machine with SSH with the following shortcut, if you need to do some debugging or any other manual procedure:
+
+```sh
+node qemu.js ssh debian_i386
+```
+
+Each machine is configured to run a VNC server available locally, which you can use to access the display, using KRDC or any other compatible viewer. Use the `info` command to get the VNC port.
+
+```sh
+node qemu.js info debian_x64
+```
+
+## Todo list
+
+After the release of version 1.3.0, the current priorities for the next major release are:
+
+- Automate Windows/AArch64 (qemu) and macOS/AArch64 (how?) tests and builds
+- Create a real-world example, using several libraries (Raylib, SQLite, libsodium) to illustrate how to work with various C API styles
+
+The following features are also planned eventually, not necessarily in that order:
+
+- Optimize passing of structs and arrays, with separate HFA-specific helper functions
+- Add simple struct type parser
+- Add more ways to manually encode and decode various types to and from byte arrays
+- Add support for unions
+- Provide better ways to automatically deal with caller/heap-allocated memory (strings, etc.)
+- Port Koffi to PowerPC (POWER9+) ABI