koffi 1.3.9 → 1.3.12

package/ChangeLog.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## Koffi 1.3.12
+
+**Main fixes:**
+
+- Fix support for Yarn package manager
+
+## Koffi 1.3.11
+
+**Main fixes:**
+
+- Fix broken parsing of `void *` when used for first parameter
+
+## Koffi 1.3.10
+
+**Main fixes:**
+
+- Fix support for callbacks with more than 4 parameters on Windows x64
+- Fix support for callbacks with multiple floating-point arguments on ARM32 platforms
+- Fix possibly incorrect conversion for uint32_t callback parameters
+
+**Other changes:**
+
+- Various documentation fixes and improvements
+
 ## Koffi 1.3.9
 
 **Main fixes:**

package/doc/benchmarks.md

@@ -10,8 +10,8 @@ Here is a quick overview of the execution time of Koffi calls on three benchmark
 
 <table style="margin: 0 auto;">
     <tr>
-        <td><a href="_static/perf_linux_20220628.png" target="_blank"><img src="_static/perf_linux_20220628.png" alt="Linux performance" style="width: 350px;"/></a></td>
-        <td><a href="_static/perf_windows_20220628.png" target="_blank"><img src="_static/perf_windows_20220628.png" alt="Windows performance" style="width: 350px;"/></a></td>
+        <td><a href="_static/perf_linux_20220628.png" target="_blank"><img src="_static/perf_linux_20220628.png" alt="Linux x86_64 performance" style="width: 350px;"/></a></td>
+        <td><a href="_static/perf_windows_20220628.png" target="_blank"><img src="_static/perf_windows_20220628.png" alt="Windows x86_64 performance" style="width: 350px;"/></a></td>
     </tr>
 </table>
 

package/doc/conf.py

@@ -31,7 +31,7 @@ html_title = project
 
 html_theme = 'furo'
 
-html_static_path = ['_static']
+html_static_path = ['static']
 
 html_theme_options = {
     'light_css_variables': {

package/doc/contribute.md

@@ -4,7 +4,7 @@
 
 Use the official repository (named Luigi, because this is a monorepo containing multiple projects) for bugs, ideas and features requests.
 
-Go here: https://github.com/Koromix/luigi/issues
+Go here: https://github.com/Koromix/luigi/
 
 ## Build from source
 

package/doc/dist/html/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 1029792c9b37058f12795e72139e7221
+config: 3958eced2fda3c6c0751ec1a88d416ca
 tags: 645f666f9bcd5a90fca523b33c5a78b7

package/doc/dist/html/_sources/benchmarks.md.txt

@@ -10,8 +10,8 @@ Here is a quick overview of the execution time of Koffi calls on three benchmark
 
 <table style="margin: 0 auto;">
     <tr>
-        <td><a href="_static/perf_linux_20220628.png" target="_blank"><img src="_static/perf_linux_20220628.png" alt="Linux performance" style="width: 350px;"/></a></td>
-        <td><a href="_static/perf_windows_20220628.png" target="_blank"><img src="_static/perf_windows_20220628.png" alt="Windows performance" style="width: 350px;"/></a></td>
+        <td><a href="static/perf_linux_20220628.png" target="_blank"><img src="static/perf_linux_20220628.png" alt="Linux x86_64 performance" style="width: 350px;"/></a></td>
+        <td><a href="static/perf_windows_20220628.png" target="_blank"><img src="static/perf_windows_20220628.png" alt="Windows x86_64 performance" style="width: 350px;"/></a></td>
     </tr>
 </table>
 

package/doc/dist/html/_sources/contribute.md.txt

@@ -4,7 +4,7 @@
 
 Use the official repository (named Luigi, because this is a monorepo containing multiple projects) for bugs, ideas and features requests.
 
-Go here: https://github.com/Koromix/luigi/issues
+Go here: https://github.com/Koromix/luigi/
 
 ## Build from source
 
@@ -111,7 +111,6 @@ node qemu.js info debian_x64
 
 The following features and improvements are planned, not necessarily in that order:
 
-- Provide better ways to automatically deal with caller/heap-allocated memory (strings, etc.)
 - Optimize passing of structs and arrays (avoid setting named properties one by one? separate HFA-specific helper functions?)
 - Automate Windows/AArch64 (qemu) and macOS/AArch64 (how? ... thanks Apple) tests
 - Create a real-world example, using several libraries (Raylib, SQLite, libsodium) to illustrate various C API styles

package/doc/dist/html/_sources/functions.md.txt

@@ -23,7 +23,7 @@ const printf = lib.func('printf', 'int', ['str', '...']);
 const atoi = lib.func('atoi', 'int', ['str']);
 ```
 
-Koffi automatically tries mangled names for non-standard x86 calling conventions. See the section [on standard calls](#synchronous-calls) for more information on this subject.
+Koffi automatically tries mangled names for non-standard x86 calling conventions. See the section on [calling conventions](#calling-conventions) for more information on this subject.
 
 ### C-like prototypes
 
@@ -36,7 +36,9 @@ const atoi = lib.func('int atoi(str)'); // The parameter name is not used by Kof
 
 You can use `()` or `(void)` for functions that take no argument.
 
-## Synchronous calls
+## Function calls
+
+### Calling conventions
 
 By default, calling a C function happens synchronously.
 
@@ -57,12 +59,12 @@ Below you can find a small example showing how to use a non-default calling conv
 const koffi = require('koffi');
 const lib = koffi.load('user32.dll');
 
-// The following two declarations are equivalent, and use Stdcall
+// The following two declarations are equivalent, and use stdcall on x86 (and the default ABI on other platforms)
 const MessageBoxA_1 = lib.stdcall('MessageBoxA', 'int', ['void *', 'str', 'str', 'uint']);
 const MessageBoxA_2 = lib.func('int __stdcall MessageBoxA(void *hwnd, str text, str caption, uint type)');
 ```
 
-## Asynchronous calls
+### Asynchronous calls
 
 You can issue asynchronous calls by calling the function through its async member. In this case, you need to provide a callback function as the last argument, with `(err, res)` parameters.
 
@@ -88,7 +90,7 @@ You can easily convert this callback-style async function to a promise-based ver
 
 Variadic functions cannot be called asynchronously.
 
-## Variadic functions
+### Variadic functions
 
 Variadic functions are declared with an ellipsis as the last argument.
 
@@ -103,7 +105,9 @@ printf('Integer %d, double %g, str %s', 'int', 6, 'double', 8.5, 'str', 'THE END
 
 On x86 platforms, only the Cdecl convention can be used for variadic functions.
 
-## Output parameters
+## C to JS conversion gotchas
+
+### Output parameters
 
 By default, Koffi will only forward arguments from Javascript to C. However, many C functions use pointer arguments for output values, or input/output values.
 
@@ -122,7 +126,7 @@ The same can be done when declaring a function with a C-like prototype string, w
 - `_Out_` for output parameters
 - `_Inout_` for dual input/output parameters
 
-### Struct example
+#### Struct example
 
 This example calls the POSIX function `gettimeofday()`, and uses the prototype-like syntax.
 
@@ -148,7 +152,7 @@ gettimeofday(tv, null);
 console.log(tv);
 ```
 
-### Opaque handle example
+#### Opaque handle example
 
 This example opens an in-memory SQLite database, and uses the node-ffi-style function declaration syntax.
 
@@ -158,9 +162,9 @@ const lib = koffi.load('sqlite.so');
 
 const sqlite3_db = koffi.handle('sqlite3_db');
 
-// Use koffi.out() on a pointer to copy out (from C to JS) after the call
-const sqlite3_open_v2 = lib.func('sqlite3_open_v2', 'int', ['str', koffi.out(koffi.pointer(sqlite3_db)), 'int', 'str']);
-const sqlite3_close_v2 = lib.func('sqlite3_close_v2', 'int', [sqlite3_db]);
+// Use koffi.out() on a double pointer to copy out (from C to JS) after the call
+const sqlite3_open_v2 = lib.func('sqlite3_open_v2', 'int', ['str', koffi.out(koffi.pointer(sqlite3_db, 2)), 'int', 'str']);
+const sqlite3_close_v2 = lib.func('sqlite3_close_v2', 'int', [koffi.pointer(sqlite3_db)]);
 
 const SQLITE_OPEN_READWRITE = 0x2;
 const SQLITE_OPEN_CREATE = 0x4;
@@ -171,6 +175,55 @@ if (sqlite3_open_v2(':memory:', db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE,
 sqlite3_close_v2(db);
 ```
 
+### Heap-allocated values
+
+Some C functions return heap-allocated values directly or through output parameters. While Koffi automatically converts values from C to JS (to a string or an object), it does not know when something needs to be freed, or how.
+
+For opaque handles, such as FILE, this does not matter because you will explicitly call `fclose()` on them. But some values (such as strings) get implicitly converted by Koffi, and you lose access to the original pointer. This creates a leak if the string is heap-allocated.
+
+To avoid this, you can instruct Koffi to call a function on the original pointer once the conversion is done, by creating a disposable type with `koffi.dispose(name, type, func)`. This creates a type derived from another type, the only difference being that *func* gets called with the original pointer once the value has been converted and is not needed anymore.
+
+The *name* can be omitted to create an anonymous disposable type. If *func* is omitted or is null, Koffi will use `koffi.free(ptr)` (which calls the standard C library *free* function under the hood).
+
+```js
+const AnonHeapStr = koffi.disposable('str'); // Anonymous type (cannot be used in function prototypes)
+const NamedHeapStr = koffi.disposable('HeapStr', 'str'); // Same thing, but named so usable in function prototypes
+const ExplicitFree = koffi.disposable('HeapStr16', 'str16', koffi.free); // You can specify any other JS function
+```
+
+The following example illustrates the use of a disposable type derived from *str*.
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('libc.so.6');
+
+// You can also use: const strdup = lib.func('const char *! asprintf(const char *str)')
+const HeapStr = koffi.disposable('str');
+const strdup = lib.cdecl('strdup', HeapStr, ['str']);
+
+let copy = strdup('Hello!');
+console.log(copy); // Prints Hello!
+```
+
+When you declare functions with the [prototype-like syntax](#c-like-prototypes), you can either use named disposables types or use the '!' shortcut qualifier with compatibles types, as shown in the example below. This qualifier creates an anonymous disposable type that calls `koffi.free(ptr)`.
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('libc.so.6');
+
+// You can also use: const strdup = lib.func('const char *! strdup(const char *str)')
+const strdup = lib.func('str! strdup(const char *str)');
+
+let copy = strdup('World!');
+console.log(copy); // Prints World!
+```
+
+Disposable types can only be created from pointer or string types.
+
+```{warning}
+Be careful on Windows: if your shared library uses a different CRT (such as msvcrt), the memory could have been allocated by a different malloc/free implementation or heap, resulting in undefined behavior if you use `koffi.free()`.
+```
+
 ## Javascript callbacks
 
 In order to pass a JS function to a C function expecting a callback, you must first create a callback type with the expected return type and parameters. The syntax is similar to the one used to load functions from a shared library.
@@ -185,9 +238,28 @@ const ExampleCallback = koffi.callback('ExampleCallback', 'void', ['int']);
 const AddDoubleFloat = koffi.callback('double AddDoubleFloat(double d, float f)');
 ```
 
-Once your callback type is declared, you can use it in struct definitions, or as function parameter and/or return type.
+Once your callback type is declared, you can use a pointer to it in struct definitions, or as function parameters and/or return types.
+
+```{note}
+Callbacks **have changed in version 2.0**.
+
+In Koffi 1.x, callbacks were defined in a way that made them usable directly as parameter and return types, obscuring the underlying pointer.
+
+Now, you must use them through a pointer: `void CallIt(CallbackType func)` in Koffi 1.x becomes `void CallIt(CallbackType *func)` in version 2.0 and newer.
+```
+
+Koffi only uses predefined static trampolines, and does not need to generate code at runtime, which makes it compatible with platforms with hardened W^X migitations (such as PaX mprotect). However, this imposes some restrictions on the maximum number of callbacks, and their duration.
+
+Thus, Koffi distinguishes two callback modes:
 
-Here is a small example with the C part and the JS part.
+- [Transient callbacks](#transient-callbacks) can only be called while the C function they are passed to is running, and are invalidated when it returns. If the C function calls the callback later, the behavior is undefined, though Koffi tries to detect such cases. If it does, an exception will be thrown, but this is no guaranteed. However, they are simple to use, and don't require any special handling.
+- [Registered callbacks](#registered-callbacks) can be called at any time, but they must be manually registered and unregistered. A limited number of registered callbacks can exist at the same time.
+
+You need to specify the correct [calling convention](#calling-conventions) on x86 platforms, or the behavior is undefined (Node will probably crash). Only *cdecl* and *stdcall* callbacks are supported.
+
+### Transient callbacks
+
+Use transient callbacks when the native C function only needs to call them while it runs (e.g. qsort, progress callback, `sqlite3_exec`). Here is a small example with the C part and the JS part.
 
 ```c
 #include <string.h>
@@ -202,10 +274,11 @@ int TransferToJS(const char *name, int age, int (*cb)(const char *str, int age))
 
 ```js
 const koffi = require('koffi');
+const lib = koffi.load('./callbacks.so'); // Fake path
 
 const TransferCallback = koffi.callback('int TransferCallback(const char *str, int age)');
 
-const TransferToJS = lib.func('int TransferToJS(const char *str, int age, TransferCallback cb)');
+const TransferToJS = lib.func('TransferToJS', 'int', ['str', 'int', koffi.pointer(TransferCallback)]);
 
 let ret = TransferToJS('Niels', 27, (str, age) => {
     console.log(str);
@@ -220,7 +293,56 @@ console.log(ret);
 //   42
 ```
 
-On x86 platforms, only Cdecl and Stdcall callbacks are supported.
+### Registered callbacks
+
+Use registered callbacks when the function needs to be called at a later time (e.g. log handler, event handler, `fopencookie/funopen`). Call `koffi.register(func, type)` to register a callback function, with two arguments: the JS function, and the callback type.
+
+When you are done, call `koffi.unregister()` (with the value returned by `koffi.register()`) to release the slot. A maximum of 16 registered callbacks can exist at the same time. Failure to do so will leak the slot, and subsequent registrations may fail (with an exception) once all slots are used.
+
+The example below shows how to register and unregister delayed callbacks.
+
+```c
+static const char *(*g_cb1)(const char *name);
+static void (*g_cb2)(const char *str);
+
+void RegisterFunctions(const char *(*cb1)(const char *name), void (*cb2)(const char *str))
+{
+    g_cb1 = cb1;
+    g_cb2 = cb2;
+}
+
+void SayIt(const char *name)
+{
+    const char *str = g_cb1(name);
+    g_cb2(str);
+}
+```
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('./callbacks.so'); // Fake path
+
+const GetCallback = koffi.callback('const char *GetCallback(const char *name)');
+const PrintCallback = koffi.callback('void PrintCallback(const char *str)');
+
+const RegisterFunctions = lib.func('void RegisterFunctions(GetCallback *cb1, PrintCallback *cb2)');
+const SayIt = lib.func('void SayIt(const char *name)');
+
+let cb1 = koffi.register(name => 'Hello ' + name + '!', koffi.pointer(GetCallback));
+let cb2 = koffi.register(console.log, 'PrintCallback *');
+
+RegisterFunctions(cb1, cb2);
+SayIt('Kyoto'); // Prints Hello Kyoto!
+
+koffi.unregister(cb1);
+koffi.unregister(cb2);
+```
+
+### Handling of exceptions
+
+If an exception happens inside the JS callback, the C API will receive 0 or NULL (depending on the return value type).
+
+Handle the exception yourself (with try/catch) if you need to handle exceptions differently.
 
 ## Thread safety
 

package/doc/dist/html/_sources/index.rst.txt

@@ -14,6 +14,8 @@ Koffi is a **fast and easy-to-use C FFI module for Node.js**, featuring:
 
 Koffi requires a recent `Node.js <https://nodejs.org/>`_ version with N-API version 8 support, see :ref:`this page<Node.js>` for more information.
 
+The source code is available here: https://github.com/Koromix/luigi/ (in the *koffi* subdirectory).
+
 Table of contents
 -----------------
 

package/doc/dist/html/_sources/memory.md.txt

@@ -7,7 +7,7 @@ For synchronous/normal calls, Koffi uses two preallocated memory blocks:
 - One to construct the C stack and assign registers, subsequently used by the platform-specific assembly code (1 MiB by default)
 - One to allocate strings and big objects/structs (2 MiB by default)
 
-Unless very big strings or objects (at least more than one page of memory) are used, no extra allocation ever happens during calls or callbacks.
+Unless very big strings or objects (at least more than one page of memory) are used, Koffi does not directly allocate any extra memory during calls or callbacks. However, please note that the JS engine (V8) might.
 
 The size (in bytes) of these preallocated blocks can be changed. Use `koffi.config()` to get an object with the settings, and `koffi.config(obj)` to apply new settings.
 

package/doc/dist/html/_sources/types.md.txt

@@ -63,7 +63,7 @@ Primitive types can be specified by name (in a string) or through `koffi.types`:
 
 ```js
 // These two lines do the same:
-let struct1 = koffi.struct({ dummy: 'int' });
+let struct1 = koffi.struct({ dummy: 'long' });
 let struct2 = koffi.struct({ dummy: koffi.types.long });
 ```
 
@@ -91,7 +91,7 @@ typedef struct A {
 const A = koffi.struct('A', {
     a: 'int',
     b: 'char',
-    c: 'str',
+    c: 'const char *', // Koffi does not care about const, it is ignored
     d: koffi.struct({
         d1: 'double',
         d2: 'double'
@@ -146,7 +146,15 @@ console.log(pos);
 
 Many C libraries use some kind of object-oriented API, with a pair of functions dedicated to create and delete objects. An obvious example of this can be found in stdio.h, with the opaque `FILE *` pointer. You can open and close files with `fopen()` and `fclose()`, and manipule the handle with other functions such as `fread()` or `ftell()`.
 
-In Koffi, you can manage this with opaque handles. Declare the handle type with `koffi.handle(name)`, and use this type either as a return type or some kind of [output parameter](functions.md#output-parameters) (with a pointer to the handle).
+In Koffi, you can manage this with opaque handles. Declare the handle type with `koffi.handle(name)`, and use a pointer to this type either as a return type or some kind of [output parameter](functions.md#output-parameters) (with a double pointer).
+
+```{note}
+Opaque handles **have changed in version 2.0**.
+
+In Koffi 1.x, opaque handles were defined in a way that made them usable directly as parameter and return types, obscuring the underlying pointer.
+
+Now, you must use them through a pointer, and use an array for output parameters. This is shown in the example below (look for the call to `ConcatNewOut` in the JS part), and is described in the section on [output parameters](functions.md#output-parameters).
+```
 
 The full example below implements an iterative string builder (concatenator) in C, and uses it from Javascript to output a mix of Hello World and FizzBuzz. The builder is hidden behind an opaque handle, and is created and destroyed using a pair of C functions: `ConcatNew` (or `ConcatNewOut`) and `ConcatFree`.
 
@@ -274,17 +282,19 @@ const koffi = require('koffi');
 const lib = koffi.load('./handles.so');
 
 const Concat = koffi.handle('Concat');
-const ConcatNew = lib.func('Concat ConcatNew()');
-const ConcatFree = lib.func('void ConcatFree(Concat c)');
-const ConcatAppend = lib.func('bool ConcatAppend(Concat c, const char *frag)');
-const ConcatBuild = lib.func('const char *ConcatBuild(Concat c)');
-const ConcatNewOut = lib.func('bool ConcatNewOut(_Out_ Concat out)');
+const ConcatNewOut = lib.func('bool ConcatNewOut(_Out_ Concat **out)');
+const ConcatNew = lib.func('Concat *ConcatNew()');
+const ConcatFree = lib.func('void ConcatFree(Concat *c)');
+const ConcatAppend = lib.func('bool ConcatAppend(Concat *c, const char *frag)');
+const ConcatBuild = lib.func('const char *ConcatBuild(Concat *c)');
 
 let c = ConcatNew();
 if (!c) {
     // This is stupid, it does the same, but try both versions (return value, output parameter)
-    if (!ConcatNewOut(c))
+    let ptr = [null];
+    if (!ConcatNewOut(ptr))
         throw new Error('Allocation failure');
+    c = ptr[0];
 }
 
 try {
@@ -368,7 +378,7 @@ Here is an example based on the Win32 API, listing files in the current director
 const koffi = require('koffi');
 const lib = koffi.load('kernel32.dll');
 
-const HANDLE = koffi.handle('HANDLE');
+const HANDLE = koffi.pointer('HANDLE', koffi.handle());
 const FILETIME = koffi.struct('FILETIME', {
     dwLowDateTime: 'uint',
     dwHighDateTime: 'uint'
@@ -509,7 +519,13 @@ Koffi exposes three functions to explore type information:
 
 - `koffi.sizeof(type)` to get the size of a type
 - `koffi.alignof(type)` to get the alignment of a type
-- `koffi.introspect(type)` to get the definition of a type (only for structs for now)
+- `koffi.introspect(type)` to get the definition of a type in an object containing: name, primitive, size, alignment, members (structs), reference (array, pointer) and length (array)
+
+```{note}
+The value returned by `introspect()` has **changed in version 2.0**.
+
+In Koffi 1.x, it could only be used with struct types and returned the object passed to koffi.struct() with the member names and types.
+```
 
 Just like before, you can refer to primitive types by their name or through `koffi.types`:
 

package/doc/dist/html/_static/basic.css

@@ -237,6 +237,16 @@ a.headerlink {
     visibility: hidden;
 }
 
+a.brackets:before,
+span.brackets > a:before{
+    content: "[";
+}
+
+a.brackets:after,
+span.brackets > a:after {
+    content: "]";
+}
+
 h1:hover > a.headerlink,
 h2:hover > a.headerlink,
 h3:hover > a.headerlink,
@@ -324,18 +334,14 @@ aside.sidebar {
 p.sidebar-title {
     font-weight: bold;
 }
-nav.contents,
-aside.topic,
 
-div.admonition, div.topic, blockquote {
+div.admonition, div.topic, aside.topic, blockquote {
     clear: left;
 }
 
 /* -- topics ---------------------------------------------------------------- */
-nav.contents,
-aside.topic,
 
-div.topic {
+div.topic, aside.topic {
     border: 1px solid #ccc;
     padding: 7px;
     margin: 10px 0 10px 0;
@@ -373,20 +379,16 @@ div.body p.centered {
 
 div.sidebar > :last-child,
 aside.sidebar > :last-child,
-nav.contents > :last-child,
-aside.topic > :last-child,
-
 div.topic > :last-child,
+aside.topic > :last-child,
 div.admonition > :last-child {
     margin-bottom: 0;
 }
 
 div.sidebar::after,
 aside.sidebar::after,
-nav.contents::after,
-aside.topic::after,
-
 div.topic::after,
+aside.topic::after,
 div.admonition::after,
 blockquote::after {
     display: block;