koffi 2.1.5 → 2.2.1

package/ChangeLog.md

@@ -2,6 +2,26 @@
 
 ## History
 
+### Koffi 2.2.1
+
+**Main fixes:**
+
+- Fix crash when [calling callback again after FFI call inside previous callback](https://github.com/Koromix/rygel/issues/15)
+
+### Koffi 2.2.0
+
+**New features:**
+
+- Add [koffi.decode()](callbacks.md#pointer-arguments) for callback pointer arguments
+- Support transparent [output string parameters](functions.md#output-parameters)
+- Add `koffi.offsetof()` utility function
+- Support optional *this* binding in `koffi.register()`
+
+**Other fixes:**
+
+- Correctly validate output parameter types
+- Fix assertion with `void *` parameters
+
 ### Koffi 2.1.5
 
 **Main fixes:**
@@ -66,7 +86,7 @@
 **Major new features:**
 
 - Add [disposable types](functions.md#heap-allocated-values) for automatic disposal of C values (such as heap-allocated strings)
-- Add support for [registered callbacks](functions.md#registered-callbacks), that can be called after the initial FFI call
+- Add support for [registered callbacks](callbacks.md#registered-callbacks), that can be called after the initial FFI call
 - Support named pointer types
 - Support complex type specifications outside of prototype parser
 

package/doc/Makefile

@@ -6,7 +6,7 @@
 SPHINXOPTS    ?=
 SPHINXBUILD   ?= sphinx-build
 SOURCEDIR     = .
-BUILDDIR      = ../build/doc
+BUILDDIR      = dist
 
 # Put it first so that "make" without argument is like "make help".
 help:

package/doc/callbacks.md

@@ -0,0 +1,175 @@
+# Callbacks
+
+In order to pass a JS function to a C function expecting a callback, you must first create a callback type with the expected return type and parameters. The syntax is similar to the one used to load functions from a shared library.
+
+```js
+const koffi = require('koffi');
+
+// With the classic syntax, this callback expects an integer and returns nothing
+const ExampleCallback = koffi.callback('ExampleCallback', 'void', ['int']);
+
+// With the prototype parser, this callback expects a double and float, and returns the sum as a double
+const AddDoubleFloat = koffi.callback('double AddDoubleFloat(double d, float f)');
+```
+
+Once your callback type is declared, you can use a pointer to it in struct definitions, or as function parameters and/or return types.
+
+```{note}
+Callbacks **have changed in version 2.0**.
+
+In Koffi 1.x, callbacks were defined in a way that made them usable directly as parameter and return types, obscuring the underlying pointer.
+
+Now, you must use them through a pointer: `void CallIt(CallbackType func)` in Koffi 1.x becomes `void CallIt(CallbackType *func)` in version 2.0 and newer.
+
+Consult the [migration guide](changes.md) for more information.
+```
+
+## Callback types
+
+Koffi only uses predefined static trampolines, and does not need to generate code at runtime, which makes it compatible with platforms with hardened W^X migitations (such as PaX mprotect). However, this imposes some restrictions on the maximum number of callbacks, and their duration.
+
+Thus, Koffi distinguishes two callback modes:
+
+- [Transient callbacks](#transient-callbacks) can only be called while the C function they are passed to is running, and are invalidated when it returns. If the C function calls the callback later, the behavior is undefined, though Koffi tries to detect such cases. If it does, an exception will be thrown, but this is no guaranteed. However, they are simple to use, and don't require any special handling.
+- [Registered callbacks](#registered-callbacks) can be called at any time, but they must be manually registered and unregistered. A limited number of registered callbacks can exist at the same time.
+
+You need to specify the correct [calling convention](functions.md#calling-conventions) on x86 platforms, or the behavior is undefined (Node will probably crash). Only *cdecl* and *stdcall* callbacks are supported.
+
+### Transient callbacks
+
+Use transient callbacks when the native C function only needs to call them while it runs (e.g. qsort, progress callback, `sqlite3_exec`). Here is a small example with the C part and the JS part.
+
+```c
+#include <string.h>
+
+int TransferToJS(const char *name, int age, int (*cb)(const char *str, int age))
+{
+    char buf[64];
+    snprintf(buf, sizeof(buf), "Hello %s!", str);
+    return cb(buf, age);
+}
+```
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('./callbacks.so'); // Fake path
+
+const TransferCallback = koffi.callback('int TransferCallback(const char *str, int age)');
+
+const TransferToJS = lib.func('TransferToJS', 'int', ['str', 'int', koffi.pointer(TransferCallback)]);
+
+let ret = TransferToJS('Niels', 27, (str, age) => {
+    console.log(str);
+    console.log('Your age is:', age);
+    return 42;
+});
+console.log(ret);
+
+// This example prints:
+//   Hello Niels!
+//   Your age is: 27
+//   42
+```
+
+### Registered callbacks
+
+*New in Koffi 2.0 (explicit this binding in Koffi 2.2)*
+
+Use registered callbacks when the function needs to be called at a later time (e.g. log handler, event handler, `fopencookie/funopen`). Call `koffi.register(func, type)` to register a callback function, with two arguments: the JS function, and the callback type.
+
+When you are done, call `koffi.unregister()` (with the value returned by `koffi.register()`) to release the slot. A maximum of 16 registered callbacks can exist at the same time. Failure to do so will leak the slot, and subsequent registrations may fail (with an exception) once all slots are used.
+
+The example below shows how to register and unregister delayed callbacks.
+
+```c
+static const char *(*g_cb1)(const char *name);
+static void (*g_cb2)(const char *str);
+
+void RegisterFunctions(const char *(*cb1)(const char *name), void (*cb2)(const char *str))
+{
+    g_cb1 = cb1;
+    g_cb2 = cb2;
+}
+
+void SayIt(const char *name)
+{
+    const char *str = g_cb1(name);
+    g_cb2(str);
+}
+```
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('./callbacks.so'); // Fake path
+
+const GetCallback = koffi.callback('const char *GetCallback(const char *name)');
+const PrintCallback = koffi.callback('void PrintCallback(const char *str)');
+
+const RegisterFunctions = lib.func('void RegisterFunctions(GetCallback *cb1, PrintCallback *cb2)');
+const SayIt = lib.func('void SayIt(const char *name)');
+
+let cb1 = koffi.register(name => 'Hello ' + name + '!', koffi.pointer(GetCallback));
+let cb2 = koffi.register(console.log, 'PrintCallback *');
+
+RegisterFunctions(cb1, cb2);
+SayIt('Kyoto'); // Prints Hello Kyoto!
+
+koffi.unregister(cb1);
+koffi.unregister(cb2);
+```
+
+Starting *with Koffi 2.2*, you can optionally specify the `this` value for the function as the first argument.
+
+```js
+class ValueStore {
+    constructor(value) { this.value = value; }
+    get() { return this.value; }
+}
+
+let store = new ValueStore(42);
+
+let cb1 = koffi.register(store.get, 'IntCallback *'); // If a C function calls cb1 it will fail because this will be undefined
+let cb2 = koffi.register(store, store.get, 'IntCallback *'); // However in this case, this will match the store object
+```
+
+## Pointer arguments
+
+*New in Koffi 2.2*
+
+Koffi does not have enough information to convert callback pointer arguments to an appropriate JS value. In this case, your JS function will receive an opaque *External* object.
+
+You can pass this value through to another C function that expects a pointer of the same type, or you can use `koffi.decode(value, offset, type, len)` to decode it into something you can use in Javascript.
+
+Some arguments are optional and this function can be called in several ways:
+
+- `koffi.decode(value, offset, type, len)`: this is the full signature, value is the JS external object, offset (optional, defaults to 0) is specified in bytes, type is the value type, and length (optional) can be used for arrays and strings.
+- `koffi.decode(value, type)`: no offset, expect NUL-terminated strings
+- `koffi.decode(value, type, len)`: use specified length when decoding strings and arrays
+- `koffi.decode(value, offset, type)`
+
+The following example sorts an array of strings (in-place) with `qsort()`:
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('libc.so.6');
+
+const SortCallback = koffi.callback('int SortCallback(const void *first, const void *second)');
+const qsort = lib.func('void qsort(_Inout_ void *array, size_t count, size_t size, SortCallback *cb)');
+
+let array = ['foo', 'bar', '123', 'foobar'];
+
+qsort(koffi.as(array, 'char **'), array.length, koffi.sizeof('void *'), (ptr1, ptr2) => {
+    let str1 = koffi.decode(ptr1, 'char *');
+    let str2 = koffi.decode(ptr2, 'char *');
+
+    return str1.localeCompare(str2);
+});
+
+console.log(array); // Prints ['123', 'bar', 'foo', 'foobar']
+```
+
+## Handling of exceptions
+
+If an exception happens inside the JS callback, the C API will receive 0 or NULL (depending on the return value type).
+
+Handle the exception yourself (with try/catch) if you need to handle exceptions differently.

package/doc/changes.md

@@ -1,5 +1,4 @@
-```{include} ../ChangeLog.md
-```
+{{ "```{include} " ~ root ~ "/ChangeLog.md" ~ "\n```" }}
 
 ## Migration guide
 
@@ -64,7 +63,7 @@ let ret = TransferToJS('Niels', 27, (str, age) => {
 console.log(ret);
 ```
 
-Koffi 1.x only supported [transient callbacks](functions.md#javascript-callbacks), you must use Koffi 2.x for registered callbacks.
+Koffi 1.x only supported [transient callbacks](callbacks.md#callbacks), you must use Koffi 2.x for registered callbacks.
 
 #### Opaque types
 

package/doc/conf.py

@@ -7,12 +7,31 @@ project = 'Koffi'
 copyright = '2022, Niels Martignène'
 author = 'Niels Martignène'
 
-with open(os.path.dirname(__file__) + '/../package.json') as f:
-    config = json.load(f)
+root = None
+version = None
+revision = None
+stable = None
 
-    version = config['version']
-    revision = config['version']
-    stable = config['stable']
+names = ['../../src/koffi/package.json', '../package.json']
+
+for name in names:
+    filename = os.path.dirname(__file__) + '/' + name
+
+    if not os.path.exists(filename):
+        continue;
+
+    with open(filename) as f:
+        config = json.load(f)
+
+        root = os.path.dirname(name)
+        version = config['version']
+        revision = config['version']
+        stable = config['stable']
+
+    break
+
+if root is None:
+    raise FileNotFoundError('Cannot find Koffi package.json')
 
 # -- General configuration ---------------------------------------------------
 
@@ -63,15 +82,19 @@ html_sidebars = {
 }
 
 html_context = {
+    "root": root,
     "stable": stable
 }
 
 # -- MyST parser options -------------------------------------------------
 
 myst_enable_extensions = [
-    'linkify'
+    'linkify',
+    'substitution'
 ]
 
+myst_substitutions = html_context
+
 myst_heading_anchors = 3
 
 myst_linkify_fuzzy_links = False

package/doc/functions.md

@@ -115,6 +115,7 @@ For simplicity, and because Javascript only has value semantics for primitive ty
 
 - [Structs](types.md#struct-types) (to/from JS objects)
 - [Opaque types](types.md#opaque-types)
+- String buffers
 
 In order to change an argument from input-only to output or input/output, use the following functions:
 
@@ -177,6 +178,44 @@ let db = out[0];
 sqlite3_close_v2(db);
 ```
 
+#### String buffer example
+
+*New in Koffi 2.2*
+
+This example calls a C function to concatenate two strings to a pre-allocated string buffer. Since JS strings are immutable, you must pass an array with a single string instead.
+
+```c
+void ConcatToBuffer(const char *str1, const char *str2, char *out)
+{
+    size_t len = 0;
+
+    for (size_t i = 0; str1[i]; i++) {
+        out[len++] = str1[i];
+    }
+    for (size_t i = 0; str2[i]; i++) {
+        out[len++] = str2[i];
+    }
+
+    out[len] = 0;
+}
+```
+
+```js
+const ConcatToBuffer = lib.func('void ConcatToBuffer(const char *str1, const char *str2, _Out_ char *out)');
+
+let str1 = 'Hello ';
+let str2 = 'Friends!';
+
+// We need to reserve space for the output buffer! Including the NUL terminator
+// because ConcatToBuffer() expects so, but Koffi can convert back to a JS string
+// without it (if we reserve the right size).
+let out = ['\0'.repeat(str1.length + str2.length + 1)];
+
+ConcatToBuffer(str1, str2, out);
+
+console.log(out[0]);
+```
+
 ### Polymorphic parameters
 
 *New in Koffi 2.1*
@@ -290,130 +329,6 @@ Disposable types can only be created from pointer or string types.
 Be careful on Windows: if your shared library uses a different CRT (such as msvcrt), the memory could have been allocated by a different malloc/free implementation or heap, resulting in undefined behavior if you use `koffi.free()`.
 ```
 
-## Javascript callbacks
-
-In order to pass a JS function to a C function expecting a callback, you must first create a callback type with the expected return type and parameters. The syntax is similar to the one used to load functions from a shared library.
-
-```js
-const koffi = require('koffi');
-
-// With the classic syntax, this callback expects an integer and returns nothing
-const ExampleCallback = koffi.callback('ExampleCallback', 'void', ['int']);
-
-// With the prototype parser, this callback expects a double and float, and returns the sum as a double
-const AddDoubleFloat = koffi.callback('double AddDoubleFloat(double d, float f)');
-```
-
-Once your callback type is declared, you can use a pointer to it in struct definitions, or as function parameters and/or return types.
-
-```{note}
-Callbacks **have changed in version 2.0**.
-
-In Koffi 1.x, callbacks were defined in a way that made them usable directly as parameter and return types, obscuring the underlying pointer.
-
-Now, you must use them through a pointer: `void CallIt(CallbackType func)` in Koffi 1.x becomes `void CallIt(CallbackType *func)` in version 2.0 and newer.
-
-Consult the [migration guide](changes.md) for more information.
-```
-
-Koffi only uses predefined static trampolines, and does not need to generate code at runtime, which makes it compatible with platforms with hardened W^X migitations (such as PaX mprotect). However, this imposes some restrictions on the maximum number of callbacks, and their duration.
-
-Thus, Koffi distinguishes two callback modes:
-
-- [Transient callbacks](#transient-callbacks) can only be called while the C function they are passed to is running, and are invalidated when it returns. If the C function calls the callback later, the behavior is undefined, though Koffi tries to detect such cases. If it does, an exception will be thrown, but this is no guaranteed. However, they are simple to use, and don't require any special handling.
-- [Registered callbacks](#registered-callbacks) can be called at any time, but they must be manually registered and unregistered. A limited number of registered callbacks can exist at the same time.
-
-You need to specify the correct [calling convention](#calling-conventions) on x86 platforms, or the behavior is undefined (Node will probably crash). Only *cdecl* and *stdcall* callbacks are supported.
-
-### Transient callbacks
-
-Use transient callbacks when the native C function only needs to call them while it runs (e.g. qsort, progress callback, `sqlite3_exec`). Here is a small example with the C part and the JS part.
-
-```c
-#include <string.h>
-
-int TransferToJS(const char *name, int age, int (*cb)(const char *str, int age))
-{
-    char buf[64];
-    snprintf(buf, sizeof(buf), "Hello %s!", str);
-    return cb(buf, age);
-}
-```
-
-```js
-const koffi = require('koffi');
-const lib = koffi.load('./callbacks.so'); // Fake path
-
-const TransferCallback = koffi.callback('int TransferCallback(const char *str, int age)');
-
-const TransferToJS = lib.func('TransferToJS', 'int', ['str', 'int', koffi.pointer(TransferCallback)]);
-
-let ret = TransferToJS('Niels', 27, (str, age) => {
-    console.log(str);
-    console.log('Your age is:', age);
-    return 42;
-});
-console.log(ret);
-
-// This example prints:
-//   Hello Niels!
-//   Your age is: 27
-//   42
-```
-
-### Registered callbacks
-
-*New in Koffi 2.0*
-
-Use registered callbacks when the function needs to be called at a later time (e.g. log handler, event handler, `fopencookie/funopen`). Call `koffi.register(func, type)` to register a callback function, with two arguments: the JS function, and the callback type.
-
-When you are done, call `koffi.unregister()` (with the value returned by `koffi.register()`) to release the slot. A maximum of 16 registered callbacks can exist at the same time. Failure to do so will leak the slot, and subsequent registrations may fail (with an exception) once all slots are used.
-
-The example below shows how to register and unregister delayed callbacks.
-
-```c
-static const char *(*g_cb1)(const char *name);
-static void (*g_cb2)(const char *str);
-
-void RegisterFunctions(const char *(*cb1)(const char *name), void (*cb2)(const char *str))
-{
-    g_cb1 = cb1;
-    g_cb2 = cb2;
-}
-
-void SayIt(const char *name)
-{
-    const char *str = g_cb1(name);
-    g_cb2(str);
-}
-```
-
-```js
-const koffi = require('koffi');
-const lib = koffi.load('./callbacks.so'); // Fake path
-
-const GetCallback = koffi.callback('const char *GetCallback(const char *name)');
-const PrintCallback = koffi.callback('void PrintCallback(const char *str)');
-
-const RegisterFunctions = lib.func('void RegisterFunctions(GetCallback *cb1, PrintCallback *cb2)');
-const SayIt = lib.func('void SayIt(const char *name)');
-
-let cb1 = koffi.register(name => 'Hello ' + name + '!', koffi.pointer(GetCallback));
-let cb2 = koffi.register(console.log, 'PrintCallback *');
-
-RegisterFunctions(cb1, cb2);
-SayIt('Kyoto'); // Prints Hello Kyoto!
-
-koffi.unregister(cb1);
-koffi.unregister(cb2);
-```
-
-### Handling of exceptions
-
-If an exception happens inside the JS callback, the C API will receive 0 or NULL (depending on the return value type).
-
-Handle the exception yourself (with try/catch) if you need to handle exceptions differently.
-
 ## Thread safety
 
 Asynchronous functions run on worker threads. You need to deal with thread safety issues if you share data between threads.

package/doc/index.rst

@@ -26,6 +26,7 @@ Table of contents
    start
    types
    functions
+   callbacks
    memory
    benchmarks
    contribute

package/doc/make.bat

@@ -8,7 +8,7 @@ if "%SPHINXBUILD%" == "" (
 	set SPHINXBUILD=sphinx-build
 )
 set SOURCEDIR=.
-set BUILDDIR=../build/doc
+set BUILDDIR=dist
 
 %SPHINXBUILD% >NUL 2>NUL
 if errorlevel 9009 (

package/doc/types.md

@@ -509,23 +509,50 @@ Read the documentation for [disposable types](functions.md#heap-allocated-values
 
 ### Type introspection
 
-*New in Koffi 2.0: `koffi.resolve()`*
-
-Koffi exposes three functions to explore type information:
-
-- `koffi.sizeof(type)` to get the size of a type
-- `koffi.alignof(type)` to get the alignment of a type
-- `koffi.introspect(type)` to get the definition of a type in an object containing: name, primitive, size, alignment, members (structs), reference (array, pointer) and length (array)
-- `koffi.resolve(type)` to get the resolved type object from a type string
+*New in Koffi 2.0: `koffi.resolve()`, new in Koffi 2.2: `koffi.offsetof()`*
 
 ```{note}
-The value returned by `introspect()` has **changed in version 2.0**.
+The value returned by `introspect()` has **changed in version 2.0 and in version 2.2**.
 
 In Koffi 1.x, it could only be used with struct types and returned the object passed to koffi.struct() with the member names and types.
 
+Starting in Koffi 2.2, each record member is exposed as an object containing the name, the type and the offset within the record.
+
 Consult the [migration guide](changes.md) for more information.
 ```
 
+Use `koffi.introspect(type)` to get detailed information about a type: name, primitive, size, alignment, members (record types), reference type (array, pointer) and length (array).
+
+```js
+const FoobarType = koffi.struct('FoobarType', {
+    a: 'int',
+    b: 'char *',
+    c: 'double'
+});
+
+console.log(koffi.introspect(FoobarType));
+
+// Expected result on 64-bit machines:
+// {
+//     name: 'FoobarType',
+//     primitive: 'Record',
+//     size: 24,
+//     alignment: 8,
+//     members: {
+//         a: { name: 'a', type: [External: 4b28a60], offset: 0 },
+//         b: { name: 'b', type: [External: 4b292e0], offset: 8 },
+//         c: { name: 'c', type: [External: 4b29260], offset: 16 }
+//     }
+// }
+```
+
+Koffi also exposes a few more utility functions to get a subset of this information:
+
+- `koffi.sizeof(type)` to get the size of a type
+- `koffi.alignof(type)` to get the alignment of a type
+- `koffi.offsetof(type, member_name)` to get the offset of a record member
+- `koffi.resolve(type)` to get the resolved type object from a type string
+
 Just like before, you can refer to primitive types by their name or through `koffi.types`:
 
 ```js

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "koffi",
-    "version": "2.1.5",
-    "stable": "2.1.5",
+    "version": "2.2.1",
+    "stable": "2.2.1",
     "description": "Fast and simple C FFI (foreign function interface) for Node.js",
     "keywords": [
         "foreign",