koffi 2.7.4 → 2.8.0

package/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## Version history
 
+### Koffi 2.8
+
+#### Koffi 2.8.0 (2024-02-12)
+
+- Support pushing pointers for string arguments
+- Add `koffi.alloc()` for [stable pointers](output.md#stable-pointers)
+
 ### Koffi 2.7
 
 #### Koffi 2.7.4 (2024-02-04)
@@ -130,7 +137,7 @@ Pre-built binaries don't work correctly in Koffi 2.5.13 to 2.5.15, skip those ve
 
 #### Koffi 2.5.11 (2023-08-03)
 
-- Support casting function pointers with [koffi.as()](polymorphism.md#input-polymorphism)
+- Support casting function pointers with [koffi.as()](pointers.md#handling-void-pointers)
 - Build in C++20 mode
 
 #### Koffi 2.5.10 (2023-08-01)
@@ -338,7 +345,7 @@ Pre-built binaries don't work correctly in Koffi 2.5.13 to 2.5.15, skip those ve
 **Main changes:**
 
 - Allow buffers (TypedArray or ArrayBuffer) values for input and/or output pointer arguments (for polymorphic arguments)
-- Support opaque buffers (TypedArray or ArrayBuffer) values in `koffi.decode()` to [decode output buffers](polymorphism.md#output-buffers)
+- Support opaque buffers (TypedArray or ArrayBuffer) values in `koffi.decode()` to [decode output buffers](output.md#output-buffers)
 - Decode non-string types as arrays when an [explicit length is passed to koffi.decode()](callbacks.md#decoding-pointer-arguments)
 
 **Other changes:**
@@ -431,7 +438,7 @@ Pre-built binaries don't work correctly in Koffi 2.5.13 to 2.5.15, skip those ve
 
 **Main changes:**
 
-- Add [koffi.as()](polymorphism.md#input-polymorphism) to support polymorphic APIs based on `void *` parameters
+- Add [koffi.as()](pointers.md#handling-void-pointers) to support polymorphic APIs based on `void *` parameters
 - Add [endian-sensitive integer types](input.md#endian-sensitive-integers): `intX_le_t`, `intX_be_t`, `uintX_le_t`, `uintX_be_t`
 - Accept typed arrays for `void *` parameters
 - Introduce `koffi.opaque()` to replace `koffi.handle()` (which remains supported until Koffi 3.0)

package/doc/functions.md

@@ -251,4 +251,3 @@ Among other thing, in the the following pages you will learn more about:
 - How Koffi translates [input parameters](input.md) to C
 - How you can [define and use pointers](pointers.md)
 - How to deal with [output parameters](output.md)
-- How to handle [polymorphic API](polymorphism.md)

package/doc/index.rst

@@ -29,7 +29,6 @@ Table of contents
    input
    pointers
    output
-   polymorphism
    unions
    variables
    callbacks

package/doc/input.md

@@ -393,7 +393,7 @@ The reverse case is also true, Koffi can convert a C fixed-size buffer to a JS s
 
 ### Dynamic arrays (pointers)
 
-In C, dynamically-sized arrays are usually passed around as pointers. Read more about [array pointers](pointers.md#array-pointers-dynamic-arrays) in the relevant section.
+In C, dynamically-sized arrays are usually passed around as pointers. Read more about [array pointers](pointers.md#dynamic-arrays) in the relevant section.
 
 ## Union types
 

package/doc/output.md

@@ -167,3 +167,111 @@ ConcatToBuffer(str1, str2, out);
 
 console.log(out[0]);
 ```
+
+## Output buffers
+
+In most cases, you can use buffers and typed arrays to provide output buffers. This works as long as the buffer only gets used while the native C function is being called. See [transient pointers](#transient-pointers) below for an example.
+
+```{warning}
+It is unsafe to keep the pointer around in the native code, or to change the contents outside of the function call where it is provided.
+
+If you need to provide a pointer that will be kept around, allocate memory with [koffi.alloc()](#stable-pointers) instead.
+```
+
+### Transient pointers
+
+*New in Koffi 2.3*
+
+You can use buffers and typed arrays for output (and input/output) pointer parameters. Simply pass the buffer as an argument and the native function will receive a pointer to its contents.
+
+Once the native function returns, you can decode the content with `koffi.decode(value, type)` as in the following example:
+
+```js
+// ES6 syntax: import koffi from 'koffi';
+const koffi = require('koffi');
+
+const lib = koffi.load('libc.so.6');
+
+const Vec3 = koffi.struct('Vec3', {
+    x: 'float32',
+    y: 'float32',
+    z: 'float32'
+})
+
+const memcpy = lib.func('void *memcpy(_Out_ void *dest, const void *src, size_t size)');
+
+let vec1 = { x: 1, y: 2, z: 3 };
+let vec2 = null;
+
+// Copy the vector in a convoluted way through memcpy
+{
+    let src = koffi.as(vec1, 'Vec3 *');
+    let dest = Buffer.allocUnsafe(koffi.sizeof(Vec3));
+
+    memcpy(dest, src, koffi.sizeof(Vec3));
+
+    vec2 = koffi.decode(dest, Vec3);
+}
+
+// CHange vector1, leaving copy alone
+[vec1.x, vec1.y, vec1.z] = [vec1.z, vec1.y, vec1.x];
+
+console.log(vec1); // { x: 3, y: 2, z: 1 }
+console.log(vec2); // { x: 1, y: 2, z: 3 }
+```
+
+See [decoding variables](variables.md#decode-to-js-values) for more information about the decode function.
+
+### Stable pointers
+
+*New in Koffi 2.8*
+
+In some cases, the native code may need to change the output buffer at a later time, maybe during a later call or from another thread.
+
+In this case, it is **not safe to use buffers or typed arrays**! 
+
+However, you can use `koffi.alloc(type, len)` to allocate memory and get a pointer that won't move, and can be safely used at any time by the native code. Use [koffi.decode()](variables.md#decode-to-js-values) to read data from the pointer when needed.
+
+The example below sets up some memory to be used as an output buffer where a concatenation function appends a string on each call.
+
+```c
+#include <assert.h>
+#include <stddef.h>
+
+static char *buf_ptr;
+static size_t buf_len;
+static size_t buf_size;
+
+void reset_buffer(char *buf, size_t size)
+{
+    assert(size > 1);
+
+    buf_ptr = buf;
+    buf_len = 0;
+    buf_size = size - 1; // Keep space for trailing NUL
+
+    buf_ptr[0] = 0;
+}
+
+void append_str(const char *str)
+{
+    for (size_t i = 0; str[i] && buf_len < buf_size; i++, buf_len++) {
+        buf_ptr[buf_len] = str[i];
+    }
+    buf_ptr[buf_len] = 0;
+}
+```
+
+```js
+const reset_buffer = lib.func('void reset_buffer(char *buf, size_t size)');
+const append_str = lib.func('void append_str(const char *str)');
+
+let output = koffi.alloc('char', 64);
+reset_buffer(output, 64);
+
+append_str('Hello');
+console.log(koffi.decode(output, 'char', -1)); // Prints Hello
+
+append_str(' World!');
+console.log(koffi.decode(output, 'char', -1)); // Prints Hello World!
+```

package/doc/pointers.md

@@ -88,7 +88,7 @@ AddInt(sum, 6);
 console.log(sum[0]); // Prints 42
 ```
 
-### Array pointers (dynamic arrays)
+### Dynamic arrays
 
 In C, dynamically-sized arrays are usually passed around as pointers. The length is either passed as an additional argument, or inferred from the array content itself, for example with a terminating sentinel value (such as a NULL pointers in the case of an array of strings).
 
@@ -132,6 +132,69 @@ console.log(total); // Prints 14
 
 By default, just like for objects, array arguments are copied from JS to C but not vice-versa. You can however change the direction as documented in the section on [output parameters](output.md).
 
+## Handling void pointers
+
+*New in Koffi 2.1*
+
+Many C functions use `void *` parameters in order to pass polymorphic objects and arrays, meaning that the data format changes can change depending on one other argument, or on some kind of struct tag member.
+
+Koffi provides two features to deal with this:
+
+- You can use `koffi.as(value, type)` to tell Koffi what kind of type is actually expected, as shown in the example below.
+- Buffers and typed JS arrays can be used as values in place everywhere a pointer is expected. See [dynamic arrays](#dynamic-arrays) for more information, for input or output.
+
+The example below shows the use of `koffi.as()` to read the header of a PNG file with `fread()` directly to a JS object.
+
+```js
+// ES6 syntax: import koffi from 'koffi';
+const koffi = require('koffi');
+
+const lib = koffi.load('libc.so.6');
+
+const FILE = koffi.opaque('FILE');
+
+const PngHeader = koffi.pack('PngHeader', {
+    signature: koffi.array('uint8_t', 8),
+    ihdr: koffi.pack({
+        length: 'uint32_be_t',
+        chunk: koffi.array('char', 4),
+        width: 'uint32_be_t',
+        height: 'uint32_be_t',
+        depth: 'uint8_t',
+        color: 'uint8_t',
+        compression: 'uint8_t',
+        filter: 'uint8_t',
+        interlace: 'uint8_t',
+        crc: 'uint32_be_t'
+    })
+});
+
+const fopen = lib.func('FILE *fopen(const char *path, const char *mode)');
+const fclose = lib.func('int fclose(FILE *fp)');
+const fread = lib.func('size_t fread(_Out_ void *ptr, size_t size, size_t nmemb, FILE *fp)');
+
+let filename = process.argv[2];
+if (filename == null)
+    throw new Error('Usage: node png.js <image.png>');
+
+let hdr = {};
+{
+    let fp = fopen(filename, 'rb');
+    if (!fp)
+        throw new Error(`Failed to open '${filename}'`);
+
+    try {
+        let len = fread(koffi.as(hdr, 'PngHeader *'), 1, koffi.sizeof(PngHeader), fp);
+        if (len < koffi.sizeof(PngHeader))
+            throw new Error('Failed to read PNG header');
+    } finally {
+        fclose(fp);
+    }
+}
+
+console.log('PNG header:', hdr);
+```
+
 ## Disposable types
 
 *New in Koffi 2.0*

package/index.js

@@ -378,8 +378,8 @@ var require_package = __commonJS({
   "build/dist/src/koffi/package.json"(exports2, module2) {
     module2.exports = {
       name: "koffi",
-      version: "2.7.4",
-      stable: "2.7.4",
+      version: "2.8.0",
+      stable: "2.8.0",
       description: "Fast and simple C FFI (foreign function interface) for Node.js",
       keywords: [
         "foreign",

package/indirect.js

@@ -378,8 +378,8 @@ var require_package = __commonJS({
   "build/dist/src/koffi/package.json"(exports2, module2) {
     module2.exports = {
       name: "koffi",
-      version: "2.7.4",
-      stable: "2.7.4",
+      version: "2.8.0",
+      stable: "2.8.0",
       description: "Fast and simple C FFI (foreign function interface) for Node.js",
       keywords: [
         "foreign",

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "koffi",
-    "version": "2.7.4",
-    "stable": "2.7.4",
+    "version": "2.8.0",
+    "stable": "2.8.0",
     "description": "Fast and simple C FFI (foreign function interface) for Node.js",
     "keywords": [
         "foreign",

package/src/core/libcc/libcc.hh

@@ -3142,10 +3142,10 @@ static inline void Log(LogLevel level, const char *ctx, const char *fmt, Args...
 #ifdef RG_DEBUG
     const char *DebugLogContext(const char *filename, int line);
 
-    #define LogDebug(...) Log(LogLevel::Debug, DebugLogContext(__FILE__, __LINE__),  __VA_ARGS__)
-    #define LogInfo(...) Log(LogLevel::Info, nullptr, __VA_ARGS__)
-    #define LogWarning(...) Log(LogLevel::Warning, DebugLogContext(__FILE__, __LINE__), __VA_ARGS__)
-    #define LogError(...) Log(LogLevel::Error, DebugLogContext(__FILE__, __LINE__), __VA_ARGS__)
+    #define LogDebug(...) Log(LogLevel::Debug, DebugLogContext(__FILE__, __LINE__) __VA_OPT__(,) __VA_ARGS__)
+    #define LogInfo(...) Log(LogLevel::Info, nullptr __VA_OPT__(,) __VA_ARGS__)
+    #define LogWarning(...) Log(LogLevel::Warning, DebugLogContext(__FILE__, __LINE__) __VA_OPT__(,) __VA_ARGS__)
+    #define LogError(...) Log(LogLevel::Error, DebugLogContext(__FILE__, __LINE__) __VA_OPT__(,) __VA_ARGS__)
 #else
     template <typename... Args>
     static inline void LogDebug(Args...) {}

package/src/koffi/CMakeLists.txt

@@ -30,7 +30,7 @@ include(CheckCXXCompilerFlag)
 set(CMAKE_CXX_STANDARD 20)
 if(MSVC)
     set(CMAKE_MSVC_RUNTIME_LIBRARY MultiThreaded)
-    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /Zc:__cplusplus /W4 /wd4200 /wd4201 /wd4127 /wd4458 /wd4706 /wd4702 /wd4324")
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /Zc:__cplusplus /Zc:preprocessor /W4 /wd4200 /wd4201 /wd4127 /wd4458 /wd4706 /wd4702 /wd4324")
 
     # ASM_MASM does not (yet) work on Windows ARM64
     if(NOT CMAKE_GENERATOR_PLATFORM MATCHES "ARM64")

package/src/koffi/src/call.cc

@@ -209,8 +209,7 @@ bool CallData::PushString(Napi::Value value, int directions, const char **out_st
 
         return true;
     } else {
-        ThrowError<Napi::TypeError>(env, "Unexpected %1 value, expected string", GetValueType(instance, value));
-        return false;
+        return PushPointer(value, instance->str_type, directions, (void **)out_str);
     }
 }
 
@@ -311,8 +310,7 @@ bool CallData::PushString16(Napi::Value value, int directions, const char16_t **
 
         return true;
     } else {
-        ThrowError<Napi::TypeError>(env, "Unexpected %1 value, expected string", GetValueType(instance, value));
-        return false;
+        return PushPointer(value, instance->str16_type, directions, (void **)out_str16);
     }
 }
 
@@ -983,7 +981,9 @@ bool CallData::PushPointer(Napi::Value value, const TypeInfo *type, int directio
         } break;
 
         case napi_external: {
-            RG_ASSERT(type->primitive == PrimitiveKind::Pointer);
+            RG_ASSERT(type->primitive == PrimitiveKind::Pointer ||
+                      type->primitive == PrimitiveKind::String ||
+                      type->primitive == PrimitiveKind::String16);
 
             if (!CheckValueTag(instance, value, type->ref.marker) &&
                     !CheckValueTag(instance, value, instance->void_type) &&

package/src/koffi/src/ffi.cc

@@ -770,6 +770,55 @@ static inline bool GetExternalPointer(Napi::Env env, Napi::Value value, void **o
     }
 }
 
+static Napi::Value CallAlloc(const Napi::CallbackInfo &info)
+{
+    Napi::Env env = info.Env();
+    InstanceData *instance = env.GetInstanceData<InstanceData>();
+
+    if (info.Length() < 2) {
+        ThrowError<Napi::TypeError>(env, "Expected 2 arguments, got %1", info.Length());
+        return env.Null();
+    }
+    if (!info[1].IsNumber()) {
+        ThrowError<Napi::TypeError>(env, "Unexpected %1 value for length, expected number", GetValueType(instance, info[1]));
+        return env.Null();
+    }
+
+    const TypeInfo *type = ResolveType(info[0]);
+    if (!type)
+        return env.Null();
+
+    if (!type->size) [[unlikely]] {
+        ThrowError<Napi::TypeError>(env, "Cannot allocate memory for zero-sized type %1", type->name);
+        return env.Null();
+    }
+
+    int32_t len = info[1].As<Napi::Number>();
+
+    if (len <= 0) [[unlikely]] {
+        ThrowError<Napi::Error>(env, "Size must be greater than 0");
+        return env.Null();
+    }
+    if (len > INT32_MAX / type->size) [[unlikely]] {
+        ThrowError<Napi::Error>(env, "Cannot allocate more than %1 objects of type %2", INT32_MAX / type->size, type->name);
+        return env.Null();
+    }
+
+    void *ptr = calloc((size_t)len, (size_t)type->size);
+
+    if (!ptr) [[unlikely]] {
+        Size size = (Size)(len * type->size);
+
+        ThrowError<Napi::Error>(env, "Failed to allocate %1 of memory", FmtMemSize((Size)size));
+        return env.Null();
+    }
+
+    Napi::External<void> external = Napi::External<void>::New(env, ptr);
+    SetValueTag(instance, external, type);
+
+    return external;
+}
+
 static Napi::Value CallFree(const Napi::CallbackInfo &info)
 {
     Napi::Env env = info.Env();
@@ -2241,6 +2290,7 @@ static Napi::Object InitModule(Napi::Env env, Napi::Object exports)
     exports.Set("inout", Napi::Function::New(env, MarkInOut, "inout"));
 
     exports.Set("disposable", Napi::Function::New(env, CreateDisposableType, "disposable"));
+    exports.Set("alloc", Napi::Function::New(env, CallAlloc, "alloc"));
     exports.Set("free", Napi::Function::New(env, CallFree, "free"));
 
     exports.Set("register", Napi::Function::New(env, RegisterCallback, "register"));
@@ -2327,6 +2377,8 @@ static Napi::Object InitModule(Napi::Env env, Napi::Object exports)
         instance->void_type = instance->types_map.FindValue("void", nullptr);
         instance->char_type = instance->types_map.FindValue("char", nullptr);
         instance->char16_type = instance->types_map.FindValue("char16", nullptr);
+        instance->str_type = instance->types_map.FindValue("char *", nullptr);
+        instance->str16_type = instance->types_map.FindValue("char16_t *", nullptr);
 
         instance->active_symbol = Napi::Symbol::New(env, "active");
 

package/src/koffi/src/ffi.hh

@@ -277,6 +277,8 @@ struct InstanceData {
     const TypeInfo *void_type;
     const TypeInfo *char_type;
     const TypeInfo *char16_type;
+    const TypeInfo *str_type;
+    const TypeInfo *str16_type;
 
     Napi::Symbol active_symbol;
 

package/doc/polymorphism.md

@@ -1,108 +0,0 @@
-# Polymorphic arguments
-
-## Input polymorphism
-
-*New in Koffi 2.1*
-
-Many C functions use `void *` parameters in order to pass polymorphic objects and arrays, meaning that the data format changes can change depending on one other argument, or on some kind of struct tag member.
-
-Koffi provides two features to deal with this:
-
-- Buffers and typed JS arrays can be used as values in place everywhere a pointer is expected. See [dynamic arrays](pointers.md#array-pointers-dynamic-arrays) for more information, for input or output.
-- You can use `koffi.as(value, type)` to tell Koffi what kind of type is actually expected.
-
-The example below shows the use of `koffi.as()` to read the header of a PNG file with `fread()`.
-
-```js
-// ES6 syntax: import koffi from 'koffi';
-const koffi = require('koffi');
-
-const lib = koffi.load('libc.so.6');
-
-const FILE = koffi.opaque('FILE');
-
-const PngHeader = koffi.pack('PngHeader', {
-    signature: koffi.array('uint8_t', 8),
-    ihdr: koffi.pack({
-        length: 'uint32_be_t',
-        chunk: koffi.array('char', 4),
-        width: 'uint32_be_t',
-        height: 'uint32_be_t',
-        depth: 'uint8_t',
-        color: 'uint8_t',
-        compression: 'uint8_t',
-        filter: 'uint8_t',
-        interlace: 'uint8_t',
-        crc: 'uint32_be_t'
-    })
-});
-
-const fopen = lib.func('FILE *fopen(const char *path, const char *mode)');
-const fclose = lib.func('int fclose(FILE *fp)');
-const fread = lib.func('size_t fread(_Out_ void *ptr, size_t size, size_t nmemb, FILE *fp)');
-
-let filename = process.argv[2];
-if (filename == null)
-    throw new Error('Usage: node png.js <image.png>');
-
-let hdr = {};
-{
-    let fp = fopen(filename, 'rb');
-    if (!fp)
-        throw new Error(`Failed to open '${filename}'`);
-
-    try {
-        let len = fread(koffi.as(hdr, 'PngHeader *'), 1, koffi.sizeof(PngHeader), fp);
-        if (len < koffi.sizeof(PngHeader))
-            throw new Error('Failed to read PNG header');
-    } finally {
-        fclose(fp);
-    }
-}
-
-console.log('PNG header:', hdr);
-```
-
-## Output buffers
-
-*New in Koffi 2.3*
-
-You can use buffers and typed arrays for output (and input/output) pointer parameters. Simply pass the buffer as an argument and the native function will receive a pointer to its contents.
-
-Once the native function returns, you can decode the content with `koffi.decode(value, type)` as in the following example:
-
-```js
-// ES6 syntax: import koffi from 'koffi';
-const koffi = require('koffi');
-
-const lib = koffi.load('libc.so.6');
-
-const Vec3 = koffi.struct('Vec3', {
-    x: 'float32',
-    y: 'float32',
-    z: 'float32'
-})
-
-const memcpy = lib.func('void *memcpy(_Out_ void *dest, const void *src, size_t size)');
-
-let vec1 = { x: 1, y: 2, z: 3 };
-let vec2 = null;
-
-// Copy the vector in a convoluted way through memcpy
-{
-    let src = koffi.as(vec1, 'Vec3 *');
-    let dest = Buffer.allocUnsafe(koffi.sizeof(Vec3));
-
-    memcpy(dest, src, koffi.sizeof(Vec3));
-
-    vec2 = koffi.decode(dest, Vec3);
-}
-
-// CHange vector1, leaving copy alone
-[vec1.x, vec1.y, vec1.z] = [vec1.z, vec1.y, vec1.x];
-
-console.log(vec1); // { x: 3, y: 2, z: 1 }
-console.log(vec2); // { x: 1, y: 2, z: 3 }
-```
-
-See [decoding variables](variables.md#decode-to-js-values) for more information about the decode function.