koffi 2.2.6-beta.3 → 2.3.0-beta.2

package/CHANGELOG.md

@@ -1,14 +1,16 @@
 # Changelog
 
-## Koffi 2.x
+## Version history
 
-### Koffi 2.2.5
+### Koffi 2.2
+
+#### Koffi 2.2.5
 
 **Main changes:**
 
 - Relicense Koffi under LGPL 3.0
 
-### Koffi 2.2.4
+#### Koffi 2.2.4
 
 **Main fixes:**
 
@@ -18,7 +20,7 @@
 
 - Reorganize documentation pages
 
-### Koffi 2.2.3
+#### Koffi 2.2.3
 
 **Main fixes:**
 
@@ -28,7 +30,7 @@
 
 - Try to use ebp/rbp as frame pointer in x86/x64 ASM code
 
-### Koffi 2.2.2
+#### Koffi 2.2.2
 
 **Main fixes:**
 
@@ -42,13 +44,13 @@
 - Check N-API version when module is loaded
 - Optimize callback unregistration
 
-### Koffi 2.2.1
+#### Koffi 2.2.1
 
 **Main fixes:**
 
 - Fix crash when [calling callback again after FFI call inside previous callback](https://github.com/Koromix/rygel/issues/15)
 
-### Koffi 2.2.0
+#### Koffi 2.2.0
 
 **New features:**
 
@@ -62,13 +64,15 @@
 - Correctly validate output parameter types
 - Fix assertion with `void *` parameters
 
-### Koffi 2.1.5
+### Koffi 2.1
+
+#### Koffi 2.1.5
 
 **Main fixes:**
 
 - Add missing README.md file to NPM package
 
-### Koffi 2.1.4
+#### Koffi 2.1.4
 
 **Main changes:**
 
@@ -79,25 +83,25 @@
 
 - Moved Koffi to a new repository: https://github.com/Koromix/rygel/
 
-### Koffi 2.1.3
+#### Koffi 2.1.3
 
 **Main changes:**
 
 - Support up to 16 output parameters (instead of 8)
 
-### Koffi 2.1.2
+#### Koffi 2.1.2
 
 **Main changes:**
 
 - Support up to 8 output parameters (instead of 4)
 
-### Koffi 2.1.1
+#### Koffi 2.1.1
 
 **Main fixes:**
 
 - Fix potential memory allocation bugs
 
-### Koffi 2.1.0
+#### Koffi 2.1.0
 
 **Main changes:**
 
@@ -115,13 +119,15 @@
 - Detect impossible parameter and return types (such as non-pointer opaque types)
 - Various documentation fixes and improvements
 
-### Koffi 2.0.1
+### Koffi 2.0
+
+#### Koffi 2.0.1
 
 **Main changes:**
 
 - Return `undefined` (instead of null) for `void` functions
 
-### Koffi 2.0.0
+#### Koffi 2.0.0
 
 **Major new features:**
 
@@ -145,21 +151,21 @@
 
 Consult the [migration guide](migration.md) for more information.
 
-## Koffi 1.x
+### Koffi 1.3
 
-### Koffi 1.3.12
+#### Koffi 1.3.12
 
 **Main fixes:**
 
 - Fix support for Yarn package manager
 
-### Koffi 1.3.11
+#### Koffi 1.3.11
 
 **Main fixes:**
 
 - Fix broken parsing of `void *` when used for first parameter
 
-### Koffi 1.3.10
+#### Koffi 1.3.10
 
 **Main fixes:**
 
@@ -171,13 +177,13 @@ Consult the [migration guide](migration.md) for more information.
 
 - Various documentation fixes and improvements
 
-### Koffi 1.3.9
+#### Koffi 1.3.9
 
 **Main fixes:**
 
 - Fix prebuild compatibility with Electron on Windows x64
 
-### Koffi 1.3.8
+#### Koffi 1.3.8
 
 **Main changes:**
 
@@ -188,7 +194,7 @@ Consult the [migration guide](migration.md) for more information.
 
 - Fix and harmonize a few error messages
 
-### Koffi 1.3.7
+#### Koffi 1.3.7
 
 **Main fixes:**
 
@@ -201,7 +207,7 @@ Consult the [migration guide](migration.md) for more information.
 - Add str/str16 type aliases for string/string16
 - Various documentation fixes and improvements
 
-### Koffi 1.3.6
+#### Koffi 1.3.6
 
 **Main fixes:**
 
@@ -213,7 +219,7 @@ Consult the [migration guide](migration.md) for more information.
 - Prebuild with Clang for Windows x64 and Linux x64 binaries
 - Various documentation improvements
 
-### Koffi 1.3.5
+#### Koffi 1.3.5
 
 **Main changes:**
 
@@ -225,13 +231,13 @@ Consult the [migration guide](migration.md) for more information.
 - Reduce default async memory stack and heap size
 - Various documentation improvements
 
-### Koffi 1.3.4
+#### Koffi 1.3.4
 
 **Main fixes:**
 
 - Fix possible OpenBSD i386 crash with `(void)` functions
 
-### Koffi 1.3.3
+#### Koffi 1.3.3
 
 **Main fixes:**
 
@@ -243,20 +249,20 @@ Consult the [migration guide](migration.md) for more information.
 - Disable unsafe compiler optimizations
 - Various documentation improvements
 
-### Koffi 1.3.2
+#### Koffi 1.3.2
 
 **Main fixes:**
 
 - Support compilation in C++14 mode (graceful degradation)
 - Support older toolchains on Linux (tested on Debian 9)
 
-### Koffi 1.3.1
+#### Koffi 1.3.1
 
 **Main fixes:**
 
 - The prebuilt binary is tested when Koffi is installed, and a rebuild happens if it fails to load
 
-### Koffi 1.3.0
+#### Koffi 1.3.0
 
 **Major changes:**
 
@@ -272,19 +278,21 @@ Consult the [migration guide](migration.md) for more information.
 - Detect floating-point ABI before using prebuilt binaries (ARM32, RISC-V)
 - Forbid duplicate member names in struct types
 
-### Koffi 1.2.4
+### Koffi 1.2
+
+#### Koffi 1.2.4
 
 **New features:**
 
 - Windows ARM64 is now supported
 
-### Koffi 1.2.3
+#### Koffi 1.2.3
 
 **New features:**
 
 - A prebuilt binary for macOS ARM64 (M1) is now included
 
-### Koffi 1.2.1
+#### Koffi 1.2.1
 
 This entry documents changes since version 1.1.0.
 
@@ -308,3 +316,16 @@ This entry documents changes since version 1.1.0.
 - Fix value `koffi.internal` to false in module (normal) builds
 - Make sure we have a redzone below the stack for all architectures
 - Use slower allocation for big objects instead of failing
+
+## Todo list
+
+The following features and improvements are planned, not necessarily in that order:
+
+- Optimize passing of structs and arrays (avoid setting named properties one by one? separate HFA-specific helper functions?)
+- Automate Windows/AArch64 (qemu) and macOS/AArch64 (how? ... thanks Apple) tests
+- Create a real-world example, using several libraries (Raylib, SQLite, libsodium) to illustrate various C API styles
+- Add simple struct type parser
+- Add more ways to manually encode and decode various types to and from byte arrays
+- Add support for unions
+- Port Koffi to PowerPC (POWER9+) ABI
+- Fix assembly unwind and CFI directives for better debugging experience

package/doc/callbacks.md

@@ -136,18 +136,16 @@ let cb2 = koffi.register(store, store.get, 'IntCallback *'); // However in this
 
 ## Pointer arguments
 
-*New in Koffi 2.2*
+*New in Koffi 2.2, changed in Koffi 2.3*
 
 Koffi does not have enough information to convert callback pointer arguments to an appropriate JS value. In this case, your JS function will receive an opaque *External* object.
 
-You can pass this value through to another C function that expects a pointer of the same type, or you can use `koffi.decode(value, offset, type, len)` to decode it into something you can use in Javascript.
+You can pass this value through to another C function that expects a pointer of the same type, or you can use `koffi.decode()` to decode it into something you can use in Javascript.
 
 Some arguments are optional and this function can be called in several ways:
 
-- `koffi.decode(value, offset, type, len)`: this is the full signature, value is the JS external object, offset (optional, defaults to 0) is specified in bytes, type is the value type, and length (optional) can be used for arrays and strings.
 - `koffi.decode(value, type)`: no offset, expect NUL-terminated strings
-- `koffi.decode(value, type, len)`: use specified length when decoding strings and arrays
-- `koffi.decode(value, offset, type)`
+- `koffi.decode(value, offset, type)`: explicit offset to add to the pointer before decoding
 
 The following example sorts an array of strings (in-place) with `qsort()`:
 
@@ -170,6 +168,15 @@ qsort(koffi.as(array, 'char **'), array.length, koffi.sizeof('void *'), (ptr1, p
 console.log(array); // Prints ['123', 'bar', 'foo', 'foobar']
 ```
 
+There is also an optional ending `length` argument that you can use in two cases:
+
+- Use it to give the number of bytes to decode in non-NUL terminated strings: `koffi.decode(value, 'char *', 5)`
+- Decode consecutive values into an array. For example, here is how you can decode an array with 3 float values: `koffi.decode(value, 'float', 3)`. This is equivalent to `koffi.decode(value, koffi.array('float', 3))`.
+
+```{note}
+In Koffi 2.2 and earlier versions, the length argument is only used to decode strings and is ignored otherwise.
+```
+
 ## Asynchronous callbacks
 
 *New in Koffi 2.2.2*

package/doc/calls.md

@@ -159,13 +159,15 @@ console.log(out[0]);
 
 ## Polymorphic parameters
 
+### Input parameters
+
 *New in Koffi 2.1*
 
 Many C functions use `void *` parameters in order to pass polymorphic objects and arrays, meaning that the data format changes can change depending on one other argument, or on some kind of struct tag member.
 
 Koffi provides two features to deal with this:
 
-- Typed JS arrays can be used as values in place everywhere `void *` is expected. See [dynamic arrays](types.md#array-pointers-dynamic-arrays) for more information, for input or output.
+- Buffers and typed JS arrays can be used as values in place everywhere a pointer is expected. See [dynamic arrays](pointers.md#array-pointers-dynamic-arrays) for more information, for input or output.
 - You can use `koffi.as(value, type)` to tell Koffi what kind of type is actually expected.
 
 The example below shows the use of `koffi.as()` to read the header of a PNG file with `fread()`.
@@ -219,6 +221,48 @@ let hdr = {};
 console.log('PNG header:', hdr);
 ```
 
+### Output parameters
+
+*New in Koffi 2.3*
+
+You can use buffers and typed arrays for output (and input/output) pointer parameters. Simply pass the buffer as an argument and the native function will receive a pointer to its contents.
+
+Once the native function returns, you can decode the content with `koffi.decode(value, type)` as in the following example:
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('libc.so.6');
+
+const Vec3 = koffi.struct('Vec3', {
+    x: 'float32',
+    y: 'float32',
+    z: 'float32'
+})
+
+const memcpy = lib.func('void *memcpy(_Out_ void *dest, const void *src, size_t size)');
+
+let vec1 = { x: 1, y: 2, z: 3 };
+let vec2 = null;
+
+// Copy the vector in a convoluted way through memcpy
+{
+    let src = koffi.as(vec1, 'Vec3 *');
+    let dest = Buffer.allocUnsafe(koffi.sizeof(Vec3));
+
+    memcpy(dest, src, koffi.sizeof(Vec3));
+
+    vec2 = koffi.decode(dest, Vec3);
+}
+
+// CHange vector1, leaving copy alone
+[vec1.x, vec1.y, vec1.z] = [vec1.z, vec1.y, vec1.x];
+
+console.log(vec1); // { x: 3, y: 2, z: 1 }
+console.log(vec2); // { x: 1, y: 2, z: 3 }
+```
+
+See [pointer arguments](callbacks.md#pointer-arguments) for more information about the decode function.
+
 ## Heap-allocated values
 
 *New in Koffi 2.0*

package/doc/contribute.md

@@ -107,19 +107,6 @@ Each machine is configured to run a VNC server available locally, which you can
 node qemu.js info debian_x64
 ```
 
-## Todo list
-
-The following features and improvements are planned, not necessarily in that order:
-
-- Optimize passing of structs and arrays (avoid setting named properties one by one? separate HFA-specific helper functions?)
-- Automate Windows/AArch64 (qemu) and macOS/AArch64 (how? ... thanks Apple) tests
-- Create a real-world example, using several libraries (Raylib, SQLite, libsodium) to illustrate various C API styles
-- Add simple struct type parser
-- Add more ways to manually encode and decode various types to and from byte arrays
-- Add support for unions
-- Port Koffi to PowerPC (POWER9+) ABI
-- Fix assembly unwind and CFI directives for better debugging experience
-
 ## Code style
 
 Koffi is programmed in a mix of C++ and assembly code (architecture-specific code). It uses [node-addon-api](https://github.com/nodejs/node-addon-api) (C++ N-API wrapper) to interact with Node.js.

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "koffi",
-    "version": "2.2.6-beta.3",
+    "version": "2.3.0-beta.2",
     "stable": "2.2.5",
     "description": "Fast and simple C FFI (foreign function interface) for Node.js",
     "keywords": [

package/src/koffi/src/call.cc

@@ -923,15 +923,17 @@ bool CallData::PushPointer(Napi::Value value, const TypeInfo *type, int directio
                     memset_safe(ptr, 0, size);
                 }
             } else if (IsRawBuffer(value)) {
-                Span<const uint8_t> buffer = GetRawBuffer(value);
+                Span<uint8_t> buffer = GetRawBuffer(value);
 
-                ptr = AllocHeap(buffer.len, 16);
+                if (directions == 1) {
+                    ptr = AllocHeap(buffer.len, 16);
 
-                if (directions & 1) {
                     if (!PushBuffer(buffer, buffer.len, type, ptr))
                         return false;
                 } else {
-                    memset_safe(ptr, 0, (size_t)buffer.len);
+                    // Fast no-copy path
+                    ptr = buffer.ptr;
+                    directions = 1;
                 }
             } else if (RG_LIKELY(type->ref.type->primitive == PrimitiveKind::Record)) {
                 Napi::Object obj = value.As<Napi::Object>();

package/src/koffi/src/ffi.cc

@@ -626,7 +626,8 @@ static Napi::Value CreateArrayType(const Napi::CallbackInfo &info)
         return env.Null();
     }
 
-    TypeInfo::ArrayHint hint;
+    const TypeInfo *type = nullptr;
+
     if (info.Length() >= 3 && !IsNullOrUndefined(info[2])) {
         if (!info[2].IsString()) {
             ThrowError<Napi::TypeError>(env, "Unexpected %1 value for hint, expected string", GetValueType(instance, info[2]));
@@ -634,6 +635,7 @@ static Napi::Value CreateArrayType(const Napi::CallbackInfo &info)
         }
 
         std::string to = info[2].As<Napi::String>();
+        TypeInfo::ArrayHint hint = {};
 
         if (to == "typed") {
             hint = TypeInfo::ArrayHint::TypedArray;
@@ -650,24 +652,13 @@ static Napi::Value CreateArrayType(const Napi::CallbackInfo &info)
             ThrowError<Napi::Error>(env, "Array conversion hint must be 'typed', 'array' or 'string'");
             return env.Null();
         }
-    } else if (TestStr(ref->name, "char") || TestStr(ref->name, "char16") ||
-                                             TestStr(ref->name, "char16_t")) {
-        hint = TypeInfo::ArrayHint::String;
+
+        type = MakeArrayType(instance, ref, len, hint);
     } else {
-        hint = TypeInfo::ArrayHint::TypedArray;
+        type = MakeArrayType(instance, ref, len);
     }
 
-    TypeInfo *type = instance->types.AppendDefault();
-
-    type->name = Fmt(&instance->str_alloc, "%1[%2]", ref->name, len).ptr;
-
-    type->primitive = PrimitiveKind::Array;
-    type->align = ref->align;
-    type->size = (int32_t)(len * ref->size);
-    type->ref.type = ref;
-    type->hint = hint;
-
-    Napi::External<TypeInfo> external = Napi::External<TypeInfo>::New(env, type);
+    Napi::External<TypeInfo> external = Napi::External<TypeInfo>::New(env, (TypeInfo *)type);
     SetValueTag(instance, external, &TypeInfoMarker);
 
     return external;
@@ -1837,10 +1828,6 @@ static Napi::Value DecodeValue(const Napi::CallbackInfo &info)
         ThrowError<Napi::TypeError>(env, "Expected %1 to 4 arguments, got %2", 2 + has_offset, info.Length());
         return env.Null();
     }
-    if (has_len && RG_UNLIKELY(!info[2u + has_offset].IsNumber())) {
-        ThrowError<Napi::TypeError>(env, "Unexpected %1 value for length, expected number", GetValueType(instance, info[2 + has_offset]));
-        return env.Null();
-    }
 
     const TypeInfo *type = ResolveType(info[1u + has_offset]);
     if (RG_UNLIKELY(!type))
@@ -1876,6 +1863,17 @@ static Napi::Value DecodeValue(const Napi::CallbackInfo &info)
     // Used for strings and arrays, ignored otherwise
     int64_t len = has_len ? info[2u + has_offset].As<Napi::Number>().Int64Value() : -1;
 
+    if (has_len) {
+        if (RG_UNLIKELY(len < 0)) {
+            ThrowError<Napi::TypeError>(env, "Length must not be a negative value");
+            return env.Null();
+        }
+
+        if (type->primitive != PrimitiveKind::String && type->primitive != PrimitiveKind::String16) {
+            type = MakeArrayType(instance, type, len);
+        }
+    }
+
 #define RETURN_INT(Type, NewCall) \
         do { \
             Type v = *(Type *)ptr; \

package/src/koffi/src/util.cc

@@ -170,6 +170,50 @@ const TypeInfo *MakePointerType(InstanceData *instance, const TypeInfo *ref, int
     return ref;
 }
 
+static const TypeInfo *MakeArrayType(InstanceData *instance, const TypeInfo *ref, Size len,
+                                     TypeInfo::ArrayHint hint, bool insert)
+{
+    RG_ASSERT(len > 0);
+    RG_ASSERT(len <= instance->max_type_size / ref->size);
+
+    TypeInfo *type = instance->types.AppendDefault();
+
+    type->name = Fmt(&instance->str_alloc, "%1[%2]", ref->name, len).ptr;
+
+    type->primitive = PrimitiveKind::Array;
+    type->align = ref->align;
+    type->size = (int32_t)(len * ref->size);
+    type->ref.type = ref;
+    type->hint = hint;
+
+    if (insert) {
+        bool inserted;
+        type = (TypeInfo *)*instance->types_map.TrySet(type->name, type, &inserted);
+        instance->types.RemoveLast(!inserted);
+    }
+
+    return type;
+}
+
+const TypeInfo *MakeArrayType(InstanceData *instance, const TypeInfo *ref, Size len)
+{
+    TypeInfo::ArrayHint hint = {};
+
+    if (TestStr(ref->name, "char") || TestStr(ref->name, "char16") ||
+                                             TestStr(ref->name, "char16_t")) {
+        hint = TypeInfo::ArrayHint::String;
+    } else {
+        hint = TypeInfo::ArrayHint::TypedArray;
+    }
+
+    return MakeArrayType(instance, ref, len, hint, true);
+}
+
+const TypeInfo *MakeArrayType(InstanceData *instance, const TypeInfo *ref, Size len, TypeInfo::ArrayHint hint)
+{
+    return MakeArrayType(instance, ref, len, hint, false);
+}
+
 bool CanPassType(const TypeInfo *type, int directions)
 {
     if (directions & 2) {

package/src/koffi/src/util.hh

@@ -55,7 +55,10 @@ static inline bool IsRegularSize(Size size, Size max)
 
 const TypeInfo *ResolveType(Napi::Value value, int *out_directions = nullptr);
 const TypeInfo *ResolveType(InstanceData *instance, Span<const char> str, int *out_directions = nullptr);
-const TypeInfo *MakePointerType(InstanceData *instance, const TypeInfo *type, int count = 1);
+
+const TypeInfo *MakePointerType(InstanceData *instance, const TypeInfo *ref, int count = 1);
+const TypeInfo *MakeArrayType(InstanceData *instance, const TypeInfo *ref, Size len);
+const TypeInfo *MakeArrayType(InstanceData *instance, const TypeInfo *ref, Size len, TypeInfo::ArrayHint hint);
 
 bool CanPassType(const TypeInfo *type, int directions);
 bool CanReturnType(const TypeInfo *type);