koffi 2.1.0-beta.1 → 2.1.0-beta.2

package/CMakeLists.txt

@@ -31,7 +31,7 @@ if(MSVC)
         enable_language(ASM_MASM)
     endif()
 else()
-    add_compile_options(-Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter)
+    add_compile_options(-Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wswitch -Werror=switch)
     if(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
         add_compile_options(-Wno-unknown-warning-option)
     endif()

package/ChangeLog.md

@@ -2,6 +2,23 @@
 
 ## History
 
+### Koffi 2.1.0 (in beta)
+
+**Main changes:**
+
+- Add [koffi.as()](functions.md#polymorphic-parameters) to support polymorphic APIs based on `void *` parameters
+- Add [endian-sensitive integer types](types.md#endian-sensitive-types): `intX_le_t`, `intX_be_t`
+- Accept typed arrays for `void *` parameters
+- Introduce `koffi.opaque()` to replace `koffi.handle()` (which remains supported until Koffi 3.0)
+
+**Other changes:**
+
+- Improve global performance with inlining and unity builds
+- Add `size_t` primitive type
+- Support member-specific alignement value in structs
+- Detect impossible parameters and return types (such as non-pointer opaque types)
+- Various documentation fixes and improvements
+
 ### Koffi 2.0.1
 
 **Main changes:**

package/doc/functions.md

@@ -105,7 +105,7 @@ printf('Integer %d, double %g, str %s', 'int', 6, 'double', 8.5, 'str', 'THE END
 
 On x86 platforms, only the Cdecl convention can be used for variadic functions.
 
-## C to JS conversion gotchas
+## Special considerations
 
 ### Output parameters
 
@@ -177,8 +177,72 @@ let db = out[0];
 sqlite3_close_v2(db);
 ```
 
+### Polymorphic parameters
+
+*New in Koffi 2.1*
+
+Many C functions use `void *` parameters in order to pass polymorphic objects and arrays, meaning that the data format changes can change depending on one other argument, or on some kind of struct tag member.
+
+Koffi provides two features to deal with this:
+
+- Typed JS arrays can be used as values in place everywhere `void *` is expected. See [dynamic arrays](types.md#array-pointers-dynamic-arrays) for more information, for input or output.
+- You can use `koffi.as(value, type)` to tell Koffi what kind of type is actually expected.
+
+The example below shows the use of `koffi.as()` to read the header of a PNG file with `fread()`.
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('libc.so.6');
+
+const FILE = koffi.opaque('FILE');
+
+const PngHeader = koffi.pack('PngHeader', {
+    signature: koffi.array('uint8_t', 8),
+    ihdr: koffi.pack({
+        length: 'uint32_be_t',
+        chunk: koffi.array('char', 4),
+        width: 'uint32_be_t',
+        height: 'uint32_be_t',
+        depth: 'uint8_t',
+        color: 'uint8_t',
+        compression: 'uint8_t',
+        filter: 'uint8_t',
+        interlace: 'uint8_t',
+        crc: 'uint32_be_t'
+    })
+});
+
+const fopen = lib.func('FILE *fopen(const char *path, const char *mode)');
+const fclose = lib.func('int fclose(FILE *fp)');
+const fread = lib.func('size_t fread(_Out_ void *ptr, size_t size, size_t nmemb, FILE *fp)');
+
+let filename = process.argv[2];
+if (filename == null)
+    throw new Error('Usage: node png.js <image.png>');
+
+let hdr = {};
+{
+
+    let fp = fopen(filename, 'rb');
+    if (!fp)
+        throw new Error(`Failed to open '${filename}'`);
+
+    try {
+        let len = fread(koffi.as(hdr, 'PngHeader *'), 1, koffi.sizeof(PngHeader), fp);
+        if (len < koffi.sizeof(PngHeader))
+            throw new Error('Failed to read PNG header');
+    } finally {
+        fclose(fp);
+    }
+}
+
+console.log('PNG header:', hdr);
+```
+
 ### Heap-allocated values
 
+*New in Koffi 2.0*
+
 Some C functions return heap-allocated values directly or through output parameters. While Koffi automatically converts values from C to JS (to a string or an object), it does not know when something needs to be freed, or how.
 
 For opaque types, such as FILE, this does not matter because you will explicitly call `fclose()` on them. But some values (such as strings) get implicitly converted by Koffi, and you lose access to the original pointer. This creates a leak if the string is heap-allocated.
@@ -299,6 +363,8 @@ console.log(ret);
 
 ### Registered callbacks
 
+*New in Koffi 2.0*
+
 Use registered callbacks when the function needs to be called at a later time (e.g. log handler, event handler, `fopencookie/funopen`). Call `koffi.register(func, type)` to register a callback function, with two arguments: the JS function, and the callback type.
 
 When you are done, call `koffi.unregister()` (with the value returned by `koffi.register()`) to release the slot. A maximum of 16 registered callbacks can exist at the same time. Failure to do so will leak the slot, and subsequent registrations may fail (with an exception) once all slots are used.

package/doc/templates/badges.html

@@ -1,5 +1,4 @@
 <div style="text-align: center; margin-top: 2em;">
-    <a href="https://www.npmjs.com/package/koffi"><img src="https://img.shields.io/badge/NPM-{{version}}-brightgreen" alt="NPM"/></a>
+    <a href="https://www.npmjs.com/package/koffi"><img src="https://img.shields.io/badge/NPM-Koffi-brightgreen" alt="NPM"/></a>
     <a href="https://github.com/Koromix/luigi/tree/master/koffi"><img src="https://img.shields.io/badge/GitHub-Koffi-ff6600" alt="GitHub"/></a>
 </div>
-

package/doc/types.md

@@ -2,45 +2,34 @@
 
 ## Primitive types
 
+### Standard types
+
 While the C standard allows for variation in the size of most integer types, Koffi enforces the same definition for most primitive types, listed below:
 
-JS type           | C type             | Bytes | Signedness | Note
------------------ | ------------------ | ----- | ---------- | ---------------------------
-Undefined         | void               | 0     |            | Only valid as a return type
-Number (integer)  | int8               | 1     | Signed     |
-Number (integer)  | int8_t             | 1     | Signed     |
-Number (integer)  | uint8              | 1     | Unsigned   |
-Number (integer)  | uint8_t            | 1     | Unsigned   |
-Number (integer)  | char               | 1     | Signed     |
-Number (integer)  | uchar              | 1     | Unsigned   |
-Number (integer)  | unsigned char      | 1     | Unsigned   |
-Number (integer)  | char16             | 2     | Signed     |
-Number (integer)  | char16_t           | 2     | Signed     |
-Number (integer)  | int16              | 2     | Signed     |
-Number (integer)  | int16_t            | 2     | Signed     |
-Number (integer)  | uint16             | 2     | Unsigned   |
-Number (integer)  | uint16_t           | 2     | Unsigned   |
-Number (integer)  | short              | 2     | Signed     |
-Number (integer)  | unsigned short     | 2     | Unsigned   |
-Number (integer)  | int32              | 4     | Signed     |
-Number (integer)  | int32_t            | 4     | Signed     |
-Number (integer)  | uint32             | 4     | Unsigned   |
-Number (integer)  | uint32_t           | 4     | Unsigned   |
-Number (integer)  | int                | 4     | Signed     |
-Number (integer)  | uint               | 4     | Unsigned   |
-Number (integer)  | unsigned int       | 4     | Unsigned   |
-Number (integer)  | int64              | 8     | Signed     |
-Number (integer)  | int64_t            | 8     | Signed     |
-Number (integer)  | uint64             | 8     | Unsigned   |
-Number (integer)  | uint64_t           | 8     | Unsigned   |
-Number (integer)  | longlong           | 8     | Signed     |
-Number (integer)  | long long          | 8     | Signed     |
-Number (integer)  | ulonglong          | 8     | Unsigned   |
-Number (integer)  | unsigned long long | 8     | Unsigned   |
-Number (float)    | float32            | 4     |            |
-Number (float)    | float64            | 8     |            |
-Number (float)    | float              | 4     |            |
-Number (float)    | double             | 8     |            |
+JS type          | C type                        | Bytes | Signedness | Note
+---------------- | ----------------------------- | ----- | ---------- | ---------------------------
+Undefined        | void                          | 0     |            | Only valid as a return type
+Number (integer) | int8, int8_t                  | 1     | Signed     |
+Number (integer) | uint8, uint8_t                | 1     | Unsigned   |
+Number (integer) | char                          | 1     | Signed     |
+Number (integer) | uchar, unsigned char          | 1     | Unsigned   |
+Number (integer) | char16, char16_t              | 2     | Signed     |
+Number (integer) | int16, int16_t                | 2     | Signed     |
+Number (integer) | uint16, uint16_t              | 2     | Unsigned   |
+Number (integer) | short                         | 2     | Signed     |
+Number (integer) | ushort, unsigned short        | 2     | Unsigned   |
+Number (integer) | int32, int32_t                | 4     | Signed     |
+Number (integer) | uint32, uint32_t              | 4     | Unsigned   |
+Number (integer) | int                           | 4     | Signed     |
+Number (integer) | uint, unsigned int            | 4     | Unsigned   |
+Number (integer) | int64, int64_t                | 8     | Signed     |
+Number (integer) | uint64, uint64_t              | 8     | Unsigned   |
+Number (integer) | longlong, long long           | 8     | Signed     |
+Number (integer) | ulonglong, unsigned long long | 8     | Unsigned   |
+Number (float)   | float32                       | 4     |            |
+Number (float)   | float64                       | 8     |            |
+Number (float)   | float                         | 4     |            |
+Number (float)   | double                        | 8     |            |
 
 Koffi also accepts BigInt values when converting from JS to C integers. If the value exceeds the range of the C type, Koffi will convert the number to an undefined value. In the reverse direction, BigInt values are automatically used when needed for big 64-bit integers.
 
@@ -56,8 +45,8 @@ Number (integer) | intptr           | Signed     | 4 or 8 bytes depending on reg
 Number (integer) | intptr_t         | Signed     | 4 or 8 bytes depending on register width
 Number (integer) | uintptr          | Unsigned   | 4 or 8 bytes depending on register width
 Number (integer) | uintptr_t        | Unsigned   | 4 or 8 bytes depending on register width
-String           | str (string)     |            | JS strings are converted to and from UTF-8
-String           | str16 (string16) |            | JS strings are converted to and from UTF-16 (LE)
+String           | str, string      |            | JS strings are converted to and from UTF-8
+String           | str16, string16  |            | JS strings are converted to and from UTF-16 (LE)
 
 Primitive types can be specified by name (in a string) or through `koffi.types`:
 
@@ -67,6 +56,25 @@ let struct1 = koffi.struct({ dummy: 'long' });
 let struct2 = koffi.struct({ dummy: koffi.types.long });
 ```
 
+### Endian-sensitive types
+
+Koffi defines a bunch of endian-sensitive types, which can be used when dealing with binary data (network payloads, binary file formats, etc.).
+
+JS type          | C type                 | Bytes | Signedness | Endianness
+---------------- | ---------------------- | ----- | ---------- | -------------
+Number (integer) | int16_le, int16_le_t   | 2     | Signed     | Little Endian
+Number (integer) | int16_be, int16_be_t   | 2     | Signed     | Big Endian
+Number (integer) | uint16_le, uint16_le_t | 2     | Unsigned   | Little Endian
+Number (integer) | uint16_be, uint16_be_t | 2     | Unsigned   | Big Endian
+Number (integer) | int32_le, int32_le_t   | 4     | Signed     | Little Endian
+Number (integer) | int32_be, int32_be_t   | 4     | Signed     | Big Endian
+Number (integer) | uint32_le, uint32_le_t | 4     | Unsigned   | Little Endian
+Number (integer) | uint32_be, uint32_be_t | 4     | Unsigned   | Big Endian
+Number (integer) | int64_le, int64_le_t   | 8     | Signed     | Little Endian
+Number (integer) | int64_be, int64_be_t   | 8     | Signed     | Big Endian
+Number (integer) | uint64_le, uint64_le_t | 8     | Unsigned   | Little Endian
+Number (integer) | uint64_be, uint64_be_t | 8     | Unsigned   | Big Endian
+
 ## Struct types
 
 ### Struct definition
@@ -328,8 +336,8 @@ In C, pointer arguments are used for differenty purposes. It is important to dis
 
 - **Struct pointers**: Use of struct pointers by C libraries fall in two cases: avoid (potentially) expensive copies, and to let the function change struct contents (output or input/output arguments).
 - **Opaque pointers**: the library does not expose the contents of the structs, and only provides you with a pointer to it (e.g. `FILE *`). Only the functions provided by the library can do something with this pointer, in Koffi we call this an opaque type. This is usually done for ABI-stability reason, and to prevent library users from messing directly with library internals.
-- **Arrays**: in C, you dynamically-sized arrays are usually passed to functions with pointers, either NULL-terminated (or any other sentinel value) or with an additional length argument.
 - **Pointers to primitive types**: This is more rare, and generally used for output or input/output arguments. The Win32 API has a lot of these.
+- **Arrays**: in C, you dynamically-sized arrays are usually passed to functions with pointers, either NULL-terminated (or any other sentinel value) or with an additional length argument.
 
 ### Struct pointers
 
@@ -357,6 +365,8 @@ console.log(pos);
 
 ### Named pointer types
 
+*New in Koffi 2.0*
+
 Some C libraries use handles, which behave as pointers to opaque structs. An example of this is the HANDLE type in the Win32 API. If you want to reproduce this behavior, you can define a **named pointer type** to an opaque type, like so:
 
 ```js
@@ -367,111 +377,6 @@ const GetHandleInformation = lib.func('bool __stdcall GetHandleInformation(HANDL
 const CloseHandle = lib.func('bool __stdcall CloseHandle(HANDLE h)');
 ```
 
-### Array pointers
-
-In C, dynamically-sized arrays are usually passed around as pointers. The length is either passed as an additional argument, or inferred from the array content itself, for example with a terminating sentinel value (such as a NULL pointers in the case of an array of strings).
-
-Koffi can translate JS arrays and TypedArrays to pointer arguments. However, because C does not have a proper notion of dynamically-sized arrays (fat pointers), you need to provide the length or the sentinel value yourself depending on the API.
-
-Here is a simple example of a C function taking a NULL-terminated list of strings as input, to calculate the total length of all strings.
-
-```c
-// Build with: clang -fPIC -o length.so -shared length.c -Wall -O2
-
-#include <stdlib.h>
-#include <stdint.h>
-#include <string.h>
-
-int64_t ComputeTotalLength(const char **strings)
-{
-    int64_t total = 0;
-
-    for (const char **ptr = strings; *ptr; ptr++) {
-        const char *str = *ptr;
-        total += strlen(str);
-    }
-
-    return total;
-}
-```
-
-```js
-const koffi = require('koffi');
-const lib = koffi.load('./length.so');
-
-const ComputeTotalLength = lib.func('int64_t ComputeTotalLength(const char **strings)');
-
-let strings = ['Get', 'Total', 'Length', null];
-let total = ComputeTotalLength(strings);
-
-console.log(total); // Prints 14
-```
-
-By default, just like for objects, array arguments are copied from JS to C but not vice-versa. You can however change the direction as documented in the section on [output parameters](functions.md#output-parameters).
-
-Here is an example based on the Win32 API, listing files in the current directory with `FindFirstFileW()` and `FindNextFileW()`:
-
-```js
-const koffi = require('koffi');
-const lib = koffi.load('kernel32.dll');
-
-const HANDLE = koffi.pointer('HANDLE', koffi.opaque());
-const FILETIME = koffi.struct('FILETIME', {
-    dwLowDateTime: 'uint',
-    dwHighDateTime: 'uint'
-});
-const WIN32_FIND_DATA = koffi.struct('WIN32_FIND_DATA', {
-    dwFileAttributes: 'uint',
-    ftCreationTime: FILETIME,
-    ftLastAccessTime: FILETIME,
-    ftLastWriteTime: FILETIME,
-    nFileSizeHigh: 'uint',
-    nFileSizeLow: 'uint',
-    dwReserved0: 'uint',
-    dwReserved1: 'uint',
-    cFileName: koffi.array('char16', 260),
-    cAlternateFileName: koffi.array('char16', 14),
-    dwFileType: 'uint', // Obsolete. Do not use.
-    dwCreatorType: 'uint', // Obsolete. Do not use
-    wFinderFlags: 'ushort' // Obsolete. Do not use
-});
-
-const FindFirstFile = lib.func('HANDLE __stdcall FindFirstFileW(str16 path, _Out_ WIN32_FIND_DATA *data)');
-const FindNextFile = lib.func('bool __stdcall FindNextFileW(HANDLE h, _Out_ WIN32_FIND_DATA *data)');
-const FindClose = lib.func('bool __stdcall FindClose(HANDLE h)');
-const GetLastError = lib.func('uint GetLastError()');
-
-function list(dirname) {
-    let filenames = [];
-
-    let data = {};
-    let h = FindFirstFile(dirname + '\\*', data);
-
-    if (!h) {
-        if (GetLastError() == 2) // ERROR_FILE_NOT_FOUND
-            return filenames;
-        throw new Error('FindFirstFile() failed');
-    }
-
-    try {
-        do {
-            if (data.cFileName != '.' && data.cFileName != '..')
-                filenames.push(data.cFileName);
-        } while (FindNextFile(h, data));
-
-        if (GetLastError() != 18) // ERROR_NO_MORE_FILES
-            throw new Error('FindNextFile() failed');
-    } finally {
-        FindClose(h);
-    }
-
-    return filenames;
-}
-
-let filenames = list('.');
-console.log(filenames);
-```
-
 ### Pointers to primitive types
 
 In javascript, it is not possible to pass a primitive value by reference to another function. This means that you cannot call a function and expect it to modify the value of one of its number or string parameter.
@@ -509,7 +414,9 @@ AddInt(sum, 6);
 console.log(sum[0]); // Prints 42
 ```
 
-## Fixed-size C arrays
+## Array types
+
+### Fixed-size C arrays
 
 Fixed-size arrays are declared with `koffi.array(type, length)`. Just like in C, they cannot be passed as functions parameters (they degenerate to pointers), or returned by value. You can however embed them in struct types.
 
@@ -541,7 +448,7 @@ console.log(ReturnFoo1({ i: 5, a16: [6, 8] })) // Prints { i: 5, a16: Int16Array
 console.log(ReturnFoo2({ i: 5, a16: [6, 8] })) // Prints { i: 5, a16: [6, 8] }
 ```
 
-### Handling of strings
+### Fixed-size string buffers
 
 Koffi can also convert JS strings to fixed-sized arrays in the following cases:
 
@@ -550,6 +457,48 @@ Koffi can also convert JS strings to fixed-sized arrays in the following cases:
 
 The reverse case is also true, Koffi can convert a C fixed-size buffer to a JS string. This happens by default for char, char16 and char16_t arrays, but you can also explicitly ask for this with the `string` array hint (e.g. `koffi.array('char', 8, 'string')`).
 
+### Array pointers (dynamic arrays)
+
+In C, dynamically-sized arrays are usually passed around as pointers. The length is either passed as an additional argument, or inferred from the array content itself, for example with a terminating sentinel value (such as a NULL pointers in the case of an array of strings).
+
+Koffi can translate JS arrays and TypedArrays to pointer arguments. However, because C does not have a proper notion of dynamically-sized arrays (fat pointers), you need to provide the length or the sentinel value yourself depending on the API.
+
+Here is a simple example of a C function taking a NULL-terminated list of strings as input, to calculate the total length of all strings.
+
+```c
+// Build with: clang -fPIC -o length.so -shared length.c -Wall -O2
+
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+
+int64_t ComputeTotalLength(const char **strings)
+{
+    int64_t total = 0;
+
+    for (const char **ptr = strings; *ptr; ptr++) {
+        const char *str = *ptr;
+        total += strlen(str);
+    }
+
+    return total;
+}
+```
+
+```js
+const koffi = require('koffi');
+const lib = koffi.load('./length.so');
+
+const ComputeTotalLength = lib.func('int64_t ComputeTotalLength(const char **strings)');
+
+let strings = ['Get', 'Total', 'Length', null];
+let total = ComputeTotalLength(strings);
+
+console.log(total); // Prints 14
+```
+
+By default, just like for objects, array arguments are copied from JS to C but not vice-versa. You can however change the direction as documented in the section on [output parameters](functions.md#output-parameters).
+
 ## Disposable types
 
 Disposable types allow you to register a function that will automatically called after each C to JS conversion performed by Koffi. This can be used to avoid leaking heap-allocated strings, for example.
@@ -560,6 +509,8 @@ Read the documentation for [disposable types](functions.md#heap-allocated-values
 
 ### Type introspection
 
+*New in Koffi 2.0: `koffi.resolve()`*
+
 Koffi exposes three functions to explore type information:
 
 - `koffi.sizeof(type)` to get the size of a type
@@ -585,4 +536,6 @@ console.log(koffi.sizeof(koffi.types.long));
 
 ### Type aliases
 
+*New in Koffi 2.0*
+
 You can alias a type with `koffi.alias(name, type)`. Aliased types are completely equivalent.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "koffi",
-  "version": "2.1.0-beta.1",
+  "version": "2.1.0-beta.2",
   "description": "Fast and simple C FFI (foreign function interface) for Node.js",
   "keywords": [
     "foreign",