@rosepetal/node-red-contrib-async-function 1.0.2 → 1.0.3

package/README.md

@@ -31,7 +31,7 @@ Drop an **async function** node into your flow. Write your code just like you wo
 
 **Skip It For:**
 - Simple math or quick transformations (the regular function node is faster).
-- When you need `context`, `flow`, or `global` storage (coming in v2.0).
+- Flows that require live context reads/writes during execution (context is snapshot-based).
 
 ## Node Options
 
@@ -62,7 +62,7 @@ return msg;
 ```
 
 ### Buffer Handling
-- **Buffers** – Any `Buffer` in `msg` is transferred through shared memory (`/dev/shm` on Linux, otherwise `os.tmpdir()`), with base64 fallback if needed.
+- **Buffers** – Worker Threads use zero-copy transfer when possible. Child Process mode and non-transferable buffers fall back to shared memory (`/dev/shm` on Linux, otherwise `os.tmpdir()`), with base64 fallback if needed.
 
 ## Typical Flow
 
@@ -86,9 +86,11 @@ return msg;
 - `console` – Logging functions
 - `setTimeout`, `setInterval` – Timers
 
-**Not Available (Yet):**
-- `context`, `flow`, `global` – Coming in v2.0
-- `node` – Node instance methods
+**Notes:**
+- `context`, `flow`, `global` are snapshot-based: reads are from the snapshot, writes are applied after the function completes
+- Snapshot includes only literal keys found in `flow.get("key")` / `global.get("key")` / `context.get("key")`
+- Context store selection is not supported (default store only)
+- `node.warn/error/log` are collected and forwarded to the main thread
 - Non-serializable objects (functions, symbols, etc.)
 
 ## Code Examples
@@ -170,7 +172,7 @@ The node shows you what's happening in real time:
 - Best for operations taking more than 10ms to run.
 - Each node maintains a fixed pool of workers—no startup delay or dynamic scaling overhead.
 - Workers are dedicated per-node, ensuring predictable performance.
-- **Binary Fast Path**: Buffers use shared memory transfer (base64 fallback), keeping messages responsive even with large payloads.
+- **Binary Fast Path**: Worker threads use zero-copy transfer when possible; shared memory is the fallback.
 - Event loop never blocks, even when processing multi-MB binary data (images, files, etc.).
 
 ## Error Handling

package/nodes/async-function.html

@@ -7,7 +7,7 @@
         defaults: {
             name: { value: '' },
             func: {
-                value: '// Write your async code here\n// The code runs in a worker thread or child process\n// Available: msg, return, async/await, require()\n// Not available: context, flow, global, node\n\nreturn msg;'
+                value: '// Write your async code here\n// The code runs in a worker thread or child process\n// Available: msg, return, async/await, require(), node.warn/error/log,\n//           flow/global/context (snapshot-based)\n\nreturn msg;'
             },
             outputs: { value: 1, validate: RED.validators.number() },
             timeout: { value: 30000, validate: RED.validators.number() },
@@ -507,8 +507,8 @@
                 <b>Tips:</b>
                 <ul style="margin-top:5px; margin-bottom:0;">
                 <li>Code runs in a worker thread or child process to prevent blocking</li>
-                <li>Available: <code>msg</code>, <code>return</code>, <code>async/await</code>, <code>require()</code></li>
-                <li>Not available: <code>context</code>, <code>flow</code>, <code>global</code>, <code>node</code></li>
+                <li>Available: <code>msg</code>, <code>return</code>, <code>async/await</code>, <code>require()</code>, <code>node</code>, <code>flow</code>, <code>global</code>, <code>context</code></li>
+                <li>Context access is snapshot-based (writes apply after execution)</li>
                 <li>Use "Child Process" runtime for native modules that require main thread (e.g., <code>gl</code>)</li>
                 <li>Return <code>msg</code> for single output, or <code>[msg1, msg2]</code> for multiple outputs</li>
                 <li>Return <code>null</code> to stop the flow</li>
@@ -546,14 +546,14 @@
         <li><code>return</code> - Return modified message or array of messages</li>
         <li><code>async/await</code> - Use async operations</li>
         <li><code>require()</code> - Import Node.js modules</li>
+        <li><code>node</code> - Node helpers (<code>warn</code>, <code>error</code>, <code>log</code>)</li>
+        <li><code>flow</code>, <code>global</code>, <code>context</code> - Snapshot-based context access (writes apply after execution)</li>
         <li><code>console</code> - Logging functions</li>
         <li><code>setTimeout</code>, <code>setInterval</code> - Timers</li>
     </ul>
 
     <h4>Not Available</h4>
     <ul>
-        <li><code>context</code>, <code>flow</code>, <code>global</code> - Context storage (v2.0 feature)</li>
-        <li><code>node</code> - Node instance methods</li>
         <li>Non-serializable objects in <code>msg</code></li>
     </ul>
 
@@ -612,7 +612,7 @@ return msg;</pre>
         <dt>Queue Size</dt>
         <dd>Maximum number of messages queued when all workers are busy (10-1000). Default: 100.</dd>
         <dt>Buffers</dt>
-        <dd>All Buffers are transferred via shared memory when possible (falls back to base64 if needed).</dd>
+        <dd>Worker threads use zero-copy transfer when possible; shared memory is used as a fallback.</dd>
     </dl>
 
     <h3>Status Display</h3>
@@ -639,9 +639,11 @@ return msg;</pre>
         <li>Best for CPU-intensive operations (>10ms execution time)</li>
         <li>For simple operations, use the standard function node</li>
         <li>Workers are pooled and reused for efficiency</li>
-        <li><b>Buffer Transfer:</b> Buffers are streamed through shared memory (base64 fallback)</li>
+        <li><b>Buffer Transfer:</b> Worker threads use zero-copy transfer when possible; shared memory is the fallback</li>
         <li>Event loop never blocks, even when processing multi-MB binary data (images, files, etc.)</li>
         <li>Message transfer is optimized by copying only referenced <code>msg.*</code> keys when possible</li>
+        <li>Context snapshots include only literal keys found in <code>flow.get("key")</code> etc.</li>
+        <li>Context snapshots use the default context store only</li>
         <li>Timing is recorded on <code>msg.performance</code> under the node label</li>
     </ul>
 

package/nodes/async-function.js

@@ -44,7 +44,7 @@ function extractMsgKeysFromCode(code) {
     return keys;
 }
 
-function buildWorkerInputMsg(originalMsg, code) {
+function buildWorkerInputMsgCore(originalMsg, code) {
     if (!originalMsg || typeof originalMsg !== 'object') {
         return originalMsg;
     }
@@ -84,6 +84,304 @@ function hrtimeDiffToMs(start) {
     return Number(diff) / 1e6;
 }
 
+function normalizeTransferMode(mode, executionMode) {
+    const normalized = typeof mode === 'string' ? mode.trim().toLowerCase() : '';
+    if (normalized === 'transfer' || normalized === 'shared' || normalized === 'copy') {
+        return normalized;
+    }
+    return executionMode === 'worker_threads' ? 'transfer' : 'shared';
+}
+
+const MSG_WRAPPER_KEY = '__rosepetal_msg';
+const CONTEXT_WRAPPER_KEY = '__rosepetal_context';
+
+function extractContextKeysFromCode(code) {
+    const flowKeys = new Set();
+    const globalKeys = new Set();
+    const contextKeys = new Set();
+
+    if (typeof code !== 'string' || !code.trim()) {
+        return { flow: flowKeys, global: globalKeys, context: contextKeys, usesContext: false };
+    }
+
+    const flowRegex = /flow\.(?:get|set)\(\s*['"]([^'"]+)['"]/g;
+    const flowBracket = /flow\[['"]([^'"]+)['"]\]/g;
+    const globalRegex = /global\.(?:get|set)\(\s*['"]([^'"]+)['"]/g;
+    const globalBracket = /global\[['"]([^'"]+)['"]\]/g;
+    const contextRegex = /context\.(?:get|set)\(\s*['"]([^'"]+)['"]/g;
+    const contextBracket = /context\[['"]([^'"]+)['"]\]/g;
+
+    let match = flowRegex.exec(code);
+    while (match) {
+        flowKeys.add(match[1]);
+        match = flowRegex.exec(code);
+    }
+
+    match = flowBracket.exec(code);
+    while (match) {
+        flowKeys.add(match[1]);
+        match = flowBracket.exec(code);
+    }
+
+    match = globalRegex.exec(code);
+    while (match) {
+        globalKeys.add(match[1]);
+        match = globalRegex.exec(code);
+    }
+
+    match = globalBracket.exec(code);
+    while (match) {
+        globalKeys.add(match[1]);
+        match = globalBracket.exec(code);
+    }
+
+    match = contextRegex.exec(code);
+    while (match) {
+        contextKeys.add(match[1]);
+        match = contextRegex.exec(code);
+    }
+
+    match = contextBracket.exec(code);
+    while (match) {
+        contextKeys.add(match[1]);
+        match = contextBracket.exec(code);
+    }
+
+    const usesContext = /\b(flow|global|context)\s*\./.test(code);
+    return { flow: flowKeys, global: globalKeys, context: contextKeys, usesContext };
+}
+
+async function buildContextSnapshot(node, code) {
+    const empty = { flow: {}, global: {}, context: {} };
+    if (!node || typeof node.context !== 'function') {
+        return { snapshot: empty, usesContext: false };
+    }
+
+    const keys = extractContextKeysFromCode(code);
+    if (!keys.usesContext) {
+        return { snapshot: empty, usesContext: false };
+    }
+
+    const ctx = node.context();
+    if (!ctx) {
+        return { snapshot: empty, usesContext: true };
+    }
+
+    const snapshot = { flow: {}, global: {}, context: {} };
+    const coerceBufferLike = (value) => {
+        if (!value || typeof value !== 'object') {
+            return value;
+        }
+        if (Buffer.isBuffer(value)) {
+            return value;
+        }
+        if (value.type === 'Buffer' && Array.isArray(value.data)) {
+            return Buffer.from(value.data);
+        }
+        if (ArrayBuffer.isView(value) && !(value instanceof DataView)) {
+            return Buffer.from(value.buffer, value.byteOffset, value.byteLength);
+        }
+        return value;
+    };
+
+    const getValueAsync = (scope, key) => {
+        if (!scope || typeof scope.get !== 'function') {
+            return Promise.resolve(undefined);
+        }
+        if (scope.get.length >= 2) {
+            return new Promise((resolve) => {
+                try {
+                    scope.get(key, (err, value) => {
+                        resolve(err ? undefined : value);
+                    });
+                } catch (_err) {
+                    resolve(undefined);
+                }
+            });
+        }
+        try {
+            const value = scope.get(key);
+            if (value && typeof value.then === 'function') {
+                return value.then((resolved) => resolved).catch(() => undefined);
+            }
+            return Promise.resolve(value);
+        } catch (_err) {
+            return Promise.resolve(undefined);
+        }
+    };
+
+    const readKeys = async (scope, keySet, target) => {
+        if (!scope || typeof scope.get !== 'function') {
+            return;
+        }
+        const entries = Array.from(keySet);
+        await Promise.all(entries.map(async (key) => {
+            try {
+                const value = await getValueAsync(scope, key);
+                target[key] = coerceBufferLike(value);
+            } catch (_err) {
+                // Ignore context read errors to avoid blocking execution
+            }
+        }));
+    };
+
+    await Promise.all([
+        readKeys(ctx.flow, keys.flow, snapshot.flow),
+        readKeys(ctx.global, keys.global, snapshot.global),
+        readKeys(ctx, keys.context, snapshot.context)
+    ]);
+
+    return { snapshot, usesContext: true };
+}
+
+function buildWorkerInputMsg(originalMsg, code, contextSnapshot, useContextWrapper) {
+    if (!useContextWrapper) {
+        return buildWorkerInputMsgCore(originalMsg, code);
+    }
+
+    const msgPayload = buildWorkerInputMsgCore(originalMsg, code);
+    return {
+        [MSG_WRAPPER_KEY]: msgPayload,
+        [CONTEXT_WRAPPER_KEY]: contextSnapshot || { flow: {}, global: {}, context: {} }
+    };
+}
+
+function rehydrateContextValue(value) {
+    if (Buffer.isBuffer(value)) {
+        return value;
+    }
+
+    if (ArrayBuffer.isView(value) && !(value instanceof DataView)) {
+        return Buffer.from(value.buffer, value.byteOffset, value.byteLength);
+    }
+
+    if (Array.isArray(value)) {
+        return value.map((item) => rehydrateContextValue(item));
+    }
+
+    if (value && typeof value === 'object') {
+        if (value.type === 'Buffer' && Array.isArray(value.data)) {
+            return Buffer.from(value.data);
+        }
+
+        const obj = Array.isArray(value) ? [] : {};
+        Object.keys(value).forEach((key) => {
+            obj[key] = rehydrateContextValue(value[key]);
+        });
+        return obj;
+    }
+
+    return value;
+}
+
+async function applyContextUpdates(node, updates, msg) {
+    if (!node || !updates || typeof updates !== 'object' || typeof node.context !== 'function') {
+        return;
+    }
+
+    const context = node.context();
+    if (!context) {
+        return;
+    }
+
+    const flowUpdates = updates.flow && typeof updates.flow === 'object' ? updates.flow : {};
+    const globalUpdates = updates.global && typeof updates.global === 'object' ? updates.global : {};
+    const contextUpdates = updates.context && typeof updates.context === 'object' ? updates.context : {};
+
+    const applyUpdates = async (scope, scopeLabel, scopeUpdates) => {
+        if (!scope || typeof scope.set !== 'function') {
+            return;
+        }
+        const setValueAsync = (key, value) => {
+            if (scope.set.length >= 3) {
+                return new Promise((resolve) => {
+                    try {
+                        scope.set(key, value, (err) => {
+                            if (err && typeof node.warn === 'function') {
+                                node.warn(`Failed to set ${scopeLabel} context "${key}": ${err.message || err}`, msg);
+                            }
+                            resolve();
+                        });
+                    } catch (err) {
+                        if (typeof node.warn === 'function') {
+                            node.warn(`Failed to set ${scopeLabel} context "${key}": ${err.message || err}`, msg);
+                        }
+                        resolve();
+                    }
+                });
+            }
+            try {
+                const result = scope.set(key, value);
+                if (result && typeof result.then === 'function') {
+                    return result.catch((err) => {
+                        if (typeof node.warn === 'function') {
+                            node.warn(`Failed to set ${scopeLabel} context "${key}": ${err.message || err}`, msg);
+                        }
+                    });
+                }
+            } catch (err) {
+                if (typeof node.warn === 'function') {
+                    node.warn(`Failed to set ${scopeLabel} context "${key}": ${err.message || err}`, msg);
+                }
+            }
+            return Promise.resolve();
+        };
+        const entries = Object.keys(scopeUpdates);
+        await Promise.all(entries.map(async (key) => {
+            const value = rehydrateContextValue(scopeUpdates[key]);
+            await setValueAsync(key, value);
+        }));
+    };
+
+    await Promise.all([
+        applyUpdates(context.flow, 'flow', flowUpdates),
+        applyUpdates(context.global, 'global', globalUpdates),
+        applyUpdates(context, 'context', contextUpdates)
+    ]);
+}
+
+function applyWorkerLogs(node, logs, msg) {
+    if (!node || !Array.isArray(logs)) {
+        return;
+    }
+
+    logs.forEach((entry) => {
+        if (!entry) {
+            return;
+        }
+
+        const level = typeof entry === 'object' && entry.level ? String(entry.level) : 'warn';
+        const message = typeof entry === 'object' && entry.message !== undefined
+            ? entry.message
+            : entry;
+        let text = '';
+        if (typeof message === 'string') {
+            text = message;
+        } else {
+            try {
+                text = JSON.stringify(message);
+            } catch (_err) {
+                text = String(message);
+            }
+        }
+
+        if (level === 'error' && typeof node.error === 'function') {
+            node.error(text, msg);
+            return;
+        }
+
+        if (level === 'log' && typeof node.log === 'function') {
+            node.log(text);
+            return;
+        }
+
+        if (typeof node.warn === 'function') {
+            node.warn(text, msg);
+        }
+    });
+}
+
+
 function applyPerformanceMetrics(node, originalMsg, targetMsg, performance) {
     if (!performance || typeof performance !== 'object') {
         return;
@@ -241,32 +539,9 @@ module.exports = function(RED) {
         }
 
         const nodeRedUserDir = resolveNodeRedUserDir(RED);
+        const transferMode = normalizeTransferMode(config.transferMode, node.executionMode);
 
-        // Verify modules are resolvable WITHOUT loading them in the main thread.
-        // Loading native modules (like 'gl') in main thread prevents them from
-        // working in worker threads due to native module registration conflicts.
-        if (node.libs.length > 0) {
-            const { createRequire } = require('module');
-            const nodeRedRequire = createRequire(path.join(nodeRedUserDir, 'package.json'));
-
-            for (const lib of node.libs) {
-                if (!lib || !lib.module || !lib.var) {
-                    continue;
-                }
-                try {
-                    // Only resolve the path - don't actually load the module
-                    nodeRedRequire.resolve(lib.module);
-                } catch (err) {
-                    node.error(`Module "${lib.module}" not found. Install it with: cd ${nodeRedUserDir} && npm install ${lib.module}`);
-                    node.status({
-                        fill: 'red',
-                        shape: 'dot',
-                        text: `Module not found: ${lib.module}`
-                    });
-                    return;
-                }
-            }
-        }
+        // Module resolution happens inside workers to avoid blocking the main thread.
 
         const startPool = () => {
             try {
@@ -276,6 +551,7 @@ module.exports = function(RED) {
                     maxQueueSize: config.maxQueueSize || 100,
                     taskTimeout: node.timeout,
                     shmThreshold: 0,
+                    transferMode,
                     libs: node.libs,
                     nodeRedUserDir
                 });
@@ -324,22 +600,29 @@ module.exports = function(RED) {
             }
 
             const timing = { start: process.hrtime.bigint() };
-            const workerMsg = buildWorkerInputMsg(msg, node.func);
+            const contextInfo = await buildContextSnapshot(node, node.func);
+            const workerMsg = buildWorkerInputMsg(msg, node.func, contextInfo.snapshot, contextInfo.usesContext);
 
             try {
                 const payload = await node.pool.executeTask(node.func, workerMsg, node.timeout);
 
                 let resultData;
                 let performanceData = null;
+                let contextUpdates = null;
+                let logs = null;
 
                 if (payload && typeof payload === 'object' && Object.prototype.hasOwnProperty.call(payload, 'result')) {
                     resultData = payload.result;
                     performanceData = payload.performance || null;
+                    contextUpdates = payload.contextUpdates || null;
+                    logs = payload.logs || null;
                 } else {
                     resultData = payload;
                 }
 
                 if (resultData === null || resultData === undefined) {
+                    await applyContextUpdates(node, contextUpdates, msg);
+                    applyWorkerLogs(node, logs, msg);
                     done();
                     return;
                 }
@@ -348,6 +631,9 @@ module.exports = function(RED) {
                 const mergedPerformance = Object.assign({}, performanceData || {});
                 mergedPerformance.totalMs = totalMs;
 
+                await applyContextUpdates(node, contextUpdates, msg);
+                applyWorkerLogs(node, logs, msg);
+
                 const output = mergeResult(msg, resultData);
                 applyPerformanceToResult(node, msg, output, mergedPerformance);
 
@@ -362,6 +648,13 @@ module.exports = function(RED) {
                     text: `Error: ${err.message.substring(0, 20)}`
                 });
 
+                if (err && err.contextUpdates) {
+                    await applyContextUpdates(node, err.contextUpdates, msg);
+                }
+                if (err && err.logs) {
+                    applyWorkerLogs(node, err.logs, msg);
+                }
+
                 // Log error
                 node.error(`Async function error: ${err.message}`, msg);
 

package/nodes/lib/child-process-pool.js

@@ -17,6 +17,7 @@ const DEFAULT_CONFIG = {
     taskTimeout: 30000,         // Default task timeout: 30s
     maxQueueSize: 100,          // Max queued messages
     shmThreshold: 0,            // Always use shared memory for Buffers
+    transferMode: 'shared',     // shared | copy (transfer not supported for child processes)
     libs: [],                   // External modules to load in workers
     nodeRedUserDir: null,       // Node-RED user directory for module resolution
     workerScript: path.join(__dirname, 'child-process-script.js')
@@ -166,7 +167,8 @@ class ChildProcessPool {
                     type: 'init',
                     libs: this.config.libs || [],
                     nodeRedUserDir: this.config.nodeRedUserDir,
-                    shmThreshold: this.config.shmThreshold
+                    shmThreshold: this.config.shmThreshold,
+                    transferMode: this.config.transferMode
                 });
             } catch (err) {
                 clearTimeout(readyTimeout);
@@ -298,7 +300,9 @@ class ChildProcessPool {
         workerState.state = WorkerState.BUSY;
         workerState.taskId = taskId;
 
-        this.serializer.sanitizeMessage(msg, null, taskId).then(sanitizedMsg => {
+        this.serializer.sanitizeMessage(msg, null, taskId, {
+            transferMode: this.config.transferMode
+        }).then(sanitizedMsg => {
             this.timeoutManager.startTimeout(taskId, timeout, () => {
                 this.handleTimeout(workerState, taskId);
             });
@@ -329,7 +333,7 @@ class ChildProcessPool {
      * @param {object} message - Message from worker
      */
     handleWorkerMessage(workerState, message) {
-        const { type, taskId, result, error, performance } = message || {};
+        const { type, taskId, result, error, performance, contextUpdates, logs } = message || {};
 
         if (type === 'result') {
             this.timeoutManager.cancelTimeout(taskId);
@@ -339,8 +343,18 @@ class ChildProcessPool {
                 this.callbacks.delete(taskId);
                 this.recycleWorker(workerState);
 
-                this.serializer.restoreBuffers(result).then(restoredResult => {
-                    callback(null, { result: restoredResult, performance: performance || null });
+                const restoreResult = this.serializer.restoreBuffers(result);
+                const restoreContext = contextUpdates
+                    ? this.serializer.restoreBuffers(contextUpdates)
+                    : Promise.resolve(contextUpdates);
+
+                Promise.all([restoreResult, restoreContext]).then(([restoredResult, restoredContext]) => {
+                    callback(null, {
+                        result: restoredResult,
+                        performance: performance || null,
+                        contextUpdates: restoredContext || null,
+                        logs: Array.isArray(logs) ? logs : null
+                    });
                 }).catch(restoreErr => {
                     callback(restoreErr instanceof Error ? restoreErr : new Error(String(restoreErr)), null);
                 });
@@ -362,7 +376,20 @@ class ChildProcessPool {
                 if (error && error.name) {
                     err.name = error.name;
                 }
-                callback(err, null);
+
+                if (contextUpdates) {
+                    this.serializer.restoreBuffers(contextUpdates).then(restoredContext => {
+                        err.contextUpdates = restoredContext;
+                        err.logs = Array.isArray(logs) ? logs : null;
+                        callback(err, null);
+                    }).catch(() => {
+                        err.logs = Array.isArray(logs) ? logs : null;
+                        callback(err, null);
+                    });
+                } else {
+                    err.logs = Array.isArray(logs) ? logs : null;
+                    callback(err, null);
+                }
             }
 
             this.recycleWorker(workerState);