node-red-contrib-prib-functions 0.23.3 → 0.26.0

package/.github/copilot-instructions.md

@@ -0,0 +1,36 @@
+# Node-RED Contrib Functions
+
+## Project Overview
+This is a Node-RED contrib package providing custom nodes for data analysis, matrix operations, transformations, monitoring, and utilities. Nodes process messages in real-time, supporting statistical metrics, data transformations, and system monitoring.
+
+## Architecture
+- **Node Structure**: Each node resides in its own directory containing:
+  - `.js`: Backend logic using Node-RED's API (e.g., `dataAnalysis/dataAnalysis.js`)
+  - `.html`: Frontend configuration UI with typed inputs (e.g., `dataAnalysis/dataAnalysis.html`)
+  - `icons/`: Custom node icons
+- **Shared Libraries**: `lib/` holds common utilities like `common.js` (dynamic property evaluation), `objectExtensions.js` (array extensions), and domain-specific modules
+- **Data Flow**: Nodes typically input `msg.payload`, output results to multiple ports (primary results, details/stats, outliers)
+
+## Key Patterns
+- **Node Registration**: Follow `RED.nodes.registerType('NodeName', function(config) { this.on('input', (msg) => { ... }); })`
+- **Configuration Properties**: Use defaults like `dataProperty: {value:"msg.payload"}`, leverage typedInput for dynamic paths
+- **Error Handling**: Call `node.error(message)` and update status: `node.status({fill:"red",shape:"ring",text:"shortMsg"})`
+- **Logging**: Instantiate `const logger = new (require("node-red-contrib-logger"))("NodeName")`; use `logger.sendInfo()` for debugging
+- **Real-time Accumulation**: Store data in `node.dataPoint[key]` objects, recalculate metrics on each input (e.g., running averages, deltas)
+- **Dynamic Evaluation**: Use `evalInFunction(node, 'propertyName')` from `lib/common.js` to access msg/flow/global properties safely
+- **Outlier Detection**: For realtime nodes, check against mean/median ± N standard deviations, output to third port
+
+## Development Workflow
+- **Testing**: Run `npm test` (mocha on `test/dataAnalysisE*.js`); use `node-red-node-test-helper` for node testing
+- **Local Development**: `npm run startNodeRed` launches test Node-RED instance with flows in `test/data/`
+- **Dependencies**: `npm ci` installs; requires Node >=22.15.0, Node-RED >=4.0.9; external libs like `avsc` (AVRO), `xlsx` (Excel), `fast-xml-parser` (XML)
+
+## Examples
+- **Data Analysis**: Accumulates arrays/objects, computes stats (avg, stddev, autocorrelation); realtime mode detects outliers beyond 3σ
+- **Transform**: Converts formats (JSON↔CSV↔XLSX↔AVRO) using dedicated libraries; supports path operations and string manipulations
+- **Matrix**: Performs linear algebra (inverse, determinant, Gaussian elimination) on 2D arrays
+
+## Deployment
+- Automated npm publish on GitHub release via `.github/workflows/npmpublish.yml`
+- Set `NODE_AUTH_TOKEN` secret for publishing</content>
+<parameter name="filePath">c:\Users\peter\repros\node-red-contrib-prib-functions\.github\copilot-instructions.md

package/README.md

@@ -75,6 +75,17 @@ example:
 
 ------------------------------------------------------------
 
+## logistic regression
+
+Perform logistic regression by setting up model with fit then using the model in other nodes to do prediction probability or class
+------------------------------------------------------------
+
+## columnar store
+
+Store data into a columnar store with sql select access
+
+------------------------------------------------------------
+
 ## Matrix
 
 Define a matrix and perform various functions
@@ -352,6 +363,10 @@ Test/example flow in test/generalTest.json
 
 ## Version
 
+0.26.0 add columnar store node with sql query capabilty
+
+0.24.0 add in logistic regression
+
 0.23.3 Removes bug in test, more translation
 
 0.23.0 Removes bug in test, more translation

package/columnar/columnar.html

@@ -0,0 +1,258 @@
+<script type="text/javascript">
+    RED.nodes.registerType('columnar', {
+        category: 'storage',
+        color: '#8BC34A',
+        defaults: {
+            name: {value:"", required:false},
+            action: {value:"readFile", required:true},
+            filePath: {value:"", required:false},
+            sqlQuery: {value:"", required:false},
+            outputProperty: {value:"result", required:false},
+            parameterMappings: {value:[], required:false}
+        },
+        inputs: 1,
+        inputLabels: "in",
+        outputs:1,
+        outputLabels: function() { return [this.outputProperty || "result"]; },
+        paletteLabel: "Columnar Store",
+        icon: "columnar.svg",
+        label: function() {
+            const actionLabels = {
+                readFile: "Read",
+                writeFile: "Write",
+                appendFile: "Append",
+                queryFile: "Query",
+                sqlQuery: "SQL Query",
+                getSchema: "Get Schema",
+                getMetadata: "Get Metadata"
+            };
+            const actionLabel = actionLabels[this.action] || this.action;
+            
+            if (this.name) {
+                return this.name;
+            }
+            
+            if (this.action === 'sqlQuery' && this.sqlQuery) {
+                // Show truncated SQL query in label
+                const sqlPreview = this.sqlQuery.length > 30 ? 
+                    this.sqlQuery.substring(0, 27) + '...' : 
+                    this.sqlQuery;
+                return sqlPreview;
+            }
+            
+            return actionLabel + " Columnar Store";
+        },
+        labelStyle: function() {
+            return "node_label_italic";
+        },
+        oneditprepare: function() {
+            // Show/hide SQL query field based on action
+            function updateSqlField() {
+                const action = $('#node-input-action').val();
+                if (action === 'sqlQuery') {
+                    $('#sql-query-row').show();
+                } else {
+                    $('#sql-query-row').hide();
+                }
+            }
+            
+            $('#node-input-action').change(updateSqlField);
+            updateSqlField(); // Initial check
+            
+        },
+        oneditsave: function() {
+            // No parameter mappings needed; explicit :msg., :flow., :global. markers only
+        }
+    });
+</script>
+
+<script type="text/x-red" data-template-name="columnar">
+    <div class="form-row">
+        <label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
+        <input type="text" id="node-input-name" placeholder="Name">
+    </div>
+    <div class="form-row">
+        <label for="node-input-action"><i class="fa fa-cogs"></i> Action</label>
+        <select type="text" id="node-input-action">
+            <option value="readFile">Read File</option>
+            <option value="writeFile">Write File</option>
+            <option value="appendFile">Append File</option>
+            <option value="queryFile">Query File</option>
+          <option value="sqlQuery">SQL Query</option>
+            <option value="getSchema">Get Schema</option>
+            <option value="getMetadata">Get Metadata</option>
+        </select>
+    </div>
+    <div class="form-row">
+        <label for="node-input-filePath">File Path</label>
+        <input type="text" id="node-input-filePath" placeholder="/path/to/file.columnar">
+    </div>
+    <div class="form-row">
+        <label for="node-input-outputProperty"><i class="fa fa-sign-out"></i> Output Property</label>
+        <input type="text" id="node-input-outputProperty" placeholder="result">
+    </div>
+    <div class="form-row" id="sql-query-row" style="display:none;">
+        <label for="node-input-sqlQuery"><i class="fa fa-code"></i> SQL Query</label>
+        <textarea type="text" id="node-input-sqlQuery" placeholder="SELECT * FROM ? WHERE ..." rows="3" style="width:100%; resize:vertical;"></textarea>
+    </div>
+</script>
+
+<script type="text/x-red" data-help-name="columnar">
+    <p>Columnar Store node for reading and writing columnar data files. This implementation is based on <a href="https://github.com/apache/parquet-format/" target="_blank">Apache Parquet specification</a> concepts including columnar storage, metadata structures, and compression principles, but provides a simplified JavaScript implementation for basic use cases.</p>
+
+    <h3>Actions</h3>
+    <ul>
+        <li><b>Read File:</b> Reads all records from a Columnar Store file</li>
+        <li><b>Write File:</b> Writes records to a Columnar Store file (overwrites existing file)</li>
+        <li><b>Append File:</b> Appends records to an existing Columnar Store file (creates file if it doesn't exist)</li>
+        <li><b>Query File:</b> Reads and filters records from a Columnar Store file</li>
+        <li><b>SQL Query:</b> Execute SQL queries with aggregates, grouping, sorting, JOINs, and filtering</li>
+        <li><b>Get Schema:</b> Retrieves the schema information from a Columnar Store file</li>
+        <li><b>Get Metadata:</b> Retrieves metadata information from a Columnar Store file</li>
+    </ul>
+
+    <h3>Configuration</h3>
+    <ul>
+      <li><b>File Path:</b> Path to the Columnar file (.columnar extension). Can be overridden in msg.payload.filePath</li>
+      <li><b>Output Property:</b> Property name on msg object where results will be stored (default: "result")</li>
+      <li><b>SQL Query:</b> (SQL Query action only) Hardcoded SQL query to execute. Can be overridden by msg.payload.sql. Hardcoded queries are parsed once and cached for better performance.</li>
+      <li><b>Parameters:</b> <b>Only</b> <code>:msg.path</code>, <code>:flow.path</code>, or <code>:global.path</code> markers are allowed in SQL. Example: <code>WHERE a.name = :msg.payload.name AND a.age > :flow.minAge</code>. No other parameter markers are supported.</li>
+    </ul>
+
+    <h3>Input/Output Formats</h3>
+
+    <h4>Read File</h4>
+    <p><b>Input:</b> Optional filePath in msg.payload</p>
+    <p><b>Output:</b></p>
+    <pre>{
+  "result": {
+    "records": [...],
+    "count": 100,
+    "schema": {...},
+    "filePath": "/path/to/file.columnar"
+  }
+}</pre>
+
+    <h4>Write File</h4>
+    <p><b>Input:</b></p>
+    <pre>{
+  "records": [
+    {"name": "John", "age": 30, "city": "NYC"},
+    {"name": "Jane", "age": 25, "city": "LA"}
+  ],
+  "filePath": "/path/to/output.columnar",
+  "schema": {...} // optional, auto-inferred if not provided
+}</pre>
+    <p><b>Output:</b></p>
+    <pre>{
+  "result": {
+    "filePath": "/path/to/output.columnar",
+    "recordsWritten": 2,
+    "schema": {...}
+  }
+}</pre>
+
+    <h4>Query File</h4>
+    <p><b>Input:</b></p>
+    <pre>{
+  "filePath": "/path/to/file.columnar",
+  "limit": 100, // optional
+  // either provide a javascript filter function or a ridMap to perform
+  // a column‑based intersection.  The columnar layout stores each field
+  // as its own buffer along with a list of row‑ids (rids).  A ridMap is an
+  // object whose keys are column names and whose values are arrays or
+  // Sets of rids; only rows present in *every* set will be returned.
+  // Example using a filter function:
+  "filter": "function(record) { return record.age > 25; }" // optional
+  // Example using a ridMap (select rows 1 and 2 from the 'age' column):
+  // "ridMap": {"age": [1,2]}
+}</pre>
+    <p><b>Output:</b></p>
+    <pre>{
+  "result": {
+    "records": [...],
+    "count": 50,
+    "totalScanned": 100,
+    "schema": {...},
+    "filtered": true
+  }
+}</pre>
+
+    <h4>SQL Query</h4>
+    <p><b>Input:</b> SQL query can be configured in the node or provided in msg.payload.sql (msg.payload.sql takes precedence)</p>
+    <pre>{
+  "filePath": "/path/to/file.columnar",
+  "sql": "SELECT * FROM ? WHERE a.name = :msg.payload.name AND a.age > :flow.minAge"
+}</pre>
+    <p><b>Parameters:</b> Use <code>:msg.path</code>, <code>:flow.path</code>, or <code>:global.path</code> in your SQL to reference values from <code>msg</code>, <code>flow</code>, or <code>global</code> context. <b>Other parameter markers are not supported and will cause an error.</b></p>
+    <p><b>Supported Features:</b></p>
+    <ul>
+      <li><b>SELECT:</b> Column names, <code>*</code>, or aggregates (COUNT, SUM, AVG, MIN, MAX, COUNT DISTINCT)</li>
+      <li><b>FROM/JOIN:</b> Main table (<code>?</code>) with optional INNER/LEFT/RIGHT/FULL OUTER JOIN on other .columnar files</li>
+      <li><b>WHERE:</b> Filter records using JavaScript expressions</li>
+      <li><b>GROUP BY:</b> Group results by one or more columns with aggregate functions</li>
+      <li><b>HAVING:</b> Filter grouped results</li>
+      <li><b>ORDER BY:</b> Sort by column(s) with ASC/DESC</li>
+      <li><b>LIMIT:</b> Restrict result count</li>
+      <li><b>Parameters:</b> Use <code>:msg.path</code>, <code>:flow.path</code>, or <code>:global.path</code> in queries. No other parameter markers are allowed.</li>
+    </ul>
+    <p><b>JOIN Syntax:</b></p>
+    <pre>[INNER|LEFT|RIGHT|FULL OUTER] JOIN 'filepath.columnar' ON table1.col = table2.col</pre>
+    <p><b>Output:</b></p>
+    <pre>{
+  "result": [ /* array of result rows */ ]
+}</pre>
+
+    <h4>Get Schema</h4>
+    <p><b>Input:</b> Optional filePath in msg.payload</p>
+    <p><b>Output:</b></p>
+    <pre>{
+  "result": {
+    "schema": {...},
+    "fields": ["name", "age", "city"],
+    "filePath": "/path/to/file.columnar"
+  }
+}</pre>
+
+    <h4>Get Metadata</h4>
+    <p><b>Input:</b> Optional filePath in msg.payload</p>
+    <p><b>Output:</b></p>
+    <pre>{
+  "result": {
+    "metadata": {...},
+    "filePath": "/path/to/file.columnar",
+    "numRows": 1000,
+    "numRowGroups": 5
+  }
+}</pre>
+
+    <h3>Data Types</h3>
+    <p>Columnar supports various data types. The node automatically infers types when writing:</p>
+    <ul>
+        <li>String → UTF8</li>
+        <li>Number → DOUBLE</li>
+        <li>Boolean → BOOLEAN</li>
+        <li>Date → TIMESTAMP</li>
+    </ul>
+
+    <h3>Implementation Notes</h3>
+    <ul>
+        <li>Based on <a href="https://github.com/apache/parquet-format/" target="_blank">Apache Parquet specification</a> concepts</li>
+        <li>Implements columnar storage for efficient data organization</li>
+        <li>Uses metadata structures inspired by Parquet format specification</li>
+        <li>Applies compression principles from the Parquet specification</li>
+        <li>Custom JavaScript implementation, not a full Apache Parquet compatible format</li>
+        <li>Best suited for basic tabular data storage and retrieval</li>
+      <li>Supports filtering via row‑id (rid) maps; query operations on
+        multiple columns efficiently intersect the provided rid sets</li>
+    </ul>
+
+    <h3>Error Handling</h3>
+    <p>Errors are reported in msg.error. Common issues:</p>
+    <ul>
+        <li>File not found</li>
+        <li>Invalid Columnar format</li>
+        <li>Schema mismatches</li>
+        <li>Permission issues</li>
+    </ul>
+</script>