@winkjs/composer 0.0.2 → 0.4.1

package/LICENSE

@@ -1,6 +1,6 @@
                            The MIT License
 
-Copyright (c) 2024-25  GRAYPE Systems Private Limited
+Copyright (c) 2024-26  GRAYPE Systems Private Limited
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

package/README.md

@@ -1,132 +1,158 @@
-# wink-composer ♫
+# winkComposer
 
-### [![Stability](https://img.shields.io/badge/stability-1--experimental-orange.svg)](https://nodejs.org/api/documentation.html#documentation_stability_index)
+[![npm version](https://img.shields.io/npm/v/@winkjs/composer.svg)](https://www.npmjs.com/package/@winkjs/composer)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-## ✨ Real-time analytics that sparks joy
+## Composable Streaming Intelligence
+[<img align="right" src="https://decisively.github.io/wink-logos/logo-title.png" width="100px" >](https://winkjs.org/)
 
-[<img align="right" src="https://decisively.github.io/wink-logos/logo-title.png" width="100px" >](http://winkjs.org/)
+winkComposer is a JavaScript framework for turning live data streams into insights and decisions — while the data is still moving. There is no batch job to wait for. On the plant floor, a delayed insight is a lost opportunity. It is open source, purpose-built for SMBs and MSMEs.
 
-WinkComposer is an upcoming open-source framework that makes building streaming intelligence playfully simple. Connect intuitive nodes, experiment freely, perfect it in an iteration or two. From edge-ready AI algorithms to built-in resilience, everything you need for production comes standard. No complexity, no black boxes - just the joy of creating exactly what you need.
 
-## Why Composability Matters
+## See It in Action
 
-**🎯 Create What You Need** — With composability, you're the inventor: Correlate vibration with weather? Done. Detect micro-trends before they become macro? Built. 
+The fastest way to understand winkComposer is to watch it work. The demos below run in your browser — real nodes, live data, no install, nothing to sign up for. Each one uses the same core that runs from edge to cloud.
 
-**💫 Countless Possibilities** — Dozens of nodes → Endless pipelines. Like few music notes creating captivating symphonies.
+| [![Bearing Health](https://composer.winkjs.org/readme-bearing-health.png)](https://composer.winkjs.org/docs/use-cases/bearing-health) | [![Server Health](https://composer.winkjs.org/readme-server-health.png)](https://composer.winkjs.org/docs/use-cases/server-health) | [![Process Quality](https://composer.winkjs.org/readme-process-quality.png)](https://composer.winkjs.org/docs/use-cases/process-quality) |
+|:---:|:---:|:---:|
+| [**Detecting Bearing Failure**](https://composer.winkjs.org/docs/use-cases/bearing-health) | [**Detecting Server Latency Degradation**](https://composer.winkjs.org/docs/use-cases/server-health) | [**Catching Process Drift**](https://composer.winkjs.org/docs/use-cases/process-quality) |
+| Predictive Maintenance | AIOps | Process Control |
 
-**🌱 Evolve Naturally** — Start: `sensor → threshold → alert`. Grow: Add intelligence: `sensor → normalize → smooth → detect → confirm → alert`. Unconstrained by vendor feature lists.
+Other end-to-end examples include [diagnosing Wi-Fi access-point health](https://composer.winkjs.org/docs/use-cases/wifi-ap-health), [classifying driving conditions](https://composer.winkjs.org/docs/use-cases/driving-modes) from OBD-II telematics, and [tracking wash-cycle quality](https://composer.winkjs.org/docs/use-cases/wash-cycle-quality).
 
-**🔍 Total Transparency** — See which node triggered, what threshold crossed, why it alerted. Every decision is explainable.
+Prefer to start with a failure mode instead of an industry? [Recipes](https://composer.winkjs.org/docs/playground/recipes) are short, runnable patterns for recurring problems. They cover gradual drift, directional trends, sudden shifts, hidden parasitic drain, PID-loop hunting, frozen sensors, and subtle process shifts — plus adaptive compression for storage.
 
-## Build Exactly What You Need
 
-Build precise analytics for your unique patterns:
+## How It Works
 
-```javascript
-// Detect bearing failure in rotating equipment
-vibration → isolateSignal → extractPeaks → measureIntensity →
-      trackChanges → triggerAlert
+A flow is a [pipeline](docs/handbook/understanding-composer.md) of small, single-purpose [nodes](docs/handbook/nodes/index.md). Each node reads fields from the message, computes its result, and adds the result onto the same message for the nodes downstream. The message that leaves the pipeline carries its whole history. A message can also stop early — a failed check drops it, and nothing downstream fires.
+
+The intelligence emerges from composition. Together the nodes cover the whole path from raw signal to decision: clean signals, extract features, detect change, fuse evidence, and act. The repertoire runs from a simple threshold, through a [Kalman filter, to a two-layer spiking neural network](docs/handbook/nodes/intelligence.md). The filter estimates what sensors cannot directly measure. The network weighs many weak signals into one confident verdict.
+
+You define such a pipeline in code — a [linear flow](docs/handbook/flow-language.md) that reads top to bottom, not a drag-and-drop graph. Explicit control signals handle orchestration, so a flow stays readable as it grows. And when many machines share one flow, each gets its own [isolated state](docs/handbook/understanding-composer.md) — one flow, thousands of assets.
+
+Results leave through emitters and storage: alerts to an MQTT broker like Mosquitto or the terminal, insights to QuestDB. [Grafana dashboards](docs/handbook/visualization.md) read from there. The stack is open source end to end.
 
-// Monitor chemical process quality
-temperature → removeNoise → smoothFast → smoothSlow → findDivergence →
-      detectChange → waitForPersistence
+The Quick Start below builds one such pipeline end to end.
 
-// Find correlated sensor behaviors
-[sensor1, sensor2] → trackCorrelation → compareToNormal → raiseFlag
 
-// Calculate real-time OEE
-machine → trackState → sumTime → countOutput →
-      checkQuality → calculateScore
+## Quick Start
+
+```bash
+npm install @winkjs/composer
 ```
 
-Each node is simple:
-- `removeNoise`: Median of last few values (removes sensor spikes).
-- `smoothFast`: Average of last 10 values (recent trend).
-- `smoothSlow`: Average of last 100 values (baseline trend).
-- `findDivergence`: Fast trend - slow trend (unusual behavior?).
-- `trackCorrelation`: Smart correlation (learns relationships between sensors)
+Requires Node.js 22 or newer.
+
+A complete flow — replay a CSV feed, clean each reading, flag a hot pump motor, confirm it is not a blip, and print an alert:
 
-_Node names and details are illustrative - actual implementations may vary._
+```javascript
+import { flow, csv, terminal } from '@winkjs/composer';
 
-Every arrow is a conscious choice. Every node has a purpose. No magic, just simple intuitive math.
+const handle = await flow( 'hello-flow' )
 
+    // Replay the CSV like a live feed: one reading every 200 ms.
+    .source( csv, { path: 'data/pump-temps.csv', delayMs: 200 } )
 
-## Your Analytics Journey
+    // Alerts print to the terminal here. A production flow points this
+    // at an MQTT broker instead, with QuestDB for storage.
+    .emitter( terminal, { verbose: true, prefix: '[pump]' } )
 
-**Start Simple**  
-`sensor → threshold → alert`
+    // One isolated pipeline per pump, and alerts name the pump in
+    // their topic. This feed has a single pump; a fleet needs no
+    // code change.
+    .assetId( 'id' )
+
+    // 1. clean — reject readings outside a sane range
+    .sanitize( 'clean', 'motor_t',
+        { failureReason: 'reject_reason' },
+        { ranges: { min: 0, max: 120 } } )
+
+    // 2. detect — flag when the motor runs hot
+    .threshold( 'tooHot', 'motor_t',
+        { active: 'is_hot' },
+        { mode: 'above', threshold: 80, hysteresis: 3 } )
+
+    // 3. confirm — hot across several readings, not one spike
+    .persistenceCheck( 'confirmHot',
+        ( msg ) => msg.is_hot,
+        { persistenceConfirmed: 'hot_confirmed' },
+        { minVotes: 3, outOfTotal: 5 } )
+
+    // 4. broadcast — print an alert once confirmed
+    .emitIf( 'alert',
+        ( msg ) => msg.hot_confirmed,
+        { target: 'terminal', insightType: 'overheat' } )
+
+    .run();
+```
 
-**Evolve Naturally**  
-`sensor → validate → normalize → threshold → debounce → alert`
+The same flow ships as a runnable project in [`examples/hello-flow`](examples/hello-flow), data included. Run it with `npm install && npm start`.
 
-**Scale Intelligently**  
-`sensor → [parallel: statistics | patterns | anomalies] → correlate → decide → act`
+Prefer zero install? [Build this flow interactively](https://composer.winkjs.org/docs/playground/hello-flow) in the browser playground.
 
-Your analytics grow with your understanding, and are never limited by any tool's feature list.
+The example above replays a CSV file. A live deployment reads from an MQTT broker instead — or runs [headless](docs/handbook/headless-flow.md), where your own code feeds the flow. Headless takes any source you already run: an OPC-UA client, a Kafka consumer.
 
-## Designed for the Real World
 
-### Industrial IoT
-From simple threshold monitoring to complex predictive maintenance - compose what fits your equipment, your patterns, your physics. **Move intelligence to the edge** with analytics that run on industrial Raspberry Pis. Works alongside your existing systems to handle your unique requirements.
+## Performance
 
-### Financial Services  
-Build fraud detection you can explain to auditors. Every decision traceable, every threshold justified.
+The numbers below come from a pure compute benchmark — every step a live message takes through an 8-node flow, from arrival to final output. **Storage and MQTT I/O are excluded** — these are compute ceilings, not full-deployment numbers.
 
-### Smart Infrastructure
-Energy optimization with transparent logic. See why the system made each decision.
+| Configuration        | Throughput              |
+|----------------------|-------------------------|
+| Raspberry Pi 5       | ~100K messages/second   |
+| Modern server        | ~1.2M messages/second   |
+| Tracking 10K assets  | ~500K messages/second   |
 
-## Production-Grade Resilience
-Real streams have noise. Sensors fail. Networks hiccup. WinkComposer handles it all:
+Two configurations produced these numbers. The first two rows interleave 10 asset pipelines in random order — 4.5 million messages (10 pipelines × 900 data points × 500 iterations). The third row runs the same flow with 10,000 assets — 10,000 isolated states alive at once, 9 million messages, under 140 MB of heap. Pipelines are created dynamically as each asset first appears; timing uses `process.hrtime.bigint()`. The same pipeline [runs in your browser](https://composer.winkjs.org/docs/benchmark) — browser performance varies from native Node.js due to JIT differences.
 
-- **Node-level fault isolation** - One bad reading doesn't crash your pipeline
-- **Automatic state recovery** - Snapshot and restore without missing a beat
-- **Built-in observability** - Know what's happening in every node, every partition
-- **Adaptive execution** - Maintains responsiveness under varying load
 
+## Built for the Real World
 
-## Core Principles
+Write a flow once and run it anywhere Node.js does — an industrial-grade Raspberry Pi, a production server, a Kubernetes cluster. Each asset's state is isolated, so a fault in one stays in one. Messages [queue locally](docs/handbook/resilience.md) when the network drops and drain cleanly on reconnect. Shutdown is ordered and deterministic: sources close first, storage last. A misconfigured flow fails when you define it, not in production — unknown options, output collisions, and bad triggers are all caught at definition time.
 
-| Principle | What it means |
-|-----------|---------------|
-| **Atomic Operations** | Each node does one thing perfectly |
-| **Pure Functions** | Predictable, testable, composable |
-| **Transparent Logic** | Understand every decision |
-| **Production Ready** | Built for high-throughput streaming with microsecond-scale latency |
-| **AI-Native Intelligence** | From statistical learning nodes to LLM reasoning - intelligence at every level |
-| **Built for Reality** | Node-level fault isolation, automatic recovery, and comprehensive observability |
+The test suite holds over 6,500 tests behind a 99.5% coverage gate. Integration tests run against real Mosquitto and QuestDB services, not mocks. Every npm release carries SLSA provenance — a public, verifiable link from the package back to the exact source commit and the build that produced it. `npm audit signatures` verifies it.
 
-## What's Coming
+The documentation also relates winkComposer's concepts to two industrial standards: [ISO 13374](https://composer.winkjs.org/docs/reference/iso-13374-mapping) for condition monitoring, the [NIST AI RMF](https://composer.winkjs.org/docs/reference/nist-ai-rmf-mapping) for AI risk management.
 
-### 🎉 **Opening Our Doors in 2025**
-WinkComposer is transitioning to open source development in 2025. Join the [early conversation](https://github.com/winkjs/composer/discussions) and help shape the future of composable streaming analytics. You can even reach out directly at wink@graype.in.
+Flows can run headless inside your own application — no broker, no services. Your code feeds messages in from any source you already run. Already running a historian (the plant's time-series system of record)? winkComposer adds value beside it — it watches the live stream and acts, while the historian keeps the record.
 
-**Together, we're building:**
 
-- **Rich Node Library**: Statistical, filtering, detection, transformation, and control nodes
-- **Edge-Ready Intelligence**: Advanced algorithms like smart correlation, change point detection, and regression inference - AI-like capabilities without heavyweight frameworks
-- **Pipeline Builder**: Define your analytics flows declaratively (visual tools planned)
-- **Edge-Ready Performance**: Process data efficiently, even on resource-constrained devices (**tested on Raspberry Pi**)
-- **Knowledge Graph Integration**: Build digital twins with Neo4j/ArangoDB for contextual intelligence
-- **AI-Native Reasoning**: When patterns emerge, LLMs explain why. When anomalies occur, get actionable recommendations. Open-source models keep intelligence at the edge.
+## Documentation
 
+The documentation lives in two places: the [documentation site](https://composer.winkjs.org) for interactive learning, and the in-repo handbook for version-matched reference.
 
-## The Vision
+| Resource | What it covers |
+|---|---|
+| [**Hello Flow!**](https://composer.winkjs.org/docs/playground/hello-flow) | Build a 4-node temperature monitor from scratch — smooth, detect, confirm, broadcast — with an interactive demo running real nodes in your browser. The natural starting point. |
+| [**Recipes**](https://composer.winkjs.org/docs/playground/recipes) | Short, runnable patterns for recurring failure modes — each names the nodes it uses and runs in the browser. |
+| [**Explore Nodes**](https://composer.winkjs.org/docs/playground/explore-nodes) | Single-node sandboxes — drag a slider, watch the node respond in real time. Covers the Kalman 1D filter and the kernel convolution node, with more to come. |
+| [**Under the Hood**](https://composer.winkjs.org/docs/concepts/under-the-hood) | How messages flow and get enriched node by node, how bad data and throwing functions are handled without crashing the pipeline, how per-asset isolation works, and timestamp requirements. |
+| [**Flow Language**](https://composer.winkjs.org/docs/concepts/flow-language) | The complete DSL reference — flow anatomy, node call signatures, dynamic options (tunables), single vs. multi-field processing, naming policies, and node processing types. |
+| [**Composition Patterns**](https://composer.winkjs.org/docs/concepts/composition-patterns) | Proven node combinations for recurring problems: noise-tolerant alarms, drift detection, adaptive diagnostics, layered flows, and downsampling for storage. Includes clear guidance on when to use passIf, emitIf, or controller/disable. |
+| [**Semantics**](https://composer.winkjs.org/docs/concepts/semantics) | How to define what computed values mean — types, units, physical ranges, operational limits — as a single source of truth shared by storage, dashboards, and query engines. Covers the facts-vs-decisions design principle. |
+| [**Node Index**](https://composer.winkjs.org/docs/reference/node-index) | Every node grouped by category — Signal Conditioning, Detection, Feature Extraction, Intelligence, and more — with what each computes and what it adds to the message. |
+| [**Handbook**](docs/handbook/index.md) | The complete reference, right here in the repo and version-matched: the handbook at any release tag describes exactly that release's code. |
+| [**Changelog**](CHANGELOG.md) | Release notes for every published version — what changed, what broke, and what to do about it. |
+| [**Roadmap**](ROADMAP.md) | What is planned next, item by item, with complexity and status. |
 
-Finally, build solutions that fit your exact problems, not the other way around.
 
-Every industry has unique patterns. Every process has specific physics. Every business has particular rules. WinkComposer gives you the building blocks to create exactly what you need.
+## Get Involved
 
-Domain-specific platforms excel at standard use cases. WinkComposer shines when you need something unique, beyond standard use cases. Standard use cases always included - they're just the beginning of what's possible!
+| [**Star winkComposer**](https://github.com/winkjs/composer) | [**Follow @winkjs**](https://github.com/winkjs) | [**Discussions**](https://github.com/winkjs/composer/discussions) |
+|:---:|:---:|:---:|
+| Support open-source streaming intelligence. | Stay updated on releases and ecosystem developments. | Questions, ideas, or feedback — all welcome. |
 
+Want to contribute code or docs? Start with the [contributing guide](CONTRIBUTING.md). The project follows the [Contributor Covenant](CODE_OF_CONDUCT.md).
 
 
 ## About winkJS
 
-WinkComposer joins the winkJS family - production-grade open-source packages for Natural Language Processing, Machine Learning, and Statistical Analysis in JavaScript. Built for reliability with ~100% test coverage.
+[winkJS](https://winkjs.org) is the open-source home of high-performance JavaScript tools. winkNLP covers natural language processing; winkComposer covers streaming intelligence. Both follow the same discipline: developer-friendly APIs, measured performance, and well-tested code.
 
 
-## Copyright & License
+## License
 
-**Wink Composer** is copyright 2017-25 [GRAYPE Systems Private Limited](https://graype.in/).
+winkComposer is released under the [MIT License](LICENSE).
 
-It is licensed under the terms of the MIT License.
+Copyright (c) 2024-26 GRAYPE Systems Private Limited.

package/package.json

@@ -1,30 +1,36 @@
 {
     "name": "@winkjs/composer",
-    "version": "0.0.2",
-    "description": "Composable Streaming Intelligence.",
+    "version": "0.4.1",
+    "description": "Composable Streaming Intelligence — turn streaming data into actionable insights, from edge to cloud",
     "keywords": [
+        "streaming",
+        "streaming-intelligence",
+        "streaming-analytics",
+        "real-time",
+        "edge-computing",
         "iot",
-        "statistical-learning",
-        "stream-processing",
-        "statistical-analysis",
-        "event-driven",
-        "composable",
-        "real-time-analytics",
+        "iiot",
+        "industrial-iot",
+        "time-series",
+        "signal-processing",
         "anomaly-detection",
-        "edge-computing",
-        "edgi-ai",
-        "streaming-analytics",
-        "industrial-iot"
+        "predictive-maintenance",
+        "mqtt",
+        "questdb",
+        "winkjs",
+        "wink"
     ],
     "main": "src/composer.js",
+    "module": "src/composer.js",
+    "exports": {
+        ".": "./src/composer.js"
+    },
     "scripts": {
         "pretest": "npm run lint",
-        "test": "nyc --reporter=html --reporter=lcov --reporter=text mocha ./test/",
-        "coverage": "nyc report --reporter=text-lcov | coveralls",
-        "sourcedocs": "docker -i src -o ./sourcedocs --sidebar yes",
-        "lint": "eslint ./src/*.js ./test/*.js",
-        "perf": "node benchmark/run.js",
-        "dtslint": "dtslint types",
+        "test": "mocha --recursive 'src/**/*.specs.js' --ignore 'src/**/slow-*.specs.js'",
+        "test:hardening": "node --expose-gc node_modules/.bin/mocha --recursive 'src/**/slow-*.specs.js' --timeout 300000 --exit",
+        "test:coverage": "c8 --reporter=html mocha --recursive 'src/**/*.specs.js' --ignore 'src/**/slow-*.specs.js'",
+        "lint": "eslint 'src/**/*.js'",
         "docs-serve": "docsify serve ./docs"
     },
     "repository": {
@@ -33,19 +39,41 @@
     },
     "author": "Sanjaya Kumar Saxena",
     "license": "MIT",
+    "type": "module",
+    "engines": {
+        "node": ">=22"
+    },
+    "files": [
+        "src/composer.js",
+        "src/core/**/*.js",
+        "src/flow/**/*.js",
+        "src/nodes/**/*.js",
+        "src/tools/**/*.js",
+        "src/windowing/**/*.js",
+        "!src/**/*.specs.js",
+        "!src/**/test/"
+    ],
     "bugs": {
         "url": "https://github.com/winkjs/composer/issues"
     },
-    "homepage": "http://winkjs.org/composer/#/",
+    "homepage": "https://composer.winkjs.org/",
+    "publishConfig": {
+        "access": "public"
+    },
     "devDependencies": {
-        "benchmark": "^2.1.4",
-        "chai": "^4.3.7",
-        "coveralls": "^3.1.1",
-        "docker": "^1.0.0",
-        "docsify-cli": "^4.4.4",
-        "dtslint": "^4.2.1",
-        "eslint": "^8.36.0",
-        "mocha": "^10.2.0",
-        "nyc": "^15.1.0"
+        "c8": "^10.1.3",
+        "chai": "^5.2.0",
+        "eslint": "^9.21.0",
+        "eslint-plugin-security": "^4.0.0",
+        "mocha": "^11.1.0",
+        "sinon": "^19.0.2"
+    },
+    "dependencies": {
+        "@questdb/nodejs-client": "^4.2.0",
+        "classic-level": "^3.0.0",
+        "glob": "^13.0.6",
+        "mqtt": "^5.15.1",
+        "msgpackr": "^1.11.10",
+        "pg": "^8.20.0"
     }
 }

package/src/composer.js

@@ -0,0 +1,75 @@
+// src/composer.js
+
+/**
+ * @fileoverview Main entry point for @winkjs/composer.
+ *
+ * Provides flat exports for the DSL, source adapters, emitter adapters,
+ * codecs, and tunable helpers.
+ *
+ * @example
+ * import { flow, csv, mqttEmitter } from '@winkjs/composer';
+ *
+ * flow('myPipeline')
+ *     .sanitize(...)
+ *     .threshold(...)
+ *     .source(csv, { path: './data.csv' })
+ *     .emitter(mqttEmitter, { brokerUrl: 'mqtt://localhost' })
+ *     .run();
+ */
+
+// DSL
+export { flow } from './flow/flow.js';
+
+// Headless driver — feed a flow that has no source (see docs/handbook/headless-flow.md)
+export { headlessDriver } from './flow/driver.js';
+
+// Emitters
+export * as mqttEmitter from './core/emitter-manager/mqtt/index.js';
+export * as terminal from './core/emitter-manager/terminal/index.js';
+
+// Sources
+export * as csv from './core/source-manager/csv/index.js';
+export * as mqttSource from './core/source-manager/mqtt/index.js';
+
+// Storage
+export { default as questdbAdapter } from './core/storage-manager/questdb/index.js';
+
+// Semantics
+export { loadSemantics } from './core/semantics/loader.js';
+
+// Codecs
+export { jsonCodec, msgpackCodec } from './core/codec/index.js';
+
+// Tunable helpers
+export {
+    lookupByField,
+    pickByField,
+    scaleBy,
+    chooseWhen,
+    clampTo,
+    fromField,
+    offsetBy
+} from './core/tunable/helpers.js';
+
+// Transform helpers
+export {
+    square,
+    abs,
+    sqrt,
+    log,
+    log10,
+    reciprocal,
+    negate
+} from './nodes/transform/helpers.js';
+
+// Training utilities
+export {
+    computeStandardParams,
+    computeMinMaxParams,
+    computeRobustParams,
+    scale,
+    standardize
+} from './tools/training/scale.js';
+export { sigmoid, train, predict, classify } from './tools/training/logistic.js';
+export { confusionMatrix, classificationMetrics, sweepThresholds } from './tools/training/metrics.js';
+

package/src/core/batch/aoo-encoder.js

@@ -0,0 +1,148 @@
+// core/batch/aoo-encoder/index.js
+// Purpose:
+//   Array-of-Objects (AoO) encoder for efficient batch transmission of tabular data.
+//   Optimized for downsampled time-series data where rows have consistent shape.
+//   Designed to work with msgpackr's structure learning for maximum compression.
+//
+// Design Philosophy:
+//   - Zero allocations in hot path - reuse all objects and arrays
+//   - Stable object shapes - enables msgpackr structure learning
+//   - Predictable field order - consistent compression ratios
+//   - Null preservation - maintains shape even with missing data
+//
+// Use Case:
+//   Downsampled sensor data often arrives as rows of readings:
+//   [{ timestamp: 1234, temp: 23.5, pressure: 101.3 }, ...]
+//   This encoder batches these rows efficiently for MQTT transmission.
+//
+// Compression Strategy:
+//   By maintaining stable object shapes and field order, msgpackr learns the
+//   structure after ~5 messages. Subsequent messages only send structure ID + values,
+//   achieving 80-90% size reduction compared to JSON.
+
+/**
+ * Create an Array-of-Objects encoder for batch data transmission.
+ *
+ * @param {Object} options - Encoder configuration
+ * @param {number} options.maxRows - Maximum rows per batch (pre-allocation size)
+ * @param {string[]} options.fieldOrder - Field names in consistent order
+ * @param {Object} options.codec - Codec instance from createCodec()
+ *
+ * @returns {Object} Encoder with encode() method and fieldOrder reference
+ *
+ * @example
+ * const encoder = createAooEncoder({
+ *   maxRows: 100,
+ *   fieldOrder: ['timestamp', 'temperature', 'pressure'],
+ *   codec: msgpackCodec
+ * });
+ *
+ * const { payload, props } = encoder.encode('sensor-01', 1, readings, 50);
+ * mqttClient.publish('telemetry/batch', payload, { properties: props });
+ */
+export const createAooEncoder = function ( { maxRows, fieldOrder, codec } ) {
+    // Input validation - fail fast on misconfiguration
+    if ( !Array.isArray( fieldOrder ) || fieldOrder.length === 0 ) {
+        throw new Error( 'fieldOrder must be non-empty array of field names' );
+    }
+    if ( !maxRows || maxRows <= 0 ) {
+        throw new Error( 'maxRows must be positive integer' );
+    }
+    if ( !codec || typeof codec.pack !== 'function' ) {
+        throw new Error( 'codec must have pack() method');
+    }
+
+    // Pre-allocate row objects with stable shape.
+    // Critical for msgpackr structure learning:
+    // - Same fields in same order = same structure ID
+    // - null instead of undefined = consistent serialization
+    // - Object.create(null) = no prototype pollution
+    const rows = new Array( maxRows );
+    for ( let i = 0; i < maxRows; i += 1 ) {
+        const r = Object.create( null );
+        // Initialize all fields to null (not undefined!)
+        // This ensures consistent object shape even when values are missing
+        for ( let k = 0; k < fieldOrder.length; k += 1 ) {
+            r[ fieldOrder[ k ] ] = null;
+        }
+        rows[ i ] = r;
+    }
+
+    // Pre-allocate batch array that references row objects.
+    // We'll adjust its length per batch but never reallocate.
+    // slice() here is a one-time shallow copy of references.
+    const batch = rows.slice();
+
+    // Cache MQTT properties to avoid repeated calls.
+    // These properties hint the wire format to subscribers.
+    const props = codec.properties || {};
+
+    /**
+     * Encode source rows into a compressed batch payload.
+     *
+     * Design decisions:
+     *   - Copy values field-by-field to maintain stable shape
+     *   - Preserve null but normalize undefined to null
+     *   - Reuse pre-allocated objects to avoid GC
+     *   - Temporarily adjust batch.length without reallocation
+     *
+     * @param {string} producerId - Identifies the data source (e.g., device ID)
+     * @param {number} seq - Sequence number for gap detection on receiver side
+     * @param {Array} srcRows - Source rows to encode (not modified)
+     * @param {number} count - Number of rows to encode (may be < srcRows.length)
+     *
+     * @returns {Object} { payload: Uint8Array|Buffer, props: Object }
+     *                   payload: Compressed binary data
+     *                   props: MQTT v5 properties for content type hint
+     */
+    const encode = function ( producerId, seq, srcRows, count ) {
+        // Validate count doesn't exceed pre-allocated capacity
+        if ( count > maxRows ) {
+            throw new Error( `Cannot encode ${count} rows, max capacity is ${maxRows}` );
+        }
+
+        // Copy source data into pre-allocated row objects
+        for ( let i = 0; i < count; i += 1 ) {
+            const src = srcRows[ i ];
+            const dst = rows[ i ];
+
+            // Copy only declared fields in consistent order
+            for ( let k = 0; k < fieldOrder.length; k += 1 ) {
+                const key = fieldOrder[ k ];
+                const v = src[ key ];
+                // FIXED: Normalize undefined to null for consistent serialization
+                dst[ key ] = ( v === undefined ) ? null : v;
+            }
+
+            // Point batch array to this row (reference, not copy)
+            batch[ i ] = dst;
+        }
+
+        // Temporarily adjust batch length for this encoding
+        const cap = batch.length;
+        batch.length = count;
+
+        // Pack the message with consistent structure
+        const payload = codec.pack( {
+            producerId: producerId,
+            seq: seq,
+            items: batch
+        } );
+
+        // Restore batch array to full capacity for next call
+        batch.length = cap;
+
+        return {
+            payload: payload,
+            props: props
+        };
+    }; // encode()
+
+    // Public API
+    // Expose encode function and field order for reference.
+    // slice() returns a defensive copy of fieldOrder.
+    return {
+        encode: encode,
+        fieldOrder: fieldOrder.slice()
+    };
+};

package/src/core/batch/index.js

@@ -0,0 +1,12 @@
+// core/batch/index.js
+// Purpose:
+//   Batch data handling subsystem for efficient transmission of tabular data.
+//   Provides sliding windows for history retention and encoders for compression.
+
+export { createAooEncoder } from './aoo-encoder.js';
+export { createWindow } from './sliding-window.js';
+
+// These work together:
+// 1. Window maintains recent history (sliding-window.js)
+// 2. Encoder compresses for transmission (aoo-encoder.js)
+// 3. Both use same field order for structure stability