xy-scale 1.4.2 → 1.4.31

package/README.md

@@ -1,196 +1,130 @@
-# **xy-scale.js**  
-**Machine Learning Data Preparation Toolkit**  
-*XY Splitting, Feature Weighting, Standardization, and MinMax Scaling in JavaScript*
+# xy-scale.js
 
----
+Machine learning data preparation helpers for JavaScript.
 
 ## Overview
 
-`xy-scale.js` is a robust toolkit designed for scaling and preparing datasets in JavaScript, specifically tailored for machine learning applications. It offers essential utilities for:
+`xy-scale.js` now focuses on turning already-prepared row objects into flat `X` and `Y` arrays for training or production use.
 
-- **Dataset Splitting:** Dividing data into training and testing subsets.
-- **Scaling:** Handling both numerical and categorical data.
-- **Preprocessing:** Preparing time-series data for modeling.
-
-### Key Features
-
-- **Modular Functions:** `parseTrainingXY` and `parseProductionX` allow for flexible handling of features and labels.
-- **Custom Scaling:** Enable custom scaling, feature weighting, and data transformation.
-- **Feature Grouping:** Organize features into meaningful categories for streamlined processing.
-
----
+The library no longer scales values internally. Your `arrObj` input, or the objects returned by your callbacks, should already contain the numeric or boolean values you want to feed into a model.
 
 ## Installation
 
-Install the toolkit via npm:
-
 ```bash
 npm install xy-scale
 ```
 
----
-
-## Main Functions
-
-### 1. `parseTrainingXY`
-
-Processes a dataset for supervised learning by scaling and splitting it into training and testing subsets. It supports configurable options for feature grouping, weighting, and scaling.
-
-**Parameters:**
+## Exports
 
-- `arrObj` (Array of Objects): Input data containing features (X) and labels (Y) e.g, `[{high: 10, low: 8, close: 9, dayOfTheWeek: 'monday'}, {high: 11, low: 8, close: 10, dayOfTheWeek: 'tuesday'}]`. Each Item an be a `number`, `string` or `boolean`.
-- `trainingSplit` (Number, optional): Fraction of data for training (default: `0.8`).
-- `yCallbackFunc` (Function): Extracts Y values from each row. Returns `null` or `undefined` to exclude a row.
-- `xCallbackFunc` (Function): Extracts X values from each row. Returns `null` or `undefined` to exclude a row.
-- `groups` (Object, optional): Groups continuous values features into categories after `yCallbackFunc` and `xCallbackFunc` callbacks are applied (e.g., `{ ohlc: ['open', 'high', 'low', 'close'] }`). Each feature belonging to a group will be MinMax scaled or standardized using the group's properties (`min`, `max`, `std`).
-- `shuffle` (Boolean, optional): Randomizes data order after `yCallbackFunc` and `xCallbackFunc` callbacks are applied (default: `false`).
-- `repeat` (Object, optional): Determines feature repetition after `yCallbackFunc` and `xCallbackFunc` callbacks are applied.
-- `balancing` (String, optional): Handles inbalanced datasets applying `oversample` or `undersample` to `X` and `Y` (defaults to `null`);
-
-**Returns:**
-
-- `trainX`, `trainY`: Scaled training features and labels.
-- `testX`, `testY`: Scaled testing features and labels.
-- `configX`, `configY`: Scaling configuration objects for features and labels.
-- `keyNamesX`, `keyNamesY`: Key names reflecting feature repetition and grouping.
-
----
+```javascript
+import { parseTrainingXY, parseProductionX, arrayToTimesteps } from 'xy-scale';
+```
 
-### 2. `parseProductionX`
+## Main functions
 
-Parses and scales unseen production data for feature preparation.
+### parseTrainingXY
 
-**Parameters:**
+Builds supervised-learning datasets and splits them into training and testing arrays.
 
-- `arrObj` (Array of Objects): Input data containing features (X) and labels (Y) e.g, `[{high: 10, low: 8, close: 9, dayOfTheWeek: 'monday'}, {high: 11, low: 8, close: 10, dayOfTheWeek: 'tuesday'}]`. Each Item an be a `number`, `string` or `boolean`.
-- `repeat` (Object, optional): Determines feature repetition after `yCallbackFunc` and `xCallbackFunc` callbacks are applied.
-- `xCallbackFunc` (Function): Extracts X values from each row. Returns `null` or `undefined` to exclude a row.
-- `groups` (Object, optional): Groups continuous values features into categories after `yCallbackFunc` and `xCallbackFunc` callbacks are applied (e.g., `{ ohlc: ['open', 'high', 'low', 'close'] }`). Each feature belonging to a group will be MinMax scaled or standardized using the group's properties (`min`, `max`, `std`).
-- `shuffle` (Boolean, optional): Randomizes data order after `yCallbackFunc` and `xCallbackFunc` callbacks are applied (default: `false`).
+#### Parameters
 
-**Returns:**
+- `arrObj` (Array<Object>): Source dataset.
+- `trainingSplit` (Number, optional): Fraction of rows used for training. Default: `0.8`.
+- `yCallbackFunc` (Function, optional): Builds the output object for each row. Returning `null` or `undefined` skips the row.
+- `xCallbackFunc` (Function, optional): Builds the feature object for each row. Returning `null` or `undefined` skips the row.
+- `validateRows` (Function, optional): Extra row filter executed before the callbacks.
+- `shuffle` (Boolean, optional): Shuffles `X` and `Y` together before splitting. Default: `false`.
+- `balancing` (String, optional): Accepts `oversample` or `undersample`.
+- `state` (Object, optional): Shared mutable state passed into callbacks.
 
-- `X`: Scaled feature array.
-- `configX`: Scaling configuration for features.
-- `keyNamesX`: Key names reflecting feature repetition and grouping.
+#### Returns
 
----
+- `trainX`, `trainY`
+- `testX`, `testY`
+- `configX`: `{ keyNames: [...] }`
+- `configY`: `{ keyNames: [...] }`
 
-### 3. `descaleArrayObj`
+`configX.keyNames` and `configY.keyNames` preserve the object-key order used when flattening each callback result into an array.
 
-Reconstructs original data values from scaled arrays.
+### parseProductionX
 
-**Parameters:**
+Builds production-ready feature arrays from already-prepared rows.
 
-- `scaled` (Array): Scaled data array (e.g., `trainX`, `trainY`, etc.).
-- `config` (Object): Configuration object returned from `parseTrainingXY` or `parseProductionX`.
-- `keyNames` (Array): Key names reflecting feature repetition and grouping.
+#### Parameters
 
-**Returns:**
+- `arrObj` (Array<Object>): Source dataset.
+- `xCallbackFunc` (Function, optional): Builds the feature object for each row. Returning `null`, `undefined`, or `false` skips the row.
+- `validateRows` (Function, optional): Extra row filter executed before the callback.
+- `shuffle` (Boolean, optional): Shuffles the final `X` rows. Default: `false`.
+- `state` (Object, optional): Shared mutable state passed into the callback.
 
-- An array of objects resembling the original input data structure.
+#### Returns
 
----
+- `X`
+- `configX`: `{ keyNames: [...] }`
 
-### 4. `arrayToTimesteps`
+### arrayToTimesteps
 
-Converts a flat array into sequences for time-series modeling.
+Converts a flat array into overlapping sequences for time-series models.
 
-**Parameters:**
+#### Parameters
 
 - `arr` (Array): Input array.
 - `timeSteps` (Number): Length of each sequence.
   - If `timeSteps === 0`, returns the original array.
   - If `timeSteps < 0`, throws an error.
 
-**Returns:**
+#### Returns
 
 - An array of overlapping sub-arrays, each containing `timeSteps` elements.
 
----
-
-## Usage Example
+## Usage example
 
 ```javascript
-import { parseTrainingXY, parseProductionX, descaleArrayObj, arrayToTimesteps } from 'xy-scale';
-import { loadFile } from './fs.js';
+import { parseTrainingXY, arrayToTimesteps } from 'xy-scale';
 import * as tf from '@tensorflow/tfjs-node';
 
-(async () => {
-    const myArray = await loadFile({ fileName: '1d-spy.json', pathName: 'datasets' });
-
-    // Callback function for parsing `X` features
-    const xCallbackFunc = ({ objRow, index }) => {
-        const curr = objRow[index];
-        const prev = objRow[index - 1];
-        if (typeof prev === 'undefined') return null;
-
-        const { open, high, low, close, volume } = curr;
-        return {
-            open,
-            high,
-            low,
-            close,
-            change: open - prev.close,
-            top: high - Math.max(open, close),
-            bottom: low - Math.min(open, close),
-            body: open - close,
-        };
-    };
-
-    // Callback function for parsing `Y` labels
-    const yCallbackFunc = ({ objRow, index }) => {
-        const curr = objRow[index];
-        const next = objRow[index + 1];
-        if (typeof next === 'undefined') return null;
-
-        return {
-            label_1: next.open > curr.close,
-            label_2: next.close > curr.close,
-        };
-    };
-
-    // Parse and scale training data
-    const {
-        trainX,
-        trainY,
-        testX,
-        testY,
-        configX,
-        keyNamesX,
-    } = parseTrainingXY({
-        arrObj: myArray,
-        trainingSplit: 0.9,
-        yCallbackFunc,
-        xCallbackFunc,
-        groups: { ohlc: ['open', 'high', 'low', 'close'] },
-        shuffle: true,
-        repeat: { close: 20 },
-        balancing: null, //oversample or undersample
-    });
-
-    // Time-stepping and TensorFlow integration
-    const timeSteps = 10;
-    const timeSteppedTrainX = arrayToTimesteps(trainX, timeSteps);
-    const trimmedTrainY = trainY.slice(timeSteps - 1);
-
-    const inputX = tf.tensor3d(timeSteppedTrainX, [timeSteppedTrainX.length, timeSteps, trainX[0].length]);
-    const targetY = tf.tensor2d(trimmedTrainY, [trimmedTrainY.length, trainY[0].length]);
-
-    console.log('configX', configX);
-    console.log('inputX', inputX);
-    console.log('targetY', targetY);
-})();
+const candles = [
+  { closeScaled: 0.41, volumeScaled: 0.22, targetUp: 1 },
+  { closeScaled: 0.45, volumeScaled: 0.25, targetUp: 0 },
+  { closeScaled: 0.48, volumeScaled: 0.28, targetUp: 1 },
+  { closeScaled: 0.51, volumeScaled: 0.31, targetUp: 1 },
+  { closeScaled: 0.49, volumeScaled: 0.27, targetUp: 0 },
+  { closeScaled: 0.54, volumeScaled: 0.35, targetUp: 1 },
+];
+
+const { trainX, trainY, testX, testY, configX, configY } = parseTrainingXY({
+  arrObj: candles,
+  trainingSplit: 0.8,
+  shuffle: true,
+  xCallbackFunc: ({ objRow, index }) => ({
+    close: objRow[index].closeScaled,
+    volume: objRow[index].volumeScaled,
+  }),
+  yCallbackFunc: ({ objRow, index }) => ({
+    target: objRow[index].targetUp,
+  }),
+});
+
+const timeSteps = 3;
+const timeSteppedTrainX = arrayToTimesteps(trainX, timeSteps);
+const trimmedTrainY = trainY.slice(timeSteps - 1);
+
+const inputX = tf.tensor3d(timeSteppedTrainX, [timeSteppedTrainX.length, timeSteps, trainX[0].length]);
+const targetY = tf.tensor2d(trimmedTrainY, [trimmedTrainY.length, trainY[0].length]);
+
+console.log(configX.keyNames);
+console.log(configY.keyNames);
+console.log(testX, testY);
+console.log(inputX, targetY);
 ```
 
----
-
-## Additional Information
-
-For more detailed documentation, examples, and support, please visit the [GitHub repository](https://github.com/jaimelias/xy-scale).
+## Notes
 
----
+- `parseTrainingXY` and `parseProductionX` do not scale values.
+- If you need scaling, do it before passing data into this library.
+- Callback return objects are flattened with `Object.values(...)`, using the same key order stored in `configX.keyNames` and `configY.keyNames`.
 
 ## License
 
-This project is licensed under the [MIT License](LICENSE).
+This project is licensed under the MIT License.

package/dist/xy-scale.min.js

@@ -1 +1 @@
-var XY_Scale;(()=>{"use strict";var e={d:(t,n)=>{for(var r in n)e.o(n,r)&&!e.o(t,r)&&Object.defineProperty(t,r,{enumerable:!0,get:n[r]})},o:(e,t)=>Object.prototype.hasOwnProperty.call(e,t),r:e=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})}},t={};e.r(t),e.d(t,{arrayToTimesteps:()=>c,parseProductionX:()=>l,parseTrainingXY:()=>u});const n=({arrObj:e,repeat:t={},minmaxRange:n,groups:o={},customMinMaxRanges:s=null,excludes:i=new Set})=>{const u=[...e],l=u.length,c=u[0],p="object"==typeof s&&null!==s;if(0===l)return{scaledOutput:[],scaledConfig:{}};const f=Object.keys(c),m=f.map((e=>t.hasOwnProperty(e)?Math.max(t[e],1):1)),d=m.reduce(((e,t)=>e+t),0),g={arrObjLen:l,rangeMin:n[0],rangeMax:n[1],inputTypes:{},min:{},max:{},groupMinMax:{},repeat:t,groups:o,inputKeyNames:f,outputKeyNames:new Array(d),repeatedKeyNames:m};let h=0;for(let e=0;e<g.inputKeyNames.length;e++)for(let t=0;t<g.repeatedKeyNames[e];t++)g.outputKeyNames[h++]=g.inputKeyNames[e];r(g.groups);const y=["number","boolean"];for(const e of g.inputKeyNames){if(i.has(e)){g.inputTypes[e]="excluded";continue}const t=typeof c[e],n=a(e,g.groups);if(!y.includes(t))throw new Error(`Invalid input type "${t}" provided for key "${e}". Only accepting `);g.inputTypes[e]=t,p&&s.hasOwnProperty(e)?n?g.groupMinMax[n]=s[e]:(g.min[e]=s[e].min,g.max[e]=s[e].max):n?g.groupMinMax[n]={min:1/0,max:-1/0}:(g.min[e]=1/0,g.max[e]=-1/0)}for(const e of u)for(const t of g.inputKeyNames){if("excluded"===g.inputTypes[t])continue;let n=e[t];"boolean"===g.inputTypes[t]&&(e[t]=Number(n));const r=a(t,g.groups);(!1===p||p&&!s.hasOwnProperty(t))&&(r?(g.groupMinMax[r].min=Math.min(g.groupMinMax[r].min,n),g.groupMinMax[r].max=Math.max(g.groupMinMax[r].max,n)):(g.min[t]=Math.min(g.min[t],n),g.max[t]=Math.max(g.max[t],n)))}const x=new Array(l);for(let e=0;e<l;e++){const t=u[e],n=new Array(g.outputKeyNames.length);let r=0;for(let e=0;e<g.inputKeyNames.length;e++){const o=g.inputKeyNames[e],s=t[o];if("excluded"===g.inputTypes[o]){n[r++]=s;continue}const i=a(o,g.groups);let u,l;i?(u=g.groupMinMax[i].min,l=g.groupMinMax[i].max):(u=g.min[o],l=g.max[o]);const c=l!==u?g.rangeMin+(s-u)/(l-u)*(g.rangeMax-g.rangeMin):g.rangeMin,p=g.repeatedKeyNames[e];for(let e=0;e<p;e++)n[r++]=c}x[e]=n}return{scaledOutput:x,scaledConfig:g}},r=e=>{const t=new Set,n=[];for(const[r,a]of Object.entries(e))t.add(r),n.push(r),a.forEach((e=>{t.add(e),n.push(e)}));if(t.size!==n.length)throw new Error("Duplicate value found between properties in validateUniqueProperties function.")},a=(e,t)=>{for(const[n,r]of Object.entries(t))if(r.includes(e))return n;return null},o=(e,t)=>{const n=Object.keys(e);if(!Array.isArray(t))throw new Error('Property "excludes" must be an array.');for(const e of t)if(!n.includes(e))throw new Error(`An item in "excludes" property was not found in "arrObj".\n\nexcludes: ${JSON.stringify(t)}\n\narrObj: ${JSON.stringify(n)}`)},s=(e,{min:t=-1/0,max:n=1/0},r)=>{if(!Array.isArray(e))throw new Error(`Invalid property. "${r}" expected an array.`);if(e.length<t)throw new Error(`Invalid property value. Array "${r}" expected at least ${n} items.`);if(e.length>n)throw new Error(`Invalid property value. Array "${r}" expected at max ${n} items.`);return!0},i=e=>{for(const[t,n]of Object.entries(e)){if("number"==typeof n&&Number.isNaN(n))throw new Error(`Invalid value at index 0 property "${t}": value is "${n}". Expected a numeric value.`);if(null===n)throw new Error(`Invalid value at index 0 property "${t}": value is "${n}".`)}return!0},u=({arrObj:e=[],trainingSplit:t=.8,repeat:r={},yCallbackFunc:a=e=>e,xCallbackFunc:u=e=>e,validateRows:l=()=>!0,groups:c={},shuffle:p=!1,minmaxRange:f=[0,1],balancing:m="",state:d={},customMinMaxRanges:g={},excludes:h=[]})=>{let y=[],x=[];s(e,{min:5},"parseTrainingXY"),i(e[0]),o(e[0],h);for(let t=0;t<e.length;t++){if(!l({objRow:e,index:t,state:d}))continue;const n=u({objRow:e,index:t,state:d}),r=a({objRow:e,index:t,state:d});null!=n&&null!=r&&(y.push(n),x.push(r))}if(p){const{shuffledX:e,shuffledY:t}=((e,t)=>{if(e.length!==t.length)throw new Error("X and Y arrays must have the same length");const n=Array.from({length:e.length},((e,t)=>t));for(let e=n.length-1;e>0;e--){const t=Math.floor(Math.random()*(e+1));[n[e],n[t]]=[n[t],n[e]]}return{shuffledX:n.map((t=>e[t])),shuffledY:n.map((e=>t[e]))}})(y,x);y=e,x=t}const M=new Set(h);let{scaledOutput:b,scaledConfig:w}=n({arrObj:y,repeat:r,groups:c,minmaxRange:f,customMinMaxRanges:g,excludes:M});const O=x.length,v=new Array(O),j={keyNames:Object.keys(x[0])};for(let e=0;e<O;e++)v[e]=Object.values(x[e]);const N=Math.floor(b.length*t);let E=b.slice(0,N),X=v.slice(0,N),R=b.slice(N),S=v.slice(N);if(m){let e;if("oversample"===m)e=((e,t)=>{const n={},r={};t.forEach(((a,o)=>{n[a]||(n[a]=0,r[a]=[]),n[a]++,r[a].push([e[o],t[o]])}));const a=Math.max(...Object.values(n)),o=[],s=[];return Object.keys(r).forEach((e=>{const t=r[e],n=t.length;for(let e=0;e<a;e++){const r=t[e%n];o.push(r[0]),s.push(r[1])}})),{X:o,Y:s}})(E,X),E=e.X,X=e.Y;else{if("undersample"!==m)throw Error('balancing argument only accepts "false", "oversample" and "undersample". Defaults to "false".');e=((e,t)=>{const n={},r={};t.forEach(((a,o)=>{n[a]||(n[a]=0,r[a]=[]),n[a]++,r[a].push([e[o],t[o]])}));const a=Math.min(...Object.values(n)),o=[],s=[];return Object.keys(r).forEach((e=>{const t=r[e];for(let e=0;e<a;e++){const n=t[e];o.push(n[0]),s.push(n[1])}})),{X:o,Y:s}})(E,X),E=e.X,X=e.Y}}return{trainX:E,trainY:X,testX:R,testY:S,configX:w,configY:j}},l=({arrObj:e=[],repeat:t={},xCallbackFunc:r=e=>e,validateRows:a=()=>!0,groups:u={},shuffle:l=!1,minmaxRange:c=[0,1],state:p={},customMinMaxRanges:f,excludes:m=[]})=>{let d=[];s(e,{min:5},"parseProductionX"),i(e[0]),o(e[0],m);for(let t=0;t<e.length;t++){if(!a(e[t]))continue;const n=r({objRow:e,index:t,state:p});null!=n&&!1!==n&&d.push(n)}l&&(d=(e=>{const t=[...e];for(let e=t.length-1;e>0;e--){const n=Math.floor(Math.random()*(e+1));[t[e],t[n]]=[t[n],t[e]]}return t})(d));const{scaledOutput:g,scaledConfig:h}=n({arrObj:d,repeat:t,groups:u,minmaxRange:c,customMinMaxRanges:f,excludes:new Set(m)});return{X:g,configX:h}},c=(e,t)=>{if(0===t)return e;if(t<0)throw new Error("timeSteps must be greater than 0");const n=[];for(let r=0;r<=e.length-t;r++)n.push(e.slice(r,r+t));return n};XY_Scale=t})();
+var XY_Scale;(()=>{"use strict";var e={d:(t,r)=>{for(var n in r)e.o(r,n)&&!e.o(t,n)&&Object.defineProperty(t,n,{enumerable:!0,get:r[n]})},o:(e,t)=>Object.prototype.hasOwnProperty.call(e,t),r:e=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})}},t={};e.r(t),e.d(t,{arrayToTimesteps:()=>l,parseProductionX:()=>o,parseTrainingXY:()=>a});const r=(e,{min:t=-1/0,max:r=1/0},n)=>{if(!Array.isArray(e))throw new Error(`Invalid property. "${n}" expected an array.`);if(e.length<t)throw new Error(`Invalid property value. Array "${n}" expected at least ${r} items.`);if(e.length>r)throw new Error(`Invalid property value. Array "${n}" expected at max ${r} items.`);return!0},n=e=>{for(const[t,r]of Object.entries(e)){if("number"==typeof r&&Number.isNaN(r))throw new Error(`Invalid value at index 0 property "${t}": value is "${r}". Expected a numeric value.`);if(null===r)throw new Error(`Invalid value at index 0 property "${t}": value is "${r}".`)}return!0},a=({arrObj:e=[],trainingSplit:t=.8,yCallbackFunc:a=e=>e,xCallbackFunc:o=e=>e,validateRows:l=()=>!0,shuffle:s=!1,balancing:i="",state:c={}})=>{let u=[],f=[];r(e,{min:5},"parseTrainingXY"),n(e[0]);for(let t=0;t<e.length;t++){if(!l({objRow:e,index:t,state:c}))continue;const r=o({objRow:e,index:t,state:c}),n=a({objRow:e,index:t,state:c});null!=r&&null!=n&&(u.push(r),f.push(n))}if(s){const{shuffledX:e,shuffledY:t}=((e,t)=>{if(e.length!==t.length)throw new Error("X and Y arrays must have the same length");const r=Array.from({length:e.length},((e,t)=>t));for(let e=r.length-1;e>0;e--){const t=Math.floor(Math.random()*(e+1));[r[e],r[t]]=[r[t],r[e]]}return{shuffledX:r.map((t=>e[t])),shuffledY:r.map((e=>t[e]))}})(u,f);u=e,f=t}const h=u.length,p=f.length,d=new Array(h),y=new Array(p),m={keyNames:h?Object.keys(u[0]):[]},b={keyNames:p?Object.keys(f[0]):[]};for(let e=0;e<h;e++)d[e]=Object.values(u[e]);for(let e=0;e<p;e++)y[e]=Object.values(f[e]);const g=Math.floor(d.length*t);let v=d.slice(0,g),w=y.slice(0,g),j=d.slice(g),O=y.slice(g);if(i){let e;if("oversample"===i)e=((e,t)=>{const r={},n={};t.forEach(((a,o)=>{r[a]||(r[a]=0,n[a]=[]),r[a]++,n[a].push([e[o],t[o]])}));const a=Math.max(...Object.values(r)),o=[],l=[];return Object.keys(n).forEach((e=>{const t=n[e],r=t.length;for(let e=0;e<a;e++){const n=t[e%r];o.push(n[0]),l.push(n[1])}})),{X:o,Y:l}})(v,w),v=e.X,w=e.Y;else{if("undersample"!==i)throw Error('balancing argument only accepts "false", "oversample" and "undersample". Defaults to "false".');e=((e,t)=>{const r={},n={};t.forEach(((a,o)=>{r[a]||(r[a]=0,n[a]=[]),r[a]++,n[a].push([e[o],t[o]])}));const a=Math.min(...Object.values(r)),o=[],l=[];return Object.keys(n).forEach((e=>{const t=n[e];for(let e=0;e<a;e++){const r=t[e];o.push(r[0]),l.push(r[1])}})),{X:o,Y:l}})(v,w),v=e.X,w=e.Y}}return{trainX:v,trainY:w,testX:j,testY:O,configX:m,configY:b}},o=({arrObj:e=[],xCallbackFunc:t=e=>e,validateRows:a=()=>!0,shuffle:o=!1,state:l={}})=>{let s=[];r(e,{min:5},"parseProductionX"),n(e[0]);for(let r=0;r<e.length;r++){if(!a(e[r]))continue;const n=t({objRow:e,index:r,state:l});null!=n&&!1!==n&&s.push(n)}o&&(s=(e=>{const t=[...e];for(let e=t.length-1;e>0;e--){const r=Math.floor(Math.random()*(e+1));[t[e],t[r]]=[t[r],t[e]]}return t})(s));const i=s.length,c=new Array(i),u={keyNames:i?Object.keys(s[0]):[]};for(let e=0;e<i;e++)c[e]=Object.values(s[e]);return{X:c,configX:u}},l=(e,t)=>{if(0===t)return e;if(t<0)throw new Error("timeSteps must be greater than 0");const r=[];for(let n=0;n<=e.length-t;n++)r.push(e.slice(n,n+t));return r};XY_Scale=t})();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "xy-scale",
-  "version": "1.4.2",
+  "version": "1.4.31",
   "main": "./index.js",
   "type": "module",
   "scripts": {
@@ -14,7 +14,7 @@
     "@tensorflow/tfjs-node": "^4.22.0",
     "ml-confusion-matrix": "^2.0.0",
     "ml-knn": "^3.0.0",
-    "ohlcv-indicators": "^3.4.7",
+    "ohlcv-indicators": "^3.5.59",
     "webpack": "^5.96.1",
     "webpack-cli": "^5.1.4"
   },

package/src/datasets.js

@@ -1,32 +1,24 @@
-import { scaleArrayObj } from "./scale.js";
 import { arrayShuffle, xyArrayShuffle } from "./utilities.js";
 import { oversampleXY, undersampleXY } from "./balancing.js";
 import { validateFirstRow, validateArray } from "./validators.js";
-import { validateExcludes } from "./validators.js";
 
 //ADD A PARAM max correlation that will measure the correlation between variables if defined
 
 export const parseTrainingXY = ({ 
     arrObj = [], //array of objects
     trainingSplit = 0.8, //numberic float between 0.01 and 0.99
-    repeat = {}, //accepted key pair object with number as values
     yCallbackFunc = row => row, //accepted callback functions
     xCallbackFunc = row => row, //accepted callback functions
     validateRows = () => true,//accepted callback functions
-    groups = {},//accepted object of arrays
     shuffle = false,//only booleans
-    minmaxRange = [0, 1],
     balancing = '',//accepted null, "oversample" or "undersample"
     state = {}, //accepted object or classes
-    customMinMaxRanges = {},
-    excludes = [],//each item must be a string
 }) => {
     let X = [];
     let Y = [];
 
     validateArray(arrObj, {min: 5}, 'parseTrainingXY')
     validateFirstRow(arrObj[0])
-    validateExcludes(arrObj[0], excludes)
 
     //if parsedX and parsedY is undefined or null the current row will be excluded from training or production
     for (let x = 0; x < arrObj.length; x++) {
@@ -49,18 +41,20 @@ export const parseTrainingXY = ({
         Y = shuffledY
     }
 
-    const excludesSet = new Set(excludes)
-
-    let {
-        scaledOutput: scaledX, 
-        scaledConfig: configX
-    } = scaleArrayObj({arrObj: X, repeat, groups, minmaxRange, customMinMaxRanges, excludes: excludesSet})
-
-
+    const xLen = X.length
     const yLen = Y.length
+    const flatX = new Array(xLen)
     const flatY = new Array(yLen)
+    const configX = {
+        keyNames: xLen ? Object.keys(X[0]) : []
+    }
     const configY = {
-        keyNames: Object.keys(Y[0])
+        keyNames: yLen ? Object.keys(Y[0]) : []
+    }
+
+    for(let idx = 0; idx < xLen; idx++)
+    {
+        flatX[idx] = Object.values(X[idx])
     }
 
     for(let idx = 0; idx < yLen; idx++)
@@ -68,11 +62,11 @@ export const parseTrainingXY = ({
         flatY[idx] = Object.values(Y[idx])
     }
 
-    const splitIndex = Math.floor(scaledX.length * trainingSplit)
+    const splitIndex = Math.floor(flatX.length * trainingSplit)
 
-    let trainX = scaledX.slice(0, splitIndex)
+    let trainX = flatX.slice(0, splitIndex)
     let trainY = flatY.slice(0, splitIndex)
-    let testX = scaledX.slice(splitIndex)
+    let testX = flatX.slice(splitIndex)
     let testY = flatY.slice(splitIndex)
 
 
@@ -113,21 +107,15 @@ export const parseTrainingXY = ({
 
 export const parseProductionX = ({ 
     arrObj = [], 
-    repeat = {}, 
     xCallbackFunc = row => row,
     validateRows = () => true,
-    groups = {},
     shuffle = false,
-    minmaxRange = [0, 1],
     state = {},
-    customMinMaxRanges,
-    excludes = []
 }) => {
     let X = [];
 
     validateArray(arrObj, {min: 5}, 'parseProductionX')
     validateFirstRow(arrObj[0])
-    validateExcludes(arrObj[0], excludes)
 
     for (let x = 0; x < arrObj.length; x++) {
 
@@ -145,17 +133,20 @@ export const parseProductionX = ({
         X = arrayShuffle(X)
     }
 
+    const xLen = X.length
+    const flatX = new Array(xLen)
+    const configX = {
+        keyNames: xLen ? Object.keys(X[0]) : []
+    }
 
-    // Scale X
-    const {
-        scaledOutput: scaledX, 
-        scaledConfig: configX
-    } = scaleArrayObj({arrObj: X, repeat, groups, minmaxRange, customMinMaxRanges, excludes: new Set(excludes)})
-
+    for(let idx = 0; idx < xLen; idx++)
+    {
+        flatX[idx] = Object.values(X[idx])
+    }
 
     // Split into training and testing sets
     return {
-        X: scaledX,
+        X: flatX,
         configX,
     }
-};
+};

package/test/test.js

@@ -1,39 +1,34 @@
 import OHLCV_INDICATORS from 'ohlcv-indicators'
-import KNN from 'ml-knn'
-import { ConfusionMatrix } from 'ml-confusion-matrix';
-
 import { parseTrainingXY } from "../src/datasets.js"
-import {arrayToTimesteps} from '../src/timeSteps.js'
 import { loadFile } from "./fs.js"
-import * as tf from '@tensorflow/tfjs-node'
 
 const test = async () => {
 
-    const ohlcv = (await loadFile({fileName: 'btc-1d.json', pathName: 'datasets'}))
-
-    const indicators = new OHLCV_INDICATORS({input: ohlcv, ticker: 'BTC', precision: false})
+    const input = (await loadFile({fileName: 'btc-1d.json', pathName: 'datasets'}))
 
-    indicators
-        .rsi(40)
-        .bollingerBands(20, 2)
-        .ema(50)
-        .sma(200)
-        .sma(300)
-        .scaler(200, ['open', 'high', 'low', 'close'], {group: true})
-        .crossPairs([
-            {fast: 'rsi_40', slow: 30},
-            {fast: 'price', slow: 'sma_200'},
-            {fast: 'price', slow: 'sma_300'},
-            {fast: 'price', slow: 'bollinger_bands_upper'}
-        ])
+    const indicators = new OHLCV_INDICATORS({input, precision: false})
 
-    const parsedOhlcv = indicators.getData({dateFormat: 'milliseconds'})
+    const weights5 = {
+      ret_gap: [2, 2, 1.75, 1.5],
+      ret_change: [2, 2, 1.75, 1.5]
+    }
 
-    //console.log(parsedOhlcv.slice(-1))
+    const weights3 = {
+      ret_ema_5: [1.5, 1.5, 1.2],
+      ret_sma_200: [1.5, 1.5, 1.2], 
+      ret_atr_14_upper: [1.5, 1.5, 1.2], 
+    }
 
-    const {scaledGroups} = indicators
+    indicators
+      .volumeDelta()
+      .atr(14, {upper: 1})
+      .ema(5)
+      .sma(200)
+      .priceFeatures({colKeys: ['ema_5', 'sma_200', 'atr_14_upper']})
+      .scaler('zscore', 5, {group: true, lag: true, colKeys: ['ret_change', 'ret_upper_wick', 'ret_lower_wick', 'ret_gap', 'ret_body', 'ret_range'], weights: weights5})
+      .scaler('zscore', 3, {group: true, lag: true, colKeys: ['ret_change', 'ret_ema_5', 'ret_sma_200', 'ret_atr_14_upper'], weights: weights3})
 
-    //console.log(scaledGroups)
+    const arrObj = indicators.getData()
 
     const {
         trainX,
@@ -41,156 +36,58 @@ const test = async () => {
         testX,
         testY,
         configX,
-        keyNamesX,
     } = parseTrainingXY({
-        arrObj: parsedOhlcv,
+        arrObj,
         trainingSplit: 0.50,
         yCallbackFunc,
         xCallbackFunc,
         validateRows: ({objRow, index}) => {
+
             const curr = objRow[index]
             const prev = objRow[index - 1]
 
-            if(typeof prev === 'undefined') return false
+            if(typeof prev === 'undefined') return false //return false or null or undefined to continue to skip this row
+
+            if(!Number.isNaN(curr.sma_200) && !Number.isNaN(prev.sma_200)  && curr.sma_200 > prev.sma_200) return true //return true to include this row in dataset
 
-            return curr.ema_50 > curr.sma_300 && curr.sma_200 > curr.sma_300 && (curr.price_x_sma_300 === -1 || curr.price_x_sma_300 === 1)
+            return false
         },
         shuffle: true,
-        minmaxRange: [0, 1],
         balancing: null,
-        groups: scaledGroups,
-        excludes: ['high2'],
-        correlation: {corrExcludes: ['price_x_sma_300', 'price_x_sma_200']}
     });
 
-    console.log(configX.outputKeyNames)
-    console.log(configX.inputTypes)
-    //console.log(configX)
-
-    //console.log('trainX', trainX[0])
-
-
-
-/* 
-    tensorflowExample({
-        trainX,
-        trainY,
-        testX,
-        testY,
-        configX,
-        keyNamesX,
-    })
-
-    classifiersExample({
-        trainX,
-        trainY,
-        testX,
-        testY,
-        configX,
-        keyNamesX,
-    })
- */
- 
-}
-
-const classifiersExample = ({
-    trainX,
-    trainY,
-    testX,
-    testY,
-    configX,
-    keyNamesX,
-}) => {
-    const model = new KNN(trainX, trainY)
-
-    const predictions = model.predict(testX)
-    const compare = ConfusionMatrix.fromLabels(testY.flat(), predictions.flat())
-
-    //console.log(testY.flat(), predictions.flat())
-
-    console.log(compare.getAccuracy())
-}
-
-const tensorflowExample = ({
-    trainX,
-    trainY,
-    testX,
-    testY,
-    configX,
-    keyNamesX,
-}) => {
-
-    const timeSteps = 10
-    const colsX = trainX[0].length
-    const colsY = trainY[0].length
-    const timeSteppedTrainX = arrayToTimesteps(trainX, timeSteps)
-    const trimedTrainY = trainY.slice(timeSteps-1)
-
-    const inputX = tf.tensor3d(timeSteppedTrainX, [timeSteppedTrainX.length, timeSteps, colsX])
-    const targetY = tf.tensor2d(trimedTrainY,  [trimedTrainY.length, colsY])
-
-    //console.log('trainX', trainX)
-    //console.log('configX', configX)
-    //console.log('inputX', inputX)
-    //console.log('inputX', targetY)   
+    console.log(configX.keyNames)
+    console.log('row_1', {features: trainX[0], labels: trainY[0]})
 
+    console.log(trainY.length, trainX.length)
 }
 
-//callback function used to prepare X before scaling
+//callback function used to prepare X before flattening
 const xCallbackFunc = ({ objRow, index }) => {
     const curr = objRow[index]
-    const prev = objRow[index - 1]
 
-    //returning null or undefined will exclude current row X and Y from training
-    if(typeof prev === 'undefined') return null
-
-    //console.log(((curr.sma_300 - curr.low) / curr.low) * 100)
-
-    const output = {
-        high: curr.high,
-        ema50IsUp: curr.ema_50 > prev.ema_50,
-        ema50GtSma200: curr.ema_50 > curr.sma_200,
-        ema50GtSma300: curr.ema_50 > curr.sma_300,
-        sma200IsUp: curr.sma_200 > prev.sma_200,
-        sma200GtSma300: curr.sma_200 > prev.sma_300,
-        sma_300IsUp: curr.sma_300 > prev.sma_300
-    }
+    const output = {}
 
-    for(const [key, value]  of Object.entries(curr))
-    {
-        if(key.includes('minmax'))
-        {
-            output[key] = value
+    for(const [k, v] of Object.entries(curr)) {
+        if(k.startsWith('zscore_')) {
+            output[k] = v
         }
     }
 
-    return output
+    return output //returning null or undefined will exclude current row X and Y from training
 }
 
-//callback function used to prepare Y before scaling
+//callback function used to prepare Y before flattening
 const yCallbackFunc = ({ objRow, index }) => {
-    const curr = objRow[index]
-    const next = new Array(60).fill(0).map((_, i) => objRow[index + 1 + i])
-
-    //returning null or undefined will exclude current row X and Y from training
-    if (next.some(o => typeof o === 'undefined')) return null;
 
-    const priceTp = curr.sma_300 * 1.3
-    const entryPrice = curr.sma_300 * 0.96
-    const tp = next.some(o => o.close > entryPrice && (o.high > priceTp))
-    const sl = next.some(o => (o.low - entryPrice)/entryPrice < -0.10 && o.low < entryPrice)
+    const next = objRow[index + 1]
 
-    const highestHigh = Math.max(...next.map(o => o.high))
-    const lowestLow = Math.min(...next.slice(0, 5).map(o => o.low))
-
-
-    if(lowestLow > entryPrice) return null
-
-    //console.log([curr.date, (lowestLow - entryPrice)/entryPrice])
+    //returning null or undefined will exclude current row X and Y from training
+    if (typeof next === 'undefined') return null
     
     return {
-        result: Number(tp === true && sl === false)
+        result: Number(next.close > next.open)
     }
 }
 
-test()
+test()

package/src/scale.js

@@ -1,286 +0,0 @@
-export const scaleArrayObj = ({ arrObj, repeat = {}, minmaxRange, groups = {}, customMinMaxRanges = null, excludes = new Set() }) => {
-    
-    const arrObjClone = [...arrObj]
-    const arrObjLen = arrObjClone.length;
-    const firstRow = arrObjClone[0]
-    const validCustomMinMaxRanges = typeof customMinMaxRanges === 'object' && customMinMaxRanges !== null
-
-    if (arrObjLen === 0) {
-        return {
-            scaledOutput: [],
-            scaledConfig: {}
-        };
-    }
-    
-    const inputKeyNames = Object.keys(firstRow);
-
-    const repeatedKeyNames = inputKeyNames.map(key => {
-        return repeat.hasOwnProperty(key) ? Math.max(repeat[key], 1) : 1;
-    });
-
-    const countRepeatedKeyNames = repeatedKeyNames.reduce((sum, rep) => sum + rep, 0);
-
-    const config = {
-        arrObjLen,
-        rangeMin: minmaxRange[0], 
-        rangeMax: minmaxRange[1],
-        inputTypes: {},
-        min: {},
-        max: {},
-        groupMinMax: {},
-        repeat,
-        groups,
-        inputKeyNames,
-        outputKeyNames: new Array(countRepeatedKeyNames),
-        repeatedKeyNames
-    }
-        
-    let keyNamesIdx = 0;
-
-    for (let i = 0; i < config.inputKeyNames.length; i++) {
-        for (let w = 0; w < config.repeatedKeyNames[i]; w++) {
-            config.outputKeyNames[keyNamesIdx++] = config.inputKeyNames[i];
-        }
-    }
-
-    validateUniqueProperties(config.groups);
-
-    const validInputTypes = ['number', 'boolean']
-
-    for (const key of config.inputKeyNames) {
-
-        if(excludes.has(key))
-        {
-            config.inputTypes[key] = 'excluded'
-            continue
-        }
-
-        const firstType = typeof firstRow[key]
-        const thisGroup = findGroup(key, config.groups);
-
-
-        if(!validInputTypes.includes(firstType))
-        {
-            throw new Error(`Invalid input type "${firstType}" provided for key "${key}". Only accepting `)
-        }
-
-        config.inputTypes[key] = firstType;
-        
-        if(validCustomMinMaxRanges && customMinMaxRanges.hasOwnProperty(key))
-        {
-            if (thisGroup)
-            {
-                config.groupMinMax[thisGroup] = customMinMaxRanges[key]
-            }
-            else
-            {
-                config.min[key] = customMinMaxRanges[key].min;
-                config.max[key] = customMinMaxRanges[key].max;                   
-            }
-        }
-        else {
-            if (thisGroup) {
-                config.groupMinMax[thisGroup] = { min: Infinity, max: -Infinity };
-            } 
-            else {
-                config.min[key] = Infinity;
-                config.max[key] = -Infinity;
-            }
-        }
-    }
-
-    for (const obj of arrObjClone) {
-        for (const key of config.inputKeyNames) {
-
-            if (config.inputTypes[key] === 'excluded')
-            {
-                continue;
-            }
-
-            let value = obj[key];
-
-            if (config.inputTypes[key] === 'boolean') {
-                obj[key] = Number(value);
-            }
-
-            const thisGroup = findGroup(key, config.groups);
-
-            if(validCustomMinMaxRanges === false || (validCustomMinMaxRanges && !customMinMaxRanges.hasOwnProperty(key)))
-            {
-                if (thisGroup) {
-                    config.groupMinMax[thisGroup].min = Math.min(config.groupMinMax[thisGroup].min, value);
-                    config.groupMinMax[thisGroup].max = Math.max(config.groupMinMax[thisGroup].max, value);
-                } else {
-                    config.min[key] = Math.min(config.min[key], value);
-                    config.max[key] = Math.max(config.max[key], value);
-                }
-            }
-
-        }
-    }
-
-    const scaledOutput = new Array(arrObjLen);
-
-    
-    for (let i = 0; i < arrObjLen; i++) {
-        const obj = arrObjClone[i];
-        const scaledRow = new Array(config.outputKeyNames.length);
-        let idx = 0;
-
-        for (let j = 0; j < config.inputKeyNames.length; j++) {
-            const key = config.inputKeyNames[j]
-            const value = obj[key]
-
-            if (config.inputTypes[key] === 'excluded')
-            {
-                scaledRow[idx++] = value
-                continue
-            }
-
-            const thisGroup = findGroup(key, config.groups);
-            let minValue, maxValue
-
-            if (thisGroup) {
-                minValue = config.groupMinMax[thisGroup].min
-                maxValue = config.groupMinMax[thisGroup].max
-            } else {
-                minValue = config.min[key]
-                maxValue = config.max[key]
-            }
-
-            const scaledValue =
-                maxValue !== minValue
-                    ? config.rangeMin + ((value - minValue) / (maxValue - minValue)) * (config.rangeMax - config.rangeMin)
-                    : config.rangeMin;
-
-            const rep = config.repeatedKeyNames[j]
-
-            for (let w = 0; w < rep; w++) {
-                scaledRow[idx++] = scaledValue
-            }
-            
-        }
-        scaledOutput[i] = scaledRow
-    }
-
-    
-
-    return {
-        scaledOutput,
-        scaledConfig: config
-    }
-}
-
-
-const validateUniqueProperties = obj => {
-    const uniqueValues = new Set();
-    const allValues = [];
-
-    for (const [key, arr] of Object.entries(obj)) {
-        uniqueValues.add(key);
-        allValues.push(key);
-
-        arr.forEach(v => {
-            uniqueValues.add(v);
-            allValues.push(v);
-        });
-    }
-
-    if (uniqueValues.size !== allValues.length) {
-        throw new Error('Duplicate value found between properties in validateUniqueProperties function.');
-    }
-};
-
-const findGroup = (key, groups) => {
-    for (const [groupK, groupV] of Object.entries(groups)) {
-        if (groupV.includes(key)) {
-            return groupK;
-        }
-    }
-    return null;
-};
-
-
-const validateConfig = config => {
-
-    if(!config) return false 
-
-    const requiredKeys = [
-        "rangeMin",
-        "rangeMax",
-        "inputTypes",
-        "min",
-        "max",
-        "groupMinMax",
-        "repeat",
-        "groups",
-        "inputKeyNames",
-        "outputKeyNames",
-        "repeatedKeyNames"
-    ];
-
-    // Check for missing keys
-    for (const key of requiredKeys) {
-        if (!config.hasOwnProperty(key)) {
-            throw new Error(`Missing key "${key}" in config.`);
-        }
-    }
-
-    const { 
-        rangeMin, 
-        rangeMax, 
-        inputTypes, 
-        min, 
-        max, 
-        groupMinMax, 
-        repeat, 
-        groups,
-        inputKeyNames,
-        outputKeyNames,
-        repeatedKeyNames
-    } = config;
-
-    // Validate rangeMin and rangeMax are numbers and in proper order
-    if (typeof rangeMin !== 'number' || typeof rangeMax !== 'number') {
-        throw new Error("rangeMin and rangeMax must be numbers.");
-    }
-    if (rangeMin >= rangeMax) {
-        throw new Error("rangeMin must be less than rangeMax.");
-    }
-
-    // Helper to check if a value is a plain object (and not null or an array)
-    const isPlainObject = (obj) => typeof obj === 'object' && obj !== null && !Array.isArray(obj);
-
-    if (!isPlainObject(inputTypes)) {
-        throw new Error("inputTypes must be an object.");
-    }
-    if (!isPlainObject(min)) {
-        throw new Error("min must be an object.");
-    }
-    if (!isPlainObject(max)) {
-        throw new Error("max must be an object.");
-    }
-    if (!isPlainObject(groupMinMax)) {
-        throw new Error("groupMinMax must be an object.");
-    }
-    if (!isPlainObject(repeat)) {
-        throw new Error("repeat must be an object.");
-    }
-    if (!isPlainObject(groups)) {
-        throw new Error("groups must be an object.");
-    }
-    if(!Array.isArray(inputKeyNames))
-    {
-        throw new Error("inputKeyNames must be an array.");
-    }
-    if(!Array.isArray(outputKeyNames))
-    {
-        throw new Error("outputKeyNames must be an array.");
-    }
-    if(!Array.isArray(repeatedKeyNames))
-    {
-        throw new Error("repeatedKeyNames must be an array.");
-    }
-
-    return true;
-}