knnopt 1.0.2__tar.gz

knnopt-1.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jianbo Huang (黄建波: jianbo.huang22@gmail.com)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

knnopt-1.0.2/MANIFEST.in

@@ -0,0 +1,7 @@
+include README.md
+include LICENSE
+recursive-include recursive-include examples *
+
+
+# Exclude Python cache files from sdist
+global-exclude __pycache__ *.py[cod]

knnopt-1.0.2/PKG-INFO

@@ -0,0 +1,629 @@
+Metadata-Version: 2.4
+Name: knnopt
+Version: 1.0.2
+Summary: Optimized KNN Regressor/Classifier with NLopt weight optimization and CSV file process utilities
+Author: Jianbo Huang
+Author-email: jianbo.huang22@gmail.com
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: scikit-learn
+Requires-Dist: tabulate
+Requires-Dist: joblib
+Requires-Dist: nlopt
+Dynamic: author
+Dynamic: author-email
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# knnopt
+
+**Optimized KNN Regressor/Classifier with NLopt weight optimization**
+
+`knnopt` is a Python package that extends the standard K-Nearest Neighbors (KNN) algorithm by automatically optimizing **feature-target weights** 
+using the [NLopt](https://nlopt.readthedocs.io/en/latest/) library.  
+This allows KNN models to adapt feature importance for each target variable, improving accuracy and robustness.
+Test and validation cases show that it has better accuracy than other commonly-used machine-learning
+models for most of the test cases, while on other cases, its accuracy has similar level
+with other models. Our validation cases show that, it does not have over-fitting issues. Test and
+validation accuracy is consistent across different cases, and performs well for cases
+with big sample records such as 10000 records, as well as cases with small number of records
+around 100. 
+
+This package also contain a number of utilities for handling CSV data file. They are designed to help
+with general data science and machine learning tasks.
+
+---
+
+## 🚀 Features
+
+- **Supports Regression and Classification**
+- **Automatic Feature Weight Optimization** per target using NLopt
+- **Scales Features** with `MinMaxScaler` for consistent distance metrics
+- **Saves Models and Scalers** for later prediction
+- **Performance Tracking** via `performance.csv`
+- **Customizable KNN Parameters** (`n_neighbors`, `weights`, `metric`)
+- **Choice of NLopt Optimizers** (e.g., `LN_COBYLA`, `LD_MMA`, etc.)
+- **CSV utilities** a set of utilities for handling CSV data, useful for data analysis and machine learning tasks
+
+---
+
+## 🛠 Technology Overview
+
+This model is implemented as a Python class KnnOpt. It consists of methods for loading training data,
+scale dataset, optimization objective function, optimization model, feature-target correlation,
+train model, performance metrics calculation, targets prediction with new, untrained feature records,
+model validation with untrained dataset which contains both features and targets columns.
+
+Apart from the class implementation, this package also contains three CLI utility files for model training, 
+prediction and validation, and a set of utilities for CSV file handling and training data preperation.
+The knnopt-model-related utilities are:
+
+```
+cli_predict.py  cli_train.py  cli_validate.py 
+```
+These utilities allow us to train, validate KnnOpt model, and use the trained model for targets
+prediction using commands run on terminals. This approach simplifies the usage of machine-learning
+models, especially helpful for researchers/engineers who do not have much experiences in 
+data science and computer languages. 
+
+This package also includes utilites for preparing CSV data files prior to train a model.
+They can be used for any data analysis and machine-learning jobs involving CSV data. The utility files are:
+
+
+```combine_csv.py  csv_diff.py  data_processing.py  get_dataset_cls.py  
+   get_dataset.py  pearson_coeffs.py  split_csv_by_row.py  split_csv.py  
+   train_validate_split.py
+```
+
+## KnnOpt Model creation
+
+The syntax to create a KnnOpt model is:
+
+```python
+  model = KnnOpt(
+        features=['f0','f1','f2'...], # feature-column names list
+        targets=['t0','t1','t2'...], # target-column names list
+        model_type='reg', # cls
+        n_neighbors=5, #any suitable int >= 1, 3 to 7 are suitable choices
+        weights='distance', #uniform
+        metric='euclidean', #manhattan
+        optimizer='LN_COBYLA' #LN_BOBYQA, GN_DIRECT
+    )
+```
+The most influential parameter is: n_neighbors. The weights, metric and optimizer default values are suitable for majority cases.
+
+## Train KnnOpt Model
+
+Once KnnOpt model instance is created, the next step is to train the model. The syntax is:
+
+```python
+model.train(csv_path, test_size=0.3)
+
+```
+where csv_path is the absolute path to the training data file, test_size is fraction of data used for model testing.
+Note that:
+
+- All the model-related data files, results files, weights, pipeline file will be saved in the same directory as the training data file.
+- test_size has a substantial effect on model accuracy. A value range of [0.2, 0.35] is a suitable choice.
+- The train method will automatically call data pre-processing method to scale the data
+- All data columns in the training data file needs to contain numerical data only
+- If your CSV data file contain categorical data, and/or missing data, you can use the data_processing utility to pre-process the data file first.
+- The test accuracy or test error metrics are the real measurement of model accuracy. High train accuracy does not mean the model
+is accurate, because the target values for evaluation are used in training the model.
+- In evaluating the test accuracy/error, the target values are not used for model training, so it is a real representation of the
+model performance.
+- During training, the weights of feature(i) to target(j), W(i,j) is optimized in such a way that the test error becomes minimized.
+The optimization procedure is automatically called, and the final weights array W is used to fit the model.
+- The feature-target weight can be seen as the importance factor of a feature influencing a target, therefore, optimized weights will
+help KNN model to find the best match between a new record and the neighbouring records, thus increase prediction accuracy. 
+- The training process will create and store 4 files in the case directory:
+```
+ knn_models.pkl  scaler.pkl   weights.npy   performance.csv 
+```
+
+## Workflow of the model training process
+
+1. Load CSV containing features and targets.
+1. Scale Features using MinMaxScaler.
+1. Split Data into train/test sets.
+1. Optimize Feature Weights for each target using NLopt:
+1. Minimizes RMSE for regression. Maximizes accuracy/F1 for classification.
+1. Train KNN Models with optimized weights.
+1. Save Outputs in case directory:
+- **scaler.pkl** — feature scaler
+- **weights.npy** — optimized weights
+- **knn_models.pkl** — trained KNN model
+- **performance.csv** — metrics for each target
+
+
+## Validate KnnOpt Model
+
+The test accuracy is a good indication of machine-learning model performance, but it does not ensure the prediction will
+have the same accuracy due to potential parameter overfitting issue. To check whether a KnnOpt model is overfitted, a validation method is created. 
+The idea is that we keep a fraction of the sample data which contains both features and targets out of the train-test loop, therefore it is
+true representation of model prediction except that we know the true targets. To validate KnnOpt model against a given dataset, the procedure is:
+
+- Split the whole data into training data and validation data, using Scikit-learn train-test split function. You can control the fraction of the data
+to be used for validation and randomly mix up the data for a fair validation.
+- The remaining data  train data can then be fed into a KnnOpt model for training.
+- This process can be done by using this packages utility code: **train_validate_split.py**
+- The process will create two files in the case directory, while the original data file is unchanged:
+```
+train_data.csv    validation_input.csv    
+```
+- To validate the trained model, you can either use the validation utility in cli_validate.py or use the class method
+model.validate(input_csv), where input_csv is the filename which contains validation input data. The file must already be in the
+case directory
+
+## KnnOpt class methods
+
+The KnnOpt class contains the following methods:
+
+- **load_train_data**: load training dataset in given CSV file. The data file contains features columns, targets columns, and can contain
+   other columns not used for training/prediction. It also scale the data using Scikit-lean MinMaxScaler() function
+- **data_split**: split the loaded data into train and test parts
+- **objective**: objective function to minimise the test error
+- **optimize_model**: Procedure to get the optimized weights
+- **correlation**: helper method to get correlation coefficients of all features with respect to targets[t_idx]
+- **finalize_regressor**: calculate and save the performance metrics for regressor
+- **finalize_classifier**: calculate and save the performance metrics for classifier
+- **train**: scale training data, train the model, optimize weights, save model file, scalar file and weights file in case directory
+- **predict**: use the trained model to predict targets based on unseen features data
+- **validate**: use untrained input file which contains both features and targets and the trained model to predict
+targets, find the difference between true values and predicted values, output the relative errors, prediction
+accuracy to case directory. This is an effective tool to check whether the model overfits the training data.
+
+## Utilities
+
+1. Train KnnOpt model **cli_train.py**: CLI command to train a KnnOpt model. 
+
+- Usage example for regressor
+
+Under a terminal, type the following command:
+```
+cd ~/knnOptModel/knnopt/examples/metal_strength
+knn-train train_data.csv -f f0,f1,f2,f3,f4 -t target1,target2,target3,target4 -n 7 --test-size 0.2
+```
+You can also put the command in a train.sh file, make it executable, go to the directory, then under terminal window,
+type:
+```
+train.sh
+```
+Note that model_type default is 'reg', therefore for regression application, the --model-type option is not needed.
+
+- Usage example for classifier
+
+Under a terminal type the following command:
+
+```
+knnopt-train train_data.csv -f f0,f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11 \
+-t medical_event --model-type cls  -n  7 --test-size 0.25
+```
+
+Note that the 
+
+```
+--model-type cls 
+```
+is needed, as classifier is not the default model type. Put the command in a train.sh file is a better choice for reuse
+and for keeping record.
+
+2. Validate KnnOpt model **cli_validate.py**: CLI command to validate a trained KnnOpt model. 
+- Usage example for regressor: the bash file validate.sh for command is as follows
+
+```
+knnopt-validate reg ./ validate_input.csv
+```
+where ./ is the directory containing the validation input file. You can use absolute path name
+and any filename (must be with extension .csv) for the validation input file.
+
+- Usage example for classifier: the bash file validate.sh for command is as follows
+
+```
+knnopt-validate cls ./ validate_input.csv
+```
+where ./ is the directory containing the validation input file. You can use absolute path name
+and any filename (must be with extension .csv) for the validation input file.
+
+3. KnnOpt model prediction **cli_predict.py**: CLI command to predict the targets of giving features data, by using a trained KnnOpt model. 
+- Usage example for regressor: the bash file predict.sh for command is as follows
+
+
+```
+knnopt-predict reg input.csv
+```
+
+where reg is the model_type, input.csv is input data path
+
+
+- Usage example for classifier: the bash file predict.sh for command is as follows
+
+
+```
+knnopt-predict cls input.csv
+```
+
+where cls is the model_type, input.csv is input data path
+
+4. combine-csv **utils/combine_csv.py**: combine two CSV files the same number of rows but
+different column names. 
+
+- Usage:
+```
+combine-csv file1.csv file2.csv
+```
+- Both files must contain a header and have the same number of rows
+- Combined CSV data will be saved as combined_csv.csv, and in the same directory as file1.csv
+- Duplicated column names in file2.csv will be ignored
+- This utility is useful to combine feature data and target data, if they are originally not in the same file
+- It is useful for any data analytics, not specific to KnnOpt model
+
+
+5. csv-diff **utile/csv_diff.py**: find data difference between two CSV files. 
+
+- Usage example:
+
+```
+csv-diff file1 file2 --columns f1,f2,f3,f4 --eps 1.0e-4
+```
+
+where file1 and file2 are the paths of the two CSV files, columns is a string containing comma-separated column names to find the differences with.
+The two files may contain other columns, but names in **--columns** must be exist in both files. Both files must have the same number of rows.
+eps is a small number to prevent division-by-zero. The cell-wise absolute difference is normalized by the mean of the associated column.
+If **--columns** is missing, all columns will be compared.
+
+
+6. data-process **utile/data_processing.py**: CSV preprocessing utility: auto missing data handling, categorical/boolean to numeric, float rounding 
+
+- Usage example:
+
+```
+data-process input_csv -s skip-columns -e extra-category -m missing-value
+```
+
+where input_csv is the file path to the csv file to be processed. All processed data will be written in the same directory as the input_csv.
+skip-columns is a string of comma-separated names, those columns will not be handled. The utility will automatically identify category data columns,
+However, if some data columns seemingly numerical such as postcodes, but actually category, you can use the -e option to define them.
+The -s option missing-value is for defining missing value cells in the dataset, such as NA, N/A, nan, or empty string, default is empty string("").
+Processed data will be saved into file processed.csv.
+
+7. dataset-cls **utile/get_dataset_cls.py**: get CSV data from Scikit-learn  classification datasets and save as CSV files. 
+
+- Usage example:
+
+```
+dataset-cls dataset_name -c case_directory -s -r records -n noise
+```
+
+where: dataset_name includes iris, wine, breast_cancer, synthetic. -s means the data rows are randomly shuffled for model training.
+case_directory default to current working directory. records(default 500) and noise(default 0.1) are only for synthetic
+dataset.
+
+- Output data file: output data will be saved as full_data.csv, in case_directory
+
+
+
+8. dataset-reg **utile/get_dataset_reg.py**: get CSV data from Scikit-learn  regression datasets and save as CSV files. 
+
+- Usage example:
+
+```
+dataset-reg dataset_name -c case_directory -f nfeatures -s -r records -n noise
+```
+
+where: dataset_name includes wine, california_housing, synthetic. -s means the data rows are randomly shuffled for model training.
+case_directory default to current working directory. nfeatures(default 10), records(default 1000) and noise(default 0.1) are only for synthetic
+dataset.  
+
+- Output data file: output data will be saved as full_data.csv, in case_directory
+
+
+9. correlation **utile/pearson_coeffs.py**: calculates correlation coefficients of each feature column to a target column. 
+
+- Usage example:
+
+```
+correlation csv_path targets -s skip_columns
+```
+
+where: csv_path is the data file to be handled, targets is a comma-separated string containing the target names. skip_columns
+is a comma-separated string containing names of columns in the data file to be skipped for correlation.
+
+- Output data file: output data files are: correlation_coefficients.csv and normalized_coefficients.csv. They will be saved in the same directory as csv_path file.
+
+
+10. split-csv **utile/split_csv.py**: split a CSV file into two CSV files based on specified columns.
+
+- Usage example:
+```
+split-csv csv_path file1_name file2_name file2_columns -c common_columns
+```
+
+where:
+- **csv_path** : path to the input CSV file
+- **file1_name**: name of the first output CSV file
+- **file2_name**: name of the second output CSV file
+- **file2_columns** : comma-separated string containing column names for file2.csv
+- **common_columns**: Optional comma-separated list of columns to include in both files
+
+This utility is useful for spliting a file into feature file and target file.
+
+
+11. split-csv-rows **utile/split_csv_by_row.py**: split a CSV file into two files by randomly selecting rows.
+
+- Usage example:
+```
+split-csv-rows csv_path file1_name file2_name -r file2_ratio
+```
+
+where:
+- **csv_path** : path to the input CSV file
+- **file1_name**: name of the first output CSV file
+- **file2_name**: name of the second output CSV file
+- **file2_ratio**: ratio of rows to put into file2.csv (0 < ratio < 1, default 0.1)
+
+This utility is useful for spliting a file into train file and validation file.
+
+
+12. train-validate-split **utile/train_validate_split.py**: split a CSV file into train file and validate file.
+
+- Usage example:
+```
+train-validate-split csv_path -r validate_size
+```
+
+where:
+- **csv_path** : path to the input CSV file
+
+- **validate_size**: fraction of data for validation (0 < ratio < 1, default 0.1)
+
+- The splitted data will be saved into train_data.csv and validation_input.csv, stored in the same
+directory as the csv_path.
+
+This utility is useful for spliting a file into train file and validation file.
+
+
+## Examples
+
+A number of examples are provided in the examples/ folder. There are two types of examples:
+1. Examples using datasets provided by scikit-learn, for create and run those examples, do:
+```
+cd knnOptModel/knnopt/examples
+python3 knnopt_validate.py
+```
+
+There are 7 cases in this category: 4 classification cases:  breast_cancer, wine, iris, synthetic_classification and 3
+regression cases: diabetes, california_housing, synthetic_regression
+
+2. Datasets provided by this package
+
+The package includes 5 additional test and validation cases: 3 regression cases: metal_strength, heat_tr_coef, matprop,
+2 classification cases: medical_event, claim_predict.
+
+To run those cases, cd to the a case, then type ./run.sh. You may change the parameters in the run.sh file
+to see the effect of each parameter.
+
+
+
+## Installation
+
+To install this package, you can download or clone the source package. Assuming the package is stored
+in the $HOME directory, then under terminal, do:
+
+```
+cd ~/knnOptModel
+pip install .
+```
+
+
+📜 License
+This project is licensed under the MIT License — see the LICENSE file for details.
+
+🤝 Contributing
+Pull requests are welcome!
+If you have suggestions for optimizers, metrics, or additional features, feel free to open an issue.
+
+
+## 📊 KnnOpt accuracy comparison with other models
+
+### Case name: diabetes regression
+This case uses sklearn diabetes dataset. All models use test_size: 0.25, random_state: 42
+
+-----------------------------------------
+| Model name | Target Name| Test Rel MAE% |Validate Rel MAE% | Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     |  target |23.67% | 25.52% |n_neighbours=5, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    |  target |26.36% | 26.03% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVR        |  target |24.50% | 27.50% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest| target |25.07% | 24.38% |n_estimators=300,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP         | target |23.96% | 27.46% |hidden_layer_sizes: [50, 30, 20], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn       |  target |26.25% | 27.15% |n_neighbours=5, weights: distance, metric: euclidean|
+
+
+
+### Case name: california_housing regression
+It uses sklearn california_housing dataset. All models use test_size: 0.25, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Rel MAE% |Validate Rel MAE | Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     |  target |13.82% | 13.57% |n_neighbours=5, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    |  target |15.42% | 15.05% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVR        |  target |23.69% | 24.37% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest| target |22.38% | 22.24% |n_estimators=300,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP         | target |20.64% | 20.69% |hidden_layer_sizes: [50, 30, 20], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn       |  target |21.93% | 22.40% |n_neighbours=5, weights: distance, metric: euclidean|
+
+
+
+### Case name: heat_tr_coef regression
+
+The dataset comes from CFD simulation of heat transfer coefficients of two-dimensional rectangular objects
+of different dimensions and flow configs. The purpose here is to use the dataset to validate different models.
+All models use test_size: 0.3, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Rel MAE% |Validate Rel MAE | Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     |  hcoef |7.18% | 8.22% |n_neighbours=5, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    |  hcoef |9.24% | 8.11% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVR        |  hcoef  |5.61% | 7.28% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest| hcoef |10.81% | 8.21% |n_estimators=300,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP         | hcoef  |33.38% | 32.46% |hidden_layer_sizes: [50, 30, 20], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn       |  hcoef |14.76% | 10.81% |n_neighbours=5, weights: distance, metric: euclidean|
+
+
+
+### Case name: metal_strength regression
+
+The dataset correlates metal strength properties with different operational treatment conditions.
+There are 4 targets in this case. The physics background of the dataset is obsecured, but the data relationships are kept for model
+validations. All models use test_size: 0.2, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Rel MAE% |Validate Rel MAE | Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     |t1,t2,t3,t4 |6.57.2.02,5.10,8.63% |6.51,2.28,11.31,12.08% |n_neighbours=3, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | t1,t2,t3,t4 |7.84,2.09,6.2,8.19% | 7.99,1.87,11.11,12.42% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVR        |  t1,t2,t3,t4  |18.96,2.65,11.23,15.02% |20.96,3.29,15.23,16.91% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest| t1,t2,t3,t4 |6.84,1.99,5.70,8.18% |6.82,2.09,8.62,11.68% |n_estimators=300,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP         | t1,t2,t3,t4 |17.74,14.24,26.39,28.59% |18.81,7.43,31.58,18.72% |hidden_layer_sizes: [50, 30, 20], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn       |  t1,t2,t3,t4  |10.14,2.13,8.71,11.10% | 8.51,2.80,12.05,10.93% |n_neighbours=5, weights: distance, metric: euclidean|
+
+### Case name: synthetic regression
+
+The dataset is generated using sklearn make_regression function with some post transformation. All models use test_size: 0.3,
+random_state: 42, total 1000 records are generated, of which 10% is used for validation and 90% for train/test, and 30% of the remaining data is for test.
+The target is of normal distribution with mean=10, std=1, noise level 10%.
+
+
+-----------------------------------------
+| Model name | Target Name| Test Rel MAE% |Validate Rel MAE | Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     |target |2.58% |2.75% |n_neighbours=3, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | target |2.13% | 2.11% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVR        |  target |0.05% |0.06% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest| target |3.53% |3.46% |n_estimators=300,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP         |target |0.31% |0.29% |hidden_layer_sizes: [50, 30, 20], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn       |  target |3.76 | 3.92% |n_neighbours=3, weights: distance, metric: euclidean|
+
+
+### Case name: product quality reqression
+
+This case correlates the product quality parameters with its raw material composition. All  models use test_size=0.3.
+4 product characteristics are chosen for comparing the model prediction capabilities: size, strength, component6, react_rate1
+
+
+-----------------------------------------
+| Model name | Target Name| Test Rel MAE% |Validate Rel MAE | Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     | size, strength, component6, react_rate1|1.87,0.57,2.81,2.47% |3.49,0.67,1.88,3.18% |n_neighbours=3, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | size, strength, component6, react_rate1|5.07,3.51,3.57,3.15% | 5.93,2.43,2.59,3.31% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVR        | size, strength, component6, react_rate1 |3.65,0.75,3.68,3.15% |3.12,0.79,2.86,3.15% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest|size, strength, component6, react_rate1 |4.96,3.48,3.36,2.65% |7.04,4.31,2.55,3.28% |n_estimators=200,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP         |size, strength, component6, react_rate1|8.12,6.33,11.87,6.52% |5.22,8.61,11.62,7.57% |hidden_layer_sizes: [100], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn       |  size, strength, component6, react_rate1|2.29,0.68,3.57,3.39% |3.59,0.69,2.05,3.29% |n_neighbours=3, weights: distance, metric: euclidean|
+
+
+### Case name: breast_cancer classfication
+
+
+This case uses sklearn breast_cancer dataset.  All  models use test_size=0.3, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Accuracy% |Validate Accuracy| Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     | target|98.72% |100% |n_neighbours=5, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | target|97.44% | 100% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVC       | target |98.72% |98.25% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest|target |95.12% |96.49% |n_estimators=200,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP        |target|98.37% |98.25% |hidden_layer_sizes: [100], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn        | target|96.75% |94.74% |n_neighbours=3, weights: distance, metric: euclidean|
+
+
+
+### Case name: iris classification
+
+
+This case uses sklearn iris dataset.  All  models use test_size=0.3, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Accuracy% |Validate Accuracy| Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     | target|100% |100% |n_neighbours=5, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | target|97.50% | 100% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVC       | target |96.97% |100% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest|target |84.85% |100% |n_estimators=200,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP        |target|96.97% |93.33% |hidden_layer_sizes: [100], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn        | target|87.88% |100% |n_neighbours=3, weights: distance, metric: euclidean|
+
+
+
+
+### Case name: wine classification
+
+This case uses sklearn wine dataset.  All  models use test_size=0.3, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Accuracy% |Validate Accuracy| Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     | target|100% |100% |n_neighbours=4, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | target|97.92% | 100% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVC       | target |87.18% |100% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest|target |97.44% |100% |n_estimators=200,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP        |target|92.31% |100% |hidden_layer_sizes: [100], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn        | target|89.74% |94.44% |n_neighbours=4, weights: distance, metric: euclidean|
+
+
+### Case name: synthetic_classification 
+
+ It uses sklearn make_classification to generate the dtaset.  1000 records are generated,
+of which 900 is for train/test the model, and 100 for validate the model. There are 10 features and 1 target in this case. 
+All  models use test_size=0.2, random_state: 42
+
+
+-----------------------------------------
+| Model name | Target Name| Test Accuracy% |Validate Accuracy| Model parameters|
+|------------|---------|----------|----------------|----------------|
+| KnnOpt     | target|92.22% |85% |n_neighbours=7, weights: distance, metric: euclidean, optimizer: LN_COBYLA|
+| XGBoost    | target|93.33% | 84% |n_estimators=300,learning_rate=0.05,max_depth=6,subsample=0.8,colsample_bytree=0.8|
+| SVC       | target |88.89% |81% |C=200, gamma=0.05,epsilon=0.01,Kernel=RBF|
+| RandomForest|target |92.78% |85% |n_estimators=200,max_depth=6,min_samples_split=2,min_samples_leaf=1|
+| MLP        |target|88.89% |81% |hidden_layer_sizes: [100], activation: relu, solver: adam, alpha: 0.0001, learning_rate: constant, max_iter: 600|
+| Knn        | target|83.33% |80% |n_neighbours=4, weights: distance, metric: euclidean|
+
+
+
+## Concluding remarks
+
+- KnnOpt model consistently shows  stable and mostly better accuracy compare to other models both reegression and classification cases.
+- The consistently good performance is because KnnOpt introduces a weights-based importance
+factor to measure a feature with regard to a target. The weights matrix is then optimized to minimize the 
+RMAE error for unseen dataset in the training process. If the unseen dataset covers a wide range of 
+potential new data in prediction, it is then able to give good accuracy
+- The accuracy of the model depends on how close of a new data record is from the train/test data. If a data 
+record is too far away from the dataset used to train/test the model, the prediction accuracy will drop.
+
+
+
+
+
+
+
+
+