pyNIBS 0.2024.8__py3-none-any.whl

pynibs/data/network mapping configuration/output_documentation.md

@@ -0,0 +1,185 @@
+# Output Documentation
+
+This document provides an overview of the structure of the output CSV file generated by the network detection algorithm.
+The result file is created by the functions `write_nda_test_results_csv()` and `write_nda_application_results_csv()`.
+The output file contains the settings, detailed results of the analysis, and metadata. Below is a description of each column in the output file.
+In the end, a short description of a possible result validation process is included.
+
+## Application Results
+
+### Structure
+The output is organized as one row per analysis. Each row is structured as follows:
+- Experimental and configuration parameters: 15 entries (Content of the configuration file minus the last 8 entries, which are not relevant for analysis)
+- Result values: 19 entries
+- Additional Info and Metadata: 6 entries
+
+### Analysis
+The most important columns to make sense of the result are `network_type`, `found_idcs_0` and `found_idcs_1`, since this can tell us which ROI elements are involved and in what kind of interplay they have an effect.
+
+### Columns
+<u> Configuration: </u>
+
+- ``run``: Identifies the specific run or iteration of the analysis (hardcoded right now, default: 1)
+- ``bin_method``: See configuration guide.
+- ``bin_factor``: See configuration guide.
+- ``scoring_method``: See configuration guide.
+- ``scoring_thr``: See configuration guide.
+- ``scoring_interval``: See configuration guide.
+- ``acc_thr``: See configuration guide.
+- ``corr_thr``: See configuration guide.
+- ``note``: See configuration guide.
+- ``fn_flag``: See configuration guide.
+- ``subject_id``: See configuration guide.
+- ``response_specification``: See configuration guide.
+- ``efield_specification``: See configuration guide.
+
+<u> Result: </u>
+
+Let h0 and h1 denote the two found (potential) hotspots.
+- ``found_idcs_0``: ROI index of h0.
+- ``found_idcs_1``: ROI index of h1.
+- ``found_scores_0``: Hotspot score of h0. The higher, the better. Meaning and range depends on the scoring method:
+  - 'mi': mutual information score, (value > 0)
+  - 'clf': number of promising element combinations involving this element. (value < n_elms)
+  - 'regress_data': R2-value of the sigmoidal fit. (value in [0,1])
+  - 'regression': R2-value of the multivariable Gaussian fit. (value in [0,1])
+- ``found_scores_1``: Hotspot score of h1. (analogous to h0)
+- ``found_acc_0``: Accuracy of h0. (value in [0,1]) 
+Only differs from the hotspot score when using the CLF-method, in which case this denotes the decision tree classifier accuracy. (= how well the h0 e-field describes the TMS effect)
+- ``found_acc_1``: Accuracy of h1. (analogous to h1)
+- ``found_hotspots_corr``: Correlation of found hotspots. 
+(Measured as the Pearson correlation coefficient of their e-fields.) (value in [0,1])
+- ``found_hotspots_distance``: Distance between found hotspots. (Geodesic distance in mm)
+- ``found_network_type``: Type of network found, network IDs:
+    - (1) `NO`: No network ("pseudonetwork").
+    - (2) `AND`: Dual node network: Effect if elm0 AND elm1 are stimulated.
+    - (3) `1_INH_0`: Dual node network: elm1 inhibits elm0, elm0 has an effect.
+    - (4) `SH_0`: Single hotspot: Only elm0 has an effect.
+    - (5) `0_INH_1`: Dual node network: elm0 inhibits elm1, elm1 has an effect.
+    - (6) `SH_1`: Single hotspot: Only elm1 has an effect.
+    - (7) `XOR`: Dual node network: elm0 inhibits elm1, elm1 inhibits elm0. Both have an effect.
+    - (8) `OR`: Dual node network: Effect if either elm0 or elm1 is stimulated.
+- ``network_type_certainty``: Certainty of the network type identified. (value in [0,1])
+Measured as the maximal entry of the network identification vector divided by its second maximal entry. (How close was the call?)
+- ``network_type_vector_[0:8]``: Value of the network type vector for each network type. (in the code: `shape_vector`)
+The highest value belongs to the identified network.
+- ``sample_size``: The size of the sample analyzed. (n_zaps)
+
+<u> Additional: </u>
+
+- ``response_max``: Maximum response observed.
+- ``response_mean``: Mean response.
+- ``response_dev``: Standard deviation of the response.
+- ``runtime_gen``: Runtime for the data generation step.
+- ``runtime_scores``: Runtime for the scoring step.
+- ``runtime_eval``: Runtime for the evaluation step.
+
+
+## Testing Results
+
+### Structure
+The output is organized as one row per analysis. Each row is structured as follows:
+- Experimental and configuration parameters: 20 entries (Content of the configuration file minus the last 5 entries, which are not relevant for analysis)
+- Result values: 20 entries
+- Additional Info and Metadata: 6 entries
+
+### Analysis
+
+The most important columns to analyze performance are `network_eval` and `found_eval`. When both of them have the value 1 (True), the detection was completely successful. 
+
+### Columns
+<u> Configuration: </u>
+
+- ``run``: Identifies the specific run or iteration of the analysis (hardcoded right now, default: 1)
+- `hotspot_idcs_0` : See configuration guide.
+- `hotspot_idcs_1`	: See configuration guide.
+- `sample_size`	: See configuration guide.
+- `rn_seed`	: See configuration guide.
+- `dist_type`	: See configuration guide.
+- `network_type`	: See configuration guide.
+- `real_network_type`	: See configuration guide.
+- `effect_full`	: See configuration guide.
+- `effect_upper_bound`	: See configuration guide.
+- `jitter_ratio`	: See configuration guide.
+- `jitter_scale`: See configuration guide.
+- ``bin_method``: See configuration guide.
+- ``bin_factor``: See configuration guide.
+- ``scoring_method``: See configuration guide.
+- ``scoring_thr``: See configuration guide.
+- ``scoring_interval``: See configuration guide.
+- ``acc_thr``: See configuration guide.
+- ``corr_thr``: See configuration guide.
+- ``note``: See configuration guide.
+
+<u> Result: </u>
+
+Let h0 and h1 denote the two found (potential) hotspots.
+- ``found_idcs_0``: ROI index of h0.
+- ``found_idcs_1``: ROI index of h1.
+- ``found_scores_0``: Hotspot score of h0. The higher, the better. Meaning and range depends on the scoring method:
+  - 'mi': mutual information score, (value > 0)
+  - 'clf': number of promising element combinations involving this element. (value < n_elms)
+  - 'regress_data': R2-value of the sigmoidal fit. (value in [0,1])
+  - 'regression': R2-value of the multivariable Gaussian fit. (value in [0,1])
+- ``found_scores_1``: Hotspot score of h1. (analogous to h0)
+- ``found_acc_0``: Accuracy of h0. (value in [0,1]) 
+Only differs from the hotspot score when using the CLF-method, in which case this denotes the decision tree classifier accuracy. (= how well the h0 e-field describes the TMS effect)
+- ``found_acc_1``: Accuracy of h1. (analogous to h1)
+- ``found_hotspots_corr``: Correlation of found hotspots. 
+(Measured as the Pearson correlation coefficient of their e-fields.) (value in [0,1])
+- ``found_hotspots_distance``: Distance between found hotspots. (Geodesic distance in mm)
+- ``found_network_type``: Type of network found, network IDs:
+    - (1) `NO`: No network ("pseudonetwork").
+    - (2) `AND`: Dual node network: Effect if elm0 AND elm1 are stimulated.
+    - (3) `1_INH_0`: Dual node network: elm1 inhibits elm0, elm0 has an effect.
+    - (4) `SH_0`: Single hotspot: Only elm0 has an effect.
+    - (5) `0_INH_1`: Dual node network: elm0 inhibits elm1, elm1 has an effect.
+    - (6) `SH_1`: Single hotspot: Only elm1 has an effect.
+    - (7) `XOR`: Dual node network: elm0 inhibits elm1, elm1 inhibits elm0. Both have an effect.
+    - (8) `OR`: Dual node network: Effect if either elm0 or elm1 is stimulated.
+- ``network_type_certainty``: Certainty of the network type identified. (value in [0,1])
+Measured as the maximal entry of the network identification vector divided by its second maximal entry. (How close was the call?)
+- ``network_type_vector_[0:8]``: Value (int) of the network type vector for each network type. (in the code: `shape_vector`)
+The highest value belongs to the identified network.
+- ``identification_evaluation``: (boolean) Whether the network was identified correctly. (`real_network_type` == `found_network_type`)
+- ``localization_evaluation``: (boolean) Whether both hotspots were localized within tolerated area. (which is 10mm around the real hotspots)
+- `real_hotspot_dist`: Distance between real hotspots. (Geodesic distance in mm)
+- `real_hotspot_corr`: Correlation of real hotspots. 
+(Measured as the Pearson correlation coefficient of their e-fields.) (value in [0,1])
+- `real_hotspot_emax_0`:	The maximal e-field magnitude of real hotspot 0 over all trials. (As a proxy for stimulatability. The lower the e-mag max, the harder a hotspot is to detect.)
+- `real_hotspot_emax_1`:	The maximal e-field magnitude of real hotspot 1 over all trials.
+- `found_distance_0`: Distance between `found_hotspot_0` and its assigned real hotspot. (Geodesic distance in mm) (Assignment is done such that overall distances are minimized.)
+- `found_distance_1`: Distance between `found_hotspot_1` and its assigned real hotspot. (Geodesic distance in mm)
+- `num_hotspot-candidates`: The number of elements with a hotspot score > 0. (Only meaningful in `clf`-method - if even.)
+
+<u> Additional: </u>
+
+- ``response_max``: Maximum response observed.
+- ``response_mean``: Mean response.
+- ``response_dev``: Standard deviation of the response.
+- ``runtime_gen``: Runtime for the data generation step.
+- ``runtime_scores``: Runtime for the scoring step.
+- ``runtime_eval``: Runtime for the evaluation step.
+
+
+## Validation procedure 
+
+A challenging part of validation is finding a response measure and according ROI with a reasonable assumption about a network effect 
+being in place.
+Assuming that is the case, validation of NDA results in a second session could be done as follows:
+
+Based on all known e-field data `e_matrix` from the first session and the detection result, an effect map can be created using the function 
+`write_effect_map_hdf5()`.
+To create that map, for every ROI element of interest (`elm_i`), an expected effect value is estimated: 
+- Of all available stimulations/zaps, the one 
+inducing the highest e-field value on element `elm_i` is chosen. 
+- For that zap, the e-field magnitudes of the hotspot elements are taken into consideration and using the network type and 
+its effect function (see `create_artificial_response_data()`), the expected effect for that zap is computed. 
+(E.g. Assuming the hotspot elements are stimulated at this certain value, and form the network type OR, how high is the effect on the response expected to be?)
+- This effect value is assigned to the ROI element `elm_i`. 
+
+Applying this procedure to each element leads to an initial ROI effect map, on which stimulation points of interest can be chosen (e.g. maxima, mimima,...)
+For those specific stimulation configurations, a more precise effect value can be computed, since the initial map was 
+just a proxy (because the needed stimulation positions may not have been included in the first session.). 
+After measuring the response values for these stimulation points of interest, their values and deviation from the 
+baseline response can be compared to the predicted effect values. 

pynibs/data/network mapping configuration/recommendations_for_accuracy_threshold.md

@@ -0,0 +1,77 @@
+## What is the accuracy threshold?
+
+Lower bound for the accuracy (clf) or scores (other scoring methods) of hotspot elements. Below that, 
+an element will not be categorized as hotspot.
+
+## How to find out a threshold that makes sense?
+A good source of knowledge to go on is the accuracy/score of a pseudonetwork (NO effect) of comparable parameters.
+Accuracy mostly depends on the given sample size, while also being influenced by the noise level and effect strength. 
+The higher the threshold, the higher the specificity of the detection - but also the higher the chance NO network is
+detected, even when there could be one.
+
+
+## CLF-accuracies per sample size
+Using the decision tree classification scores (`clf`), the recommended accuracy thresholds are listed in the table below.
+The results were obtained by running 50 pseudonetwork trials (NO effect) with the `network_detection_algorithm_testing()`, for each sample size 
+in np.range(20,1000,20). The corresponding data is in `/data/pt_01756/studies/network_mapping/testing_NDA/15484.08/pseudonetwork_data.csv`. 
+
+The given recommendations represent the highest achieved accuracies of the tested pseudonetworks per sample size.
+To conduct a more elaborate analysis use `appprox_optimal_acc_threshold()`  in ``result_analysis.py``.
+
+
+| Sample Size | Accuracy Threshold |
+|-------------|--------------------|
+| 20          | 1                  |
+| 40          | 0.93               |
+| 60          | 0.87               |
+| 80          | 0.81               |
+| 100         | 0.77               |
+| 120         | 0.77               |
+| 140         | 0.74               |
+| 160         | 0.73               |
+| 180         | 0.72               |
+| 200         | 0.71               |
+| 220         | 0.71               |
+| 240         | 0.7                |
+| 260         | 0.69               |
+| 280         | 0.69               |
+| 300         | 0.69               |
+| 320         | 0.7                |
+| 340         | 0.67               |
+| 360         | 0.67               |
+| 380         | 0.67               |
+| 400         | 0.65               |
+| 420         | 0.65               |
+| 440         | 0.64               |
+| 460         | 0.64               |
+| 480         | 0.64               |
+| 500         | 0.63               |
+| 520         | 0.63               |
+| 540         | 0.63               |
+| 560         | 0.63               |
+| 580         | 0.63               |
+| 600         | 0.62               |
+| 620         | 0.63               |
+| 640         | 0.62               |
+| 660         | 0.62               |
+| 680         | 0.62               |
+| 700         | 0.61               |
+| 720         | 0.61               |
+| 740         | 0.61               |
+| 760         | 0.62               |
+| 780         | 0.61               |
+| 800         | 0.61               |
+| 820         | 0.61               |
+| 840         | 0.61               |
+| 860         | 0.61               |
+| 880         | 0.6                |
+| 900         | 0.6                |
+| 920         | 0.6                |
+| 940         | 0.6                |
+| 960         | 0.6                |
+| 980         | 0.6                |
+| 1000        | 0.6                |
+
+## R2 scores
+No proper analysis has been conducted, but they tend to be much lower than clf-accuracies. 
+Values of about ``0.15`` have proven sufficient.