dimx 1.0.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dimx-1.0.5/LICENSE +15 -0
- dimx-1.0.5/PKG-INFO +200 -0
- dimx-1.0.5/README.md +178 -0
- dimx-1.0.5/data/Fly80XY_norm_1061.csv +1062 -0
- dimx-1.0.5/dimx/CLI_Parser.py +234 -0
- dimx-1.0.5/dimx/CrossMapColumns.py +233 -0
- dimx-1.0.5/dimx/Evaluate.py +704 -0
- dimx-1.0.5/dimx/MDE.py +359 -0
- dimx-1.0.5/dimx/ManifoldDimExpand.py +21 -0
- dimx-1.0.5/dimx/Run.py +294 -0
- dimx-1.0.5/dimx/__init__.py +9 -0
- dimx-1.0.5/example/.ipynb_checkpoints/MDE_Fly_Example-checkpoint.ipynb +314 -0
- dimx-1.0.5/example/Evaluate_Fly.png +0 -0
- dimx-1.0.5/example/MDE_Fly.png +0 -0
- dimx-1.0.5/example/MDE_Fly_Example.ipynb +314 -0
- dimx-1.0.5/pyproject.toml +54 -0
- dimx-1.0.5/tests/.ipynb_checkpoints/MDE_test-checkpoint.ipynb +100 -0
- dimx-1.0.5/tests/MDE_test.ipynb +100 -0
- dimx-1.0.5/tests/tests.py +121 -0
dimx-1.0.5/LICENSE
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
Manifold Dimensional Expansion Non-commercial License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Okinawa Institute of Science and Technology Graduate University. All rights reserved.
|
|
4
|
+
|
|
5
|
+
Redistribution and use for non-commercial purposes in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
|
6
|
+
|
|
7
|
+
1. The software is used solely for non-commercial purposes. It may not be used indirectly for commercial use, including operation on or for a website or service that receives advertising, sponsorship, or other revenue. Results generated by or derived from use of the Software may not be used for commercial purposes without a separate commercial license. For commercial use rights, contact Okinawa Institute of Science and Technology, OIST Innovation, at tls@oist.jp.
|
|
8
|
+
|
|
9
|
+
2. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers.
|
|
10
|
+
|
|
11
|
+
3. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided with the distribution.
|
|
12
|
+
|
|
13
|
+
4. Neither the name of the Okinawa Institute of Science and Technology Graduate University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
|
14
|
+
|
|
15
|
+
THIS SOFTWARE IS PROVIDED BY THE OKINAWA INSTITUTE OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT OF PATENTS, TRADEMARKS, OR OTHER PROPRIETARY RIGHTS, ARE DISCLAIMED. NO PATENT RIGHTS ARE GRANTED, WHETHER EXPRESSLY, BY IMPLICATION, ESTOPPEL, OR OTHERWISE. IN NO EVENT SHALL THE OKINAWA INSTITUTE OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
dimx-1.0.5/PKG-INFO
ADDED
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dimx
|
|
3
|
+
Version: 1.0.5
|
|
4
|
+
Summary: Manifold Dimensional Expansion
|
|
5
|
+
Author-Email: Joseph Park <JosephPark@IEEE.org>, Gerald Pao <Gerald.Pao@OIST.jp>, Temma Fujishige <Temma.Fujishige@OIST.jp>
|
|
6
|
+
License-Expression: LicenseRef-OIST-License
|
|
7
|
+
License-File: LICENSE
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: Operating System :: OS Independent
|
|
10
|
+
Project-URL: Homepage, https://github.com/pao-unit/MDE
|
|
11
|
+
Project-URL: Documentation, https://github.com/pao-unit/MDE
|
|
12
|
+
Project-URL: Issues, https://github.com/pao-unit/MDE/issues
|
|
13
|
+
Requires-Python: >=3.11
|
|
14
|
+
Requires-Dist: pyEDM>=2.2.3
|
|
15
|
+
Requires-Dist: matplotlib
|
|
16
|
+
Requires-Dist: numpy
|
|
17
|
+
Requires-Dist: pandas>=2.0
|
|
18
|
+
Requires-Dist: pydiffmap>=0.2.0.1
|
|
19
|
+
Requires-Dist: scikit-learn
|
|
20
|
+
Requires-Dist: scipy
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
|
|
23
|
+
## Manifold Dimensional Expansion (MDE)
|
|
24
|
+
---
|
|
25
|
+
Manifold dimensional expansion is a causal discovery and dimensionality reduction technique designed to identify low dimensional maximally predictive _observables_ of a dynamical system with multivariate observations.
|
|
26
|
+
|
|
27
|
+
The algorithm is based on a greedy implementation of the generalized Takens embedding theorem. However, instead of using time delays for dimensionality expansion, _observables_ that improve the forecast skill of a target variable are added until no further improvement can be achieved. The default predictor is the [simplex](https://www.nature.com/articles/344734a0) function in [pyEDM](https://pypi.org/project/pyEDM/) providing a fully nonlinear predictor from [Empirical Dynamic Modeling (EDM)](https://en.wikipedia.org/wiki/Empirical_dynamic_modeling).
|
|
28
|
+
|
|
29
|
+
Specifically, given a target observable, scan all other observables to find the best 1-D predictor of the target, ensuring the predictor has causal inference with the target. With this 1-D vector scan all remaining observables to find the 2-D embedding with best predictability and causal inference. This greedy algorithm is iterated up to the point that no further prediction skill improvement can be produced.
|
|
30
|
+
|
|
31
|
+
Causal inference is performed by default with Convergent Cross Mapping ([CCM](https://science.sciencemag.org/content/338/6106/496)) ensuring the added observable is part of the dynamical system of the interrogated time series. The embedding dimension needed for CCM is automatically determined if parameter `E=0`, the default. Otherwise the specifed value of `E` is used. To account for unobserved variables time delay vectors of the top observables can be added.
|
|
32
|
+
|
|
33
|
+
Output is a DataFrame with a ranked list of observation vectors and predictive skill satisfying MDE criteria for the target variable.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
## Installation
|
|
37
|
+
|
|
38
|
+
`python -m pip install dimx`
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
## Usage
|
|
42
|
+
MDE is an object-oriented class implementation with command line interface (CLI) support. CLI parameters are configured through command line arguments, MDE class arguments through the MDE class constuctor.
|
|
43
|
+
|
|
44
|
+
MDE can be imported as a module and executed with `dimx.Run()` or from the command line with the`ManifoldDimExpand.py` executable as shown below.
|
|
45
|
+
|
|
46
|
+
CLI example:
|
|
47
|
+
```
|
|
48
|
+
./ManifoldDimExpand.py -d ../data/Fly80XY_norm_1061.csv
|
|
49
|
+
-rc index FWD Left_Right -D 10 -t FWD -l 1 300 -p 301 600
|
|
50
|
+
-C 10 -ccs 0.01 -emin 0.5 -P -title "MDE FWD" -v
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
MDE class constructor API example:
|
|
54
|
+
```python
|
|
55
|
+
from dimx import MDE
|
|
56
|
+
from pandas import read_csv
|
|
57
|
+
|
|
58
|
+
df = read_csv( './data/Fly80XY_norm_1061.csv' )
|
|
59
|
+
|
|
60
|
+
mde = MDE( df, target = 'FWD',
|
|
61
|
+
removeColumns = ['index','FWD','Left_Right'],
|
|
62
|
+
D = 10, lib = [1,300], pred = [301,600], ccmSeed = 12345,
|
|
63
|
+
cores = 10, plot = True, title = "MDE FWD" )
|
|
64
|
+
|
|
65
|
+
mde.Run()
|
|
66
|
+
|
|
67
|
+
mde.MDEOut
|
|
68
|
+
variables rho
|
|
69
|
+
0 TS33 0.652844
|
|
70
|
+
1 TS4 0.792290
|
|
71
|
+
2 TS17 0.823024
|
|
72
|
+
3 TS71 0.840094
|
|
73
|
+
4 TS44 0.840958
|
|
74
|
+
5 TS37 0.845765
|
|
75
|
+
6 TS9 0.846601
|
|
76
|
+
7 TS30 0.859614
|
|
77
|
+
8 TS47 0.860541
|
|
78
|
+
9 TS67 0.860230
|
|
79
|
+
```
|
|
80
|
+
---
|
|
81
|
+
## Real World Example
|
|
82
|
+
This example finds optimal observables and estimates the dimension of neural data from _Drosophila melanogaster_.
|
|
83
|
+
|
|
84
|
+
A fly expressing the calcium indicator GCaMP6f as a measure of neuronal activity was recorded walking on a Styrofoam ball. Neuronal activity across the brain was spatially segmented by independent component analysis (ICA) yielding 80 time series of neural activity from the component brain areas. Two behavioral variables were simultaneously recorded: forward speed (FWD) and left/right turning speed (Left_Right) [Aimon, S. et al. 2019](https://doi.org/10.1371/journal.pbio.2006732). A Jupyter Lab notebook is available at [MDE_Fly_Example](https://github.com/pao-unit/MDE/blob/b0eaca5bebc947676498057c679f4d3864909656/example/MDE_Fly_Example.ipynb).
|
|
85
|
+
|
|
86
|
+
### Import MDE and Evaluate classes
|
|
87
|
+
```python
|
|
88
|
+
from dimx import MDE, Evaluate
|
|
89
|
+
```
|
|
90
|
+
### Load data
|
|
91
|
+
```python
|
|
92
|
+
from pandas import read_csv
|
|
93
|
+
df = read_csv( './data/Fly80XY_norm_1061.csv' )
|
|
94
|
+
```
|
|
95
|
+
#### Instantiate and Run MDE class objects for FWD & Left_Right targets
|
|
96
|
+
We use the first 300 time series rows to create the EDM library, and perform out-of-sample prediction on time series rows 301-600. These indices are Not zero offset.
|
|
97
|
+
|
|
98
|
+
`ccmSlope` is the minimum slope of a linear fit to the CCM rho(L) curve to validate a causal driver. L is the vector of CCM library sizes at which CCM is evaluated. Default values for L are percentiles [10,15,85,90] of the number of observations (rows).
|
|
99
|
+
|
|
100
|
+
```python
|
|
101
|
+
Fly_FWD = MDE( df, # Pandas DataFrame of observables
|
|
102
|
+
target = 'FWD', # target behavior variable
|
|
103
|
+
removeColumns = ['index','FWD','Left_Right'], # variables to ignore
|
|
104
|
+
D = 12, # Max number of dimensions
|
|
105
|
+
lib = [1,300], # EDM library start,stop indices
|
|
106
|
+
pred = [301,600], # EDM prediction start,stop indices
|
|
107
|
+
ccmSlope = 0.01, # CCM convergence criteria
|
|
108
|
+
embedDimRhoMin = 0.65, # Minimum rho for CCM embedding dimension
|
|
109
|
+
crossMapRhoMin = 0.5, # Minumum rho for cross map of target : variables
|
|
110
|
+
cores = 10, # Number of cores in CrossMapColumns()
|
|
111
|
+
chunksize = 30,
|
|
112
|
+
plot = False )
|
|
113
|
+
Fly_FWD.Run()
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
```python
|
|
117
|
+
Fly_LR = MDE( df,
|
|
118
|
+
target = 'Left_Right',
|
|
119
|
+
removeColumns = ['index','FWD','Left_Right'],
|
|
120
|
+
D = 12,
|
|
121
|
+
lib = [1,600],
|
|
122
|
+
pred = [801,1000],
|
|
123
|
+
ccmSlope = 0.01,
|
|
124
|
+
embedDimRhoMin = 0.2,
|
|
125
|
+
crossMapRhoMin = 0.05,
|
|
126
|
+
cores = 10,
|
|
127
|
+
chunksize = 30,
|
|
128
|
+
plot = False )
|
|
129
|
+
Fly_LR.Run()
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
The FWD behavior suggests a dimension of D=5 observables is an appropriate low-dimensional set of obervables to predict FWD movement.
|
|
133
|
+

|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
#### Evaluate MDE components & compare to PCA & Diffusion Map
|
|
137
|
+
Here we compare out-of-sample prediction of FWD behavior with the 5 MDE identified observables as well as 5 component PCA and Diffusion Map.
|
|
138
|
+
|
|
139
|
+
```python
|
|
140
|
+
Fly_FWD_Eval = Evaluate( df,
|
|
141
|
+
columns_range = [1,81], # 0-offset range of columns for PCA, DMap
|
|
142
|
+
mde_columns = ['TS33', 'TS4', 'TS8', 'TS9', 'TS32'],
|
|
143
|
+
predictVar = 'FWD',
|
|
144
|
+
library = [1, 300], # index start,stop of observations for library
|
|
145
|
+
prediction = [301, 600], # index start,stop of predictions
|
|
146
|
+
components = 5, # Number of PCA & DMap components
|
|
147
|
+
dmap_k = 15, # diffusion_map k nearest neighbors
|
|
148
|
+
figsize = (8,6) )
|
|
149
|
+
Fly_FWD_Eval.Run()
|
|
150
|
+
Fly_FWD_Eval.Plot()
|
|
151
|
+
```
|
|
152
|
+
The MDE prediction has the lowest CAE (cumulative absolute error) to the out-of-sample observations. The diffusion map compoents are latent (not observable) and do not correspond in an obvious way to observed neural dynamics. The PCA prediction lumps the majority of the variance into a single component based on a linear decomposition. Both PCA and diffusion map predict activity during times when no FWD movement is observed, while MDE does not. Crucially, MDE predictions are not latent, but actual _observables_ of the system.
|
|
153
|
+
|
|
154
|
+

|
|
155
|
+
|
|
156
|
+
---
|
|
157
|
+
## Parameters
|
|
158
|
+
MDE parameters are defined in the MDE constructor.
|
|
159
|
+
|
|
160
|
+
| Parameter | Default | Description |
|
|
161
|
+
|---|---|---|
|
|
162
|
+
| dataFrame | None | Pandas DataFrame : column observation vectors, row observations |
|
|
163
|
+
| dataFile | None | Data file name to load |
|
|
164
|
+
| dataName | None | Data name in .npz archive |
|
|
165
|
+
| removeTime | False | Remove first column from dataFrame |
|
|
166
|
+
| noTime | False | First column of dataFrame is not time vector |
|
|
167
|
+
| columnNames | [] | dataFrame columns to process |
|
|
168
|
+
| initDataColumns | [] | If reading .npz omit these leading columns |
|
|
169
|
+
| removeColumns | [] | Columns to ignore |
|
|
170
|
+
| D | 3 | Maximum number of MDE dimensions |
|
|
171
|
+
| target | None | Target variable |
|
|
172
|
+
| lib | [] | EDM library indices. Default to all rows |
|
|
173
|
+
| pred | [] | EDM prediction indices. Default to all rows |
|
|
174
|
+
| Tp | 1 | Prediction time interval |
|
|
175
|
+
| tau | -1 | CCM embedding time delay |
|
|
176
|
+
| exclusionRadius | 0 | CCM library temporal exlcusion radius |
|
|
177
|
+
| sample | 20 | CCM random library samples to average |
|
|
178
|
+
| pLibSizes | [10, 15, 85, 90] | Percentiles of CCM library sizes |
|
|
179
|
+
| noCCM | False | Disable CCM |
|
|
180
|
+
| ccmSlope | 0.01 | Slope of CCM(LibSizes) convergence |
|
|
181
|
+
| ccmSeed | None | CCM random seed |
|
|
182
|
+
| E | 0 | CCM embedding dimension. If 0 compute automatically |
|
|
183
|
+
| crossMapRhoMin | 0.5 | Minimum rho for cross map acceptance |
|
|
184
|
+
| embedDimRhoMin | 0.5 | Minimum rho for CCM embedding dimension |
|
|
185
|
+
| maxE | 15 | Maximum embedding dimension for CCM |
|
|
186
|
+
| firstEMax | False | CCM embedding dimension is first local peak in rho(E) |
|
|
187
|
+
| timeDelay | 0 | Add N=timeDelay time delays |
|
|
188
|
+
| cores | 5 | number of multiprocessing CPU in CrossMapColumns() |
|
|
189
|
+
| mpMethod | None | multiprocessing context method in CrossMapColumns() |
|
|
190
|
+
| chunksize | 1 | multiprocessing chunksize in CrossMapColumns() |
|
|
191
|
+
| outDir | None | Output file directory |
|
|
192
|
+
| outFile | None | MDE object pickle file |
|
|
193
|
+
| outCSV | None | CSV of MDE output |
|
|
194
|
+
| logFile | None | Log file |
|
|
195
|
+
| consoleOut | True | Echo output to console |
|
|
196
|
+
| verbose | False | Verbose mode |
|
|
197
|
+
| debug | False | Debug mode |
|
|
198
|
+
| plot | False | Plot MDE result |
|
|
199
|
+
| title | None | Plot title |
|
|
200
|
+
| args | None | ArgumentParser object from CLI_Parser |
|
dimx-1.0.5/README.md
ADDED
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
## Manifold Dimensional Expansion (MDE)
|
|
2
|
+
---
|
|
3
|
+
Manifold dimensional expansion is a causal discovery and dimensionality reduction technique designed to identify low dimensional maximally predictive _observables_ of a dynamical system with multivariate observations.
|
|
4
|
+
|
|
5
|
+
The algorithm is based on a greedy implementation of the generalized Takens embedding theorem. However, instead of using time delays for dimensionality expansion, _observables_ that improve the forecast skill of a target variable are added until no further improvement can be achieved. The default predictor is the [simplex](https://www.nature.com/articles/344734a0) function in [pyEDM](https://pypi.org/project/pyEDM/) providing a fully nonlinear predictor from [Empirical Dynamic Modeling (EDM)](https://en.wikipedia.org/wiki/Empirical_dynamic_modeling).
|
|
6
|
+
|
|
7
|
+
Specifically, given a target observable, scan all other observables to find the best 1-D predictor of the target, ensuring the predictor has causal inference with the target. With this 1-D vector scan all remaining observables to find the 2-D embedding with best predictability and causal inference. This greedy algorithm is iterated up to the point that no further prediction skill improvement can be produced.
|
|
8
|
+
|
|
9
|
+
Causal inference is performed by default with Convergent Cross Mapping ([CCM](https://science.sciencemag.org/content/338/6106/496)) ensuring the added observable is part of the dynamical system of the interrogated time series. The embedding dimension needed for CCM is automatically determined if parameter `E=0`, the default. Otherwise the specifed value of `E` is used. To account for unobserved variables time delay vectors of the top observables can be added.
|
|
10
|
+
|
|
11
|
+
Output is a DataFrame with a ranked list of observation vectors and predictive skill satisfying MDE criteria for the target variable.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
## Installation
|
|
15
|
+
|
|
16
|
+
`python -m pip install dimx`
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
## Usage
|
|
20
|
+
MDE is an object-oriented class implementation with command line interface (CLI) support. CLI parameters are configured through command line arguments, MDE class arguments through the MDE class constuctor.
|
|
21
|
+
|
|
22
|
+
MDE can be imported as a module and executed with `dimx.Run()` or from the command line with the`ManifoldDimExpand.py` executable as shown below.
|
|
23
|
+
|
|
24
|
+
CLI example:
|
|
25
|
+
```
|
|
26
|
+
./ManifoldDimExpand.py -d ../data/Fly80XY_norm_1061.csv
|
|
27
|
+
-rc index FWD Left_Right -D 10 -t FWD -l 1 300 -p 301 600
|
|
28
|
+
-C 10 -ccs 0.01 -emin 0.5 -P -title "MDE FWD" -v
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
MDE class constructor API example:
|
|
32
|
+
```python
|
|
33
|
+
from dimx import MDE
|
|
34
|
+
from pandas import read_csv
|
|
35
|
+
|
|
36
|
+
df = read_csv( './data/Fly80XY_norm_1061.csv' )
|
|
37
|
+
|
|
38
|
+
mde = MDE( df, target = 'FWD',
|
|
39
|
+
removeColumns = ['index','FWD','Left_Right'],
|
|
40
|
+
D = 10, lib = [1,300], pred = [301,600], ccmSeed = 12345,
|
|
41
|
+
cores = 10, plot = True, title = "MDE FWD" )
|
|
42
|
+
|
|
43
|
+
mde.Run()
|
|
44
|
+
|
|
45
|
+
mde.MDEOut
|
|
46
|
+
variables rho
|
|
47
|
+
0 TS33 0.652844
|
|
48
|
+
1 TS4 0.792290
|
|
49
|
+
2 TS17 0.823024
|
|
50
|
+
3 TS71 0.840094
|
|
51
|
+
4 TS44 0.840958
|
|
52
|
+
5 TS37 0.845765
|
|
53
|
+
6 TS9 0.846601
|
|
54
|
+
7 TS30 0.859614
|
|
55
|
+
8 TS47 0.860541
|
|
56
|
+
9 TS67 0.860230
|
|
57
|
+
```
|
|
58
|
+
---
|
|
59
|
+
## Real World Example
|
|
60
|
+
This example finds optimal observables and estimates the dimension of neural data from _Drosophila melanogaster_.
|
|
61
|
+
|
|
62
|
+
A fly expressing the calcium indicator GCaMP6f as a measure of neuronal activity was recorded walking on a Styrofoam ball. Neuronal activity across the brain was spatially segmented by independent component analysis (ICA) yielding 80 time series of neural activity from the component brain areas. Two behavioral variables were simultaneously recorded: forward speed (FWD) and left/right turning speed (Left_Right) [Aimon, S. et al. 2019](https://doi.org/10.1371/journal.pbio.2006732). A Jupyter Lab notebook is available at [MDE_Fly_Example](https://github.com/pao-unit/MDE/blob/b0eaca5bebc947676498057c679f4d3864909656/example/MDE_Fly_Example.ipynb).
|
|
63
|
+
|
|
64
|
+
### Import MDE and Evaluate classes
|
|
65
|
+
```python
|
|
66
|
+
from dimx import MDE, Evaluate
|
|
67
|
+
```
|
|
68
|
+
### Load data
|
|
69
|
+
```python
|
|
70
|
+
from pandas import read_csv
|
|
71
|
+
df = read_csv( './data/Fly80XY_norm_1061.csv' )
|
|
72
|
+
```
|
|
73
|
+
#### Instantiate and Run MDE class objects for FWD & Left_Right targets
|
|
74
|
+
We use the first 300 time series rows to create the EDM library, and perform out-of-sample prediction on time series rows 301-600. These indices are Not zero offset.
|
|
75
|
+
|
|
76
|
+
`ccmSlope` is the minimum slope of a linear fit to the CCM rho(L) curve to validate a causal driver. L is the vector of CCM library sizes at which CCM is evaluated. Default values for L are percentiles [10,15,85,90] of the number of observations (rows).
|
|
77
|
+
|
|
78
|
+
```python
|
|
79
|
+
Fly_FWD = MDE( df, # Pandas DataFrame of observables
|
|
80
|
+
target = 'FWD', # target behavior variable
|
|
81
|
+
removeColumns = ['index','FWD','Left_Right'], # variables to ignore
|
|
82
|
+
D = 12, # Max number of dimensions
|
|
83
|
+
lib = [1,300], # EDM library start,stop indices
|
|
84
|
+
pred = [301,600], # EDM prediction start,stop indices
|
|
85
|
+
ccmSlope = 0.01, # CCM convergence criteria
|
|
86
|
+
embedDimRhoMin = 0.65, # Minimum rho for CCM embedding dimension
|
|
87
|
+
crossMapRhoMin = 0.5, # Minumum rho for cross map of target : variables
|
|
88
|
+
cores = 10, # Number of cores in CrossMapColumns()
|
|
89
|
+
chunksize = 30,
|
|
90
|
+
plot = False )
|
|
91
|
+
Fly_FWD.Run()
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
```python
|
|
95
|
+
Fly_LR = MDE( df,
|
|
96
|
+
target = 'Left_Right',
|
|
97
|
+
removeColumns = ['index','FWD','Left_Right'],
|
|
98
|
+
D = 12,
|
|
99
|
+
lib = [1,600],
|
|
100
|
+
pred = [801,1000],
|
|
101
|
+
ccmSlope = 0.01,
|
|
102
|
+
embedDimRhoMin = 0.2,
|
|
103
|
+
crossMapRhoMin = 0.05,
|
|
104
|
+
cores = 10,
|
|
105
|
+
chunksize = 30,
|
|
106
|
+
plot = False )
|
|
107
|
+
Fly_LR.Run()
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
The FWD behavior suggests a dimension of D=5 observables is an appropriate low-dimensional set of obervables to predict FWD movement.
|
|
111
|
+

|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
#### Evaluate MDE components & compare to PCA & Diffusion Map
|
|
115
|
+
Here we compare out-of-sample prediction of FWD behavior with the 5 MDE identified observables as well as 5 component PCA and Diffusion Map.
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
Fly_FWD_Eval = Evaluate( df,
|
|
119
|
+
columns_range = [1,81], # 0-offset range of columns for PCA, DMap
|
|
120
|
+
mde_columns = ['TS33', 'TS4', 'TS8', 'TS9', 'TS32'],
|
|
121
|
+
predictVar = 'FWD',
|
|
122
|
+
library = [1, 300], # index start,stop of observations for library
|
|
123
|
+
prediction = [301, 600], # index start,stop of predictions
|
|
124
|
+
components = 5, # Number of PCA & DMap components
|
|
125
|
+
dmap_k = 15, # diffusion_map k nearest neighbors
|
|
126
|
+
figsize = (8,6) )
|
|
127
|
+
Fly_FWD_Eval.Run()
|
|
128
|
+
Fly_FWD_Eval.Plot()
|
|
129
|
+
```
|
|
130
|
+
The MDE prediction has the lowest CAE (cumulative absolute error) to the out-of-sample observations. The diffusion map compoents are latent (not observable) and do not correspond in an obvious way to observed neural dynamics. The PCA prediction lumps the majority of the variance into a single component based on a linear decomposition. Both PCA and diffusion map predict activity during times when no FWD movement is observed, while MDE does not. Crucially, MDE predictions are not latent, but actual _observables_ of the system.
|
|
131
|
+
|
|
132
|
+

|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
## Parameters
|
|
136
|
+
MDE parameters are defined in the MDE constructor.
|
|
137
|
+
|
|
138
|
+
| Parameter | Default | Description |
|
|
139
|
+
|---|---|---|
|
|
140
|
+
| dataFrame | None | Pandas DataFrame : column observation vectors, row observations |
|
|
141
|
+
| dataFile | None | Data file name to load |
|
|
142
|
+
| dataName | None | Data name in .npz archive |
|
|
143
|
+
| removeTime | False | Remove first column from dataFrame |
|
|
144
|
+
| noTime | False | First column of dataFrame is not time vector |
|
|
145
|
+
| columnNames | [] | dataFrame columns to process |
|
|
146
|
+
| initDataColumns | [] | If reading .npz omit these leading columns |
|
|
147
|
+
| removeColumns | [] | Columns to ignore |
|
|
148
|
+
| D | 3 | Maximum number of MDE dimensions |
|
|
149
|
+
| target | None | Target variable |
|
|
150
|
+
| lib | [] | EDM library indices. Default to all rows |
|
|
151
|
+
| pred | [] | EDM prediction indices. Default to all rows |
|
|
152
|
+
| Tp | 1 | Prediction time interval |
|
|
153
|
+
| tau | -1 | CCM embedding time delay |
|
|
154
|
+
| exclusionRadius | 0 | CCM library temporal exlcusion radius |
|
|
155
|
+
| sample | 20 | CCM random library samples to average |
|
|
156
|
+
| pLibSizes | [10, 15, 85, 90] | Percentiles of CCM library sizes |
|
|
157
|
+
| noCCM | False | Disable CCM |
|
|
158
|
+
| ccmSlope | 0.01 | Slope of CCM(LibSizes) convergence |
|
|
159
|
+
| ccmSeed | None | CCM random seed |
|
|
160
|
+
| E | 0 | CCM embedding dimension. If 0 compute automatically |
|
|
161
|
+
| crossMapRhoMin | 0.5 | Minimum rho for cross map acceptance |
|
|
162
|
+
| embedDimRhoMin | 0.5 | Minimum rho for CCM embedding dimension |
|
|
163
|
+
| maxE | 15 | Maximum embedding dimension for CCM |
|
|
164
|
+
| firstEMax | False | CCM embedding dimension is first local peak in rho(E) |
|
|
165
|
+
| timeDelay | 0 | Add N=timeDelay time delays |
|
|
166
|
+
| cores | 5 | number of multiprocessing CPU in CrossMapColumns() |
|
|
167
|
+
| mpMethod | None | multiprocessing context method in CrossMapColumns() |
|
|
168
|
+
| chunksize | 1 | multiprocessing chunksize in CrossMapColumns() |
|
|
169
|
+
| outDir | None | Output file directory |
|
|
170
|
+
| outFile | None | MDE object pickle file |
|
|
171
|
+
| outCSV | None | CSV of MDE output |
|
|
172
|
+
| logFile | None | Log file |
|
|
173
|
+
| consoleOut | True | Echo output to console |
|
|
174
|
+
| verbose | False | Verbose mode |
|
|
175
|
+
| debug | False | Debug mode |
|
|
176
|
+
| plot | False | Plot MDE result |
|
|
177
|
+
| title | None | Plot title |
|
|
178
|
+
| args | None | ArgumentParser object from CLI_Parser |
|