moseq2-model 1.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- moseq2_model-1.4.0/LICENSE.md +49 -0
- moseq2_model-1.4.0/PKG-INFO +84 -0
- moseq2_model-1.4.0/README.md +43 -0
- moseq2_model-1.4.0/moseq2_model/__init__.py +1 -0
- moseq2_model-1.4.0/moseq2_model/_yaml.py +46 -0
- moseq2_model-1.4.0/moseq2_model/cli.py +326 -0
- moseq2_model-1.4.0/moseq2_model/gui.py +104 -0
- moseq2_model-1.4.0/moseq2_model/helpers/__init__.py +0 -0
- moseq2_model-1.4.0/moseq2_model/helpers/data.py +324 -0
- moseq2_model-1.4.0/moseq2_model/helpers/wrappers.py +348 -0
- moseq2_model-1.4.0/moseq2_model/train/__init__.py +0 -0
- moseq2_model-1.4.0/moseq2_model/train/backend.py +711 -0
- moseq2_model-1.4.0/moseq2_model/train/models.py +227 -0
- moseq2_model-1.4.0/moseq2_model/train/util.py +413 -0
- moseq2_model-1.4.0/moseq2_model/util.py +704 -0
- moseq2_model-1.4.0/moseq2_model.egg-info/PKG-INFO +84 -0
- moseq2_model-1.4.0/moseq2_model.egg-info/SOURCES.txt +23 -0
- moseq2_model-1.4.0/moseq2_model.egg-info/dependency_links.txt +1 -0
- moseq2_model-1.4.0/moseq2_model.egg-info/entry_points.txt +2 -0
- moseq2_model-1.4.0/moseq2_model.egg-info/requires.txt +11 -0
- moseq2_model-1.4.0/moseq2_model.egg-info/top_level.txt +1 -0
- moseq2_model-1.4.0/pyproject.toml +9 -0
- moseq2_model-1.4.0/setup.cfg +7 -0
- moseq2_model-1.4.0/setup.py +77 -0
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Non-Commercial Research and Academic Use Software License and Terms of Use
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
MoSeq is a software package that includes original code created by the Harvard researchers listed below (the “Software”), and third-party code that may be obtained by End Users separately. The Software is designed to label behaviors identified from 3D pose dynamics data through the use of computational modeling and fitting approaches. The Software was developed by Alexander Wiltschko, Ph.D., Sandeep Robert Datta, Ph.D., and Matthew J. Johnson, Ph.D. at Harvard University. It is distributed for free academic and non-commercial research use by the President and Fellows of Harvard College (“Harvard”).
|
|
5
|
+
|
|
6
|
+
Using the Software indicates your agreement to be bound by the terms of this Software Use Agreement (“Agreement”). Absent your agreement to the terms below, you (the “End User”) have no rights to hold or use the Software whatsoever.
|
|
7
|
+
|
|
8
|
+
Harvard agrees to grant hereunder a limited non-exclusive license to End User for the use of the Software in the performance of End User’s internal, non-commercial research and academic use at End User’s academic or not-for-profit research institution (“Institution”) on the following terms and conditions:
|
|
9
|
+
|
|
10
|
+
1. **NO COMMERCIAL USE.** End User shall not use the Software for Commercial use and any such use of the Software is expressly prohibited. “Commercial use” includes, but is not limited to, (i) use of the Software in fee-for-service arrangements, (ii) use of the Software by core facilities or laboratories to provide research services to (or in collaboration with) for-profit third parties for a fee, and (iii) use of the Software in industry-sponsored and/or collaborative research projects in which any commercial rights are granted to the sponsor or collaborator. If End User wishes to use the Software for Commercial use, End User must execute a separate license agreement with Harvard.
|
|
11
|
+
|
|
12
|
+
_Requests for use of the Software for Commercial use, please contact:_
|
|
13
|
+
|
|
14
|
+
Office of Technology Development
|
|
15
|
+
Harvard University
|
|
16
|
+
Smith Campus Center, Suite 727E
|
|
17
|
+
1350 Massachusetts Avenue Cambridge, MA 02138 USA Telephone: (617) 495-3067
|
|
18
|
+
Facsimile: (617) 495-9568
|
|
19
|
+
E-mail: otd@harvard.edu
|
|
20
|
+
|
|
21
|
+
2. **OWNERSHIP AND COPYRIGHT NOTICE.** Harvard owns all intellectual property in the Software. End User shall gain no ownership to the Software. End User shall not remove or delete, and shall retain in the Software (including in any modifications to the Software and in any Derivative Works), the copyright, trademark, or other notices pertaining to Software as provided with the Software.
|
|
22
|
+
|
|
23
|
+
3. **DERIVATIVE WORKS.** End User may create use and distribute Derivative Works, as such term is defined under U.S. copyright laws, provided that any such Derivative Works shall be restricted to non-commercial research and academic use. End User may not distribute Derivative Works to any for-profit third parties for Commercial use.
|
|
24
|
+
|
|
25
|
+
4. **FEEDBACK.** In order to improve the Software, comments from End Users may be useful. End User agrees to provide Harvard with feedback on the End User’s use of the Software (e.g., any bugs in the Software, the user experience, etc.). Harvard is permitted to use such information provided by End User in making changes and improvements to the Software without compensation or accounting to End User.
|
|
26
|
+
|
|
27
|
+
5. **NON ASSERT.** End User acknowledges that Harvard may develop modifications to the Software that may be based on the feedback provided by End User under Section 5 above. Harvard shall not be restricted in any way by End User regarding its use of such information. End User acknowledges the right of Harvard to prepare, publish, display, reproduce, transmit and or use modifications to the Software that may be substantially similar or functionally equivalent to End User’s modifications and/or improvements if any. In the event that End User obtains patent protection for any modification or improvement to Software, End User agrees not to allege or enjoin infringement of End User’s patent against Harvard, or any of the researchers, medical or research staff, officers, directors and employees of those institutions.
|
|
28
|
+
|
|
29
|
+
6. **PUBLICATION & ATTRIBUTION.** End User has the right to publish, present, or share results from the use of the Software. In accordance with customary academic practice, End User will acknowledge Harvard as the provider of the Software and may cite the relevant reference(s) from the following list of publications:
|
|
30
|
+
|
|
31
|
+
Wiltschko AB, Johnson MJ, Iurilli G, Peterson RE, Katon JE, Pashkovski SL, Abraira VE, Adams RP and Datta SR. (2015). Mapping sub-second Structure in Behavior. Neuron, 88:1121.
|
|
32
|
+
|
|
33
|
+
Markowitz JE, Gillis WF, Beron CC, Neufeld SQ, Robertson K, Bhagat ND, Peterson RE, Peterson E, Hyun M, Linderman SW, Sabatini BL and Datta SR. (2018). The striatum organizes 3D behavior via moment-to-moment action selection. Cell, in press.
|
|
34
|
+
|
|
35
|
+
7.** NO WARRANTIES.** THE SOFTWARE IS PROVIDED "AS IS." TO THE FULLEST EXTENT PERMITTED BY LAW, HARVARD HEREBY DISCLAIMS ALL WARRANTIES OF ANY KIND (EXPRESS, IMPLIED OR OTHERWISE) REGARDING THE SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OWNERSHIP, AND NON-INFRINGEMENT. HARVARD MAKES NO WARRANTY ABOUT THE ACCURACY, RELIABILITY, COMPLETENESS, TIMELINESS, SUFFICIENCY OR QUALITY OF THE SOFTWARE. HARVARD DOES NOT WARRANT THAT THE SOFTWARE WILL OPERATE WITHOUT ERROR OR INTERRUPTION.
|
|
36
|
+
|
|
37
|
+
8. **LIMITATIONS OF LIABILITY AND REMEDIES.** USE OF THE SOFTWARE IS AT END USER’S OWN RISK. IF END USER IS DISSATISFIED WITH THE SOFTWARE, ITS EXCLUSIVE REMEDY IS TO STOP USING IT. IN NO EVENT SHALL HARVARD BE LIABLE TO END USER OR ITS INSTITUTION, IN CONTRACT, TORT OR OTHERWISE, FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR OTHER DAMAGES OF ANY KIND WHATSOEVER ARISING OUT OF OR IN CONNECTION WITH THE SOFTWARE, EVEN IF HARVARD IS NEGLIGENT OR OTHERWISE AT FAULT, AND REGARDLESS OF WHETHER HARVARD IS ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
|
|
38
|
+
|
|
39
|
+
9. **INDEMNIFICATION.** To the extent permitted by law, End User shall indemnify, defend and hold harmless Harvard, their corporate affiliates, current or future directors, trustees, officers, faculty, medical and professional staff, employees, students and agents and their respective successors, heirs and assigns (the "Indemnitees"), against any liability, damage, loss or expense (including reasonable attorney's fees and expenses of litigation) incurred by or imposed upon the Indemnitees or any one of them in connection with any claims, suits, actions, demands or judgments arising from End User’s breach of this Agreement or its Institution’s use of the Software except to the extent caused by the gross negligence or willful misconduct of Harvard. This indemnification provision shall survive expiration or termination of this Agreement.
|
|
40
|
+
|
|
41
|
+
10. **GOVERNING LAW.** This Agreement shall be construed and governed by the laws of the Commonwealth of Massachusetts regardless of otherwise applicable choice of law standards.
|
|
42
|
+
|
|
43
|
+
11. **NON-USE OF NAME.** Nothing in this License and Terms of Use shall be construed as granting End Users or their Institutions any rights or licenses to use any trademarks, service marks or logos associated with the Software. End User may not use the terms “Harvard” (or a substantially similar term) in any way that is inconsistent with the permitted uses described herein. End Users may not use any name or emblem of Harvard or any of its schools or subdivisions for any purpose, or to falsely suggest any relationship between End User (or its Institution) and Harvard, or in any manner that would infringe or violate any of its rights.
|
|
44
|
+
|
|
45
|
+
12. End User represents and warrants that it has the legal authority to enter into this License and Terms of Use on behalf of itself and its Institution.
|
|
46
|
+
|
|
47
|
+
***
|
|
48
|
+
|
|
49
|
+
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: moseq2-model
|
|
3
|
+
Version: 1.4.0
|
|
4
|
+
Summary: Modeling for the best
|
|
5
|
+
Author: Datta Lab
|
|
6
|
+
License: LicenseRef-Harvard-Academic-Non-Commercial
|
|
7
|
+
Platform: any
|
|
8
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
9
|
+
Classifier: Intended Audience :: Science/Research
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
16
|
+
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
17
|
+
Requires-Python: >=3.10
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
License-File: LICENSE.md
|
|
20
|
+
Requires-Dist: jax-moseq>=0.3.5
|
|
21
|
+
Requires-Dist: jax
|
|
22
|
+
Requires-Dist: numpy
|
|
23
|
+
Requires-Dist: scipy
|
|
24
|
+
Requires-Dist: h5py>=3
|
|
25
|
+
Requires-Dist: click>=8
|
|
26
|
+
Requires-Dist: joblib
|
|
27
|
+
Requires-Dist: cytoolz
|
|
28
|
+
Requires-Dist: tqdm>=4.48.0
|
|
29
|
+
Requires-Dist: matplotlib
|
|
30
|
+
Requires-Dist: ruamel.yaml>=0.17
|
|
31
|
+
Dynamic: author
|
|
32
|
+
Dynamic: classifier
|
|
33
|
+
Dynamic: description
|
|
34
|
+
Dynamic: description-content-type
|
|
35
|
+
Dynamic: license
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
Dynamic: platform
|
|
38
|
+
Dynamic: requires-dist
|
|
39
|
+
Dynamic: requires-python
|
|
40
|
+
Dynamic: summary
|
|
41
|
+
|
|
42
|
+
# MoSeq2-model
|
|
43
|
+
|
|
44
|
+
[](https://codecov.io/gh/dattalab/moseq2_model)
|
|
45
|
+
[](https://zenodo.org/badge/latestdoi/81390468)
|
|
46
|
+
|
|
47
|
+
# Installation
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
pip install moseq2-model
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Python 3.10 or newer, on Linux. macOS and Windows support is being verified.
|
|
54
|
+
|
|
55
|
+
Most people install the whole pipeline in one step with `pip install
|
|
56
|
+
moseq2-app`, which brings this package with it. See [the installation
|
|
57
|
+
guide](https://github.com/dattalab/moseq2-app/wiki/MoSeq2-Installation).
|
|
58
|
+
|
|
59
|
+
# [Documentation: MoSeq2 Wiki](https://github.com/dattalab/moseq2-app/wiki)
|
|
60
|
+
You can find more information about MoSeq Pipeline, installation, step-by-step instructions, documentation for Command Line Interface(CLI), tutorials etc in [MoSeq2 Wiki](https://github.com/dattalab/moseq2-app/wiki).
|
|
61
|
+
|
|
62
|
+
You can run `moseq2-model --version` to check the current version and `moseq2-model --help` to see all the commands.
|
|
63
|
+
```bash
|
|
64
|
+
Usage: moseq2-model [OPTIONS] COMMAND [ARGS]...
|
|
65
|
+
|
|
66
|
+
Options:
|
|
67
|
+
--version Show the version and exit. [default: False]
|
|
68
|
+
--help Show this message and exit. [default: False]
|
|
69
|
+
|
|
70
|
+
Commands:
|
|
71
|
+
count-frames Counts number of frames in given h5 file (pca_scores)
|
|
72
|
+
kappa-scan Batch fit multiple models scanning over different syllable...
|
|
73
|
+
learn-model Trains ARHMM on PCA Scores with given training parameters
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
# Community Support and Contributing
|
|
77
|
+
- Please join [](https://moseqworkspace.slack.com) to post questions and interactive with MoSeq developers and users.
|
|
78
|
+
- If you encounter bugs, errors or issues, please submit a Bug report [here](https://github.com/dattalab/moseq2-app/issues/new/choose). We encourage you to check out the [troubleshooting and tips section](https://github.com/dattalab/moseq2-app/wiki/Troubleshooting-and-Tips) and search your issues in [the existing issues](https://github.com/dattalab/moseq2-app/issues) first.
|
|
79
|
+
- If you want to see certain features in MoSeq or you have new ideas, please submit a Feature request [here](https://github.com/dattalab/moseq2-app/issues/new/choose).
|
|
80
|
+
- If you want to contribute to our codebases, please check out our [Developer Guidelines](https://github.com/dattalab/moseq2-app/wiki/MoSeq-Developer-Guidelines).
|
|
81
|
+
- Please tell us what you think by filling out [this user survey](https://forms.gle/FbtEN8E382y8jF3p6).
|
|
82
|
+
|
|
83
|
+
# License
|
|
84
|
+
MoSeq is freely available for academic use under a license provided by Harvard University. Please refer to the license file for details. If you are interested in using MoSeq for commercial purposes please contact Bob Datta directly at srdatta@hms.harvard.edu, who will put you in touch with the appropriate people in the Harvard Technology Transfer office.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# MoSeq2-model
|
|
2
|
+
|
|
3
|
+
[](https://codecov.io/gh/dattalab/moseq2_model)
|
|
4
|
+
[](https://zenodo.org/badge/latestdoi/81390468)
|
|
5
|
+
|
|
6
|
+
# Installation
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
pip install moseq2-model
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Python 3.10 or newer, on Linux. macOS and Windows support is being verified.
|
|
13
|
+
|
|
14
|
+
Most people install the whole pipeline in one step with `pip install
|
|
15
|
+
moseq2-app`, which brings this package with it. See [the installation
|
|
16
|
+
guide](https://github.com/dattalab/moseq2-app/wiki/MoSeq2-Installation).
|
|
17
|
+
|
|
18
|
+
# [Documentation: MoSeq2 Wiki](https://github.com/dattalab/moseq2-app/wiki)
|
|
19
|
+
You can find more information about MoSeq Pipeline, installation, step-by-step instructions, documentation for Command Line Interface(CLI), tutorials etc in [MoSeq2 Wiki](https://github.com/dattalab/moseq2-app/wiki).
|
|
20
|
+
|
|
21
|
+
You can run `moseq2-model --version` to check the current version and `moseq2-model --help` to see all the commands.
|
|
22
|
+
```bash
|
|
23
|
+
Usage: moseq2-model [OPTIONS] COMMAND [ARGS]...
|
|
24
|
+
|
|
25
|
+
Options:
|
|
26
|
+
--version Show the version and exit. [default: False]
|
|
27
|
+
--help Show this message and exit. [default: False]
|
|
28
|
+
|
|
29
|
+
Commands:
|
|
30
|
+
count-frames Counts number of frames in given h5 file (pca_scores)
|
|
31
|
+
kappa-scan Batch fit multiple models scanning over different syllable...
|
|
32
|
+
learn-model Trains ARHMM on PCA Scores with given training parameters
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
# Community Support and Contributing
|
|
36
|
+
- Please join [](https://moseqworkspace.slack.com) to post questions and interactive with MoSeq developers and users.
|
|
37
|
+
- If you encounter bugs, errors or issues, please submit a Bug report [here](https://github.com/dattalab/moseq2-app/issues/new/choose). We encourage you to check out the [troubleshooting and tips section](https://github.com/dattalab/moseq2-app/wiki/Troubleshooting-and-Tips) and search your issues in [the existing issues](https://github.com/dattalab/moseq2-app/issues) first.
|
|
38
|
+
- If you want to see certain features in MoSeq or you have new ideas, please submit a Feature request [here](https://github.com/dattalab/moseq2-app/issues/new/choose).
|
|
39
|
+
- If you want to contribute to our codebases, please check out our [Developer Guidelines](https://github.com/dattalab/moseq2-app/wiki/MoSeq-Developer-Guidelines).
|
|
40
|
+
- Please tell us what you think by filling out [this user survey](https://forms.gle/FbtEN8E382y8jF3p6).
|
|
41
|
+
|
|
42
|
+
# License
|
|
43
|
+
MoSeq is freely available for academic use under a license provided by Harvard University. Please refer to the license file for details. If you are interested in using MoSeq for commercial purposes please contact Bob Datta directly at srdatta@hms.harvard.edu, who will put you in touch with the appropriate people in the Harvard Technology Transfer office.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "1.4.0"
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
"""YAML reading and writing, in one place.
|
|
2
|
+
|
|
3
|
+
``ruamel.yaml`` removed the module-level ``safe_load``/``safe_dump`` functions in
|
|
4
|
+
0.18 in favour of instantiating a parser. Routing every call through here keeps
|
|
5
|
+
that detail in one file rather than at each of the call sites.
|
|
6
|
+
|
|
7
|
+
The parser stays ``ruamel``, deliberately. ``ruamel``'s safe loader is YAML 1.2;
|
|
8
|
+
PyYAML is YAML 1.1 and reads unquoted ``yes``/``no``/``on``/``off`` as booleans
|
|
9
|
+
and ``06:00:00`` as a sexagesimal integer. moseq2 config files contain both
|
|
10
|
+
shapes -- ``wall_time: 06:00:00`` is interpolated straight into an ``sbatch -t``
|
|
11
|
+
argument by ``util.get_parameter_strings`` -- so switching parsers would quietly
|
|
12
|
+
change how existing user configs are read.
|
|
13
|
+
"""
|
|
14
|
+
|
|
15
|
+
import io
|
|
16
|
+
|
|
17
|
+
from ruamel.yaml import YAML
|
|
18
|
+
|
|
19
|
+
# ``pure=True`` selects the Python implementation, matching what the module-level
|
|
20
|
+
# ``safe_load`` used and keeping behaviour identical whether or not the C
|
|
21
|
+
# extension is available.
|
|
22
|
+
_yaml = YAML(typ="safe", pure=True)
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
def read_yaml(path_or_stream):
|
|
26
|
+
"""Parse a YAML file, given a path or an open stream."""
|
|
27
|
+
if hasattr(path_or_stream, "read"):
|
|
28
|
+
return _yaml.load(path_or_stream)
|
|
29
|
+
with open(path_or_stream, "r") as handle:
|
|
30
|
+
return _yaml.load(handle)
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def write_yaml(data, path_or_stream):
|
|
34
|
+
"""Write ``data`` as YAML, given a path or an open stream."""
|
|
35
|
+
if hasattr(path_or_stream, "write"):
|
|
36
|
+
_yaml.dump(data, path_or_stream)
|
|
37
|
+
return
|
|
38
|
+
with open(path_or_stream, "w") as handle:
|
|
39
|
+
_yaml.dump(data, handle)
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def dump_yaml(data):
|
|
43
|
+
"""Serialize ``data`` to a YAML string."""
|
|
44
|
+
stream = io.StringIO()
|
|
45
|
+
_yaml.dump(data, stream)
|
|
46
|
+
return stream.getvalue()
|
|
@@ -0,0 +1,326 @@
|
|
|
1
|
+
"""
|
|
2
|
+
CLI for modeling the data using AR-HMM.
|
|
3
|
+
"""
|
|
4
|
+
|
|
5
|
+
import os
|
|
6
|
+
import click
|
|
7
|
+
from os.path import join
|
|
8
|
+
from moseq2_model.util import count_frames as count_frames_wrapper
|
|
9
|
+
from moseq2_model.helpers.wrappers import (
|
|
10
|
+
learn_model_wrapper,
|
|
11
|
+
kappa_scan_fit_models_wrapper,
|
|
12
|
+
apply_model_wrapper,
|
|
13
|
+
)
|
|
14
|
+
|
|
15
|
+
orig_init = click.core.Option.__init__
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
def new_init(self, *args, **kwargs):
|
|
19
|
+
orig_init(self, *args, **kwargs)
|
|
20
|
+
self.show_default = True
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
click.core.Option.__init__ = new_init
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
@click.group()
|
|
27
|
+
@click.version_option()
|
|
28
|
+
def cli():
|
|
29
|
+
pass
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
@cli.command(
|
|
33
|
+
name="count-frames", help="Count the number of frames in given h5 file (pca_scores)"
|
|
34
|
+
)
|
|
35
|
+
@click.argument("input_file", type=click.Path(exists=True))
|
|
36
|
+
@click.option(
|
|
37
|
+
"--var-name",
|
|
38
|
+
type=str,
|
|
39
|
+
default="scores",
|
|
40
|
+
help="Variable name in input file with PCs",
|
|
41
|
+
)
|
|
42
|
+
def count_frames(input_file, var_name):
|
|
43
|
+
# Count the number of frames in the input file.
|
|
44
|
+
count_frames_wrapper(input_file=input_file, var_name=var_name)
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def modeling_parameters(function):
|
|
48
|
+
|
|
49
|
+
function = click.option(
|
|
50
|
+
"--hold-out",
|
|
51
|
+
"-h",
|
|
52
|
+
is_flag=True,
|
|
53
|
+
default=False,
|
|
54
|
+
help="Hold out one fold (set by nfolds) for computing heldout likelihood",
|
|
55
|
+
)(function)
|
|
56
|
+
function = click.option(
|
|
57
|
+
"--hold-out-seed",
|
|
58
|
+
type=int,
|
|
59
|
+
default=-1,
|
|
60
|
+
help="Random seed for holding out data (set for reproducibility)",
|
|
61
|
+
)(function)
|
|
62
|
+
function = click.option(
|
|
63
|
+
"--nfolds", type=int, default=5, help="Number of folds for split"
|
|
64
|
+
)(function)
|
|
65
|
+
function = click.option(
|
|
66
|
+
"--ncpus",
|
|
67
|
+
"-c",
|
|
68
|
+
type=int,
|
|
69
|
+
default=0,
|
|
70
|
+
help="Number of cores to use for resampling",
|
|
71
|
+
)(function)
|
|
72
|
+
function = click.option(
|
|
73
|
+
"--num-iter",
|
|
74
|
+
"-n",
|
|
75
|
+
type=int,
|
|
76
|
+
default=100,
|
|
77
|
+
help="Number of interations to resample model",
|
|
78
|
+
)(function)
|
|
79
|
+
function = click.option(
|
|
80
|
+
"--var-name",
|
|
81
|
+
type=str,
|
|
82
|
+
default="scores",
|
|
83
|
+
help="Variable name in input file with PCs",
|
|
84
|
+
)(function)
|
|
85
|
+
function = click.option(
|
|
86
|
+
"--e-step",
|
|
87
|
+
is_flag=True,
|
|
88
|
+
default=False,
|
|
89
|
+
help="Compute the expected state sequence for each recordings",
|
|
90
|
+
)(function)
|
|
91
|
+
function = click.option(
|
|
92
|
+
"--save-every",
|
|
93
|
+
"-s",
|
|
94
|
+
type=int,
|
|
95
|
+
default=-1,
|
|
96
|
+
help="Increment to save labels and model object (-1 for just last)",
|
|
97
|
+
)(function)
|
|
98
|
+
function = click.option(
|
|
99
|
+
"--save-model",
|
|
100
|
+
type=bool,
|
|
101
|
+
default=True,
|
|
102
|
+
help="Save model object at the end of training",
|
|
103
|
+
)(function)
|
|
104
|
+
function = click.option(
|
|
105
|
+
"--max-states", "-m", type=int, default=100, help="Maximum number of states"
|
|
106
|
+
)(function)
|
|
107
|
+
function = click.option(
|
|
108
|
+
"--npcs", type=int, default=10, help="Number of PCs to use"
|
|
109
|
+
)(function)
|
|
110
|
+
function = click.option(
|
|
111
|
+
"--whiten",
|
|
112
|
+
"-w",
|
|
113
|
+
type=str,
|
|
114
|
+
default="all",
|
|
115
|
+
help="Whiten PCs: (e)each session (a)ll combined or (n)o whitening",
|
|
116
|
+
)(function)
|
|
117
|
+
function = click.option(
|
|
118
|
+
"--progressbar", "-p", type=bool, default=True, help="Show model progress"
|
|
119
|
+
)(function)
|
|
120
|
+
function = click.option(
|
|
121
|
+
"--percent-split",
|
|
122
|
+
type=int,
|
|
123
|
+
default=0,
|
|
124
|
+
help="Training-validation split percentage used when not holding out data and when this parameter > 0.",
|
|
125
|
+
)(function)
|
|
126
|
+
function = click.option(
|
|
127
|
+
"--load-groups",
|
|
128
|
+
type=bool,
|
|
129
|
+
default=True,
|
|
130
|
+
help="If groups should be loaded with the PC scores.",
|
|
131
|
+
)(function)
|
|
132
|
+
function = click.option(
|
|
133
|
+
"--gamma",
|
|
134
|
+
"-g",
|
|
135
|
+
type=float,
|
|
136
|
+
default=1e3,
|
|
137
|
+
help="Gamma; hierarchical dirichlet process hyperparameter (try not to change it).",
|
|
138
|
+
)(function)
|
|
139
|
+
function = click.option(
|
|
140
|
+
"--alpha",
|
|
141
|
+
"-a",
|
|
142
|
+
type=float,
|
|
143
|
+
default=5.7,
|
|
144
|
+
help="Alpha; hierarchical dirichlet process hyperparameter (try not to change it).",
|
|
145
|
+
)(function)
|
|
146
|
+
function = click.option(
|
|
147
|
+
"--noise-level",
|
|
148
|
+
type=float,
|
|
149
|
+
default=0,
|
|
150
|
+
help="Additive white gaussian noise to input data for regularization. Not generally used",
|
|
151
|
+
)(function)
|
|
152
|
+
function = click.option(
|
|
153
|
+
"--nlags", type=int, default=3, help="Number of lags to use"
|
|
154
|
+
)(function)
|
|
155
|
+
function = click.option(
|
|
156
|
+
"--separate-trans",
|
|
157
|
+
is_flag=True,
|
|
158
|
+
default=False,
|
|
159
|
+
help="Use separate transition matrix for each group",
|
|
160
|
+
)(function)
|
|
161
|
+
function = click.option(
|
|
162
|
+
"--robust",
|
|
163
|
+
is_flag=True,
|
|
164
|
+
default=False,
|
|
165
|
+
help="Use robust AR-HMM model. More tolerant to noise",
|
|
166
|
+
)(function)
|
|
167
|
+
function = click.option(
|
|
168
|
+
"--check-every",
|
|
169
|
+
type=int,
|
|
170
|
+
default=5,
|
|
171
|
+
help="Increment to record training and validation log-likelihoods.",
|
|
172
|
+
)(function)
|
|
173
|
+
|
|
174
|
+
return function
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
@cli.command(
|
|
178
|
+
name="learn-model", help="Train ARHMM on PC Scores with given training parameters"
|
|
179
|
+
)
|
|
180
|
+
@click.argument("input_file", type=click.Path(exists=True))
|
|
181
|
+
@click.argument(
|
|
182
|
+
"dest_file", type=click.Path(file_okay=True, writable=True, resolve_path=True)
|
|
183
|
+
)
|
|
184
|
+
@modeling_parameters
|
|
185
|
+
@click.option(
|
|
186
|
+
"--kappa",
|
|
187
|
+
"-k",
|
|
188
|
+
type=float,
|
|
189
|
+
default=None,
|
|
190
|
+
help="Kappa; hyperparameter used to set syllable duration. Larger k = longer syllable lengths",
|
|
191
|
+
)
|
|
192
|
+
@click.option(
|
|
193
|
+
"--checkpoint-freq",
|
|
194
|
+
type=int,
|
|
195
|
+
default=-1,
|
|
196
|
+
help="save model checkpoint every n iterations",
|
|
197
|
+
)
|
|
198
|
+
@click.option(
|
|
199
|
+
"--use-checkpoint",
|
|
200
|
+
is_flag=True,
|
|
201
|
+
default=False,
|
|
202
|
+
help="indicate whether to use previously saved checkpoint",
|
|
203
|
+
)
|
|
204
|
+
@click.option(
|
|
205
|
+
"--index",
|
|
206
|
+
"-i",
|
|
207
|
+
type=click.Path(),
|
|
208
|
+
default="",
|
|
209
|
+
help="Path to moseq2-index.yaml for group definitions",
|
|
210
|
+
)
|
|
211
|
+
@click.option(
|
|
212
|
+
"--default-group",
|
|
213
|
+
type=str,
|
|
214
|
+
default="n/a",
|
|
215
|
+
help="Default group name to use for separate-trans",
|
|
216
|
+
)
|
|
217
|
+
@click.option(
|
|
218
|
+
"--verbose",
|
|
219
|
+
"-v",
|
|
220
|
+
is_flag=True,
|
|
221
|
+
default=False,
|
|
222
|
+
help="Print syllable log-likelihoods during training.",
|
|
223
|
+
)
|
|
224
|
+
def learn_model(input_file, dest_file, **config_data):
|
|
225
|
+
# Train the ARHMM using PC scores located in the INPUT_FILE, and saves the model to DEST_FILE
|
|
226
|
+
|
|
227
|
+
learn_model_wrapper(input_file, dest_file, config_data)
|
|
228
|
+
|
|
229
|
+
|
|
230
|
+
@cli.command(name="apply-model", help="Apply pre-trained ARHMM to PC scores.")
|
|
231
|
+
@click.argument("model_file", type=click.Path(exists=True))
|
|
232
|
+
@click.argument("pc_file", type=click.Path(exists=True))
|
|
233
|
+
@click.argument(
|
|
234
|
+
"dest_file", type=click.Path(file_okay=True, writable=True, resolve_path=True)
|
|
235
|
+
)
|
|
236
|
+
@click.option(
|
|
237
|
+
"--var-name",
|
|
238
|
+
type=str,
|
|
239
|
+
default="scores",
|
|
240
|
+
help="Variable name in input file with PCs",
|
|
241
|
+
)
|
|
242
|
+
@click.option(
|
|
243
|
+
"--index",
|
|
244
|
+
"-i",
|
|
245
|
+
type=click.Path(),
|
|
246
|
+
default="",
|
|
247
|
+
help="Path to moseq2-index.yaml for group definitions",
|
|
248
|
+
)
|
|
249
|
+
@click.option(
|
|
250
|
+
"--load-groups",
|
|
251
|
+
type=bool,
|
|
252
|
+
default=True,
|
|
253
|
+
help="If groups should be loaded with the PC scores.",
|
|
254
|
+
)
|
|
255
|
+
def apply_model(model_file, pc_file, dest_file, **config_data):
|
|
256
|
+
# Apply the ARHMM located in MODEL_FILE to the PC scores in PC_FILE, and saves the results to DEST_FILE
|
|
257
|
+
|
|
258
|
+
apply_model_wrapper(model_file, pc_file, dest_file, config_data)
|
|
259
|
+
|
|
260
|
+
|
|
261
|
+
@cli.command(
|
|
262
|
+
name="kappa-scan",
|
|
263
|
+
help="Batch train multiple model to scan over different kappa values.",
|
|
264
|
+
)
|
|
265
|
+
@click.argument("input_file", type=click.Path(exists=True))
|
|
266
|
+
@click.argument("output_dir", type=click.Path(exists=False))
|
|
267
|
+
@click.option(
|
|
268
|
+
"--index",
|
|
269
|
+
"-i",
|
|
270
|
+
type=click.Path(),
|
|
271
|
+
default="",
|
|
272
|
+
help="Path to moseq2-index.yaml for session metadata and group information",
|
|
273
|
+
)
|
|
274
|
+
@click.option(
|
|
275
|
+
"--out-script",
|
|
276
|
+
type=click.Path(),
|
|
277
|
+
default="train_out.sh",
|
|
278
|
+
help="Name of bash script file to save model training commands.",
|
|
279
|
+
)
|
|
280
|
+
@click.option(
|
|
281
|
+
"--n-models", type=int, default=10, help="Number of models to train in kappa scan."
|
|
282
|
+
)
|
|
283
|
+
@click.option(
|
|
284
|
+
"--prefix",
|
|
285
|
+
type=str,
|
|
286
|
+
default="",
|
|
287
|
+
help="Batch command string to prefix model training command (slurm only).",
|
|
288
|
+
)
|
|
289
|
+
@click.option(
|
|
290
|
+
"--cluster-type",
|
|
291
|
+
type=click.Choice(["local", "slurm"]),
|
|
292
|
+
default="local",
|
|
293
|
+
help="Platform to train models on",
|
|
294
|
+
)
|
|
295
|
+
@click.option(
|
|
296
|
+
"--scan-scale",
|
|
297
|
+
type=click.Choice(["log", "linear"]),
|
|
298
|
+
default="log",
|
|
299
|
+
help="Scale to scan kappa values at.",
|
|
300
|
+
)
|
|
301
|
+
@click.option(
|
|
302
|
+
"--min-kappa",
|
|
303
|
+
type=float,
|
|
304
|
+
default=None,
|
|
305
|
+
help="Minimum kappa value to begin scan from.",
|
|
306
|
+
)
|
|
307
|
+
@click.option(
|
|
308
|
+
"--max-kappa", type=float, default=None, help="Maximum kappa value to end scan on."
|
|
309
|
+
)
|
|
310
|
+
@click.option("--memory", type=str, default="5GB", help="RAM (slurm only)")
|
|
311
|
+
@click.option("--wall-time", type=str, default="3:00:00", help="Wall time (slurm only)")
|
|
312
|
+
@click.option(
|
|
313
|
+
"--partition", type=str, default="short", help="Partition name (slurm only)"
|
|
314
|
+
)
|
|
315
|
+
@click.option("--get-cmd", is_flag=True, default=False, help="Print scan command strings.")
|
|
316
|
+
@click.option("--run-cmd", is_flag=True, default=False, help="Run scan command strings.")
|
|
317
|
+
@modeling_parameters
|
|
318
|
+
def kappa_scan_fit_models(input_file, output_dir, **config_data):
|
|
319
|
+
# Scan through the kappa hyperparameter to find the kappa that best matches the changepoint duration distribution.
|
|
320
|
+
|
|
321
|
+
config_data["out_script"] = join(output_dir, config_data["out_script"])
|
|
322
|
+
kappa_scan_fit_models_wrapper(input_file, config_data, output_dir)
|
|
323
|
+
|
|
324
|
+
|
|
325
|
+
if __name__ == "__main__":
|
|
326
|
+
cli()
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
"""
|
|
2
|
+
GUI front-end functions for training ARHMM.
|
|
3
|
+
"""
|
|
4
|
+
|
|
5
|
+
from moseq2_model.cli import learn_model, kappa_scan_fit_models
|
|
6
|
+
from os.path import dirname, join, exists
|
|
7
|
+
from moseq2_model.helpers.wrappers import (
|
|
8
|
+
learn_model_wrapper,
|
|
9
|
+
kappa_scan_fit_models_wrapper,
|
|
10
|
+
apply_model_wrapper,
|
|
11
|
+
)
|
|
12
|
+
from moseq2_model._yaml import read_yaml
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
def learn_model_command(progress_paths, get_cmd=True, verbose=False):
|
|
16
|
+
"""
|
|
17
|
+
Train ARHMM from within a Jupyter notebook using parameters specified in the notebook.
|
|
18
|
+
|
|
19
|
+
Args:
|
|
20
|
+
progress_paths (dict): notebook progress dict that contains paths to the pc scores, config, and index files.
|
|
21
|
+
get_cmd (bool): flag to return the kappa scan learn-model commands.
|
|
22
|
+
verbose (bool): compute modeling summary - can slow down training.
|
|
23
|
+
|
|
24
|
+
Returns:
|
|
25
|
+
None or kappa scan command
|
|
26
|
+
"""
|
|
27
|
+
|
|
28
|
+
# Load proper input variables
|
|
29
|
+
input_file = progress_paths["scores_path"]
|
|
30
|
+
dest_file = progress_paths["model_path"]
|
|
31
|
+
config_file = progress_paths["config_file"]
|
|
32
|
+
index = progress_paths["index_file"]
|
|
33
|
+
output_dir = progress_paths["base_model_path"]
|
|
34
|
+
|
|
35
|
+
assert exists(
|
|
36
|
+
input_file
|
|
37
|
+
), "PCA Score not found; set the correct path in progress_paths['scores_path']"
|
|
38
|
+
assert exists(
|
|
39
|
+
config_file
|
|
40
|
+
), "Config file not found; set the correct path in progress_paths['config_file']"
|
|
41
|
+
assert exists(
|
|
42
|
+
index
|
|
43
|
+
), "Index file not found; set the correct path in progress_paths['index_file']"
|
|
44
|
+
|
|
45
|
+
with open(config_file, "r") as f:
|
|
46
|
+
config_data = read_yaml(f)
|
|
47
|
+
|
|
48
|
+
# Get default CLI params
|
|
49
|
+
params = {tmp.name: tmp.default for tmp in learn_model.params if not tmp.required}
|
|
50
|
+
# merge default params and config data, preferring values in config data
|
|
51
|
+
config_data = {**params, **config_data}
|
|
52
|
+
|
|
53
|
+
config_data["verbose"] = verbose
|
|
54
|
+
config_data["index"] = index
|
|
55
|
+
|
|
56
|
+
if output_dir is None:
|
|
57
|
+
print("Output directory not specified, saving models to base directory.")
|
|
58
|
+
output_dir = dirname(index)
|
|
59
|
+
|
|
60
|
+
if config_data["kappa"] == "scan":
|
|
61
|
+
assert config_data["scan_scale"] in (
|
|
62
|
+
"log",
|
|
63
|
+
"linear",
|
|
64
|
+
), 'scan scale must be "log" or "linear"'
|
|
65
|
+
|
|
66
|
+
# Get default CLI params
|
|
67
|
+
params = {
|
|
68
|
+
tmp.name: tmp.default
|
|
69
|
+
for tmp in kappa_scan_fit_models.params
|
|
70
|
+
if not tmp.required
|
|
71
|
+
}
|
|
72
|
+
# merge default params and config data, preferring values in config data
|
|
73
|
+
config_data = {**params, **config_data}
|
|
74
|
+
|
|
75
|
+
config_data["out_script"] = join(output_dir, config_data["out_script"])
|
|
76
|
+
config_data["get_cmd"] = get_cmd
|
|
77
|
+
|
|
78
|
+
command = kappa_scan_fit_models_wrapper(input_file, config_data, output_dir)
|
|
79
|
+
return command
|
|
80
|
+
else:
|
|
81
|
+
learn_model_wrapper(input_file, dest_file, config_data)
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def apply_model_command(progress_paths, model_file):
|
|
85
|
+
"""Apply a pre-trained ARHMM to a new dataset from within a Jupyter notebook.
|
|
86
|
+
|
|
87
|
+
Args:
|
|
88
|
+
progress_paths (dict): notebook progress dict that contains paths to the pc scores, config, and index files.
|
|
89
|
+
model_file (str): path to the pre-trained ARHMM.
|
|
90
|
+
"""
|
|
91
|
+
|
|
92
|
+
# Load proper input variables
|
|
93
|
+
pc_file = progress_paths["scores_path"]
|
|
94
|
+
dest_file = progress_paths["model_path"]
|
|
95
|
+
config_file = progress_paths["config_file"]
|
|
96
|
+
index = progress_paths["index_file"]
|
|
97
|
+
output_dir = progress_paths["base_model_path"]
|
|
98
|
+
|
|
99
|
+
# load config data
|
|
100
|
+
with open(config_file, "r") as f:
|
|
101
|
+
config_data = read_yaml(f)
|
|
102
|
+
|
|
103
|
+
# apply model to data
|
|
104
|
+
apply_model_wrapper(model_file, pc_file, dest_file, config_data)
|
|
File without changes
|