actirhythm-toolkit 0.1.0__tar.gz

actirhythm_toolkit-0.1.0/PKG-INFO

@@ -0,0 +1,477 @@
+Metadata-Version: 2.4
+Name: actirhythm-toolkit
+Version: 0.1.0
+Summary: Reproducible accelerometer analysis pipeline for circadian and behavioral rhythm studies
+Home-page: https://github.com/nerminjukan/masters-thesis
+Author: Nermin Jukan
+License: MIT
+Project-URL: Source, https://github.com/nerminjukan/masters-thesis
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: scipy>=1.7.0
+Requires-Dist: pyarrow>=5.0.0
+Requires-Dist: matplotlib>=3.4.0
+Requires-Dist: seaborn>=0.11.0
+Requires-Dist: statsmodels>=0.13.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: hmmlearn>=0.2.7
+Requires-Dist: pyyaml>=5.4.0
+Requires-Dist: loguru>=0.5.3
+Provides-Extra: ml
+Requires-Dist: xgboost>=1.5.0; extra == "ml"
+Requires-Dist: pomegranate>=0.14.8; extra == "ml"
+Requires-Dist: CosinorPy>=1.1; extra == "ml"
+Provides-Extra: glmm
+Requires-Dist: pymer4>=0.7.0; extra == "glmm"
+Requires-Dist: polars>=1.0.0; extra == "glmm"
+Requires-Dist: rpy2>=3.6.0; extra == "glmm"
+Requires-Dist: great-tables>=0.23.0; extra == "glmm"
+Provides-Extra: notebooks
+Requires-Dist: jupyter>=1.0.0; extra == "notebooks"
+Requires-Dist: ipykernel>=6.0.0; extra == "notebooks"
+Requires-Dist: nbformat>=5.1.0; extra == "notebooks"
+Requires-Dist: tqdm>=4.62.0; extra == "notebooks"
+Provides-Extra: dev
+Requires-Dist: pytest>=6.2.0; extra == "dev"
+Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
+Requires-Dist: python-dotenv>=0.19.0; extra == "dev"
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Masters Thesis Analysis Pipeline
+
+A reproducible Python-based data analysis pipeline for thesis research, focusing on time series analysis, hidden Markov models, cosinor analysis, and machine learning.
+
+**Designed for accelerometer data analysis** with support for 3-axis accelerometer readings and activity metrics.
+
+## ⚠️ Important: Methodological Corrections
+
+**This pipeline has been updated with critical methodological corrections** for biological and statistical validity. If you're using this for animal activity rhythm analysis:
+
+1. **READ FIRST**: [`METHODOLOGY_CORRECTIONS.md`](METHODOLOGY_CORRECTIONS.md) - Detailed explanation of all corrections
+2. **QUICK START**: [`QUICK_START.md`](QUICK_START.md) - Quick reference guide for the corrected pipeline
+
+### Key Corrections Implemented
+- ✅ Feature standardization to prevent HMM state flickering
+- ✅ Biological state validation (replaces unreliable AIC/BIC selection)
+- ✅ Minimum dwell-time filtering for realistic behavioral dynamics
+- ✅ State-based cosinor analysis (correct approach - applied AFTER state identification)
+- ✅ Per-individual feature computation for multi-subject studies
+- ✅ Complete documentation of methodological choices
+
+**Previous pipeline order (incorrect):** Raw → Features → Cosinor → HMM  
+**Corrected pipeline order:** Raw → Features → Standardization → HMM → Dwell-time Filter → Biological Validation → State-Based Cosinor
+
+See [`METHODOLOGY_CORRECTIONS.md`](METHODOLOGY_CORRECTIONS.md) for full details.
+
+---
+
+## Data Format
+
+The pipeline is configured to work with accelerometer data in CSV format with the following columns:
+
+- **ActMindata**: Tag-derived activity metric (PRIMARY ACTIVITY SIGNAL)
+- **XAccel, YAccel, ZAccel**: 3-axis accelerometer readings (UNSIGNED RELATIVE VALUES, 0-255)
+- **PostChg**: Posture change indicator
+- **PostCt**: Posture count
+- **ACTEndTimeAllS**: Timestamp of the reading (e.g., "7/8/23 10:46")
+- **serial**: Device serial number
+- **subject**: Subject identifier
+
+### IMPORTANT: Accelerometer Data Assumptions
+
+**CRITICAL**: XAccel/YAccel/ZAccel are **UNSIGNED RELATIVE VALUES** (typically 0-255), NOT calibrated acceleration in physical units.
+
+**What this means**:
+- Absolute axis values encode device orientation + gravity (NOT activity intensity)
+- DO NOT compute VeDBA, ODBA, or raw magnitude: `sqrt(X² + Y² + Z²)` is INCORRECT
+- DO NOT square or directly combine axes assuming zero-centered data
+- USE ActMindata as your primary activity signal (validated by tag manufacturer)
+- DERIVE relative movement features from axes (variance, std, absolute differences)
+
+**Correct workflow**:
+```python
+# CORRECT: Use ActMindata as primary signal
+df['activity'] = df['ActMindata']
+
+# CORRECT: Add relative movement features from axes
+df = features.add_accelerometer_movement_features(df, subject_column='subject')
+
+# INCORRECT: Do not compute magnitude from unsigned axes
+# df['activity'] = sqrt(XAccel² + YAccel² + ZAccel²)  # WRONG!
+```
+
+A sample dataset is provided in `data/raw/sample_accelerometer_data.csv`.
+
+## Project Structure
+
+```
+masters-thesis/
+├── data/
+│   ├── raw/              # Raw data files (not committed to git)
+│   └── processed/        # Processed data files (not committed to git)
+├── notebooks/            # Jupyter notebooks for analysis
+│   ├── 00_intake.ipynb           # Data intake
+│   ├── 01_qc_eda.ipynb          # Quality control and EDA
+│   ├── 02_features.ipynb         # Feature engineering (⚠️ WITH STANDARDIZATION)
+│   ├── 03_hmm_hsmm.ipynb        # HMM/HSMM modeling (⚠️ BIOLOGICAL VALIDATION)
+│   ├── 04_cosinor.ipynb         # Cosinor analysis (⚠️ STATE-BASED)
+│   └── 05_glmm_ml.ipynb         # GLMM and ML models
+├── src/                  # Source code modules
+│   ├── __init__.py
+│   ├── io.py            # Data I/O functions
+│   ├── qc.py            # Quality control
+│   ├── features.py      # Feature engineering (⚠️ NEW: standardization, log transform)
+│   ├── models_hmm.py    # HMM/HSMM (⚠️ NEW: dwell-time filter, biological validation)
+│   ├── cosinor.py       # Cosinor analysis (⚠️ NEW: state-based methods)
+│   ├── effects.py       # GLMM and ML models
+│   └── eval.py          # Evaluation utilities
+├── outputs/             # Generated outputs (plots, results)
+├── logs/                # Log files
+├── docs/                # Documentation
+├── METHODOLOGY_CORRECTIONS.md  # ⚠️ CRITICAL: Read this for methodological details
+├── QUICK_START.md              # Quick reference guide
+├── requirements.txt     # Python dependencies
+├── setup.py            # Package setup
+└── README.md           # This file
+```
+
+⚠️ = Contains critical methodological corrections
+
+## Features
+
+### Reproducible Pipeline
+
+The pipeline implements the following workflow:
+
+1. **Data Intake** (`00_intake.ipynb`)
+   - Load accelerometer data from CSV files
+   - Use ActMindata as primary activity signal (NOT magnitude from axes)
+   - Parse timestamps
+   - Initial data exploration
+   - Logging of parameters and random seeds
+
+2. **Quality Control** (`01_qc_eda.ipynb`)
+   - Detection of missing values
+   - Detection and removal of duplicates
+   - Time gap analysis
+   - Exploratory data analysis with visualizations
+
+3. **Feature Engineering** (`02_features.ipynb`)
+   - Use ActMindata as primary activity signal (NOT axis magnitude)
+   - Add relative movement features from accelerometer axes (variance, std, changes)
+   - Rolling statistics (mean, std, min, max) per individual
+   - Time-based features (hour, day of week, etc.)
+   - Log transformation and z-score standardization per individual
+   - Lag and difference features
+
+4. **HMM/HSMM Analysis** (`03_hmm_hsmm.ipynb`)
+   - Input: ActMin + relative movement features (standardized)
+   - Gaussian Hidden Markov Models with K=2..5 states
+   - Minimum dwell-time filtering to prevent state flickering
+   - Biological state validation (not just IC-based selection)
+   - State labeling and interpretation
+   - State sequence prediction (Viterbi algorithm)
+   - Posterior probability estimation
+
+5. **Cosinor Analysis** (`04_cosinor.ipynb`)
+   - Applied AFTER behavioral state identification (CORRECT approach)
+   - State-based cosinor: probability of active states
+   - Per-state cosinor: state-conditioned activity rhythms
+   - MESOR (mean level) estimation per state
+   - Amplitude and acrophase extraction per state
+   - Multi-period analysis
+   - Statistical significance testing
+
+6. **GLMM and Machine Learning** (`05_glmm_ml.ipynb`)
+   - Generalized Linear Mixed Models (GLMM)
+   - Random Forest regression/classification
+   - XGBoost models
+   - Model comparison and evaluation
+   - Feature importance analysis
+
+### Key Modules
+
+- **io.py**: Functions for loading and saving data in multiple formats
+  - `load_accelerometer_data()`: Load accelerometer CSV data with format validation
+  - `load_raw_data()`: Generic loader for CSV, Excel, Parquet, JSON, Feather
+  - `save_processed_data()`: Save processed data
+- **qc.py**: Quality control checks (gaps, duplicates, validation)
+- **features.py**: Feature engineering utilities
+  - `add_accelerometer_movement_features()`: Compute relative movement features (variance, std, changes) from XAccel/YAccel/ZAccel
+  - `add_rolling_statistics_per_individual()`: Rolling window features per subject
+  - `standardize_features()`: Z-score normalization per individual (CRITICAL for HMM)
+  - `log_transform_activity()`: Log transform for count data
+  - `add_time_features()`: Extract time-based features
+  - ~~`calculate_accelerometer_activity()`~~: DEPRECATED - do not use for unsigned axes
+- **models_hmm.py**: HMM/HSMM implementation with model selection
+  - `fit_hmm_pipeline()`: Complete HMM workflow
+  - `apply_minimum_dwell_time()`: Post-process to prevent state flickering
+  - `analyze_state_characteristics()`: Biological validation of states
+  - `select_states_biologically()`: State selection based on interpretability
+- **cosinor.py**: Cosinor analysis for circadian rhythms
+  - `fit_cosinor_per_state()`: State-based cosinor (CORRECT approach)
+  - `fit_cosinor_per_individual_per_state()`: Per-individual state-based analysis
+  - `fit_cosinor()`: Basic cosinor fitting
+- **effects.py**: Mixed effects models and machine learning
+- **eval.py**: Evaluation metrics and visualization utilities
+
+## Installation
+
+1. Clone the repository:
+```bash
+git clone https://github.com/nerminjukan/masters-thesis.git
+cd masters-thesis
+```
+
+2. Create a virtual environment:
+```bash
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+3. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+Or install as a package:
+```bash
+pip install -e .
+```
+
+Optional extras:
+
+```bash
+pip install -e .[dev]
+pip install -e .[notebooks]
+pip install -e .[ml]
+pip install -e .[glmm]
+```
+
+Validated packaging states:
+
+```bash
+# Base CLI/runtime only
+pip install -e .
+
+# Add optional machine-learning/research extras
+pip install -e .[ml]
+
+# Add GLMM support used by optional pymer4 workflows
+pip install -e .[glmm]
+```
+
+GLMM note:
+
+```text
+The Python extra installs the pymer4-side Python dependencies, but a working R installation
+and required R packages are still needed for pymer4-backed models. In a clean Windows test
+environment, pymer4 additionally required R packages such as tibble.
+```
+
+You can diagnose the GLMM environment with:
+
+```bash
+actirhythm glmm-doctor
+```
+
+### Run as an Installed Package (CLI)
+
+After installation, run the full workflow from the project root with a single command:
+
+```bash
+actirhythm
+```
+
+Equivalent explicit commands:
+
+```bash
+actirhythm full --run-version v3
+actirhythm run --run-version v3
+```
+
+Run stages separately:
+
+```bash
+actirhythm preprocess --run-version v3
+actirhythm analytics --run-version v3
+```
+
+Advanced options:
+
+```bash
+# Show what will run and where outputs will go (no execution)
+actirhythm full --run-version v4 --dry-run
+
+# Resume safely: skip stages whose expected outputs already exist
+actirhythm full --run-version v4 --skip-existing
+
+# Override input/output locations
+actirhythm preprocess \
+   --data-revised-dir ./my-revised-data \
+   --config ./config.yaml \
+   --raw-data-file ./data/raw/fallback.csv \
+   --processed-base-dir ./custom-processed \
+   --output-base-dir ./custom-runs \
+   --run-version experiment-01
+
+# Run analytics from an explicit HMM parquet file
+actirhythm analytics \
+   --hmm-input-file ./custom-processed/experiment-01/hmm_results_improved.parquet \
+   --output-base-dir ./custom-runs \
+   --run-version experiment-01
+
+# Use a non-default config file
+actirhythm full --project-root /path/to/masters-thesis --config ./config.custom.yaml --dry-run
+
+# Check optional GLMM prerequisites on the current machine
+actirhythm glmm-doctor
+```
+
+Release validation:
+
+```bash
+python -m build
+python -m twine check dist/*
+```
+
+Run from another folder by passing the project root:
+
+```bash
+actirhythm full --project-root /path/to/masters-thesis --run-version v3
+```
+
+## Usage
+
+### Running the Pipeline
+
+Execute notebooks in sequence:
+
+```bash
+cd notebooks
+jupyter notebook
+```
+
+Run notebooks in order (00 → 01 → 02 → 03 → 04 → 05).
+
+### Using Modules in Code
+
+```python
+from src import io, qc, features, models_hmm, cosinor, effects, eval
+
+# Load accelerometer data
+df = io.load_accelerometer_data('data/raw/sample_accelerometer_data.csv')
+
+# CORRECT: Use ActMindata as primary activity signal
+df['activity'] = df['ActMindata']
+
+# CORRECT: Add relative movement features from accelerometer axes
+# These capture variance and changes, NOT absolute magnitude
+df = features.add_accelerometer_movement_features(
+    df, 
+    subject_column='subject',
+    windows=[6, 12, 24],
+    standardize=True
+)
+
+# Quality control
+qc_report = qc.qc_report(df, time_column='timestamp')
+
+# Feature engineering - use ActMin, NOT magnitude
+df_features = features.add_rolling_statistics_per_individual(
+    df, 
+    'activity',  # ActMin
+    subject_column='subject',
+    windows=[6, 12, 24]
+)
+df_features = features.add_time_features(df_features, 'timestamp')
+
+# Standardize features before HMM (CRITICAL)
+df_features = features.log_transform_activity(df_features, 'activity', offset=1.0)
+df_features = features.standardize_features(
+    df_features, 
+    ['activity_log'],
+    group_by='subject'
+)
+
+# HMM analysis with standardized features
+feature_cols = ['activity_log_standardized']
+# Add standardized movement features
+movement_features = [col for col in df_features.columns if 'movement_var' in col and '_standardized' in col]
+feature_cols.extend(movement_features[:3])
+
+X = df_features[feature_cols].dropna().values
+hmm_results = models_hmm.fit_hmm_pipeline(X, state_range=range(2, 6))
+
+# Apply dwell-time filter
+states_filtered = models_hmm.apply_minimum_dwell_time(
+    hmm_results['states'], 
+    min_dwell=4
+)
+
+# State-based cosinor (CORRECT - applied AFTER state identification)
+df_with_states = df.copy()
+df_with_states['hmm_state'] = states_filtered
+cosinor_results = cosinor.fit_cosinor_per_state(
+    df_with_states,
+    time_column='timestamp',
+    state_column='hmm_state',
+    activity_column='activity'
+)
+
+# Machine learning
+rf_results = effects.fit_random_forest(X_train, y_train, X_test, y_test)
+```
+
+## Reproducibility
+
+All analyses use fixed random seeds (default: 42) for reproducibility. Parameters and seeds are logged in each notebook for full traceability.
+
+## Dependencies
+
+Core dependencies:
+- numpy, pandas, scipy
+- matplotlib, seaborn
+- scikit-learn, xgboost
+- hmmlearn, pomegranate
+- statsmodels, pymer4
+- jupyter, loguru
+
+See `requirements.txt` for complete list.
+
+## Contributing
+
+This is a thesis project. For questions or suggestions, please open an issue.
+
+## License
+
+This project is part of a master's thesis research.
+
+## Author
+
+Masters Thesis Author
+
+## Citation
+
+If you use this pipeline in your research, please cite appropriately.