cehrgpt 0.1.1__tar.gz → 0.1.3__tar.gz

cehrgpt-0.1.3/PKG-INFO

@@ -0,0 +1,238 @@
+Metadata-Version: 2.4
+Name: cehrgpt
+Version: 0.1.3
+Summary: CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines
+Author-email: Chao Pang <chaopang229@gmail.com>, Xinzhuo Jiang <xj2193@cumc.columbia.edu>, Krishna Kalluri <kk3326@cumc.columbia.edu>, Elise Minto <em3697@cumc.columbia.edu>, Jason Patterson <jp3477@cumc.columbia.edu>, Nishanth Parameshwar Pavinkurve <np2689@cumc.columbia.edu>, Karthik Natarajan <kn2174@cumc.columbia.edu>
+License: MIT License
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10.0
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cehrbert>=1.4.8
+Requires-Dist: cehrbert_data==0.0.11
+Requires-Dist: openai==1.54.3
+Requires-Dist: optuna==4.0.0
+Requires-Dist: transformers==4.44.1
+Requires-Dist: tokenizers==0.19.0
+Requires-Dist: peft==0.10.0
+Requires-Dist: lightgbm
+Requires-Dist: polars
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-subtests; extra == "dev"
+Requires-Dist: rootutils; extra == "dev"
+Requires-Dist: hypothesis; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Provides-Extra: flash-attn
+Requires-Dist: flash_attn; extra == "flash-attn"
+Dynamic: license-file
+
+# CEHRGPT
+
+[![PyPI - Version](https://img.shields.io/pypi/v/cehrgpt)](https://pypi.org/project/cehrgpt/)
+![Python](https://img.shields.io/badge/-Python_3.11-blue?logo=python&logoColor=white)
+[![tests](https://github.com/knatarajan-lab/cehrgpt/actions/workflows/tests.yaml/badge.svg)](https://github.com/knatarajan-lab/cehrgpt/actions/workflows/tests.yaml)
+[![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](https://github.com/knatarajan-lab/cehrgpt/blob/main/LICENSE)
+[![contributors](https://img.shields.io/github/contributors/knatarajan-lab/cehrgpt.svg)](https://github.com/knatarajan-lab/cehrgpt/graphs/contributors)
+
+CEHRGPT is a multi-task foundation model for structured electronic health records (EHR) data that supports three capabilities: feature representation, zero-shot prediction, and synthetic data generation.
+
+## 🎯 Key Capabilities
+
+### Feature Representation
+Extract meaningful patient embeddings from sequences of medical events using **linear probing** techniques for downstream tasks such as disease prediction, patient clustering, and risk stratification.
+
+### Zero-Shot Prediction
+Generate outcome predictions directly from prompts without requiring task-specific training, enabling rapid evaluation in low-label clinical settings.
+
+### Synthetic Data Generation
+Generate comprehensive patient profiles including demographics, medical history, treatment courses, and outcomes while implementing advanced privacy-preserving techniques to ensure generated data contains no identifiable information.
+The platform is fully compatible with the OMOP Common Data Model for seamless integration with existing healthcare systems.
+## 🚀 Installation
+
+Clone the repository and install dependencies:
+
+```bash
+git clone https://github.com/knatarajan-lab/cehrgpt.git
+cd cehrgpt
+pip install .
+```
+
+## 📋 Prerequisites
+
+Before getting started, set up the required environment variables:
+
+```bash
+export CEHRGPT_HOME=$(git rev-parse --show-toplevel)
+export OMOP_DIR=""                    # Path to your OMOP data
+export CEHR_GPT_DATA_DIR=""          # Path for processed data storage
+export CEHR_GPT_MODEL_DIR=""         # Path for model storage
+```
+
+Create the dataset cache directory:
+```bash
+mkdir $CEHR_GPT_DATA_DIR/dataset_prepared
+```
+
+## 🏗️ Model Training
+
+### Step 1: Generate Pre-training Data from OMOP
+
+Generate the training data following the [Data Generation Instruction](./data_generation.md).
+
+### Step 2: Pre-train CEHR-GPT
+
+Train the foundation model:
+
+```bash
+python -u -m cehrgpt.runners.hf_cehrgpt_pretrain_runner \
+  --model_name_or_path $CEHR_GPT_MODEL_DIR \
+  --tokenizer_name_or_path $CEHR_GPT_MODEL_DIR \
+  --output_dir $CEHR_GPT_MODEL_DIR \
+  --data_folder "$CEHR_GPT_DATA_DIR/patient_sequence/train" \
+  --dataset_prepared_path "$CEHR_GPT_DATA_DIR/dataset_prepared" \
+  --do_train true --seed 42 \
+  --dataloader_num_workers 16 --dataloader_prefetch_factor 8 \
+  --hidden_size 768 --num_hidden_layers 14 --max_position_embeddings 4096 \
+  --evaluation_strategy epoch --save_strategy epoch \
+  --sample_packing --max_tokens_per_batch 16384 \
+  --warmup_ratio 0.01 --weight_decay 0.01 \
+  --num_train_epochs 50 --learning_rate 0.0002 \
+  --use_early_stopping --early_stopping_threshold 0.001
+```
+
+> **Tip**: Increase `max_position_embeddings` for longer context windows based on your use case.
+
+## 🎯 Feature Representation
+
+CEHR-GPT enables extraction of meaningful patient embeddings from medical event sequences using **linear probing** techniques for downstream prediction tasks. The feature representation pipeline includes label generation, patient sequence extraction, and linear regression model training on the extracted representations.
+
+For detailed instructions including cohort creation, patient feature extraction, and linear probing evaluation, please follow the [Feature Representation Guide](./feature_representation.md).
+
+## 🔮 Zero-Shot Prediction
+
+CEHR-GPT can generate outcome predictions directly from clinical prompts without requiring task-specific training, making it ideal for rapid evaluation in low-label clinical settings. The zero-shot prediction capability performs time-to-event analysis by processing patient sequences and generating risk predictions based on learned medical patterns.
+
+For complete setup instructions including label generation, sequence preparation, and prediction execution, please follow the [Zero-Shot Prediction Guide](./zero_shot_prediction.md).
+
+## 🧬 Synthetic Data Generation
+
+CEHR-GPT generates comprehensive synthetic patient profiles including demographics, medical history, treatment courses, and outcomes while implementing advanced privacy-preserving techniques. The synthetic data maintains statistical fidelity to real patient populations without containing identifiable information, and outputs are fully compatible with the OMOP Common Data Model.
+
+For step-by-step instructions on generating synthetic sequences and converting them to OMOP format, please follow the [Synthetic Data Generation Guide](./synthetic_data_generation.md).
+
+## 📊 MEDS Support
+
+CEHR-GPT supports the Medical Event Data Standard (MEDS) format for enhanced interoperability.
+
+### Prerequisites
+
+Configure MEDS-specific environment variables:
+
+```bash
+export CEHR_GPT_MODEL_DIR=""    # CEHR-GPT model directory
+export MEDS_DIR=""              # MEDS data directory
+export MEDS_READER_DIR=""       # MEDS reader output directory
+```
+
+### Step 1: Create MIMIC MEDS Data
+
+Transform MIMIC files to MEDS format following the [MEDS_transforms](https://github.com/mmcdermott/MEDS_transforms/) repository instructions.
+
+### Step 2: Prepare MEDS Reader
+
+Convert MEDS data for CEHR-GPT compatibility:
+
+```bash
+meds_reader_convert $MEDS_DIR $MEDS_READER_DIR --num_threads 10
+```
+
+### Step 3: Pre-train with MEDS Data
+
+Execute pre-training using MEDS format:
+
+```bash
+python -u -m cehrgpt.runners.hf_cehrgpt_pretrain_runner \
+  --model_name_or_path $CEHR_GPT_MODEL_DIR \
+  --tokenizer_name_or_path $CEHR_GPT_MODEL_DIR \
+  --output_dir $CEHR_GPT_MODEL_DIR \
+  --data_folder $MEDS_READER_DIR \
+  --dataset_prepared_path "$CEHR_GPT_MODEL_DIR/dataset_prepared" \
+  --do_train true --seed 42 \
+  --dataloader_num_workers 16 --dataloader_prefetch_factor 8 \
+  --hidden_size 768 --num_hidden_layers 14 --max_position_embeddings 8192 \
+  --evaluation_strategy epoch --save_strategy epoch \
+  --sample_packing --max_tokens_per_batch 16384 \
+  --warmup_steps 500 --weight_decay 0.01 \
+  --num_train_epochs 50 --learning_rate 0.0002 \
+  --use_early_stopping --early_stopping_threshold 0.001 \
+  --is_data_in_meds --inpatient_att_function_type day \
+  --att_function_type day --include_inpatient_hour_token \
+  --include_auxiliary_token --include_demographic_prompt \
+  --meds_to_cehrbert_conversion_type "MedsToBertMimic4"
+```
+
+### Step 4: Generate MEDS Trajectories
+
+#### Environment Setup
+
+Configure trajectory generation environment:
+
+```bash
+export MEDS_LABEL_COHORT_DIR=""     # Cohort labels directory (parquet files)
+export MEDS_TRAJECTORY_DIR=""       # Trajectory output directory
+```
+
+#### Generate Synthetic Trajectories
+
+Create patient trajectories with the trained model:
+
+```bash
+python -u -m cehrgpt.generation.cehrgpt_conditional_generation \
+  --cohort_folder $MEDS_LABEL_COHORT_DIR \
+  --data_folder $MEDS_READER_DIR \
+  --dataset_prepared_path "$CEHR_GPT_MODEL_DIR/dataset_prepared" \
+  --model_name_or_path $CEHR_GPT_MODEL_DIR \
+  --tokenizer_name_or_path $CEHR_GPT_MODEL_DIR \
+  --output_dir $MEDS_TRAJECTORY_DIR \
+  --per_device_eval_batch_size 16 \
+  --num_of_trajectories_per_sample 2 \
+  --generation_input_length 4096 \
+  --generation_max_new_tokens 4096 \
+  --is_data_in_meds \
+  --att_function_type day --inpatient_att_function_type day \
+  --meds_to_cehrbert_conversion_type MedsToBertMimic4 \
+  --include_auxiliary_token --include_demographic_prompt \
+  --include_inpatient_hour_token
+```
+
+> **Important**: Ensure `generation_input_length` + `generation_max_new_tokens` ≤ `max_position_embeddings` (8192).
+
+#### Parameter Reference
+
+- `generation_input_length`: Input context length for generation
+- `generation_max_new_tokens`: Maximum new tokens to generate
+- `num_of_trajectories_per_sample`: Number of trajectories per patient sample
+
+## 📖 Citation
+
+If you use CEHRGPT in your research, please cite:
+
+```bibtex
+@article{cehrgpt2024,
+  title={CEHRGPT: Synthetic Data Generation for Electronic Health Records},
+  author={Natarajan, K and others},
+  journal={arXiv preprint arXiv:2402.04400},
+  year={2024}
+}
+```
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

cehrgpt-0.1.3/README.md

@@ -0,0 +1,203 @@
+# CEHRGPT
+
+[![PyPI - Version](https://img.shields.io/pypi/v/cehrgpt)](https://pypi.org/project/cehrgpt/)
+![Python](https://img.shields.io/badge/-Python_3.11-blue?logo=python&logoColor=white)
+[![tests](https://github.com/knatarajan-lab/cehrgpt/actions/workflows/tests.yaml/badge.svg)](https://github.com/knatarajan-lab/cehrgpt/actions/workflows/tests.yaml)
+[![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](https://github.com/knatarajan-lab/cehrgpt/blob/main/LICENSE)
+[![contributors](https://img.shields.io/github/contributors/knatarajan-lab/cehrgpt.svg)](https://github.com/knatarajan-lab/cehrgpt/graphs/contributors)
+
+CEHRGPT is a multi-task foundation model for structured electronic health records (EHR) data that supports three capabilities: feature representation, zero-shot prediction, and synthetic data generation.
+
+## 🎯 Key Capabilities
+
+### Feature Representation
+Extract meaningful patient embeddings from sequences of medical events using **linear probing** techniques for downstream tasks such as disease prediction, patient clustering, and risk stratification.
+
+### Zero-Shot Prediction
+Generate outcome predictions directly from prompts without requiring task-specific training, enabling rapid evaluation in low-label clinical settings.
+
+### Synthetic Data Generation
+Generate comprehensive patient profiles including demographics, medical history, treatment courses, and outcomes while implementing advanced privacy-preserving techniques to ensure generated data contains no identifiable information.
+The platform is fully compatible with the OMOP Common Data Model for seamless integration with existing healthcare systems.
+## 🚀 Installation
+
+Clone the repository and install dependencies:
+
+```bash
+git clone https://github.com/knatarajan-lab/cehrgpt.git
+cd cehrgpt
+pip install .
+```
+
+## 📋 Prerequisites
+
+Before getting started, set up the required environment variables:
+
+```bash
+export CEHRGPT_HOME=$(git rev-parse --show-toplevel)
+export OMOP_DIR=""                    # Path to your OMOP data
+export CEHR_GPT_DATA_DIR=""          # Path for processed data storage
+export CEHR_GPT_MODEL_DIR=""         # Path for model storage
+```
+
+Create the dataset cache directory:
+```bash
+mkdir $CEHR_GPT_DATA_DIR/dataset_prepared
+```
+
+## 🏗️ Model Training
+
+### Step 1: Generate Pre-training Data from OMOP
+
+Generate the training data following the [Data Generation Instruction](./data_generation.md).
+
+### Step 2: Pre-train CEHR-GPT
+
+Train the foundation model:
+
+```bash
+python -u -m cehrgpt.runners.hf_cehrgpt_pretrain_runner \
+  --model_name_or_path $CEHR_GPT_MODEL_DIR \
+  --tokenizer_name_or_path $CEHR_GPT_MODEL_DIR \
+  --output_dir $CEHR_GPT_MODEL_DIR \
+  --data_folder "$CEHR_GPT_DATA_DIR/patient_sequence/train" \
+  --dataset_prepared_path "$CEHR_GPT_DATA_DIR/dataset_prepared" \
+  --do_train true --seed 42 \
+  --dataloader_num_workers 16 --dataloader_prefetch_factor 8 \
+  --hidden_size 768 --num_hidden_layers 14 --max_position_embeddings 4096 \
+  --evaluation_strategy epoch --save_strategy epoch \
+  --sample_packing --max_tokens_per_batch 16384 \
+  --warmup_ratio 0.01 --weight_decay 0.01 \
+  --num_train_epochs 50 --learning_rate 0.0002 \
+  --use_early_stopping --early_stopping_threshold 0.001
+```
+
+> **Tip**: Increase `max_position_embeddings` for longer context windows based on your use case.
+
+## 🎯 Feature Representation
+
+CEHR-GPT enables extraction of meaningful patient embeddings from medical event sequences using **linear probing** techniques for downstream prediction tasks. The feature representation pipeline includes label generation, patient sequence extraction, and linear regression model training on the extracted representations.
+
+For detailed instructions including cohort creation, patient feature extraction, and linear probing evaluation, please follow the [Feature Representation Guide](./feature_representation.md).
+
+## 🔮 Zero-Shot Prediction
+
+CEHR-GPT can generate outcome predictions directly from clinical prompts without requiring task-specific training, making it ideal for rapid evaluation in low-label clinical settings. The zero-shot prediction capability performs time-to-event analysis by processing patient sequences and generating risk predictions based on learned medical patterns.
+
+For complete setup instructions including label generation, sequence preparation, and prediction execution, please follow the [Zero-Shot Prediction Guide](./zero_shot_prediction.md).
+
+## 🧬 Synthetic Data Generation
+
+CEHR-GPT generates comprehensive synthetic patient profiles including demographics, medical history, treatment courses, and outcomes while implementing advanced privacy-preserving techniques. The synthetic data maintains statistical fidelity to real patient populations without containing identifiable information, and outputs are fully compatible with the OMOP Common Data Model.
+
+For step-by-step instructions on generating synthetic sequences and converting them to OMOP format, please follow the [Synthetic Data Generation Guide](./synthetic_data_generation.md).
+
+## 📊 MEDS Support
+
+CEHR-GPT supports the Medical Event Data Standard (MEDS) format for enhanced interoperability.
+
+### Prerequisites
+
+Configure MEDS-specific environment variables:
+
+```bash
+export CEHR_GPT_MODEL_DIR=""    # CEHR-GPT model directory
+export MEDS_DIR=""              # MEDS data directory
+export MEDS_READER_DIR=""       # MEDS reader output directory
+```
+
+### Step 1: Create MIMIC MEDS Data
+
+Transform MIMIC files to MEDS format following the [MEDS_transforms](https://github.com/mmcdermott/MEDS_transforms/) repository instructions.
+
+### Step 2: Prepare MEDS Reader
+
+Convert MEDS data for CEHR-GPT compatibility:
+
+```bash
+meds_reader_convert $MEDS_DIR $MEDS_READER_DIR --num_threads 10
+```
+
+### Step 3: Pre-train with MEDS Data
+
+Execute pre-training using MEDS format:
+
+```bash
+python -u -m cehrgpt.runners.hf_cehrgpt_pretrain_runner \
+  --model_name_or_path $CEHR_GPT_MODEL_DIR \
+  --tokenizer_name_or_path $CEHR_GPT_MODEL_DIR \
+  --output_dir $CEHR_GPT_MODEL_DIR \
+  --data_folder $MEDS_READER_DIR \
+  --dataset_prepared_path "$CEHR_GPT_MODEL_DIR/dataset_prepared" \
+  --do_train true --seed 42 \
+  --dataloader_num_workers 16 --dataloader_prefetch_factor 8 \
+  --hidden_size 768 --num_hidden_layers 14 --max_position_embeddings 8192 \
+  --evaluation_strategy epoch --save_strategy epoch \
+  --sample_packing --max_tokens_per_batch 16384 \
+  --warmup_steps 500 --weight_decay 0.01 \
+  --num_train_epochs 50 --learning_rate 0.0002 \
+  --use_early_stopping --early_stopping_threshold 0.001 \
+  --is_data_in_meds --inpatient_att_function_type day \
+  --att_function_type day --include_inpatient_hour_token \
+  --include_auxiliary_token --include_demographic_prompt \
+  --meds_to_cehrbert_conversion_type "MedsToBertMimic4"
+```
+
+### Step 4: Generate MEDS Trajectories
+
+#### Environment Setup
+
+Configure trajectory generation environment:
+
+```bash
+export MEDS_LABEL_COHORT_DIR=""     # Cohort labels directory (parquet files)
+export MEDS_TRAJECTORY_DIR=""       # Trajectory output directory
+```
+
+#### Generate Synthetic Trajectories
+
+Create patient trajectories with the trained model:
+
+```bash
+python -u -m cehrgpt.generation.cehrgpt_conditional_generation \
+  --cohort_folder $MEDS_LABEL_COHORT_DIR \
+  --data_folder $MEDS_READER_DIR \
+  --dataset_prepared_path "$CEHR_GPT_MODEL_DIR/dataset_prepared" \
+  --model_name_or_path $CEHR_GPT_MODEL_DIR \
+  --tokenizer_name_or_path $CEHR_GPT_MODEL_DIR \
+  --output_dir $MEDS_TRAJECTORY_DIR \
+  --per_device_eval_batch_size 16 \
+  --num_of_trajectories_per_sample 2 \
+  --generation_input_length 4096 \
+  --generation_max_new_tokens 4096 \
+  --is_data_in_meds \
+  --att_function_type day --inpatient_att_function_type day \
+  --meds_to_cehrbert_conversion_type MedsToBertMimic4 \
+  --include_auxiliary_token --include_demographic_prompt \
+  --include_inpatient_hour_token
+```
+
+> **Important**: Ensure `generation_input_length` + `generation_max_new_tokens` ≤ `max_position_embeddings` (8192).
+
+#### Parameter Reference
+
+- `generation_input_length`: Input context length for generation
+- `generation_max_new_tokens`: Maximum new tokens to generate
+- `num_of_trajectories_per_sample`: Number of trajectories per patient sample
+
+## 📖 Citation
+
+If you use CEHRGPT in your research, please cite:
+
+```bibtex
+@article{cehrgpt2024,
+  title={CEHRGPT: Synthetic Data Generation for Electronic Health Records},
+  author={Natarajan, K and others},
+  journal={arXiv preprint arXiv:2402.04400},
+  year={2024}
+}
+```
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

cehrgpt-0.1.3/constraints.txt

@@ -0,0 +1 @@
+xformers==0  # blocks installation

cehrgpt-0.1.3/data_generation.md

@@ -0,0 +1,119 @@
+# CEHR-GPT Data Generation
+
+This guide covers the process of generating pre-training data for CEHR-GPT from OMOP-formatted healthcare datasets.
+
+## Prerequisites
+
+Before starting data generation, ensure you have:
+
+1. **Spark Environment**: Configured Apache Spark (see [Spark Setup README](./spark_setup.md))
+2. **OMOP Data**: Healthcare data in OMOP Common Data Model format
+3. **Environment Variables**: Required paths and directories set up
+
+## Required Environment Variables
+
+Set up the necessary directory paths:
+
+```bash
+# CEHR-GPT installation directory
+export CEHRGPT_HOME=$(git rev-parse --show-toplevel)
+
+# OMOP input data directory
+export OMOP_DIR="/path/to/omop/data"
+
+# Output directory for processed data
+export CEHR_GPT_DATA_DIR="/path/to/output/data"
+```
+
+## Step 1: Configure Spark for Data Processing
+
+Set up Spark environment variables optimized for healthcare data processing:
+
+```bash
+# Worker configuration
+export SPARK_WORKER_INSTANCES="1"
+export SPARK_MASTER="local[16]"
+export SPARK_WORKER_CORES="16"
+export SPARK_EXECUTOR_CORES="4"
+
+# Memory configuration
+export SPARK_DRIVER_MEMORY="20g"
+export SPARK_EXECUTOR_MEMORY="20g"
+
+export SPARK_SUBMIT_OPTIONS="--master $SPARK_MASTER --driver-memory $SPARK_DRIVER_MEMORY --executor-memory $SPARK_EXECUTOR_MEMORY --executor-cores $SPARK_EXECUTOR_CORES"
+```
+
+### Configuration Guidelines
+
+**Memory Allocation:**
+- **Small datasets (< 1M patients)**: 8GB driver/executor memory
+- **Medium datasets (1-10M patients)**: 12-16GB driver/executor memory
+- **Large datasets (> 10M patients)**: 20-32GB driver/executor memory
+
+**Core Allocation:**
+- Adjust `SPARK_WORKER_CORES` based on available CPU cores
+- Keep `SPARK_EXECUTOR_CORES` at 2-4 for optimal performance
+- Reserve 2-4 cores for system processes
+
+## Step 2: Generate Pre-training Data
+
+Execute the data generation script:
+
+```bash
+sh $CEHRGPT_HOME/scripts/create_cehrgpt_pretraining_data.sh \
+  --input_folder $OMOP_DIR \
+  --output_folder $CEHR_GPT_DATA_DIR \
+  --start_date "1985-01-01"
+```
+
+### Script Parameters
+
+- `--input_folder`: Directory containing OMOP-formatted data files
+- `--output_folder`: Directory where processed data will be saved
+- `--start_date`: Earliest date for including patient records (format: YYYY-MM-DD)
+
+## Performance Optimization
+
+### For Large Datasets
+
+```bash
+# Increase parallelism
+export SPARK_SQL_SHUFFLE_PARTITIONS="800"
+
+# Enable dynamic allocation
+export SPARK_CONF_spark_dynamicAllocation_enabled="true"
+export SPARK_CONF_spark_dynamicAllocation_minExecutors="2"
+export SPARK_CONF_spark_dynamicAllocation_maxExecutors="20"
+```
+
+### Memory Optimization
+
+```bash
+# Tune garbage collection
+export SPARK_CONF_spark_executor_extraJavaOptions="-XX:+UseG1GC -XX:+PrintGCDetails"
+
+# Optimize serialization
+export SPARK_CONF_spark_serializer_objectStreamReset="100"
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**Out of Memory Errors:**
+```bash
+# Increase driver memory
+export SPARK_DRIVER_MEMORY="20g"
+
+# Increase executor memory
+export SPARK_EXECUTOR_MEMORY="16g"
+```
+
+**Slow Performance:**
+```bash
+# Increase parallelism
+export SPARK_WORKER_CORES="32"
+
+# Enable adaptive query execution
+export SPARK_CONF_spark_sql_adaptive_enabled="true"
+```

cehrgpt-0.1.3/feature_representation.md

@@ -0,0 +1,109 @@
+# CEHR-GPT Feature Representation using Linear Probing
+
+This guide covers the process of extracting meaningful patient embeddings from healthcare sequences using **linear probing** techniques for downstream prediction tasks such as disease prediction, patient clustering, and risk stratification.
+
+## Prerequisites
+
+Ensure you have:
+
+1. **Trained CEHR-GPT Model**: Pre-trained model available at `$CEHR_GPT_MODEL_DIR`
+2. **OMOP Data**: Healthcare data processed and ready for feature extraction
+3. **Environment Setup**: Required environment variables configured
+
+## Required Environment Variables
+
+Set up the necessary directory paths:
+
+```bash
+# CEHR-GPT installation directory (auto-detect from git repository)
+export CEHRGPT_HOME=$(git rev-parse --show-toplevel)
+export CEHR_GPT_MODEL_DIR="/path/to/trained/model"
+
+# Data directories
+export OMOP_DIR="/path/to/omop/data"
+export CEHR_GPT_DATA_DIR="/path/to/processed/data"
+export CEHRGPT_FEATURES_DIR="/path/to/extracted/features"
+```
+
+## Step 1: Generate Prediction Labels
+
+Create heart failure readmission labels compatible with MEDS schema for downstream prediction tasks:
+
+```bash
+python -u -m cehrbert_data.prediction_cohorts.hf_readmission \
+   -c hf_readmission -i $OMOP_DIR -o $OMOP_DIR/labels \
+   -dl 1985-01-01 -du 2023-12-31 \
+   -l 18 -u 100 -ow 730 -ps 1 -pw 30 \
+   --is_new_patient_representation \
+   --should_construct_artificial_visits \
+   --include_concept_list \
+   --is_remove_index_prediction_starts \
+   --meds_format \
+   --exclude_features
+```
+
+### Parameter Explanation
+
+- `-c hf_readmission`: Cohort name for heart failure readmission prediction
+- `-i $OMOP_DIR`: Input directory containing OMOP data
+- `-o $OMOP_DIR/labels`: Output directory for generated labels
+- `-dl/-du`: Date range for patient inclusion (1985-2023)
+- `-l 18 -u 100`: Age limits (18-100 years)
+- `-ow 730`: Observation window in days (2 years)
+- `-ps 1 -pw 30`: Prediction start (1 day) and window (30 days)
+- `--is_remove_index_prediction_starts`: Remove cases where outcome events occur before prediction start date
+- `--include_concept_list`: Include only concepts that are allowed in the model vocabulary
+- `--meds_format`: Output in MEDS-compatible format
+
+
+
+## Step 2: Extract Patient Features
+
+Extract patient sequences using a 2-year observation window, focusing on key clinical events:
+
+```bash
+sh $CEHRGPT_HOME/scripts/extract_features_gpt.sh \
+  --cohort-folder $OMOP_DIR/labels \
+  --input-dir $OMOP_DIR \
+  --output-dir "$CEHR_GPT_DATA_DIR/phenotype_cehrgpt_sequences" \
+  --patient-splits-folder "$OMOP_DIR/patient_splits" \
+  --ehr-tables "condition_occurrence procedure_occurrence drug_exposure" \
+  --observation-window 730
+```
+> **Tip**: This step requires pyspark, and please refer to **Spark Environment**: Configured Apache Spark (see [Spark Setup README](./spark_setup.md))
+
+### Key Parameters
+
+- `--cohort-folder`: Directory containing prediction labels
+- `--input-dir`: Source OMOP data directory
+- `--output-dir`: Output directory for extracted sequences
+- `--patient-splits-folder`: Pre-defined train/validation/test splits
+- `--ehr-tables`: Clinical tables to include in feature extraction
+- `--observation-window`: Observation period in days (730 = 2 years)
+
+
+
+## Step 3: Run Feature Extraction and Linear Probing
+
+Execute CEHR-GPT feature extraction and train a linear regression model on the extracted patient representations:
+
+```bash
+sh $CEHRGPT_HOME/run_cehrgpt.sh \
+  --base_dir="$CEHR_GPT_DATA_DIR/phenotype_cehrgpt_sequences" \
+  --dataset_prepared_path="$CEHR_GPT_DATA_DIR/dataset_prepared" \
+  --model_path=$CEHR_GPT_MODEL_DIR \
+  --output_dir=$CEHRGPT_FEATURES_DIR \
+  --preprocessing_workers=8 \
+  --model_name="cehrgpt"
+```
+
+This step performs both feature extraction from patient sequences and trains a linear regression model on the extracted patient representations for downstream prediction tasks.
+
+### Parameter Details
+
+- `--base_dir`: Directory containing prepared patient sequences
+- `--dataset_prepared_path`: Path for preprocessed datasets
+- `--model_path`: Location of trained CEHR-GPT model
+- `--output_dir`: Output directory for extracted features and embeddings
+- `--preprocessing_workers`: Number of parallel workers for data preprocessing
+- `--model_name`: Model identifier for feature extraction

{cehrgpt-0.1.1 → cehrgpt-0.1.3}/pyproject.toml

@@ -28,7 +28,7 @@ classifiers = [
 ]
 
 dependencies = [
-    "cehrbert==1.4.5",
+    "cehrbert>=1.4.8",
     "cehrbert_data==0.0.11",
     "openai==1.54.3",
     "optuna==4.0.0",

cehrgpt-0.1.3/sample_configs/credential_file_sample.ini

@@ -0,0 +1,5 @@
+[DEFAULT]
+base_url = jdbc:jtds:sqlserver://omop.dbmi.columbia.edu:1433;databaseName=your_database
+driver = net.sourceforge.jtds.jdbc.Driver
+user = username
+password = password