pyaging 0.0.6__tar.gz → 0.0.8__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of pyaging might be problematic. Click here for more details.

@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: pyaging
3
- Version: 0.0.6
3
+ Version: 0.0.8
4
4
  Summary: A Python-based compendium of GPU-optimized aging clocks.
5
5
  License: BSD
6
6
  Author: Lucas Paulo de Lima Camillo
@@ -48,6 +48,30 @@ Description-Content-Type: text/markdown
48
48
 
49
49
  With a growing number of aging clocks and biomarkers of aging, comparing and analyzing them can be challenging. `pyaging` simplifies this process, allowing researchers to input various molecular layers (DNA methylation, histone ChIP-Seq, ATAC-seq, transcriptomics, etc.) and quickly analyze them using multiple aging clocks, thanks to its GPU-backed infrastructure. This makes it an ideal tool for large datasets and multi-layered analysis.
50
50
 
51
+ ## ❓ Can't find an aging clock?
52
+
53
+ If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
54
+
55
+ ## 💬 Community Discussion
56
+ For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
57
+
58
+ ## 📖 Citation
59
+
60
+ To cite `pyaging`, please use the following:
61
+
62
+ ```
63
+ @article {de_Lima_Camillo_pyaging,
64
+ author = {Lucas Paulo de Lima Camillo},
65
+ title = {pyaging: a Python-based compendium of GPU-optimized aging clocks},
66
+ elocation-id = {2023.11.28.569069},
67
+ year = {2023},
68
+ doi = {10.1101/2023.11.28.569069},
69
+ publisher = {Cold Spring Harbor Laboratory},
70
+ URL = {https://www.biorxiv.org/content/early/2023/11/29/2023.11.28.569069},
71
+ journal = {bioRxiv}
72
+ }
73
+ ```
74
+
51
75
  ## 📝 To-Do List
52
76
 
53
77
  - [X] Incorporate more murine DNA methylation clocks
@@ -57,10 +81,4 @@ With a growing number of aging clocks and biomarkers of aging, comparing and ana
57
81
  - [ ] Integrate scAge clocks (this is proving to be difficult)
58
82
  - [ ] Integrate scRNAseq clocks (and datasets)
59
83
 
60
- ## ❓ Can't find an aging clock?
61
-
62
- If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
63
-
64
- ## 💬 Community Discussion
65
- For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
66
84
 
@@ -23,6 +23,30 @@
23
23
 
24
24
  With a growing number of aging clocks and biomarkers of aging, comparing and analyzing them can be challenging. `pyaging` simplifies this process, allowing researchers to input various molecular layers (DNA methylation, histone ChIP-Seq, ATAC-seq, transcriptomics, etc.) and quickly analyze them using multiple aging clocks, thanks to its GPU-backed infrastructure. This makes it an ideal tool for large datasets and multi-layered analysis.
25
25
 
26
+ ## ❓ Can't find an aging clock?
27
+
28
+ If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
29
+
30
+ ## 💬 Community Discussion
31
+ For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
32
+
33
+ ## 📖 Citation
34
+
35
+ To cite `pyaging`, please use the following:
36
+
37
+ ```
38
+ @article {de_Lima_Camillo_pyaging,
39
+ author = {Lucas Paulo de Lima Camillo},
40
+ title = {pyaging: a Python-based compendium of GPU-optimized aging clocks},
41
+ elocation-id = {2023.11.28.569069},
42
+ year = {2023},
43
+ doi = {10.1101/2023.11.28.569069},
44
+ publisher = {Cold Spring Harbor Laboratory},
45
+ URL = {https://www.biorxiv.org/content/early/2023/11/29/2023.11.28.569069},
46
+ journal = {bioRxiv}
47
+ }
48
+ ```
49
+
26
50
  ## 📝 To-Do List
27
51
 
28
52
  - [X] Incorporate more murine DNA methylation clocks
@@ -32,9 +56,3 @@ With a growing number of aging clocks and biomarkers of aging, comparing and ana
32
56
  - [ ] Integrate scAge clocks (this is proving to be difficult)
33
57
  - [ ] Integrate scRNAseq clocks (and datasets)
34
58
 
35
- ## ❓ Can't find an aging clock?
36
-
37
- If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
38
-
39
- ## 💬 Community Discussion
40
- For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
@@ -1,5 +1,7 @@
1
1
  import anndata
2
2
  import gc
3
+ import warnings
4
+ from anndata import ImplicitModificationWarning
3
5
 
4
6
  from ._pred_utils import *
5
7
 
@@ -64,9 +66,10 @@ def predict_age(
64
66
  silence_logger("predict_age")
65
67
  logger.first_info("Starting predict_age function")
66
68
 
67
- # Ensure clock_names is a list
69
+ # Ensure clock_names is a list with lowercase names
68
70
  if isinstance(clock_names, str):
69
71
  clock_names = [clock_names]
72
+ clock_names = [clock_name.lower() for clock_name in clock_names]
70
73
 
71
74
  # Set device for PyTorch operations
72
75
  device = set_torch_device(logger)
@@ -75,9 +78,9 @@ def predict_age(
75
78
  logger.info(f"🕒 Processing clock: {clock_name}", indent_level=1)
76
79
 
77
80
  # Load and prepare the clock
78
- clock_name = clock_name.lower()
79
81
  (
80
82
  features,
83
+ reference_feature_values,
81
84
  weight_dict,
82
85
  preprocessing,
83
86
  postprocessing,
@@ -86,30 +89,23 @@ def predict_age(
86
89
  ) = load_clock(clock_name, dir, logger, indent_level=2)
87
90
 
88
91
  # Check and update adata for missing features
89
- original_var_names = list(adata.var_names)
90
92
  adata = check_features_in_adata(
91
- adata, clock_name, features, logger, indent_level=2
93
+ adata,
94
+ clock_name,
95
+ features,
96
+ reference_feature_values,
97
+ logger,
98
+ indent_level=2,
92
99
  )
93
100
 
94
101
  # Apply preprocessing
95
- if preprocessing:
96
- adata = preprocess_data(
97
- adata,
98
- preprocessing,
99
- preprocessing_helper,
100
- features,
101
- logger,
102
- indent_level=2,
103
- )
104
-
105
- # Filter features and then extract data matrix
106
- x_numpy = filter_features_and_extract_data(
107
- adata, preprocessing, features, logger, indent_level=2
108
- )
109
-
110
- # Convert numpy array to tensor
111
- x_tensor = convert_numpy_array_to_tensor(
112
- x_numpy, device, logger, indent_level=2
102
+ adata = preprocess_data(
103
+ adata,
104
+ preprocessing,
105
+ preprocessing_helper,
106
+ features,
107
+ logger,
108
+ indent_level=2,
113
109
  )
114
110
 
115
111
  # Initialize and configure the model
@@ -119,37 +115,31 @@ def predict_age(
119
115
 
120
116
  # Perform age prediction using the model
121
117
  predicted_ages_tensor = predict_ages_with_model(
122
- clock_model, x_tensor, logger, indent_level=2
118
+ clock_model, adata, features, device, logger, indent_level=2
123
119
  )
124
120
 
125
- # Convert adata tensor to numpy array
121
+ # Convert torch tensor to numpy array
126
122
  predicted_ages = convert_tensor_to_numpy_array(
127
123
  predicted_ages_tensor, logger, indent_level=2
128
124
  )
129
125
 
130
- # Apply postprocessing if specified
131
- if postprocessing:
132
- predicted_ages = postprocess_data(
133
- predicted_ages,
134
- postprocessing,
135
- postprocessing_helper,
136
- logger,
137
- indent_level=2,
138
- )
126
+ # Apply postprocessing
127
+ predicted_ages = postprocess_data(
128
+ predicted_ages,
129
+ postprocessing,
130
+ postprocessing_helper,
131
+ logger,
132
+ indent_level=2,
133
+ )
139
134
 
140
135
  # Add predicted ages to adata
141
136
  add_pred_ages_adata(adata, predicted_ages, clock_name, logger, indent_level=2)
142
137
 
143
- # Load all clocks metadata
144
- all_clock_metadata = load_clock_metadata(dir, logger, indent_level=2)
145
-
146
138
  # Add clock metadata to adata object
147
- add_clock_metadata_adata(
148
- adata, clock_name, all_clock_metadata, logger, indent_level=2
149
- )
139
+ add_clock_metadata_adata(adata, clock_name, dir, logger, indent_level=2)
150
140
 
151
- # Reduce feature size to original features
152
- adata = adata[:, original_var_names].copy()
141
+ # Return adata to original size and number of features
142
+ adata = filter_missing_features(adata, logger, indent_level=2)
153
143
 
154
144
  # Flush memory
155
145
  gc.collect()
@@ -5,6 +5,7 @@ from torch.utils.data import TensorDataset, DataLoader
5
5
  import numpy as np
6
6
  import pandas as pd
7
7
  import anndata
8
+ from anndata.experimental.pytorch import AnnLoader
8
9
  import ntpath
9
10
  from urllib.request import urlretrieve
10
11
 
@@ -47,6 +48,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
47
48
  tuple
48
49
  A tuple containing the following components of the clock:
49
50
  - features: The features used by the clock.
51
+ - reference_feature_values: Reference values for features in case features are missing.
50
52
  - weight_dict: A dictionary of weights used in the clock's model.
51
53
  - preprocessing: Any preprocessing steps required for the clock's input data.
52
54
  - postprocessing: Any postprocessing steps applied to the clock's output.
@@ -65,7 +67,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
65
67
 
66
68
  Examples
67
69
  --------
68
- >>> features, weight_dict, _, _, _, _ = load_clock("clock1", "pyaging_data", logger)
70
+ >>> features, reference_feature_values, _, _, _, _, _ = load_clock("clock1", "pyaging_data", logger)
69
71
  >>> print(features)
70
72
  ['feature1', 'feature2', ...]
71
73
 
@@ -81,6 +83,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
81
83
 
82
84
  # Extract relevant information from the clock dictionary
83
85
  features = clock_dict["features"]
86
+ reference_feature_values = clock_dict.get("reference_feature_values", None)
84
87
  weight_dict = clock_dict["weight_dict"]
85
88
  preprocessing = clock_dict.get("preprocessing", None)
86
89
  postprocessing = clock_dict.get("postprocessing", None)
@@ -89,6 +92,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
89
92
 
90
93
  return (
91
94
  features,
95
+ reference_feature_values,
92
96
  weight_dict,
93
97
  preprocessing,
94
98
  postprocessing,
@@ -102,6 +106,7 @@ def check_features_in_adata(
102
106
  adata: anndata.AnnData,
103
107
  clock_name: str,
104
108
  features: List[str],
109
+ reference_feature_values: List[float],
105
110
  logger,
106
111
  indent_level: int = 2,
107
112
  ) -> anndata.AnnData:
@@ -110,8 +115,9 @@ def check_features_in_adata(
110
115
 
111
116
  This function checks an AnnData object (commonly used in single-cell analysis) to ensure
112
117
  that it contains all the necessary features specified in the 'features' list. If any features
113
- are missing, they are added to the AnnData object with a default value of 0. This is crucial
114
- for downstream analyses where the presence of all specified features is assumed.
118
+ are missing, they are added to the AnnData object with a default value of 0 or with a reference
119
+ value if given. This is crucial for downstream analyses where the presence of all specified
120
+ features is assumed.
115
121
 
116
122
  Parameters
117
123
  ----------
@@ -127,6 +133,10 @@ def check_features_in_adata(
127
133
  A list of features (e.g., gene names or other identifiers) that are expected to be
128
134
  present in the 'adata'.
129
135
 
136
+ reference_feature_values : list
137
+ A list of the reference values for each feature. The order must match the order of features.
138
+ If no
139
+
130
140
  logger : Logger
131
141
  A logger object used for logging information about the process, such as the number
132
142
  of missing features.
@@ -139,17 +149,17 @@ def check_features_in_adata(
139
149
  -------
140
150
  anndata.AnnData
141
151
  The updated AnnData object, which includes any missing features added with a default
142
- value of 0.
152
+ value of 0 (or reference value if provided).
143
153
 
144
154
  Notes
145
155
  -----
146
156
  This function is particularly useful in preprocessing steps where the consistency of
147
157
  data structure across different datasets is crucial. The function modifies the AnnData
148
- object in place if there are missing features and logs detailed information about
149
- these modifications.
158
+ object if there are missing features and logs detailed information about these modifications.
150
159
 
151
160
  The added features are initialized with zeros. This approach, while providing completeness,
152
- may introduce biases if not accounted for in downstream analyses.
161
+ may introduce biases if not accounted for in downstream analyses. If reference values are
162
+ provided, then they are used instead of zeros.
153
163
 
154
164
  Examples
155
165
  --------
@@ -187,32 +197,52 @@ def check_features_in_adata(
187
197
  if missing_features:
188
198
  logger.warning(
189
199
  f"{num_missing_features} out of {total_features} features "
190
- f"({percent_missing:.2f}%) are missing and will be "
191
- f"added with default value 0: {missing_features[:np.min([3, num_missing_features])]}, etc.",
200
+ f"({percent_missing:.2f}%) are missing: {missing_features[:np.min([3, num_missing_features])]}, etc.",
192
201
  indent_level=indent_level + 1,
193
202
  )
194
203
 
195
- # Create an empty AnnData object for missing features
196
- empty_data = np.zeros((adata.n_obs, num_missing_features))
204
+ # If there are reference values provided
205
+ if reference_feature_values:
206
+ logger.info(
207
+ f"Using reference feature values for {clock_name}",
208
+ indent_level=indent_level + 1,
209
+ )
210
+
211
+ # Map features to reference values
212
+ feature_value_map = dict(zip(features, reference_feature_values))
213
+
214
+ # Pre-allocate with reference values, if missing, use a default value (e.g., 0)
215
+ missing_data = np.array(
216
+ [feature_value_map.get(f, 0) for f in missing_features] * adata.n_obs
217
+ ).reshape(adata.n_obs, num_missing_features)
218
+ else:
219
+ logger.info(
220
+ f"Filling missing features entirely with 0",
221
+ indent_level=indent_level + 1,
222
+ )
223
+
224
+ # Create an empty array
225
+ missing_data = np.zeros((adata.n_obs, num_missing_features))
226
+
197
227
  adata_empty = anndata.AnnData(
198
- X=empty_data,
228
+ X=missing_data,
199
229
  obs=adata.obs,
200
230
  var=pd.DataFrame(
201
231
  np.ones((num_missing_features, 1)),
202
232
  index=missing_features,
203
233
  columns=["percent_na"],
204
234
  ),
205
- layers=dict(zip(adata.layers.keys(), [empty_data] * len(adata.layers))),
235
+ layers=dict(zip(adata.layers.keys(), [missing_data] * len(adata.layers))),
206
236
  uns=adata.uns,
207
237
  )
208
238
 
209
- # Concatenate original adata with the empty adata
239
+ # Concatenate original adata with the missing adata
210
240
  adata = anndata.concat(
211
241
  [adata, adata_empty], axis=1, merge="same", uns_merge="unique"
212
242
  )
213
243
 
214
244
  logger.info(
215
- f"Expanded adata with {num_missing_features} missing features.",
245
+ f"Expanded adata with {num_missing_features} missing features",
216
246
  indent_level=indent_level + 1,
217
247
  )
218
248
  else:
@@ -311,7 +341,9 @@ def initialize_model(
311
341
  "ocampoatac1",
312
342
  "ocampoatac2",
313
343
  "bitage",
314
- "zhang",
344
+ "zhangmortality",
345
+ "zhangen",
346
+ "zhangblup",
315
347
  "leecpc",
316
348
  "leerpc",
317
349
  "leerefinedrpc",
@@ -319,6 +351,7 @@ def initialize_model(
319
351
  "thompsonmultitissue",
320
352
  "petkovichblood",
321
353
  "stubbsmultitissue",
354
+ "linblood",
322
355
  ]:
323
356
  model = LinearModel(len(features))
324
357
  elif clock_name in [
@@ -355,6 +388,7 @@ def initialize_model(
355
388
  raise ValueError(f"Clock '{clock_name}' is not supported.")
356
389
 
357
390
  model.load_state_dict(weight_dict)
391
+ model.to(torch.float64)
358
392
  model = model.to(device)
359
393
  model.eval()
360
394
  return model
@@ -418,10 +452,10 @@ def preprocess_data(
418
452
  np.array([...])
419
453
 
420
454
  """
421
- # Skip if it has already found the preprocessing layer
422
- if f"X_{preprocessing}" in adata.layers:
455
+ # Skip if it there is no preprocessing to be done
456
+ if preprocessing is None:
423
457
  logger.info(
424
- f"Layer with {preprocessing} preprocessing is already in adata",
458
+ "There is no preprocessing to be done",
425
459
  indent_level=3,
426
460
  )
427
461
  return adata
@@ -435,9 +469,7 @@ def preprocess_data(
435
469
 
436
470
  logger.info(f"Preprocessing data with function {preprocessing}", indent_level=3)
437
471
  # Apply specified preprocessing method
438
- if preprocessing == "log1p":
439
- adata.X = np.log1p(adata.X)
440
- elif preprocessing == "tpm_norm_log1p":
472
+ if preprocessing == "tpm_norm_log1p":
441
473
  adata.X = tpm_norm_log1p(adata.X, preprocessing_helper)
442
474
  elif preprocessing == "binarize":
443
475
  adata.X = binarize(adata.X)
@@ -466,7 +498,6 @@ def preprocess_data(
466
498
  else:
467
499
  logger.error(f"Preprocessing function {preprocessing} not found.")
468
500
  raise ValueError()
469
- adata.layers[f"X_{preprocessing}"] = adata.X.copy()
470
501
 
471
502
  return adata
472
503
 
@@ -526,6 +557,14 @@ def postprocess_data(
526
557
  (num_samples, num_features)
527
558
 
528
559
  """
560
+ # Skip if it there is no postprocessing to be done
561
+ if postprocessing is None:
562
+ logger.info(
563
+ "There is no postprocessing to be done",
564
+ indent_level=3,
565
+ )
566
+ return data
567
+
529
568
  logger.info(f"Postprocessing data with function {postprocessing}", indent_level=3)
530
569
  # Apply specified postprocessing method using vectorization
531
570
  if postprocessing == "anti_log_linear":
@@ -557,7 +596,12 @@ def postprocess_data(
557
596
 
558
597
  @progress("Predict ages with model")
559
598
  def predict_ages_with_model(
560
- model: torch.nn.Module, data: torch.Tensor, logger, indent_level: int = 2
599
+ model: torch.nn.Module,
600
+ adata: torch.Tensor,
601
+ features: List[str],
602
+ device: str,
603
+ logger,
604
+ indent_level: int = 2,
561
605
  ) -> torch.Tensor:
562
606
  """
563
607
  Predict biological ages using a trained model and input data.
@@ -565,16 +609,23 @@ def predict_ages_with_model(
565
609
  This function takes a machine learning model and input data, and returns predictions made by the model.
566
610
  It's primarily used for estimating biological ages based on various biological markers. The function
567
611
  assumes that the model is already trained and the data is preprocessed according to the model's requirements.
568
- If the size of the data is large (> 1000 samples), a dataloader is used because of possible memory constraints.
612
+ A dataloader is used because of possible memory constraints.
569
613
 
570
614
  Parameters
571
615
  ----------
572
616
  model : torch.nn.Module
573
617
  A pre-trained machine learning model that can make predictions.
574
618
 
575
- data : array-like
576
- The input data on which age predictions need to be made. This should be in the format expected
577
- by the `model` (e.g., a numpy array, pandas DataFrame, or PyTorch tensor).
619
+ adata : anndata.AnnData
620
+ The AnnData object containing the dataset. Its `.X` attribute is expected to be a matrix where rows
621
+ correspond to samples and columns correspond to features.
622
+
623
+ features : list of str
624
+ A list of feature names to be included in the output array. Only these features from the AnnData object will
625
+ be extracted for age prediction.
626
+
627
+ device : str
628
+ Device to move AnnData to during inference. Eithe 'cpu' or 'cuda'.
578
629
 
579
630
  logger : Logger
580
631
  A logger object for logging the progress or any relevant information during the prediction process.
@@ -598,39 +649,26 @@ def predict_ages_with_model(
598
649
  Examples
599
650
  --------
600
651
  >>> model = load_pretrained_model()
601
- >>> predictions = predict_ages_with_model(model, data, logger)
652
+ >>> predictions = predict_ages_with_model(model, adata, features, 'cpu', logger)
602
653
  >>> print(predictions[:5])
603
654
  [34.5, 29.3, 47.8, 50.1, 42.6]
604
655
 
605
656
  """
606
- # Check if the size of data is greater than 1000
607
- if data.size(0) > 1000:
608
- logger.info(
609
- f"Number of samples ({data.size(0)}) is bigger than 1000. Predicting age with DataLoader",
610
- indent_level=indent_level + 1,
611
- )
612
-
613
- # Create a TensorDataset
614
- dataset = TensorDataset(data)
615
-
616
- # Create a DataLoader
617
- batch_size = 64
618
- dataloader = DataLoader(dataset, batch_size=batch_size)
619
-
620
- # Use the DataLoader for batched prediction
621
- predictions = []
622
- with torch.no_grad():
623
- for batch in dataloader:
624
- batch_data = batch[0]
625
- batch_pred = model(batch_data)
626
- predictions.append(batch_pred)
627
-
628
- # Concatenate all batch predictions
629
- predictions = torch.cat(predictions)
630
- else:
631
- # If data size is not greater than 1000, predict directly
632
- with torch.no_grad():
633
- predictions = model(data)
657
+ # Create an AnnLoader
658
+ use_cuda = device == "cuda"
659
+ dataloader = AnnLoader(adata, batch_size=1024, use_cuda=use_cuda)
660
+
661
+ # Use the AnnLoader for batched prediction
662
+ predictions = []
663
+ with torch.no_grad():
664
+ for batch in main_tqdm(
665
+ dataloader, indent_level=indent_level + 1, logger=logger
666
+ ):
667
+ batch_pred = model(batch[:, features].X)
668
+ predictions.append(batch_pred)
669
+
670
+ # Concatenate all batch predictions
671
+ predictions = torch.cat(predictions)
634
672
  return predictions
635
673
 
636
674
 
@@ -680,133 +718,6 @@ def convert_tensor_to_numpy_array(
680
718
  return tensor.cpu().detach().numpy().flatten()
681
719
 
682
720
 
683
- @progress("Convert numpy array to tensor")
684
- def convert_numpy_array_to_tensor(
685
- array: np.ndarray, device: str, logger, indent_level: int = 2
686
- ) -> torch.Tensor:
687
- """
688
- Convert a NumPy array to a tensor.
689
-
690
- This utility function takes a NumPy array and converts it into PyTorch tensor. It is useful in scenarios
691
- where NumPy arrays need to be processed or analyzed using PyTorch-based tools or functions. The function also
692
- moves the tensor to the device being used.
693
-
694
- Parameters
695
- ----------
696
- tensor : torch.Tensor
697
- The PyTorch tensor to be converted. This can be a tensor of any shape.
698
-
699
- device : str
700
- Device to move tensor to after. Eithe 'cpu' or 'cuda'.
701
-
702
- logger : Logger
703
- A logger object for logging the progress or any relevant information during the conversion process.
704
-
705
- indent_level : int, optional
706
- The indentation level for logging messages, by default 2.
707
-
708
- Returns
709
- -------
710
- numpy_array : ndarray
711
- A 1D NumPy array equivalent to the input tensor.
712
-
713
- Notes
714
- -----
715
- This function detaches the tensor from the computation graph, so it should be used with caution if the tensor
716
- is part of a PyTorch computational graph, such as when dealing with gradients in training neural networks.
717
-
718
- The flattening to a 1D array may lose the original shape information of the tensor, which should be considered
719
- if the shape is important for further processing.
720
-
721
- Examples
722
- --------
723
- >>> numpy_array = np.array([[1, 2], [3, 4]])
724
- >>> tensor = convert_numpy_array_to_tensor(numpy_array, logger)
725
- >>> tensor
726
- torch.tensor([1, 2, 3, 4])
727
-
728
- """
729
- tensor = torch.tensor(array, dtype=torch.float32)
730
- tensor = tensor.to(device)
731
- return tensor
732
-
733
-
734
- @progress("Filter features and extract data matrix")
735
- def filter_features_and_extract_data(
736
- adata: anndata.AnnData,
737
- preprocessing: str,
738
- features: List[str],
739
- logger,
740
- indent_level: int = 2,
741
- ) -> np.ndarray:
742
- """
743
- Filter features from an AnnData object
744
-
745
- This function processes an AnnData object, which is commonly used in bioinformatics for storing large
746
- gene expression datasets. It extracts the data matrix corresponding to a specified set of features (genes or
747
- other genomic elements) and converts this subset into a numpy array.
748
-
749
- Parameters
750
- ----------
751
- adata : anndata.AnnData
752
- The AnnData object containing the dataset. Its `.X` attribute is expected to be a matrix where rows
753
- correspond to samples and columns correspond to features.
754
-
755
- preprocessing : str
756
- The name of the preprocessing method used (if any). Used to search adata.layers for data matrix.
757
-
758
- features : list of str
759
- A list of feature names to be included in the output array. Only these features from the AnnData object will
760
- be extracted and converted.
761
-
762
- logger : Logger
763
- A logger object for logging progress or relevant information during the conversion process.
764
-
765
- indent_level : int, optional
766
- The indentation level for logging messages, by default 2.
767
-
768
- Returns
769
- -------
770
- numpy_array : ndarray
771
- A Numpy array containing the filtered data from the AnnData object, with dtype set to float32.
772
-
773
- Notes
774
- -----
775
- The AnnData object is a central data structure in many bioinformatics pipelines. It efficiently handles
776
- large datasets typical in genomics and transcriptomics. This function facilitates the integration of
777
- AnnData-based workflows with PyTorch models and functions.
778
-
779
- The dtype of the returned tensor is set to float32, which is a common choice for numerical operations in
780
- PyTorch, especially in the context of machine learning models.
781
-
782
- Examples
783
- --------
784
- >>> adata = anndata.AnnData(np.random.rand(5, 10))
785
- >>> features = ['gene1', 'gene2', 'gene3']
786
- >>> numpy_array = filter_features_and_extract_data(adata, None, features, logger)
787
- >>> numpy_array.shape
788
- array([5, 3])
789
-
790
- """
791
- if preprocessing is not None:
792
- layer_name = f"X_{preprocessing}"
793
- elif "X_imputed" in adata.layers:
794
- layer_name = "X_imputed"
795
- else:
796
- layer_name = "X_original"
797
- # x_numpy = np.array(adata[:, features].layers[layer_name])
798
- layer_data = adata[:, features].layers[layer_name]
799
-
800
- if not isinstance(layer_data, np.ndarray):
801
- layer_data = (
802
- layer_data.toarray()
803
- if hasattr(layer_data, "toarray")
804
- else np.array(layer_data, dtype=np.float32)
805
- )
806
-
807
- return layer_data
808
-
809
-
810
721
  @progress("Add predicted ages to adata")
811
722
  def add_pred_ages_adata(
812
723
  adata: anndata.AnnData,
@@ -874,11 +785,66 @@ def add_pred_ages_adata(
874
785
  adata.obs[clock_name] = predicted_ages
875
786
 
876
787
 
788
+ @progress("Return adata to original size")
789
+ def filter_missing_features(
790
+ adata: anndata.AnnData,
791
+ logger,
792
+ indent_level: int = 2,
793
+ ) -> anndata.AnnData:
794
+ """
795
+ Returns adata with original features.
796
+
797
+ This function checks for variables that have 100% of samples with missing features, and removes them
798
+ from the adata object. It is useful for returning the adata in the original state.
799
+
800
+ Parameters
801
+ ----------
802
+ adata : anndata.AnnData
803
+ The AnnData object that will be filtered.
804
+
805
+ logger : Logger
806
+ A logger object for logging the progress or relevant information during the operation.
807
+
808
+ indent_level : int, optional
809
+ The indentation level for logging messages, by default 2.
810
+
811
+ Returns
812
+ -------
813
+ anndata.AnnData
814
+ The filtered adata in which all features appear in at least one sample.
815
+
816
+ Notes
817
+ -----
818
+ During filtering, the adata object is filtered based on a column called "percent_na" which should be less
819
+ than 1. If the column is not present, an error will appear.
820
+
821
+ Examples
822
+ --------
823
+ >>> adata = anndata.AnnData(np.random.rand(5, 10))
824
+ >>> adata = filter_missing_features(adata, logger)
825
+
826
+ """
827
+ n_missing_features = sum(adata.var["percent_na"] == 1)
828
+ if n_missing_features > 0:
829
+ logger.info(
830
+ f"Removing {n_missing_features} added features",
831
+ indent_level=indent_level + 1,
832
+ )
833
+ adata = adata[:, adata.var["percent_na"] < 1].copy()
834
+ else:
835
+ logger.info(
836
+ "No missing features, so adata size did not change",
837
+ indent_level=indent_level + 1,
838
+ )
839
+
840
+ return adata
841
+
842
+
877
843
  @progress("Add clock metadata to adata.uns")
878
844
  def add_clock_metadata_adata(
879
845
  adata: anndata.AnnData,
880
846
  clock_name: str,
881
- all_clock_metadata: dict,
847
+ dir: str,
882
848
  logger,
883
849
  indent_level: int = 2,
884
850
  ) -> None:
@@ -901,9 +867,8 @@ def add_clock_metadata_adata(
901
867
  The name of the aging clock. The metadata associated with this clock will be added to
902
868
  the AnnData object.
903
869
 
904
- all_clock_metadata : dict
905
- A dictionary containing metadata for all available clocks. The function will extract metadata for
906
- the specified clock from this dictionary.
870
+ dir: str
871
+ The directory to deposit the downloaded file. Defaults to "pyaging_data".
907
872
 
908
873
  logger : Logger
909
874
  A logger object for logging the progress or relevant information during the operation.
@@ -922,18 +887,21 @@ def add_clock_metadata_adata(
922
887
  here ensures that all relevant information about the aging clocks used in the analysis is
923
888
  preserved alongside the dataset.
924
889
 
925
- It is important to ensure that the `clock_name` exists within the `all_clock_metadata` dictionary to
926
- avoid key errors.
927
-
928
890
  Examples
929
891
  --------
930
892
  >>> adata = anndata.AnnData(np.random.rand(5, 10))
931
- >>> all_clock_metadata = {'horvath2013': {'description': 'Horvath’s 2013 epigenetic clock', 'reference': 'Horvath, S. (2013)'}}
932
- >>> add_clock_metadata_adata(adata, 'horvath2013', all_clock_metadata, logger)
893
+ >>> add_clock_metadata_adata(adata, 'horvath2013', 'pyaging_data', logger)
933
894
  >>> adata.uns['horvath2013_metadata']
934
895
  {'description': 'Horvath’s 2013 epigenetic clock', 'reference': 'Horvath, S. (2013)'}
935
896
 
936
897
  """
898
+
899
+ # Load all clocks metadata
900
+ url = f"https://pyaging.s3.amazonaws.com/clocks/metadata/all_clock_metadata.pt"
901
+ download(url, dir, logger, indent_level=indent_level)
902
+ all_clock_metadata = torch.load(f"{dir}/all_clock_metadata.pt")
903
+
904
+ # Add clock metadata to adata.uns
937
905
  adata.uns[f"{clock_name}_metadata"] = all_clock_metadata[clock_name]
938
906
 
939
907
 
@@ -202,6 +202,11 @@ def create_anndata_object(
202
202
  obs_names = df.index
203
203
  var_names = df.columns
204
204
 
205
+ # Check for duplicate features
206
+ if len(np.unique(var_names)) != len(var_names):
207
+ logger.error("There are duplicate feature names!")
208
+ raise ValueError
209
+
205
210
  obs = pd.DataFrame(index=obs_names)
206
211
  var = pd.DataFrame(index=var_names)
207
212
 
@@ -313,6 +313,11 @@ def cite_clock(clock_name: str, dir: str = "pyaging_data") -> None:
313
313
  citation = clock_dict["citation"]
314
314
  logger.info(f"Citation for {clock_name}:", indent_level=2)
315
315
  logger.info(citation, indent_level=2)
316
+ logger.info("Please also consider citing pyaging :)", indent_level=2)
317
+ logger.info(
318
+ 'de Lima Camillo, Lucas Paulo. "pyaging: a Python-based compendium of GPU-optimized aging clocks." bioRxiv (2023): 2023-11.',
319
+ indent_level=2,
320
+ )
316
321
  else:
317
322
  logger.warning(f"Citation not found in {clock_name}", indent_level=2)
318
323
  else:
@@ -438,3 +443,77 @@ def get_clock_metadata(clock_name: str, dir: str = "pyaging_data") -> None:
438
443
  logger.finish_progress(f"{message} finished")
439
444
 
440
445
  logger.done()
446
+
447
+
448
+ def get_clock_weights(clock_name: str, dir: str = "pyaging_data") -> dict:
449
+ """
450
+ Loads the specified aging clock from a remote source and returns a dictionary with its
451
+ components.
452
+
453
+ This function downloads the weights and configuration of a specified aging clock from a
454
+ remote server. It then loads and returns a dictionary with various components of the clock
455
+ such as its features, preprocessing or postprocessing steps, etc.
456
+
457
+ Parameters
458
+ ----------
459
+ clock_name : str
460
+ The name of the aging clock whose metadata is to be retrieved. The name is case-insensitive.
461
+ dir : str
462
+ The directory to deposit the downloaded file. Defaults to 'pyaging_data'.
463
+
464
+ Returns
465
+ -------
466
+ dict
467
+ A dictionary containing the following components of the clock:
468
+ - features: The features used by the clock.
469
+ - reference_feature_values: Reference values for features in case features are missing.
470
+ - weight_dict: A dictionary of weights used in the clock's model.
471
+ - preprocessing: Any preprocessing steps required for the clock's input data.
472
+ - postprocessing: Any postprocessing steps applied to the clock's output.
473
+ - preprocessing_helper: Any preprocessing helper file.
474
+ - postprocessing_helper: Any postprocessing helper file.
475
+
476
+ Notes
477
+ -----
478
+ The clock's weights and configuration are assumed to be stored in a .pt (PyTorch) file
479
+ on a remote server. The URL for the clock is constructed based on the clock's name.
480
+ If the clock or its components are not found, the function may fail or return incomplete
481
+ information.
482
+
483
+ The logger is used extensively for progress tracking and information logging, enhancing
484
+ transparency and user experience.
485
+
486
+ Examples
487
+ --------
488
+ >>> clock_dict = get_clock_weights("clock1", "pyaging_data")
489
+
490
+ """
491
+
492
+ logger = LoggerManager.gen_logger("get_clock_weights")
493
+ logger.first_info("Starting get_clock_weights function")
494
+
495
+ # Load all metadata
496
+ all_clock_metadata = load_clock_metadata(dir, logger, indent_level=1)
497
+
498
+ # Lowercase clock name
499
+ clock_name = clock_name.lower()
500
+ clock_dict = all_clock_metadata[clock_name]
501
+
502
+ # Check if clock name is available
503
+ if clock_name not in all_clock_metadata.keys():
504
+ logger.error(f"Clock {clock_name} is not yet available on pyaging")
505
+ raise ValueError
506
+
507
+ # Download weights
508
+ url = f"https://pyaging.s3.amazonaws.com/clocks/weights/{clock_name}.pt"
509
+ download(url, dir, logger, indent_level=1)
510
+
511
+ # Define the path to the clock weights file
512
+ weights_path = os.path.join(dir, f"{clock_name}.pt")
513
+
514
+ # Load the clock dictionary from the file
515
+ clock_dict = torch.load(weights_path)
516
+
517
+ logger.done()
518
+
519
+ return clock_dict
@@ -1,6 +1,6 @@
1
1
  [tool.poetry]
2
2
  name = "pyaging"
3
- version = "0.0.6"
3
+ version = "0.0.8"
4
4
  description = "A Python-based compendium of GPU-optimized aging clocks."
5
5
  authors = ["Lucas Paulo de Lima Camillo <lucas_camillo@alumni.brown.edu>"]
6
6
  license = "BSD"
File without changes
File without changes
File without changes