pyaging 0.0.6__tar.gz → 0.0.7__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of pyaging might be problematic. Click here for more details.

@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: pyaging
3
- Version: 0.0.6
3
+ Version: 0.0.7
4
4
  Summary: A Python-based compendium of GPU-optimized aging clocks.
5
5
  License: BSD
6
6
  Author: Lucas Paulo de Lima Camillo
@@ -48,6 +48,30 @@ Description-Content-Type: text/markdown
48
48
 
49
49
  With a growing number of aging clocks and biomarkers of aging, comparing and analyzing them can be challenging. `pyaging` simplifies this process, allowing researchers to input various molecular layers (DNA methylation, histone ChIP-Seq, ATAC-seq, transcriptomics, etc.) and quickly analyze them using multiple aging clocks, thanks to its GPU-backed infrastructure. This makes it an ideal tool for large datasets and multi-layered analysis.
50
50
 
51
+ ## ❓ Can't find an aging clock?
52
+
53
+ If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
54
+
55
+ ## 💬 Community Discussion
56
+ For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
57
+
58
+ ## 📖 Citation
59
+
60
+ To cite `pyaging`, please use the following:
61
+
62
+ ```
63
+ @article {de_Lima_Camillo_pyaging,
64
+ author = {Lucas Paulo de Lima Camillo},
65
+ title = {pyaging: a Python-based compendium of GPU-optimized aging clocks},
66
+ elocation-id = {2023.11.28.569069},
67
+ year = {2023},
68
+ doi = {10.1101/2023.11.28.569069},
69
+ publisher = {Cold Spring Harbor Laboratory},
70
+ URL = {https://www.biorxiv.org/content/early/2023/11/29/2023.11.28.569069},
71
+ journal = {bioRxiv}
72
+ }
73
+ ```
74
+
51
75
  ## 📝 To-Do List
52
76
 
53
77
  - [X] Incorporate more murine DNA methylation clocks
@@ -57,10 +81,4 @@ With a growing number of aging clocks and biomarkers of aging, comparing and ana
57
81
  - [ ] Integrate scAge clocks (this is proving to be difficult)
58
82
  - [ ] Integrate scRNAseq clocks (and datasets)
59
83
 
60
- ## ❓ Can't find an aging clock?
61
-
62
- If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
63
-
64
- ## 💬 Community Discussion
65
- For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
66
84
 
@@ -23,6 +23,30 @@
23
23
 
24
24
  With a growing number of aging clocks and biomarkers of aging, comparing and analyzing them can be challenging. `pyaging` simplifies this process, allowing researchers to input various molecular layers (DNA methylation, histone ChIP-Seq, ATAC-seq, transcriptomics, etc.) and quickly analyze them using multiple aging clocks, thanks to its GPU-backed infrastructure. This makes it an ideal tool for large datasets and multi-layered analysis.
25
25
 
26
+ ## ❓ Can't find an aging clock?
27
+
28
+ If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
29
+
30
+ ## 💬 Community Discussion
31
+ For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
32
+
33
+ ## 📖 Citation
34
+
35
+ To cite `pyaging`, please use the following:
36
+
37
+ ```
38
+ @article {de_Lima_Camillo_pyaging,
39
+ author = {Lucas Paulo de Lima Camillo},
40
+ title = {pyaging: a Python-based compendium of GPU-optimized aging clocks},
41
+ elocation-id = {2023.11.28.569069},
42
+ year = {2023},
43
+ doi = {10.1101/2023.11.28.569069},
44
+ publisher = {Cold Spring Harbor Laboratory},
45
+ URL = {https://www.biorxiv.org/content/early/2023/11/29/2023.11.28.569069},
46
+ journal = {bioRxiv}
47
+ }
48
+ ```
49
+
26
50
  ## 📝 To-Do List
27
51
 
28
52
  - [X] Incorporate more murine DNA methylation clocks
@@ -32,9 +56,3 @@ With a growing number of aging clocks and biomarkers of aging, comparing and ana
32
56
  - [ ] Integrate scAge clocks (this is proving to be difficult)
33
57
  - [ ] Integrate scRNAseq clocks (and datasets)
34
58
 
35
- ## ❓ Can't find an aging clock?
36
-
37
- If you have recently developed an aging clock and would like it to be integrated into `pyaging`, please [email us](lucas_camillo@alumni.brown.edu). We aim to incorporate it within two weeks! We are also happy to adapt to any licensing terms for commercial entities.
38
-
39
- ## 💬 Community Discussion
40
- For coding-related queries, feedback, and discussions, please visit our [GitHub Issues](https://github.com/rsinghlab/pyaging/issues) page.
@@ -1,5 +1,7 @@
1
1
  import anndata
2
2
  import gc
3
+ import warnings
4
+ from anndata import ImplicitModificationWarning
3
5
 
4
6
  from ._pred_utils import *
5
7
 
@@ -78,6 +80,7 @@ def predict_age(
78
80
  clock_name = clock_name.lower()
79
81
  (
80
82
  features,
83
+ reference_feature_values,
81
84
  weight_dict,
82
85
  preprocessing,
83
86
  postprocessing,
@@ -86,21 +89,24 @@ def predict_age(
86
89
  ) = load_clock(clock_name, dir, logger, indent_level=2)
87
90
 
88
91
  # Check and update adata for missing features
89
- original_var_names = list(adata.var_names)
90
92
  adata = check_features_in_adata(
91
- adata, clock_name, features, logger, indent_level=2
93
+ adata,
94
+ clock_name,
95
+ features,
96
+ reference_feature_values,
97
+ logger,
98
+ indent_level=2,
92
99
  )
93
100
 
94
101
  # Apply preprocessing
95
- if preprocessing:
96
- adata = preprocess_data(
97
- adata,
98
- preprocessing,
99
- preprocessing_helper,
100
- features,
101
- logger,
102
- indent_level=2,
103
- )
102
+ adata = preprocess_data(
103
+ adata,
104
+ preprocessing,
105
+ preprocessing_helper,
106
+ features,
107
+ logger,
108
+ indent_level=2,
109
+ )
104
110
 
105
111
  # Filter features and then extract data matrix
106
112
  x_numpy = filter_features_and_extract_data(
@@ -127,15 +133,14 @@ def predict_age(
127
133
  predicted_ages_tensor, logger, indent_level=2
128
134
  )
129
135
 
130
- # Apply postprocessing if specified
131
- if postprocessing:
132
- predicted_ages = postprocess_data(
133
- predicted_ages,
134
- postprocessing,
135
- postprocessing_helper,
136
- logger,
137
- indent_level=2,
138
- )
136
+ # Apply postprocessing
137
+ predicted_ages = postprocess_data(
138
+ predicted_ages,
139
+ postprocessing,
140
+ postprocessing_helper,
141
+ logger,
142
+ indent_level=2,
143
+ )
139
144
 
140
145
  # Add predicted ages to adata
141
146
  add_pred_ages_adata(adata, predicted_ages, clock_name, logger, indent_level=2)
@@ -149,7 +154,7 @@ def predict_age(
149
154
  )
150
155
 
151
156
  # Reduce feature size to original features
152
- adata = adata[:, original_var_names].copy()
157
+ adata = filter_missing_features(adata, logger, indent_level=2)
153
158
 
154
159
  # Flush memory
155
160
  gc.collect()
@@ -47,6 +47,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
47
47
  tuple
48
48
  A tuple containing the following components of the clock:
49
49
  - features: The features used by the clock.
50
+ - reference_feature_values: Reference values for features in case features are missing.
50
51
  - weight_dict: A dictionary of weights used in the clock's model.
51
52
  - preprocessing: Any preprocessing steps required for the clock's input data.
52
53
  - postprocessing: Any postprocessing steps applied to the clock's output.
@@ -65,7 +66,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
65
66
 
66
67
  Examples
67
68
  --------
68
- >>> features, weight_dict, _, _, _, _ = load_clock("clock1", "pyaging_data", logger)
69
+ >>> features, reference_feature_values, _, _, _, _, _ = load_clock("clock1", "pyaging_data", logger)
69
70
  >>> print(features)
70
71
  ['feature1', 'feature2', ...]
71
72
 
@@ -81,6 +82,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
81
82
 
82
83
  # Extract relevant information from the clock dictionary
83
84
  features = clock_dict["features"]
85
+ reference_feature_values = clock_dict.get("reference_feature_values", None)
84
86
  weight_dict = clock_dict["weight_dict"]
85
87
  preprocessing = clock_dict.get("preprocessing", None)
86
88
  postprocessing = clock_dict.get("postprocessing", None)
@@ -89,6 +91,7 @@ def load_clock(clock_name: str, dir: str, logger, indent_level: int = 2) -> Tupl
89
91
 
90
92
  return (
91
93
  features,
94
+ reference_feature_values,
92
95
  weight_dict,
93
96
  preprocessing,
94
97
  postprocessing,
@@ -102,6 +105,7 @@ def check_features_in_adata(
102
105
  adata: anndata.AnnData,
103
106
  clock_name: str,
104
107
  features: List[str],
108
+ reference_feature_values: List[float],
105
109
  logger,
106
110
  indent_level: int = 2,
107
111
  ) -> anndata.AnnData:
@@ -110,8 +114,9 @@ def check_features_in_adata(
110
114
 
111
115
  This function checks an AnnData object (commonly used in single-cell analysis) to ensure
112
116
  that it contains all the necessary features specified in the 'features' list. If any features
113
- are missing, they are added to the AnnData object with a default value of 0. This is crucial
114
- for downstream analyses where the presence of all specified features is assumed.
117
+ are missing, they are added to the AnnData object with a default value of 0 or with a reference
118
+ value if given. This is crucial for downstream analyses where the presence of all specified
119
+ features is assumed.
115
120
 
116
121
  Parameters
117
122
  ----------
@@ -127,6 +132,10 @@ def check_features_in_adata(
127
132
  A list of features (e.g., gene names or other identifiers) that are expected to be
128
133
  present in the 'adata'.
129
134
 
135
+ reference_feature_values : list
136
+ A list of the reference values for each feature. The order must match the order of features.
137
+ If no
138
+
130
139
  logger : Logger
131
140
  A logger object used for logging information about the process, such as the number
132
141
  of missing features.
@@ -139,17 +148,17 @@ def check_features_in_adata(
139
148
  -------
140
149
  anndata.AnnData
141
150
  The updated AnnData object, which includes any missing features added with a default
142
- value of 0.
151
+ value of 0 (or reference value if provided).
143
152
 
144
153
  Notes
145
154
  -----
146
155
  This function is particularly useful in preprocessing steps where the consistency of
147
156
  data structure across different datasets is crucial. The function modifies the AnnData
148
- object in place if there are missing features and logs detailed information about
149
- these modifications.
157
+ object if there are missing features and logs detailed information about these modifications.
150
158
 
151
159
  The added features are initialized with zeros. This approach, while providing completeness,
152
- may introduce biases if not accounted for in downstream analyses.
160
+ may introduce biases if not accounted for in downstream analyses. If reference values are
161
+ provided, then they are used instead of zeros.
153
162
 
154
163
  Examples
155
164
  --------
@@ -187,32 +196,52 @@ def check_features_in_adata(
187
196
  if missing_features:
188
197
  logger.warning(
189
198
  f"{num_missing_features} out of {total_features} features "
190
- f"({percent_missing:.2f}%) are missing and will be "
191
- f"added with default value 0: {missing_features[:np.min([3, num_missing_features])]}, etc.",
199
+ f"({percent_missing:.2f}%) are missing: {missing_features[:np.min([3, num_missing_features])]}, etc.",
192
200
  indent_level=indent_level + 1,
193
201
  )
194
202
 
195
- # Create an empty AnnData object for missing features
196
- empty_data = np.zeros((adata.n_obs, num_missing_features))
203
+ # If there are reference values provided
204
+ if reference_feature_values:
205
+ logger.info(
206
+ f"Using reference feature values for {clock_name}",
207
+ indent_level=indent_level + 1,
208
+ )
209
+
210
+ # Map features to reference values
211
+ feature_value_map = dict(zip(features, reference_feature_values))
212
+
213
+ # Pre-allocate with reference values, if missing, use a default value (e.g., 0)
214
+ missing_data = np.array(
215
+ [feature_value_map.get(f, 0) for f in missing_features] * adata.n_obs
216
+ ).reshape(adata.n_obs, num_missing_features)
217
+ else:
218
+ logger.info(
219
+ f"Filling missing features entirely with 0",
220
+ indent_level=indent_level + 1,
221
+ )
222
+
223
+ # Create an empty array
224
+ missing_data = np.zeros((adata.n_obs, num_missing_features))
225
+
197
226
  adata_empty = anndata.AnnData(
198
- X=empty_data,
227
+ X=missing_data,
199
228
  obs=adata.obs,
200
229
  var=pd.DataFrame(
201
230
  np.ones((num_missing_features, 1)),
202
231
  index=missing_features,
203
232
  columns=["percent_na"],
204
233
  ),
205
- layers=dict(zip(adata.layers.keys(), [empty_data] * len(adata.layers))),
234
+ layers=dict(zip(adata.layers.keys(), [missing_data] * len(adata.layers))),
206
235
  uns=adata.uns,
207
236
  )
208
237
 
209
- # Concatenate original adata with the empty adata
238
+ # Concatenate original adata with the missing adata
210
239
  adata = anndata.concat(
211
240
  [adata, adata_empty], axis=1, merge="same", uns_merge="unique"
212
241
  )
213
242
 
214
243
  logger.info(
215
- f"Expanded adata with {num_missing_features} missing features.",
244
+ f"Expanded adata with {num_missing_features} missing features",
216
245
  indent_level=indent_level + 1,
217
246
  )
218
247
  else:
@@ -311,7 +340,8 @@ def initialize_model(
311
340
  "ocampoatac1",
312
341
  "ocampoatac2",
313
342
  "bitage",
314
- "zhang",
343
+ "zhangmortality",
344
+ "zhangen",
315
345
  "leecpc",
316
346
  "leerpc",
317
347
  "leerefinedrpc",
@@ -418,6 +448,14 @@ def preprocess_data(
418
448
  np.array([...])
419
449
 
420
450
  """
451
+ # Skip if it there is no preprocessing to be done
452
+ if preprocessing is None:
453
+ logger.info(
454
+ "There is no preprocessing to be done",
455
+ indent_level=3,
456
+ )
457
+ return adata
458
+
421
459
  # Skip if it has already found the preprocessing layer
422
460
  if f"X_{preprocessing}" in adata.layers:
423
461
  logger.info(
@@ -435,9 +473,7 @@ def preprocess_data(
435
473
 
436
474
  logger.info(f"Preprocessing data with function {preprocessing}", indent_level=3)
437
475
  # Apply specified preprocessing method
438
- if preprocessing == "log1p":
439
- adata.X = np.log1p(adata.X)
440
- elif preprocessing == "tpm_norm_log1p":
476
+ if preprocessing == "tpm_norm_log1p":
441
477
  adata.X = tpm_norm_log1p(adata.X, preprocessing_helper)
442
478
  elif preprocessing == "binarize":
443
479
  adata.X = binarize(adata.X)
@@ -526,6 +562,14 @@ def postprocess_data(
526
562
  (num_samples, num_features)
527
563
 
528
564
  """
565
+ # Skip if it there is no postprocessing to be done
566
+ if postprocessing is None:
567
+ logger.info(
568
+ "There is no postprocessing to be done",
569
+ indent_level=3,
570
+ )
571
+ return data
572
+
529
573
  logger.info(f"Postprocessing data with function {postprocessing}", indent_level=3)
530
574
  # Apply specified postprocessing method using vectorization
531
575
  if postprocessing == "anti_log_linear":
@@ -604,9 +648,9 @@ def predict_ages_with_model(
604
648
 
605
649
  """
606
650
  # Check if the size of data is greater than 1000
607
- if data.size(0) > 1000:
651
+ if data.size(0) > 1024:
608
652
  logger.info(
609
- f"Number of samples ({data.size(0)}) is bigger than 1000. Predicting age with DataLoader",
653
+ f"Number of samples ({data.size(0)}) is bigger than 1024. Predicting age with DataLoader",
610
654
  indent_level=indent_level + 1,
611
655
  )
612
656
 
@@ -614,13 +658,15 @@ def predict_ages_with_model(
614
658
  dataset = TensorDataset(data)
615
659
 
616
660
  # Create a DataLoader
617
- batch_size = 64
661
+ batch_size = 1024
618
662
  dataloader = DataLoader(dataset, batch_size=batch_size)
619
663
 
620
664
  # Use the DataLoader for batched prediction
621
665
  predictions = []
622
666
  with torch.no_grad():
623
- for batch in dataloader:
667
+ for batch in main_tqdm(
668
+ dataloader, indent_level=indent_level + 1, logger=logger
669
+ ):
624
670
  batch_data = batch[0]
625
671
  batch_pred = model(batch_data)
626
672
  predictions.append(batch_pred)
@@ -874,6 +920,49 @@ def add_pred_ages_adata(
874
920
  adata.obs[clock_name] = predicted_ages
875
921
 
876
922
 
923
+ @progress("Return adata to original size")
924
+ def filter_missing_features(
925
+ adata: anndata.AnnData,
926
+ logger,
927
+ indent_level: int = 2,
928
+ ) -> anndata.AnnData:
929
+ """
930
+ Returns adata with original features.
931
+
932
+ This function checks for variables that have 100% of samples with missing features, and removes them
933
+ from the adata object. It is useful for returning the adata in the original state.
934
+
935
+ Parameters
936
+ ----------
937
+ adata : anndata.AnnData
938
+ The AnnData object that will be filtered.
939
+
940
+ logger : Logger
941
+ A logger object for logging the progress or relevant information during the operation.
942
+
943
+ indent_level : int, optional
944
+ The indentation level for logging messages, by default 2.
945
+
946
+ Returns
947
+ -------
948
+ anndata.AnnData
949
+ The filtered adata in which all features appear in at least one sample.
950
+
951
+ Notes
952
+ -----
953
+ During filtering, the adata object is filtered based on a column called "percent_na" which should be less
954
+ than 1. If the column is not present, an error will appear.
955
+
956
+ Examples
957
+ --------
958
+ >>> adata = anndata.AnnData(np.random.rand(5, 10))
959
+ >>> adata = filter_missing_features(adata, logger)
960
+
961
+ """
962
+ adata = adata[:, adata.var["percent_na"] < 1].copy()
963
+ return adata
964
+
965
+
877
966
  @progress("Add clock metadata to adata.uns")
878
967
  def add_clock_metadata_adata(
879
968
  adata: anndata.AnnData,
@@ -202,6 +202,11 @@ def create_anndata_object(
202
202
  obs_names = df.index
203
203
  var_names = df.columns
204
204
 
205
+ # Check for duplicate features
206
+ if len(np.unique(var_names)) != len(var_names):
207
+ logger.error("There are duplicate feature names!")
208
+ raise ValueError
209
+
205
210
  obs = pd.DataFrame(index=obs_names)
206
211
  var = pd.DataFrame(index=var_names)
207
212
 
@@ -313,6 +313,11 @@ def cite_clock(clock_name: str, dir: str = "pyaging_data") -> None:
313
313
  citation = clock_dict["citation"]
314
314
  logger.info(f"Citation for {clock_name}:", indent_level=2)
315
315
  logger.info(citation, indent_level=2)
316
+ logger.info("Please also consider citing pyaging :)", indent_level=2)
317
+ logger.info(
318
+ 'de Lima Camillo, Lucas Paulo. "pyaging: a Python-based compendium of GPU-optimized aging clocks." bioRxiv (2023): 2023-11.',
319
+ indent_level=2,
320
+ )
316
321
  else:
317
322
  logger.warning(f"Citation not found in {clock_name}", indent_level=2)
318
323
  else:
@@ -438,3 +443,77 @@ def get_clock_metadata(clock_name: str, dir: str = "pyaging_data") -> None:
438
443
  logger.finish_progress(f"{message} finished")
439
444
 
440
445
  logger.done()
446
+
447
+
448
+ def get_clock_weights(clock_name: str, dir: str = "pyaging_data") -> dict:
449
+ """
450
+ Loads the specified aging clock from a remote source and returns a dictionary with its
451
+ components.
452
+
453
+ This function downloads the weights and configuration of a specified aging clock from a
454
+ remote server. It then loads and returns a dictionary with various components of the clock
455
+ such as its features, preprocessing or postprocessing steps, etc.
456
+
457
+ Parameters
458
+ ----------
459
+ clock_name : str
460
+ The name of the aging clock whose metadata is to be retrieved. The name is case-insensitive.
461
+ dir : str
462
+ The directory to deposit the downloaded file. Defaults to 'pyaging_data'.
463
+
464
+ Returns
465
+ -------
466
+ dict
467
+ A dictionary containing the following components of the clock:
468
+ - features: The features used by the clock.
469
+ - reference_feature_values: Reference values for features in case features are missing.
470
+ - weight_dict: A dictionary of weights used in the clock's model.
471
+ - preprocessing: Any preprocessing steps required for the clock's input data.
472
+ - postprocessing: Any postprocessing steps applied to the clock's output.
473
+ - preprocessing_helper: Any preprocessing helper file.
474
+ - postprocessing_helper: Any postprocessing helper file.
475
+
476
+ Notes
477
+ -----
478
+ The clock's weights and configuration are assumed to be stored in a .pt (PyTorch) file
479
+ on a remote server. The URL for the clock is constructed based on the clock's name.
480
+ If the clock or its components are not found, the function may fail or return incomplete
481
+ information.
482
+
483
+ The logger is used extensively for progress tracking and information logging, enhancing
484
+ transparency and user experience.
485
+
486
+ Examples
487
+ --------
488
+ >>> clock_dict = get_clock_weights("clock1", "pyaging_data")
489
+
490
+ """
491
+
492
+ logger = LoggerManager.gen_logger("get_clock_weights")
493
+ logger.first_info("Starting get_clock_weights function")
494
+
495
+ # Load all metadata
496
+ all_clock_metadata = load_clock_metadata(dir, logger, indent_level=1)
497
+
498
+ # Lowercase clock name
499
+ clock_name = clock_name.lower()
500
+ clock_dict = all_clock_metadata[clock_name]
501
+
502
+ # Check if clock name is available
503
+ if clock_name not in all_clock_metadata.keys():
504
+ logger.error(f"Clock {clock_name} is not yet available on pyaging")
505
+ raise ValueError
506
+
507
+ # Download weights
508
+ url = f"https://pyaging.s3.amazonaws.com/clocks/weights/{clock_name}.pt"
509
+ download(url, dir, logger, indent_level=1)
510
+
511
+ # Define the path to the clock weights file
512
+ weights_path = os.path.join(dir, f"{clock_name}.pt")
513
+
514
+ # Load the clock dictionary from the file
515
+ clock_dict = torch.load(weights_path)
516
+
517
+ logger.done()
518
+
519
+ return clock_dict
@@ -1,6 +1,6 @@
1
1
  [tool.poetry]
2
2
  name = "pyaging"
3
- version = "0.0.6"
3
+ version = "0.0.7"
4
4
  description = "A Python-based compendium of GPU-optimized aging clocks."
5
5
  authors = ["Lucas Paulo de Lima Camillo <lucas_camillo@alumni.brown.edu>"]
6
6
  license = "BSD"
File without changes
File without changes
File without changes