cap-anndata 0.1.1__tar.gz → 0.2.0__tar.gz

{cap_anndata-0.1.1 → cap_anndata-0.2.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: cap_anndata
-Version: 0.1.1
-Summary: Partial read of AnnData files for low-memory operations with large datasets.
+Version: 0.2.0
+Summary: Partial read/write of AnnData (h5ad) files for low-memory operations with large datasets.
 Home-page: https://github.com/cellannotation/cap-anndata
 Author: R. Mukhin, A. Isaev
 Author-email: roman@ebookapplications.com
@@ -25,41 +25,65 @@ CAP-AnnData enriches the AnnData ecosystem by offering tailored functionalities
 
 ## Getting Started
 
+### Installation
+Install CAP-AnnData via pip:
+
+```commandline
+pip install -U cap-anndata
+```
+
 ### Running Tests
-Ensure the integrity and reliability of CAP-AnnData on your system by running the unit tests in `test/unit_test.py`.
+Ensure the integrity and reliability of CAP-AnnData on your system by running the unit tests via `pytest` from the root of the repo.
+
+```commandline
+pip install pytest
+pytest test
+```
 
 Make sure Python 3.9 or newer is used, along with all requirements specified in requirements.txt
 
 ## How-TO:
 
-#### 1. Read AnnData File Dataframes
+#### 1. Access AnnData File DataFrames
 
 ##### Basic Reading
 By default, `CapAnnData` does not automatically read any data. To begin working with dataframes, you need to explicitly read the data from the AnnData file. You can read the entire dataframe or select specific columns. For partial reading, provide a list of column names.
 
 ```python
-import h5py
-from cap_anndata import CapAnnData
+from cap_anndata import read_h5ad
 
 file_path = "your_data.h5ad"
-with h5py.File(file_path, 'r') as file:
-    cap_adata = CapAnnData(file)
-
+with read_h5ad(file_path=file_path, edit=False) as cap_adata:
+    # Get the list of all obs columns in AnnData file
+    cap_adata.obs_keys()  # ['a', 'b', 'c']
     # Read all columns of 'obs'
     cap_adata.read_obs()
+    # Get the list of columns of DataFrame in memory
+    cap_adata.obs.columns  # ['a', 'b', 'c']
 
+    # Get the list of all var columns in AnnData file
+    cap_adata.var_keys()  # ['d', 'e', 'f']
     # Read specific columns of 'var'
-    cap_adata.read_var(columns=['gene_expression', 'dispersion'])
-
-    # Read all columns of raw.var
-    cap_adata.read_var(raw=True)
+    cap_adata.read_var(columns=['d'])
+    cap_adata.var.columns  # ['d']
+    # Read additional column
+    cap_adata.read_var(columns=['e'])
+    cap_adata.var.columns  # ['d', 'e']
+
+    # Read column and reset the in-memory DataFrame before that
+    cap_adata.read_var(columns=['f'], reset=True)
+    cap_adata.var.columns  # ['f']
+
+    # Read no columns of raw.var (only the index)
+    cap_adata.raw.read_var(columns=[])
 ```
 
-##### Non-existing columns
+##### Difference between `obs_keys()` and `obs.columns`
+`obs_keys()` returns the list of columns in the on-disc AnnData file, while `obs.columns` returns the list of columns in the in-memory DataFrame. The two lists may differ if you read only specific columns. If you modify the in-memory DataFrame, the `obs_keys()` will reflect the changes. BTW it is recommended to check the `obs_keys()` before the `overwrite()` call to avoid the AnnData file damage.
 
-If a column doesn't exist in the file, no error will be raised but the column will be missing in the resulting Dataframe. So, the list of columns saying more like "try to read this columns from the file". It is needed because we there is no way yet to check if the column exists before the read. 
+If a column doesn't exist in the file, no error will be raised but the column will be missing in the resulting DataFrame. So, the list of columns saying more like "try to read this columns from the file". It is needed because we there is no way yet to check if the column exists before the read. Exactly the same behavior is for the `var_keys()` and `var.columns`. 
 
-#### 2. Modify the AnnData File Dataframes In-Place
+#### 2. Modify the AnnData File DataFrames In-Place
 
 You can directly modify the dataframe by adding, renaming, or removing columns.
 
@@ -68,13 +92,14 @@ You can directly modify the dataframe by adding, renaming, or removing columns.
 cap_adata.obs['new_col'] = [value1, value2, value3]
 
 # Rename a column
-cap_adata.rename_column('old_col_name', 'new_col_name')
+cap_adata.obs.rename_column('old_col_name', 'new_col_name')
 
 # Remove a column
-cap_adata.remove_column('col_to_remove')
+cap_adata.obs.remove_column('col_to_remove')
 ```
 
 After modifications, you can overwrite the changes back to the AnnData file. If a value doesn't exist, it will be created.
+Note: `read_h5ad` must be called with `edit=True` argument to open `.h5ad` file in `r+` mode.
 
 ```python
 # overwrite all values which were read
@@ -84,7 +109,7 @@ cap_adata.overwrite()
 cap_adata.overwrite(['obs', 'var'])
 ```
 
-The full list of supported fields: `X`, `raw.X`, `obs`, `var`, `raw.var`, `obsm`, `uns`.
+The full list of supported fields: `obs`, `var`, `raw.var`, `obsm`, `uns`.
 
 #### 3. How to Read Few Columns but Overwrite One in a Dataframe
 
@@ -100,14 +125,19 @@ cap_adata.obs.drop(columns='sample', inplace=True)
 
 # Overwrite changes
 cap_adata.overwrite(['obs'])
+
+# NOTE that the line 
+# cap_adata.read_obs(columns=['sample'], reset=True)
+# Will override in-memory changes with values from the AnnData file
 ```
 
 #### 4. How to work with X and raw.X
 
-The CapAnnData package won't read any field by default. However, the `X` and `raw.X` will be linked to the backed matrices automatically upon the first request to those fields.
+The CapAnnData package won't read any field by default. However, the `X` and `raw.X` will be linked to the backed matrices automatically upon the first request to those fields. 
+The X object will be returned as the `h5py.Dataset` or `AnnData.experimental.sparse_dataset`.
 
 ```python
-with h5py.File(path) as file:
+with read_h5ad(file_path=file_path, edit=False) as cap_adata:
     # self.X is None here
     cap_adata = CapAnnData(file)  
 
@@ -135,13 +165,13 @@ s_ = np.s_[mask, :5]
 
 #### 5. How to handle obsm embeddings matrixes
 
-By the default the CapAnnData will not read the embeddings matrix. The link to the h5py objects will be created upon the first call of the `.obsm` property. Alike the AnnData package the call like `cap_adata.obsm["X_tsne"]` will not return the in-memory matrix but will return the backed version instead. We can get the information about the name and shape of the embeddings without taking the whole matrixes in the memory!
+By the default the CapAnnData will not read the embeddings matrix. 
+The link to the h5py objects will be created upon the first call of the `.obsm` property. 
+Alike the AnnData package the call like `cap_adata.obsm["X_tsne"]` will not return the in-memory matrix but will return the backed version instead. 
+It is possible to get the information about the name and shape of the embeddings without taking the whole matrix in the memory.
 
 ```python
-with h5py.File(path) as file:
-    # initialization
-    cap_adata = CapAnnData(file) 
-
+with read_h5ad(file_path=file_path, edit=False) as cap_adata:
     # will return the list of strings
     obsm_keys = cap_adata.obsm_keys()  
 
@@ -158,10 +188,7 @@ with h5py.File(path) as file:
 The `CapAnnData` class will lazely link the uns section upon the first call but ***WILL NOT*** read it into memory. Instead, the dictionary of the pairs `{'key': "__NotLinkedObject"}` will be creted. It allow to get the list of keys before the actual read. To read the uns section in the memory the `.read_uns(keys)` method must be called.
 
 ```python
-with h5py.File(path) as file:
-    # initialization
-    cap_adata = CapAnnData(file) 
-
+with read_h5ad(file_path=file_path, edit=True) as cap_adata:
     # will return the keys() object
     keys = cap_adata.uns.keys()  
 
@@ -197,3 +224,28 @@ To save `uns` changes the method `CapAnnData.overwrite()` must be called.
 cap_adata.overwrite()  # all in-memory fields will be overwritten
 cap_adata.overwrite(["uns"])  # overwrite the uns secion only
 ```
+
+#### 7. Join and Merge DataFrames
+
+Cap-AnnData provides enhanced methods for joining and merging dataframes, preserving column order and data integrity
+
+```python
+from cap_anndata import CapAnnDataDF
+import pandas as pd
+
+data1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
+data2 = pd.DataFrame({'D': [7, 8, 9], 'E': [10, 11, 12]})
+cap_anndata_df1 = CapAnnDataDF.from_df(data1, column_order=['A', 'B', 'C'])
+
+cap_df = cap_anndata_df1.join(data2, how='left')
+
+cap_df.columns  # ['A', 'B', 'D', 'E']
+cap_df.column_order  # ['A', 'B', 'C', 'D', 'E']
+
+data3 = pd.DataFrame({'A': [2, 3, 4], 'D': [10, 11, 12]})
+cap_df = cap_anndata_df1.merge(data3, on='A')
+
+cap_df.columns  # ['A', 'B', 'D']
+cap_df.column_order  # ['A', 'B', 'C', 'D']
+cap_df.shape  # (2, 3)
+```

{cap_anndata-0.1.1 → cap_anndata-0.2.0}/README.md

@@ -5,41 +5,65 @@ CAP-AnnData enriches the AnnData ecosystem by offering tailored functionalities
 
 ## Getting Started
 
+### Installation
+Install CAP-AnnData via pip:
+
+```commandline
+pip install -U cap-anndata
+```
+
 ### Running Tests
-Ensure the integrity and reliability of CAP-AnnData on your system by running the unit tests in `test/unit_test.py`.
+Ensure the integrity and reliability of CAP-AnnData on your system by running the unit tests via `pytest` from the root of the repo.
+
+```commandline
+pip install pytest
+pytest test
+```
 
 Make sure Python 3.9 or newer is used, along with all requirements specified in requirements.txt
 
 ## How-TO:
 
-#### 1. Read AnnData File Dataframes
+#### 1. Access AnnData File DataFrames
 
 ##### Basic Reading
 By default, `CapAnnData` does not automatically read any data. To begin working with dataframes, you need to explicitly read the data from the AnnData file. You can read the entire dataframe or select specific columns. For partial reading, provide a list of column names.
 
 ```python
-import h5py
-from cap_anndata import CapAnnData
+from cap_anndata import read_h5ad
 
 file_path = "your_data.h5ad"
-with h5py.File(file_path, 'r') as file:
-    cap_adata = CapAnnData(file)
-
+with read_h5ad(file_path=file_path, edit=False) as cap_adata:
+    # Get the list of all obs columns in AnnData file
+    cap_adata.obs_keys()  # ['a', 'b', 'c']
     # Read all columns of 'obs'
     cap_adata.read_obs()
+    # Get the list of columns of DataFrame in memory
+    cap_adata.obs.columns  # ['a', 'b', 'c']
 
+    # Get the list of all var columns in AnnData file
+    cap_adata.var_keys()  # ['d', 'e', 'f']
     # Read specific columns of 'var'
-    cap_adata.read_var(columns=['gene_expression', 'dispersion'])
-
-    # Read all columns of raw.var
-    cap_adata.read_var(raw=True)
+    cap_adata.read_var(columns=['d'])
+    cap_adata.var.columns  # ['d']
+    # Read additional column
+    cap_adata.read_var(columns=['e'])
+    cap_adata.var.columns  # ['d', 'e']
+
+    # Read column and reset the in-memory DataFrame before that
+    cap_adata.read_var(columns=['f'], reset=True)
+    cap_adata.var.columns  # ['f']
+
+    # Read no columns of raw.var (only the index)
+    cap_adata.raw.read_var(columns=[])
 ```
 
-##### Non-existing columns
+##### Difference between `obs_keys()` and `obs.columns`
+`obs_keys()` returns the list of columns in the on-disc AnnData file, while `obs.columns` returns the list of columns in the in-memory DataFrame. The two lists may differ if you read only specific columns. If you modify the in-memory DataFrame, the `obs_keys()` will reflect the changes. BTW it is recommended to check the `obs_keys()` before the `overwrite()` call to avoid the AnnData file damage.
 
-If a column doesn't exist in the file, no error will be raised but the column will be missing in the resulting Dataframe. So, the list of columns saying more like "try to read this columns from the file". It is needed because we there is no way yet to check if the column exists before the read. 
+If a column doesn't exist in the file, no error will be raised but the column will be missing in the resulting DataFrame. So, the list of columns saying more like "try to read this columns from the file". It is needed because we there is no way yet to check if the column exists before the read. Exactly the same behavior is for the `var_keys()` and `var.columns`. 
 
-#### 2. Modify the AnnData File Dataframes In-Place
+#### 2. Modify the AnnData File DataFrames In-Place
 
 You can directly modify the dataframe by adding, renaming, or removing columns.
 
@@ -48,13 +72,14 @@ You can directly modify the dataframe by adding, renaming, or removing columns.
 cap_adata.obs['new_col'] = [value1, value2, value3]
 
 # Rename a column
-cap_adata.rename_column('old_col_name', 'new_col_name')
+cap_adata.obs.rename_column('old_col_name', 'new_col_name')
 
 # Remove a column
-cap_adata.remove_column('col_to_remove')
+cap_adata.obs.remove_column('col_to_remove')
 ```
 
 After modifications, you can overwrite the changes back to the AnnData file. If a value doesn't exist, it will be created.
+Note: `read_h5ad` must be called with `edit=True` argument to open `.h5ad` file in `r+` mode.
 
 ```python
 # overwrite all values which were read
@@ -64,7 +89,7 @@ cap_adata.overwrite()
 cap_adata.overwrite(['obs', 'var'])
 ```
 
-The full list of supported fields: `X`, `raw.X`, `obs`, `var`, `raw.var`, `obsm`, `uns`.
+The full list of supported fields: `obs`, `var`, `raw.var`, `obsm`, `uns`.
 
 #### 3. How to Read Few Columns but Overwrite One in a Dataframe
 
@@ -80,14 +105,19 @@ cap_adata.obs.drop(columns='sample', inplace=True)
 
 # Overwrite changes
 cap_adata.overwrite(['obs'])
+
+# NOTE that the line 
+# cap_adata.read_obs(columns=['sample'], reset=True)
+# Will override in-memory changes with values from the AnnData file
 ```
 
 #### 4. How to work with X and raw.X
 
-The CapAnnData package won't read any field by default. However, the `X` and `raw.X` will be linked to the backed matrices automatically upon the first request to those fields.
+The CapAnnData package won't read any field by default. However, the `X` and `raw.X` will be linked to the backed matrices automatically upon the first request to those fields. 
+The X object will be returned as the `h5py.Dataset` or `AnnData.experimental.sparse_dataset`.
 
 ```python
-with h5py.File(path) as file:
+with read_h5ad(file_path=file_path, edit=False) as cap_adata:
     # self.X is None here
     cap_adata = CapAnnData(file)  
 
@@ -115,13 +145,13 @@ s_ = np.s_[mask, :5]
 
 #### 5. How to handle obsm embeddings matrixes
 
-By the default the CapAnnData will not read the embeddings matrix. The link to the h5py objects will be created upon the first call of the `.obsm` property. Alike the AnnData package the call like `cap_adata.obsm["X_tsne"]` will not return the in-memory matrix but will return the backed version instead. We can get the information about the name and shape of the embeddings without taking the whole matrixes in the memory!
+By the default the CapAnnData will not read the embeddings matrix. 
+The link to the h5py objects will be created upon the first call of the `.obsm` property. 
+Alike the AnnData package the call like `cap_adata.obsm["X_tsne"]` will not return the in-memory matrix but will return the backed version instead. 
+It is possible to get the information about the name and shape of the embeddings without taking the whole matrix in the memory.
 
 ```python
-with h5py.File(path) as file:
-    # initialization
-    cap_adata = CapAnnData(file) 
-
+with read_h5ad(file_path=file_path, edit=False) as cap_adata:
     # will return the list of strings
     obsm_keys = cap_adata.obsm_keys()  
 
@@ -138,10 +168,7 @@ with h5py.File(path) as file:
 The `CapAnnData` class will lazely link the uns section upon the first call but ***WILL NOT*** read it into memory. Instead, the dictionary of the pairs `{'key': "__NotLinkedObject"}` will be creted. It allow to get the list of keys before the actual read. To read the uns section in the memory the `.read_uns(keys)` method must be called.
 
 ```python
-with h5py.File(path) as file:
-    # initialization
-    cap_adata = CapAnnData(file) 
-
+with read_h5ad(file_path=file_path, edit=True) as cap_adata:
     # will return the keys() object
     keys = cap_adata.uns.keys()  
 
@@ -177,3 +204,28 @@ To save `uns` changes the method `CapAnnData.overwrite()` must be called.
 cap_adata.overwrite()  # all in-memory fields will be overwritten
 cap_adata.overwrite(["uns"])  # overwrite the uns secion only
 ```
+
+#### 7. Join and Merge DataFrames
+
+Cap-AnnData provides enhanced methods for joining and merging dataframes, preserving column order and data integrity
+
+```python
+from cap_anndata import CapAnnDataDF
+import pandas as pd
+
+data1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
+data2 = pd.DataFrame({'D': [7, 8, 9], 'E': [10, 11, 12]})
+cap_anndata_df1 = CapAnnDataDF.from_df(data1, column_order=['A', 'B', 'C'])
+
+cap_df = cap_anndata_df1.join(data2, how='left')
+
+cap_df.columns  # ['A', 'B', 'D', 'E']
+cap_df.column_order  # ['A', 'B', 'C', 'D', 'E']
+
+data3 = pd.DataFrame({'A': [2, 3, 4], 'D': [10, 11, 12]})
+cap_df = cap_anndata_df1.merge(data3, on='A')
+
+cap_df.columns  # ['A', 'B', 'D']
+cap_df.column_order  # ['A', 'B', 'C', 'D']
+cap_df.shape  # (2, 3)
+```

{cap_anndata-0.1.1 → cap_anndata-0.2.0}/cap_anndata/__init__.py

@@ -1,6 +1,10 @@
 from .backed_df import CapAnnDataDF
 from .backed_uns import CapAnnDataUns
 from .cap_anndata import CapAnnData
+from .reader import (
+    read_directly,
+    read_h5ad,
+)
 
 
 __all__ = ["CapAnnData"]

cap_anndata-0.2.0/cap_anndata/backed_df.py

@@ -0,0 +1,69 @@
+import pandas as pd
+import numpy as np
+from typing import List, Any
+import logging
+
+from pandas._typing import Self
+from pandas.core.generic import bool_t
+
+logger = logging.getLogger(__name__)
+
+
+class CapAnnDataDF(pd.DataFrame):
+    """
+    The class to expand the pandas DataFrame behaviour to support partial
+    reading and writing of AnnData obs and var (raw.var) fields.
+    The main feature of the class is handling <column-order> attribute
+    which must be a copy of h5py.Group attribute
+    """
+
+    _metadata = ["column_order"]
+
+    def rename_column(self, old_name: str, new_name: str) -> None:
+        i = np.where(self.column_order == old_name)[0]
+        self.column_order[i] = new_name
+        self.rename(columns={old_name: new_name}, inplace=True)
+
+    def remove_column(self, col_name: str) -> None:
+        i = np.where(self.column_order == col_name)[0]
+        self.column_order = np.delete(self.column_order, i)
+        self.drop(columns=[col_name], inplace=True)
+
+    def __setitem__(self, key, value) -> None:
+        if key not in self.column_order:
+            self.column_order = np.append(self.column_order, key)
+        return super().__setitem__(key, value)
+
+    @classmethod
+    def from_df(cls, df: pd.DataFrame, column_order: List[str] = None) -> Self:
+        if column_order is None:
+            column_order = df.columns.to_numpy()
+
+        new_inst = cls(df)
+        new_inst.column_order = column_order
+        return new_inst
+
+    def join(self, other: Any, **kwargs) -> Self:
+        result = super().join(other=other, **kwargs)
+        if isinstance(other, CapAnnDataDF):
+            new_columns = [
+                col for col in other.column_order if col not in self.column_order
+            ]
+        else:
+            new_columns = [col for col in other.columns if col not in self.column_order]
+        column_order = np.append(self.column_order, new_columns)
+        return self.from_df(result, column_order=column_order)
+
+    def merge(self, right, **kwargs) -> Self:
+        result = super().merge(right=right, **kwargs)
+        if isinstance(right, CapAnnDataDF):
+            new_columns = [
+                col for col in right.column_order if col not in self.column_order
+            ]
+        else:
+            new_columns = [col for col in right.columns if col not in self.column_order]
+        column_order = np.append(self.column_order, new_columns)
+        return self.from_df(result, column_order=column_order)
+
+    def copy(self, deep: bool_t | None = True) -> Self:
+        return self.from_df(super().copy(deep=deep), column_order=self.column_order)