ngio 0.2.9__tar.gz → 0.3.0a1__tar.gz

{ngio-0.2.9 → ngio-0.3.0a1}/.github/workflows/build_docs.yml

@@ -3,7 +3,7 @@ name: Build Docs
 on:
   push:
     branches:
-      - main
+      - dev
     tags:
       - "v*"
 

{ngio-0.2.9 → ngio-0.3.0a1}/.github/workflows/ci.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+      - dev
     tags:
       - "v*"
   pull_request:

ngio-0.3.0a1/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## [Unreleased]
+
+### Added
+
+- Initial version of the changelog.

{ngio-0.2.9 → ngio-0.3.0a1}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: ngio
-Version: 0.2.9
+Version: 0.3.0a1
 Summary: Next Generation file format IO
-Project-URL: homepage, https://github.com/lorenzocerrone/ngio
-Project-URL: repository, https://github.com/lorenzocerrone/ngio
+Project-URL: homepage, https://github.com/fractal-analytics-platform/ngio
+Project-URL: repository, https://github.com/fractal-analytics-platform/ngio
 Author-email: Lorenzo Cerrone <lorenzo.cerrone@uzh.ch>
 License: BSD-3-Clause
 License-File: LICENSE
@@ -50,6 +50,7 @@ Requires-Dist: mkdocs; extra == 'docs'
 Requires-Dist: mkdocs-autorefs; extra == 'docs'
 Requires-Dist: mkdocs-git-committers-plugin-2; extra == 'docs'
 Requires-Dist: mkdocs-git-revision-date-localized-plugin; extra == 'docs'
+Requires-Dist: mkdocs-include-markdown-plugin; extra == 'docs'
 Requires-Dist: mkdocs-jupyter; extra == 'docs'
 Requires-Dist: mkdocs-material; extra == 'docs'
 Requires-Dist: mkdocstrings[python]; extra == 'docs'

ngio-0.3.0a1/docs/changelog.md

@@ -0,0 +1,5 @@
+{%
+    include-markdown "../CHANGELOG.md"
+    start="<!--intro-start-->"
+    end="<!--intro-end-->"
+%}

{ngio-0.2.9 → ngio-0.3.0a1}/docs/code_of_conduct.md

@@ -1,2 +1,4 @@
+# Code of Conduct
+
 !!! warning
     The library is still in the early stages of development, the code of conduct is not yet established.

{ngio-0.2.9 → ngio-0.3.0a1}/docs/getting_started/1_ome_zarr_containers.md

@@ -89,7 +89,7 @@ Examples of the OME-Zarr metadata access:
 
 To access images, labels, and tables, you can use the `get_image`, `get_label`, and `get_table` methods of the `OME-Zarr Container` object.
 
-A variety of examples and additional information can be found in the [Images and Labels](./2_images.md), and [Tables](../3_tables.md) sections.
+A variety of examples and additional information can be found in the [Images and Labels](./2_images.md), and [Tables](./3_tables.md) sections.
 
 ## Creating derived images
 

{ngio-0.2.9 → ngio-0.3.0a1}/docs/getting_started/3_tables.md

@@ -190,7 +190,7 @@ Tables (differently from Images and Labels) can be purely in memory objects, and
     >>> from ngio.tables import FeatureTable
     >>> import pandas as pd
     >>> example_data = pd.DataFrame({"label": [1, 2, 3], "area": [100, 200, 300]})
-    >>> feature_table = FeatureTable(dataframe=example_data)
+    >>> feature_table = FeatureTable(table_data=example_data)
     >>> feature_table
     >>> print(feature_table) # markdown-exec: hide
     ```
@@ -203,7 +203,7 @@ Tables (differently from Images and Labels) can be purely in memory objects, and
     >>> from ngio.tables import GenericTable
     >>> import pandas as pd
     >>> example_data = pd.DataFrame({"area": [100, 200, 300], "perimeter": [50, 60, 70]})
-    >>> generic_table = GenericTable(dataframe=example_data)
+    >>> generic_table = GenericTable(table_data=example_data)
     >>> generic_table
     >>> print(generic_table) # markdown-exec: hide
     ```
@@ -212,7 +212,7 @@ Tables (differently from Images and Labels) can be purely in memory objects, and
     >>> from ngio.tables import GenericTable
     >>> import anndata as ad
     >>> adata = ad.AnnData(X=np.random.rand(10, 5), obs=pd.DataFrame({"cell_type": ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]}))
-    >>> generic_table = GenericTable(anndata=adata)
+    >>> generic_table = GenericTable(table_data=adata)
     >>> generic_table
     >>> print(generic_table) # markdown-exec: hide
     ```

ngio-0.3.0a1/docs/table_specs/backend.md

@@ -0,0 +1,112 @@
+# Table Backends
+
+In ngio we implemented four different table backends. Each table backend is a python class that can serialize tabular data into OME-Zarr containers.
+
+These backends are wrappers around existing tooling implemented in `anndata`, `pandas`, and `polars`.
+Currently, we provide a thin layer of metadata and table normalization to ensure that tables are serialized/deserialized in a consistent way across the different backends and across different table objects.
+
+In particular, we provide the metadata that describes the intended index key and type of the table for each backend.
+
+## AnnData Backend
+
+AnnData is a widely used format in single-cell genomics, and can natively store complex tabular data in a Zarr group. The AnnData backend in ngio is a wrapper around the `anndata` library, which performs some table normalization for consistency and compatibility with the ngio table specifications.
+
+The following normalization steps are applied to each table before saving it to the AnnData backend:
+
+- We separate the table in two parts: The floating point columns are casted to `float32` and stored as `X` in the AnnData object, while the categorical, boolean, and integer columns are stored as `obs`.
+- The index column is cast to a string, and the name and original type is stored in the zarr attributes.
+
+AnnData backend metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "str", // Either "int" or "str"
+}
+```
+
+Additionally, the AnnData package will write some additional metadata to the group attributes
+
+```json
+{
+    "encoding-type": "anndata",
+    "encoding-version": "0.1.0",
+}
+```
+
+## Parquet Backend
+
+The Parquet backend is a high-performance columnar storage format that is widely used in big data processing. It is designed to efficiently store large datasets and can be used with various data processing frameworks.
+Another advantage of the Parquet backend is that it can be used lazily, meaning that the data is not loaded into memory until it is needed. This can be useful for working with large datasets that do not fit into memory.
+
+Parquet backend metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "parquet", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+The Zarr group directory will contain the Parquet file, and the metadata will be stored in the group attributes.
+
+```bash
+table.zarr          # Zarr group for the table
+├── table.parquet   # Parquet file containing the table data
+├── .zattrs         # Zarr group attributes containing the metadata
+└── .zgroup         # Zarr group metadata
+```
+
+## CSV Backend
+
+The CSV backend is a plain text format that is widely used for tabular data. It is easy to read and write, and can be used across many different tools.
+
+The CSV backen in ngio follows closely the same specifications as the Parquet backend, with the following metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "csv", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+The Zarr group directory will contain the CSV file, and the metadata will be stored in the group attributes.
+
+```bash
+table.zarr         # Zarr group for the table
+├── table.csv      # CSV file containing the table data
+├── .zattrs        # Zarr group attributes containing the metadata
+└── .zgroup        # Zarr group metadata
+```
+
+## JSON Backend
+
+The JSON backend serializes the table data into the Zarr group attributes as a JSON object. This backend is useful for tiny tables.
+
+JSON backend metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "json", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "int" // Either "int" or "str"
+}
+```
+
+The table will be stored in a subgroup of the Zarr group, and the metadata will be stored in the group attributes. Storing the table in a subgroup instead of a standalone json file allows for easier access via the Zarr API.
+
+```bash
+table.zarr          # Zarr group for the table
+└── table           # Zarr subgroup containing the table data
+    ├── .zattrs     # the json table data serialized as a JSON object
+    └── .zgroup     # Zarr group metadata
+├── .zattrs         # Zarr group attributes containing the metadata
+└── .zgroup         # Zarr group metadata
+```

ngio-0.3.0a1/docs/table_specs/overview.md

@@ -0,0 +1,92 @@
+# Tables Overview
+
+Ngio's architecture is designed to tightly integrate image and tabular data. For this purpose we developed custom specifications for serializing and deserializing tabular data into OME-Zarr containers, and semantically typed tables derived from the [fractal table specification](https://fractal-analytics-platform.github.io/fractal-tasks-core/tables/).
+
+## Architecture
+
+The ngio tables architectures is composed of three main components:
+
+### 1. Table Backends
+
+A backend module is a class that can serialize tabular data into OME-Zarr containers. We currently support four on-disk file formats:
+
+- **AnnData**: Commonly used in single-cell genomics and was the standard table for the initial Fractal table spec.
+- **Parquet**: A columnar storage file format optimized for large datasets.
+- **CSV**: A simple text format for tabular data, easily human readable and writable.
+- **JSON**: A lightweight data interchange format that both readable and efficient for small tables.
+
+A more detailed description of the backend module can be found in the [Table Backends documentation](backend.md).
+
+### 2. In-Memory Table Objects
+
+These are Python objects that represent the tabular data in memory. They provide a convenient interface for manipulating and analyzing the data without needing to interact directly with the underlying file format. We support the following in-memory table objects:
+
+- **Pandas DataFrame**: The most commonly used data structure for tabular data in Python.
+- **Polars LazyFrame**: A fast DataFrame implementation that allows for lazy evaluation and efficient computation on large datasets.
+- **AnnData**: A specialized data structure for single-cell genomics data, which goes beyond simple tabular data.
+
+We also provide utilities to convert between these in-memory representations in a standardized way based on the table type specifications/metadata.
+
+### 3. Table Type Specifications
+
+These specifications define structured tables that standardize common table types used in image analysis. We have defined five table types so far:
+
+- **Generic Tables**: A flexible table type that can represent any tabular data. See more in the [Generic Tables documentation](table_types/generic_table.md).
+- **ROI Tables**: A table type specifically designed for representing Regions of Interest (ROIs) in images. See more in the [ROI Tables documentation](table_types/roi_table.md).
+- **Masking ROI Tables**: A specialized table type for representing ROIs that are associated with specific labels in a OME-Zarr label image. See more in the [Masking ROI Tables documentation](table_types/masking_roi_table.md).
+- **Feature Tables**: A table type for representing features extracted from images. This table is also associated with a specific label image. See more in the [Feature Tables documentation](table_types/feature_table.md).
+- **Condition Tables**: A table to represent experimental conditions or metadata associated with images or experiments. See more in the [Condition Tables documentation](table_types/condition_table.md).
+
+## Tables Groups
+
+Tables in OME-Zarr images are organized into groups of tables. Each group is saved in a Zarr group, and can be associated with a specific image or plate. The tables groups are:
+
+- **Image Tables**: These tables are a sub group of the OME-Zarr image group and contain metadata or features related only to that specific image. The `.zarr` hierarchy is based on image [specification in NGFF 0.4](https://ngff.openmicroscopy.org/0.4/index.html#image-layout). The subgroup structure is based on the approach of the OME-Zarr `labels` group.
+
+```bash
+image.zarr        # Zarr group for a OME-Zarr image
+|
+├── 0             # Zarr array for multiscale level 0
+├── ...
+├── N             # Zarr array for multiscale level N
+|
+├── labels        # Zarr subgroup with a list of labels associated to this image
+|   ├── label_A   # Zarr subgroup for a given label
+|   ├── label_B   # Zarr subgroup for a given label
+|   └── ...
+|
+└── tables        # Zarr subgroup with a list of tables associated to this image
+    ├── table_1   # Zarr subgroup for a given table
+    ├── table_2   # Zarr subgroup for a given table
+    └── ...
+```
+
+- **Plate Tables**: These tables are a sub group of the OME-Zarr plate group and contain metadata or features related only to that specific plate.
+  
+```bash
+plate.zarr       # Zarr group for a OME-Zarr HCS plate
+|
+├── A             # Row A of the plate
+|   ├── 1         # Column 0 of row A
+|   |   ├── 0     # Acquisition 0 of column A1
+|   |   ├── 1     # Acquisition 1 of column A1
+|   |   └── ...   # Other acquisitions of column A1
+...
+├── tables        # Zarr subgroup with a list of tables associated to this plate
+|   ├── table_1   # Zarr subgroup for a given table
+|   ├── table_2   # Zarr subgroup for a given table
+|   └── ...
+└── ...
+```
+
+If a plate table contains per image information, the table should contain a `row`, `column`, and `path_in_well` columns.
+
+## Tables Group Attributes
+
+The Zarr attributes of the tables group must include the key tables, pointing to the list of all tables (this simplifies discovery of tables associated to the current OME-Zarr image or plate), as in
+
+```json
+{
+    "tables": ["table_1", "table_2"]
+}
+```

ngio-0.3.0a1/docs/table_specs/table_types/condition_table.md

@@ -0,0 +1,28 @@
+# Condition Table
+
+A condition table is a simple table that can be used to represent experimental conditions or metadata associated with images or experiments. It is a flexible table type that can be used to store any kind of metadata related to the images or experiments.
+
+Example condition table:
+
+| Cell Type | Drug     | Dose |
+|-----------|-----------|------|
+| A         | Drug A   | 10   |
+| A         | Drug B   | 20   |
+
+## Specifications
+
+### V1
+
+A condition table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // Condition table metadata
+    "type": "condition_table",
+    "table_version": "1",
+    // Backend metadata
+    "backend": "csv", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the condition table, which is used to identify each row.
+    "index_type": "int" // Either "int" or "str"
+}
+```

ngio-0.3.0a1/docs/table_specs/table_types/custom_table.md

@@ -0,0 +1,6 @@
+# Add a Custom Table
+
+Ngio allows users to define custom tables that can be used to store any kind of tabular data. Custom tables are flexible and can be used to represent any kind of data that does not fit into the predefined table types.
+
+!!! warning
+    The library is still in the early stages and full documentation for custom tables is not yet available.

ngio-0.3.0a1/docs/table_specs/table_types/feature_table.md

@@ -0,0 +1,54 @@
+# Feature Tables
+
+A feature table is a table type for representing per object features in an image. Each row in a feature table corresponds to a specific label in the label image.
+
+Feature tables can optionally include metadata to specify the type of features stored in each column:
+
+- `measurement`: A quantitative measurement of the object, such as area, perimeter, or intensity.
+- `categorical`: A categorical feature of the object, such as a classification label or a type.
+- `metadata`: Additional free-from columns that can be used to store any other information about the object, but that should not be used for analysis/classification purposes.
+
+These feature types inform casting of the values when serialising a table and can be used in downstream analysis to select specific subsets of features. The feature type can be explicitly specified in the feature table metadata. Alternatively, if a column is not specified, we apply the following casting rules:
+
+- If the column contains only numeric values, it is considered a `measurement`.
+- If the column contains string or boolean values, it is considered a `categorical`.
+- The index column is considered a `categorical` feature.
+
+## Specifications
+
+### V1
+
+A feature table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // Feature table metadata
+    "type": "feature_table",
+    "table_version": "1",
+    "region": {"path": "../labels/label_DAPI"}, // Path to the label image associated with this feature table
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "label", 
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+Additionally, it can include feature type information such as:
+
+```json
+{
+    "categorical_columns": [
+        "label",
+        "cell_type",
+    ],
+    "measurement_columns": [
+        "area",
+        "perimeter",
+        "intensity_mean",
+        "intensity_std"
+    ],
+    "metadata_columns": [
+        "description",
+    ],
+}
+```

ngio-0.3.0a1/docs/table_specs/table_types/generic_table.md

@@ -0,0 +1,23 @@
+# Generic Tables
+
+A generic table is a flexible table type that can represent any tabular data. It is not tied to any specific domain or use case, making it suitable for a wide range of custom applications.
+
+Generic tables can used as a safe fallback when trying to read a table that does not match any other specific table type.
+
+## Specifications
+
+### V1
+
+A generic table should include the following metadata fields in the group attributes:
+
+```json
+{
+    // Generic table metadata
+    "type": "generic_table",
+    "table_version": "1",
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the generic table, which is used to identify each row.
+    "index_type": "int" // Either "int" or "str"
+}
+```

ngio-0.3.0a1/docs/table_specs/table_types/masking_roi_table.md

@@ -0,0 +1,35 @@
+# Masking ROI Tables
+
+A masking ROI table is a specialized table type for representing Regions of Interest (ROIs) that are associated with specific labels in a label image.
+Each row in a masking ROI table corresponds to a specific label in the label image.
+
+Masking ROI tables can be used for several purposes, such as:
+
+- Feature extraction from specific regions in the image.
+- Masking specific regions in the image for further processing. For example a masking ROI table could store the ROIs for specific tissues, and for each of these ROIs we would like to perform cell segmentation.
+
+## Specifications
+
+### V1
+
+A ROI table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // ROI table metadata
+    "type": "masking_roi_table",
+    "table_version": "1",
+    "region": {"path": "../labels/label_DAPI"}, // Path to the label image associated with this masking ROI table
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "label", // The default index key for the ROI table, which is used to identify each ROI. 
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+Moreover the ROI table must include the following columns:
+
+- `x_micrometer`, `y_micrometer`, `z_micrometer`: the top-left corner coordinates of the ROI in micrometers.
+- `len_x_micrometer`, `len_y_micrometer`, `len_z_micrometer`: the size of the ROI in micrometers along each axis.
+
+Additionally, each ROI can include the following optional columns: see [ROI Table](./roi_table.md).

ngio-0.3.0a1/docs/table_specs/table_types/roi_table.md

@@ -0,0 +1,39 @@
+# ROI Table
+
+A ROI table defines regions of space which are axes-aligned bounding boxes in the image space.
+
+ROI tables can be used for several purposes, such as:
+
+- Storing information about the Microscope Field of View (FOV).
+- Storing arbitrary regions of interest (ROIs).
+- Use them as masks for other processes, such as segmentation or feature extraction.
+
+## Specifications
+
+### V1
+
+A ROI table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // ROI table metadata
+    "type": "roi_table",
+    "table_version": "1",
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "FieldIndex", // The default index key for the ROI table, which is used to identify each ROI. 
+    "index_type": "str", // Either "int" or "str"
+}
+```
+
+Moreover the ROI table must include the following columns:
+
+- `x_micrometer`, `y_micrometer`, `z_micrometer`: the top-left corner coordinates of the ROI in micrometers.
+- `len_x_micrometer`, `len_y_micrometer`, `len_z_micrometer`: the size of the ROI in micrometers along each axis.
+
+Additionally, each ROI can include the following optional columns:
+
+- `x_micrometer_original` and `y_micrometer_original` which are the original coordinates of the ROI in micrometers. These are typically used when the data is saved in different coordinates during conversion, e.g. to avoid overwriting data from overlapping ROIs.
+- `translation_x`, `translation_y` and `translation_z`, which are used during registration of multiplexing acquisitions.
+
+The user can also add additional columns to the ROI table, but these columns will not be exposed in the ROI table API.

{ngio-0.2.9 → ngio-0.3.0a1}/mkdocs.yml

@@ -52,6 +52,7 @@ plugins:
   - search
   - autorefs
   - markdown-exec
+  - include-markdown
   - mkdocstrings:
       handlers:
         python:
@@ -99,6 +100,16 @@ nav:
     - tutorials/feature_extraction.ipynb
     - tutorials/hcs_processing.ipynb
 
+  - Table Specifications:
+    - "Overview": table_specs/overview.md
+    - "Table Backends": table_specs/backend.md
+    - "Table Types":
+      - "ROI Table": table_specs/table_types/roi_table.md
+      - "Masking ROI Table": table_specs/table_types/masking_roi_table.md
+      - "Feature Table": table_specs/table_types/feature_table.md
+      - "Condition Table": table_specs/table_types/condition_table.md
+      - "Add Custom Table": table_specs/table_types/custom_table.md
+  
   - API Reference:
     - "ngio": api/ngio.md
     - "ngio.images": api/images.md
@@ -106,6 +117,8 @@ nav:
     - "ngio.hcs": api/hcs.md
     - "ngio.utils": api/utils.md
     - "ngio.common": api/common.md
+    - changelog.md
+
   - Contributing:
     - "Contributing Guide": contributing.md
     - "Code of Conduct": code_of_conduct.md

{ngio-0.2.9 → ngio-0.3.0a1}/pyproject.toml

@@ -71,6 +71,7 @@ dev = [
 
 docs = [
     "mkdocs",
+    "mkdocs-include-markdown-plugin",
     "mkdocs-material",
     "mkdocstrings[python]",
     "mkdocs-jupyter",
@@ -87,8 +88,8 @@ docs = [
 
 
 [project.urls]
-homepage = "https://github.com/lorenzocerrone/ngio"
-repository = "https://github.com/lorenzocerrone/ngio"
+homepage = "https://github.com/fractal-analytics-platform/ngio"
+repository = "https://github.com/fractal-analytics-platform/ngio"
 
 # Entry points
 # https://peps.python.org/pep-0621/#entry-points

{ngio-0.2.9 → ngio-0.3.0a1}/src/ngio/common/__init__.py

@@ -24,6 +24,15 @@ from ngio.common._slicer import (
     numpy_get_slice,
     numpy_set_slice,
 )
+from ngio.common._table_ops import (
+    concatenate_image_tables,
+    concatenate_image_tables_as,
+    concatenate_image_tables_as_async,
+    concatenate_image_tables_async,
+    conctatenate_tables,
+    list_image_tables,
+    list_image_tables_async,
+)
 from ngio.common._zoom import dask_zoom, numpy_zoom
 
 __all__ = [
@@ -34,6 +43,11 @@ __all__ = [
     "SliceTransform",
     "compute_and_slices",
     "compute_masking_roi",
+    "concatenate_image_tables",
+    "concatenate_image_tables_as",
+    "concatenate_image_tables_as_async",
+    "concatenate_image_tables_async",
+    "conctatenate_tables",
     "consolidate_pyramid",
     "dask_get_slice",
     "dask_set_slice",
@@ -41,6 +55,8 @@ __all__ = [
     "get_masked_pipe",
     "get_pipe",
     "init_empty_pyramid",
+    "list_image_tables",
+    "list_image_tables_async",
     "numpy_get_slice",
     "numpy_set_slice",
     "numpy_zoom",