ngio 0.3.0a0__tar.gz → 0.3.1__tar.gz

ngio-0.3.1/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## [Unreleased]
+
+### Added
+
+- Initial version of the changelog.

{ngio-0.3.0a0 → ngio-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ngio
-Version: 0.3.0a0
+Version: 0.3.1
 Summary: Next Generation file format IO
 Project-URL: homepage, https://github.com/fractal-analytics-platform/ngio
 Project-URL: repository, https://github.com/fractal-analytics-platform/ngio
@@ -50,6 +50,7 @@ Requires-Dist: mkdocs; extra == 'docs'
 Requires-Dist: mkdocs-autorefs; extra == 'docs'
 Requires-Dist: mkdocs-git-committers-plugin-2; extra == 'docs'
 Requires-Dist: mkdocs-git-revision-date-localized-plugin; extra == 'docs'
+Requires-Dist: mkdocs-include-markdown-plugin; extra == 'docs'
 Requires-Dist: mkdocs-jupyter; extra == 'docs'
 Requires-Dist: mkdocs-material; extra == 'docs'
 Requires-Dist: mkdocstrings[python]; extra == 'docs'
@@ -62,7 +63,7 @@ Requires-Dist: pytest-cov; extra == 'test'
 Requires-Dist: scikit-image; extra == 'test'
 Description-Content-Type: text/markdown
 
-# NGIO - Next Generation file format IO
+# Ngio - Next Generation file format IO
 
 [![License](https://img.shields.io/pypi/l/ngio.svg?color=green)](https://github.com/lorenzocerrone/ngio/raw/main/LICENSE)
 [![PyPI](https://img.shields.io/pypi/v/ngio.svg?color=green)](https://pypi.org/project/ngio)
@@ -70,36 +71,69 @@ Description-Content-Type: text/markdown
 [![CI](https://github.com/fractal-analytics-platform/ngio/actions/workflows/ci.yml/badge.svg)](https://github.com/fractal-analytics-platform/ngio/actions/workflows/ci.yml)
 [![codecov](https://codecov.io/gh/fractal-analytics-platform/ngio/graph/badge.svg?token=FkmF26FZki)](https://codecov.io/gh/fractal-analytics-platform/ngio)
 
-NGIO is a Python library to streamline OME-Zarr image analysis workflows.
-
-**Main Goals:**
-
-- Abstract object base API for handling OME-Zarr files
-- Powerful iterators for processing data using common access patterns
-- Tight integration with [Fractal's Table Fractal](https://fractal-analytics-platform.github.io/fractal-tasks-core/tables/)
-- Validation of OME-Zarr files
-
-To get started, check out the [Getting Started](https://fractal-analytics-platform.github.io/ngio/getting-started/) guide. Or checkout our [Documentation](https://fractal-analytics-platform.github.io/ngio/)
-
-## 🚧 Ngio is Under active Development 🚧
-
-### Roadmap
-
-| Feature | Status | ETA | Description |
-|---------|--------|-----|-------------|
-| Metadata Handling | ✅ | | Read, Write, Validate OME-Zarr Metadata (0.4 supported, 0.5 ready) |
-| OME-Zarr Validation | ✅ | | Validate OME-Zarr files for compliance with the OME-Zarr Specification + Compliance between Metadata and Data |
-| Base Image Handling | ✅ | | Load data from OME-Zarr files, retrieve basic metadata, and write data |
-| ROI Handling | ✅ | | Common ROI models |
-| Label Handling | ✅ | Mid-September | Based on Image Handling |
-| Table Validation | ✅ | Mid-September | Validate Table fractal V1 + Compliance between Metadata and Data |
-| Table Handling | ✅ | Mid-September | Read, Write ROI, Features, and Masked Tables |
-| Basic Iterators | Ongoing | End-September | Read and Write Iterators for common access patterns |
-| Base Documentation | ✅ | End-September | API Documentation and Examples |
-| Beta Ready Testing | ✅ | End-September | Beta Testing; Library is ready for testing, but the API is not stable |
-| Streaming from Fractal | Ongoing | December | Ngio can stream OME-Zarr from fractal |
-| Mask Iterators | Ongoing | Early 2025 | Iterators over Masked Tables |
-| Advanced Iterators | Not started | mid-2025 | Iterators for advanced access patterns |
-| Parallel Iterators | Not started | mid-2025 | Concurrent Iterators for parallel read and write |
-| Full Documentation | Not started | 2025 | Complete Documentation |
-| Release 1.0 (Commitment to API) | Not started | 2025 | API is stable; breaking changes will be avoided |
+ngio is a Python library designed to simplify bioimage analysis workflows, offering an intuitive interface for working with OME-Zarr files.
+
+## What is Ngio?
+
+Ngio is built for the [OME-Zarr](https://ngff.openmicroscopy.org/) file format, a modern, cloud-optimized format for biological imaging data. OME-Zarr stores large, multi-dimensional microscopy images and metadata in an efficient and scalable way.
+
+Ngio's mission is to streamline working with OME-Zarr files by providing a simple, object-based API for opening, exploring, and manipulating OME-Zarr images and high-content screening (HCS) plates. It also offers comprehensive support for labels, tables and regions of interest (ROIs), making it easy to extract and analyze specific regions in your data.
+
+## Key Features
+
+### 📊 Simple Object-Based API
+
+- Easily open, explore, and manipulate OME-Zarr images and HCS plates
+- Create and derive new images and labels with minimal boilerplate code
+
+### 🔍 Rich Tables and Regions of Interest (ROI) Support
+
+- Extract and analyze specific regions of interest
+- Tight integration with [Tabular Data](https://fractal-analytics-platform.github.io/ngio/stable/table_specs/overview/)
+
+### 🔄 Scalable Data Processing (Coming Soon)
+
+- Powerful iterators for processing data at scale
+- Efficient memory management for large datasets
+
+## Installation
+
+You can install ngio via pip:
+
+```bash
+pip install ngio
+```
+
+To get started check out the [Quickstart Guide](https://fractal-analytics-platform.github.io/ngio/stable/getting_started/0_quickstart/).
+
+## Supported OME-Zarr versions
+
+Currently, ngio only supports OME-Zarr v0.4. Support for version 0.5 and higher is planned for future releases.
+
+## Development Status
+
+!!! warning
+    Ngio is under active development and is not yet stable. The API is subject to change, and bugs and breaking changes are expected.
+    We follow [Semantic Versioning](https://semver.org/). Which means for 0.x releases potentially breaking changes can be introduced in minor releases.
+
+### Available Features
+
+- ✅ OME-Zarr metadata handling and validation
+- ✅ Image and label access across pyramid levels
+- ✅ ROI and table support
+- ✅ Streaming from remote sources
+- ✅ Documentation and examples
+
+### Upcoming Features
+
+- Advanced image processing iterators
+- Parallel processing capabilities
+- Support for OME-Zarr v0.5 and Zarr v3
+
+## Contributors
+
+Ngio is developed at the [BioVisionCenter](https://www.biovisioncenter.uzh.ch/en.html), University of Zurich, by [@lorenzocerrone](https://github.com/lorenzocerrone) and [@jluethi](https://github.com/jluethi).
+
+## License
+
+Ngio is released under the BSD-3-Clause License. See [LICENSE](https://github.com/fractal-analytics-platform/ngio/blob/main/LICENSE) for details.

ngio-0.3.1/README.md

@@ -0,0 +1,74 @@
+# Ngio - Next Generation file format IO
+
+[![License](https://img.shields.io/pypi/l/ngio.svg?color=green)](https://github.com/lorenzocerrone/ngio/raw/main/LICENSE)
+[![PyPI](https://img.shields.io/pypi/v/ngio.svg?color=green)](https://pypi.org/project/ngio)
+[![Python Version](https://img.shields.io/pypi/pyversions/ngio.svg?color=green)](https://python.org)
+[![CI](https://github.com/fractal-analytics-platform/ngio/actions/workflows/ci.yml/badge.svg)](https://github.com/fractal-analytics-platform/ngio/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/fractal-analytics-platform/ngio/graph/badge.svg?token=FkmF26FZki)](https://codecov.io/gh/fractal-analytics-platform/ngio)
+
+ngio is a Python library designed to simplify bioimage analysis workflows, offering an intuitive interface for working with OME-Zarr files.
+
+## What is Ngio?
+
+Ngio is built for the [OME-Zarr](https://ngff.openmicroscopy.org/) file format, a modern, cloud-optimized format for biological imaging data. OME-Zarr stores large, multi-dimensional microscopy images and metadata in an efficient and scalable way.
+
+Ngio's mission is to streamline working with OME-Zarr files by providing a simple, object-based API for opening, exploring, and manipulating OME-Zarr images and high-content screening (HCS) plates. It also offers comprehensive support for labels, tables and regions of interest (ROIs), making it easy to extract and analyze specific regions in your data.
+
+## Key Features
+
+### 📊 Simple Object-Based API
+
+- Easily open, explore, and manipulate OME-Zarr images and HCS plates
+- Create and derive new images and labels with minimal boilerplate code
+
+### 🔍 Rich Tables and Regions of Interest (ROI) Support
+
+- Extract and analyze specific regions of interest
+- Tight integration with [Tabular Data](https://fractal-analytics-platform.github.io/ngio/stable/table_specs/overview/)
+
+### 🔄 Scalable Data Processing (Coming Soon)
+
+- Powerful iterators for processing data at scale
+- Efficient memory management for large datasets
+
+## Installation
+
+You can install ngio via pip:
+
+```bash
+pip install ngio
+```
+
+To get started check out the [Quickstart Guide](https://fractal-analytics-platform.github.io/ngio/stable/getting_started/0_quickstart/).
+
+## Supported OME-Zarr versions
+
+Currently, ngio only supports OME-Zarr v0.4. Support for version 0.5 and higher is planned for future releases.
+
+## Development Status
+
+!!! warning
+    Ngio is under active development and is not yet stable. The API is subject to change, and bugs and breaking changes are expected.
+    We follow [Semantic Versioning](https://semver.org/). Which means for 0.x releases potentially breaking changes can be introduced in minor releases.
+
+### Available Features
+
+- ✅ OME-Zarr metadata handling and validation
+- ✅ Image and label access across pyramid levels
+- ✅ ROI and table support
+- ✅ Streaming from remote sources
+- ✅ Documentation and examples
+
+### Upcoming Features
+
+- Advanced image processing iterators
+- Parallel processing capabilities
+- Support for OME-Zarr v0.5 and Zarr v3
+
+## Contributors
+
+Ngio is developed at the [BioVisionCenter](https://www.biovisioncenter.uzh.ch/en.html), University of Zurich, by [@lorenzocerrone](https://github.com/lorenzocerrone) and [@jluethi](https://github.com/jluethi).
+
+## License
+
+Ngio is released under the BSD-3-Clause License. See [LICENSE](https://github.com/fractal-analytics-platform/ngio/blob/main/LICENSE) for details.

ngio-0.3.1/docs/changelog.md

@@ -0,0 +1,5 @@
+{%
+    include-markdown "../CHANGELOG.md"
+    start="<!--intro-start-->"
+    end="<!--intro-end-->"
+%}

{ngio-0.3.0a0 → ngio-0.3.1}/docs/code_of_conduct.md

@@ -1,2 +1,4 @@
+# Code of Conduct
+
 !!! warning
     The library is still in the early stages of development, the code of conduct is not yet established.

{ngio-0.3.0a0 → ngio-0.3.1}/docs/getting_started/1_ome_zarr_containers.md

@@ -89,7 +89,7 @@ Examples of the OME-Zarr metadata access:
 
 To access images, labels, and tables, you can use the `get_image`, `get_label`, and `get_table` methods of the `OME-Zarr Container` object.
 
-A variety of examples and additional information can be found in the [Images and Labels](./2_images.md), and [Tables](../3_tables.md) sections.
+A variety of examples and additional information can be found in the [Images and Labels](./2_images.md), and [Tables](./3_tables.md) sections.
 
 ## Creating derived images
 

{ngio-0.3.0a0 → ngio-0.3.1}/docs/getting_started/3_tables.md

@@ -190,7 +190,7 @@ Tables (differently from Images and Labels) can be purely in memory objects, and
     >>> from ngio.tables import FeatureTable
     >>> import pandas as pd
     >>> example_data = pd.DataFrame({"label": [1, 2, 3], "area": [100, 200, 300]})
-    >>> feature_table = FeatureTable(dataframe=example_data)
+    >>> feature_table = FeatureTable(table_data=example_data)
     >>> feature_table
     >>> print(feature_table) # markdown-exec: hide
     ```
@@ -203,7 +203,7 @@ Tables (differently from Images and Labels) can be purely in memory objects, and
     >>> from ngio.tables import GenericTable
     >>> import pandas as pd
     >>> example_data = pd.DataFrame({"area": [100, 200, 300], "perimeter": [50, 60, 70]})
-    >>> generic_table = GenericTable(dataframe=example_data)
+    >>> generic_table = GenericTable(table_data=example_data)
     >>> generic_table
     >>> print(generic_table) # markdown-exec: hide
     ```
@@ -212,7 +212,7 @@ Tables (differently from Images and Labels) can be purely in memory objects, and
     >>> from ngio.tables import GenericTable
     >>> import anndata as ad
     >>> adata = ad.AnnData(X=np.random.rand(10, 5), obs=pd.DataFrame({"cell_type": ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]}))
-    >>> generic_table = GenericTable(anndata=adata)
+    >>> generic_table = GenericTable(table_data=adata)
     >>> generic_table
     >>> print(generic_table) # markdown-exec: hide
     ```

{ngio-0.3.0a0 → ngio-0.3.1}/docs/index.md

@@ -1,4 +1,3 @@
-
 ngio is a Python library designed to simplify bioimage analysis workflows, offering an intuitive interface for working with OME-Zarr files.
 
 ## What is Ngio?
@@ -30,12 +29,14 @@ Refer to the [Getting Started](getting_started/0_quickstart.md) guide to integra
 For more advanced usage and API documentation, see our [API Reference](api/ngio.md).
 
 ## Supported OME-Zarr versions
+
 Currently, ngio only supports OME-Zarr v0.4. Support for version 0.5 and higher is planned for future releases.
 
 ## Development Status
 
 !!! warning
     Ngio is under active development and is not yet stable. The API is subject to change, and bugs and breaking changes are expected.
+    We follow [Semantic Versioning](https://semver.org/). Which means for 0.x releases potentially breaking changes can be introduced in minor releases.
 
 ### Available Features
 

ngio-0.3.1/docs/table_specs/backend.md

@@ -0,0 +1,113 @@
+# Table Backends
+
+In ngio we implemented four different table backends. Each table backend is a python class that can serialize tabular data into OME-Zarr containers.
+
+These backends are wrappers around existing tooling implemented in `anndata`, `pandas`, and `polars`.
+Currently, we provide a thin layer of metadata and table normalization to ensure that tables are serialized/deserialized in a consistent way across the different backends and across different table objects.
+
+In particular, we provide the metadata that describes the intended index key and type of the table for each backend.
+
+## AnnData Backend
+
+AnnData is a widely used format in single-cell genomics, and can natively store complex tabular data in a Zarr group. The AnnData backend in ngio is a wrapper around the `anndata` library, which performs some table normalization for consistency and compatibility with the ngio table specifications.
+
+The following normalization steps are applied to each table before saving it to the AnnData backend:
+
+- We separate the table in two parts: The floating point columns are casted to `float32` and stored as `X` in the AnnData object, while the categorical, boolean, and integer columns are stored as `obs`.
+- The index column is cast to a string, and is stored in the `obs` index.
+- The index column name must match the `index_key` specified in the metadata.
+
+AnnData backend metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "str", // Either "int" or "str"
+}
+```
+
+Additionally, the AnnData package will write some additional metadata to the group attributes
+
+```json
+{
+    "encoding-type": "anndata",
+    "encoding-version": "0.1.0",
+}
+```
+
+## Parquet Backend
+
+The Parquet backend is a high-performance columnar storage format that is widely used in big data processing. It is designed to efficiently store large datasets and can be used with various data processing frameworks.
+Another advantage of the Parquet backend is that it can be used lazily, meaning that the data is not loaded into memory until it is needed. This can be useful for working with large datasets that do not fit into memory.
+
+Parquet backend metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "parquet", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+The Zarr group directory will contain the Parquet file, and the metadata will be stored in the group attributes.
+
+```bash
+table.zarr          # Zarr group for the table
+├── table.parquet   # Parquet file containing the table data
+├── .zattrs         # Zarr group attributes containing the metadata
+└── .zgroup         # Zarr group metadata
+```
+
+## CSV Backend
+
+The CSV backend is a plain text format that is widely used for tabular data. It is easy to read and write, and can be used across many different tools.
+
+The CSV backen in ngio follows closely the same specifications as the Parquet backend, with the following metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "csv", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+The Zarr group directory will contain the CSV file, and the metadata will be stored in the group attributes.
+
+```bash
+table.zarr         # Zarr group for the table
+├── table.csv      # CSV file containing the table data
+├── .zattrs        # Zarr group attributes containing the metadata
+└── .zgroup        # Zarr group metadata
+```
+
+## JSON Backend
+
+The JSON backend serializes the table data into the Zarr group attributes as a JSON object. This backend is useful for tiny tables.
+
+JSON backend metadata:
+
+```json
+{
+    // Backend metadata
+    "backend": "json", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the table, which is used to identify each row.
+    "index_type": "int" // Either "int" or "str"
+}
+```
+
+The table will be stored in a subgroup of the Zarr group, and the metadata will be stored in the group attributes. Storing the table in a subgroup instead of a standalone json file allows for easier access via the Zarr API.
+
+```bash
+table.zarr          # Zarr group for the table
+└── table           # Zarr subgroup containing the table data
+    ├── .zattrs     # the json table data serialized as a JSON object
+    └── .zgroup     # Zarr group metadata
+├── .zattrs         # Zarr group attributes containing the metadata
+└── .zgroup         # Zarr group metadata
+```

ngio-0.3.1/docs/table_specs/overview.md

@@ -0,0 +1,92 @@
+# Tables Overview
+
+Ngio's architecture is designed to tightly integrate image and tabular data. For this purpose we developed custom specifications for serializing and deserializing tabular data into OME-Zarr containers, and semantically typed tables derived from the [fractal table specification](https://fractal-analytics-platform.github.io/fractal-tasks-core/tables/).
+
+## Architecture
+
+The ngio tables architectures is composed of three main components:
+
+### 1. Table Backends
+
+A backend module is a class that can serialize tabular data into OME-Zarr containers. We currently support four on-disk file formats:
+
+- **AnnData**: Commonly used in single-cell genomics and was the standard table for the initial Fractal table spec.
+- **Parquet**: A columnar storage file format optimized for large datasets.
+- **CSV**: A simple text format for tabular data, easily human readable and writable.
+- **JSON**: A lightweight data interchange format that both readable and efficient for small tables.
+
+A more detailed description of the backend module can be found in the [Table Backends documentation](backend.md).
+
+### 2. In-Memory Table Objects
+
+These are Python objects that represent the tabular data in memory. They provide a convenient interface for manipulating and analyzing the data without needing to interact directly with the underlying file format. We support the following in-memory table objects:
+
+- **Pandas DataFrame**: The most commonly used data structure for tabular data in Python.
+- **Polars LazyFrame**: A fast DataFrame implementation that allows for lazy evaluation and efficient computation on large datasets.
+- **AnnData**: A specialized data structure for single-cell genomics data, which goes beyond simple tabular data.
+
+We also provide utilities to convert between these in-memory representations in a standardized way based on the table type specifications/metadata.
+
+### 3. Table Type Specifications
+
+These specifications define structured tables that standardize common table types used in image analysis. We have defined five table types so far:
+
+- **Generic Tables**: A flexible table type that can represent any tabular data. See more in the [Generic Tables documentation](table_types/generic_table.md).
+- **ROI Tables**: A table type specifically designed for representing Regions of Interest (ROIs) in images. See more in the [ROI Tables documentation](table_types/roi_table.md).
+- **Masking ROI Tables**: A specialized table type for representing ROIs that are associated with specific labels in a OME-Zarr label image. See more in the [Masking ROI Tables documentation](table_types/masking_roi_table.md).
+- **Feature Tables**: A table type for representing features extracted from images. This table is also associated with a specific label image. See more in the [Feature Tables documentation](table_types/feature_table.md).
+- **Condition Tables**: A table to represent experimental conditions or metadata associated with images or experiments. See more in the [Condition Tables documentation](table_types/condition_table.md).
+
+## Tables Groups
+
+Tables in OME-Zarr images are organized into groups of tables. Each group is saved in a Zarr group, and can be associated with a specific image or plate. The tables groups are:
+
+- **Image Tables**: These tables are a sub group of the OME-Zarr image group and contain metadata or features related only to that specific image. The `.zarr` hierarchy is based on image [specification in NGFF 0.4](https://ngff.openmicroscopy.org/0.4/index.html#image-layout). The subgroup structure is based on the approach of the OME-Zarr `labels` group.
+
+```bash
+image.zarr        # Zarr group for a OME-Zarr image
+|
+├── 0             # Zarr array for multiscale level 0
+├── ...
+├── N             # Zarr array for multiscale level N
+|
+├── labels        # Zarr subgroup with a list of labels associated to this image
+|   ├── label_A   # Zarr subgroup for a given label
+|   ├── label_B   # Zarr subgroup for a given label
+|   └── ...
+|
+└── tables        # Zarr subgroup with a list of tables associated to this image
+    ├── table_1   # Zarr subgroup for a given table
+    ├── table_2   # Zarr subgroup for a given table
+    └── ...
+```
+
+- **Plate Tables**: These tables are a sub group of the OME-Zarr plate group and contain metadata or features related only to that specific plate.
+  
+```bash
+plate.zarr       # Zarr group for a OME-Zarr HCS plate
+|
+├── A             # Row A of the plate
+|   ├── 1         # Column 0 of row A
+|   |   ├── 0     # Acquisition 0 of column A1
+|   |   ├── 1     # Acquisition 1 of column A1
+|   |   └── ...   # Other acquisitions of column A1
+...
+├── tables        # Zarr subgroup with a list of tables associated to this plate
+|   ├── table_1   # Zarr subgroup for a given table
+|   ├── table_2   # Zarr subgroup for a given table
+|   └── ...
+└── ...
+```
+
+If a plate table contains per image information, the table should contain a `row`, `column`, and `path_in_well` columns.
+
+## Tables Group Attributes
+
+The Zarr attributes of the tables group must include the key tables, pointing to the list of all tables (this simplifies discovery of tables associated to the current OME-Zarr image or plate), as in
+
+```json
+{
+    "tables": ["table_1", "table_2"]
+}
+```

ngio-0.3.1/docs/table_specs/table_types/condition_table.md

@@ -0,0 +1,28 @@
+# Condition Table
+
+A condition table is a simple table that can be used to represent experimental conditions or metadata associated with images or experiments. It is a flexible table type that can be used to store any kind of metadata related to the images or experiments.
+
+Example condition table:
+
+| Cell Type | Drug     | Dose |
+|-----------|-----------|------|
+| A         | Drug A   | 10   |
+| A         | Drug B   | 20   |
+
+## Specifications
+
+### V1
+
+A condition table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // Condition table metadata
+    "type": "condition_table",
+    "table_version": "1",
+    // Backend metadata
+    "backend": "csv", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the condition table, which is used to identify each row.
+    "index_type": "int" // Either "int" or "str"
+}
+```

ngio-0.3.1/docs/table_specs/table_types/custom_table.md

@@ -0,0 +1,6 @@
+# Add a Custom Table
+
+Ngio allows users to define custom tables that can be used to store any kind of tabular data. Custom tables are flexible and can be used to represent any kind of data that does not fit into the predefined table types.
+
+!!! warning
+    The library is still in the early stages and full documentation for custom tables is not yet available.

ngio-0.3.1/docs/table_specs/table_types/feature_table.md

@@ -0,0 +1,54 @@
+# Feature Tables
+
+A feature table is a table type for representing per object features in an image. Each row in a feature table corresponds to a specific label in the label image.
+
+Feature tables can optionally include metadata to specify the type of features stored in each column:
+
+- `measurement`: A quantitative measurement of the object, such as area, perimeter, or intensity.
+- `categorical`: A categorical feature of the object, such as a classification label or a type.
+- `metadata`: Additional free-from columns that can be used to store any other information about the object, but that should not be used for analysis/classification purposes.
+
+These feature types inform casting of the values when serialising a table and can be used in downstream analysis to select specific subsets of features. The feature type can be explicitly specified in the feature table metadata. Alternatively, if a column is not specified, we apply the following casting rules:
+
+- If the column contains only numeric values, it is considered a `measurement`.
+- If the column contains string or boolean values, it is considered a `categorical`.
+- The index column is considered a `categorical` feature.
+
+## Specifications
+
+### V1
+
+A feature table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // Feature table metadata
+    "type": "feature_table",
+    "table_version": "1",
+    "region": {"path": "../labels/label_DAPI"}, // Path to the label image associated with this feature table
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "label", 
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+Additionally, it can include feature type information such as:
+
+```json
+{
+    "categorical_columns": [
+        "label",
+        "cell_type",
+    ],
+    "measurement_columns": [
+        "area",
+        "perimeter",
+        "intensity_mean",
+        "intensity_std"
+    ],
+    "metadata_columns": [
+        "description",
+    ],
+}
+```

ngio-0.3.1/docs/table_specs/table_types/generic_table.md

@@ -0,0 +1,23 @@
+# Generic Tables
+
+A generic table is a flexible table type that can represent any tabular data. It is not tied to any specific domain or use case, making it suitable for a wide range of custom applications.
+
+Generic tables can used as a safe fallback when trying to read a table that does not match any other specific table type.
+
+## Specifications
+
+### V1
+
+A generic table should include the following metadata fields in the group attributes:
+
+```json
+{
+    // Generic table metadata
+    "type": "generic_table",
+    "table_version": "1",
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "index", // The default index key for the generic table, which is used to identify each row.
+    "index_type": "int" // Either "int" or "str"
+}
+```

ngio-0.3.1/docs/table_specs/table_types/masking_roi_table.md

@@ -0,0 +1,35 @@
+# Masking ROI Tables
+
+A masking ROI table is a specialized table type for representing Regions of Interest (ROIs) that are associated with specific labels in a label image.
+Each row in a masking ROI table corresponds to a specific label in the label image.
+
+Masking ROI tables can be used for several purposes, such as:
+
+- Feature extraction from specific regions in the image.
+- Masking specific regions in the image for further processing. For example a masking ROI table could store the ROIs for specific tissues, and for each of these ROIs we would like to perform cell segmentation.
+
+## Specifications
+
+### V1
+
+A ROI table must include the following metadata fields in the group attributes:
+
+```json
+{
+    // ROI table metadata
+    "type": "masking_roi_table",
+    "table_version": "1",
+    "region": {"path": "../labels/label_DAPI"}, // Path to the label image associated with this masking ROI table
+    // Backend metadata
+    "backend": "annadata", // the backend used to store the table, e.g. "annadata", "parquet", etc..
+    "index_key": "label", // The default index key for the ROI table, which is used to identify each ROI. 
+    "index_type": "int", // Either "int" or "str"
+}
+```
+
+Moreover the ROI table must include the following columns:
+
+- `x_micrometer`, `y_micrometer`, `z_micrometer`: the top-left corner coordinates of the ROI in micrometers.
+- `len_x_micrometer`, `len_y_micrometer`, `len_z_micrometer`: the size of the ROI in micrometers along each axis.
+
+Additionally, each ROI can include the following optional columns: see [ROI Table](./roi_table.md).