mango-metadata-from-tables 1.0.0__tar.gz

mango_metadata_from_tables-1.0.0/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.3
+Name: mango-metadata-from-tables
+Version: 1.0.0
+Summary: Use this package to process tabular files in which each row represents an iRODS data object or collection and each column contains either an identifier or metadata to add to this item. It supports plain text files and Excel files, which could be stored locally or in iRODS itself.
+Author: Mariana Montes, Jef Scheepers
+Requires-Dist: click>=8.3.2
+Requires-Dist: jinja2>=3.1.6
+Requires-Dist: mango-mdschema==1.1.0
+Requires-Dist: odfpy>=1.4.1
+Requires-Dist: openpyxl==3.1.5
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: pygments>=2.20.0
+Requires-Dist: python-irodsclient>=3.2.0
+Requires-Dist: pyyaml==6.0.2
+Requires-Dist: rich==13.9.4
+Requires-Python: >=3.10
+Project-URL: repository, https://github.com/kuleuven/mango-metadata-from-tables
+Description-Content-Type: text/markdown
+
+# Python package to extract metadata from tables
+
+Use this package to process tabular files in which each row represents an iRODS data object or collection
+and each column contains either an identifier or metadata to add to this item.
+It supports plain text files and Excel files, which could be stored locally or in iRODS itself.
+
+To get started, create a virtual environment with pip and install the dependencies described in the [requirements file](./requirements.txt):
+
+```sh
+python -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+Now, you can run the script with the command `mango-metadata-from-tables`.  
+
+
+## Usage
+
+This module can run on the command line with two commands: `setup` and `run`.
+
+The `setup` command takes as arguments the path to a tabular file (local or in iRODS) and
+the desired path for the output YAML, asks the user questions about how to
+parse the tabular file, and outputs a configuration file.
+
+This configuration file can then be provided as the `--config` option to the `run`
+command in order to standardize tabular files and properly obtain paths to data objects
+and attach metadata to them based on the columns of these files.
+
+**Note:** Empty values in the table are ignored, and will not be added as metadata.  
+In some cases, users may find it meaningful to include the absence of a value as contextual information.  
+In those cases, we advice to use a value like "Unknown", "Not applicable" or "NA" in your table instead.   
+
+### Examples
+
+### A small csv file
+
+The following file simulates having [a small semicolon-separated file](./tests/testdata/testdata.csv)
+with absolute paths in a "dataobject" column and a few columns with metadata.
+
+First, with the `setup` command, we answer a few questions on how to parse the tabular file
+and create a "test-config.yaml" configuration file that keeps track of the answers.
+
+Then, with the `run` command, we use the information on the configuration YAML file to parse
+the tabular file and, because it's just a "dry run", we simulate adding the metadata to each
+data object or collection. Note that this `run` command could then also be used on other tabular files
+with the same properties as the original one.
+
+```sh
+mango-metadata-from-tables setup testdata/testdata.csv test-config.yaml --sep ";"
+mango-metadata-from-tables run testdata/testdata.csv --config test-config.yaml --dry-run
+```
+
+### A larger Excel file with multiple sheets
+
+In this second example the file is an [Excel file with multiple sheets](./tests/testdata/bigger-testdata.xlsx),
+including one that has no relevant metadata. Again, with the `setup` command we indicate
+how the Excel should be parsed and record the answers in a YAML configuration file.
+Then, with the `run` command we parse the Excel and simulate adding the metadata.
+
+```sh
+mango-metadata-from-tables setup testdata/bigger-testdata.xlsx bigger-test-config.yaml
+mango-metadata-from-tables run testdata/bigger-testdata.xlsx --config bigger-test-config.yaml --dry-run
+```
+
+## `setup`
+
+The configuration file can be created as follows:
+
+```sh
+mango-metadata-from-tables setup filename output_path
+```
+
+In this case `filename` is the path to a tabular file (csv, tsv, Excel...),
+stored either locally or in iRODS. If it lives in iRODS, the `--irods` flag should be used,
+so that an iRODS session is started:
+
+```sh
+mango-metadata-from-tables setup /zone/home/project/path/to/tabular output_path --irods
+```
+
+If the tabular file is a plain text file, it is possible to specify a column separator
+with the `--sep` option, which has "," as a default. If a wrong separator is provided and
+the parser finds a single column, it will warn you and give you the possibility to correct it.
+
+```sh
+mango-metadata-from-tables setup testdata/testdata.csv test-output.yml --sep ";"
+```
+
+If the file can be found and opened as a dataframe, the user will be prompted with questions
+that will later guide preprocessing of equivalent tabular files:
+
+- If there are multiple sheets in an Excel file, which one(s) should be used?
+- Which of the columns contains a unique identifier of the data objects that metadata has to be attached to?
+- If the unique identifier is not an absolute path, is it a relative path or part of filename?
+And if so, within which collection should the data objects be found?
+- Should any columns be whitelisted or blacklisted?
+
+The final YAML will be printed on the console and saved as a file locally.
+
+## `run`
+
+Given a path to a tabular file with metadata and a YAML with the settings to preprocess it,
+metadata can be added with the `run` command:
+
+```sh
+mango-metadata-from-tables  run path_to_tabular --config path/to/config.yml
+```
+
+For testing purposes, it is possible to use
+the `--dry-run` flag, which simulates the preprocessing and identification of metadata and
+prints a small report at the end.
+An iRODS session will be initiated always, so **make sure you have a valid active iRODS Session**.
+For the testdata, that is the icts zone in quality.
+
+
+```sh
+mango-metadata-from-tables  run path_to_tabular --config path/to/config.yml --dry-run
+```
+
+It is not necessary to rerun both `setup` and `run` for each tabular file:
+if you have several tabular files with the same properties, and that thus can be described by the same
+YAML configuration file, you just need to run `setup` with one of them, and then
+`run` with each of the tabular files and the same configuration file.

mango_metadata_from_tables-1.0.0/README.md

@@ -0,0 +1,124 @@
+# Python package to extract metadata from tables
+
+Use this package to process tabular files in which each row represents an iRODS data object or collection
+and each column contains either an identifier or metadata to add to this item.
+It supports plain text files and Excel files, which could be stored locally or in iRODS itself.
+
+To get started, create a virtual environment with pip and install the dependencies described in the [requirements file](./requirements.txt):
+
+```sh
+python -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+Now, you can run the script with the command `mango-metadata-from-tables`.  
+
+
+## Usage
+
+This module can run on the command line with two commands: `setup` and `run`.
+
+The `setup` command takes as arguments the path to a tabular file (local or in iRODS) and
+the desired path for the output YAML, asks the user questions about how to
+parse the tabular file, and outputs a configuration file.
+
+This configuration file can then be provided as the `--config` option to the `run`
+command in order to standardize tabular files and properly obtain paths to data objects
+and attach metadata to them based on the columns of these files.
+
+**Note:** Empty values in the table are ignored, and will not be added as metadata.  
+In some cases, users may find it meaningful to include the absence of a value as contextual information.  
+In those cases, we advice to use a value like "Unknown", "Not applicable" or "NA" in your table instead.   
+
+### Examples
+
+### A small csv file
+
+The following file simulates having [a small semicolon-separated file](./tests/testdata/testdata.csv)
+with absolute paths in a "dataobject" column and a few columns with metadata.
+
+First, with the `setup` command, we answer a few questions on how to parse the tabular file
+and create a "test-config.yaml" configuration file that keeps track of the answers.
+
+Then, with the `run` command, we use the information on the configuration YAML file to parse
+the tabular file and, because it's just a "dry run", we simulate adding the metadata to each
+data object or collection. Note that this `run` command could then also be used on other tabular files
+with the same properties as the original one.
+
+```sh
+mango-metadata-from-tables setup testdata/testdata.csv test-config.yaml --sep ";"
+mango-metadata-from-tables run testdata/testdata.csv --config test-config.yaml --dry-run
+```
+
+### A larger Excel file with multiple sheets
+
+In this second example the file is an [Excel file with multiple sheets](./tests/testdata/bigger-testdata.xlsx),
+including one that has no relevant metadata. Again, with the `setup` command we indicate
+how the Excel should be parsed and record the answers in a YAML configuration file.
+Then, with the `run` command we parse the Excel and simulate adding the metadata.
+
+```sh
+mango-metadata-from-tables setup testdata/bigger-testdata.xlsx bigger-test-config.yaml
+mango-metadata-from-tables run testdata/bigger-testdata.xlsx --config bigger-test-config.yaml --dry-run
+```
+
+## `setup`
+
+The configuration file can be created as follows:
+
+```sh
+mango-metadata-from-tables setup filename output_path
+```
+
+In this case `filename` is the path to a tabular file (csv, tsv, Excel...),
+stored either locally or in iRODS. If it lives in iRODS, the `--irods` flag should be used,
+so that an iRODS session is started:
+
+```sh
+mango-metadata-from-tables setup /zone/home/project/path/to/tabular output_path --irods
+```
+
+If the tabular file is a plain text file, it is possible to specify a column separator
+with the `--sep` option, which has "," as a default. If a wrong separator is provided and
+the parser finds a single column, it will warn you and give you the possibility to correct it.
+
+```sh
+mango-metadata-from-tables setup testdata/testdata.csv test-output.yml --sep ";"
+```
+
+If the file can be found and opened as a dataframe, the user will be prompted with questions
+that will later guide preprocessing of equivalent tabular files:
+
+- If there are multiple sheets in an Excel file, which one(s) should be used?
+- Which of the columns contains a unique identifier of the data objects that metadata has to be attached to?
+- If the unique identifier is not an absolute path, is it a relative path or part of filename?
+And if so, within which collection should the data objects be found?
+- Should any columns be whitelisted or blacklisted?
+
+The final YAML will be printed on the console and saved as a file locally.
+
+## `run`
+
+Given a path to a tabular file with metadata and a YAML with the settings to preprocess it,
+metadata can be added with the `run` command:
+
+```sh
+mango-metadata-from-tables  run path_to_tabular --config path/to/config.yml
+```
+
+For testing purposes, it is possible to use
+the `--dry-run` flag, which simulates the preprocessing and identification of metadata and
+prints a small report at the end.
+An iRODS session will be initiated always, so **make sure you have a valid active iRODS Session**.
+For the testdata, that is the icts zone in quality.
+
+
+```sh
+mango-metadata-from-tables  run path_to_tabular --config path/to/config.yml --dry-run
+```
+
+It is not necessary to rerun both `setup` and `run` for each tabular file:
+if you have several tabular files with the same properties, and that thus can be described by the same
+YAML configuration file, you just need to run `setup` with one of them, and then
+`run` with each of the tabular files and the same configuration file.

mango_metadata_from_tables-1.0.0/pyproject.toml

@@ -0,0 +1,44 @@
+[project]
+name = "mango-metadata-from-tables"
+version = "1.0.0"
+description = "Use this package to process tabular files in which each row represents an iRODS data object or collection and each column contains either an identifier or metadata to add to this item. It supports plain text files and Excel files, which could be stored locally or in iRODS itself."
+readme = "README.md"
+authors = [
+    { name = "Mariana Montes" },
+    { name = "Jef Scheepers" }  
+]
+requires-python = ">=3.10"
+dependencies = [
+    "click>=8.3.2",
+    "jinja2>=3.1.6",
+    "mango-mdschema==1.1.0",
+    "odfpy>=1.4.1",
+    "openpyxl==3.1.5",
+    "pandas>=2.2.3",
+    "pygments>=2.20.0",
+    "python-irodsclient>=3.2.0",
+    "pyyaml==6.0.2",
+    "rich==13.9.4",
+]
+
+[project.scripts]
+mango-metadata-from-tables = "mango_metadata_from_tables.cli:mdtab"
+
+[project.urls]
+repository = "https://github.com/kuleuven/mango-metadata-from-tables"
+
+[build-system]
+requires = ["uv_build>=0.9.10,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+
+[dependency-groups]
+dev = [
+    "pylint>=4.0.8",
+    "pytest>=9.0.3,<9.1.0",
+    "pytest-cases>=3.9.1",
+    "pytest-cov>=7.1.0",
+]
+

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/__init__.py

@@ -0,0 +1,19 @@
+from enum import StrEnum
+
+from rich.console import Console
+
+DATAOBJECT = "dataobject"
+EXCLUDE_NONSCHEMA_MD = "exclude_non_schema_metadata"
+EXCLUDE_INVALID_SCHEMA_MD = "exclude_invalid_schema_metadata"
+
+
+class ItemType(StrEnum):
+    DATAOBJECT = "data object"
+    COLLECTION = "collection"
+
+
+console = Console()
+
+
+def main() -> None:
+    print("Hello from mango-metadata-from-tables!")

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/__main__.py

@@ -0,0 +1,4 @@
+if __name__ == "__main__":
+    from mango_metadata_from_tables.cli import mdtab
+
+    mdtab()

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/cli.py

@@ -0,0 +1,20 @@
+import click
+from .run import run
+from .create_config import setup
+
+
+# These are the functions that would be called from the command line :)
+# (and use click)
+@click.group()
+def mdtab():
+    """Process tabular files to add iRODS metadata to data objects."""
+    pass
+
+
+mdtab.add_command(setup)
+mdtab.add_command(run)
+
+# endregion
+
+if __name__ == "__main__":
+    mdtab()

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/create_config.py

@@ -0,0 +1,149 @@
+import click
+from rich.prompt import Prompt, Confirm
+from rich.console import Group
+from rich.syntax import Syntax
+from .prompts import (
+    select_sheets,
+    classify_target_item_column,
+    filter_columns,
+    ask_multivalue_columns,
+    list_columns_with_character,
+    ask_about_schemas,
+)
+from rich.markdown import Markdown
+from .preprocessing import get_sheets
+from . import console
+
+
+@click.command()
+@click.option("--sep", default=",", help="Separator for plain text files.")
+@click.option(
+    "--irods/--no-irods", default=False, help="Whether an iRODS session is needed."
+)
+@click.argument("example")
+@click.argument("output", type=click.File("w"))
+def setup(example, output, sep=",", irods=False):
+    """
+    Generate configuration file.
+
+    EXAMPLE is the path to the tabular file to parse.
+
+    OUTPUT is the path where the configuration file will be saved.
+    """
+    import yaml
+
+    while True:
+        sheets = get_sheets(example, sep, irods)
+        if len(sheets) > 1:
+            break
+        column_names = list(sheets.values())[0].columns
+        if len(column_names) > 1:
+            break
+        update_separator = Confirm.ask(
+            f"Your sheet has only one column: `{column_names[0]}`, \
+                would you like to provide another separator?"
+        )
+        if update_separator:
+            sep = Prompt.ask("Which separator would you like to try now?") or " "
+        else:
+            break
+
+    # select which sheets to use, if there are more than one
+    selection_of_sheets = select_sheets(sheets)
+    sheets = {k: v for k, v in sheets.items() if k in selection_of_sheets}
+
+    # get info on the item column, if there is any
+    path_info = classify_target_item_column(sheets)
+    item_column = path_info["item_column"]
+
+    # if there is a dedicated item column,
+    # only keep the sheets that contain that column
+    if item_column:
+        sheets = {k: v for k, v in sheets.items() if item_column in v.columns}
+
+    # start config yaml with the info we have
+    for_yaml = {
+        "sheets": list(sheets.keys()),
+        "separator": sep,
+        "item_type": path_info["item_type"].name,
+        "path_column": {
+            "column_name": item_column,
+            "path_type": path_info["path_type"],
+            "pattern": path_info["pattern"],
+            "workdir": path_info["workdir"],
+        },
+    }
+
+    # ask if any columns need to be blacklisted OR whitelisted
+    all_column_names = list(
+        set(
+            col
+            for sheet in sheets.values()
+            for col in sheet.columns
+            if col != item_column
+        )
+    )
+    column_filter = filter_columns(all_column_names)
+
+    # calculating columns based on filter
+    if column_filter.get("whitelist", False):
+        filtered_columns = [
+            col for col in all_column_names if col in column_filter["whitelist"]
+        ]
+    elif column_filter.get("blacklist", False):
+        filtered_columns = [
+            col for col in all_column_names if col not in column_filter["blacklist"]
+        ]
+    else:
+        filtered_columns = all_column_names
+
+    # update yaml with column information
+    for_yaml.update(column_filter)
+
+    if Confirm.ask(
+        "Do your sheet(s) have columns which may contain multiple values per row?"
+    ):
+
+        valid_separator_chosen = False
+        while not valid_separator_chosen:
+            multivalue_separator = Prompt.ask(
+                "What is the separator of your columns with multiple values?"
+            )
+            if multivalue_separator == sep:
+                print(
+                    "This separator cannot be used, because it is used to separate your columns."
+                )
+                continue
+            columns_with_separator = list_columns_with_character(
+                sheets.values(), filtered_columns, multivalue_separator
+            )
+            if len(columns_with_separator) == 0:
+                print(
+                    "This separator cannot be used, because it does not appear in your file."
+                )
+
+            else:
+                valid_separator_chosen = True
+
+        # ask for multivalue columns
+        multivalue_columns = ask_multivalue_columns(
+            list(col for col in columns_with_separator if col != item_column)
+        )
+        # update yaml with multivalue columns information
+        for_yaml["multivalue_separator"] = multivalue_separator
+        for_yaml["multivalue_columns"] = multivalue_columns
+
+    # ask about schema metadata
+    if mango_schema_info := ask_about_schemas():
+        for_yaml["mango_schema"] = mango_schema_info
+
+    # create yaml from the dictionary
+    yml = yaml.dump(for_yaml, default_flow_style=False, indent=2)
+    # Make a group and indicate where it is saved
+    panel_group = Group(
+        Markdown("# This is your config yaml"),
+        Syntax(yml, "yaml"),
+        Markdown(f"_It will be saved in `{output.name}`._"),
+    )
+    console.print(panel_group)
+    click.echo(yml, file=output)

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/dataframe2avus.py

@@ -0,0 +1,119 @@
+import pandas as pd
+from irods.session import iRODSSession
+from mango_mdschema import Schema
+from irods.meta import iRODSMeta, AVUOperation
+from collections.abc import Generator
+from . import console, DATAOBJECT, ItemType
+
+
+def unlist_value(value: list, field) -> str | int:
+    """Unlist values for non repeatable fields"""
+    if len(value) == 1 and not field.repeatable:
+        return value[0]
+
+
+def dict_to_avus(
+    row: dict,
+    schema: Schema = None,
+    exclude_non_schema_metadata: bool = True,
+    exclude_invalid_schema_metadata: bool = False,
+) -> list[iRODSMeta]:
+    """Convert a dictionary of metadata name-value pairs into a list of iRODSMeta"""
+    if schema is not None:
+        dict_to_validate = {
+            k: unlist_value(v, schema.fields[k])
+            for k, v in row.items()
+            if k in schema.fields
+        }
+        valid_schema_metadata = schema.validate(
+            dict_to_validate
+        )  # dict with metadata that passed the schema
+        for k, v in valid_schema_metadata.items():
+            if v is None:
+                line1 = f"Found a value '{dict_to_validate[k]}' of column '{k}' that does not match the schema."
+                line2 = (
+                    "It will be excluded."
+                    if exclude_invalid_schema_metadata
+                    else "It will be added as non-schema metadata."
+                )
+                console.print(f"{line1} {line2}")
+        schema_avus = schema.to_avus(valid_schema_metadata)
+        if len(schema_avus) > 0:
+            schema_avus += [
+                iRODSMeta(f"{schema.prefix}.{schema.name}.__version__", schema.version)
+            ]
+
+        # create empty dict if other metadata is ignored; otherwise dict of metadata that did not pass
+        def is_invalid_schema_metadata(k):
+            return k in dict_to_validate and valid_schema_metadata.get(k, None) is None
+
+        def is_nonschema_metadata(k):
+            return k not in schema.fields
+
+        invalid_schema_metadata = (
+            {}
+            if exclude_invalid_schema_metadata
+            else {k: v for k, v in row.items() if is_invalid_schema_metadata(k)}
+        )
+        nonschema_metadata = (
+            {}
+            if exclude_non_schema_metadata
+            else {k: v for k, v in row.items() if is_nonschema_metadata(k)}
+        )
+
+        other_metadata = {**nonschema_metadata, **invalid_schema_metadata}
+    else:  # if there is no schema
+        schema_avus = []  # no schema metadata
+        other_metadata = row  # all metadata
+
+    non_schema_avus = [
+        iRODSMeta(str(key), str(value_item))
+        for key, value in other_metadata.items()  # empty if all metadata is from schema or the other metadata is ignored
+        for value_item in value
+        if not pd.isna(value_item)
+    ]
+    return schema_avus + non_schema_avus
+
+
+def generate_rows(
+    dataframe: pd.DataFrame, multivalue_columns: list, multivalue_separator: str
+) -> Generator[tuple]:
+    """Yield a tuple of filename and metadata-dictionary from a dataframe"""
+    for _, row in dataframe.iterrows():
+        md_dict = {}
+        for k, v in row.items():
+            if k != DATAOBJECT:
+                if k in multivalue_columns and isinstance(v, str):
+                    md_dict[k] = [
+                        val.strip()
+                        for val in v.split(multivalue_separator)
+                        if val.strip()
+                    ]
+                else:
+                    md_dict[k] = [v]
+        yield (row[DATAOBJECT], md_dict)
+
+
+def apply_metadata_to_data_object(
+    path: str,
+    avu_dict: dict,
+    schema_instructions: dict,
+    session: iRODSSession,
+    item_type: ItemType = ItemType.DATAOBJECT,
+):
+    """Add metadata from a dictionary to a given data object or collection"""
+    manager = (
+        session.data_objects
+        if item_type == ItemType.DATAOBJECT
+        else session.collections
+    )
+    try:
+        obj = manager.get(path)
+        avus = dict_to_avus(avu_dict, **schema_instructions)
+        obj.metadata.apply_atomic_operations(
+            *[AVUOperation(operation="add", avu=item) for item in avus]
+        )
+        return avus
+    except Exception as e:
+        print(e)
+        return 0

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/preprocessing.py

@@ -0,0 +1,263 @@
+import io
+import os
+import pandas as pd
+import click
+import jinja2
+import datetime
+import pathlib
+from irods.session import iRODSSession
+from irods.column import Criterion
+from irods.models import Collection, DataObject
+import yaml
+from mango_mdschema.schema import Schema, get_mango_schema
+from .read_table import parse_tabular_file
+from . import (
+    DATAOBJECT,
+    EXCLUDE_NONSCHEMA_MD,
+    EXCLUDE_INVALID_SCHEMA_MD,
+    console,
+    ItemType,
+)
+from typing import Callable
+
+
+def search_objects_with_identifier(
+    session: iRODSSession, workingdirectory: str, identifier: str, exact_match: bool
+):
+    """Searches a given project for data objects starting with a certain identifier
+
+
+    Arguments
+    ---------
+    session: obj
+        An iRODSSession object
+
+    workingdirectory: str
+        Path to the collection in iRODS
+
+    identifier: str
+        The identifier you want to search for
+
+    Returns
+    -------
+    paths: str
+        A list of data object paths matching the identifier
+    """
+
+    operator = "=" if exact_match else "like"
+    query = (
+        session.query(DataObject.name, Collection.name)
+        .filter(Criterion("like", Collection.name, workingdirectory + "%"))
+        .filter(Criterion(operator, DataObject.name, identifier + "%"))
+    )
+    paths = [f"{result[Collection.name]}/{result[DataObject.name]}" for result in query]
+    return paths
+
+
+def query_dataobjects_with_filename(
+    session, df, filename_column, workingdirectory, exact_match=True
+):
+    """
+    Queries data objects in iRODS based on identifiers in the dataframe,
+    and creates a row for each result with the accompanying metadata.
+    """
+
+    new_rows = []
+    for index, identifier in enumerate(df[filename_column]):
+        paths = search_objects_with_identifier(
+            session, workingdirectory, identifier, exact_match
+        )
+        for path in paths:
+            new_row = df.iloc[index].drop(filename_column)
+            new_row[DATAOBJECT] = path
+            # create a 1 row dataframe, which needs to be transposed (hence the T)
+            new_rows.append(new_row.to_frame().T)
+    if len(new_rows) > 0:
+        new_df = pd.concat(new_rows, ignore_index=True)
+    else:
+        columns = [column for column in df.columns if column != filename_column]
+        columns.append(DATAOBJECT)
+        new_df = pd.DataFrame(columns=columns)
+    return new_df
+
+
+def render_single_path_from_pattern(
+    row: pd.Series, pattern: str, env: jinja2.Environment
+) -> str:
+    """Render a path from a pattern using a single DataFrame row."""
+
+    path_template = env.from_string(pattern)
+    return path_template.render(row.to_dict())
+
+
+def create_path_based_on_pattern(
+    df: pd.DataFrame, pattern: str, env: jinja2.Environment
+):
+    """Create a column for data object paths based on info from other columns"""
+
+    path_template = env.from_string(pattern)
+    constructed_paths = [
+        path_template.render(row.to_dict()) for _, row in df.iterrows()
+    ]
+    df[DATAOBJECT] = constructed_paths
+    return df
+
+
+def chain_collection_and_filename(
+    df: pd.DataFrame, filename_column: str, workingdirectory: str
+):
+    """Renames the column with the relative data object or collection path and completes it with the parent collection path"""
+    df = df.rename(columns={filename_column: DATAOBJECT})
+    df[DATAOBJECT] = [str(pathlib.PurePosixPath(workingdirectory, x)) for x in df[DATAOBJECT]]
+    return df
+
+
+# filters for creating patterns
+
+
+def date_format(value, input_format="%d-%m-%Y", output_format="%Y-%m-%d"):
+    """
+    Return a date in the specified output format
+
+    If the value is a datetime object, it will be converted immediately.
+    If the value is a string, it will be converted to a datetime object
+    according to the input format, and then converted to the output format.
+    """
+
+    if isinstance(value, str):
+        value = datetime.datetime.strptime(value, input_format)
+    # If already a datetime object, skip conversion
+    return value.strftime(output_format)
+
+
+def create_jinja_environment_with_filters():
+    jinja_environment = jinja2.Environment()
+    jinja_environment.filters["date_format"] = date_format
+    return jinja_environment
+
+
+def validate_schema_columns(sheets: dict[pd.DataFrame], schema: Schema) -> list[str]:
+    if schema is None:
+        return list(sheets.keys())
+
+    required_fields = [
+        field_name
+        for field_name, field in schema.fields.items()
+        if field.required and not field.default
+    ]
+    sheets_for_schema = [
+        sheetname
+        for sheetname, sheet in sheets.items()
+        if all(field_name in sheet.columns for field_name in required_fields)
+    ]
+    if len(sheets_for_schema) == 0:
+        raise KeyError(
+            "None of the sheets contain all the required fields of the schema."
+        )
+    return sheets_for_schema
+
+
+def process_tabular_file(
+    filename: str | pathlib.Path,
+    config: io.StringIO | click.File,
+    session: iRODSSession | None = None,
+) -> dict:
+    """Apply the preprocessing to a file based on a configuration file"""
+    yml = yaml.safe_load(config)
+    sheets = parse_tabular_file(filename, session, yml.get("separator", None))
+    item_type = yml.get("item_type", ItemType.DATAOBJECT.name)
+    sheets_to_return = {}
+    for sheetname, sheet in sheets.items():
+        if sheetname not in yml["sheets"]:
+            continue
+        path_column_name = yml["path_column"]["column_name"]
+        if (
+            item_type == ItemType.DATAOBJECT.name
+            and yml["path_column"]["path_type"] == "part"
+        ):
+            if session is None:
+                raise ValueError("Cannot query paths with no iRODS session")
+            sheet = query_dataobjects_with_filename(
+                session,
+                sheet,
+                path_column_name,
+                yml["path_column"]["workdir"],
+                exact_match=False,
+            )
+            if sheet.empty:
+                continue
+        elif yml["path_column"]["path_type"] == "relative":
+            sheet = chain_collection_and_filename(
+                sheet, path_column_name, yml["path_column"]["workdir"]
+            )
+        elif yml["path_column"]["path_type"] == "pattern":
+            env = create_jinja_environment_with_filters()
+            sheet = create_path_based_on_pattern(
+                sheet, yml["path_column"]["pattern"], env
+            )
+        else:
+            sheet = sheet.rename(columns={path_column_name: DATAOBJECT})
+
+        if "whitelist" in yml:
+            sheet = sheet[
+                [c for c in sheet.columns if c in [DATAOBJECT] + yml["whitelist"]]
+            ]
+        elif "blacklist" in yml:
+            sheet = sheet[[c for c in sheet.columns if c not in yml["blacklist"]]]
+        sheets_to_return[sheetname] = sheet
+
+    multivalue_columns = yml.get("multivalue_columns", [])
+    multivalue_separator = yml.get("multivalue_separator", "")
+    schema_info = yml.get("mango_schema", {})
+    if "path" in schema_info:
+
+        def get_schema(path):
+            match path:
+                case {"realm": realm, "schema": schema}:
+                    return Schema(
+                        get_mango_schema(session, realm=realm, schema_name=schema)
+                    )
+                case str(path):
+                    return Schema(path) if os.path.exists(path) else None
+
+        schema = get_schema(schema_info["path"])
+    else:
+        schema = None
+    if schema:
+        schema_instructions = {
+            "schema": schema,
+            EXCLUDE_NONSCHEMA_MD: schema_info.get(EXCLUDE_NONSCHEMA_MD, True),
+            EXCLUDE_INVALID_SCHEMA_MD: schema_info.get(
+                EXCLUDE_INVALID_SCHEMA_MD, False
+            ),
+        }
+
+    else:
+        console.print("No schema found, metadata will be added as is.")
+        schema_instructions = {}
+
+    processed_config_data = {
+        "item_type": ItemType[yml.get("item_type", ItemType.DATAOBJECT)],
+        "sheets": sheets_to_return,
+        "multivalue_columns": multivalue_columns,
+        "multivalue_separator": multivalue_separator,
+        "schema_instructions": schema_instructions,
+    }
+    return processed_config_data
+
+
+# only connect to irods if requested
+def get_sheets(example: str, sep=",", irods=False):
+    """Parse a tabular file in iRODS or locally, with the right separator"""
+    if irods:
+        try:
+            env_file = os.environ["IRODS_ENVIRONMENT_FILE"]
+        except KeyError:
+            env_file = os.path.expanduser("~/.irods/irods_environment.json")
+
+        ssl_settings = {}
+        with iRODSSession(irods_env_file=env_file, **ssl_settings) as session:
+            sheets = parse_tabular_file(example, session, sep)
+    else:
+        sheets = parse_tabular_file(example, separator=sep)
+    return sheets

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/prompts.py

@@ -0,0 +1,289 @@
+import os.path
+import pandas as pd
+from . import console, ItemType, EXCLUDE_INVALID_SCHEMA_MD, EXCLUDE_NONSCHEMA_MD
+from rich.markdown import Markdown
+from rich.prompt import Prompt, Confirm
+from .preprocessing import (
+    create_jinja_environment_with_filters,
+    render_single_path_from_pattern,
+)
+from typing import Set
+
+
+def explain_multiple_choice():
+    console.print(
+        "Type one answer at a time, pressing Enter afterwards. \
+            Press Enter twice when you are done.",
+        style="italic magenta",
+    )
+
+
+def select_sheets(sheet_collection: dict) -> list:
+    """Ask user to choose which sheets to use from an Excel"""
+    selection_of_sheets = list(sheet_collection.keys())
+    if len(sheet_collection) == 1:
+        if selection_of_sheets[0] == "single_sheet":
+            console.print(
+                "You have provided a plain text file, no multiple sheets, great work!"
+            )
+        else:
+            console.print(
+                Markdown(
+                    f"The file you provided has only one sheet: `{selection_of_sheets[0]}`."
+                )
+            )
+        return selection_of_sheets[0]
+    all_sheets = Confirm.ask("Would you like to use all of the available sheets?")
+    if all_sheets:
+        return selection_of_sheets
+    explain_multiple_choice()
+    selected_sheets = []
+    while True:
+        selected_sheet = Prompt.ask(
+            "Which of the available sheets would you like to select?",
+            choices=selection_of_sheets + [""],
+        )
+        if selected_sheet:
+            selected_sheets.append(selected_sheet)
+        else:
+            break
+    return selected_sheets
+
+
+def identify_item_column(sheet_collection: dict) -> str:
+    """Ask user which column contains the unique data object or collection information"""
+    columns = set([col for sheet in sheet_collection.values() for col in sheet.columns])
+    dfs = "dataframe has" if len(sheet_collection) == 1 else "dataframes have"
+    cols = "1 column" if len(columns) == 1 else f"{len(columns)} columns"
+    column_intro = f"Your {dfs} {cols}:\n\n"
+    column_list = "\n\n".join(f"- {col}" for col in columns)
+    console.print(Markdown(column_intro + column_list))
+    return Prompt.ask(
+        "Which column contains an unique identifier for the target item?",
+        choices=columns,
+    )
+
+
+def test_pattern_on_first_column(sheet_collection: dict[pd.DataFrame], pattern: str):
+    """
+    Apply a pattern on the first row of the first dataframe of a dictionary of dataframes.
+    """
+
+    df = list(sheet_collection.values())[0]
+    row = df.iloc[0]
+    env = create_jinja_environment_with_filters()
+    try:
+        result = render_single_path_from_pattern(row, pattern, env)
+    except Exception as e:
+        message = f"The following error occured while trying to apply the pattern to a row:\n {type(e).__name__}: {repr(e.args)}"
+        print(message)
+        result = None
+    return result
+
+
+def classify_target_item_column(
+    sheet_collection: dict,
+) -> dict:
+
+    import re
+
+    item_type = Prompt.ask(
+        "Will you be annotating data objects or colletions?", choices=ItemType
+    )
+
+    message = """
+    In order to add metadata to your TARGET_ITEMs, each row needs
+    to have a reference to your TARGET_ITEM.
+
+    For your table, how can we find the TARGET_ITEM in each row?
+
+    1) A column contains the absolute path to the TARGET_ITEM
+    2) A column contains the relative path to the TARGET_ITEM
+    3) The absolute path of the data TARGET_ITEM can be reconstructed by combining 
+       info of multiple columns and strings.
+    """
+    message = message.replace("TARGET_ITEM", item_type)
+    choice_mapping = {"1": "absolute", "2": "relative", "3": "pattern"}
+
+    if item_type == ItemType.DATAOBJECT.value:
+        message += "4) A column contains part of the data object name\n"
+        choice_mapping["4"] = "part"
+
+    answer = Prompt.ask(message, choices=list(choice_mapping.keys()))
+    path_type = choice_mapping[answer]
+    workdir = ""
+    pattern = ""
+    if path_type == "pattern":
+        item_column = ""
+        pattern_question = """
+    Provide a path pattern using double curly braces ({{ }}) to reference column names.
+    Example: '/zone/home/project/{{ lab }}_{{ experiment }}.txt' will use values from the 'lab' and 'experiment' columns in each row.
+    You can also use filters to modify the values of the columns before using them in the path.
+    For more information, see the documentation in docs/construct_path_from_columns.md.\n"""
+        pattern_okay = False
+        while not pattern_okay:
+            pattern = Prompt.ask(pattern_question)
+            preview = test_pattern_on_first_column(sheet_collection, pattern)
+            if preview is None:
+                print("The pattern you provided is not valid.")
+            else:
+                pattern_ok_message = (
+                    f"Based on the pattern you provided, the first row in your file contains the following path: {preview}."
+                    "\n Does this look okay?"
+                )
+                pattern_okay = Confirm.ask(pattern_ok_message)
+
+        print(
+            f"Great! Data objects will be found by combining columns and strings in the following pattern: {pattern}"
+        )
+
+    else:
+        item_column = identify_item_column(sheet_collection)
+        if path_type in ["relative", "part"]:
+            while not re.match("/[a-z_]+/home/[^/]+/?", workdir):
+                workdir = Prompt.ask(
+                    "What is the absolute path of the collection where we can find these data objects? \
+                    (It should start with `/{zone}/home/{project}/...`)"
+                )
+        if path_type == "relative":
+            console.print(
+                Markdown(
+                    f"Great! The relative paths in `{item_column}` will be chained to `{workdir}`!"
+                )
+            )
+        elif path_type == "part":
+            console.print(
+                Markdown(
+                    f"Great! Data objects will be found by querying the contents of `{item_column}` within `{workdir}`!"
+                )
+            )
+    enum_mapping = {x.value: x for x in ItemType}
+
+    return {
+        "item_type": enum_mapping[item_type],
+        "item_column": item_column,
+        "path_type": path_type,
+        "pattern": pattern,
+        "workdir": workdir,
+    }
+
+
+def filter_columns(columns: list) -> dict:
+    """Ask user to blacklist or whitelist columns"""
+    filter_how = Prompt.ask(
+        "Would you like to whitelist or blacklist some columns?",
+        choices=["whitelist", "blacklist", "neither"],
+        default="neither",
+    )
+    if filter_how == "neither":
+        return {}
+    explain_multiple_choice()
+    # using a set to get a list without duplicates
+    filter_what = set()
+    # creating a 'choices'-list so we don't modify the original columns list
+    choices = columns.copy() + [""]
+    while any(c for c in choices):
+        ans = Prompt.ask(
+            f"Which column(s) would you like to {filter_how}?", choices=choices
+        )
+        if ans:
+            filter_what.add(ans)
+            choices.remove(ans)
+        else:
+            break
+    # convert set back to list because a set cannot
+    # be added to a yml
+    filter_what = list(filter_what)
+    if len(filter_what) == 0:
+        return {}
+    return {filter_how: filter_what}
+
+
+def ask_multivalue_columns(columns: list) -> list:
+    """Ask user whether the sheets contain any colums with multiple values"""
+
+    explain_multiple_choice()
+    # using a set to get a list without duplicates
+    multivalue_columns = set()
+    choices = columns.copy() + [""]
+    # creating a 'choices'-list so we don't modify the original columns list
+    while any(c for c in choices):
+        ans = Prompt.ask(
+            "Which column(s) can contain multiple values?",
+            choices=choices,
+        )
+        if ans:
+            multivalue_columns.add(ans)
+            choices.remove(ans)
+        else:
+            break
+
+    # convert set back to list because a set cannot
+    # be added to a yml
+    multivalue_columns = list(multivalue_columns)
+    return multivalue_columns
+
+
+def list_columns_with_character(
+    dfs: list[pd.DataFrame], eligible_columns: list, character: str
+) -> Set[str]:
+    """
+    Given a list of pandas DataFrames, return a set of column names
+    where at least one value contains the specified character.
+    """
+
+    return set(
+        col
+        for df in dfs
+        for col in df.columns
+        if col in eligible_columns
+        and df[col].dtype == object
+        and df[col].astype(str).str.contains(character, na=False, regex=False).any()
+    )  # not sure about how this gets split in rows
+
+
+def ask_about_schemas() -> dict | None:
+    if not Confirm.ask(
+        "Do you have a ManGO metadata schema to validate your metadata?"
+    ):
+        return
+    if Confirm.ask(
+        "Does the schema exist in ManGO? (We cannot verify the correctness at this stage yet)"
+    ):
+        realm = Prompt.ask(
+            "Please provide the name of the project/realm the schema belongs to"
+        )
+        schema = Prompt.ask(
+            f"Please provide the name of the published schema in the {realm} realm"
+        )
+        # @todo validate against iRODS?
+        schema_file = {"realm": realm, "schema": schema}
+    else:
+        schema_file = ""
+        while not os.path.exists(schema_file):
+            # TODO add mango-mdschema validation OF the schema file
+            schema_file = Prompt.ask("Please provide a valid path for your schema: ")
+            if not schema_file:
+                print("Changed your mind? We won't use a schema then!")
+                break
+    if not schema_file:
+        return
+    invalid_schema_metadata_question = (
+        "Should we discard invalid schema values? "
+        "(Otherwise, they will be added as non-schema metadata, "
+        "e.g. 'size=medium' instead of 'mgs.schema.size=medium')"
+    )
+    exclude_invalid_schema_metadata = Confirm.ask(
+        invalid_schema_metadata_question, default=False
+    )
+    nonschema_metadata_question = (
+        "Should we discard the columns not covered by schema? "
+        "(If you say no, they will be added as non-schema metadata):"
+    )
+
+    exclude_nonschema_metadata = Confirm.ask(nonschema_metadata_question, default=True)
+    return {
+        "path": schema_file,
+        EXCLUDE_NONSCHEMA_MD: exclude_nonschema_metadata,
+        EXCLUDE_INVALID_SCHEMA_MD: exclude_invalid_schema_metadata,
+    }

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/read_table.py

@@ -0,0 +1,65 @@
+import pandas as pd
+import pathlib
+from irods.exception import DataObjectDoesNotExist, CollectionDoesNotExist
+from irods.data_object import iRODSDataObject
+
+
+def create_file_object(path: str | pathlib.Path, session=None):
+    """Turn path to file into a file-like object.
+
+    Args:
+        session: (iRODSSession or None): session to connect to iRODS.
+            Session can be set to None for testing non-irods functionalities.
+        path (str): Path to tabular file
+
+    Raises:
+        FileNotFoundError: If the file cannot be found locally or in ManGO.
+
+    Returns:
+        pathlib.Path or irods.iRODSDataObject: File-like object to read metadata from.
+    """
+    ppath = pathlib.Path(path)
+    if ppath.suffix not in [".xlsx", ".csv", ".tsv"]:
+        raise IOError("Filetype not accepted")
+    if ppath.exists():
+        return ppath
+    if session:
+        try:
+            return session.data_objects.get(path)
+        except DataObjectDoesNotExist or CollectionDoesNotExist as e:
+            raise e
+    raise FileNotFoundError
+
+
+def parse_tabular_file(path: str | pathlib.Path, session=None, separator: str = ","):
+    """Parse tabular file.
+
+    Args:
+        path (str): Path to the tabular file.
+        session (iRODSSession or None): session to connect to iRODS.
+            If it is none, it is assumed that we are testing
+        separator (str, optional): Separator for plain text files.. Defaults to ",".
+
+    Raises:
+        IOError: If the file cannot be parsed (it is not .xlsx, .csv or .tsv) it won't be read.
+
+    Returns:
+        dict: Dictionary of pandas.DataFrames with sheet names as keys.
+    """
+
+    file = create_file_object(path, session)
+    if file.name.endswith("xlsx"):
+        # Local excel files are binary and should be opened with 'rb'.
+        # However, iRODS implemented their 'open' method differently,
+        # and there you should use just 'r' instead
+        reading_mode = "r" if isinstance(file, iRODSDataObject) else "rb"
+        with file.open(reading_mode) as f:
+            sheets = pd.read_excel(f, sheet_name=None)
+        if any(x.strip() != x for x in sheets.keys()):
+            sheets = {k.strip(): v for k, v in sheets.items()}
+    else:
+        # these types are not binary and should be opened with 'r'
+       sheets = {"single_sheet": pd.read_csv(str(path), sep=separator)}
+    for sheet in sheets.values():
+        sheet.columns = sheet.columns.str.strip()
+    return sheets

mango_metadata_from_tables-1.0.0/src/mango_metadata_from_tables/run.py

@@ -0,0 +1,113 @@
+import io
+import os.path
+import pathlib
+from irods.session import iRODSSession
+import click
+from rich.markdown import Markdown
+from rich.progress import track
+from .dataframe2avus import dict_to_avus, generate_rows, apply_metadata_to_data_object
+from .preprocessing import process_tabular_file, validate_schema_columns
+from . import console
+
+# region Chains
+
+
+@click.command()
+@click.option(
+    "--config",
+    type=click.File("r"),
+    required=True,
+    help="Configuration file created by `setup`.",
+)
+@click.option("--dry-run", is_flag=True, help="Simulate applying the metadata.")
+@click.argument("filename")
+def run(filename, config, dry_run=False):
+    """Apply metadata from a tabular file to data objects.
+
+    FILENAME is the path to the tabular file containing the metadata.
+    It should have some column with a unique identifier for the data objects,
+    and columns for other metadata fields.
+    """
+
+    tabular_file = pathlib.Path(filename)
+    if not tabular_file.exists():
+        raise FileExistsError
+    try:
+        env_file = os.environ["IRODS_ENVIRONMENT_FILE"]
+    except KeyError:
+        env_file = os.path.expanduser("~/.irods/irods_environment.json")
+
+    ssl_settings = {}
+    with iRODSSession(irods_env_file=env_file, **ssl_settings) as session:
+        apply_metadata_from_table(tabular_file, config, dry_run, session)
+
+
+def apply_metadata_from_table(
+    filename: str | pathlib.Path,
+    config: io.StringIO | click.File,
+    dry_run: bool = False,
+    session: iRODSSession | None = None,
+):
+    if session is None:
+        console.print("No session found, we'll try a dry-run!")
+    processed_config_data = process_tabular_file(filename, config, session)
+
+    sheets = processed_config_data["sheets"]
+    item_type = processed_config_data["item_type"]
+    multivalue_columns = processed_config_data["multivalue_columns"]
+    multivalue_separator = processed_config_data["multivalue_separator"]
+    schema_instructions = processed_config_data["schema_instructions"]
+    sheets_for_schemas = validate_schema_columns(
+        sheets, schema_instructions.get("schema", None)
+    )
+    for sheetname, sheet in sheets.items():
+        sheet_schema_instructions = (
+            schema_instructions if sheetname in sheets_for_schemas else {}
+        )
+        progress_message = f"Adding metadata from {sheetname + ' in ' if len(sheets) > 1 else ''}`{filename}`..."
+        n = 0
+        errors = 0
+        min_avus = None
+        max_avus = None
+
+        # loop over each row printing a progress bar
+        for item, md_dict in track(
+            generate_rows(sheet, multivalue_columns, multivalue_separator),
+            description=progress_message,
+        ):
+            if session and not dry_run:
+                avus = apply_metadata_to_data_object(
+                    item, md_dict, sheet_schema_instructions, session, item_type
+                )
+                yield {"item": item, "avus": avus}
+            else:
+                console.print(
+                    f"Creating the following AVUs for {item_type.value} {item}:"
+                )
+                avus = dict_to_avus(md_dict, **sheet_schema_instructions)
+                print(avus)
+                yield {"item": item, "avus": avus}
+            if avus:
+                n += 1
+                if max_avus is None or len(avus) > max_avus:
+                    max_avus = len(avus)
+                if min_avus is None or len(avus) < min_avus:
+                    min_avus = len(avus)
+            else:
+                errors += 1
+
+        avu_length_range = (
+            max_avus if min_avus == max_avus else f"{min_avus} to {max_avus}"
+        )
+        console.print(
+            Markdown(
+                # This calculation may not be correct anymore in case of multiple values,
+                # since the md_dict of each object can now have a different length
+                f"{'Simulated' if dry_run else 'Applied'} {avu_length_range} AVUs for each of {n} {item_type.value}s"
+            )
+        )
+        if errors > 0:
+            console.print(
+                f"{errors} data objects were skipped because the paths were not valid!",
+                style="red bold",
+            )