niagads-metadata-validator 0.2.0__tar.gz

niagads_metadata_validator-0.2.0/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.3
+Name: niagads-metadata-validator
+Version: 0.2.0
+Summary: JSON Schema based validation of dataset metadata developed to support submissions to the NIAGADS Data Sharing Service
+License: GNU GPLv3
+Author: fossilfriend
+Author-email: egreenfest@gmail.com
+Requires-Python: >=3.11,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: jsonschema (>=4.23.0,<5.0.0)
+Requires-Dist: openpyxl (>=3.1.5,<4.0.0)
+Requires-Dist: pandas (>=2.2.3,<3.0.0)
+Requires-Dist: python-dateutil (>=2.9.0.post0,<3.0.0)
+Requires-Dist: strenum (>=0.4.15,<0.5.0)
+Project-URL: Bug Reports, https://github.com/NIAGADS/niagads-pylib/issues
+Project-URL: Homepage, https://github.com/NIAGADS/niagads-pylib
+Project-URL: Source, https://github.com/NIAGADS/niagads-pylib
+Description-Content-Type: text/markdown
+
+<!-- markdownlint-disable -->
+
+<a href="https://github.com/NIAGADS/niagads-pylib/blob/main/bases/niagads/metadata_validator_tool/core.py#L0"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+
+# NIAGADS JSON Schema based metadata validation tools
+
+This tool allows the user to perform [JSON Schema](https://json-schema.org/)-based validation of a sample or file manifest metadata file arranged in tabular format (with a header row that has field names matching the validation schema). 
+
+The tool works for delimited text files (.tab, .csv., .txt) as well as excel (.xls, .xlsx) files. 
+
+This tool can be run as a script or can also be imported as a module.  When run as a script, results are piped to STDOUT unless the `--log` option is specified. 
+
+## Requirements
+
+* Python: >3.12,<4.0
+
+---
+
+## Usage    
+
+### command-line
+
+Run with the `--help` option to get full USAGE information
+
+```bash
+validate-metadata --help
+```
+
+### module
+
+Import package into your python script.
+
+```python
+import niagads.metadata_validator_tool.core as mv_tool
+```
+
+Use [`mv_tool.initialize_validator`](#function-initialize_validator) to initialize and retrieve a validator object for further manipulation. Use [`mv_tool.run`](#function-run) to initialize and run a validation with default configuration. See [validator documentation](https://github.com/NIAGADS/niagads-pylib/blob/main/components/niagads/metadata_validator/README.md) for more information about validator properties and member functions.
+
+
+
+Example code, with schema and metadata files are availble in the code repostory:  [examples/niagads-metadata-validator](https://github.com/NIAGADS/niagads-pylib/blob/e58808f2ef2b412e68ef66ff214683783d2f7576/projects/examples/niagads-metadata-validator/example.ipynb).
+
+---
+
+## API Reference
+
+---
+
+## <kbd>function</kbd> `get_templated_schema_file`
+
+```python
+get_templated_schema_file(dir: str, template: str) → str
+```
+
+Verify that templated schema file `{schemaDir}/{vType}.json` exists. 
+
+
+
+**Args:**
+ 
+ - <b>`path`</b> (str):  path to directory containing schema file 
+ - <b>`template`</b> (str):  template name 
+
+
+
+**Raises:**
+ 
+ - <b>`FileExistsError`</b>:  if the schema file does not exist 
+
+
+
+**Returns:**
+ 
+ - <b>`str`</b>:  schema file name 
+
+
+---
+
+## <kbd>function</kbd> `get_templated_metadata_file`
+
+```python
+get_templated_metadata_file(
+    prefix: str,
+    template: str,
+    extensions: List[str] = ['xlsx', 'xls', 'txt', 'csv', 'tab']
+) → str
+```
+
+Find metadata file based on templated name `{prefix}{validator_type}.{ext}`. 
+
+
+
+**Args:**
+ 
+ - <b>`path`</b> (str):  file path; may include prefix/file pattern to match (e.g. /files/study1/experiment1-) 
+ - <b>`template`</b> (str):  template name 
+ - <b>`extensions`</b> (List[str], optional):  allowable file extensions. Defaults to ["xlsx", "xls", "txt", "csv", "tab"]. 
+
+
+
+**Raises:**
+ 
+ - <b>`FileNotFoundError`</b>:  if metadata file does not exist 
+
+
+
+**Returns:**
+ 
+ - <b>`str`</b>:  metadata file name 
+
+
+---
+
+## <kbd>function</kbd> `initialize_validator`
+
+```python
+initialize_validator(
+    file: str,
+    schema: str,
+    metadataType: MetadataValidatorType,
+    idField: str = None
+) → Union[BiosourcePropertiesValidator, FileManifestValidator]
+```
+
+Initialize and return a metadata validator. 
+
+
+
+**Args:**
+ 
+ - <b>`file`</b> (str):  metadata file name 
+ - <b>`schema`</b> (str):  JSONschema file name 
+ - <b>`metadataType`</b> (MetadataValidatorType):  type of metadata to be validated 
+ - <b>`idField`</b> (str, optional):  biosource id field in the metadata file; required for `BIOSOURCE_PROPERTIES` validation. Defaults to None. 
+
+
+
+**Raises:**
+ 
+ - <b>`RuntimeError`</b>:  if `metadataType == 'BIOSOURCE_PROPERTIES'` and no `idField` was provided 
+ - <b>`ValueError`</b>:  if invalid `metadataType` is specified 
+
+
+
+**Returns:**
+ 
+ - <b>`Union[BiosourcePropertiesValidator, FileManifestValidator]`</b>:  the validator object 
+
+
+---
+
+
+## <kbd>function</kbd> `run`
+
+```python
+run(
+    file: str,
+    schema: str,
+    metadataType: str,
+    idField: str = None,
+    failOnError: bool = False
+)
+```
+
+Run validation. 
+
+Validator initialization fully encapsulated.  Returns validation result. 
+
+
+
+**Args:**
+ 
+ - <b>`file`</b> (str):  metadata file name 
+ - <b>`schema`</b> (str):  JSONschema file name 
+ - <b>`metadataType`</b> (MetadataValidatorType):  type of metadata to be validated 
+ - <b>`idField`</b> (str, optional):  biosource id field in the metadata file; required for `BIOSOURCE_PROPERTIES` valdiatoin. Defaults to None. 
+ - <b>`failOnError`</b> (bool, optional):  raise an exception on validation error if true, otherwise returns list of validation errors. Defaults to False. 
+
+
+
+**Returns:**
+ 
+ - <b>`list`</b>:  list of validation errors 
+
+
+---
+
+## <kbd>class</kbd> `MetadataValidatorType`
+Enum defining types of supported tabular metadata files. 
+
+```python
+BIOSOURCE_PROPERTIES = '''biosource properties file;
+a file that maps a sample or participant to descriptive properties
+(e.g., phenotype or material) or a ISA-TAB-like sample file'''
+
+FILE_MANIFEST = "file manifest or a sample-data-relationship (SDRF) file"
+``` 
+
+
+
+
+
+
+
+---
+
+_This file was automatically generated via [lazydocs](https://github.com/ml-tooling/lazydocs)._
+

niagads_metadata_validator-0.2.0/README.md

@@ -0,0 +1,207 @@
+<!-- markdownlint-disable -->
+
+<a href="https://github.com/NIAGADS/niagads-pylib/blob/main/bases/niagads/metadata_validator_tool/core.py#L0"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+
+# NIAGADS JSON Schema based metadata validation tools
+
+This tool allows the user to perform [JSON Schema](https://json-schema.org/)-based validation of a sample or file manifest metadata file arranged in tabular format (with a header row that has field names matching the validation schema). 
+
+The tool works for delimited text files (.tab, .csv., .txt) as well as excel (.xls, .xlsx) files. 
+
+This tool can be run as a script or can also be imported as a module.  When run as a script, results are piped to STDOUT unless the `--log` option is specified. 
+
+## Requirements
+
+* Python: >3.12,<4.0
+
+---
+
+## Usage    
+
+### command-line
+
+Run with the `--help` option to get full USAGE information
+
+```bash
+validate-metadata --help
+```
+
+### module
+
+Import package into your python script.
+
+```python
+import niagads.metadata_validator_tool.core as mv_tool
+```
+
+Use [`mv_tool.initialize_validator`](#function-initialize_validator) to initialize and retrieve a validator object for further manipulation. Use [`mv_tool.run`](#function-run) to initialize and run a validation with default configuration. See [validator documentation](https://github.com/NIAGADS/niagads-pylib/blob/main/components/niagads/metadata_validator/README.md) for more information about validator properties and member functions.
+
+
+
+Example code, with schema and metadata files are availble in the code repostory:  [examples/niagads-metadata-validator](https://github.com/NIAGADS/niagads-pylib/blob/e58808f2ef2b412e68ef66ff214683783d2f7576/projects/examples/niagads-metadata-validator/example.ipynb).
+
+---
+
+## API Reference
+
+---
+
+## <kbd>function</kbd> `get_templated_schema_file`
+
+```python
+get_templated_schema_file(dir: str, template: str) → str
+```
+
+Verify that templated schema file `{schemaDir}/{vType}.json` exists. 
+
+
+
+**Args:**
+ 
+ - <b>`path`</b> (str):  path to directory containing schema file 
+ - <b>`template`</b> (str):  template name 
+
+
+
+**Raises:**
+ 
+ - <b>`FileExistsError`</b>:  if the schema file does not exist 
+
+
+
+**Returns:**
+ 
+ - <b>`str`</b>:  schema file name 
+
+
+---
+
+## <kbd>function</kbd> `get_templated_metadata_file`
+
+```python
+get_templated_metadata_file(
+    prefix: str,
+    template: str,
+    extensions: List[str] = ['xlsx', 'xls', 'txt', 'csv', 'tab']
+) → str
+```
+
+Find metadata file based on templated name `{prefix}{validator_type}.{ext}`. 
+
+
+
+**Args:**
+ 
+ - <b>`path`</b> (str):  file path; may include prefix/file pattern to match (e.g. /files/study1/experiment1-) 
+ - <b>`template`</b> (str):  template name 
+ - <b>`extensions`</b> (List[str], optional):  allowable file extensions. Defaults to ["xlsx", "xls", "txt", "csv", "tab"]. 
+
+
+
+**Raises:**
+ 
+ - <b>`FileNotFoundError`</b>:  if metadata file does not exist 
+
+
+
+**Returns:**
+ 
+ - <b>`str`</b>:  metadata file name 
+
+
+---
+
+## <kbd>function</kbd> `initialize_validator`
+
+```python
+initialize_validator(
+    file: str,
+    schema: str,
+    metadataType: MetadataValidatorType,
+    idField: str = None
+) → Union[BiosourcePropertiesValidator, FileManifestValidator]
+```
+
+Initialize and return a metadata validator. 
+
+
+
+**Args:**
+ 
+ - <b>`file`</b> (str):  metadata file name 
+ - <b>`schema`</b> (str):  JSONschema file name 
+ - <b>`metadataType`</b> (MetadataValidatorType):  type of metadata to be validated 
+ - <b>`idField`</b> (str, optional):  biosource id field in the metadata file; required for `BIOSOURCE_PROPERTIES` validation. Defaults to None. 
+
+
+
+**Raises:**
+ 
+ - <b>`RuntimeError`</b>:  if `metadataType == 'BIOSOURCE_PROPERTIES'` and no `idField` was provided 
+ - <b>`ValueError`</b>:  if invalid `metadataType` is specified 
+
+
+
+**Returns:**
+ 
+ - <b>`Union[BiosourcePropertiesValidator, FileManifestValidator]`</b>:  the validator object 
+
+
+---
+
+
+## <kbd>function</kbd> `run`
+
+```python
+run(
+    file: str,
+    schema: str,
+    metadataType: str,
+    idField: str = None,
+    failOnError: bool = False
+)
+```
+
+Run validation. 
+
+Validator initialization fully encapsulated.  Returns validation result. 
+
+
+
+**Args:**
+ 
+ - <b>`file`</b> (str):  metadata file name 
+ - <b>`schema`</b> (str):  JSONschema file name 
+ - <b>`metadataType`</b> (MetadataValidatorType):  type of metadata to be validated 
+ - <b>`idField`</b> (str, optional):  biosource id field in the metadata file; required for `BIOSOURCE_PROPERTIES` valdiatoin. Defaults to None. 
+ - <b>`failOnError`</b> (bool, optional):  raise an exception on validation error if true, otherwise returns list of validation errors. Defaults to False. 
+
+
+
+**Returns:**
+ 
+ - <b>`list`</b>:  list of validation errors 
+
+
+---
+
+## <kbd>class</kbd> `MetadataValidatorType`
+Enum defining types of supported tabular metadata files. 
+
+```python
+BIOSOURCE_PROPERTIES = '''biosource properties file;
+a file that maps a sample or participant to descriptive properties
+(e.g., phenotype or material) or a ISA-TAB-like sample file'''
+
+FILE_MANIFEST = "file manifest or a sample-data-relationship (SDRF) file"
+``` 
+
+
+
+
+
+
+
+---
+
+_This file was automatically generated via [lazydocs](https://github.com/ml-tooling/lazydocs)._

niagads_metadata_validator-0.2.0/niagads/arg_parser/__init__.py

@@ -0,0 +1,3 @@
+from niagads.arg_parser import core
+
+__all__ = ["core"]

niagads_metadata_validator-0.2.0/niagads/arg_parser/core.py

@@ -0,0 +1,41 @@
+"""helpers for argparse args, including custom actions"""
+
+import json
+from argparse import ArgumentTypeError
+
+from niagads.enums.core import CaseInsensitiveEnum
+
+
+def json_type(value: str) -> dict:
+    """
+    convert a JSON string argument value to an object
+
+    Args:
+        value (str): JSON string
+
+    Raises:
+        argparse.ArgumentTypeError
+
+    Returns:
+        dict: decoded JSON
+    """
+    try:
+        return json.decodes(value)
+    except:
+        raise ArgumentTypeError("Invalid JSON: " + value)
+
+
+def case_insensitive_enum_type(enumType: CaseInsensitiveEnum):
+    """check that the string belongs to the `enumType`"""
+
+    def type_func(value):
+        try:
+            matchedEnum: CaseInsensitiveEnum = enumType(value)
+            return matchedEnum.value
+
+        except:
+            raise ArgumentTypeError(
+                f"invalid choice: '{value} (choose from [{', '.join(enumType.list())}]"
+            )
+
+    return type_func

niagads_metadata_validator-0.2.0/niagads/csv_parser/__init__.py

@@ -0,0 +1,3 @@
+from niagads.csv_parser import core
+
+__all__ = ["core"]

niagads_metadata_validator-0.2.0/niagads/csv_parser/core.py

@@ -0,0 +1,123 @@
+import logging
+import json
+
+from csv import Sniffer, Dialect
+from pandas import read_csv, DataFrame
+
+from niagads.dict_utils.core import convert_str2numeric_values
+from niagads.pd_dataframe.core import strip
+
+
+class CSVFileParser:
+    """
+    parser for CSV files; mainly to add the following functionality:
+
+    * infer delimiter
+    * to_json (leveraging pandas)
+    """
+
+    def __init__(self, file: str, sep: str = None, debug: bool = False):
+        """
+        init new CSVParser
+
+        Args:
+            file (str): file name (full path)
+            sep (str, optional): delimiter; if None will attempt to infer.  Defaults to None.
+            debug (bool, optional): enable debug mode. Defaults to False.
+        """
+        self._debug = debug
+        self.logger = logging.getLogger(__name__)
+        self.__file = file
+        self.__sep = sep
+        self.__na = None  # missing value string representation
+        self.__strip = False  # flag for trimming leading & trailing whitespace
+
+    def na(self, value: str):
+        """
+        fill NA's with specified value when using pandas conversions
+
+        Args:
+            value (str): value to fill (e.g., 'NULL', 'NA', '.')
+        """
+        self.__na = value
+
+    def strip(self, strip=True):
+        """
+        flag indicating whether to iterate over all fields and
+        trim leading and trailing spaces when converting to JSON or CSV
+
+        Args:
+            strip (bool, optional): trim leading and trailing spaces from all fields. Defaults to True.
+        """
+        self.__strip = strip
+
+    def to_json(self, transpose=False, returnStr=False, **kwargs):
+        """
+        converts the CSV file to JSON
+
+        Args:
+            transpose (bool, optional): transpose the worksheet?
+            returnStr (bool, optional): return jsonStr instead of object
+            **kwargs (optional): arguments to pass to `pandas` `read_csv` see
+                (see https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html))
+
+        Returns:
+            if `returnStr` returns JSON string instead of object
+        """
+
+        # orient='records' returns indexes; e.g. [index: {row data}] so need to extract the values
+        jsonStr = self.to_pandas_df(transpose, **kwargs).to_json(orient="records")
+
+        # convert strings to numeric so can do typing validation
+        jsonObj = json.loads(jsonStr)
+        if isinstance(jsonObj, list):
+            jsonObj = [convert_str2numeric_values(r) for r in json.loads(jsonStr)]
+        else:
+            jsonObj = convert_str2numeric_values(jsonObj)
+
+        return json.dumps(jsonObj) if returnStr else json.loads(jsonStr)
+
+    def __trim(self, df: DataFrame):
+        """
+        trims trailing spaces if set in options
+
+        Args:
+            df (DataFrame): pandas data frame
+        """
+        return strip(df) if self.__strip else df
+
+    def sniff(self):
+        """
+        'sniff' out / infer the delimitier
+        """
+        if self.__sep is not None:
+            return self.__sep
+        else:
+            with open(self.__file, "r") as fh:
+                dialect: Dialect = Sniffer().sniff(fh.read(1024))
+                fh.seek(0)
+                return dialect.delimiter
+
+    def to_pandas_df(self, transpose=False, **kwargs) -> DataFrame:
+        """
+        _summary_
+
+        Args:
+            transpose (str): transpose the worksheet
+            **kwargs: must match expected args for pandas.read_excel
+                (see https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html)
+
+        Returns:
+            DataFrame: CSV data in data frame format
+        """
+        if kwargs is None:
+            kwargs = {}
+
+        if "delimiter" not in kwargs:
+            kwargs["delimiter"] = self.sniff() if self.__sep is None else self.__sep
+
+        # raise error if False
+        df: DataFrame = read_csv(self.__file, **kwargs)
+        if self.__na is not None:
+            df.fillna(self.__na)
+        return self.__trim(df.T) if transpose else self.__trim(df)

niagads_metadata_validator-0.2.0/niagads/csv_validator/__init__.py

@@ -0,0 +1,3 @@
+from niagads.csv_validator import core
+
+__all__ = ["core"]