tsdf 0.5.2__tar.gz → 0.6.1__tar.gz

{tsdf-0.5.2 → tsdf-0.6.1}/PKG-INFO

@@ -1,19 +1,18 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: tsdf
-Version: 0.5.2
+Version: 0.6.1
 Summary: A Python library that provides methods for encoding and decoding TSDF (Time Series Data Format) data, which allows you to easily create, manipulate and serialize TSDF files in your Python code.
-Home-page: https://github.com/biomarkersParkinson/tsdf
 License: Apache-2.0
+License-File: LICENSE
 Keywords: Time Series Data Format (TSDF),binary data,digital sensors
-Author: Peter Kok
-Author-email: p.kok@esciencecenter.nl
-Requires-Python: >=3.9,<4.0
+Author: Vedran Kasalica
+Author-email: v.kaslica@esciencecenter.nl
+Requires-Python: >=3.11,<3.13
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Requires-Dist: numpy (>=1.24.1,<2.0.0)
 Requires-Dist: pandas (>=2.1.3,<3.0.0)
 Project-URL: Repository, https://github.com/biomarkersParkinson/tsdf
@@ -25,7 +24,7 @@ Description-Content-Type: text/markdown
 | Badges | |
 |:----:|----|
 | **Packages and Releases** | [![Latest release](https://img.shields.io/github/release/biomarkersparkinson/tsdf.svg)](https://github.com/biomarkersparkinson/tsdf/releases/latest) [![PyPI](https://img.shields.io/pypi/v/tsdf.svg)](https://pypi.python.org/pypi/tsdf/)  [![Static Badge](https://img.shields.io/badge/RSD-tsdf-lib)](https://research-software-directory.org/software/tsdf) |
-| **Build Status** | [![](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/) ![Python package](https://github.com/biomarkersparkinson/tsdf/workflows/Python%20package/badge.svg) |
+| **Build Status** | [![](https://img.shields.io/badge/python-3.11%2C3.12-blue.svg)](https://www.python.org/downloads/) ![Build and test](https://github.com/biomarkersparkinson/tsdf/workflows/build-and-test.yml/badge.svg) |
 | **DOI** | [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7867899.svg)](https://doi.org/10.5281/zenodo.7867899) |
 | **License** |  [![GitHub license](https://img.shields.io/github/license/biomarkersParkinson/tsdf)](https://github.com/biomarkersparkinson/tsdf/blob/main/LICENSE) |
 | **Fairness** |  [![fair-software.eu](https://img.shields.io/badge/fair--software.eu-%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F-green)](https://fair-software.eu) [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/8083/badge)](https://www.bestpractices.dev/projects/8083) |
@@ -34,11 +33,16 @@ Description-Content-Type: text/markdown
 
 A package ([documentation](https://biomarkersparkinson.github.io/tsdf/)) to load TSDF data ([specification](https://arxiv.org/abs/2211.11294)) into Python.
 
+## Overview
+The [tsdf package](10.5281/zenodo.7867899) is a comprehensively documented reference implementation of the Time Series Data Format (TSDF) standard [[1]](https://arxiv.org/abs/2211.11294). TSDF simplifies data storage and exchange of multi-channel digital sensor data, thereby promoting interpretability and reproducibility of scientific results. Sensor measurements and timestamps are stored as raw tabular binary array files. To ensure unambiguous reconstruction, binary array files are accompanied by human-readable JavaScript Object Notation (JSON) metadata files, which contain a set of mandatory fields limited to essential sensor measurement information.
+
+The tsdf Python package implements functions for reading and writing TSDF files. It guarantees formatting and metadata consistency. It enforces usage of the essential metadata such as study identification, time frame, data channel descriptions and data attributes corresponding to the binary data.
+
 ## Installation
 
 ### Using `pip`
 
-The package is available in PyPi and requires [Python 3.9](https://www.python.org/downloads/) or higher. It can be installed using:
+The package is available in PyPi and requires [Python 3.11](https://www.python.org/downloads/) or higher. It can be installed using:
 
 ```bash
 $ pip install tsdf
@@ -59,17 +63,19 @@ poetry run pytest
 
 ### Building the documentation
 
-We use [mkdocs](https://www.mkdocs.org/) to build the documentation. If you want to build the documentation locally, the following commands will prove useful:
+We use [Sphinx](https://www.sphinx-doc.org/) to build the documentation. Use this command to build the documentation locally:
 
 ```bash
-mkdocs build       # build the documentation
-mkdocs serve       # serve the documentation on a local server
-mkdocs gh-deploy   # deploy the documentation to GitHub pages
+poetry run make html --directory docs
 ```
 
 ## Contributing
 
-Interested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.
+We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for more details on coding standards, how to get started, and the submission process.
+
+## Code of Conduct
+
+To ensure a welcoming and respectful community, all contributors and participants are expected to adhere to our [Code of Conduct](CONDUCT.md). By participating in this project, you agree to abide by its terms.
 
 ## License
 

{tsdf-0.5.2 → tsdf-0.6.1}/README.md

@@ -4,7 +4,7 @@
 | Badges | |
 |:----:|----|
 | **Packages and Releases** | [![Latest release](https://img.shields.io/github/release/biomarkersparkinson/tsdf.svg)](https://github.com/biomarkersparkinson/tsdf/releases/latest) [![PyPI](https://img.shields.io/pypi/v/tsdf.svg)](https://pypi.python.org/pypi/tsdf/)  [![Static Badge](https://img.shields.io/badge/RSD-tsdf-lib)](https://research-software-directory.org/software/tsdf) |
-| **Build Status** | [![](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/) ![Python package](https://github.com/biomarkersparkinson/tsdf/workflows/Python%20package/badge.svg) |
+| **Build Status** | [![](https://img.shields.io/badge/python-3.11%2C3.12-blue.svg)](https://www.python.org/downloads/) ![Build and test](https://github.com/biomarkersparkinson/tsdf/workflows/build-and-test.yml/badge.svg) |
 | **DOI** | [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7867899.svg)](https://doi.org/10.5281/zenodo.7867899) |
 | **License** |  [![GitHub license](https://img.shields.io/github/license/biomarkersParkinson/tsdf)](https://github.com/biomarkersparkinson/tsdf/blob/main/LICENSE) |
 | **Fairness** |  [![fair-software.eu](https://img.shields.io/badge/fair--software.eu-%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F-green)](https://fair-software.eu) [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/8083/badge)](https://www.bestpractices.dev/projects/8083) |
@@ -13,11 +13,16 @@
 
 A package ([documentation](https://biomarkersparkinson.github.io/tsdf/)) to load TSDF data ([specification](https://arxiv.org/abs/2211.11294)) into Python.
 
+## Overview
+The [tsdf package](10.5281/zenodo.7867899) is a comprehensively documented reference implementation of the Time Series Data Format (TSDF) standard [[1]](https://arxiv.org/abs/2211.11294). TSDF simplifies data storage and exchange of multi-channel digital sensor data, thereby promoting interpretability and reproducibility of scientific results. Sensor measurements and timestamps are stored as raw tabular binary array files. To ensure unambiguous reconstruction, binary array files are accompanied by human-readable JavaScript Object Notation (JSON) metadata files, which contain a set of mandatory fields limited to essential sensor measurement information.
+
+The tsdf Python package implements functions for reading and writing TSDF files. It guarantees formatting and metadata consistency. It enforces usage of the essential metadata such as study identification, time frame, data channel descriptions and data attributes corresponding to the binary data.
+
 ## Installation
 
 ### Using `pip`
 
-The package is available in PyPi and requires [Python 3.9](https://www.python.org/downloads/) or higher. It can be installed using:
+The package is available in PyPi and requires [Python 3.11](https://www.python.org/downloads/) or higher. It can be installed using:
 
 ```bash
 $ pip install tsdf
@@ -38,17 +43,19 @@ poetry run pytest
 
 ### Building the documentation
 
-We use [mkdocs](https://www.mkdocs.org/) to build the documentation. If you want to build the documentation locally, the following commands will prove useful:
+We use [Sphinx](https://www.sphinx-doc.org/) to build the documentation. Use this command to build the documentation locally:
 
 ```bash
-mkdocs build       # build the documentation
-mkdocs serve       # serve the documentation on a local server
-mkdocs gh-deploy   # deploy the documentation to GitHub pages
+poetry run make html --directory docs
 ```
 
 ## Contributing
 
-Interested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.
+We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for more details on coding standards, how to get started, and the submission process.
+
+## Code of Conduct
+
+To ensure a welcoming and respectful community, all contributors and participants are expected to adhere to our [Code of Conduct](CONDUCT.md). By participating in this project, you agree to abide by its terms.
 
 ## License
 

{tsdf-0.5.2 → tsdf-0.6.1}/pyproject.toml

@@ -1,10 +1,15 @@
 [tool.poetry]
 name = "tsdf"
-version = "0.5.2"
+version = "0.6.1"
 description = "A Python library that provides methods for encoding and decoding TSDF (Time Series Data Format) data, which allows you to easily create, manipulate and serialize TSDF files in your Python code."
-authors = ["Peter Kok <p.kok@esciencecenter.nl>",
-            "Pablo Rodríguez <p.rodriguez-sanchez@esciencecenter.nl>",
-            "Vedran Kasalica <v.kaslica@esciencecenter.nl>"]
+authors = ["Vedran Kasalica <v.kaslica@esciencecenter.nl>",
+           "Pablo Rodríguez <p.rodriguez-sanchez@esciencecenter.nl>",
+           "Luc Evers <luc.evers@radboudumc.nl>",
+           "Erik Post <erik.post@radboudumc.nl>",
+           "Yordan Raykov <yordan.raykov@nottingham.ac.uk>",
+           "Max Little <maxl@mit.edu>",
+           "Peter Kok <p.kok@esciencecenter.nl>",
+            ]
 license = "Apache-2.0"
 classifiers = [
     "License :: OSI Approved :: Apache Software License",
@@ -16,23 +21,31 @@ keywords = ["Time Series Data Format (TSDF)", "binary data", "digital sensors"]
 repository = "https://github.com/biomarkersParkinson/tsdf"
 
 [tool.poetry.dependencies]
-python = "^3.9"
+python = ">=3.11,<3.13"
 numpy = "^1.24.1"
 pandas = "^2.1.3"
 
 [tool.poetry.group.dev.dependencies]
+coverage = "^7.0.0"
+matplotlib = "^3.6.3"
+pytype = "^2024.4.11"
+myst-parser = "^3.0.1"
+
+[tool.poetry.group.testing.dependencies]
+ipykernel = "^6.19.2"
 pytest = "^7.2.0"
 pytest-cov = "^4.0.0"
 pytest-datadir = "^1.4.1"
-jupyter = "^1.0.0"
-ipykernel = "^6.19.2"
-coverage = "^7.0.0"
-matplotlib = "^3.6.3"
-mkdocs = "^1.4.2"
-mkdocs-jupyter = "^0.22.0"
+
+[tool.poetry.group.docs.dependencies]
+sphinx = "^7.3.7"
+myst-nb = "^1.1.0"
+sphinx-autoapi = "^3.1.2"
+sphinx-rtd-theme = "^2.0.0"
 
 [tool.poetry.scripts]
 validate-tsdf = "tsdf.validator:main"
+#docs = "sphinx.cmd.build:main"
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]

{tsdf-0.5.2 → tsdf-0.6.1}/src/tsdf/parse_metadata.py

@@ -6,6 +6,8 @@ Reference: https://arxiv.org/abs/2211.11294
 
 import os
 from typing import Any, Dict, List
+import re
+from dateutil import parser
 
 from tsdf import constants
 from tsdf import tsdfmetadata
@@ -82,7 +84,7 @@ def _read_struct(
     # levels of the TSDF structure.
     # Extend the mapping recursively with values provided at those levels.
     for key, value in remaining_data.items():
-        if _is_a_list(value):
+        if isinstance(value, list):
             for each_value in value:
                 all_streams = all_streams | _read_struct(
                     each_value, defined_properties.copy(), source_path, version
@@ -135,17 +137,6 @@ def _contains_file_name(data: Any) -> bool:
     return False
 
 
-def _is_a_list(value) -> bool:
-    """
-    Function returns True if the value is a list, otherwise it returns False.
-
-    :param value: value to be checked.
-
-    :return: True if the value is a list, otherwise False.
-    """
-    return isinstance(value, list)
-
-
 def contains_tsdf_mandatory_fields(dictionary: Dict[str, Any]) -> bool:
     """
     Verifies that all the mandatory properties for TSDF metadata are provided,
@@ -242,4 +233,33 @@ def confirm_dir_of_metadata(metadatas: List["tsdfmetadata.TSDFMetadata"]) -> Non
         if init_metadata.file_name == curr_metadata.file_name:
             raise tsdfmetadata.TSDFMetadataFieldValueError(
                 "Two metadata objects cannot reference the same binary file (file_name)."
+                #TODO: why not?
             )
+
+def is_iso8601(date_string: str) -> bool:
+    """
+    Checks if the given date string is in ISO8601 format.
+
+    :param date_string: date string to be checked.
+    """
+    # Note that we need both the regex and the parser to validate the date string
+    # The regex only still allows for invalid dates, e.g. 2021-02-29
+    # The parser is too lenient in accepting different formats
+    iso8601_regex = r"^(-?(?:[1-9][0-9]*)?[0-9]{4})-(1[0-2]|0[1-9])-(3[01]|0[1-9]|[12][0-9])(T(2[0-3]|[01][0-9]):[0-5][0-9]:[0-5][0-9](?:\.[0-9]+)?(?:Z|[+-](?:2[0-3]|[01][0-9]):[0-5][0-9])?)?$"
+    if re.match(iso8601_regex, date_string):
+        try:
+            parser.parse(date_string)
+            return True
+        except tsdfmetadata.TSDFMetadataFieldValueError:
+            return False
+    return False
+
+def validate_datetimes(metadata: tsdfmetadata.TSDFMetadata) -> bool:
+    """
+    Validates the start and end date format of the TSDFMetaData object.
+    """
+    if not is_iso8601(metadata.start_iso8601):
+        raise tsdfmetadata.TSDFMetadataFieldValueError(f"Invalid start_iso8601: {metadata.start_iso8601}")
+    if not is_iso8601(metadata.end_iso8601):
+        raise tsdfmetadata.TSDFMetadataFieldValueError(f"Invalid end_iso8601: {metadata.end_iso8601}")
+    return True

{tsdf-0.5.2 → tsdf-0.6.1}/src/tsdf/read_tsdf.py

@@ -6,6 +6,7 @@ Reference: https://arxiv.org/abs/2211.11294
 
 import json
 import os
+from pathlib import Path
 from typing import Dict, List
 from tsdf import file_utils 
 from tsdf.constants import METADATA_NAMING_PATTERN
@@ -71,7 +72,7 @@ def load_metadatas_from_dir(
     return metadatas
 
 
-def load_metadata_from_path(path: str) -> Dict[str, TSDFMetadata]:
+def load_metadata_from_path(path: Path) -> Dict[str, TSDFMetadata]:
     """
     Loads a TSDF metadata file, returns a dictionary
 

{tsdf-0.5.2 → tsdf-0.6.1}/src/tsdf/tsdfmetadata.py

@@ -1,5 +1,7 @@
 import copy
 from typing import Any, Dict, List
+from datetime import datetime
+from dateutil import parser
 
 from tsdf import parse_metadata
 
@@ -48,11 +50,11 @@ class TSDFMetadata:
 
     file_dir_path: str
     """ A reference to the directory path, so we don't need it again when reading associated binary files. """
-    metadata_file_name: str
+    metadata_file_name: str #TODO: do we need this?? / is it used?
     """ A reference to the source path, so we don't need it again when reading associated binary files. """
 
     def __init__(
-        self, dictionary: Dict[str, Any], dir_path: str, metadata_file_name: str = ""
+        self, dictionary: Dict[str, Any], dir_path: str, metadata_file_name: str = "", do_validate: bool = True
     ) -> None:
         """
         The default constructor takes a dictionary as an argument and creates each
@@ -61,14 +63,34 @@ class TSDFMetadata:
 
         :param dictionary: dictionary containing TSDF metadata.
         :param dir_path: path to the directory where the metadata file is stored.
-        :param file_name: (optional) name of the metadata file.
+        :param metadata_file_name: (optional) name of the metadata file.
+        :param do_validate: (optional) flag to validate the metadata.
         """
-        parse_metadata.contains_tsdf_mandatory_fields(dictionary) #TODO: how to load a dict that is not complete yet?
+
+        # Copy the attributes from the dictionary to the object
         for key, value in dictionary.items():
             setattr(self, key, value)
         self.file_dir_path = dir_path
         self.metadata_file_name = metadata_file_name
 
+        # Validate the metadata
+        if do_validate:
+            if not self.validate():
+                raise TSDFMetadataFieldValueError("The provided metadata is invalid.")
+
+
+    def validate(self) -> bool:
+        isValid: bool = True
+
+        # Validate presence of mandatory fields
+        dict = self.get_plain_tsdf_dict_copy()
+        isValid = isValid and parse_metadata.contains_tsdf_mandatory_fields(dict)
+
+        # Validate datetimes
+        isValid = isValid and parse_metadata.validate_datetimes(self)
+
+        return isValid
+
     def get_plain_tsdf_dict_copy(self) -> Dict[str, Any]:
         """
         Method returns the a copy of the dict containing fields needed for the TSDF file.
@@ -81,3 +103,41 @@ class TSDFMetadata:
         if simple_dict.get("metadata_file_name") is not None:
             simple_dict.pop("metadata_file_name")
         return simple_dict
+
+    def set_start_datetime(self, date_time: datetime) -> None:
+        """
+        Sets the start date of the recording in ISO8601 format.
+        :param date_time: datetime object containing the start date.
+        """
+        self.start_iso8601 = date_time.isoformat()
+
+    def get_start_datetime(self) -> datetime:
+        """
+        Returns the start date of the recording as a datetime object.
+        :return: datetime object containing the start date.
+        """
+        return parser.parse(self.start_iso8601)
+
+    def set_end_datetime(self, date_time: datetime) -> None:
+        """
+        Sets the end date of the recording in ISO8601 format.
+        :param date_time: datetime object containing the end date.
+        """
+        self.end_iso8601 = date_time.isoformat()
+
+    def get_end_datetime(self) -> datetime:
+        """
+        Returns the end date of the recording as a datetime object.
+        :return: datetime object containing the end date.
+        """
+        return parser.parse(self.end_iso8601)
+
+    start = property(get_start_datetime, set_start_datetime, doc=
+        """
+        Start time of the recording.
+        """)
+
+    end = property(get_end_datetime, set_end_datetime, doc=
+        """
+        End time of the recording.
+        """)

{tsdf-0.5.2 → tsdf-0.6.1}/src/tsdf/write_binary.py

@@ -5,26 +5,24 @@ Reference: https://arxiv.org/abs/2211.11294
 """
 
 import os
-from typing import Any, Dict
+from typing import Any, Dict, List
 import numpy as np
 import pandas as pd
-from tsdf import numpy_utils 
+from tsdf import numpy_utils
 
 from tsdf.tsdfmetadata import TSDFMetadata
 
 
 def write_dataframe_to_binaries(
-    file_dir: str, df: pd.DataFrame, metadatas: [TSDFMetadata]
+    file_dir: str, df: pd.DataFrame, metadatas: List[TSDFMetadata]
 ) -> None:
     """
     Save binary file based on the provided pandas DataFrame.
 
-    :param file_dir: path to the directory where the file will be saved.
-    :param file_name: name of the file to be saved.
-    :param data: pandas DataFrame containing the data.
-    :param metadata: dictionary containing the metadata.
-
-    :return: TSDFMetadata object.
+    :param file_dir:    path to the directory where the file will be saved.
+    :param df:          pandas DataFrame containing the data.
+    :param metadatas:   list of metadata objects to be saved, also contains
+                        channels to be retrieved from dataframe.
     """
     for metadata in metadatas:
         file_name = metadata.file_name

{tsdf-0.5.2 → tsdf-0.6.1}/src/tsdf/write_tsdf.py

@@ -19,6 +19,9 @@ def write_metadata(metadatas: List[TSDFMetadata], file_name: str) -> None:
 
     :raises TSDFMetadataFieldValueError: if the metadata files cannot be combined (e.g. they have no common fields) or if the list of TSDFMetadata objects is empty.
     """
+    for meta in metadatas:
+        meta.validate()
+
     if len(metadatas) == 0:
         raise TSDFMetadataFieldValueError(
             "Metadata cannot be saved, as the list of TSDFMetadata objects is empty."
@@ -42,7 +45,7 @@ def write_metadata(metadatas: List[TSDFMetadata], file_name: str) -> None:
         )
 
     if len(plain_meta) > 0:
-        overlap["sensors"] = _calculate_ovelaps_rec(plain_meta)
+        overlap["sensors"] = _calculate_overlaps_rec(plain_meta)
     file_utils.write_to_file(overlap, metadatas[0].file_dir_path, file_name)
 
 
@@ -76,7 +79,7 @@ def _extract_common_fields(metadatas: List[Dict[str, Any]]) -> Dict[str, Any]:
     return meta_overlap
 
 
-def _calculate_ovelaps_rec(metadatas: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+def _calculate_overlaps_rec(metadatas: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
     """
     A recursive call that optimises the structure of the TSDF metadata, by grouping common values. For the input the list of dictionaries
     corresponds to a list of "flat" metadata dictionaries. The output is a list of dictionaries (potentially of length 1) that contain
@@ -107,11 +110,11 @@ def _calculate_ovelaps_rec(metadatas: List[Dict[str, Any]]) -> List[Dict[str, An
     # Handle the first group
     first_overlap = _extract_common_fields(first_group)
     if len(first_group) > 0:
-        first_overlap["sensors"] = _calculate_ovelaps_rec(first_group)
+        first_overlap["sensors"] = _calculate_overlaps_rec(first_group)
     final_metadata.append(first_overlap)
 
     # Handle the rest of the elements
-    second_overlap = _calculate_ovelaps_rec(second_grop)
+    second_overlap = _calculate_overlaps_rec(second_grop)
     final_metadata.extend(second_overlap)
 
     return final_metadata