tacoreader 0.1.0__tar.gz

tacoreader-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2024, Cesar Aybar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

tacoreader-0.1.0/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.1
+Name: tacoreader
+Version: 0.1.0
+Summary: A Python package to read Cloud-Optimized Datasets
+Home-page: https://github.com/tacofoundation/tacoreader
+Author: Cesar Aybar
+Author-email: cesar.aybar@uv.es
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: geopandas (>=1.0.1)
+Requires-Dist: pyarrow (>=17.0.0)
+Requires-Dist: requests (>=2.32.3)
+Requires-Dist: shapely (>=2.0.6)
+Project-URL: Documentation, https://tacofoundation.github.io/tacoreader/
+Project-URL: Repository, https://github.com/tacofoundation/tacoreader
+Description-Content-Type: text/markdown
+
+# tacoreader
+
+[![Release](https://img.shields.io/github/v/release/csaybar/tacoreader)](https://img.shields.io/github/v/release/csaybar/tacoreader)
+[![Build status](https://img.shields.io/github/actions/workflow/status/csaybar/tacoreader/main.yml?branch=main)](https://github.com/csaybar/tacoreader/actions/workflows/main.yml?query=branch%3Amain)
+[![codecov](https://codecov.io/gh/csaybar/tacoreader/branch/main/graph/badge.svg)](https://codecov.io/gh/csaybar/tacoreader)
+[![Commit activity](https://img.shields.io/github/commit-activity/m/csaybar/tacoreader)](https://img.shields.io/github/commit-activity/m/csaybar/tacoreader)
+[![License](https://img.shields.io/github/license/csaybar/tacoreader)](https://img.shields.io/github/license/csaybar/tacoreader)
+
+Read TACO datasets
+
+- **Github repository**: <https://github.com/csaybar/tacoreader/>
+- **Documentation** <https://csaybar.github.io/tacoreader/>
+
+## Getting started with your project
+
+First, create a repository on GitHub with the same name as this project, and then run the following commands:
+
+```bash
+git init -b main
+git add .
+git commit -m "init commit"
+git remote add origin git@github.com:csaybar/tacoreader.git
+git push -u origin main
+```
+
+Finally, install the environment and the pre-commit hooks with
+
+```bash
+make install
+```
+
+You are now ready to start development on your project!
+The CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.
+
+To finalize the set-up for publishing to PyPI or Artifactory, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/publishing/#set-up-for-pypi).
+For activating the automatic documentation with MkDocs, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/mkdocs/#enabling-the-documentation-on-github).
+To enable the code coverage reports, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/codecov/).
+
+## Releasing a new version
+
+- Create an API Token on [PyPI](https://pypi.org/).
+- Add the API Token to your projects secrets with the name `PYPI_TOKEN` by visiting [this page](https://github.com/csaybar/tacoreader/settings/secrets/actions/new).
+- Create a [new release](https://github.com/csaybar/tacoreader/releases/new) on Github.
+- Create a new tag in the form `*.*.*`.
+- For more details, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/cicd/#how-to-trigger-a-release).
+
+---
+
+Repository initiated with [fpgmaas/cookiecutter-poetry](https://github.com/fpgmaas/cookiecutter-poetry).
+

tacoreader-0.1.0/README.md

@@ -0,0 +1,49 @@
+# tacoreader
+
+[![Release](https://img.shields.io/github/v/release/csaybar/tacoreader)](https://img.shields.io/github/v/release/csaybar/tacoreader)
+[![Build status](https://img.shields.io/github/actions/workflow/status/csaybar/tacoreader/main.yml?branch=main)](https://github.com/csaybar/tacoreader/actions/workflows/main.yml?query=branch%3Amain)
+[![codecov](https://codecov.io/gh/csaybar/tacoreader/branch/main/graph/badge.svg)](https://codecov.io/gh/csaybar/tacoreader)
+[![Commit activity](https://img.shields.io/github/commit-activity/m/csaybar/tacoreader)](https://img.shields.io/github/commit-activity/m/csaybar/tacoreader)
+[![License](https://img.shields.io/github/license/csaybar/tacoreader)](https://img.shields.io/github/license/csaybar/tacoreader)
+
+Read TACO datasets
+
+- **Github repository**: <https://github.com/csaybar/tacoreader/>
+- **Documentation** <https://csaybar.github.io/tacoreader/>
+
+## Getting started with your project
+
+First, create a repository on GitHub with the same name as this project, and then run the following commands:
+
+```bash
+git init -b main
+git add .
+git commit -m "init commit"
+git remote add origin git@github.com:csaybar/tacoreader.git
+git push -u origin main
+```
+
+Finally, install the environment and the pre-commit hooks with
+
+```bash
+make install
+```
+
+You are now ready to start development on your project!
+The CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.
+
+To finalize the set-up for publishing to PyPI or Artifactory, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/publishing/#set-up-for-pypi).
+For activating the automatic documentation with MkDocs, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/mkdocs/#enabling-the-documentation-on-github).
+To enable the code coverage reports, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/codecov/).
+
+## Releasing a new version
+
+- Create an API Token on [PyPI](https://pypi.org/).
+- Add the API Token to your projects secrets with the name `PYPI_TOKEN` by visiting [this page](https://github.com/csaybar/tacoreader/settings/secrets/actions/new).
+- Create a [new release](https://github.com/csaybar/tacoreader/releases/new) on Github.
+- Create a new tag in the form `*.*.*`.
+- For more details, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/cicd/#how-to-trigger-a-release).
+
+---
+
+Repository initiated with [fpgmaas/cookiecutter-poetry](https://github.com/fpgmaas/cookiecutter-poetry).

tacoreader-0.1.0/pyproject.toml

@@ -0,0 +1,107 @@
+[tool.poetry]
+name = "tacoreader"
+version = "0.1.0"
+description = "A Python package to read Cloud-Optimized Datasets"
+authors = ["Cesar Aybar <cesar.aybar@uv.es>"]
+repository = "https://github.com/tacofoundation/tacoreader"
+documentation = "https://tacofoundation.github.io/tacoreader/"
+readme = "README.md"
+packages = [
+  {include = "tacoreader"}
+]
+
+[tool.poetry.dependencies]
+python = ">=3.9,<4.0"
+pyarrow = ">=17.0.0"
+geopandas = ">=1.0.1"
+shapely = ">=2.0.6"
+requests = ">=2.32.3"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^7.2.0"
+pytest-cov = "^4.0.0"
+deptry = "^0.16.2"
+mypy = "^1.5.1"
+pre-commit = "^3.4.0"
+tox = "^4.11.1"
+
+[tool.poetry.group.docs.dependencies]
+mkdocs = "^1.4.2"
+mkdocs-material = "^9.2.7"
+mkdocstrings = {extras = ["python"], version = "^0.26.1"}
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.mypy]
+files = ["tacoreader"]
+disallow_untyped_defs = "True"
+disallow_any_unimported = "True"
+no_implicit_optional = "True"
+check_untyped_defs = "True"
+warn_return_any = "True"
+warn_unused_ignores = "True"
+show_error_codes = "True"
+
+
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+target-version = "py39"
+line-length = 120
+fix = true
+select = [
+    # flake8-2020
+    "YTT",
+    # flake8-bandit
+    "S",
+    # flake8-bugbear
+    "B",
+    # flake8-builtins
+    "A",
+    # flake8-comprehensions
+    "C4",
+    # flake8-debugger
+    "T10",
+    # flake8-simplify
+    "SIM",
+    # isort
+    "I",
+    # mccabe
+    "C90",
+    # pycodestyle
+    "E", "W",
+    # pyflakes
+    "F",
+    # pygrep-hooks
+    "PGH",
+    # pyupgrade
+    "UP",
+    # ruff
+    "RUF",
+    # tryceratops
+    "TRY",
+]
+ignore = [
+    # LineTooLong
+    "E501",
+    # DoNotAssignLambda
+    "E731",
+]
+
+[tool.ruff.format]
+preview = true
+
+[tool.coverage.report]
+skip_empty = true
+
+[tool.coverage.run]
+branch = true
+source = ["tacoreader"]
+
+
+[tool.ruff.per-file-ignores]
+"tests/*" = ["S101"]

tacoreader-0.1.0/tacoreader/__init__.py

@@ -0,0 +1,18 @@
+from tacoreader.croissant import read_croissant
+from tacoreader.datacard import read_datacard
+from tacoreader.datacite import read_datacite
+from tacoreader.load import load, load_metadata
+from tacoreader.stac import read_stac
+
+__all__ = [
+    "load",
+    "load_metadata",
+    "read_datacard",
+    "read_stac",
+    "read_croissant",
+    "read_datacite",
+]
+
+__version__ = "0.1.0"
+
+

tacoreader-0.1.0/tacoreader/croissant.py

@@ -0,0 +1,2 @@
+def read_croissant():
+    print("Do not implement it yet")

tacoreader-0.1.0/tacoreader/datacard.py

@@ -0,0 +1,241 @@
+import pathlib
+from typing import List, Union
+
+from tacoreader.load import load_metadata
+
+try:
+    from mdutils.mdutils import MdUtils
+except ImportError:
+    raise ImportError("Please install the mdutils package with: pip install mdutils")
+
+
+def read_datacard(
+    file: Union[str, pathlib.Path, List[pathlib.Path], List[str]],
+    outfile: Union[str, pathlib.Path],
+) -> pathlib.Path:
+
+    # Create a JSON file
+    taco_object: dict = load_metadata(file)
+
+    # Convert the output file to a pathlib.Path object
+    if isinstance(outfile, str):
+        outfile = pathlib.Path(outfile)
+
+    # Create the README.md file
+    md_file = MdUtils(file_name=outfile)
+
+    # --- YAML Header ---
+    md_file.new_line("---")
+    md_file.new_line("license:")
+    for item in taco_object["licenses"]:
+        md_file.new_line(f"  - {item}")
+    md_file.new_line("language:")
+    md_file.new_line("- en")
+    if taco_object["keywords"]:
+        md_file.new_line("tags:")
+        for tag in taco_object["keywords"]:
+            md_file.new_line(f"  - {tag}")
+    md_file.new_line(f'pretty_name: {taco_object["id"]}')
+    md_file.new_line("---")
+
+    # --- Title and Description ---
+    md_file.new_header(level=1, title=taco_object["id"])
+    if taco_object["title"]:
+        md_file.new_line(f'**{taco_object["title"]}**', bold_italics_code="b")
+    md_file.new_paragraph(taco_object["description"])
+
+    # --- Code Snippet ---
+    md_file.new_header(level=2, title="🌮 TACO Snippet")
+    md_file.new_paragraph("Load this dataset using the `tacoreader` library.")
+    md_file.new_line("```python")
+    md_file.new_line("import tacoreader")
+    md_file.new_line("dataset = tacoreader.load('...')")
+    md_file.new_line("```")
+
+    # example in R
+    md_file.new_line("\n")
+    md_file.new_paragraph("Or in R:")
+    md_file.new_line("```r")
+    md_file.new_line("library(tacoreader)")
+    md_file.new_line("dataset <- tacoreader::load('...')")
+    md_file.new_line("```")
+
+    # --- Sensor Information ---
+    if taco_object["optical_data"]:
+        md_file.new_header(level=2, title="🛰️ Sensor Information")
+        md_file.new_paragraph(
+            f'The sensor related to the dataset: **{taco_object["optical_data"]["sensor"]}**'
+        )
+
+    # --- Task Information ---
+    md_file.new_header(level=2, title="🎯 Task")
+    md_file.new_paragraph(
+        f'The task associated with this dataset: **{taco_object["task"]}**'
+    )
+
+    # --- Raw Repository Link ---
+    if taco_object["raw_link"]:
+        md_file.new_header(level=2, title="📂 Original Data Repository")
+        md_file.new_paragraph(
+            f'Source location of the raw data:**{md_file.new_inline_link(link=taco_object["raw_link"]["href"])}**'
+        )
+
+    # --- Discussion Link ---
+    if taco_object["discuss_link"]:
+        md_file.new_header(level=2, title="💬 Discussion")
+        md_file.new_paragraph(
+            f'Insights or clarifications about the dataset: **{md_file.new_inline_link(link=taco_object["discuss_link"]["href"])}**'
+        )
+
+    # --- Split Strategy ---
+    if taco_object["split_strategy"]:
+        md_file.new_header(level=2, title="🔀 Split Strategy")
+        md_file.new_paragraph(
+            f'How the dataset is divided for training, validation, and testing: **{taco_object["split_strategy"]}**'
+        )
+
+    # --- Scientific Publications ---
+    if taco_object["scientific"]["publications"]:
+        md_file.new_header(level=2, title="📚 Scientific Publications")
+        md_file.new_paragraph("Publications that reference or describe the dataset.")
+        for idx, pub in enumerate(taco_object["scientific"]["publications"], start=1):
+            # Add the publication information
+            md_file.new_paragraph(f"### Publication {idx:02d}")
+            md_file.new_line("- **DOI**: " + md_file.new_inline_link(link=pub["doi"]))
+            md_file.new_line("- **Summary**: " + pub["summary"])
+            md_file.new_line("- **BibTeX Citation**:")
+            md_file.new_line("```bibtex")
+            md_file.new_line(pub["citation"].strip("\n"))
+            md_file.new_line("```")
+            md_file.new_line("\n")
+
+    # --- Data Providers ---
+    if taco_object["providers"]:
+        md_file.new_header(level=2, title="🤝 Data Providers")
+        md_file.new_paragraph(
+            "Organizations or individuals responsible for the dataset."
+        )
+
+        # Define table headers
+        table_headers = ["**Name**", "**Role**", "**URL**"]
+        table_data = [table_headers]
+
+        # Populate table with provider data
+        for provider in taco_object["providers"]:
+            table_data.append(
+                [
+                    provider["name"] or "N/A",
+                    (
+                        ", ".join(provider["roles"])
+                        if isinstance(provider["roles"], list)
+                        else provider["roles"] or "N/A"
+                    ),
+                    (
+                        md_file.new_inline_link(link=provider["links"][0]["href"])
+                        if provider["links"]
+                        else "N/A"
+                    ),
+                ]
+            )
+
+        # Flatten the list for Markdown formatting
+        flat_table_data = [cell for row in table_data for cell in row]
+
+        # Create the table
+        md_file.new_table(
+            columns=3, rows=len(table_data), text=flat_table_data, text_align="left"
+        )
+
+    # --- Curators ---
+    if taco_object["curators"]:
+        md_file.new_header(level=2, title="🧑‍🔬 Curators")
+        md_file.new_paragraph(
+            "Responsible for structuring the dataset in the TACO format."
+        )
+
+        # Define table headers
+        table_headers = ["**Name**", "**Organization**", "**URL**"]
+        table_data = [table_headers]
+
+        # Populate table with curator data
+        for curator in taco_object["curators"]:
+            table_data.append(
+                [
+                    curator["name"] or "N/A",
+                    curator["organization"] or "N/A",
+                    (
+                        md_file.new_inline_link(link=curator["links"][0]["href"])
+                        if curator["links"]
+                        else "N/A"
+                    ),
+                ]
+            )
+
+        # Flatten the list for Markdown formatting
+        flat_table_data = [cell for row in table_data for cell in row]
+        # Create the table
+        md_file.new_table(
+            columns=3, rows=len(table_data), text=flat_table_data, text_align="left"
+        )
+
+    # --- Labels ---
+    if taco_object["labels"]:
+        md_file.new_header(level=2, title="🏷️ Labels")
+        md_file.new_paragraph(taco_object["labels"]["label_description"])
+        table_headers = ["**Name**", "**Category**", "**Description**"]
+        table_data = [table_headers]
+        for item in taco_object["labels"]["label_classes"]:
+            table_data.append(
+                [
+                    item["name"] or "N/A",
+                    (
+                        str(item["category"]) if item["category"] is not None else "N/A"
+                    ),  # Zero is a valid category
+                    item["description"] or "N/A",
+                ]
+            )
+
+        # Flatten the list for Markdown formatting
+        flat_table_data = [cell for row in table_data for cell in row]
+        md_file.new_table(
+            columns=3, rows=len(table_data), text=flat_table_data, text_align="left"
+        )
+
+    # --- Optical Bands ---
+    if taco_object["optical_data"]:
+        md_file.new_header(level=2, title="🌈 Optical Bands")
+        md_file.new_paragraph("Spectral bands related to the sensor.")
+        table_headers = [
+            "**Name**",
+            "**Common Name**",
+            "**Description**",
+            "**Center Wavelength**",
+            "**Full Width Half Max**",
+            "**Index**",
+        ]
+        table_data = [table_headers]
+
+        for item in taco_object["optical_data"]["bands"]:
+            table_data.append(
+                [
+                    item["name"] or "N/A",
+                    item["common_name"] or "N/A",
+                    item["description"] or "N/A",
+                    item["center_wavelength"] or "N/A",
+                    item["full_width_half_max"] or "N/A",
+                    str(item["index"]) if item["index"] is not None else "N/A",
+                ]
+            )
+
+        # Flatten the list for Markdown formatting
+        flat_table_data = [cell for row in table_data for cell in row]
+        md_file.new_table(
+            columns=6, rows=len(table_data), text=flat_table_data, text_align="left"
+        )
+
+    # Export the data to the output file
+    file = md_file.get_md_text().replace("\n\n\n  \n", "").replace("  \n", "\n")
+    with open(outfile, "w") as f:
+        f.write(file)
+
+    return outfile

tacoreader-0.1.0/tacoreader/datacite.py

@@ -0,0 +1,2 @@
+def read_datacite():
+    print("Do not implement it yet")

tacoreader-0.1.0/tacoreader/load.py

@@ -0,0 +1,151 @@
+import pathlib
+import re
+from typing import List, Union
+
+import geopandas as gpd
+import pandas as pd
+import requests
+
+from tacoreader import load_local, load_remote, load_utils
+
+
+def load(file: Union[str, pathlib.Path, List[pathlib.Path], List[str]]) -> pd.DataFrame:
+    """Load the dataframe of a tortilla file.
+
+    Args:
+        file (Union[str, pathlib.Path, List]): The path of
+            the tortilla file. If the file is split into multiple
+            parts, a list of paths is accepted. Also, multiple
+            parts can be read by putting a asterisk (*) at the end
+            of the file name. For example, "file*.tortilla". In this
+            case, the function will create a list will all the partitions
+            before the reading process.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+
+    # Transform our snippet into a list of files
+    # If it is not a snippet, it will return the same file
+    file = load_utils.snippet2files(file=file)
+
+    if isinstance(file, list):
+        if load_utils.is_valid_url(file):
+            dataframe = load_remote.remote_files2dataframe(file)
+        else:
+            dataframe = load_local.local_files2dataframe(file)
+    elif isinstance(file, (str, pathlib.Path)):
+        if load_utils.is_valid_url(file):
+            dataframe = load_remote.remote_file2dataframe(file)
+        else:
+            dataframe = load_local.local_file2dataframe(file)
+    else:
+        raise ValueError("Invalid file type. Must be a list, string or pathlib.Path.")
+
+    # Clean up the dataframe
+    dataframe = load_utils.sort_columns_add_geometry(dataframe)
+
+    return TortillaDataFrame(dataframe)
+
+
+def lazy_load(offset: int, file: Union[str, pathlib.Path]) -> pd.DataFrame:
+    """Lazy load a tortilla file.
+
+    Useful for datasets that have tortillas as samples (tortillas inside tortillas).
+    The offset is used to read a specific part of the main tortilla file.
+
+    Args:
+        offset (int): The byte offset where the reading process will start.
+        file (Union[str, pathlib.Path]): The path tot the main tortilla file.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+
+    if load_utils.is_valid_url(file):
+        dataframe = load_remote.remote_lazyfile2dataframe(offset, file)
+    else:
+        dataframe = load_local.local_lazyfile2dataframe(offset, file)
+
+    # Clean up the dataframe
+    dataframe = load_utils.sort_columns_add_geometry(dataframe)
+
+    return TortillaDataFrame(dataframe)
+
+
+def load_metadata(
+    file: Union[str, pathlib.Path, List[pathlib.Path], List[str]]
+) -> dict:
+    """Load the metadata of a tortilla or taco file.
+
+    Args:
+        file (Union[str, pathlib.Path, List]): The path of
+            the taco file. If the file is split into multiple
+            parts, a list of paths is accepted. Also, multiple
+            parts can be read by putting a asterisk (*) at the end
+            of the file name. For example, "file*.tortilla". In this
+            case, the function will create a list will all the partitions
+            before the reading process.
+
+    Returns:
+        dict: The metadata of the taco file.
+    """
+
+    # Transform our snippet into a list of files
+    # If it is not a snippet, it will return the same file
+    file = load_utils.snippet2files(file=file)
+
+    # Load the metadata
+    if isinstance(file, list):
+        if load_utils.is_valid_url(file):
+            metadata = load_remote.remote_files2metadata(file)
+        else:
+            metadata = load_local.local_files2metadata(file)
+    elif isinstance(file, (str, pathlib.Path)):
+        if load_utils.is_valid_url(file):
+            metadata = load_remote.remote_file2metadata(file)
+        else:
+            metadata = load_local.local_file2metadata(file)
+    else:
+        raise ValueError("Invalid file type. Must be a list, string or pathlib.Path.")
+
+    return metadata
+
+
+class TortillaDataFrame(gpd.GeoDataFrame):
+    @property
+    def _constructor(self):
+        return TortillaDataFrame
+
+    @staticmethod
+    def get_internal_path(row):
+        pattern: re.Pattern = re.compile(r"/vsisubfile/(\d+)_(\d+),(.+)")
+        offset, length, path = pattern.match(row["internal:subfile"]).groups()
+
+        # If it is a curl file, remove the first 9 characters
+        if path.startswith("/vsicurl/"):
+            path = path[9:]
+
+        return int(offset), int(length), path
+
+    def read(self, idx):
+        row = self.iloc[idx]
+        if row["internal:file_format"] == "TORTILLA":
+            offset, length, path = self.get_internal_path(row)
+            return lazy_load(row["tortilla:offset"], path)
+        elif row["internal:file_format"] == "BYTES":
+
+            # Obtain the offset, length and internal path
+            offset, length, path = self.get_internal_path(row)
+
+            # Get the bytes
+            if load_utils.is_valid_url(path):
+                headers = {"Range": f"bytes={offset}-{offset + length - 1}"}
+                response: requests.Response = requests.get(path, headers=headers)
+                return response.content
+            else:
+                with open(path, "rb") as f:
+                    f.seek(int(offset))
+                    return f.read(int(length))
+        else:
+            return row["internal:subfile"]

tacoreader-0.1.0/tacoreader/load_local.py

@@ -0,0 +1,202 @@
+import json
+import pathlib
+from typing import List, Union
+
+import pandas as pd
+import pyarrow as pa
+import pyarrow.parquet as pq
+
+
+def local_file2dataframe(file: Union[str, pathlib.Path]):
+    """Read the dataframe of tortilla file given a local path.
+
+    Args:
+        files (Union[str, pathlib.Path]): A local path pointing to the
+            tortilla file.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+    with open(file, "rb") as f:
+        static_bytes = f.read(50)
+
+        # SPLIT the static bytes
+        MB: bytes = static_bytes[:2]
+        FO: bytes = static_bytes[2:10]
+        FL: bytes = static_bytes[10:18]
+        DF: str = static_bytes[18:42].strip().decode()
+        # DP: str = static_bytes[42:50]
+
+        if MB != b"#y":
+            raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+        # Read the NEXT 8 bytes of the file
+        footer_offset: int = int.from_bytes(FO, "little")
+
+        # Seek to the FOOTER offset
+        f.seek(footer_offset)
+
+        # Select the FOOTER length
+        # Read the FOOTER
+        footer_length: int = int.from_bytes(FL, "little")
+        dataframe = pq.read_table(pa.BufferReader(f.read(footer_length))).to_pandas()
+
+        # Convert dataset to DataFrame
+        dataframe["internal:file_format"] = DF
+        dataframe["internal:mode"] = "local"
+        dataframe["internal:subfile"] = dataframe.apply(
+            lambda row: f"/vsisubfile/{row['tortilla:offset']}_{row['tortilla:length']},{file}",
+            axis=1,
+        )
+
+    return dataframe
+
+
+def local_files2dataframe(files: Union[List[str], List[pathlib.Path]]):
+    """Read the dataframe of tortilla files given local paths.
+
+    Args:
+        files (Union[List[str], List[pathlib.Path]]): A list of local
+            paths pointing to the tortilla files.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+
+    # Merge the dataframe of the files
+    container = []
+    for file in files:
+        with open(file, "rb") as f:
+            static_bytes = f.read(50)
+
+            # SPLIT the static bytes
+            MB: bytes = static_bytes[:2]
+            FO: bytes = static_bytes[2:10]
+            FL: bytes = static_bytes[10:18]
+            DF: str = static_bytes[18:42].strip().decode()
+            # DP: str = static_bytes[42:50]
+
+            if MB != b"#y":
+                raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+            # Read the NEXT 8 bytes of the file
+            footer_offset: int = int.from_bytes(FO, "little")
+
+            # Seek to the FOOTER offset
+            f.seek(footer_offset)
+
+            # Select the FOOTER length
+            # Read the FOOTER
+            footer_length: int = int.from_bytes(FL, "little")
+            dataframe = pq.read_table(
+                pa.BufferReader(f.read(footer_length))
+            ).to_pandas()
+
+            # Convert dataset to DataFrame
+            dataframe["internal:file_format"] = DF
+            dataframe["internal:mode"] = "local"
+            dataframe["internal:subfile"] = dataframe.apply(
+                lambda row: f"/vsisubfile/{row['tortilla:offset']}_{row['tortilla:length']},{file}",
+                axis=1,
+            )
+            container.append(dataframe)
+
+    return pd.concat(container, ignore_index=True)
+
+
+def local_lazyfile2dataframe(
+    offset: int, file: Union[str, pathlib.Path]
+) -> pd.DataFrame:
+    """Read the dataframe of tortilla file that is a subfile
+    of a larger file.
+
+    Args:
+        offset (int): The offset of the subfile.
+        file (Union[str, pathlib.Path]): A local path pointing to the
+            main tortilla file.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+
+    with open(file, "rb") as f:
+        # Seek to the OFFSET
+        f.seek(offset)
+
+        static_bytes = f.read(50)
+
+        # SPLIT the static bytes
+        MB: bytes = static_bytes[:2]
+        FO: bytes = static_bytes[2:10]
+        FL: bytes = static_bytes[10:18]
+        DF: str = static_bytes[18:42].strip().decode()
+        # DP: str = static_bytes[42:50]
+
+        if MB != b"#y":
+            raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+        # Read the NEXT 8 bytes of the file
+        footer_offset: int = int.from_bytes(FO, "little") + offset
+
+        # Seek to the FOOTER offset
+        f.seek(footer_offset)
+
+        # Select the FOOTER length
+        # Read the FOOTER
+        footer_length: int = int.from_bytes(FL, "little")
+        dataframe = pq.read_table(pa.BufferReader(f.read(footer_length))).to_pandas()
+
+        # Fix the offset
+        dataframe["tortilla:offset"] = dataframe["tortilla:offset"] + offset
+
+        # Convert dataset to DataFrame
+        dataframe["internal:file_format"] = DF
+        dataframe["internal:mode"] = "local"
+        dataframe["internal:subfile"] = dataframe.apply(
+            lambda row: f"/vsisubfile/{row['tortilla:offset']}_{row['tortilla:length']},{file}",
+            axis=1,
+        )
+
+    return dataframe
+
+
+def local_file2metadata(file: Union[str, pathlib.Path]) -> dict:
+    """Read the dataframe of a taco file given a local path.
+
+    Args:
+        file (Union[str, pathlib.Path]): A local path pointing to the
+            taco file.
+
+    Returns:
+        dict: The metadata of the taco file.
+    """
+    with open(file, "rb") as f:
+        f.seek(50)
+
+        # Read the Collection offset (CO)
+        CO: int = int.from_bytes(f.read(8), "little")
+
+        # Read the Collection length (CL)
+        CL: int = int.from_bytes(f.read(8), "little")
+
+        # Seek to the Collection offset
+        f.seek(CO)
+
+        # Read the Collection (JSON UTF-8 encoded)
+        collection: dict = json.loads(f.read(CL).decode())
+
+    return collection
+
+
+def local_files2metadata(files: Union[List[str], List[pathlib.Path]]) -> dict:
+    """Read the metadata of taco files given local paths.
+
+    Args:
+        files (Union[List[str], List[pathlib.Path]]): A list of local
+            paths pointing to the taco files.
+
+    Returns:
+        dict: The metadata of the taco file.
+    """
+
+    return local_file2dataframe(files[0])

tacoreader-0.1.0/tacoreader/load_remote.py

@@ -0,0 +1,205 @@
+import json
+import pathlib
+from typing import List, Union
+
+import pandas as pd
+import pyarrow as pa
+import pyarrow.parquet as pq
+import requests
+
+
+def remote_file2dataframe(file: str) -> pd.DataFrame:
+    """Read the dataframe of a tortilla file given a URL. The
+        server must support HTTP Range requests.
+
+    Args:
+        files (str): A URL pointing to the tortilla file.
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+    # Fetch the first 8 bytes of the file
+    headers = {"Range": "bytes=0-50"}
+    response: requests.Response = requests.get(file, headers=headers)
+    static_bytes: bytes = response.content
+
+    # SPLIT the static bytes
+    MB: bytes = static_bytes[:2]
+    FO: bytes = static_bytes[2:10]
+    FL: bytes = static_bytes[10:18]
+    DF: str = static_bytes[18:42].strip().decode()
+
+    # Check if the file is a tortilla
+    if MB != b"#y":
+        raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+    # Interpret the bytes as a little-endian integer
+    footer_offset: int = int.from_bytes(FO, "little")
+    footer_length: int = int.from_bytes(FL, "little")
+
+    # Fetch the footer
+    headers = {"Range": f"bytes={footer_offset}-{footer_offset + footer_length - 1}"}
+    with requests.get(file, headers=headers) as response:
+        # Interpret the response as a parquet table
+        dataframe = pq.read_table(pa.BufferReader(response.content)).to_pandas()
+
+    # Add the file format and mode
+    dataframe["internal:file_format"] = DF
+    dataframe["internal:mode"] = "online"
+    dataframe["internal:subfile"] = dataframe.apply(
+        lambda row: f"/vsisubfile/{row['tortilla:offset']}_{row['tortilla:length']},/vsicurl/{file}",
+        axis=1,
+    )
+    return dataframe
+
+
+def remote_files2dataframe(files: List[str]) -> pd.DataFrame:
+    """Read the dataframe of tortillas files given a set of URLs. The
+        server must support HTTP Range requests.
+
+    Args:
+        files (List[str]): A list of URLs pointing to the
+            tortilla files.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+
+    container = []
+    for file in files:
+
+        # Fetch the first 8 bytes of the file
+        headers = {"Range": "bytes=0-50"}
+        response: requests.Response = requests.get(file, headers=headers)
+        static_bytes: bytes = response.content
+
+        # SPLIT the static bytes
+        MB: bytes = static_bytes[:2]
+        FO: bytes = static_bytes[2:10]
+        FL: bytes = static_bytes[10:18]
+        DF: str = static_bytes[18:42].strip().decode()
+
+        # Check if the file is a tortilla
+        if MB != b"#y":
+            raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+        # Interpret the bytes as a little-endian integer
+        footer_offset: int = int.from_bytes(FO, "little")
+        footer_length: int = int.from_bytes(FL, "little")
+
+        # Fetch the footer
+        headers = {"Range": f"bytes={footer_offset}-{footer_offset + footer_length}"}
+        with requests.get(file, headers=headers) as response:
+
+            # Interpret the response as a parquet table
+            dataframe = pq.read_table(pa.BufferReader(response.content)).to_pandas()
+
+        # Add the file format and mode
+        dataframe["internal:file_format"] = DF
+        dataframe["internal:mode"] = "online"
+        dataframe["internal:subfile"] = dataframe.apply(
+            lambda row: f"/vsisubfile/{row['tortilla:offset']}_{row['tortilla:length']},/vsicurl/{file}",
+            axis=1,
+        )
+        container.append(dataframe)
+
+    return pd.concat(container, ignore_index=True)
+
+
+def remote_lazyfile2dataframe(
+    offset: int, file: Union[str, pathlib.Path]
+) -> pd.DataFrame:
+    """Read the dataframe of tortilla file that is a subfile
+    of a larger file.
+
+    Args:
+        offset (int): The offset of the subfile.
+        file (Union[str, pathlib.Path]): A local path pointing to the
+            main tortilla file.
+
+    Returns:
+        pd.DataFrame: The dataframe of the tortilla file.
+    """
+
+    # Fetch the first 8 bytes of the file
+    initb, endb = offset, offset + 50
+    headers = {"Range": f"bytes={initb}-{endb}"}
+    response: requests.Response = requests.get(file, headers=headers)
+    static_bytes: bytes = response.content
+
+    # SPLIT the static bytes
+    MB: bytes = static_bytes[:2]
+    FO: bytes = static_bytes[2:10]
+    FL: bytes = static_bytes[10:18]
+    DF: str = static_bytes[18:42].strip().decode()
+
+    # Check if the file is a tortilla
+    if MB != b"#y":
+        raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+    # Interpret the bytes as a little-endian integer
+    footer_offset: int = int.from_bytes(FO, "little") + offset
+    footer_length: int = int.from_bytes(FL, "little")
+
+    # Fetch the footer
+    headers = {"Range": f"bytes={footer_offset}-{footer_offset + footer_length - 1}"}
+    with requests.get(file, headers=headers) as response:
+        # Interpret the response as a parquet table
+        dataframe = pq.read_table(pa.BufferReader(response.content)).to_pandas()
+
+    # Fix the offset
+    dataframe["tortilla:offset"] = dataframe["tortilla:offset"] + offset
+
+    # Add the file format and mode
+    dataframe["internal:file_format"] = DF
+    dataframe["internal:mode"] = "online"
+    dataframe["internal:subfile"] = dataframe.apply(
+        lambda row: f"/vsisubfile/{row['tortilla:offset']}_{row['tortilla:length']},/vsicurl/{file}",
+        axis=1,
+    )
+
+    return dataframe
+
+
+def remote_file2metadata(file: str) -> dict:
+    """Read the metadata of a taco file given a URL. The
+        server must support HTTP Range requests.
+
+    Args:
+        file (str): A URL pointing to the taco file.
+
+    Returns:
+        dict: The metadata of the taco file.
+    """
+    # Fetch the first 8 bytes of the file
+    headers = {"Range": "bytes=0-66"}
+    response: requests.Response = requests.get(file, headers=headers)
+    static_bytes: bytes = response.content
+
+    # SPLIT the static bytes
+    MB: bytes = static_bytes[:2]
+    CO: int = int.from_bytes(static_bytes[50:58], "little")
+    CL: int = int.from_bytes(static_bytes[58:66], "little")
+
+    # Check if the file is a tortilla
+    if MB != b"#y":
+        raise ValueError("You are not a tortilla 🫓 or a TACO 🌮")
+
+    # Read the Collection (JSON UTF-8 encoded)
+    headers = {"Range": f"bytes={CO}-{CO + CL}"}
+    collection: dict = json.loads(requests.get(file, headers=headers).content.decode())
+
+    return collection
+
+
+def remote_files2metadata(files: List[str]) -> dict:
+    """Read the metadata of taco files given a set of URLs. The server
+        must support HTTP Range requests.
+
+    Args:
+        files (List[str]): A list of URLs pointing to the
+            taco files.
+
+    Returns:
+        dict: The metadata of the taco file.
+    """
+    return remote_file2metadata(files[0])

tacoreader-0.1.0/tacoreader/load_utils.py

@@ -0,0 +1,160 @@
+import pathlib
+import re
+import urllib
+from typing import List, Tuple, Union
+
+import geopandas as gpd
+import requests
+import shapely.wkt
+
+
+def is_valid_url(url: Union[str, List[str]]) -> bool:
+    """Check if a URL or list of URLs is valid.
+
+    Args:
+        url (Union[str, List[str]]): The URL(s) to check. It can
+            be a single URL or a list of URLs.
+
+    Returns:
+        bool: True if all URLs are valid, False otherwise.
+    """
+    if isinstance(url, pathlib.Path):
+        return False
+
+    if isinstance(url, list):
+        return all(is_valid_url(single_url) for single_url in url)
+
+    try:
+        result = urllib.parse.urlparse(url)
+        return all([result.scheme in ["http", "https"], result.netloc])
+    except ValueError:
+        return False
+
+
+def split_name_and_path(file: Union[str, pathlib.Path]) -> Tuple[str, str]:
+    """
+    Split a file path or URL into its name and path components.
+
+    Args:
+        file (Union[str, pathlib.Path]): The input file or URL.
+
+    Returns:
+        Tuple[str, str]: A tuple containing the name and path.
+    """
+    if isinstance(file, (str, pathlib.Path)):
+        # Handle URLs
+        parsed = urllib.parse.urlparse(file)
+        if parsed.scheme in {"http", "https"}:
+            # Extract the name and path from a URL
+            name = pathlib.PurePosixPath(parsed.path).name
+            path = f"{parsed.scheme}://{parsed.netloc}{pathlib.PurePosixPath(parsed.path).parent.as_posix()}"
+        else:
+            # Handle local file paths
+            file_path = pathlib.Path(file)
+            name = file_path.name
+            path = file_path.parent.as_posix()
+        return name, path
+    else:
+        raise ValueError("Input must be a string or pathlib.Path.")
+
+
+def snippet2files(
+    file: Union[str, pathlib.Path, List[str], List[pathlib.Path]]
+) -> Union[List[pathlib.Path], List[str], str, pathlib.Path]:
+    """Convert snippets of a multi-part file to a list of files.
+
+    Args:
+        file (Union[str, pathlib.Path, List[str], List[pathlib.Path]]): A file, a
+            list of files, or a snippet of a multi-part file.
+
+    Raises:
+        FileNotFoundError: If a file is not found.
+        FileNotFoundError: If a partial file is missing.
+
+    Returns:
+        List[pathlib.Path]: A list of files or a single path. The path can be a
+            local path or a URL.
+    """
+
+    # Check if the file is a list
+    if isinstance(file, list):
+        files = file
+    else:
+        # Does the file finish with: *.tortilla?
+        if re.match(r".*\*\.tortilla$", str(file)):
+
+            # Split in name and path
+            name, path = split_name_and_path(file)
+
+            # Get the filename without the snippet (i.e., *.tortilla)
+            filename: str = re.sub(r"\*\.tortilla$", "", name)
+
+            # check if file is a url
+            if is_valid_url(path):
+                # It is expected that the file is a multi-part file in the same url path
+                dumbfile: str = f"{path}/{filename}.0000.part.tortilla"
+                headers = {"Range": "bytes=42-50"}
+                response: requests.Response = requests.get(dumbfile, headers=headers)
+                npartitions: int = int.from_bytes(response.content, "little")
+
+                # Check if all parts are there
+                files = []
+                for d in range(npartitions):
+                    partial_file = f"{path}/{filename}.{str(d).zfill(4)}.part.tortilla"
+                    files.append(partial_file)
+            else:
+                # Get all files in the directory
+                file = pathlib.Path(file)
+                filename = pathlib.Path(filename)
+
+                # It is expected that the file is a multi-part file in the same directory
+                dumbfile: pathlib.Path = file.resolve().parent / (
+                    filename.stem + ".0000.part.tortilla"
+                )
+
+                # check how many parts are there
+                with open(dumbfile, "rb") as f:
+                    f.seek(42)
+                    npartitions: int = int.from_bytes(f.read(8), "little")
+
+                # Check if all parts are there
+                files = []
+                for d in range(npartitions):
+                    partial_file = filename.with_suffix(
+                        f".{str(d).zfill(4)}.part.tortilla"
+                    )
+                    if partial_file.exists():
+                        files.append(partial_file)  # Add the file to the list
+                    else:
+                        raise FileNotFoundError(f"Missing partial file: {partial_file}")
+
+        else:
+            files = file
+
+    return files
+
+
+def sort_columns_add_geometry(metadata):
+    """Sort the columns of a metadata DataFrame.
+    Also, convert the "stac:centroid" column to a geometry column.
+
+    Args:
+        metadata (pd.DataFrame): The metadata DataFrame.
+
+    Returns:
+        pd.DataFrame: The metadata DataFrame with sorted columns.
+    """
+    if "stac:centroid" in metadata.columns:
+        metadata = gpd.GeoDataFrame(
+            data=metadata,
+            geometry=metadata["stac:centroid"].apply(shapely.wkt.loads),
+            crs="EPSG:4326",
+        )
+    columns = metadata.columns
+    prefixes = ["internal:", "tortilla:", "stac:", "rai:"]
+    sorted_columns = [
+        col for prefix in prefixes for col in columns if col.startswith(prefix)
+    ]
+    rest = [col for col in columns if col not in sorted_columns and col != "geometry"]
+    columns = sorted_columns + rest + (["geometry"] if "geometry" in columns else [])
+    return metadata[columns]

tacoreader-0.1.0/tacoreader/stac.py

@@ -0,0 +1,2 @@
+def read_stac():
+    print("Do not implement it yet")