gamslib 0.2.6__tar.gz

gamslib-0.2.6/.gitignore

@@ -0,0 +1,19 @@
+# python generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# venv
+.venv
+.coverage
+.python-version
+requirements-dev.lock
+requirements.lock
+.vscode/settings.json
+
+
+
+foo.py

gamslib-0.2.6/.gitlab-ci.yml

@@ -0,0 +1,50 @@
+variables:
+    RYE_VERSION: "0.42"
+    RYE_IMAGE: "jfxs/rye"
+
+stages:
+  - test
+  - lint
+
+# The Docker image (always latest or use the variables?)
+image: jfxs/rye
+
+# A hidden job template that can be used by different Python versions
+.test-template:
+  stage: test
+  script:
+    - echo "Rye environment version"
+    - rye --version
+    - echo "Pinning ${PYTHON_VERSION}"
+    - rye pin ${PYTHON_VERSION}
+    - rye sync
+    - echo "Using python version:"
+    - python --version
+    - echo "Running tests with Rye"
+    - rye test
+
+# Specific jobs for Python versions
+test-py_3.11:
+  extends: .test-template
+  variables:
+    PYTHON_VERSION: "3.11"
+
+test-py_3.12:
+  extends: .test-template
+  variables:
+    PYTHON_VERSION: "3.12"
+
+# test-py_3.13:
+#   extends: .test-template
+#   variables:
+#     PYTHON_VERSION: "3.13"
+
+lint:
+  stage: lint
+  script:
+    - echo "Rye environment version"
+    - rye --version
+    - echo "Using python version:"
+    - python --version
+    - echo "Running rye lint"
+    - rye lint

gamslib-0.2.6/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
+## [Unreleased]
+
+### Added
+
+### Changed
+
+### Removed
+
+### Fixed
+
+## [0.2.6] - 2024-12-06
+
+### Added
+
+- CHANGELOG.md
+- More tests
+
+### Changed
+
+- Extend pyproject.toml
+  

gamslib-0.2.6/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Fabio Tosques, Gunter Vasold
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gamslib-0.2.6/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.3
+Name: gamslib
+Version: 0.2.6
+Summary: Modules and subpackages used in various GAMS5 related projects
+Project-URL: Homepage, https://github.com/DHGraz/gamslib
+Project-URL: Repository, https://github.com/DHGraz/gamslib
+Project-URL: Issues, https://github.com/DHGraz/gamslib/issues
+Project-URL: Changelog, https://github.com/DHGraz/gamslib/blob/main/CHANGELOG.md
+Author-email: Gunter Vasold <gunter.vasold@uni-graz.at>, Fabio Tosques <fabio.tosques@uni-graz.at>
+Keywords: GAMS
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Requires-Dist: pydantic>=2.10.1
+Requires-Dist: pylightxl>=1.61
+Description-Content-Type: text/markdown
+
+# gamslib
+
+gamslib is a collection of GAMS related modules and packages, which are used in multiple other packages.
+
+
+gamslib is available on pypi.org and can be installed via pip:
+
+```
+pip install gamslib
+```
+
+Currently these subpackages are available (more to come):
+
+## objectcsv
+
+Handle object and datastream metadata in object csv files.
+
+When creating bags for GAMS, we provide some metadata in csv
+files (which are not part of the bag, btw).
+
+The objectcsv package provides tools to handle this metadata.
+
+  * The ObjectCSV class represents the object    
+    and datastream csv data for a single object. It is created by providing the 
+    path to the  object directory. 
+  * The manage_csv module can be used to collect csv data from all objects 
+    into a single file, which makes editing the data more efficient.
+    It also has a function to update the csv files in the object directories 
+    based on the collected data.
+  * The xlsx module can be used to convert the csv files to xlsx files 
+    and vice versa. This is useful for editing the data in a spreadsheet 
+    without the hassles of importing and exporting the csv files, which 
+    led to encoding problems in the past.
+
+## projectconfiguration
+
+This package contains a central class `Configuration` that represents the 
+project configuration. To create this object, the function 
+`load_configuration(OBJECT_ROOT, [PATH_TO_TOML_FILE])` should be used.
+
+The function tries to find the project configuration file, validates 
+its content, and creates the central Configuration object with all 
+sub-objects (Each TOML inline table is provided as its own sub-object). 
+These sub-objects are currently:
+
+  * general
+  * metadata
+
+A basic configuration file can be generated via the `create_condiguration()` function.

gamslib-0.2.6/README.md

@@ -0,0 +1,49 @@
+# gamslib
+
+gamslib is a collection of GAMS related modules and packages, which are used in multiple other packages.
+
+
+gamslib is available on pypi.org and can be installed via pip:
+
+```
+pip install gamslib
+```
+
+Currently these subpackages are available (more to come):
+
+## objectcsv
+
+Handle object and datastream metadata in object csv files.
+
+When creating bags for GAMS, we provide some metadata in csv
+files (which are not part of the bag, btw).
+
+The objectcsv package provides tools to handle this metadata.
+
+  * The ObjectCSV class represents the object    
+    and datastream csv data for a single object. It is created by providing the 
+    path to the  object directory. 
+  * The manage_csv module can be used to collect csv data from all objects 
+    into a single file, which makes editing the data more efficient.
+    It also has a function to update the csv files in the object directories 
+    based on the collected data.
+  * The xlsx module can be used to convert the csv files to xlsx files 
+    and vice versa. This is useful for editing the data in a spreadsheet 
+    without the hassles of importing and exporting the csv files, which 
+    led to encoding problems in the past.
+
+## projectconfiguration
+
+This package contains a central class `Configuration` that represents the 
+project configuration. To create this object, the function 
+`load_configuration(OBJECT_ROOT, [PATH_TO_TOML_FILE])` should be used.
+
+The function tries to find the project configuration file, validates 
+its content, and creates the central Configuration object with all 
+sub-objects (Each TOML inline table is provided as its own sub-object). 
+These sub-objects are currently:
+
+  * general
+  * metadata
+
+A basic configuration file can be generated via the `create_condiguration()` function.

gamslib-0.2.6/coverage.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+pytest --cov --cov-report term-missing --cov=gamslib tests

gamslib-0.2.6/pyproject.toml

@@ -0,0 +1,53 @@
+[project]
+name = "gamslib"
+version = "0.2.6"
+description = "Modules and subpackages used in various GAMS5 related projects"
+authors = [
+    { name = "Gunter Vasold", email = "gunter.vasold@uni-graz.at" },
+    { name = "Fabio Tosques", email = "fabio.tosques@uni-graz.at" }
+]
+dependencies = [
+    "pylightxl>=1.61",
+    "pydantic>=2.10.1",
+]
+readme = "README.md"
+requires-python = ">= 3.11"
+keywords = ["GAMS"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Topic :: Software Development :: Libraries",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+
+[project.urls]
+Homepage = "https://github.com/DHGraz/gamslib"
+# Documentation = "https://readthedocs.org"
+Repository = "https://github.com/DHGraz/gamslib"
+Issues = "https://github.com/DHGraz/gamslib/issues"
+Changelog = "https://github.com/DHGraz/gamslib/blob/main/CHANGELOG.md"
+
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.rye]
+managed = true
+dev-dependencies = [
+    "pylint>=3.3.1",
+    "pytest>=8.3.3",
+    "pytest-datadir>=1.5.0",
+    "pytest-cov>=6.0.0",
+    "pyright>=1.1.390",
+    "toml>=0.10.2",
+]
+
+[tool.hatch.metadata]
+allow-direct-references = true
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/gamslib"]

gamslib-0.2.6/src/gamslib/__init__.py

@@ -0,0 +1,6 @@
+"""A library with modules which a needed in other GAMS packages.
+
+The gamslib package contains modules which are needed in other GAMS packages.
+
+  * The objectcsv module provides tools to handle object and datastream metadata
+"""

gamslib-0.2.6/src/gamslib/objectcsv/__init__.py

@@ -0,0 +1,57 @@
+"""Handle object and datastream metadata in csv files.
+
+When creating bags for GAMS, we provide some metadata in csv
+files (which are not part of the bag, btw).
+
+The objectcsv package provides tools to handle this metadata.
+
+  * The ObjectCSV class represents the object and datastream csv data
+    for a single object. It is created by providing the path to the
+    object directory. It is composed of two classes:
+
+    * ObjectCSVFile represents the object metadata. It hold typically
+      a single ObjectData object, but can hold multiple objects if needed.
+    * DatastreamsCSVFile represents the datastream metadata. It holds
+        typically multiple DSData objects, one for each datastream.
+  * The dublincore_csv module represents the object metadata stored in
+    the objects 'DC.xml' file. It provides useful functions for acessing
+    DC data e.g. for prefered languages etc.
+  * The create_csv module can be used to initally create the csv files for
+    all objects
+  * The manage_csv module can be used collect csv data from all objects
+    into a single file, which makes editing the data more efficient.
+    It also has a function to update the csv files in the object directories
+    based on the collected data.
+  * The xlsx module can be used to convert the csv files to xlsx files
+    and vice versa. This is useful for editing the data in a spreadsheet
+    without the hassles of importing and exporting the csv files, which
+    led to encoding problems in the past.
+
+The "public" functions and classes from the submodules are directly
+available in the objectcsv:
+
+  * ObjectCSV
+  * ObjectData
+  * DSData
+  * create_csv_files
+  * collect_csv_data
+  * update_csv_files
+  * csv_to_xlsx
+  * xlsx_to_csv
+"""
+
+from .objectcsv import ObjectCSV, ObjectData, DSData
+from .create_csv import create_csv_files
+from .manage_csv import collect_csv_data, update_csv_files
+from .xlsx import csv_to_xlsx, xlsx_to_csv
+
+__all__ = [
+    "ObjectCSV",
+    "ObjectData",
+    "DSData",
+    "create_csv_files",
+    "collect_csv_data",
+    "update_csv_files",
+    "csv_to_xlsx",
+    "xlsx_to_csv",
+]

gamslib-0.2.6/src/gamslib/objectcsv/create_csv.py

@@ -0,0 +1,169 @@
+"""Create object.csv and datastreams.csv files.
+
+This module creates the object.csv and datastreams.csv files for one or many given
+object folder. It uses data from the DC.xml file and the project configuration 
+to fill in the metadata. When not enough information is available, some fields 
+will be left blank or filled with default values.
+"""
+
+import logging
+import mimetypes
+import re
+from pathlib import Path
+
+from gamslib.projectconfiguration import Configuration
+
+from .objectcsv import DSData, ObjectCSV, ObjectData
+from .dublincore import DublinCore
+from .utils import find_object_folders
+
+logger = logging.getLogger()
+
+DEFAULT_RIGHTS = (
+    "Creative Commons Attribution-NonCommercial 4.0 "
+    "(https://creativecommons.org/licenses/by-nc/4.0/)"
+)
+DEFAULT_SOURCE = "local"
+DEFAULT_OBJECT_TYPE = "text"
+
+NAMESPACES = {
+    "dc": "http://purl.org/dc/elements/1.1/",
+}
+
+
+def get_rights(config: Configuration, dc: DublinCore) -> str:
+    """Get the rights from various sources.
+
+    Lookup in this ortder:
+
+      1. Check if set in dublin core
+      2. Check if set in the configuration
+      3. Use a default value.
+    """
+    rights = dc.get_element_as_str("rights", default="")
+    if not rights:  # empty string is a valid value
+        if config.metadata.rights:
+            rights = config.metadata.rights
+        else:
+            rights = DEFAULT_RIGHTS
+    return rights
+
+
+def extract_dsid(datastream: Path | str, keep_extension=True) -> str:
+    """Extract and validate the datastream id from a datastream path.
+
+    If remove_extension is True, the file extension is removed from the PID.
+    """
+    if isinstance(datastream, str):
+        datastream = Path(datastream)
+
+    pid = datastream.name
+
+    if not keep_extension:
+        # not everything after the last dot is an extension :-(
+        mtype = mimetypes.guess_type(datastream)[0]
+        if mtype is None:
+            known_extensions = []
+        else:
+            known_extensions = mimetypes.guess_all_extensions(mtype)
+        if datastream.suffix in known_extensions:
+            pid = pid.removesuffix(datastream.suffix)
+            logger.debug("Removed extension '%s' for ID: %s", datastream.suffix, pid)
+        else:
+            parts = pid.split(".")
+            if re.match(r"^[a-zA-Z]+\w?$", parts[-1]):
+                pid = ".".join(parts[:-1])
+                logger.debug("Removed extension for ID: %s", parts[0])
+            else:
+                logger.warning(
+                    "'%s' does not look like an extension. Keeping it in PID.", pid[-1]
+                )
+
+    if re.match(r"^[a-zA-Z0-9]+[-.%_a-zA-Z0-9]+[a-zA-Z0-9]+$", pid) is None:
+        raise ValueError(f"Invalid PID: '{pid}'")
+
+    logger.debug(
+        "Extracted PID: %s from %s (keep_extension=%s)", pid, datastream, keep_extension
+    )
+    return pid
+
+
+def collect_object_data(pid: str, config: Configuration, dc: DublinCore) -> ObjectData:
+    """Find data for the object.csv by examining dc file and configuration.
+
+    This is the place to change the resolving order for data from other sources.
+    """
+
+    title = "; ".join(dc.get_element("title", default=pid))
+    description = "; ".join(dc.get_element("description", default=""))
+
+    return ObjectData(
+        recid=pid,
+        title=title,
+        project=config.metadata.project_id,
+        description=description,
+        creator=config.metadata.creator,
+        rights=get_rights(config, dc),
+        source=DEFAULT_SOURCE,
+        objectType=DEFAULT_OBJECT_TYPE,
+    )
+
+
+def collect_datastream_data(
+    ds_file: Path, config: Configuration, dc: DublinCore
+) -> DSData:
+    """Collect data for a single datastream."""
+    dsid = extract_dsid(ds_file, config.general.dsid_keep_extension)
+
+    # I think it's not possible to derive a ds title or description from the DC file
+    # title = "; ".join(dc.get_element("title", default=dsid)) # ??
+    # description = "; ".join(dc.get_element("description", default="")) #??
+
+    return DSData(
+        dspath=str(ds_file.relative_to(ds_file.parents[1])),  # objectsdir
+        dsid=dsid,
+        title="",
+        description="",
+        mimetype=mimetypes.guess_type(ds_file)[0] or "",
+        creator=config.metadata.creator,
+        rights=get_rights(config, dc),
+    )
+
+
+def create_csv(
+    object_directory: Path, configuration: Configuration
+) -> ObjectCSV | None:
+    """Generate the csv file containing the preliminary metadata for a single object."""
+    objectcsv = ObjectCSV(object_directory)
+
+    # Avoid that existing (and potentially already edited) metadata is replaced
+    if not objectcsv.is_new():
+        logger.info(
+            "CSV files for object '%s' already exist. Will not be re-created.",
+            objectcsv.object_id,
+        )
+        return None
+
+    dc = DublinCore(object_directory / "DC.xml")
+    objectcsv.add_objectdata(
+        collect_object_data(objectcsv.object_id, configuration, dc)
+    )
+    for ds_file in object_directory.glob("*"):
+        if ds_file.is_file() and ds_file.name not in ("object.csv", "datastreams.csv"):
+            objectcsv.add_datastream(
+                # collect_datastream_data(ds_file, objectcsv.object_id, configuration, dc)
+                collect_datastream_data(ds_file, configuration, dc)
+            )
+    objectcsv.write()
+    return objectcsv
+
+
+def create_csv_files(root_folder: Path, config: Configuration) -> list[ObjectCSV]:
+    """Create the CSV files for all objects below root_folder."""
+    extended_objects: list[ObjectCSV] = []
+    for path in find_object_folders(root_folder):
+        extended_obj = create_csv(path, config)
+
+        if extended_obj is not None:
+            extended_objects.append(extended_obj)
+    return extended_objects