dsp-tools 8.2.0.post10__tar.gz → 18.1.0.post5__tar.gz

dsp_tools-18.1.0.post5/PKG-INFO

@@ -0,0 +1,90 @@
+Metadata-Version: 2.3
+Name: dsp-tools
+Version: 18.1.0.post5
+Summary: DSP-TOOLS is a Python package with a command line interface that helps you interact with a DaSCH service platform (DSP) server.
+Author: DaSCH - Swiss National Data and Service Center for the Humanities
+Author-email: DaSCH - Swiss National Data and Service Center for the Humanities <info@dasch.swiss>
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
+Classifier: Operating System :: OS Independent
+Requires-Dist: argparse
+Requires-Dist: dotenv
+Requires-Dist: jinja2
+Requires-Dist: jsonpath-ng
+Requires-Dist: jsonschema
+Requires-Dist: loguru
+Requires-Dist: lxml
+Requires-Dist: networkx
+Requires-Dist: openpyxl
+Requires-Dist: packaging
+Requires-Dist: pandas[excel]
+Requires-Dist: polars
+Requires-Dist: pyld
+Requires-Dist: pyyaml
+Requires-Dist: rdflib
+Requires-Dist: regex
+Requires-Dist: requests
+Requires-Dist: rustworkx
+Requires-Dist: tqdm
+Requires-Dist: typing-extensions
+Requires-Python: >=3.12
+Project-URL: Documentation, https://docs.dasch.swiss/latest/DSP-TOOLS/
+Project-URL: Homepage, https://www.dasch.swiss/
+Project-URL: Repository, https://github.com/dasch-swiss/dsp-tools.git
+Description-Content-Type: text/markdown
+
+[![DSP-TOOLS version on PyPI](https://img.shields.io/pypi/v/dsp-tools.svg)](https://pypi.org/project/dsp-tools/) 
+[![Supported Python versions](https://img.shields.io/pypi/pyversions/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
+[![License](https://img.shields.io/pypi/l/dsp-tools.svg)](https://pypi.org/project/dsp-tools/) 
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) 
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![mypy](https://img.shields.io/badge/mypy-blue)](https://github.com/python/mypy) 
+
+# DSP-TOOLS Documentation
+
+## Installing `dsp-tools`
+
+To install the latest version, run:
+
+```bash
+pip3 install dsp-tools
+```
+
+To update to the latest version run:
+
+```bash
+pip3 install --upgrade dsp-tools
+```
+
+> 🚨 If your Python version is older than ours,
+> pip will silently install an outdated version of DSP-TOOLS.  
+> DSP-TOOLS requires one of these Python versions: 
+> [![](https://img.shields.io/pypi/pyversions/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)  
+> The most recent version of DSP-TOOLS is 
+> [![](https://img.shields.io/pypi/v/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
+
+
+
+The `dsp-tools` package provides you with functionalities in the command line 
+to interact with the [DSP-API](https://github.com/dasch-swiss/dsp-api), both remote and locally.
+Additionally, it contains the `xmllib` which helps you construct the XML file required for a mass upload.
+
+
+## Where To Start?
+
+`dsp-tools` provides you with the following core functionalities.
+
+- **Running a Local Stack:** If you want to run your own DSP stack locally, take a look [here](./local-stack.md).
+- **Data Modelling:** There are several ways to create a data model with `dsp-tools`
+    - Take a look at the technical specification for the [JSON file](./data-model/json-project/overview.md).
+    - Or take a look at our tool to [convert Excel files into the JSON format](./data-model/excel2json.md).
+    - You can create a data model on the DSP-APP. To re-use that data model on another server 
+      you can use the CLI command described [here](./data-model/data-model-cli.md#get).
+- **Data for Mass-Upload:**
+    - If you want to create the XML file required for a mass-upload onto DSP, take a look at the [`xmllib`](./xmllib-docs/overview.md).
+    - You can find an in-depth explanation of our XML file format [here](./data-file/xml-data-file.md).
+      Please note, that we recommend to use the `xmllib` library to create the file 
+      as we will ensure interoperability between the DSP-API requirements and your input.
+    - If you want to validate and upload your XML file take a look [here](./data-file/data-file-commands.md).
+      Please note, that only DaSCH employees are permitted to upload data on a production server.

dsp_tools-18.1.0.post5/docs/index.md

@@ -0,0 +1,54 @@
+[![DSP-TOOLS version on PyPI](https://img.shields.io/pypi/v/dsp-tools.svg)](https://pypi.org/project/dsp-tools/) 
+[![Supported Python versions](https://img.shields.io/pypi/pyversions/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
+[![License](https://img.shields.io/pypi/l/dsp-tools.svg)](https://pypi.org/project/dsp-tools/) 
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) 
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![mypy](https://img.shields.io/badge/mypy-blue)](https://github.com/python/mypy) 
+
+# DSP-TOOLS Documentation
+
+## Installing `dsp-tools`
+
+To install the latest version, run:
+
+```bash
+pip3 install dsp-tools
+```
+
+To update to the latest version run:
+
+```bash
+pip3 install --upgrade dsp-tools
+```
+
+> 🚨 If your Python version is older than ours,
+> pip will silently install an outdated version of DSP-TOOLS.  
+> DSP-TOOLS requires one of these Python versions: 
+> [![](https://img.shields.io/pypi/pyversions/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)  
+> The most recent version of DSP-TOOLS is 
+> [![](https://img.shields.io/pypi/v/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
+
+
+
+The `dsp-tools` package provides you with functionalities in the command line 
+to interact with the [DSP-API](https://github.com/dasch-swiss/dsp-api), both remote and locally.
+Additionally, it contains the `xmllib` which helps you construct the XML file required for a mass upload.
+
+
+## Where To Start?
+
+`dsp-tools` provides you with the following core functionalities.
+
+- **Running a Local Stack:** If you want to run your own DSP stack locally, take a look [here](./local-stack.md).
+- **Data Modelling:** There are several ways to create a data model with `dsp-tools`
+    - Take a look at the technical specification for the [JSON file](./data-model/json-project/overview.md).
+    - Or take a look at our tool to [convert Excel files into the JSON format](./data-model/excel2json.md).
+    - You can create a data model on the DSP-APP. To re-use that data model on another server 
+      you can use the CLI command described [here](./data-model/data-model-cli.md#get).
+- **Data for Mass-Upload:**
+    - If you want to create the XML file required for a mass-upload onto DSP, take a look at the [`xmllib`](./xmllib-docs/overview.md).
+    - You can find an in-depth explanation of our XML file format [here](./data-file/xml-data-file.md).
+      Please note, that we recommend to use the `xmllib` library to create the file 
+      as we will ensure interoperability between the DSP-API requirements and your input.
+    - If you want to validate and upload your XML file take a look [here](./data-file/data-file-commands.md).
+      Please note, that only DaSCH employees are permitted to upload data on a production server.

dsp_tools-18.1.0.post5/pyproject.toml

@@ -0,0 +1,207 @@
+# See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml
+
+[project]
+name = "dsp-tools"
+version = "18.1.0.post5"
+description = "DSP-TOOLS is a Python package with a command line interface that helps you interact with a DaSCH service platform (DSP) server."
+authors = [
+    { name = "DaSCH - Swiss National Data and Service Center for the Humanities", email = "info@dasch.swiss" },
+]
+readme = "docs/index.md"
+classifiers = [
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+    "Operating System :: OS Independent",
+]
+requires-python = ">=3.12"
+dependencies = [
+    "argparse",
+    "dotenv",
+    "jinja2",
+    "jsonpath-ng",
+    "jsonschema",
+    "loguru",
+    "lxml",
+    "networkx",
+    "openpyxl",
+    "packaging",
+    "pandas[excel]",
+    # pandas: "excel" is an extra that contains xlrd that is necessary for reading old .xls Excel files
+    "polars",
+    "pyld",
+    "pyyaml",
+    "rdflib",
+    "regex",
+    "requests",
+    "rustworkx",
+    "tqdm",
+    "typing-extensions",
+]
+
+
+[project.urls]
+Homepage = "https://www.dasch.swiss/"
+Documentation = "https://docs.dasch.swiss/latest/DSP-TOOLS/"
+Repository = "https://github.com/dasch-swiss/dsp-tools.git"
+
+
+[project.scripts]
+dsp-tools = "dsp_tools.cli.entry_point:main" # definition of the CLI entry point
+
+
+[build-system]
+# Tells “frontend” build tools (like pip, build, or uv) what “backend” build tool to use.
+requires = ["uv_build>=0.8.13"]
+build-backend = "uv_build"
+
+
+[dependency-groups]
+dev = [
+    "darglint",
+    "docker",
+    "mkdocs-include-markdown-plugin",
+    "mkdocs-material",
+    "mkdocs",
+    "mkdocstrings[python]",
+    "mypy",
+    "pandas-stubs",
+    "pre-commit",
+    "pytest-cov",
+    "pytest-unordered",
+    "pytest-xdist",
+    "pytest",
+    "requests-mock",
+    "ruff",
+    "testcontainers",
+    "tomlkit",
+    "types-docker",
+    "types-jsonschema",
+    "types-lxml",
+    "types-networkx",
+    "types-openpyxl",
+    "types-pyyaml",
+    "types-regex",
+    "types-requests",
+    "types-tqdm",
+    "viztracer",
+    "vulture",
+    "yamllint",
+    "pytest-order",
+]
+
+
+[tool.pytest.ini_options]
+addopts = ["--import-mode=importlib"]
+# see https://docs.pytest.org/en/latest/explanation/goodpractices.html#tests-outside-application-code
+pythonpath = [".", "src", "test"]
+
+
+[tool.mypy]
+show_column_numbers = true
+strict = true
+enable_error_code = ["possibly-undefined"]
+exclude = [
+    "src/dsp_tools/legacy_models/datetimestamp.py",
+    "src/dsp_tools/legacy_models/langstring.py",
+    "src/dsp_tools/legacy_models/projectContext.py",
+    "src/dsp_tools/commands/project/legacy_models/context.py",
+    "src/dsp_tools/commands/project/legacy_models/group.py",
+    "src/dsp_tools/commands/project/legacy_models/helpers.py",
+    "src/dsp_tools/commands/project/legacy_models/listnode.py",
+    "src/dsp_tools/commands/project/legacy_models/ontology.py",
+    "src/dsp_tools/commands/project/legacy_models/project.py",
+    "src/dsp_tools/commands/project/legacy_models/propertyclass.py",
+    "src/dsp_tools/commands/project/legacy_models/resourceclass.py",
+    "src/dsp_tools/commands/project/legacy_models/user.py",
+]
+
+
+[[tool.mypy.overrides]]
+# For these packages, no type stubs are available yet:
+module = ["jsonpath_ng.*", "viztracer.*", "pyld.*", "testcontainers.*"]
+ignore_missing_imports = true
+
+
+[tool.ruff]
+line-length = 120
+
+
+[tool.ruff.lint]
+select = [
+    "RUF",   # ruff-specific rules
+    "PL",    # pylint
+    "I",     # isort
+    "E",     # pycodestyle errors
+    "F",     # pyflakes
+    "A",     # flake8-builtins
+    "BLE",   # flake8-blind-except
+    "ARG",   # flake8-unused-arguments
+    "S",     # flake8-bandit plugin which checks for security issues
+    "YTT",   # flake8-2020 plugin, which checks for misuse of `sys.version` or `sys.version_info`
+    "ASYNC", # flake8-async plugin, which checks for bad async / asyncio practices
+    "ISC",   # flake8-implicit-str-concat plugin, which checks for problematic string concatenation
+    "INP",   # flake8-no-pep420 plugin, which checks for missing __init__.py files
+    "PIE",   # flake8-pie plugin, which does miscellaneous checks
+    "PT",    # flake8-pytest-style plugin
+    "TID",   # flake8-tidy-imports plugin
+    "ICN",   # flake8-import-conventions plugin, which checks for import conventions
+    "PGH",   # pygrep-hooks: A collection of fast, cheap, regex based pre-commit hooks
+    "UP",    # pyupgrade
+    "D101",  # pydocstyle: undocumented-public-class
+    "D102",  # pydocstyle: undocumented-public-method
+    "D103",  # pydocstyle: undocumented-public-function
+    "D417",  # pydocstyle: undocumented-param
+    "B023",  # flake8-bugbear: function-uses-loop-variable
+    "FIX",   # flake8-fixme: checks for FIXME, TODO, XXX, etc.
+]
+ignore = [
+    "ISC001", # flake8-implicit-str-concat: single-line-implicit-string-concatenation  # incompatible with the formatter
+    "S105",   # flake8-bandit: hardcoded password
+    "S106",   # flake8-bandit: hardcoded password
+    "S603",   # flake8-bandit: subprocess-without-shell-equals-true
+    "UP007",  # forbid use of typing.Union and typing.Optional
+    "UP015",  # Unnecessary mode="r" argument in open()
+    "UP045",  # Forced use of "|" instead of typing.Optional and typing.Union
+]
+
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = [
+    "F401",    # unused-import
+    "F403",    # undefined-local-with-import-star
+    "PLC0414", # useless-import-alias
+]
+"testdata/**" = [
+    "INP001", # implicit-namespace-package    # there are some python files, but no __init__.py
+]
+"test/*" = [
+    "D103", # pydocstyle: undocumented-public-function
+    "D102", # pydocstyle: undocumented-public-method
+    "D101", # pydocstyle: undocumented-public-class
+    "S101", # flake8-bandit: use of assert
+]
+"!src/dsp_tools/xmllib/**.py" = ["D103"]  # Ignore everywhere except for the `src/dsp_tools/xmllib/` directory.
+"src/dsp_tools/legacy_models/**" = [
+    "PLW"
+]
+
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+
+[tool.ruff.lint.isort]
+force-single-line = true
+
+
+[tool.vulture]
+paths = ["src/dsp_tools", ".vulture_whitelist.py"]
+exclude = [
+    "src/dsp_tools/xmllib/",
+    "src/dsp_tools/error/xmllib_warnings.py",
+    "src/dsp_tools/error/xmllib_warnings_util.py",
+    "src/dsp_tools/commands/excel2xml/",
+    "test/",
+]
+sort_by_size = true  # sort unused classes and functions by size 

dsp_tools-18.1.0.post5/src/dsp_tools/__init__.py

@@ -0,0 +1,5 @@
+from dsp_tools import xmllib
+from dsp_tools.commands import excel2xml
+from dsp_tools.config.warnings_config import initialize_warnings
+
+initialize_warnings()

dsp_tools-18.1.0.post5/src/dsp_tools/cli/args.py

@@ -0,0 +1,47 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from dataclasses import field
+from enum import Enum
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class ServerCredentials:
+    """Contains credentials to connect to a server."""
+
+    user: str
+    password: str
+    server: str
+    dsp_ingest_url: str = "http://0.0.0.0:3340"
+
+
+@dataclass(frozen=True)
+class ValidateDataConfig:
+    """Contains the configuration for validate data."""
+
+    xml_file: Path
+    save_graph_dir: Path | None
+    severity: ValidationSeverity
+    ignore_duplicate_files_warning: bool
+    is_on_prod_server: bool
+    skip_ontology_validation: bool
+    do_not_request_resource_metadata_from_db: bool
+
+
+class ValidationSeverity(Enum):
+    ERROR = 3
+    WARNING = 2
+    INFO = 1
+
+
+@dataclass
+class PathDependencies:
+    required_files: list[Path] = field(default_factory=list)
+    required_directories: list[Path] = field(default_factory=list)
+
+
+@dataclass
+class NetworkRequirements:
+    api_url: str
+    always_requires_docker: bool = False

dsp_tools-18.1.0.post5/src/dsp_tools/cli/call_action.py

@@ -0,0 +1,82 @@
+import argparse
+
+from loguru import logger
+
+from dsp_tools.cli.call_action_files_only import call_excel2json
+from dsp_tools.cli.call_action_files_only import call_excel2lists
+from dsp_tools.cli.call_action_files_only import call_excel2properties
+from dsp_tools.cli.call_action_files_only import call_excel2resources
+from dsp_tools.cli.call_action_files_only import call_id2iri
+from dsp_tools.cli.call_action_files_only import call_old_excel2json
+from dsp_tools.cli.call_action_files_only import call_old_excel2lists
+from dsp_tools.cli.call_action_with_network import call_create
+from dsp_tools.cli.call_action_with_network import call_get
+from dsp_tools.cli.call_action_with_network import call_ingest_files
+from dsp_tools.cli.call_action_with_network import call_ingest_xmlupload
+from dsp_tools.cli.call_action_with_network import call_resume_xmlupload
+from dsp_tools.cli.call_action_with_network import call_start_stack
+from dsp_tools.cli.call_action_with_network import call_stop_stack
+from dsp_tools.cli.call_action_with_network import call_upload_files
+from dsp_tools.cli.call_action_with_network import call_validate_data
+from dsp_tools.cli.call_action_with_network import call_xmlupload
+
+
+def call_requested_action(args: argparse.Namespace) -> bool:  # noqa: PLR0912 (too many branches)
+    """
+    Call the appropriate function of DSP-TOOLS.
+
+    Args:
+        args: parsed CLI arguments
+
+    Raises:
+        BaseError: from the called function
+        InputError: from the called function
+        DockerNotReachableError: from the called function
+        LocalDspApiNotReachableError: from the called function
+        unexpected errors from the called methods and underlying libraries
+
+    Returns:
+        success status
+    """
+    match args.action:
+        # commands with Docker / API interactions
+        case "start-stack":
+            result = call_start_stack(args)
+        case "stop-stack":
+            result = call_stop_stack()
+        case "create":
+            result = call_create(args)
+        case "get":
+            result = call_get(args)
+        case "validate-data":
+            result = call_validate_data(args)
+        case "xmlupload":
+            result = call_xmlupload(args)
+        case "resume-xmlupload":
+            result = call_resume_xmlupload(args)
+        case "upload-files":
+            result = call_upload_files(args)
+        case "ingest-files":
+            result = call_ingest_files(args)
+        case "ingest-xmlupload":
+            result = call_ingest_xmlupload(args)
+        # commands that do not require docker
+        case "excel2json":
+            result = call_excel2json(args)
+        case "old-excel2json":
+            result = call_old_excel2json(args)
+        case "excel2lists":
+            result = call_excel2lists(args)
+        case "old-excel2lists":
+            result = call_old_excel2lists(args)
+        case "excel2resources":
+            result = call_excel2resources(args)
+        case "excel2properties":
+            result = call_excel2properties(args)
+        case "id2iri":
+            result = call_id2iri(args)
+        case _:
+            print(f"ERROR: Unknown action '{args.action}'")
+            logger.error(f"Unknown action '{args.action}'")
+            result = False
+    return result

dsp_tools-18.1.0.post5/src/dsp_tools/cli/call_action_files_only.py

@@ -0,0 +1,74 @@
+import argparse
+from pathlib import Path
+
+from dsp_tools.cli.args import PathDependencies
+from dsp_tools.cli.utils import check_path_dependencies
+from dsp_tools.commands.excel2json.lists.make_lists import excel2lists
+from dsp_tools.commands.excel2json.old_lists import old_excel2lists
+from dsp_tools.commands.excel2json.project import excel2json
+from dsp_tools.commands.excel2json.project import old_excel2json
+from dsp_tools.commands.excel2json.properties import excel2properties
+from dsp_tools.commands.excel2json.resources import excel2resources
+from dsp_tools.commands.id2iri import id2iri
+
+
+def call_id2iri(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies([Path(args.xmlfile), Path(args.mapping)]))
+    return id2iri(
+        xml_file=args.xmlfile,
+        json_file=args.mapping,
+        remove_resource_if_id_in_mapping=args.remove_resources,
+    )
+
+
+def call_excel2properties(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies([Path(args.excelfile)]))
+    _, _, success = excel2properties(
+        excelfile=args.excelfile,
+        path_to_output_file=args.properties_section,
+    )
+    return success
+
+
+def call_excel2resources(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies([Path(args.excelfile)]))
+    _, _, success = excel2resources(
+        excelfile=args.excelfile,
+        path_to_output_file=args.resources_section,
+    )
+    return success
+
+
+def call_old_excel2lists(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies(required_directories=[Path(args.excelfolder)]))
+    _, success = old_excel2lists(
+        excelfolder=args.excelfolder,
+        path_to_output_file=args.lists_section,
+        verbose=args.verbose,
+    )
+    return success
+
+
+def call_excel2lists(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies(required_directories=[Path(args.excelfolder)]))
+    _, success = excel2lists(
+        excelfolder=args.excelfolder,
+        path_to_output_file=args.lists_section,
+    )
+    return success
+
+
+def call_excel2json(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies(required_directories=[Path(args.excelfolder)]))
+    return excel2json(
+        data_model_files=args.excelfolder,
+        path_to_output_file=args.project_definition,
+    )
+
+
+def call_old_excel2json(args: argparse.Namespace) -> bool:
+    check_path_dependencies(PathDependencies(required_directories=[Path(args.excelfolder)]))
+    return old_excel2json(
+        data_model_files=args.excelfolder,
+        path_to_output_file=args.project_definition,
+    )