oceanprotocol-job-details 0.2.5__tar.gz → 0.2.7__tar.gz

oceanprotocol_job_details-0.2.7/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.4
+Name: oceanprotocol-job-details
+Version: 0.2.7
+Summary: A Python package to get details from OceanProtocol jobs
+Project-URL: Homepage, https://github.com/AgrospAI/oceanprotocol-job-details
+Project-URL: Issues, https://github.com/AgrospAI/oceanprotocol-job-details/issues
+Author-email: Agrospai <agrospai@udl.cat>, Christian López García <christian.lopez@udl.cat>
+License: Copyright 2025 Agrospai
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Requires-Dist: dataclasses-json>=0.6.7
+Requires-Dist: dependency-injector>=4.48.2
+Requires-Dist: orjson>=3.11.3
+Description-Content-Type: text/markdown
+
+A Python package to get details from OceanProtocol jobs
+
+---
+
+## Installation
+
+```
+pip install oceanprotocol-job-details
+```
+
+## Usage 
+
+As a simple library, we only need to import `JobDetails` and load it, it will:
+
+1. Fetch the needed parameters to populate the `JobDetails` instance from the environment variables or use the passed values to the `load()` method.
+1. Look for the files corresponding to the passed DIDs in the filesystem according to the [Ocean Protocol Structure](#oceanprotocol-structure) and load them into the `JobDetails` instance.
+
+
+### Minimal Example
+
+```python
+from oceanprotocol_job_details import JobDetails
+
+job_details = JobDetails.load()
+```
+
+### Custom Input Parameters
+
+If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:
+
+```python
+from dataclasses import dataclass
+from oceanprotocol_job_details import JobDetails
+
+
+@dataclass
+class InputParameters:
+    foobar: str
+
+
+job_details = JobDetails[InputParameters].load(InputParameters)
+
+# Usage
+job_details.input_parameters.foobar
+```
+
+```python
+from dataclasses import dataclass
+from oceanprotocol_job_details import JobDetails
+
+
+@dataclass
+class Foo:
+    bar: str
+
+
+@dataclass
+class InputParameters:
+    # Allows for nested types
+    foo: Foo
+
+
+job_details = JobDetails[InputParameters].load(InputParameters)
+
+# Usage
+job_details.input_parameters.foo.bar
+```
+
+The values to fill the custom `InputParameters` will be parsed from the `algoCustomData.json` located next to the input data directories. 
+
+### Iterating Input Files the clean way
+
+```python
+from oceanprotocol_job_details import JobDetails
+
+
+job_details = JobDetails.load()
+
+for idx, file_path in job_details.next_file():
+    ...
+
+# Or if you just want one file path
+_, file_path = job_details.next_file()
+```
+
+## OceanProtocol Structure
+
+```bash
+data        # Root /data directory
+├── ddos    # Contains the loaded dataset's DDO
+│   ├── 17feb...e42 # DDO file
+│   └── ... # One DDO per loaded dataset
+├── inputs  # Datasets dir
+│   ├── 17feb...e42 # Dir holding the data of its name DID, contains files named 0..X
+│   │   └── 0 # Data file
+│   └── algoCustomData.json # Custom algorithm input data
+├── logs    # Algorithm output logs dir
+└── outputs # Algorithm output files dir
+```
+
+> **_Note:_** Even though it's possible that the algorithm is passed multiple datasets, right now the implementation only allows to use **one dataset** per algorithm execution, so **normally** the executing job will only have **one ddo**, **one dir** inside inputs, and **one data file** named `0`.

oceanprotocol_job_details-0.2.7/README.md

@@ -0,0 +1,101 @@
+A Python package to get details from OceanProtocol jobs
+
+---
+
+## Installation
+
+```
+pip install oceanprotocol-job-details
+```
+
+## Usage 
+
+As a simple library, we only need to import `JobDetails` and load it, it will:
+
+1. Fetch the needed parameters to populate the `JobDetails` instance from the environment variables or use the passed values to the `load()` method.
+1. Look for the files corresponding to the passed DIDs in the filesystem according to the [Ocean Protocol Structure](#oceanprotocol-structure) and load them into the `JobDetails` instance.
+
+
+### Minimal Example
+
+```python
+from oceanprotocol_job_details import JobDetails
+
+job_details = JobDetails.load()
+```
+
+### Custom Input Parameters
+
+If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:
+
+```python
+from dataclasses import dataclass
+from oceanprotocol_job_details import JobDetails
+
+
+@dataclass
+class InputParameters:
+    foobar: str
+
+
+job_details = JobDetails[InputParameters].load(InputParameters)
+
+# Usage
+job_details.input_parameters.foobar
+```
+
+```python
+from dataclasses import dataclass
+from oceanprotocol_job_details import JobDetails
+
+
+@dataclass
+class Foo:
+    bar: str
+
+
+@dataclass
+class InputParameters:
+    # Allows for nested types
+    foo: Foo
+
+
+job_details = JobDetails[InputParameters].load(InputParameters)
+
+# Usage
+job_details.input_parameters.foo.bar
+```
+
+The values to fill the custom `InputParameters` will be parsed from the `algoCustomData.json` located next to the input data directories. 
+
+### Iterating Input Files the clean way
+
+```python
+from oceanprotocol_job_details import JobDetails
+
+
+job_details = JobDetails.load()
+
+for idx, file_path in job_details.next_file():
+    ...
+
+# Or if you just want one file path
+_, file_path = job_details.next_file()
+```
+
+## OceanProtocol Structure
+
+```bash
+data        # Root /data directory
+├── ddos    # Contains the loaded dataset's DDO
+│   ├── 17feb...e42 # DDO file
+│   └── ... # One DDO per loaded dataset
+├── inputs  # Datasets dir
+│   ├── 17feb...e42 # Dir holding the data of its name DID, contains files named 0..X
+│   │   └── 0 # Data file
+│   └── algoCustomData.json # Custom algorithm input data
+├── logs    # Algorithm output logs dir
+└── outputs # Algorithm output files dir
+```
+
+> **_Note:_** Even though it's possible that the algorithm is passed multiple datasets, right now the implementation only allows to use **one dataset** per algorithm execution, so **normally** the executing job will only have **one ddo**, **one dir** inside inputs, and **one data file** named `0`.

{oceanprotocol_job_details-0.2.5 → oceanprotocol_job_details-0.2.7}/oceanprotocol_job_details/di.py

@@ -20,6 +20,7 @@ class Container(containers.DeclarativeContainer):
         dids=config.dids,
         transformation_did=config.transformation_did,
         paths=paths,
+        logger=config.logger,
     )
 
     files = providers.Factory(

{oceanprotocol_job_details-0.2.5 → oceanprotocol_job_details-0.2.7}/oceanprotocol_job_details/loaders/impl/ddo.py

@@ -28,5 +28,5 @@ class DDOLoader:
         ddos = []
         for path in self._ddo_paths:
             with open(path, "r") as f:
-                ddos.append(DDO.from_json(f.read()))  # type: ignore
+                ddos.append(DDO.from_json(f.read()))
         return ddos

{oceanprotocol_job_details-0.2.5 → oceanprotocol_job_details-0.2.7}/oceanprotocol_job_details/loaders/impl/files.py

@@ -2,7 +2,8 @@ from __future__ import annotations
 
 import json
 from dataclasses import InitVar, dataclass, field
-from typing import TYPE_CHECKING, Sequence, final
+from logging import Logger
+from typing import TYPE_CHECKING, final
 
 from oceanprotocol_job_details.paths import Paths
 
@@ -23,7 +24,10 @@ class FilesLoader:
     paths: Paths
     """Path configurations of the project"""
 
-    _dids: Sequence[str] = field(init=False)
+    logger: Logger
+    """Logger to use"""
+
+    _dids: str = field(init=False)
     _transformation_did: str = field(init=False)
 
     def __post_init__(
@@ -31,11 +35,17 @@ class FilesLoader:
         dids: str | None,
         transformation_did: str | None,
     ) -> None:
-        assert dids, "Missing DIDs"
-        assert transformation_did, "Missing transformation DID"
+        def _load_dids(dids, logger):
+            if dids:
+                return json.loads(dids)
+
+            logger.info("Missing DIDS, Inferring DIDS from input DDOs")
+            return [f.parts[-1] for f in self.paths.ddos.iterdir()]
 
-        object.__setattr__(self, "_dids", json.loads(dids))
         object.__setattr__(self, "_transformation_did", transformation_did)
+        object.__setattr__(self, "_dids", _load_dids(dids, self.logger))
+
+        assert self._dids, "Missing input DIDs"
 
     def load(self) -> Files:
         from oceanprotocol_job_details.ocean import DIDPaths, Files

{oceanprotocol_job_details-0.2.5 → oceanprotocol_job_details-0.2.7}/oceanprotocol_job_details/ocean.py

@@ -1,9 +1,9 @@
 from __future__ import annotations
 
-import logging
 import os
 from dataclasses import dataclass, field
 from functools import cached_property
+from logging import Logger, getLogger
 from pathlib import Path
 from typing import (
     Any,
@@ -24,14 +24,7 @@ from dataclasses_json import dataclass_json
 from oceanprotocol_job_details.di import Container
 from oceanprotocol_job_details.paths import Paths
 
-T = TypeVar("T")
-
-logging.basicConfig(
-    level=logging.INFO,
-    format="%(asctime)s [%(threadName)s] [%(levelname)s]  %(message)s",
-    handlers=[logging.StreamHandler()],
-)
-logger = logging.getLogger(__name__)
+InputParemetersT = TypeVar("InputParemetersT")
 
 
 @dataclass_json
@@ -234,7 +227,7 @@ class _EmptyJobDetails: ...
 @final
 @dataclass_json
 @dataclass(frozen=True)
-class JobDetails(Generic[T]):
+class JobDetails(Generic[InputParemetersT]):
     files: Files
     """The input filepaths"""
 
@@ -245,7 +238,7 @@ class JobDetails(Generic[T]):
     """Configuration paths"""
 
     # Store the type explicitly to avoid issues
-    _type: Type[T] = field(repr=False)
+    _type: Type[InputParemetersT] = field(repr=False)
 
     secret: str | None = None
     """Shh it's a secret"""
@@ -260,8 +253,8 @@ class JobDetails(Generic[T]):
                 yield (idx, file)
 
     @cached_property
-    def input_parameters(self) -> T:
-        """Read the input parameters and return them in an instance of the dataclass T"""
+    def input_parameters(self) -> InputParemetersT:
+        """Read the input parameters and return them in an instance of the dataclass InputParemetersT"""
 
         with open(self.paths.algorithm_custom_parameters, "r") as f:
             raw = f.read().strip()
@@ -281,13 +274,14 @@ class JobDetails(Generic[T]):
     @classmethod
     def load(
         cls,
-        _type: Type[T] | None = None,
+        _type: Type[InputParemetersT] | None = None,
         *,
         base_dir: str | None = None,
         dids: str | None = None,
         transformation_did: str | None = None,
         secret: str | None = None,
-    ) -> JobDetails[T]:
+        logger: Logger | None = None,
+    ) -> JobDetails[InputParemetersT]:
         """Load a JobDetails instance that holds the runtime details.
 
         Loading it will check the following:
@@ -299,7 +293,6 @@ class JobDetails(Generic[T]):
         - DIDS: The DIDs of the inputs
         - TRANSFORMATION_DID: The DID of the transformation algorithm
         - SECRET (optional): A really secret secret
-
         """
 
         if _type is None:
@@ -309,10 +302,11 @@ class JobDetails(Generic[T]):
         container.config.from_dict(
             {
                 "base_dir": base_dir or os.environ.get("BASE_DIR", None),
-                "dids": dids or os.environ.get("DIDS"),
+                "dids": dids or os.environ.get("DIDS", None),
                 "transformation_did": transformation_did
-                or os.environ.get("TRANSFORMATION_DID"),
-                "secret": secret or os.environ.get("SECRET"),
+                or os.environ.get("TRANSFORMATION_DID", None),
+                "secret": secret or os.environ.get("SECRET", None),
+                "logger": logger or getLogger(__name__),
             }
         )
 

{oceanprotocol_job_details-0.2.5 → oceanprotocol_job_details-0.2.7}/oceanprotocol_job_details/paths.py

@@ -10,8 +10,8 @@ class Paths:
 
     _base: Path = field(init=False)
 
-    def __post_init__(self, base_dir: Path | None) -> None:
-        self._base = base_dir or Path("/data")
+    def __post_init__(self, base_dir: str | Path | None) -> None:
+        self._base = Path(base_dir) if base_dir else Path("/data")
 
     @property
     def data(self) -> Path:

{oceanprotocol_job_details-0.2.5 → oceanprotocol_job_details-0.2.7}/pyproject.toml

@@ -1,8 +1,9 @@
 [project]
 name = "oceanprotocol-job-details"
-version = "0.2.5"
+version = "0.2.7"
 description = "A Python package to get details from OceanProtocol jobs"
 authors = [
+    { name = "Agrospai", email = "agrospai@udl.cat" },
     { name = "Christian López García", email = "christian.lopez@udl.cat" },
 ]
 requires-python = ">=3.10"

oceanprotocol_job_details-0.2.5/PKG-INFO

@@ -1,76 +0,0 @@
-Metadata-Version: 2.4
-Name: oceanprotocol-job-details
-Version: 0.2.5
-Summary: A Python package to get details from OceanProtocol jobs
-Project-URL: Homepage, https://github.com/AgrospAI/oceanprotocol-job-details
-Project-URL: Issues, https://github.com/AgrospAI/oceanprotocol-job-details/issues
-Author-email: Christian López García <christian.lopez@udl.cat>
-License: Copyright 2025 Agrospai
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-License-File: LICENSE
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Requires-Python: >=3.10
-Requires-Dist: dataclasses-json>=0.6.7
-Requires-Dist: dependency-injector>=4.48.2
-Requires-Dist: orjson>=3.11.3
-Description-Content-Type: text/markdown
-
-A Python package to get details from OceanProtocol jobs
-
----
-
-## Installation
-
-```
-pip install oceanprotocol-job-details
-```
-
-## Usage 
-
-As a simple library, we only need to import the main object and use it once:
-
-```Python
-from oceanprotocol_job_details import JobDetails
-
-# Having no algorithm input parameters
-job_details = JobDetails.load()
-
-```
-
-If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:
-
-```Python
-
-from dataclasses import dataclass
-from oceanprotocol_job_details import JobDetails
-
-
-@dataclass
-class InputParameters:
-    name: str
-    age: int
-
-
-job_details: JobDetails[InputParameters] = JobDetails.load(InputParameters)
-
-# Usage (is type hinted)
-job_details.input_parameters.name
-job_details.input_parameters.age
-
-```
-
-Assumes the directory structure of OceanProtocol algorithms.
-
-### Core functionalities
-
-Given the Ocean Protocol job details structure, parses the passed algorithm parameters into an object to use in your algorithms.
-
-1. Input parameter JSON parsing and validation
-1. Metadata and service extraction from the directory structure.

oceanprotocol_job_details-0.2.5/README.md

@@ -1,52 +0,0 @@
-A Python package to get details from OceanProtocol jobs
-
----
-
-## Installation
-
-```
-pip install oceanprotocol-job-details
-```
-
-## Usage 
-
-As a simple library, we only need to import the main object and use it once:
-
-```Python
-from oceanprotocol_job_details import JobDetails
-
-# Having no algorithm input parameters
-job_details = JobDetails.load()
-
-```
-
-If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:
-
-```Python
-
-from dataclasses import dataclass
-from oceanprotocol_job_details import JobDetails
-
-
-@dataclass
-class InputParameters:
-    name: str
-    age: int
-
-
-job_details: JobDetails[InputParameters] = JobDetails.load(InputParameters)
-
-# Usage (is type hinted)
-job_details.input_parameters.name
-job_details.input_parameters.age
-
-```
-
-Assumes the directory structure of OceanProtocol algorithms.
-
-### Core functionalities
-
-Given the Ocean Protocol job details structure, parses the passed algorithm parameters into an object to use in your algorithms.
-
-1. Input parameter JSON parsing and validation
-1. Metadata and service extraction from the directory structure.