oceanprotocol-job-details 0.2.3__tar.gz → 0.3.2__tar.gz

oceanprotocol_job_details-0.3.2/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: oceanprotocol-job-details
+Version: 0.3.2
+Summary: A Python package to get details from OceanProtocol jobs
+Project-URL: Homepage, https://github.com/AgrospAI/oceanprotocol-job-details
+Project-URL: Issues, https://github.com/AgrospAI/oceanprotocol-job-details/issues
+Author-email: Agrospai <agrospai@udl.cat>, Christian López García <christian.lopez@udl.cat>
+License: Copyright 2025 Agrospai
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Requires-Dist: aiofiles>=25.1.0
+Requires-Dist: dependency-injector>=4.48.2
+Requires-Dist: orjson>=3.11.3
+Requires-Dist: pydantic-settings>=2.12.0
+Requires-Dist: pydantic>=2.12.5
+Description-Content-Type: text/markdown
+
+A Python package to get details from OceanProtocol jobs
+
+---
+
+## Installation
+
+```bash
+pip install oceanprotocol-job-details
+```
+
+```bash
+uv add oceanprotocol-job-details
+```
+
+## Usage
+
+As a simple library, we only need to import `load_job_details` and run it. It will:
+
+1. Fetch the needed parameters to populate the `JobDetails` instance from the environment variables or use the passed values to the function.
+1. Look for the files corresponding to the passed DIDs in the filesystem according to the [Ocean Protocol Structure](#oceanprotocol-structure) and load them into the `JobDetails` instance.
+
+### Minimal Example
+
+```python
+from oceanprotocol_job_details import load_job_details
+
+class InputParameters(BaseModel): ...
+
+job_details = load_job_details({}, InputParameters)
+```
+
+### Custom Input Parameters
+
+If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:
+
+```python
+from pydantic import BaseModel
+from oceanprotocol_job_details import load_job_details
+
+
+class Foo(BaseModel):
+    bar: str
+
+
+class InputParameters(BaseModel):
+    # Allows for nested types
+    foo: Foo
+
+
+job_details = load_job_details({}, InputParameters)
+
+# Usage
+job_details.input_parameters.foo
+job_details.input_parameters.foo.bar
+```
+
+The values to fill the custom `InputParameters` will be parsed from the `algoCustomData.json` located next to the input data directories.
+
+### Iterating Input Files the clean way
+
+```python
+from oceanprotocol_job_details import load_job_details
+
+
+job_details = load_job_details
+
+for idx, file_path in job_details.inputs():
+    ...
+
+_, file_path = next(job_details.inputs())
+```
+
+## OceanProtocol Structure
+
+```bash
+data        # Root /data directory
+├── ddos    # Contains the loaded dataset's DDO
+│   ├── 17feb...e42 # DDO file
+│   └── ... # One DDO per loaded dataset
+├── inputs  # Datasets dir
+│   ├── 17feb...e42 # Dir holding the data of its name DID, contains files named 0..X
+│   │   └── 0 # Data file
+│   └── algoCustomData.json # Custom algorithm input data
+├── logs    # Algorithm output logs dir
+└── outputs # Algorithm output files dir
+```
+
+> **_Note:_** Even though it's possible that the algorithm is passed multiple datasets, right now the implementation only allows to use **one dataset** per algorithm execution, so **normally** the executing job will only have **one ddo**, **one dir** inside inputs, and **one data file** named `0`.

oceanprotocol_job_details-0.3.2/README.md

@@ -0,0 +1,88 @@
+A Python package to get details from OceanProtocol jobs
+
+---
+
+## Installation
+
+```bash
+pip install oceanprotocol-job-details
+```
+
+```bash
+uv add oceanprotocol-job-details
+```
+
+## Usage
+
+As a simple library, we only need to import `load_job_details` and run it. It will:
+
+1. Fetch the needed parameters to populate the `JobDetails` instance from the environment variables or use the passed values to the function.
+1. Look for the files corresponding to the passed DIDs in the filesystem according to the [Ocean Protocol Structure](#oceanprotocol-structure) and load them into the `JobDetails` instance.
+
+### Minimal Example
+
+```python
+from oceanprotocol_job_details import load_job_details
+
+class InputParameters(BaseModel): ...
+
+job_details = load_job_details({}, InputParameters)
+```
+
+### Custom Input Parameters
+
+If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:
+
+```python
+from pydantic import BaseModel
+from oceanprotocol_job_details import load_job_details
+
+
+class Foo(BaseModel):
+    bar: str
+
+
+class InputParameters(BaseModel):
+    # Allows for nested types
+    foo: Foo
+
+
+job_details = load_job_details({}, InputParameters)
+
+# Usage
+job_details.input_parameters.foo
+job_details.input_parameters.foo.bar
+```
+
+The values to fill the custom `InputParameters` will be parsed from the `algoCustomData.json` located next to the input data directories.
+
+### Iterating Input Files the clean way
+
+```python
+from oceanprotocol_job_details import load_job_details
+
+
+job_details = load_job_details
+
+for idx, file_path in job_details.inputs():
+    ...
+
+_, file_path = next(job_details.inputs())
+```
+
+## OceanProtocol Structure
+
+```bash
+data        # Root /data directory
+├── ddos    # Contains the loaded dataset's DDO
+│   ├── 17feb...e42 # DDO file
+│   └── ... # One DDO per loaded dataset
+├── inputs  # Datasets dir
+│   ├── 17feb...e42 # Dir holding the data of its name DID, contains files named 0..X
+│   │   └── 0 # Data file
+│   └── algoCustomData.json # Custom algorithm input data
+├── logs    # Algorithm output logs dir
+└── outputs # Algorithm output files dir
+```
+
+> **_Note:_** Even though it's possible that the algorithm is passed multiple datasets, right now the implementation only allows to use **one dataset** per algorithm execution, so **normally** the executing job will only have **one ddo**, **one dir** inside inputs, and **one data file** named `0`.

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/__init__.py

@@ -0,0 +1,4 @@
+from .helpers import create_container, load_job_details
+from .ocean import JobDetails
+
+__all__ = [JobDetails, load_job_details, create_container]  # type: ignore

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/di.py

@@ -0,0 +1,39 @@
+from typing import Generic, TypeVar
+from dependency_injector import containers, providers
+from pydantic import BaseModel
+
+from oceanprotocol_job_details.loaders.impl.ddo import DDOLoader
+from oceanprotocol_job_details.loaders.impl.files import FilesLoader
+from oceanprotocol_job_details.loaders.impl.job_details import JobDetailsLoader
+from oceanprotocol_job_details.domain import Paths
+
+
+InputParametersT = TypeVar("InputParametersT", bound=BaseModel)
+
+
+class Container(containers.DeclarativeContainer, Generic[InputParametersT]):
+    config = providers.Configuration()
+
+    paths = providers.Singleton(Paths, base_dir=config.base_dir)
+
+    file_loader = providers.Singleton(
+        FilesLoader,
+        dids=config.dids,
+        transformation_did=config.transformation_did,
+        paths=paths,
+        logger=config.logger,
+    )
+
+    files = providers.Factory(lambda loader: loader.load(), loader=file_loader)
+    ddo_loader = providers.Factory(DDOLoader, files=files)
+    ddos = providers.Factory(lambda loader: loader.load(), loader=ddo_loader)
+
+    job_details_loader: providers.Factory[JobDetailsLoader[InputParametersT]] = (
+        providers.Factory(
+            JobDetailsLoader,
+            files=files,
+            secret=config.secret,
+            paths=paths,
+            ddos=ddos,
+        )
+    )

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/domain.py

@@ -0,0 +1,187 @@
+# mypy: disable-error-code=explicit-any
+from dataclasses import InitVar, dataclass, field
+from pathlib import Path
+from typing import Generator, List, Optional, Sequence, TypeAlias, TypeVar
+
+from pydantic import BaseModel, ConfigDict, Field, JsonValue
+
+
+class Credential(BaseModel):
+    type: str
+    values: list[str]
+
+
+class Credentials(BaseModel):
+    allow: list[Credential]
+    deny: list[Credential]
+
+
+class DockerContainer(BaseModel):
+    image: str
+    tag: str
+    entrypoint: str
+
+
+class Algorithm(BaseModel):
+    container: DockerContainer
+    language: str
+    version: str
+    consumerParameters: JsonValue
+
+
+class Metadata(BaseModel):
+    description: str
+    name: str
+    type: str
+    author: str
+    license: str
+    algorithm: Optional[Algorithm] = None
+    tags: Optional[list[str]] = None
+    created: Optional[str] = None
+    updated: Optional[str] = None
+    copyrightHolder: Optional[str] = None
+    links: Optional[list[str]] = None
+    contentLanguage: Optional[str] = None
+    categories: Optional[list[str]] = None
+
+
+class ConsumerParameters(BaseModel):
+    name: str
+    type: str
+    label: str
+    required: bool
+    description: str
+    default: str
+    option: Optional[list[str]] = None
+
+
+class Service(BaseModel):
+    id: str
+    type: str
+    timeout: int
+    files: str
+    datatokenAddress: str
+    serviceEndpoint: str
+    additionalInformation: Optional[str] = None
+    name: Optional[str] = None
+    description: Optional[str] = None
+
+
+class Event(BaseModel):
+    tx: str
+    block: int
+    from_: str = Field(alias="from")
+    contract: str
+    datetime: str
+
+    model_config = ConfigDict(populate_by_name=True)
+
+
+class NFT(BaseModel):
+    address: str
+    name: str
+    symbol: str
+    state: int
+    tokenURI: str
+    owner: str
+    created: str
+
+
+class DataToken(BaseModel):
+    address: str
+    name: str
+    symbol: str
+    serviceId: str
+
+
+class Price(BaseModel):
+    value: int
+
+
+class Stats(BaseModel):
+    allocated: int
+    orders: int
+    price: Price
+
+
+class Purgatory(BaseModel):
+    state: bool
+
+
+class DDO(BaseModel):
+    id: str
+    context: list[str] = Field(alias="@context")
+    nftAddress: str
+    chainId: int
+    version: str
+    metadata: Metadata
+    services: list[Service]
+    credentials: Credentials
+    event: Event
+    nft: NFT
+    datatokens: list[DataToken]
+    stats: Stats
+    purgatory: Purgatory
+
+    model_config = ConfigDict(populate_by_name=True)
+
+
+@dataclass(frozen=True)
+class DIDPaths:
+    did: str
+    ddo: Path = field(repr=False)
+
+    files: InitVar[Generator[Path, None, None]]
+
+    _input: List[Path] = field(init=False, repr=False)
+
+    def __post_init__(self, files: Generator[Path, None, None]) -> None:
+        assert self.ddo.exists(), f"DDO {self.ddo} does not exist"
+
+        object.__setattr__(self, "_input", list(files))
+
+    @property
+    def input_files(self) -> List[Path]:
+        return self._input
+
+    def __len__(self) -> int:
+        return len(self._input)
+
+
+Files: TypeAlias = Sequence[DIDPaths]
+
+
+@dataclass(frozen=True)
+class Paths:
+    """Configuration class for the Ocean Protocol Job Details"""
+
+    base_dir: InitVar[Path | None] = None
+
+    _base: Path = field(init=False, repr=False)
+
+    def __post_init__(self, base_dir: Path | None) -> None:
+        object.__setattr__(self, "_base", base_dir if base_dir else Path("/data"))
+
+    @property
+    def data(self) -> Path:
+        return self._base
+
+    @property
+    def inputs(self) -> Path:
+        return self.data / "inputs"
+
+    @property
+    def ddos(self) -> Path:
+        return self.data / "ddos"
+
+    @property
+    def outputs(self) -> Path:
+        return self.data / "outputs"
+
+    @property
+    def logs(self) -> Path:
+        return self.data / "logs"
+
+    @property
+    def algorithm_custom_parameters(self) -> Path:
+        return self.inputs / "algoCustomData.json"

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/helpers.py

@@ -0,0 +1,29 @@
+from typing import Any, Dict, Type, TypeVar
+
+from pydantic import BaseModel, JsonValue
+
+from oceanprotocol_job_details.di import Container
+from oceanprotocol_job_details.ocean import JobDetails
+from oceanprotocol_job_details.settings import JobSettings
+
+InputParametersT = TypeVar("InputParametersT", bound=BaseModel)
+
+
+def create_container(config: Dict[str, Any]) -> Container[InputParametersT]:  # type: ignore[explicit-any]
+    """Return a fully configured Container from a config dict."""
+    container = Container[InputParametersT]()
+    settings = JobSettings(**config)
+    container.config.from_pydantic(settings)
+    return container
+
+
+def load_job_details(
+    config: Dict[str, JsonValue],
+    input_type: Type[InputParametersT],
+) -> JobDetails[InputParametersT]:
+    """
+    Load JobDetails for a given input_type using the config.
+    Returns a fully initialized JobDetails instance.
+    """
+    container: Container[InputParametersT] = create_container(config)
+    return container.job_details_loader(input_type=input_type).load()

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/loaders/impl/ddo.py

@@ -0,0 +1,24 @@
+from __future__ import annotations
+
+from dataclasses import InitVar, dataclass, field
+from pathlib import Path
+from typing import final
+
+from oceanprotocol_job_details.domain import DDO, Files
+
+
+@final
+@dataclass(frozen=True)
+class DDOLoader:
+    files: InitVar[Files]
+    """The files to load the DDOs from"""
+
+    _ddo_paths: list[Path] = field(init=False)
+
+    def __post_init__(self, files: Files) -> None:
+        assert files is not None and len(files) != 0, "Missing files"
+
+        object.__setattr__(self, "_ddo_paths", [f.ddo for f in files])
+
+    def load(self) -> list[DDO]:
+        return [DDO.model_validate_json(p.read_text()) for p in self._ddo_paths]

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/loaders/impl/files.py

@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+from dataclasses import InitVar, dataclass, field
+from logging import Logger
+from pathlib import Path
+from typing import Literal, final
+
+from oceanprotocol_job_details.domain import DIDPaths, Files, Paths
+
+
+@final
+@dataclass(frozen=True)
+class FilesLoader:
+    paths: Paths
+    """Path configurations of the project"""
+
+    logger: Logger = field(repr=False)
+    """Logger to use"""
+
+    dids: list[str]
+    """Input DIDs"""
+
+    transformation_did: InitVar[str | None] = None
+    """DID for the transformation algorithm"""
+
+    _transformation_did: str = field(init=False)
+
+    def __post_init__(self, transformation_did: str | None) -> None:
+        object.__setattr__(self, "_transformation_did", transformation_did)
+
+        assert self.dids, "Missing input DIDs"
+
+    def calculate_path(self, did: str, path_type: Literal["input", "ddo"]) -> Path:
+        match path_type:
+            case "ddo":
+                return self.paths.ddos / did
+            case "input":
+                return self.paths.inputs / did
+
+    def load(self) -> Files:
+        return [
+            DIDPaths(
+                did=did,
+                ddo=self.calculate_path(did, "ddo"),
+                files=self.calculate_path(did, "input").iterdir(),
+            )
+            for did in self.dids
+        ]

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/loaders/impl/job_details.py

@@ -0,0 +1,28 @@
+from dataclasses import dataclass, field
+from typing import Generic, Type, TypeVar, final
+
+from pydantic import BaseModel
+
+from oceanprotocol_job_details.domain import DDO, Files, Paths
+from oceanprotocol_job_details.ocean import JobDetails
+
+T = TypeVar("T", bound=BaseModel)
+
+
+@final
+@dataclass(frozen=True)
+class JobDetailsLoader(Generic[T]):
+    input_type: Type[T] = field(repr=False)
+    files: Files
+    secret: str
+    paths: Paths
+    ddos: list[DDO]
+
+    def load(self) -> JobDetails[T]:
+        return JobDetails[T](
+            files=self.files,
+            secret=self.secret,
+            ddos=self.ddos,
+            paths=self.paths,
+            input_type=self.input_type,
+        )

{oceanprotocol_job_details-0.2.3 → oceanprotocol_job_details-0.3.2}/oceanprotocol_job_details/loaders/loader.py

@@ -4,10 +4,6 @@ T = TypeVar("T", covariant=True)
 
 
 class Loader(Protocol[T]):
-
     def load(self) -> T:
         """Load an instance of the given type"""
         ...
-
-
-del T

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/ocean.py

@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+import asyncio
+from functools import cached_property
+from pathlib import Path
+from typing import Generator, Generic, Tuple, Type, TypeVar, final
+
+import aiofiles
+from pydantic import BaseModel, ConfigDict, Secret, model_validator
+
+from oceanprotocol_job_details.domain import DDO, Files, Paths
+
+InputParemetersT = TypeVar("InputParemetersT", bound=BaseModel)
+
+
+@final
+class JobDetails(BaseModel, Generic[InputParemetersT]):  # type: ignore[explicit-any]
+    files: Files
+    ddos: list[DDO]
+    paths: Paths
+    input_type: Type[InputParemetersT]
+    secret: Secret[str] | None = None
+
+    model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True)
+
+    @model_validator(mode="after")
+    def validate_type(self) -> JobDetails[InputParemetersT]:
+        assert issubclass(self.input_type, BaseModel), (
+            f"{self.input_type} must be subtype of pydantic.BaseModel"
+        )
+        return self
+
+    def inputs(self) -> Generator[Tuple[int, Path], None, None]:
+        yield from (
+            (idx, file)
+            for idx, files in enumerate(self.files)
+            for file in files.input_files
+        )
+
+    @cached_property
+    def input_parameters(self) -> InputParemetersT:
+        return asyncio.run(self.ainput_parameters())
+
+    async def ainput_parameters(self) -> InputParemetersT:
+        path = self.paths.algorithm_custom_parameters
+        async with aiofiles.open(path) as f:
+            raw = await f.read()
+
+        raw = raw.strip()
+        assert raw is not None, f"Empty file {path}"
+        return self.input_type.model_validate_json(raw)

oceanprotocol_job_details-0.3.2/oceanprotocol_job_details/settings.py

@@ -0,0 +1,31 @@
+# mypy: disable-error-code=call-overload
+from logging import Logger, getLogger
+from pathlib import Path
+
+import orjson
+from pydantic import Field, field_validator
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class JobSettings(BaseSettings):  # type: ignore[explicit-any]
+    base_dir: Path = Field(alias="BASE_DIR")
+    dids: list[str] = Field(alias="DIDS")
+    transformation_did: str = Field(alias="TRANSFORMATION_DID")
+    secret: str | None = Field(default=None, alias="SECRET")
+    logger: Logger = Field(default_factory=lambda: getLogger(__name__))
+
+    model_config = SettingsConfigDict(
+        extra="forbid",
+        validate_default=True,
+        populate_by_name=True,
+        arbitrary_types_allowed=True,
+    )
+
+    @field_validator("dids", mode="before")
+    @classmethod
+    def split_dids(cls, v: list[str] | str) -> list[str]:
+        if isinstance(v, str):
+            data = orjson.loads(v)
+            assert isinstance(data, list)
+            return data
+        return v