quapp-hpc 0.0.1__tar.gz

quapp_hpc-0.0.1/LICENSE

@@ -0,0 +1,8 @@
+The MIT License (MIT)
+Copyright © CITYNOW Co. Ltd. All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

quapp_hpc-0.0.1/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.4
+Name: quapp-hpc
+Version: 0.0.1
+Summary: Quapp HPC library — Slurm integration for Quapp Platform
+Author-email: "CITYNOW Co. Ltd." <corp@citynow.vn>
+License: The MIT License (MIT)
+        Copyright © CITYNOW Co. Ltd. All rights reserved.
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+Project-URL: Homepage, https://quapp.cloud/
+Keywords: quapp,quapp-hpc,slurm,hpc
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.13,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: quapp-common==0.0.13
+Requires-Dist: requests>=2.31.0
+Requires-Dist: boto3>=1.28.0
+Provides-Extra: dev
+Requires-Dist: black; extra == "dev"
+Requires-Dist: bumpver; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: pip-tools; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# quapp-hpc
+
+Python library cho Quapp HPC functions — cầu nối giữa Quapp FaaS platform và Slurm HPC cluster.
+
+## Architecture
+
+```
+ksvc (Docker)
+├── index.py                  FastAPI server
+├── quapp_hpc/
+│   ├── factory/
+│   │   └── hpc_handler_factory.py   Entry point cho user
+│   ├── component/backend/
+│   │   └── hpc_invocation.py        Orchestrates job lifecycle
+│   └── model/
+│       ├── provider/slurm_provider.py   Auth headers, base URL
+│       └── device/slurm_device.py       Submit → Poll → S3 download
+└── function/
+    └── handler.py            User-defined processing() + post_processing()
+```
+
+## Luồng thực thi
+
+```
+index.py nhận HTTP POST
+    → HpcHandlerFactory.create_handler(event, processing_fn, post_processing_fn)
+    → InvocationHandler.handle()
+    → HpcInvocation.submit_job()
+        1. processing_fn(invocation_input)  → bash script string
+        2. SlurmDevice._create_job(script)  → POST Slurm REST API → slurm_job_id
+        3. SlurmDevice._get_job_result()    → poll mỗi 30s → COMPLETED/FAILED
+        4. SlurmDevice._download_s3_result()→ boto3 get s3://$S3_BUCKET/$JOB_UUID/output.json
+        5. post_processing_fn(s3_result)    → final response
+```
+
+## Environment variables (từ K8s Secret `slurm-credentials`)
+
+| Var | Ví dụ | Mô tả |
+|-----|-------|-------|
+| `SLURM_API_URL` | `http://10.1.0.15:6820` | Slurm REST API base URL |
+| `SLURM_JWT` | `eyJ...` | JWT token cho Slurm auth |
+| `SLURM_USERNAME` | `quapp-svc` | Slurm username |
+| `SLURM_ACCOUNT` | `quapp` | Slurm account/allocation |
+| `S3_BUCKET` | `quapp-slurm-output-dev` | S3 bucket cho job output |
+| `AWS_REGION` | `ap-southeast-1` | AWS region |
+| `SLURM_POLL_SEC` | `30` | Polling interval (giây) |
+| `SLURM_TIMEOUT_SEC` | `21600` | Max wait time (giây, default 6h) |
+
+## invocation_input schema
+
+Xem chi tiết tại [`../qapp-sdk-templates/slurm-hpc/README.md`](../../qapp-sdk-templates/slurm-hpc/README.md).
+
+Tóm tắt:
+```json
+{
+  "resources": { "partition", "nodes", "cpus_per_task", "gpus", "memory_gb", "time_limit" },
+  "container": { "type": "sif"|"docker"|"none", "image": "..." },
+  "job":       { "type": "script"|"command", "script"|"command": "...", "environment": {}, "input_s3_paths": [] }
+}
+```
+
+## Slurm REST API
+
+- Version: `v0.0.40`
+- Submit: `POST {SLURM_API_URL}/slurm/v0.0.40/job/submit`
+- Status: `GET {SLURM_API_URL}/slurm/v0.0.40/job/{job_id}`
+- Auth headers: `X-SLURM-USER-NAME`, `X-SLURM-USER-TOKEN`
+
+### Job state mapping
+
+| Slurm state | Quapp state |
+|---|---|
+| PENDING, CONFIGURING, RUNNING, COMPLETING | RUNNING |
+| COMPLETED | DONE |
+| FAILED, CANCELLED, TIMEOUT, NODE_FAIL, PREEMPTED | ERROR |
+
+## S3 output pattern
+
+Job script phải upload kết quả:
+```bash
+aws s3 cp /tmp/output.json s3://$S3_BUCKET/$JOB_UUID/output.json
+```
+
+`$JOB_UUID` và `$S3_BUCKET` được inject tự động bởi `SlurmDevice._create_job()` qua Slurm `environment` array.
+
+## K8s Secret
+
+```yaml
+# infrastructure/quapp-job-scheduler/k8s/cts/slurm-secret.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: slurm-credentials
+  namespace: quapp-functions-dev
+stringData:
+  SLURM_JWT: "<generate: sudo scontrol token username=quapp-svc lifespan=2592000>"
+  SLURM_API_URL: "http://10.1.0.15:6820"
+  SLURM_USERNAME: "quapp-svc"
+  SLURM_ACCOUNT: "quapp"
+  S3_BUCKET: "quapp-slurm-output-dev"
+  AWS_REGION: "ap-southeast-1"
+```
+
+## DB seed required
+
+Chạy script trước khi deploy:
+```
+infrastructure/quapp-functions-backend/docs/db/seed_slurm_hpc.sql
+```

quapp_hpc-0.0.1/README.md

@@ -0,0 +1,109 @@
+# quapp-hpc
+
+Python library cho Quapp HPC functions — cầu nối giữa Quapp FaaS platform và Slurm HPC cluster.
+
+## Architecture
+
+```
+ksvc (Docker)
+├── index.py                  FastAPI server
+├── quapp_hpc/
+│   ├── factory/
+│   │   └── hpc_handler_factory.py   Entry point cho user
+│   ├── component/backend/
+│   │   └── hpc_invocation.py        Orchestrates job lifecycle
+│   └── model/
+│       ├── provider/slurm_provider.py   Auth headers, base URL
+│       └── device/slurm_device.py       Submit → Poll → S3 download
+└── function/
+    └── handler.py            User-defined processing() + post_processing()
+```
+
+## Luồng thực thi
+
+```
+index.py nhận HTTP POST
+    → HpcHandlerFactory.create_handler(event, processing_fn, post_processing_fn)
+    → InvocationHandler.handle()
+    → HpcInvocation.submit_job()
+        1. processing_fn(invocation_input)  → bash script string
+        2. SlurmDevice._create_job(script)  → POST Slurm REST API → slurm_job_id
+        3. SlurmDevice._get_job_result()    → poll mỗi 30s → COMPLETED/FAILED
+        4. SlurmDevice._download_s3_result()→ boto3 get s3://$S3_BUCKET/$JOB_UUID/output.json
+        5. post_processing_fn(s3_result)    → final response
+```
+
+## Environment variables (từ K8s Secret `slurm-credentials`)
+
+| Var | Ví dụ | Mô tả |
+|-----|-------|-------|
+| `SLURM_API_URL` | `http://10.1.0.15:6820` | Slurm REST API base URL |
+| `SLURM_JWT` | `eyJ...` | JWT token cho Slurm auth |
+| `SLURM_USERNAME` | `quapp-svc` | Slurm username |
+| `SLURM_ACCOUNT` | `quapp` | Slurm account/allocation |
+| `S3_BUCKET` | `quapp-slurm-output-dev` | S3 bucket cho job output |
+| `AWS_REGION` | `ap-southeast-1` | AWS region |
+| `SLURM_POLL_SEC` | `30` | Polling interval (giây) |
+| `SLURM_TIMEOUT_SEC` | `21600` | Max wait time (giây, default 6h) |
+
+## invocation_input schema
+
+Xem chi tiết tại [`../qapp-sdk-templates/slurm-hpc/README.md`](../../qapp-sdk-templates/slurm-hpc/README.md).
+
+Tóm tắt:
+```json
+{
+  "resources": { "partition", "nodes", "cpus_per_task", "gpus", "memory_gb", "time_limit" },
+  "container": { "type": "sif"|"docker"|"none", "image": "..." },
+  "job":       { "type": "script"|"command", "script"|"command": "...", "environment": {}, "input_s3_paths": [] }
+}
+```
+
+## Slurm REST API
+
+- Version: `v0.0.40`
+- Submit: `POST {SLURM_API_URL}/slurm/v0.0.40/job/submit`
+- Status: `GET {SLURM_API_URL}/slurm/v0.0.40/job/{job_id}`
+- Auth headers: `X-SLURM-USER-NAME`, `X-SLURM-USER-TOKEN`
+
+### Job state mapping
+
+| Slurm state | Quapp state |
+|---|---|
+| PENDING, CONFIGURING, RUNNING, COMPLETING | RUNNING |
+| COMPLETED | DONE |
+| FAILED, CANCELLED, TIMEOUT, NODE_FAIL, PREEMPTED | ERROR |
+
+## S3 output pattern
+
+Job script phải upload kết quả:
+```bash
+aws s3 cp /tmp/output.json s3://$S3_BUCKET/$JOB_UUID/output.json
+```
+
+`$JOB_UUID` và `$S3_BUCKET` được inject tự động bởi `SlurmDevice._create_job()` qua Slurm `environment` array.
+
+## K8s Secret
+
+```yaml
+# infrastructure/quapp-job-scheduler/k8s/cts/slurm-secret.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: slurm-credentials
+  namespace: quapp-functions-dev
+stringData:
+  SLURM_JWT: "<generate: sudo scontrol token username=quapp-svc lifespan=2592000>"
+  SLURM_API_URL: "http://10.1.0.15:6820"
+  SLURM_USERNAME: "quapp-svc"
+  SLURM_ACCOUNT: "quapp"
+  S3_BUCKET: "quapp-slurm-output-dev"
+  AWS_REGION: "ap-southeast-1"
+```
+
+## DB seed required
+
+Chạy script trước khi deploy:
+```
+infrastructure/quapp-functions-backend/docs/db/seed_slurm_hpc.sql
+```

quapp_hpc-0.0.1/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=61.0.0", "wheel==0.45.1"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "quapp-hpc"
+version = "0.0.1"
+description = "Quapp HPC library — Slurm integration for Quapp Platform"
+readme = "README.md"
+authors = [{ name = "CITYNOW Co. Ltd.", email = "corp@citynow.vn" }]
+license = { file = "LICENSE" }
+classifiers = [
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+]
+keywords = ["quapp", "quapp-hpc", "slurm", "hpc"]
+dependencies = [
+    "quapp-common==0.0.13",
+    "requests>=2.31.0",
+    "boto3>=1.28.0",
+]
+requires-python = ">=3.10,<3.13"
+
+[project.optional-dependencies]
+dev = ["black", "bumpver", "isort", "pip-tools", "pytest"]
+
+[tool.setuptools.packages.find]
+include = ["quapp_hpc*", "quapp_slurm_hpc*"]
+exclude = ["*.md", "*.yml", "*.yaml", "*.toml", "tests*", ".gitignore"]
+
+[project.urls]
+Homepage = "https://quapp.cloud/"

quapp_hpc-0.0.1/quapp_hpc/__init__.py

@@ -0,0 +1,3 @@
+from .factory.hpc_handler_factory import HpcHandlerFactory
+
+__all__ = ["HpcHandlerFactory"]

quapp_hpc-0.0.1/quapp_hpc/component/backend/hpc_invocation.py

@@ -0,0 +1,43 @@
+from quapp_common.component.backend.invocation import Invocation
+from quapp_common.data.request.invocation_request import InvocationRequest
+from quapp_common.model.provider.provider import Provider
+
+from ...factory.hpc_device_factory import HpcDeviceFactory
+from ...factory.hpc_provider_factory import HpcProviderFactory
+
+
+class HpcInvocation(Invocation):
+
+    def __init__(self, request_data: InvocationRequest):
+        super().__init__(request_data)
+        raw = request_data.input or {}
+        job = raw.get("job", {})
+        # s3Bucket intentionally dropped — S3 bucket is system config, read by
+        # SlurmDevice from the S3_BUCKET env var (slurm-credentials secret).
+        # User input must not override system fields (HPC design philosophy).
+        self._hpc_config = {
+            "resources":      raw.get("resources", {}),
+            "container":      raw.get("container", {}),
+            "environment":    job.get("environment", {}),
+            "input_s3_paths": job.get("input_s3_paths", []),
+        }
+
+    def _export_circuit(self, circuit):
+        pass
+
+    def _get_qubit_amount(self, circuit) -> int:
+        return 0
+
+    def _create_provider(self) -> Provider:
+        return HpcProviderFactory.create_provider(
+            provider_type=self.backend_information.provider_tag,
+            authentication=self.backend_information.authentication,
+        )
+
+    def _create_device(self, provider: Provider):
+        return HpcDeviceFactory.create_device(
+            provider=provider,
+            device_specification=self.backend_information.device_name,
+            job_uuid=self.job_id,
+            hpc_config=self._hpc_config,
+        )

quapp_hpc-0.0.1/quapp_hpc/component/backend/slurm_job_fetching.py

@@ -0,0 +1,95 @@
+import json
+import os
+
+import boto3
+import requests
+from quapp_common.component.backend.job_fetching import JobFetching
+from quapp_common.config.logging_config import job_logger
+from quapp_common.data.request.job_fetching_request import JobFetchingRequest
+from quapp_common.enum.status.job_status import JobStatus
+
+from ...model.provider.slurm_provider import SlurmProvider
+
+SLURM_JWT  = os.getenv("SLURM_JWT", "")
+S3_BUCKET  = os.getenv("S3_BUCKET",  "quapp-slurm-output-dev")
+AWS_REGION = os.getenv("AWS_REGION", "ap-southeast-1")
+
+_SLURM_TERMINAL_TO_JOB_STATUS = {
+    "COMPLETED":   JobStatus.DONE.value,
+    "FAILED":      JobStatus.ERROR.value,
+    "CANCELLED":   JobStatus.ERROR.value,
+    "TIMEOUT":     JobStatus.ERROR.value,
+    "NODE_FAIL":   JobStatus.ERROR.value,
+    "PREEMPTED":   JobStatus.ERROR.value,
+}
+
+
+class _SlurmJobResult:
+    """Lazy S3 download of the Slurm job's output.json.
+
+    result() downloads from S3 only when called (after DONE is confirmed).
+    usage() returns None so JobFetching.__get_execution_time() doesn't raise.
+    """
+
+    def __init__(self, job_uuid: str, s3_bucket: str = S3_BUCKET, aws_region: str = AWS_REGION):
+        self._job_uuid = job_uuid
+        self._s3_bucket = s3_bucket
+        self._aws_region = aws_region
+        self._logger = job_logger(job_uuid)
+
+    def usage(self):
+        return None
+
+    def result(self):
+        s3_key = f"{self._job_uuid}/output.json"
+        self._logger.info(f"Downloading result from s3://{self._s3_bucket}/{s3_key}")
+        s3 = boto3.client("s3", region_name=self._aws_region)
+        resp = s3.get_object(Bucket=self._s3_bucket, Key=s3_key)
+        return json.loads(resp["Body"].read())
+
+
+class SlurmJobFetching(JobFetching):
+    """Polls Slurm REST API for job status (IBM-pattern, triggered by watchdog)."""
+
+    def __init__(self, request_data: JobFetchingRequest):
+        super().__init__(request_data)
+        self.job_id = request_data.job_id
+        self._logger = job_logger(request_data.job_id)
+
+    # ── Abstract implementations ──────────────────────────────────────────────
+
+    def _collect_provider(self) -> SlurmProvider:
+        jwt = (self.provider_authentication or {}).get("slurm_jwt") or SLURM_JWT
+        if not jwt:
+            raise ValueError("SLURM_JWT not set — cannot authenticate with Slurm API")
+        return SlurmProvider(jwt_token=jwt)
+
+    def _retrieve_job(self, provider: SlurmProvider) -> dict:
+        return {"slurm_job_id": self.provider_job_id, "provider": provider}
+
+    def _get_job_status(self, job: dict) -> str:
+        provider: SlurmProvider = job["provider"]
+        slurm_job_id = job["slurm_job_id"]
+
+        resp = requests.get(
+            f"{provider.base_url}/job/{slurm_job_id}",
+            headers=provider.auth_headers(),
+            timeout=15,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+
+        jobs = data.get("jobs", [data])
+        if not jobs:
+            return "UNKNOWN"
+
+        raw_state = jobs[0].get("job_state", "UNKNOWN")
+        if isinstance(raw_state, list):
+            raw_state = raw_state[0] if raw_state else "UNKNOWN"
+
+        state = str(raw_state).strip()
+        self._logger.info(f"Slurm job {slurm_job_id} state: {state}")
+        return _SLURM_TERMINAL_TO_JOB_STATUS.get(state, state)
+
+    def _get_job_result(self, job: dict) -> _SlurmJobResult:
+        return _SlurmJobResult(job_uuid=self.job_id, s3_bucket=S3_BUCKET, aws_region=AWS_REGION)

quapp_hpc-0.0.1/quapp_hpc/factory/hpc_device_factory.py

@@ -0,0 +1,19 @@
+from quapp_common.config.logging_config import job_logger
+from quapp_common.model.provider.provider import Provider
+
+from ..model.device.slurm_device import SlurmDevice
+
+logger = job_logger('HpcDeviceFactory')
+
+
+class HpcDeviceFactory:
+
+    @staticmethod
+    def create_device(
+        provider: Provider,
+        device_specification: str,
+        job_uuid: str,
+        hpc_config: dict = None,
+    ) -> SlurmDevice:
+        logger.debug(f"Creating SlurmDevice: partition={device_specification}")
+        return SlurmDevice(provider, device_specification, job_uuid, hpc_config=hpc_config)

quapp_hpc-0.0.1/quapp_hpc/factory/hpc_handler_factory.py

@@ -0,0 +1,30 @@
+from quapp_common.config.logging_config import job_logger
+from quapp_common.factory.handler_factory import HandlerFactory
+from quapp_common.handler.handler import Handler
+
+from ..handler.invocation_handler import InvocationHandler
+from ..handler.job_fetching_handler import SlurmJobFetchingHandler
+
+
+class HpcHandlerFactory(HandlerFactory):
+
+    @staticmethod
+    def create_handler(event, circuit_preparation_fn, post_processing_fn) -> Handler:
+        request_data = event.json()
+        logger = job_logger(request_data.get("jobId"))
+
+        provider_job_id = request_data.get("providerJobId")
+
+        if provider_job_id:
+            logger.debug(f"HpcHandlerFactory: job fetching (slurmJobId={provider_job_id})")
+            return SlurmJobFetchingHandler(
+                request_data=request_data,
+                post_processing_fn=post_processing_fn,
+            )
+
+        logger.debug("HpcHandlerFactory: initial invocation")
+        return InvocationHandler(
+            request_data=request_data,
+            circuit_preparation_fn=circuit_preparation_fn,
+            post_processing_fn=post_processing_fn,
+        )

quapp_hpc-0.0.1/quapp_hpc/factory/hpc_provider_factory.py

@@ -0,0 +1,24 @@
+import os
+from quapp_common.enum.provider_tag import ProviderTag
+from quapp_common.config.logging_config import job_logger
+
+from ..model.provider.slurm_provider import SlurmProvider
+
+logger = job_logger('HpcProviderFactory')
+
+SLURM_JWT = os.getenv("SLURM_JWT", "")
+
+
+class HpcProviderFactory:
+
+    @staticmethod
+    def create_provider(provider_type: ProviderTag, authentication: dict):
+        logger.debug(f"Creating HPC provider: {provider_type}")
+
+        if provider_type == ProviderTag.QUAPP_HPC:
+            jwt = authentication.get("slurm_jwt") or SLURM_JWT
+            if not jwt:
+                raise ValueError("SLURM_JWT not set — cannot authenticate with Slurm API")
+            return SlurmProvider(jwt_token=jwt)
+
+        raise NotImplementedError(f"Unsupported HPC provider: {provider_type}")

quapp_hpc-0.0.1/quapp_hpc/handler/invocation_handler.py

@@ -0,0 +1,23 @@
+from quapp_common.data.request.invocation_request import InvocationRequest
+from quapp_common.handler.handler import Handler
+
+from ..component.backend.hpc_invocation import HpcInvocation
+
+
+class InvocationHandler(Handler):
+
+    def __init__(self, request_data: dict, circuit_preparation_fn, post_processing_fn):
+        super().__init__(request_data, post_processing_fn)
+        self.circuit_preparation_fn = circuit_preparation_fn
+
+    def handle(self):
+        self.logger.debug("HPC InvocationHandler: start")
+        try:
+            invocation_request = InvocationRequest(self.request_data)
+            HpcInvocation(invocation_request).submit_job(
+                circuit_preparation_fn=self.circuit_preparation_fn,
+                post_processing_fn=self.post_processing_fn,
+            )
+        except Exception as exc:
+            self.logger.exception(f"HPC invocation failed: {exc}")
+            raise

quapp_hpc-0.0.1/quapp_hpc/handler/job_fetching_handler.py

@@ -0,0 +1,18 @@
+from quapp_common.data.request.job_fetching_request import JobFetchingRequest
+from quapp_common.handler.handler import Handler
+
+from ..component.backend.slurm_job_fetching import SlurmJobFetching
+
+
+class SlurmJobFetchingHandler(Handler):
+    """Handles watchdog-triggered job status polls for Slurm jobs."""
+
+    def __init__(self, request_data: dict, post_processing_fn):
+        super().__init__(request_data, post_processing_fn)
+
+    def handle(self):
+        self.logger.debug("SlurmJobFetchingHandler: start")
+        request = JobFetchingRequest(self.request_data)
+        return SlurmJobFetching(request).fetch(
+            post_processing_fn=self.post_processing_fn
+        )

quapp_hpc-0.0.1/quapp_hpc/model/device/slurm_device.py

@@ -0,0 +1,321 @@
+import base64
+import json
+import os
+import shlex
+import time
+
+import boto3
+import requests
+from quapp_common.config.logging_config import job_logger
+from quapp_common.data.device.circuit_running_option import CircuitRunningOption
+from quapp_common.enum.status.job_status import JobStatus
+from quapp_common.model.device.device import Device
+from quapp_common.model.provider.provider import Provider
+
+from ..provider.slurm_provider import SlurmProvider, SLURM_ACCOUNT
+
+# ── System config (read from K8s secret slurm-credentials at ksvc startup) ───
+S3_BUCKET         = os.getenv("S3_BUCKET",               "quapp-slurm-output-dev")
+AWS_REGION        = os.getenv("AWS_REGION",               "ap-southeast-1")
+SLURM_POLL_SEC    = int(os.getenv("SLURM_POLL_SEC",      "30"))
+SLURM_TIMEOUT_SEC = int(os.getenv("SLURM_TIMEOUT_SEC",   "21600"))  # 6 hours
+SLURM_TIME_LIMIT  = int(os.getenv("SLURM_TIME_LIMIT_MIN", "60"))
+
+# ── Compute runtime config (injected by builder when creating ksvc) ──────────
+# Builder sets these env vars based on the function's qapp_compute_runtime row.
+# The path is resolved per-function: /data/containers/functions/<fn_hash>_<fn_tag>.sif
+# CTS knative_builder/function/handler.sh adds these via --env to `kn ksvc create`.
+COMPUTE_SIF_PATH   = os.getenv("COMPUTE_SIF_PATH")
+COMPUTE_ENTRYPOINT = os.getenv("COMPUTE_ENTRYPOINT", "python3 /tmp/quapp_job.py")
+COMPUTE_SCRIPT_EXT = os.getenv("COMPUTE_SCRIPT_EXT", "py")
+
+_TERMINAL_STATES = {"COMPLETED", "FAILED", "CANCELLED", "TIMEOUT", "NODE_FAIL", "PREEMPTED"}
+_DONE_STATE      = "COMPLETED"
+
+_SLURM_TERMINAL_TO_JOB_STATUS = {
+    "COMPLETED":   JobStatus.DONE.value,
+    "FAILED":      JobStatus.ERROR.value,
+    "CANCELLED":   JobStatus.ERROR.value,
+    "TIMEOUT":     JobStatus.ERROR.value,
+    "NODE_FAIL":   JobStatus.ERROR.value,
+    "PREEMPTED":   JobStatus.ERROR.value,
+}
+
+_COLLECT_PY = (
+    "import json, os; "
+    "ec = int(open('/tmp/quapp_exit_code.txt').read().strip()) "
+    "if os.path.exists('/tmp/quapp_exit_code.txt') else 0; "
+    "out = open('/tmp/quapp_stdout.txt').read() "
+    "if os.path.exists('/tmp/quapp_stdout.txt') else ''; "
+    "err = open('/tmp/quapp_stderr.txt').read() "
+    "if os.path.exists('/tmp/quapp_stderr.txt') else ''; "
+    "json.dump({'exit_code': ec, 'stdout': out, 'stderr': err}, open('/tmp/output.json','w'))"
+)
+
+
+def _resolve_compute_container(container_cfg: dict) -> tuple[str, str, str]:
+    """Resolve Apptainer image path, entrypoint command, and script extension.
+
+    Resolution order (first match wins):
+      1. User override via invocation `container`:
+           container.type = 'sif'    + container.image  →  /data/containers/<image>
+           container.type = 'docker' + container.image  →  docker://<image>
+      2. Builder-injected env vars (per-function SIF from build step):
+           COMPUTE_SIF_PATH + COMPUTE_ENTRYPOINT + COMPUTE_SCRIPT_EXT
+      3. Hard error — no valid configuration. Container is mandatory; running user
+         scripts on bare metal is not supported (see HPC design philosophy).
+    """
+    container_type = (container_cfg or {}).get("type", "").lower()
+    user_image     = (container_cfg or {}).get("image", "")
+    user_entry     = (container_cfg or {}).get("entrypoint")
+    user_ext       = (container_cfg or {}).get("script_extension")
+
+    # 1. User override — explicit container choice in invocation input
+    if container_type in ("sif", "docker"):
+        if not user_image:
+            raise ValueError(
+                f"container.type={container_type!r} requires container.image"
+            )
+        if container_type == "sif":
+            sif_path = f"/data/containers/{user_image}"
+        else:
+            sif_path = f"docker://{user_image}"
+        return (
+            sif_path,
+            user_entry or COMPUTE_ENTRYPOINT,
+            user_ext   or COMPUTE_SCRIPT_EXT,
+        )
+
+    # 2. Builder-injected default — per-function SIF built at deploy time
+    if COMPUTE_SIF_PATH:
+        return (COMPUTE_SIF_PATH, COMPUTE_ENTRYPOINT, COMPUTE_SCRIPT_EXT)
+
+    # 3. No path resolved → fail explicitly. Running outside a container is not
+    # supported per the Quapp-HPC design philosophy (container-first sandboxing).
+    raise RuntimeError(
+        "No compute container configured. Either the function pod is missing the "
+        "COMPUTE_SIF_PATH env var (builder did not produce a per-function SIF — "
+        "rebuild required) or the invocation must explicitly provide "
+        "container.type and container.image."
+    )
+
+
+class SlurmDevice(Device):
+
+    def __init__(
+        self,
+        provider: Provider,
+        device_specification: str,
+        job_uuid: str,
+        hpc_config: dict = None,
+    ):
+        super().__init__(provider, device_specification)
+        self.job_uuid:  str          = job_uuid
+        self.slurm:     SlurmProvider = provider
+        self.logger                  = job_logger(job_uuid)
+        self.hpc_config              = hpc_config or {}
+        # S3 bucket is a system field — read only from env (no input override).
+        # Per HPC design philosophy: system fields are not user-controlled.
+        self.s3_bucket               = S3_BUCKET
+
+    # ── Abstract method implementations ──────────────────────────────────────
+
+    def _is_simulator(self) -> bool:
+        return False
+
+    def _create_job(self, circuit: str, options: CircuitRunningOption) -> dict:
+        """Wrap circuit (user script string) in SBATCH infrastructure and submit to Slurm REST API."""
+        bash = self._build_sbatch_script(circuit)
+
+        payload = {
+            "job": {
+                "name":    f"quapp-{self.job_uuid[:8]}",
+                "account": SLURM_ACCOUNT,
+                "environment": [
+                    "PATH=/usr/bin:/bin:/usr/local/bin",
+                    f"AWS_DEFAULT_REGION={AWS_REGION}",
+                ],
+                "time_limit": {
+                    "number":   SLURM_TIME_LIMIT,
+                    "set":      True,
+                    "infinite": False,
+                },
+                "partition": self.device,
+                "current_working_directory": "/data/jobs",
+                "standard_output": f"/data/jobs/{self.job_uuid}.out",
+                "standard_error":  f"/data/jobs/{self.job_uuid}.err",
+            },
+            "script": bash,
+        }
+
+        resp = requests.post(
+            f"{self.slurm.base_url}/job/submit",
+            headers=self.slurm.auth_headers(),
+            json=payload,
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+        slurm_job_id = data.get("job_id") or data.get("jobId")
+
+        if not slurm_job_id:
+            raise RuntimeError(f"Slurm submit response missing job_id: {data}")
+
+        self.logger.info(f"Slurm job submitted: slurm_job_id={slurm_job_id}")
+        return {"slurm_job_id": str(slurm_job_id), "job_uuid": self.job_uuid}
+
+    def _get_provider_job_id(self, job: dict) -> str:
+        return job["slurm_job_id"]
+
+    def _get_job_status(self, job: dict) -> str:
+        state = self._fetch_slurm_state(job["slurm_job_id"])
+        return _SLURM_TERMINAL_TO_JOB_STATUS.get(state, state)
+
+    def _get_job_result(self, job: dict):
+        """Block until Slurm job finishes, then download result from S3."""
+        slurm_job_id = job["slurm_job_id"]
+        job_uuid     = job["job_uuid"]
+
+        elapsed = 0
+        while elapsed < SLURM_TIMEOUT_SEC:
+            state = self._fetch_slurm_state(slurm_job_id)
+            self.logger.debug(f"Slurm job {slurm_job_id} state: {state}")
+
+            if state == _DONE_STATE:
+                return self._download_s3_result(job_uuid)
+
+            if state in _TERMINAL_STATES:
+                raise RuntimeError(f"Slurm job {slurm_job_id} ended with state: {state}")
+
+            time.sleep(SLURM_POLL_SEC)
+            elapsed += SLURM_POLL_SEC
+
+        raise TimeoutError(f"Slurm job {slurm_job_id} timed out after {SLURM_TIMEOUT_SEC}s")
+
+    def _produce_histogram_data(self, job_result) -> None:
+        return None
+
+    def _calculate_execution_time(self, job_result) -> None:
+        self.execution_time = None
+
+    # ── SBATCH script builder ─────────────────────────────────────────────────
+
+    def _build_sbatch_script(self, user_script: str) -> str:
+        """Build the full SBATCH bash script that wraps the user's computation.
+
+        Container resolution (see _resolve_compute_container):
+          - Default: per-function SIF built at deploy time, identified by env
+            COMPUTE_SIF_PATH + COMPUTE_ENTRYPOINT + COMPUTE_SCRIPT_EXT
+          - Override: invocation `container` block can point at a shared SIF or
+            pull a Docker image at runtime
+
+        The user script string is base64-encoded into the SBATCH script and
+        written to /tmp/quapp_job.<ext> on the compute node, then executed
+        through Apptainer with the configured entrypoint.
+        """
+        resources      = self.hpc_config.get("resources", {})
+        container      = self.hpc_config.get("container", {})
+        environment    = self.hpc_config.get("environment", {})
+        input_s3_paths = self.hpc_config.get("input_s3_paths", [])
+
+        sif_path, entrypoint, script_ext = _resolve_compute_container(container)
+        script_file = f"/tmp/quapp_job.{script_ext}"
+
+        # Entrypoint may reference the canonical filename; rewrite to actual extension
+        entrypoint_resolved = entrypoint.replace("/tmp/quapp_job.py", script_file) \
+                                       .replace("/tmp/quapp_job.sh", script_file)
+
+        self.logger.info(
+            "Compute container resolved: sif=%s entrypoint=%s script_file=%s",
+            sif_path, entrypoint_resolved, script_file,
+        )
+
+        lines = ["#!/bin/bash"]
+
+        # ── SBATCH directives ─────────────────────────────────────────────────
+        gpus = int(resources.get("gpus", 0))
+        memory_mb = (resources.get("memoryMb")
+                     or resources.get("memory_mb")
+                     or int(resources.get("memory_gb", 4)) * 1024)
+        lines += [
+            f"#SBATCH --account=quapp",
+            f"#SBATCH --partition={resources.get('partition', 'cpu')}",
+            f"#SBATCH --nodes={resources.get('nodes', 1)}",
+            f"#SBATCH --ntasks={resources.get('ntasks', 1)}",
+            f"#SBATCH --cpus-per-task={resources.get('cpus_per_task', 1)}",
+            f"#SBATCH --mem={memory_mb}M",
+            f"#SBATCH --time={resources.get('walltime') or resources.get('time_limit', '01:00:00')}",
+        ]
+        if gpus > 0:
+            gpu_type = resources.get("gpu_type", "")
+            gres = f"gpu:{gpu_type}:{gpus}" if gpu_type else f"gpu:{gpus}"
+            lines.append(f"#SBATCH --gres={gres}")
+        lines.append("")
+
+        # ── Runtime setup ─────────────────────────────────────────────────────
+        lines += ["set -euo pipefail", "module load apptainer 2>/dev/null || true", ""]
+
+        # ── Download S3 input files ───────────────────────────────────────────
+        if input_s3_paths:
+            lines.append("mkdir -p /tmp/quapp_inputs")
+            for path in input_s3_paths:
+                lines.append(f"aws s3 cp {shlex.quote(path)} /tmp/quapp_inputs/")
+            lines.append("")
+
+        # ── Extra environment variables ───────────────────────────────────────
+        for k, v in environment.items():
+            lines.append(f"export {k}={shlex.quote(str(v))}")
+        if environment:
+            lines.append("")
+
+        # ── Write and run user script inside Apptainer ────────────────────────
+        b64 = base64.b64encode(user_script.encode()).decode()
+        exec_prefix = f"apptainer exec {shlex.quote(sif_path)}" \
+            if not sif_path.startswith("docker://") \
+            else f"apptainer exec {sif_path}"
+        lines += [
+            f"echo {shlex.quote(b64)} | base64 -d > {script_file}",
+            f"chmod +x {script_file}",
+            "",
+            f"{exec_prefix} {entrypoint_resolved} > /tmp/quapp_stdout.txt 2>/tmp/quapp_stderr.txt || true",
+            "echo $? > /tmp/quapp_exit_code.txt",
+            "",
+        ]
+
+        # ── Collect output and upload to S3 ──────────────────────────────────
+        s3_uri = f"s3://{self.s3_bucket}/{self.job_uuid}/output.json"
+        lines += [
+            f"python3 -c {shlex.quote(_COLLECT_PY)}",
+            f"aws s3 cp /tmp/output.json {shlex.quote(s3_uri)}",
+        ]
+
+        return "\n".join(lines)
+
+    # ── Helpers ───────────────────────────────────────────────────────────────
+
+    def _fetch_slurm_state(self, slurm_job_id: str) -> str:
+        resp = requests.get(
+            f"{self.slurm.base_url}/job/{slurm_job_id}",
+            headers=self.slurm.auth_headers(),
+            timeout=15,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+
+        jobs = data.get("jobs", [data])
+        if not jobs:
+            return "UNKNOWN"
+
+        raw_state = jobs[0].get("job_state", "UNKNOWN")
+        if isinstance(raw_state, list):
+            raw_state = raw_state[0] if raw_state else "UNKNOWN"
+
+        return str(raw_state).strip()
+
+    def _download_s3_result(self, job_uuid: str) -> dict:
+        s3_key = f"{job_uuid}/output.json"
+        self.logger.info(f"Downloading result from s3://{self.s3_bucket}/{s3_key}")
+
+        s3   = boto3.client("s3", region_name=AWS_REGION)
+        resp = s3.get_object(Bucket=self.s3_bucket, Key=s3_key)
+        return json.loads(resp["Body"].read())

quapp_hpc-0.0.1/quapp_hpc/model/provider/slurm_provider.py

@@ -0,0 +1,40 @@
+import os
+import requests
+from quapp_common.config.logging_config import job_logger
+from quapp_common.enum.provider_tag import ProviderTag
+from quapp_common.model.provider.provider import Provider
+
+logger = job_logger('SlurmProvider')
+
+SLURM_API_URL  = os.getenv("SLURM_API_URL",  "http://10.1.0.15:6820")
+SLURM_API_VER  = os.getenv("SLURM_API_VER",  "v0.0.40")
+SLURM_USERNAME = os.getenv("SLURM_USERNAME", "quapp-svc")
+SLURM_ACCOUNT  = os.getenv("SLURM_ACCOUNT",  "quapp")
+
+
+class SlurmProvider(Provider):
+
+    def __init__(self, jwt_token: str):
+        super().__init__(ProviderTag.QUAPP_HPC)
+        self.jwt_token = jwt_token
+        self.base_url  = f"{SLURM_API_URL}/slurm/{SLURM_API_VER}"
+
+    def get_backend(self, device_specification: str) -> str:
+        # device_specification maps to Slurm partition name
+        return device_specification or "compute"
+
+    def collect_provider(self):
+        resp = requests.get(
+            f"{SLURM_API_URL}/ping",
+            timeout=10,
+        )
+        resp.raise_for_status()
+        logger.debug(f"Slurm API reachable: {resp.json()}")
+        return self
+
+    def auth_headers(self) -> dict:
+        return {
+            "X-SLURM-USER-NAME":  SLURM_USERNAME,
+            "X-SLURM-USER-TOKEN": self.jwt_token,
+            "Content-Type":       "application/json",
+        }

quapp_hpc-0.0.1/quapp_hpc.egg-info/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.4
+Name: quapp-hpc
+Version: 0.0.1
+Summary: Quapp HPC library — Slurm integration for Quapp Platform
+Author-email: "CITYNOW Co. Ltd." <corp@citynow.vn>
+License: The MIT License (MIT)
+        Copyright © CITYNOW Co. Ltd. All rights reserved.
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+Project-URL: Homepage, https://quapp.cloud/
+Keywords: quapp,quapp-hpc,slurm,hpc
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.13,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: quapp-common==0.0.13
+Requires-Dist: requests>=2.31.0
+Requires-Dist: boto3>=1.28.0
+Provides-Extra: dev
+Requires-Dist: black; extra == "dev"
+Requires-Dist: bumpver; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: pip-tools; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# quapp-hpc
+
+Python library cho Quapp HPC functions — cầu nối giữa Quapp FaaS platform và Slurm HPC cluster.
+
+## Architecture
+
+```
+ksvc (Docker)
+├── index.py                  FastAPI server
+├── quapp_hpc/
+│   ├── factory/
+│   │   └── hpc_handler_factory.py   Entry point cho user
+│   ├── component/backend/
+│   │   └── hpc_invocation.py        Orchestrates job lifecycle
+│   └── model/
+│       ├── provider/slurm_provider.py   Auth headers, base URL
+│       └── device/slurm_device.py       Submit → Poll → S3 download
+└── function/
+    └── handler.py            User-defined processing() + post_processing()
+```
+
+## Luồng thực thi
+
+```
+index.py nhận HTTP POST
+    → HpcHandlerFactory.create_handler(event, processing_fn, post_processing_fn)
+    → InvocationHandler.handle()
+    → HpcInvocation.submit_job()
+        1. processing_fn(invocation_input)  → bash script string
+        2. SlurmDevice._create_job(script)  → POST Slurm REST API → slurm_job_id
+        3. SlurmDevice._get_job_result()    → poll mỗi 30s → COMPLETED/FAILED
+        4. SlurmDevice._download_s3_result()→ boto3 get s3://$S3_BUCKET/$JOB_UUID/output.json
+        5. post_processing_fn(s3_result)    → final response
+```
+
+## Environment variables (từ K8s Secret `slurm-credentials`)
+
+| Var | Ví dụ | Mô tả |
+|-----|-------|-------|
+| `SLURM_API_URL` | `http://10.1.0.15:6820` | Slurm REST API base URL |
+| `SLURM_JWT` | `eyJ...` | JWT token cho Slurm auth |
+| `SLURM_USERNAME` | `quapp-svc` | Slurm username |
+| `SLURM_ACCOUNT` | `quapp` | Slurm account/allocation |
+| `S3_BUCKET` | `quapp-slurm-output-dev` | S3 bucket cho job output |
+| `AWS_REGION` | `ap-southeast-1` | AWS region |
+| `SLURM_POLL_SEC` | `30` | Polling interval (giây) |
+| `SLURM_TIMEOUT_SEC` | `21600` | Max wait time (giây, default 6h) |
+
+## invocation_input schema
+
+Xem chi tiết tại [`../qapp-sdk-templates/slurm-hpc/README.md`](../../qapp-sdk-templates/slurm-hpc/README.md).
+
+Tóm tắt:
+```json
+{
+  "resources": { "partition", "nodes", "cpus_per_task", "gpus", "memory_gb", "time_limit" },
+  "container": { "type": "sif"|"docker"|"none", "image": "..." },
+  "job":       { "type": "script"|"command", "script"|"command": "...", "environment": {}, "input_s3_paths": [] }
+}
+```
+
+## Slurm REST API
+
+- Version: `v0.0.40`
+- Submit: `POST {SLURM_API_URL}/slurm/v0.0.40/job/submit`
+- Status: `GET {SLURM_API_URL}/slurm/v0.0.40/job/{job_id}`
+- Auth headers: `X-SLURM-USER-NAME`, `X-SLURM-USER-TOKEN`
+
+### Job state mapping
+
+| Slurm state | Quapp state |
+|---|---|
+| PENDING, CONFIGURING, RUNNING, COMPLETING | RUNNING |
+| COMPLETED | DONE |
+| FAILED, CANCELLED, TIMEOUT, NODE_FAIL, PREEMPTED | ERROR |
+
+## S3 output pattern
+
+Job script phải upload kết quả:
+```bash
+aws s3 cp /tmp/output.json s3://$S3_BUCKET/$JOB_UUID/output.json
+```
+
+`$JOB_UUID` và `$S3_BUCKET` được inject tự động bởi `SlurmDevice._create_job()` qua Slurm `environment` array.
+
+## K8s Secret
+
+```yaml
+# infrastructure/quapp-job-scheduler/k8s/cts/slurm-secret.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: slurm-credentials
+  namespace: quapp-functions-dev
+stringData:
+  SLURM_JWT: "<generate: sudo scontrol token username=quapp-svc lifespan=2592000>"
+  SLURM_API_URL: "http://10.1.0.15:6820"
+  SLURM_USERNAME: "quapp-svc"
+  SLURM_ACCOUNT: "quapp"
+  S3_BUCKET: "quapp-slurm-output-dev"
+  AWS_REGION: "ap-southeast-1"
+```
+
+## DB seed required
+
+Chạy script trước khi deploy:
+```
+infrastructure/quapp-functions-backend/docs/db/seed_slurm_hpc.sql
+```

quapp_hpc-0.0.1/quapp_hpc.egg-info/SOURCES.txt

@@ -0,0 +1,27 @@
+LICENSE
+README.md
+pyproject.toml
+quapp_hpc/__init__.py
+quapp_hpc.egg-info/PKG-INFO
+quapp_hpc.egg-info/SOURCES.txt
+quapp_hpc.egg-info/dependency_links.txt
+quapp_hpc.egg-info/requires.txt
+quapp_hpc.egg-info/top_level.txt
+quapp_hpc/component/__init__.py
+quapp_hpc/component/backend/__init__.py
+quapp_hpc/component/backend/hpc_invocation.py
+quapp_hpc/component/backend/slurm_job_fetching.py
+quapp_hpc/factory/__init__.py
+quapp_hpc/factory/hpc_device_factory.py
+quapp_hpc/factory/hpc_handler_factory.py
+quapp_hpc/factory/hpc_provider_factory.py
+quapp_hpc/handler/__init__.py
+quapp_hpc/handler/invocation_handler.py
+quapp_hpc/handler/job_fetching_handler.py
+quapp_hpc/model/__init__.py
+quapp_hpc/model/device/__init__.py
+quapp_hpc/model/device/slurm_device.py
+quapp_hpc/model/provider/__init__.py
+quapp_hpc/model/provider/slurm_provider.py
+quapp_slurm_hpc/__init__.py
+quapp_slurm_hpc/runner.py

quapp_hpc-0.0.1/quapp_hpc.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

quapp_hpc-0.0.1/quapp_hpc.egg-info/requires.txt

@@ -0,0 +1,10 @@
+quapp-common==0.0.13
+requests>=2.31.0
+boto3>=1.28.0
+
+[dev]
+black
+bumpver
+isort
+pip-tools
+pytest

quapp_hpc-0.0.1/quapp_hpc.egg-info/top_level.txt

@@ -0,0 +1,2 @@
+quapp_hpc
+quapp_slurm_hpc

quapp_hpc-0.0.1/quapp_slurm_hpc/runner.py

@@ -0,0 +1,142 @@
+"""CLI runner — invoked by the sbatch script inside the Apptainer container.
+
+The Java SlurmPlatformManager generates a sbatch script that runs:
+
+    apptainer exec docker://<image> python -m quapp_slurm_hpc.runner \
+        --input-file  /data/jobs/<jobId>/input.json  \
+        --output-file /data/jobs/<jobId>/output.json \
+        --handler-dir /home/app/function
+
+This module:
+  1. Reads the invocation input from --input-file (JSON written by the sbatch script).
+  2. Loads handler.py from --handler-dir.
+  3. Calls handler.processing(invocation_input) → bash script string.
+  4. Executes the bash script, captures stdout / stderr / exit_code.
+  5. Calls handler.post_processing(raw_result) if it exists.
+  6. Writes the final result dict to --output-file.
+  7. Uploads output.json to S3 (key: {QUAPP_JOB_ID}/output.json) when SLURM_S3_BUCKET is set.
+"""
+import argparse
+import importlib.util
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import traceback
+from pathlib import Path
+from types import ModuleType
+
+
+def _load_handler(handler_dir: str) -> ModuleType:
+    handler_path = Path(handler_dir) / "handler.py"
+    if not handler_path.exists():
+        raise FileNotFoundError(
+            f"handler.py not found in {handler_dir!r}. "
+            "Ensure the function image was built with the handler in /home/app/function."
+        )
+    if str(handler_dir) not in sys.path:
+        sys.path.insert(0, str(handler_dir))
+    spec = importlib.util.spec_from_file_location("handler", handler_path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def _run_script(script: str) -> dict:
+    """Write script to a temp file and execute it with bash, capturing output."""
+    with tempfile.NamedTemporaryFile(
+        mode="w", suffix=".sh", delete=False, encoding="utf-8"
+    ) as f:
+        f.write(script)
+        tmp_path = f.name
+    try:
+        result = subprocess.run(
+            ["bash", tmp_path],
+            capture_output=True,
+            text=True,
+        )
+        return {
+            "exit_code": result.returncode,
+            "stdout": result.stdout,
+            "stderr": result.stderr,
+        }
+    finally:
+        Path(tmp_path).unlink(missing_ok=True)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Quapp HPC runner — executes handler.py inside an Apptainer container"
+    )
+    parser.add_argument("--input-file",  required=True, help="Path to input JSON file")
+    parser.add_argument("--output-file", required=True, help="Path to write output JSON")
+    parser.add_argument("--handler-dir", required=True, help="Directory containing handler.py")
+    args = parser.parse_args()
+
+    output: dict = {}
+    try:
+        with open(args.input_file, encoding="utf-8") as f:
+            invocation_input = json.load(f)
+
+        handler = _load_handler(args.handler_dir)
+
+        if not hasattr(handler, "processing"):
+            raise AttributeError(
+                "handler.py must define a processing(invocation_input: dict) -> str function"
+            )
+
+        script = handler.processing(invocation_input)
+        if not isinstance(script, str):
+            raise TypeError(
+                f"handler.processing() must return a str (bash script), got {type(script).__name__}"
+            )
+
+        raw_result = _run_script(script)
+
+        if hasattr(handler, "post_processing"):
+            output = handler.post_processing(raw_result)
+        else:
+            output = raw_result
+
+    except Exception:
+        output = {
+            "exit_code": 1,
+            "stdout": "",
+            "stderr": traceback.format_exc(),
+        }
+
+    with open(args.output_file, "w", encoding="utf-8") as f:
+        json.dump(output, f)
+
+    _upload_to_s3(args.output_file)
+
+
+def _upload_to_s3(output_file: str) -> None:
+    """Upload output.json to S3 so the Java backend can read the result.
+
+    Requires SLURM_S3_BUCKET and QUAPP_JOB_ID env vars (injected by sbatch script).
+    Skips silently when the bucket is not configured.
+    """
+    s3_bucket = os.getenv("SLURM_S3_BUCKET", "")
+    job_uuid = os.getenv("QUAPP_JOB_ID", "")
+    if not s3_bucket or not job_uuid:
+        return
+
+    try:
+        import boto3
+        s3 = boto3.client(
+            "s3",
+            region_name=os.getenv("SLURM_S3_REGION", "ap-southeast-1"),
+            aws_access_key_id=os.getenv("SLURM_S3_ACCESS_KEY"),
+            aws_secret_access_key=os.getenv("SLURM_S3_SECRET_KEY"),
+        )
+        s3_key = f"{job_uuid}/output.json"
+        s3.upload_file(output_file, s3_bucket, s3_key)
+        print(f"[runner] Uploaded output to s3://{s3_bucket}/{s3_key}", flush=True)
+    except Exception as exc:
+        print(f"[runner] WARNING: S3 upload failed: {exc}", file=sys.stderr, flush=True)
+
+
+if __name__ == "__main__":
+    main()

quapp_hpc-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+