Py4HEAppE 2.9.0__tar.gz → 3.1.0__tar.gz

py4heappe-3.1.0/.gitlab/stages/.generation.yml

@@ -0,0 +1,34 @@
+## GENERATION STAGE
+
+API_client_generation:
+  stage: API_generation
+  image: eclipse-temurin:11-jdk
+  when: manual
+  rules:
+    - if: '$CI_COMMIT_BRANCH != "develop" && $CI_COMMIT_BRANCH != "main"'
+  before_script:
+    - |
+      if [ -x "$(command -v dnf)" ]; then
+        dnf -y install wget zip
+      elif [ -x "$(command -v yum)" ]; then
+        yum -y install wget zip
+      else
+        export DEBIAN_FRONTEND=noninteractive
+        apt-get update -qq
+        apt-get install -y --no-install-recommends wget zip
+        rm -rf /var/lib/apt/lists/*
+      fi
+    - wget https://repo1.maven.org/maven2/io/swagger/codegen/v3/swagger-codegen-cli/3.0.71/swagger-codegen-cli-3.0.71.jar -O /swagger-codegen-cli.jar
+  script:
+    - echo "Generation API client"
+    - java -jar /swagger-codegen-cli.jar generate -i Tools/APIWrapperGen/swagger.json -l python -t Tools/APIWrapperGen/Templates/python/ -o ./app/client -c Tools/APIWrapperGen/config.json
+    - cd app/client/py4heappe
+    - zip -r ../../../APIWrapperGen.zip .
+  tags:
+    - Docker
+    - Centos9
+    - IT4I_Network
+  artifacts:
+    paths:
+      - APIWrapperGen.zip
+    expire_in: 1 days

py4heappe-3.1.0/CHANGELOG.md

@@ -0,0 +1,141 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## 3.1.0
+
+### Added
+- CLI integration tests (live CLI examples) under `tests/` with helper `tests/live_cli_support.py` and a `tests/.env` template to simplify running live examples.
+- Updated README with instructions for running CLI tests and environment setup.
+
+### Changed
+- Documented test-runner defaults (use Python's `unittest`) and optional `pytest` support.
+
+
+## 3.0.0
+
+### Compatible with HEAppE V6.5.X, V6.4.X, V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Changed
+- Updated Py4HEAppE wrapper to support new changes in HEAppE Version 6.5.0 [See HEAppE v6.4.0 CHANGELOG.md](https://github.com/It4innovations/HEAppE/blob/master/CHANGELOG.md#v650)
+
+
+## 2.9.0
+
+### Compatible with HEAppE V6.4.X, V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Added
+- Added `--full/--minimal` to `py4heappe Job InitJobSpecification` so the initializer can generate either the original minimal template or a full swagger-shaped job specification.
+- Added `--save-result-job-specification` to `py4heappe Job Create` to persist the final parsed job specification JSON in the current working directory before submission.
+
+## 2.8.0
+
+### Compatible with HEAppE V6.4.X, V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Added
+- Added `py4heappe Job InitJobSpecification` to generate a minimal JSON job specification template in the current directory or in a user-provided destination.
+
+### Changed
+- Expanded `py4heappe Job Create` to accept an optional JSON job specification file and merge CLI overrides on top of values loaded from that file.
+- Added optional CLI overrides for the extended HEAppE V6 job specification, including indexed overrides for task attributes and nested collections such as environment variables, template parameter values, required nodes, task parallelization parameters, and task dependencies.
+- Switched job specification parsing to build requests from the generated HEAppE V6 models, including generated-model handling for `Priority` and `DependsOn`.
+- Replaced the previous `Job Remove` behavior with `Job Delete`, including support for the `--archive-logs` option.
+
+## 2.7.1
+
+### Compatible with HEAppE V6.4.X, V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Fixed
+- Fixed syntax error (`IndentationError`) in `management_api.py` caused by an empty `if` block for single file upload parameters.
+- Fixed the Swagger Codegen mustache template to correctly support single file (`isBinary`), list of files (`items.isBinary`), and regular form parameters.
+
+## 2.7.0
+
+### Compatible with HEAppE V6.4.X, V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Changed
+- Updated Py4HEAppE wrapper to support new changes in HEAppE V6.4.0 [See HEAppE v6.4.0 CHANGELOG.md](https://github.com/It4innovations/HEAppE/blob/master/CHANGELOG.md#v640)
+
+
+## 2.6.0
+
+### Compatible with HEAppE V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Added
+
+- **File Transfer CLI Commands**: Added full support for the `FileTransfer` HEAppE API suite to manage cluster data channels and job files:
+  - `py4heappe FileTransfer Single/Interactive` - Initiates and creates a secure file transfer tunnel via SSH and make a one-time transfer of file(s) or folder (Single) or interactively multiple times (Interactive) (`POST /heappe/FileTransfer/RequestFileTransfer`).
+  - Automatically closes an active file transfer tunnel after transfer (`POST /heappe/FileTransfer/CloseFileTransfer`).
+  - `py4heappe FileTransfer DownloadParts` - Downloads specific parts of job files from the cluster (`POST /heappe/FileTransfer/DownloadPartsOfJobFilesFromCluster`).
+  - `py4heappe FileTransfer ListChanged` - Retrieves a list of all files modified during job execution (`GET /heappe/FileTransfer/ListChangedFilesForJob`).
+  - `py4heappe FileTransfer Download` - Downloads a specific individual file directly from the cluster (`POST /heappe/FileTransfer/DownloadFileFromCluster`).
+  - `py4heappe FileTransfer Stream` - Uploads files directly into a designated job execution directory (`POST /heappe/FileTransfer/UploadFilesToJobExecutionDir`).
+
+## 2.5.2
+
+### Compatible with HEAppE V6.3.X, V6.2.X, V6.2.1, V5.0.X, V4.3.X and V4.2.X
+
+### Added
+
+- Enhanced `ListAvailableClusters` response with cluster-specific storage paths (`ScratchStoragePath` and `ProjectStoragePath`) for each project.
+- Storage paths are now returned in a structured collection `ClusterProjectStoragePaths` within the `ProjectExt` model, including Cluster ID and Name.
+
+## 2.5.1
+
+### Compatible with HEAppE V6.2.X, V6.0.X, V5.0.X, V4.3.X and V4.2.X
+
+### Changed
+
+- Dependencies relaxation.
+
+## 2.5.0
+
+### Compatible with HEAppE V6.2.X, V6.0.X, V5.0.X, V4.3.X and V4.2.X
+
+## 2.4.0
+
+### Compatible with HEAppE V6.1.X, V6.0.X, V5.0.X, V4.3.X and V4.2.X
+
+## V2.3.0
+
+### Compatible with HEAppE V6.0.X, V5.0.X, V4.3.X and V4.2.X
+
+## V2.2.0
+
+### Compatible with HEAppE V5.0.X, V4.3.X and V4.2.X
+
+### Changed
+
+- Update requirements package
+- Adding File Commands Groups in CLI (support Job Management)
+
+## V2.1.2
+
+### Compatible with HEAppE V5.0.X, V4.3.X and V4.2.X
+
+### Changed
+
+- Requirements package (python-dotenv)
+
+## V2.1.1
+
+### Compatible with HEAppE V5.0.X, V4.3.X and V4.2.X
+
+### Fixed
+
+- Deserialization response for HEAppE 4.X.X
+
+## V2.1.0
+
+### Compatible with HEAppE V5.0.X, V4.3.X and V4.2.X
+
+## V2.0.0
+
+### Compatible with HEAppE V5.0.X
+
+## V1.0.0
+
+### Compatible with HEAppE V4.2.X and V4.3.X

py4heappe-3.1.0/PKG-INFO

@@ -0,0 +1,364 @@
+Metadata-Version: 2.4
+Name: Py4HEAppE
+Version: 3.1.0
+Summary: Py4HEAppE is client library for easy access to HEAppE Middleware
+Author-email: IT4Innovations <support.heappe@it4i.cz>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/It4innovations/Py4HEAppE
+Project-URL: Repository, https://github.com/It4innovations/Py4HEAppE.git
+Project-URL: Issues, https://github.com/It4innovations/Py4HEAppE/issues
+Keywords: LEXIS,HEAppE,Client Library
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: paramiko>=4.0.0
+Requires-Dist: scp==0.15.0
+Requires-Dist: urllib3>=2.0.2
+Requires-Dist: certifi>=2025.1.31
+Requires-Dist: six>=1.16.0
+Requires-Dist: click>=8.1.0
+Requires-Dist: typer>=0.15.0
+Requires-Dist: python-dotenv>=0.21.1
+Requires-Dist: validators==0.35.0
+Requires-Dist: typing-extensions==4.16.0
+Dynamic: license-file
+
+<img align="right" width="35%" src="https://raw.githubusercontent.com/It4innovations/Py4HEAppE/refs/heads/master/docs/imgs/logo.png">
+
+# Py4HEAppE (Python for HEAppE Middleware)
+Release: 3.1.0
+
+Py4HEAppE provides both a command-line interface and versioned Python wrappers for the [HEAppE](https://heappe.eu) middleware API.
+
+You can use it in two ways:
+- HEAppE CLI commands for interactive and scripting use
+- Versioned Python API wrappers for direct integration into your own applications
+
+## Supported HEAppE Versions
+| Py4HEAppE | HEAppE Version                                         | Notes |
+|:---------:|:-------------------------------------------------------| :---- |
+|   3.1.X   | 6.5.X, 6.4.X, 6.3.X, 6.2.X, 6.2.1, 5.0.X, 4.3.X, 4.2.X | Without Admin CLI section |
+|   2.9.X   | 6.4.X, 6.3.X, 6.2.X, 6.2.1, 5.0.X, 4.3.X, 4.2.X        | Without Admin CLI section |
+|   2.8.X   | 6.4.X, 6.3.X, 6.2.X, 6.2.1, 5.0.X, 4.3.X, 4.2.X        | Without Admin CLI section |
+|   2.7.X   | 6.4.X, 6.3.X, 6.2.X, 6.2.1, 5.0.X, 4.3.X, 4.2.X        | Without Admin CLI section |
+|   2.6.X   | 6.3.X, 6.2.X, 6.1.X, 5.0.X, 4.3.X, 4.2.X               | Without Admin CLI section |
+|   2.5.X   | 6.3.X, 6.2.X, 6.1.X, 5.0.X, 4.3.X, 4.2.X               | Without Admin CLI section |
+|   2.4.X   | 6.1.X, 6.0.X, 5.0.X, 4.3.X, 4.2.X                      | Without Admin CLI section |
+|   2.3.X   | 6.0.X, 5.0.X, 4.3.X, 4.2.X                             | Without Admin CLI section |
+|   2.2.X   | 5.0.X, 4.3.X, 4.2.X                                    | Without Admin CLI section |
+|   2.1.X   | 5.0.X, 4.3.X, 4.2.X                                    | Without Admin, Job, and File CLI sections |
+|   2.0.X   | 5.0.X                                                  | Without Admin, Job, and File CLI sections |
+|   1.X.X   | 4.3.X, 4.2.X                                           | Without Admin, Job, and File CLI sections |
+
+## Requirements
+- Python 3.11 or newer
+- Access to a deployed HEAppE instance
+- HEAppE instance URL
+- HEAppE accounting string for the target computational project
+
+## Installation
+If the `py4heappe` executable is not on your `PATH`, use the full path to the installed script or add the script directory to `PATH`.
+
+On Windows, use `py4heappe.exe` instead of `py4heappe`.
+
+## HEAppE CLI
+The CLI is intended for users who want to work with HEAppE directly from a terminal without writing Python code.
+
+### Initial Setup
+Before using the CLI for the first time, initialize the HEAppE instance URL and the accounting string:
+
+```shell
+py4heappe Conf Init
+```
+
+### Available Command Groups
+```shell
+py4heappe --help
+```
+
+Current command groups:
+
+- `Conf` for local CLI configuration
+- `Auth` for authentication
+- `CmdTemp` for command template operations
+- `Info` for version and cluster information
+- `Job` for job lifecycle management
+- `FileTransfer` for job file upload and download operations
+- `Report` for resource usage and reporting
+
+### Useful Help Commands
+
+```shell
+py4heappe Auth --help
+py4heappe Info --help
+py4heappe Job --help
+py4heappe FileTransfer --help
+py4heappe Report --help
+```
+
+### Basic CLI Examples
+
+Authenticate with username and password:
+
+```shell
+py4heappe Auth UserPass
+```
+
+Show the HEAppE API version:
+
+```shell
+py4heappe Info Version
+```
+
+List available clusters:
+
+```shell
+py4heappe Info ClusterInfo
+```
+
+List your jobs:
+
+```shell
+py4heappe Job List
+```
+
+### Create a Job from a JSON Template
+
+Py4HEAppE supports a JSON-first workflow for job creation.
+
+Generate a minimal job specification template in the current directory:
+
+```shell
+py4heappe Job InitJobSpecification
+```
+
+Generate the full swagger-shaped job specification template instead:
+
+```shell
+py4heappe Job InitJobSpecification --full
+```
+
+Or generate it in a specific directory or target file:
+
+```shell
+py4heappe Job InitJobSpecification --file-destination ./job_specs
+py4heappe Job InitJobSpecification --file-destination ./job_specs/demo_job.json
+```
+
+Create a job from the JSON file and override selected values from the command line:
+
+```shell
+py4heappe Job Create --json-job-spec-file ./job_specification.json \
+    --name demo-job \
+    --cluster-id 2 \
+    --project-id 1 \
+    --task-name 0:demo-task \
+    --task-max-cores 0:128 \
+    --walltime-limit 0:1800 \
+    --cluster-node-type-id 0:18 \
+    --cmd-template-id 0:3 \
+    --cmd-template-parameters inputParam:testValue
+```
+
+CLI overrides always take precedence over values loaded from `--json-job-spec-file`.
+
+Save the final parsed job specification that will be submitted to HEAppE:
+
+```shell
+py4heappe Job Create --json-job-spec-file ./job_specification.json \
+    --save-result-job-specification
+```
+
+This writes `resultJobSpecJson.json` into the current working directory.
+
+### Override Format for Nested Task Data
+
+Some `Job Create` options work on indexed task items and nested collections.
+
+Examples:
+
+```shell
+# Task scalar override
+py4heappe Job Create --json-job-spec-file ./job_specification.json --task-name 0:main-task
+
+# Task environment variable override
+py4heappe Job Create --json-job-spec-file ./job_specification.json \
+  --task-environment-variable 0:0:Name:OMP_NUM_THREADS \
+  --task-environment-variable 0:0:Value:8
+
+# Task template parameter override
+py4heappe Job Create --json-job-spec-file ./job_specification.json \
+  --task-template-parameter-value 0:0:CommandParameterIdentifier:inputParam \
+  --task-template-parameter-value 0:0:ParameterValue:testValue
+```
+
+### File Transfer Examples
+
+List files changed during job execution:
+
+```shell
+py4heappe FileTransfer ListChanged --help
+```
+
+Download a single file from a job directory:
+
+```shell
+py4heappe FileTransfer Download --help
+```
+
+Upload files directly into the job execution directory:
+
+```shell
+py4heappe FileTransfer Stream --help
+```
+
+## Python API Wrapper
+
+Py4HEAppE also ships versioned Python wrappers generated from the HEAppE API.
+
+Use the wrapper that matches your target HEAppE version:
+
+```python
+import py4heappe.heappe_v6.core as hp
+# or:
+# import py4heappe.heappe_v5.core as hp
+# import py4heappe.heappe_v4.core as hp
+```
+
+There is no single unversioned `py4heappe.core` module. Use the versioned wrapper explicitly.
+
+### Minimal Python Example
+
+```python
+import json
+
+import py4heappe.heappe_v6.core as hp
+
+configuration = hp.Configuration()
+configuration.host = "https://heappe.example.org"
+api_client = hp.ApiClient(configuration)
+
+try:
+    auth_api = hp.UserAndLimitationManagementApi(api_client)
+    management_api = hp.ManagementApi(api_client)
+    cluster_api = hp.ClusterInformationApi(api_client)
+
+    auth_body = {
+        "_preload_content": False,
+        "body": {
+            "Credentials": {
+                "Username": "username",
+                "Password": "password",
+            }
+        },
+    }
+
+    auth_response = auth_api.heappe_user_and_limitation_management_authenticate_user_password_post(
+        **auth_body
+    )
+    session_code = json.loads(auth_response.data)
+
+    version_response = management_api.heappe_management_version_information_get(
+        _preload_content=False,
+        SessionCode=session_code,
+    )
+    print(json.dumps(json.loads(version_response.data), indent=2))
+
+    cluster_response = cluster_api.heappe_cluster_information_list_available_clusters_get(
+        _preload_content=False,
+        SessionCode=session_code,
+    )
+    print(json.dumps(json.loads(cluster_response.data), indent=2))
+
+finally:
+    api_client.pool.close()
+    api_client.pool.join()
+```
+
+### Create a Job Through the Python Wrapper
+
+```python
+import json
+
+import py4heappe.heappe_v6.core as hp
+
+configuration = hp.Configuration()
+configuration.host = "https://heappe.example.org"
+api_client = hp.ApiClient(configuration)
+
+try:
+    job_api = hp.JobManagementApi(api_client)
+
+    body = {
+        "_preload_content": False,
+        "body": {
+            "SessionCode": "your-session-code",
+            "JobSpecification": {
+                "Name": "demo-job",
+                "ProjectId": 1,
+                "ClusterId": 2,
+                "FileTransferMethodId": 1,
+                "Tasks": [
+                    {
+                        "Name": "demo-task",
+                        "MinCores": 1,
+                        "MaxCores": 16,
+                        "WalltimeLimit": 1800,
+                        "StandardOutputFile": "stdout",
+                        "StandardErrorFile": "stderr",
+                        "ProgressFile": "stdprog",
+                        "LogFile": "stdlog",
+                        "ClusterNodeTypeId": 18,
+                        "CommandTemplateId": 3,
+                        "TemplateParameterValues": [
+                            {
+                                "CommandParameterIdentifier": "inputParam",
+                                "ParameterValue": "testValue",
+                            }
+                        ],
+                    }
+                ],
+            },
+        },
+    }
+
+    response = job_api.heappe_job_management_create_job_post(**body)
+    job_id = json.loads(response.data)["Id"]
+    print(f"Created job: {job_id}")
+
+finally:
+    api_client.pool.close()
+    api_client.pool.join()
+```
+
+## Running CLI tests
+
+The repository includes live CLI integration tests under `tests/` that require a reachable HEAppE instance and several environment variables.
+
+Quick steps:
+
+- Create a `tests/.env` from the example and edit it:
+  - Unix: `cp tests/.env.example tests/.env`
+  - Windows (PowerShell): `Copy-Item tests\.env.example tests\.env`
+- Populate required values in `tests/.env` (or export the variables). At minimum set `PY4HEAPPE_RUN_LIVE_CLI_TESTS=1` and `PY4HEAPPE_TEST_URL` and `PY4HEAPPE_TEST_PROJECT`. To allow tests that change remote state also set `PY4HEAPPE_TEST_RUN_MUTATING_COMMANDS=1`.
+- Run the CLI tests (default, using the standard library's unittest):
+  - Run all CLI tests: `python -m unittest discover -s tests -p "test_cli_*.py" -v`
+  - Run a single test module: `python -m unittest tests.test_cli_auth -v`
+
+- Optional: run with pytest (pytest can run unittest-style tests and offers improved output):
+  - Install pytest: `python -m pip install -U pytest`
+  - Run with pytest: `python -m pytest -q tests/test_cli_*.py`
+
+Do NOT commit `tests/.env` with real credentials or session tokens.
+
+## Notes
+
+- CLI configuration is stored locally after `py4heappe Conf Init`.
+- Most CLI commands require a valid session, so authenticate before running job or file-transfer operations.
+- For detailed option lists, use the built-in `--help` on each command group and subcommand.
+
+## Acknowledgement
+
+This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through e-INFRA CZ (ID:90254).