etiket-sync-agent-qualibrate 0.0.1__tar.gz

etiket_sync_agent_qualibrate-0.0.1/LICENCE

@@ -0,0 +1,34 @@
+All Rights Reserved License
+Copyright ©️ 2026 Quantum Machines Ltd. All Rights Reserved.
+
+Agreement to Terms
+By accessing, downloading, installing, or viewing this Software (including its source code, binaries, or application files), you acknowledge and agree to the terms outlined below.
+
+Terms and Conditions
+This software and its source code (the "Software") are the exclusive property of Quantum Machines Ltd. and are protected by copyright and other intellectual property laws.
+
+License and Testing Exemptions
+Commercial License: If you have entered into a separate commercial license agreement with Quantum Machines Ltd., the terms of that agreement shall supersede the restrictions listed below.
+Testing Permission: If you have obtained written permission for testing/evaluation from Quantum Machines Ltd., you are permitted to install and use the Software for evaluation purposes. However, this permission strictly excludes the right to modify, alter, create derivative works, or reverse engineer the Software.
+
+Prohibited Actions
+Unless explicitly authorized by the exemptions above, you are NOT permitted to:
+	•	Copy, reproduce, or duplicate the Software in any form (except as reasonably necessary for viewing or authorized installation)
+	•	Modify, alter, or create derivative works based on the Software
+	•	Distribute, publish, or share the Software with others
+	•	Reverse engineer, decompile, or disassemble the Software
+	•	Use the Software for any commercial or non-commercial purposes
+	•	Transfer, sell, lease, or sublicense the Software
+	•	Remove or alter any copyright notices or proprietary markings
+
+Viewing Only 
+For those without a commercial license or written testing permission, the Software is made available for viewing and reference purposes only. Any access to view the source code or application does not grant any rights to use, copy, or modify the Software.
+
+No Implied Rights
+No rights are granted by implication, estoppel, or otherwise. All rights not expressly granted are reserved by the copyright holder.
+
+Governing Law 
+This license and any disputes arising from it shall be governed by the laws of the Netherlands.
+
+Disclaimer 
+This Software is provided "AS IS" without warranty of any kind. The copyright holder disclaims all warranties and shall not be liable for any damages arising from the use or inability to use this Software.

etiket_sync_agent_qualibrate-0.0.1/PKG-INFO

@@ -0,0 +1,136 @@
+Metadata-Version: 2.4
+Name: etiket_sync_agent_qualibrate
+Version: 0.0.1
+Summary: QUAlibrate connector for eTiKeT sync agent
+Author: QHarbor team
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://qharbor.nl
+Project-URL: Documentation, https://docs.qharbor.nl
+Keywords: etiket,sync,connector,qualibrate,quantum
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENCE
+Requires-Dist: etiket_sync_agent>=0.3.0b1
+Requires-Dist: numpy
+Requires-Dist: xarray
+Requires-Dist: h5netcdf
+Requires-Dist: Pillow
+Dynamic: license-file
+
+# eTiKeT Sync Agent - QUAlibrate Connector
+
+Connector for synchronizing [QUAlibrate](https://qua-platform.github.io/qualibrate/) calibration data with the eTiKeT platform. This connector scans a QUAlibrate data directory for node/workflow snapshots and syncs them to the cloud.
+
+## Installation
+
+Install the QUAlibrate connector using the eTiKeT Sync SDK:
+
+```python
+from etiket_sdk.sync import Connectors
+
+# Install the latest version
+Connectors.install_from_pypi("etiket-sync-agent-qualibrate")
+```
+
+The package is automatically discovered by `etiket_sync_agent` through the entry-point system. Once installed, you can verify with:
+
+```python
+# List installed connectors
+print(Connectors.list())
+
+# Get details for the QUAlibrate connector
+connector = Connectors.get("etiket_sync_agent_qualibrate")
+print(connector)
+```
+
+### Updating the Connector
+
+```python
+# Update to the latest version
+Connectors.update_from_pypi("etiket-sync-agent-qualibrate")
+```
+
+## What Gets Synchronized
+
+Each QUAlibrate snapshot is a directory containing a `node.json` (metadata) and a `data.json` (results, with references to `.npz`/`.h5`/`.png`/`.json` files). When a snapshot is synced, the following data is extracted and uploaded:
+
+| QUAlibrate Data | eTiKeT Field | Description |
+|-----------------|--------------|-------------|
+| Snapshot folder name | `alt_uid` | Unique identifier for the snapshot |
+| `metadata.name` (or folder name) | `name` | Name of the dataset |
+| `created_at` / `metadata.run_start` | `collected` | When the snapshot was created |
+| `metadata.type_of_execution` | `keywords` | `node` or `workflow` |
+| `metadata.description` | `description` | Snapshot description |
+| `type_of_execution`, `status` | `attributes` | Small, searchable scalar metadata |
+| `data.json` → `raw_data` | HDF5 file | Raw measurement arrays combined into a single netCDF (`raw_data.h5`) |
+
+### Data Processing
+
+- **Reference resolution**: `data.json` reference strings (`./arrays.npz#a.b.c`) are resolved into real numpy arrays, xarray datasets, and images via the vendored loader.
+- **Raw-data extraction**: Only the `raw_data` entry of the results tree is converted to netCDF. Other entries (e.g. `fit_results`) are intentionally excluded.
+- **Array-to-dataset convention** (npz format): this is a best-effort reconstruction. Within a group of sibling numpy arrays, the leading 1-D arrays are treated as the coordinate axes in order (x, y, z, ...) and the remaining arrays as the measured values over the grid those axes span. If shapes don't fit this convention, arrays fall back to standalone variables with independent dimensions.
+- **Combining**: per-measurement datasets are merged into one flat netCDF. Data variables are prefixed by their results-path; shared sweep axes stay shared (`join="exact"`). On conflict, every dataset is fully namespaced (lossless, collision-proof).
+- **Workflows**: workflow snapshots typically carry no array data, in which case no netCDF file is uploaded.
+
+---
+
+## Configuration
+
+The QUAlibrate connector requires a `QualibrateConfigData` configuration with the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `data_directory` | `Path` or `str` | Yes | Path to the QUAlibrate data directory (the root containing snapshot folders) |
+| `is_server_folder` | `bool` | Yes | Whether this is a server folder (e.g. on a network drive of the university) |
+
+A scope is **required** for this connector (`scope_requirement = REQUIRED`); scope mapping is not supported.
+
+The configuration is validated on creation: `data_directory` must exist and be a directory, and it must not overlap (be identical to, a parent of, or a subdirectory of) the path of an existing QUAlibrate sync source.
+
+### Example Configuration
+
+Example using the `etiket-sdk` package:
+
+```python
+from etiket_sdk.sync import SyncSources
+
+SyncSources.create(
+    name="my_qualibrate_source",
+    connector_identifier="etiket_sync_agent_qualibrate",
+    config_data={
+        "data_directory": "~/.qualibrate/user_storage",
+        "is_server_folder": False
+    },
+    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+)
+```
+
+## Live Sync
+
+Live synchronization is **not** supported by this connector. Snapshots are synced once they are complete (i.e. once a `node.json` is present in the snapshot directory).
+
+## Requirements
+
+- Python >= 3.10
+- numpy
+- xarray
+- h5netcdf
+- Pillow
+
+## License
+
+Copyright © 2025 QHarbor. All Rights Reserved. See [LICENCE](LICENCE) for details.

etiket_sync_agent_qualibrate-0.0.1/README.md

@@ -0,0 +1,103 @@
+# eTiKeT Sync Agent - QUAlibrate Connector
+
+Connector for synchronizing [QUAlibrate](https://qua-platform.github.io/qualibrate/) calibration data with the eTiKeT platform. This connector scans a QUAlibrate data directory for node/workflow snapshots and syncs them to the cloud.
+
+## Installation
+
+Install the QUAlibrate connector using the eTiKeT Sync SDK:
+
+```python
+from etiket_sdk.sync import Connectors
+
+# Install the latest version
+Connectors.install_from_pypi("etiket-sync-agent-qualibrate")
+```
+
+The package is automatically discovered by `etiket_sync_agent` through the entry-point system. Once installed, you can verify with:
+
+```python
+# List installed connectors
+print(Connectors.list())
+
+# Get details for the QUAlibrate connector
+connector = Connectors.get("etiket_sync_agent_qualibrate")
+print(connector)
+```
+
+### Updating the Connector
+
+```python
+# Update to the latest version
+Connectors.update_from_pypi("etiket-sync-agent-qualibrate")
+```
+
+## What Gets Synchronized
+
+Each QUAlibrate snapshot is a directory containing a `node.json` (metadata) and a `data.json` (results, with references to `.npz`/`.h5`/`.png`/`.json` files). When a snapshot is synced, the following data is extracted and uploaded:
+
+| QUAlibrate Data | eTiKeT Field | Description |
+|-----------------|--------------|-------------|
+| Snapshot folder name | `alt_uid` | Unique identifier for the snapshot |
+| `metadata.name` (or folder name) | `name` | Name of the dataset |
+| `created_at` / `metadata.run_start` | `collected` | When the snapshot was created |
+| `metadata.type_of_execution` | `keywords` | `node` or `workflow` |
+| `metadata.description` | `description` | Snapshot description |
+| `type_of_execution`, `status` | `attributes` | Small, searchable scalar metadata |
+| `data.json` → `raw_data` | HDF5 file | Raw measurement arrays combined into a single netCDF (`raw_data.h5`) |
+
+### Data Processing
+
+- **Reference resolution**: `data.json` reference strings (`./arrays.npz#a.b.c`) are resolved into real numpy arrays, xarray datasets, and images via the vendored loader.
+- **Raw-data extraction**: Only the `raw_data` entry of the results tree is converted to netCDF. Other entries (e.g. `fit_results`) are intentionally excluded.
+- **Array-to-dataset convention** (npz format): this is a best-effort reconstruction. Within a group of sibling numpy arrays, the leading 1-D arrays are treated as the coordinate axes in order (x, y, z, ...) and the remaining arrays as the measured values over the grid those axes span. If shapes don't fit this convention, arrays fall back to standalone variables with independent dimensions.
+- **Combining**: per-measurement datasets are merged into one flat netCDF. Data variables are prefixed by their results-path; shared sweep axes stay shared (`join="exact"`). On conflict, every dataset is fully namespaced (lossless, collision-proof).
+- **Workflows**: workflow snapshots typically carry no array data, in which case no netCDF file is uploaded.
+
+---
+
+## Configuration
+
+The QUAlibrate connector requires a `QualibrateConfigData` configuration with the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `data_directory` | `Path` or `str` | Yes | Path to the QUAlibrate data directory (the root containing snapshot folders) |
+| `is_server_folder` | `bool` | Yes | Whether this is a server folder (e.g. on a network drive of the university) |
+
+A scope is **required** for this connector (`scope_requirement = REQUIRED`); scope mapping is not supported.
+
+The configuration is validated on creation: `data_directory` must exist and be a directory, and it must not overlap (be identical to, a parent of, or a subdirectory of) the path of an existing QUAlibrate sync source.
+
+### Example Configuration
+
+Example using the `etiket-sdk` package:
+
+```python
+from etiket_sdk.sync import SyncSources
+
+SyncSources.create(
+    name="my_qualibrate_source",
+    connector_identifier="etiket_sync_agent_qualibrate",
+    config_data={
+        "data_directory": "~/.qualibrate/user_storage",
+        "is_server_folder": False
+    },
+    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+)
+```
+
+## Live Sync
+
+Live synchronization is **not** supported by this connector. Snapshots are synced once they are complete (i.e. once a `node.json` is present in the snapshot directory).
+
+## Requirements
+
+- Python >= 3.10
+- numpy
+- xarray
+- h5netcdf
+- Pillow
+
+## License
+
+Copyright © 2025 QHarbor. All Rights Reserved. See [LICENCE](LICENCE) for details.

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate/__init__.py

@@ -0,0 +1,5 @@
+from .qualibrate_sync_class import QualibrateSync
+from .qualibrate_config_class import QualibrateConfigData
+
+__all__ = ["QualibrateSync", "QualibrateConfigData"]
+__version__ = "0.0.1"

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate/_qualibrate_loader.py

@@ -0,0 +1,226 @@
+"""Vendored, dependency-light copy of qualibrate's node data loader.
+
+We only need qualibrate's reference-resolution logic (turning the
+``./arrays.npz#a.b.c`` style references in ``data.json`` back into real
+numpy / xarray / image objects). Installing the full ``qualibrate`` package to
+get this drags in the whole web stack (fastapi, uvicorn, sqlalchemy, ...), which
+conflicts with the sync-agent environment, so the relevant pieces are vendored
+here instead.
+
+Ported from ``qualibrate.core.utils.node.{content,path_solver,loaders}``
+(qualibrate 1.4.0). Only ``read_node_data`` and its dependencies are included;
+the parameter/pydantic machinery from the original ``content.py`` is omitted.
+
+Runtime deps: numpy (required), xarray + Pillow (optional, used if installed).
+"""
+
+import json
+import logging
+from abc import ABC, abstractmethod
+from importlib.util import find_spec
+from itertools import chain
+from os import PathLike
+from pathlib import Path
+from typing import Any, Optional, Sequence
+
+import numpy as np
+from numpy.lib.npyio import NpzFile
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["read_node_data"]
+
+
+# ---------------------------------------------------------------------------
+# Loaders (extension -> object)
+# ---------------------------------------------------------------------------
+
+class BaseLoader(ABC):
+    file_extensions: tuple[str, ...]
+
+    @classmethod
+    def is_loader_support_extension(cls, extension: str) -> bool:
+        return extension.lower() in cls.file_extensions
+
+    @classmethod
+    def validate_file_exists(cls, file_path: Path) -> None:
+        if not file_path.is_file():
+            raise FileNotFoundError(f"Can't load file '{file_path}' by loader {cls.__name__}")
+
+    @abstractmethod
+    def load(self, path: Path, **kwargs: Any) -> Any:
+        raise NotImplementedError
+
+
+class JSONLoader(BaseLoader):
+    file_extensions = (".json",)
+
+    def load(self, path: Path, **kwargs: Any) -> Any:
+        self.__class__.validate_file_exists(path)
+        with open(path) as f:
+            return json.load(f)
+
+
+class NumpyArrayLoader(BaseLoader):
+    file_extensions = (".npy", ".npz")
+
+    def __init__(self) -> None:
+        self.filepath_to_array: dict[Path, Any] = {}
+
+    def load(self, path: Path, **kwargs: Any) -> Any:
+        if path in self.filepath_to_array:
+            file_content = self.filepath_to_array[path]
+        else:
+            self.__class__.validate_file_exists(path)
+            file_content = np.load(path)
+            self.filepath_to_array[path] = file_content
+        subref = kwargs.get("subref")
+        if subref is None:
+            return file_content
+        if not isinstance(file_content, NpzFile):
+            raise ValueError(f"Loaded file {path} is not representation of multiple NumPy arrays")
+        return file_content.get(subref)
+
+
+class XarrayLoader(BaseLoader):
+    file_extensions: tuple[str, ...] = (".h5",)
+
+    def load(self, path: Path, **kwargs: Any) -> Any:
+        # Faithful to qualibrate: opens the dataset and ignores any "#group"
+        # subref (the default save mode writes one file per dataset).
+        if find_spec("xarray") is None:
+            return None
+        from xarray import open_dataset
+
+        self.__class__.validate_file_exists(path)
+        return open_dataset(path)
+
+
+class ImageLoader(BaseLoader):
+    file_extensions: tuple[str, ...] = (".png",)
+
+    def load(self, path: Path, **kwargs: Any) -> Any:
+        if find_spec("PIL") is None:
+            return None
+        from PIL import Image
+
+        self.__class__.validate_file_exists(path)
+        return Image.open(path)
+
+
+# QuamLoader is intentionally omitted: quam state is referenced from node.json,
+# not from data.json, so read_node_data never needs it.
+DEFAULT_LOADERS: list[type[BaseLoader]] = [
+    JSONLoader,
+    NumpyArrayLoader,
+    ImageLoader,
+    XarrayLoader,
+]
+
+SUPPORTED_EXTENSIONS = set(chain.from_iterable(loader.file_extensions for loader in DEFAULT_LOADERS))
+
+
+# ---------------------------------------------------------------------------
+# Path helpers
+# ---------------------------------------------------------------------------
+
+def get_data_filepath(node_path: Path) -> Path:
+    return node_path / "data.json"
+
+
+def resolve_and_check_relative(base_path: Path, subpath: "PathLike[str]") -> Path:
+    """Build a full path from base + subpath; raise if it escapes base_path."""
+    full = (base_path / Path(subpath)).resolve()
+    base_path_resolved = base_path.resolve()
+    if not full.is_relative_to(base_path_resolved):
+        raise FileNotFoundError(f"Subpath {subpath} isn't relative to base {base_path}.")
+    return full
+
+
+# ---------------------------------------------------------------------------
+# Reference resolution
+# ---------------------------------------------------------------------------
+
+def _read_raw_node_file(node_filepath: Path) -> dict[str, Any]:
+    try:
+        with node_filepath.open("r") as f:
+            return dict(json.load(f))
+    except json.JSONDecodeError as ex:
+        logger.exception("Can't read json by path %s", node_filepath, exc_info=ex)
+        raise
+
+
+def _get_filename_and_subreference(filepath: "str | Path") -> tuple[Path, Optional[str]]:
+    """``"./arrays.npz#a.b.c"`` -> ``(Path("./arrays.npz"), "a.b.c")``."""
+    filepath = Path(filepath)
+    name, *subref = filepath.name.split("#")
+    clear_filepath = filepath.with_name(name)
+    return clear_filepath, "#".join(subref) if subref else None
+
+
+def _check_supported_reference(filename: Any, supported_extensions: set[str]) -> bool:
+    if not isinstance(filename, str) or "." not in filename:
+        return False
+    filepath, _ = _get_filename_and_subreference(filename)
+    return filepath.suffix in supported_extensions
+
+
+def _read_reference(reference: str, loaders: Sequence[BaseLoader], node_dir: Path) -> Any:
+    filepath, subreference = _get_filename_and_subreference(reference)
+    try:
+        filepath = resolve_and_check_relative(node_dir, filepath)
+    except FileNotFoundError as ex:
+        logger.exception("File %s can't be located inside node directory.", filepath, exc_info=ex)
+        return reference
+    supported_loaders = [ldr for ldr in loaders if ldr.is_loader_support_extension(filepath.suffix)]
+    for loader in supported_loaders:
+        try:
+            result = loader.load(filepath, subref=subreference)
+            if result is None:
+                continue
+            return result
+        except Exception as ex:
+            logger.exception("Can't load reference (%s) by loader %s", reference, loader, exc_info=ex)
+    logger.warning("Reference %s was not loaded. Kept reference value.", reference)
+    return reference
+
+
+def _resolve_references(
+    raw_data: dict[str, Any],
+    loaders: Sequence[BaseLoader],
+    supported_extensions: set[str],
+    node_dir: Path,
+) -> None:
+    for key, value in raw_data.items():
+        if isinstance(value, dict):
+            _resolve_references(value, loaders, supported_extensions, node_dir)
+            raw_data[key] = value
+        if not _check_supported_reference(value, supported_extensions):
+            continue
+        raw_data[key] = _read_reference(value, loaders, node_dir)
+
+
+def read_node_data(
+    node_dir: Path,
+    node_id: int,
+    base_path: Path,
+    custom_loaders: Optional[Sequence[type[BaseLoader]]] = None,
+) -> Optional[dict[str, Any]]:
+    """Read ``data.json`` and resolve its references into real objects.
+
+    Returns the results dict (with numpy/xarray/image objects in place of the
+    reference strings), or None if there is no data.json.
+    """
+    data_filepath = get_data_filepath(node_dir)
+    if not data_filepath.is_file():
+        logger.error("Data file for node %s wasn't found in %s", node_id, base_path)
+        return None
+    results = _read_raw_node_file(data_filepath)
+    loader_classes: list[type[BaseLoader]] = [*custom_loaders] if custom_loaders is not None else []
+    loader_classes.extend(DEFAULT_LOADERS)
+    loaders_instances = [loader() for loader in loader_classes]
+    supported_extensions = set(
+        chain.from_iterable(loader.file_extensions for loader in DEFAULT_LOADERS)
+    ) | SUPPORTED_EXTENSIONS
+    _resolve_references(results, loaders_instances, supported_extensions, node_dir)
+    return results

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate/qualibrate_config_class.py

@@ -0,0 +1,74 @@
+import dataclasses
+import pathlib
+import etiket_sync_agent_qualibrate
+
+from typing import Optional
+
+# Imports that you will likely need
+from etiket_sync_agent.crud.sync_sources import crud_sync_sources, SyncSources
+from etiket_sync_agent.db import get_db_session_context
+
+@dataclasses.dataclass
+class QualibrateConfigData:
+    """Template config dataclass for a sync connector.
+
+    Add your configuration fields here and implement the validate method
+    to perform sanity checks and cross-check with existing sources.
+    """
+    data_directory: pathlib.Path
+    is_server_folder : bool
+
+    def __post_init__(self):
+        # Ensure that data_directory is a pathlib.Path object
+        if not isinstance(self.data_directory, pathlib.Path):
+            self.data_directory = pathlib.Path(self.data_directory)
+
+    async def validate(self, current_sync_source: Optional[SyncSources] = None):
+        """
+        Validates the qualibrate configuration.
+
+        Checks:
+        1. If the data_directory exists and is a directory.
+        2. If the data_directory conflicts with an existing qualibrate sync source
+            (i.e., it's identical, a subdirectory, or a parent directory).
+
+        Raises:
+            ValueError: If any validation check fails.
+
+        Returns:
+            True if all checks pass.
+        """
+        try:
+            abs_root_dir = self.data_directory.expanduser().resolve(strict=True)
+        except FileNotFoundError:
+            raise ValueError(f"The specified root directory does not exist: {self.data_directory}")
+
+        # check if the root directory exists and is a directory.
+        if not abs_root_dir.is_dir():
+            raise ValueError(f"The specified path is not a directory: {abs_root_dir}")
+
+        # check if the folder is not yet added/is part of a folder that is already added.
+        async with get_db_session_context() as session:
+            sync_sources = await crud_sync_sources.list_sync_sources(session)
+            for sync_source in sync_sources:
+                if sync_source.connector == etiket_sync_agent_qualibrate.__name__ and (current_sync_source is None or sync_source.id != current_sync_source.id):
+                    # Assuming config_data stores the path as a string and is always present/valid.
+                    existing_path_str = sync_source.config_data['data_directory']
+                    existing_path = pathlib.Path(existing_path_str).expanduser().resolve()
+                    
+                    # Check for conflicts (using samefile for case-insensitive filesystem support)
+                    try:
+                        is_same = abs_root_dir.samefile(existing_path)
+                    except FileNotFoundError:
+                        is_same = False  # existing path was deleted
+
+                    if is_same:
+                        raise ValueError(f"The directory '{abs_root_dir}' is already added as sync source '{sync_source.name}'.")
+                    # Check if the new path is inside an existing path
+                    if abs_root_dir.is_relative_to(existing_path):
+                        raise ValueError(f"The directory '{abs_root_dir}' is inside the directory '{existing_path}' added by sync source '{sync_source.name}'.")
+                    # Check if an existing path is inside the new path
+                    if existing_path.is_relative_to(abs_root_dir):
+                        raise ValueError(f"The directory '{existing_path}' added by sync source '{sync_source.name}' is inside the specified directory '{abs_root_dir}'.")
+
+        return True

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate/qualibrate_sync_class.py

@@ -0,0 +1,266 @@
+import json
+import typing
+import datetime
+from pathlib import Path
+
+import numpy as np
+import xarray as xr
+
+from etiket_sync_agent.connectors.sync_source_abstract import SyncSourceFileBase, ScopeRequirement
+from etiket_sync_agent.sync.sync_records.manager import SyncRecordManager
+from etiket_sync_agent.schemas import SyncItemSchema
+from etiket_sync_agent.sync.sync_utilities import sync_utilities, dataset_info, file_info, FileType
+
+# vendored qualibrate loader (resolves `./file.npz#a.b.c` references) — avoids
+# depending on the full qualibrate package (which drags in fastapi/uvicorn/etc.)
+from ._qualibrate_loader import read_node_data
+
+from .qualibrate_config_class import QualibrateConfigData
+
+FILE_GENERATOR = "QUAlibrate"
+
+
+class QualibrateSync(SyncSourceFileBase):
+    sync_agent_name: typing.ClassVar[str] = "QUAlibrate"
+    config_data_class: typing.ClassVar[typing.Type[QualibrateConfigData]] = QualibrateConfigData
+    scope_requirement: typing.ClassVar[ScopeRequirement] = ScopeRequirement.REQUIRED
+    live_sync_implemented: typing.ClassVar[bool] = False
+    supports_scope_mapping = False
+    level: typing.ClassVar[int] = 2
+    has_owner: typing.ClassVar[bool] = True
+    is_single_file = False
+
+    @staticmethod
+    def rootPath(config_data: QualibrateConfigData) -> Path:
+        return Path(config_data.data_directory)
+
+    @staticmethod
+    async def checkLiveDataset(config_data: QualibrateConfigData, sync_item: SyncItemSchema, max_priority: bool) -> bool:
+        """
+        Always false since we don't have a mechanism for live datasets.
+        """
+        return False
+
+    @staticmethod
+    async def syncDatasetNormal(config_data: QualibrateConfigData, sync_item: SyncItemSchema, sync_record: SyncRecordManager):
+        root = Path(config_data.data_directory)
+        node_dir = (root / sync_item.dataIdentifier).resolve()
+
+        # scope_requirement is REQUIRED, so a scope must be present.
+        scope_uuid = sync_item.scopeUUID
+        if scope_uuid is None:
+            raise ValueError("A scope is required for the qualibrate sync source.")
+
+        # --- 1. Collect dataset information (node.json) ------------------------
+        node_json_path = node_dir / "node.json"
+        if not node_json_path.is_file():
+            sync_record.add_log(f"No node.json in {node_dir}; not a qualibrate snapshot (yet), skipping.")
+            return
+
+        with sync_record.task("Reading qualibrate snapshot (node.json)"):
+            node_info = json.loads(node_json_path.read_text())
+
+        metadata = node_info.get("metadata", {}) or {}
+        snapshot_id = node_info.get("id")
+        exec_type = metadata.get("type_of_execution", "node")  # "node" | "workflow"
+        created = _parse_dt(node_info.get("created_at") or metadata.get("run_start"))
+
+        # --- 2. Create / update the dataset record -----------------------------
+        ds_info = dataset_info(
+            name=metadata.get("name") or node_dir.name,
+            datasetUUID=sync_item.datasetUUID,
+            scopeUUID=scope_uuid,
+            created=created,
+            alt_uid=node_dir.name,
+            description=metadata.get("description"),
+            keywords=[exec_type],
+            attributes=_build_attributes(node_info, metadata, exec_type),
+        )
+        await sync_utilities.create_or_update_dataset(False, sync_item, ds_info, sync_record)
+
+        # --- 3. Upload every raw file ------------------------------------------
+        with sync_record.task("Uploading raw snapshot files"):
+            await _upload_raw_files(node_dir, sync_item, created, sync_record)
+
+        # --- 4. Build the raw-data netCDF for the plotting system -------------
+        # data.json -> load all arrays -> extract the relevant data ->
+        # arrays as vars/coords -> merge into one struct (single netCDF).
+        with sync_record.task("Loading results via qualibrate loaders"):
+            try:
+                results = read_node_data(node_dir, snapshot_id or 0, root) or {}
+            except Exception as e:  # never let a bad results tree kill the sync
+                sync_record.add_error("Failed to load results via qualibrate loaders", e)
+                results = {}
+
+        datasets = _split_results(results)
+
+        if datasets:
+            with sync_record.task("Combining raw data into a single netCDF dataset"):
+                combined = _combine_raw(datasets, sync_record)
+            if combined is not None:
+                await sync_utilities.upload_xarray(
+                    combined, sync_item,
+                    file_info(name="raw_data", fileName="raw_data.h5",
+                              created=created, fileType=FileType.HDF5_NETCDF,
+                              file_generator=FILE_GENERATOR),
+                    sync_record,
+                )
+        else:
+            sync_record.add_log("No array/xarray raw data found (expected for workflows).")
+
+        sync_record.add_log(f"Synchronized qualibrate {exec_type} snapshot {snapshot_id}.")
+
+    @staticmethod
+    async def syncDatasetLive(config_data: QualibrateConfigData, sync_item: SyncItemSchema, sync_record: SyncRecordManager):
+        # Implement the live synchronization flow if applicable.
+        raise NotImplementedError
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _filetype_for_suffix(path: Path) -> FileType:
+    return {
+        ".h5": FileType.HDF5_NETCDF,
+        ".nc": FileType.HDF5_NETCDF,
+        ".hdf5": FileType.HDF5_NETCDF,
+        ".json": FileType.JSON,
+        ".npy": FileType.NDARRAY,
+        ".npz": FileType.NDARRAY,
+        ".txt": FileType.TEXT,
+    }.get(path.suffix.lower(), FileType.UNKNOWN)
+
+
+async def _upload_raw_files(
+    node_dir: Path,
+    sync_item: SyncItemSchema,
+    created: datetime.datetime,
+    sync_record: SyncRecordManager,
+) -> None:
+    """Upload every file in the snapshot folder verbatim, under a "raw/" name.
+
+    Preserves the original bytes and folder structure so the complete snapshot
+    is always captured, even if the curated combine/JSON steps drop something.
+    """
+    for path in sorted(node_dir.rglob("*")):
+        if not path.is_file() or path.name.startswith("."):
+            continue
+        rel = path.relative_to(node_dir).as_posix()
+        await sync_utilities.upload_file(
+            str(path), sync_item,
+            file_info(name=f"raw/{rel}", fileName=path.name,
+                      created=created, fileType=_filetype_for_suffix(path),
+                      file_generator=FILE_GENERATOR),
+            sync_record,
+        )
+
+
+def _parse_dt(value: typing.Optional[str]) -> datetime.datetime:
+    if value:
+        try:
+            return datetime.datetime.fromisoformat(value)
+        except ValueError:
+            pass
+    return datetime.datetime.now().astimezone()
+
+
+def _build_attributes(node_info: dict, metadata: dict, exec_type: str) -> dict:
+    """Small, searchable scalar metadata. Values are stringified by dataset_info.
+    """
+    attrs: dict = {"type_of_execution": exec_type}
+    for key in ("status"):
+        if metadata.get(key) is not None:
+            attrs[key] = metadata[key]
+    return attrs
+
+
+def _split_results(tree: typing.Mapping, prefix: str = "") -> dict:
+    """Walk the results tree and turn array-like leaves into xr.Datasets.
+
+    - xr.Dataset / xr.DataArray values are taken as-is.
+    - A dict whose direct children include numpy arrays is treated as one
+      measurement: by convention the FIRST array is the independent x-axis
+      (a coordinate) and the remaining arrays are values measured over it.
+    Everything else (scalars, figures, ...) is ignored here because it is
+    already captured by the verbatim raw-file upload.
+
+    Returns {dotted_path: xr.Dataset}.
+    """
+    datasets: dict = {}
+    array_items = [(k, v) for k, v in tree.items() if isinstance(v, np.ndarray)]
+    for key, value in tree.items():
+        path = f"{prefix}{key}"
+        if isinstance(value, xr.Dataset):
+            datasets[path] = value
+        elif isinstance(value, xr.DataArray):
+            datasets[path] = value.to_dataset(name=str(key))
+        elif isinstance(value, dict):
+            datasets.update(_split_results(value, prefix=f"{path}."))
+        # numpy arrays are handled as a sibling group below
+    if array_items:
+        group_path = prefix.rstrip(".") or "raw_data"
+        datasets[group_path] = _arrays_to_dataset(array_items)
+    return datasets
+
+
+def _arrays_to_dataset(array_items: list) -> xr.Dataset:
+    """Build an xr.Dataset from a group of sibling numpy arrays.
+
+    Convention (npz format): the leading 1-D arrays are the coordinate axes in
+    order (x, y, z, ...) and the remaining arrays are the measured values over
+    the grid those axes span. The number of axes is taken from the highest-rank
+    array (1 for a line sweep, 2 for a chevron map, ...). Falls back to
+    standalone variables with independent dims if the shapes don't fit.
+    """
+    arrays = [(name, np.asarray(v)) for name, v in array_items if v is not None]
+    if not arrays:
+        return xr.Dataset()
+
+    n_axes = max(v.ndim for _, v in arrays)  # x, y, z, ...
+    axes, data = arrays[:n_axes], arrays[n_axes:]
+    expected_shape = tuple(v.shape[0] for _, v in axes)
+    axes_ok = len(axes) == n_axes and all(v.ndim == 1 for _, v in axes)
+    data_ok = bool(data) and all(v.shape == expected_shape for _, v in data)
+
+    if axes_ok and data_ok:
+        axis_names = [n for n, _ in axes]
+        return xr.Dataset(
+            {n: (axis_names, v) for n, v in data},
+            coords={n: v for n, v in axes},
+        )
+
+    return xr.Dataset({n: ([f"{n}.dim_{i}" for i in range(v.ndim)], v) for n, v in arrays})
+
+
+def _namespace_all(path: str, ds: xr.Dataset) -> xr.Dataset:
+    names = set(ds.dims) | set(ds.coords) | set(ds.data_vars)
+    return ds.rename({n: f"{path}.{n}" for n in names})
+
+
+def _combine_raw(datasets: dict, sync_record: SyncRecordManager) -> typing.Optional[xr.Dataset]:
+    """Combine the per-measurement datasets into one flat xr.Dataset.
+
+    Strategy ladder:
+      1. prefix only data_vars by their results-path, keep coords shared, merge
+         with join="exact" so a sweep axis shared by parallel measurements stays
+         shared, but mismatched same-named axes raise instead of silently
+         outer-joining (union + NaN padding);
+      2. on conflict, fully namespace every dataset (no shared axes, but lossless
+         and collision-proof).
+    """
+    prepared = [ds.rename({v: f"{path}.{v}" for v in ds.data_vars}) for path, ds in datasets.items()]
+
+    if not prepared:
+        return None
+    if len(prepared) == 1:
+        return prepared[0]
+
+    try:
+        return typing.cast(xr.Dataset, xr.merge(prepared, join="exact", compat="no_conflicts", combine_attrs="drop_conflicts"))
+    except Exception as e:
+        sync_record.add_log(f"Shared-coordinate merge failed ({e}); namespacing fully.")
+        fully = [_namespace_all(path, ds) for path, ds in datasets.items()]
+        return typing.cast(xr.Dataset, xr.merge(fully, combine_attrs="drop_conflicts"))
+
+

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate.egg-info/PKG-INFO

@@ -0,0 +1,136 @@
+Metadata-Version: 2.4
+Name: etiket_sync_agent_qualibrate
+Version: 0.0.1
+Summary: QUAlibrate connector for eTiKeT sync agent
+Author: QHarbor team
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://qharbor.nl
+Project-URL: Documentation, https://docs.qharbor.nl
+Keywords: etiket,sync,connector,qualibrate,quantum
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENCE
+Requires-Dist: etiket_sync_agent>=0.3.0b1
+Requires-Dist: numpy
+Requires-Dist: xarray
+Requires-Dist: h5netcdf
+Requires-Dist: Pillow
+Dynamic: license-file
+
+# eTiKeT Sync Agent - QUAlibrate Connector
+
+Connector for synchronizing [QUAlibrate](https://qua-platform.github.io/qualibrate/) calibration data with the eTiKeT platform. This connector scans a QUAlibrate data directory for node/workflow snapshots and syncs them to the cloud.
+
+## Installation
+
+Install the QUAlibrate connector using the eTiKeT Sync SDK:
+
+```python
+from etiket_sdk.sync import Connectors
+
+# Install the latest version
+Connectors.install_from_pypi("etiket-sync-agent-qualibrate")
+```
+
+The package is automatically discovered by `etiket_sync_agent` through the entry-point system. Once installed, you can verify with:
+
+```python
+# List installed connectors
+print(Connectors.list())
+
+# Get details for the QUAlibrate connector
+connector = Connectors.get("etiket_sync_agent_qualibrate")
+print(connector)
+```
+
+### Updating the Connector
+
+```python
+# Update to the latest version
+Connectors.update_from_pypi("etiket-sync-agent-qualibrate")
+```
+
+## What Gets Synchronized
+
+Each QUAlibrate snapshot is a directory containing a `node.json` (metadata) and a `data.json` (results, with references to `.npz`/`.h5`/`.png`/`.json` files). When a snapshot is synced, the following data is extracted and uploaded:
+
+| QUAlibrate Data | eTiKeT Field | Description |
+|-----------------|--------------|-------------|
+| Snapshot folder name | `alt_uid` | Unique identifier for the snapshot |
+| `metadata.name` (or folder name) | `name` | Name of the dataset |
+| `created_at` / `metadata.run_start` | `collected` | When the snapshot was created |
+| `metadata.type_of_execution` | `keywords` | `node` or `workflow` |
+| `metadata.description` | `description` | Snapshot description |
+| `type_of_execution`, `status` | `attributes` | Small, searchable scalar metadata |
+| `data.json` → `raw_data` | HDF5 file | Raw measurement arrays combined into a single netCDF (`raw_data.h5`) |
+
+### Data Processing
+
+- **Reference resolution**: `data.json` reference strings (`./arrays.npz#a.b.c`) are resolved into real numpy arrays, xarray datasets, and images via the vendored loader.
+- **Raw-data extraction**: Only the `raw_data` entry of the results tree is converted to netCDF. Other entries (e.g. `fit_results`) are intentionally excluded.
+- **Array-to-dataset convention** (npz format): this is a best-effort reconstruction. Within a group of sibling numpy arrays, the leading 1-D arrays are treated as the coordinate axes in order (x, y, z, ...) and the remaining arrays as the measured values over the grid those axes span. If shapes don't fit this convention, arrays fall back to standalone variables with independent dimensions.
+- **Combining**: per-measurement datasets are merged into one flat netCDF. Data variables are prefixed by their results-path; shared sweep axes stay shared (`join="exact"`). On conflict, every dataset is fully namespaced (lossless, collision-proof).
+- **Workflows**: workflow snapshots typically carry no array data, in which case no netCDF file is uploaded.
+
+---
+
+## Configuration
+
+The QUAlibrate connector requires a `QualibrateConfigData` configuration with the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `data_directory` | `Path` or `str` | Yes | Path to the QUAlibrate data directory (the root containing snapshot folders) |
+| `is_server_folder` | `bool` | Yes | Whether this is a server folder (e.g. on a network drive of the university) |
+
+A scope is **required** for this connector (`scope_requirement = REQUIRED`); scope mapping is not supported.
+
+The configuration is validated on creation: `data_directory` must exist and be a directory, and it must not overlap (be identical to, a parent of, or a subdirectory of) the path of an existing QUAlibrate sync source.
+
+### Example Configuration
+
+Example using the `etiket-sdk` package:
+
+```python
+from etiket_sdk.sync import SyncSources
+
+SyncSources.create(
+    name="my_qualibrate_source",
+    connector_identifier="etiket_sync_agent_qualibrate",
+    config_data={
+        "data_directory": "~/.qualibrate/user_storage",
+        "is_server_folder": False
+    },
+    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+)
+```
+
+## Live Sync
+
+Live synchronization is **not** supported by this connector. Snapshots are synced once they are complete (i.e. once a `node.json` is present in the snapshot directory).
+
+## Requirements
+
+- Python >= 3.10
+- numpy
+- xarray
+- h5netcdf
+- Pillow
+
+## License
+
+Copyright © 2025 QHarbor. All Rights Reserved. See [LICENCE](LICENCE) for details.

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENCE
+README.md
+pyproject.toml
+etiket_sync_agent_qualibrate/__init__.py
+etiket_sync_agent_qualibrate/_qualibrate_loader.py
+etiket_sync_agent_qualibrate/qualibrate_config_class.py
+etiket_sync_agent_qualibrate/qualibrate_sync_class.py
+etiket_sync_agent_qualibrate.egg-info/PKG-INFO
+etiket_sync_agent_qualibrate.egg-info/SOURCES.txt
+etiket_sync_agent_qualibrate.egg-info/dependency_links.txt
+etiket_sync_agent_qualibrate.egg-info/entry_points.txt
+etiket_sync_agent_qualibrate.egg-info/requires.txt
+etiket_sync_agent_qualibrate.egg-info/top_level.txt

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[etiket_sync_agent.connectors]
+qualibrate_sync_agent = etiket_sync_agent_qualibrate:QualibrateSync

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate.egg-info/requires.txt

@@ -0,0 +1,5 @@
+etiket_sync_agent>=0.3.0b1
+numpy
+xarray
+h5netcdf
+Pillow

etiket_sync_agent_qualibrate-0.0.1/etiket_sync_agent_qualibrate.egg-info/top_level.txt

@@ -0,0 +1 @@
+etiket_sync_agent_qualibrate

etiket_sync_agent_qualibrate-0.0.1/pyproject.toml

@@ -0,0 +1,60 @@
+[build-system]
+requires = ["setuptools>=69.0.0", "setuptools-scm"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "etiket_sync_agent_qualibrate"
+description = "QUAlibrate connector for eTiKeT sync agent"
+readme = "README.md"
+authors = [{name = "QHarbor team"}]
+license = "LicenseRef-Proprietary"
+license-files = ["LICENCE"]
+requires-python = ">=3.10"
+dynamic = ["version"]
+keywords = [
+    "etiket",
+    "sync",
+    "connector",
+    "qualibrate",
+    "quantum",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS :: MacOS X",
+    "Operating System :: Microsoft :: Windows",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "etiket_sync_agent>=0.3.0b1",
+    "numpy",
+    "xarray",
+    "h5netcdf",
+    "Pillow",
+]
+
+[project.entry-points."etiket_sync_agent.connectors"]
+qualibrate_sync_agent = "etiket_sync_agent_qualibrate:QualibrateSync"
+
+[project.urls]
+Homepage = "https://qharbor.nl"
+Documentation = "https://docs.qharbor.nl"
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["etiket_sync_agent_qualibrate*"]
+
+[tool.setuptools.dynamic]
+version = {attr = "etiket_sync_agent_qualibrate.__version__"}

etiket_sync_agent_qualibrate-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+