aipmodel 0.2.68__tar.gz → 0.2.69__tar.gz

{aipmodel-0.2.68 → aipmodel-0.2.69}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aipmodel
-Version: 0.2.68
+Version: 0.2.69
 Summary: SDK for model registration, versioning, and storage
 Home-page: https://github.com/AIP-MLOPS/model-registry
 Author: AIP MLOPS Team

{aipmodel-0.2.68 → aipmodel-0.2.69}/README.md

@@ -1,6 +1,6 @@
 # AIP Model SDK
 
-This SDK provides a robust interface for registering, uploading, downloading, listing, and deleting machine learning models using ClearML and S3 (Ceph) as storage. It supports **automatic versioning**, **categorization**, **tagging**, **metadata transparency**, **web link generation**, and fail-fast connectivity checks.
+This SDK provides a robust interface for registering, uploading, downloading, listing, and deleting machine learning models using Mlops-Core and S3 (Ceph) as storage. It supports **automatic versioning**, **categorization**, **tagging**, **metadata transparency**, **web link generation**, and fail-fast connectivity checks.
 
 ---
 
@@ -20,11 +20,13 @@ pip install aipmodel
 To use the SDK, you must set up your environment variables. You can create a `.env` file in your project root:
 
 ```env
-# ClearML API Credentials
-CLEARML_API_HOST="http://your-clearml-server:8008"
-CLEARML_WEB_HOST="http://your-clearml-web-ui:8080"  # Required for generating model web links
-CLEARML_ACCESS_KEY="your_access_key"
-CLEARML_SECRET_KEY="your_secret_key"
+# Mlops-Core API Credentials
+# (The legacy CLEARML_API_HOST / CLEARML_WEB_HOST env var names and constructor
+# kwargs are still accepted as a deprecated fallback for backward compatibility.)
+MLOPS_CORE_API_HOST="http://your-mlops-core-server:8008"
+MLOPS_CORE_WEB_HOST="http://your-mlops-core-web-ui:8080"  # Required for generating model web links
+MLOPS_CORE_ACCESS_KEY="your_access_key"
+MLOPS_CORE_SECRET_KEY="your_secret_key"
 
 # Ceph / S3 Storage
 CEPH_ENDPOINT_URL="https://your-ceph-endpoint.com"
@@ -59,7 +61,7 @@ S3_BOTO_MAX_ATTEMPTS="10"        # botocore request-level retry attempts (adapti
 
 ```
 
-> **Note:** `CLEARML_WEB_HOST` is new. If not provided, the SDK will default to the API host, which might result in incorrect clickable links in the output.
+> **Note:** `MLOPS_CORE_WEB_HOST` is new. If not provided, the SDK will default to the API host, which might result in incorrect clickable links in the output. The legacy `CLEARML_API_HOST` / `CLEARML_WEB_HOST` names still work but are deprecated; when both old and new names are set, the `MLOPS_CORE_*` name wins.
 
 > Every constant the SDK relies on (timeouts, categories, transfer tuning, ACL) is overridable via `.env` — nothing important is hardcoded.
 
@@ -97,7 +99,7 @@ reset). The SDK now **retries these automatically**, so `add_model`,
 
 `add_model` is **concurrency-safe** — you can run many adds at the same time:
 
-- Adds with **different model names** run **fully in parallel** (own ClearML id,
+- Adds with **different model names** run **fully in parallel** (own Mlops-Core id,
   own S3 prefix — no interference).
 - Adds with the **same `(project, model_name)`** are **serialized** by a
   process-wide lock. Otherwise two simultaneous same-name adds would race on the
@@ -116,7 +118,7 @@ reset). The SDK now **retries these automatically**, so `add_model`,
   automatically on the next add to that model.
 - The lock is **per-process**. One server process (e.g. a single `uvicorn`
   worker) is fully protected; for multiple workers/replicas add a distributed
-  lock (ClearML/Redis).
+  lock (Mlops-Core/Redis).
 
 ## Access Control (ACL)
 
@@ -143,7 +145,7 @@ Requires `ACL_ENABLED=true` and that you have `admin`-level access on the model.
 
 ## Cross-User Visibility (Public Models)
 
-Every model lives in its owner's own ClearML project (`project_<owner_username>`) — that part hasn't changed. What changed is **how `list_models`/`get_model_id_by_name`/`get_model_info`/`get_model` look things up**:
+Every model lives in its owner's own Mlops-Core project (`project_<owner_username>`) — that part hasn't changed. What changed is **how `list_models`/`get_model_id_by_name`/`get_model_info`/`get_model` look things up**:
 
 - **`list_models()`** (default `include_public=True`): returns your own models (private + public) **plus** every other user's model that is marked `public=True`. Other users' **private** models are never returned, under any setting. Each entry's `owner` field is resolved from the model's actual project, so models from other users show their real owner, not yours.
   - Pass `include_public=False` to get the exact pre-existing behavior: your own project only, nothing from other users.
@@ -152,7 +154,7 @@ Every model lives in its owner's own ClearML project (`project_<owner_username>`
 - **`get_model_link`** now resolves the link using the model's *actual* owning project (previously it always used your own project ID, which silently produced a wrong link for any model you didn't own — this never came up before because nothing could return another user's model anyway).
 - Write operations (`add_model`, `delete_model`, `set_model_public`, `set_latest_version`) are **unchanged** — they still only ever operate within your own project by name. To act on a model you don't own (e.g. as an ACL-granted admin), pass its `model_id` directly (already supported) rather than its name.
 
-**Performance note:** when `include_public=True` (the default), `list_models()` makes one extra unscoped ClearML query (`models.get_all` with no project filter) to find other users' public models — on a very large ClearML deployment this returns every model company-wide before filtering down to public ones. If that ever becomes slow, set `include_public=False` and only fetch other users' models on demand via `owner=`.
+**Performance note:** when `include_public=True` (the default), `list_models()` makes one extra unscoped Mlops-Core query (`models.get_all` with no project filter) to find other users' public models — on a very large Mlops-Core deployment this returns every model company-wide before filtering down to public ones. If that ever becomes slow, set `include_public=False` and only fetch other users' models on demand via `owner=`.
 
 **Backward compatibility:** `get_model_id_by_name`, `get_model`, `get_model_info` keep their exact old signatures plus one new optional trailing argument — no existing call site breaks. `list_models()`'s default *output* now includes more rows than before (other users' public models) since that was the explicit point of this change; pass `include_public=False` if your code depends on the exact old row set.
 
@@ -174,8 +176,8 @@ print("\n[STEP 1] Initialize MLOps Manager")
 # verbose=True enables detailed logging and transparency mode.
 manager = MLOpsManager(
     user_token=os.getenv("USER_TOKEN"), 
-    CLEARML_API_HOST=os.getenv("CLEARML_API_HOST"),
-    CLEARML_WEB_HOST=os.getenv("CLEARML_WEB_HOST"), # For generating UI links
+    MLOPS_CORE_API_HOST=os.getenv("MLOPS_CORE_API_HOST"),
+    MLOPS_CORE_WEB_HOST=os.getenv("MLOPS_CORE_WEB_HOST"), # For generating UI links
     CEPH_ENDPOINT_URL=os.getenv("CEPH_ENDPOINT_URL"),
     USER_MANAGEMENT_API=os.getenv("USER_MANAGEMENT_API"),
     verbose=True 
@@ -234,7 +236,7 @@ print(f" -> Registered ID: {hf_model_id}")
 
 # --- STEP 6: Get Detailed Info ---
 print("\n[STEP 6] Get Detailed Model Info")
-# Displays ID, Public status, Total Versions, Size, and ClearML Web Link
+# Displays ID, Public status, Total Versions, Size, and Mlops-Core Web Link
 manager.get_model_info("demo_model_local")
 
 # --- STEP 7: List All Models ---
@@ -287,7 +289,7 @@ print("\n[STEP 10] Deletion")
 # Delete a specific version (Removes files + Metadata entry)
 manager.delete_model(model_name="demo_model_local", version="v1.0")
 
-# Delete the entire model (Removes all versions + ClearML ID)
+# Delete the entire model (Removes all versions + Mlops-Core ID)
 manager.delete_model(model_name="demo_model_hf")
 
 ```
@@ -300,14 +302,14 @@ manager.delete_model(model_name="demo_model_hf")
 
 | Function | Input Arguments | Description |
 | --- | --- | --- |
-| **`__init__`** | `user_token`, `CLEARML_API_HOST`, `CLEARML_WEB_HOST`, `CEPH_ENDPOINT_URL`, `USER_MANAGEMENT_API`, `verbose`, `ACL_API_HOST`, `ACL_ENABLED` | Initializes connections. Performs fail-fast health checks. `ACL_*` args (or env vars) toggle ACL enforcement (default: disabled). |
+| **`__init__`** | `user_token`, `MLOPS_CORE_API_HOST`, `MLOPS_CORE_WEB_HOST`, `CEPH_ENDPOINT_URL`, `USER_MANAGEMENT_API`, `verbose`, `ACL_API_HOST`, `ACL_ENABLED` | Initializes connections. Performs fail-fast health checks. `ACL_*` args (or env vars) toggle ACL enforcement (default: disabled). The legacy `CLEARML_API_HOST` / `CLEARML_WEB_HOST` kwargs are still accepted as deprecated aliases (the `MLOPS_CORE_*` kwargs win if both are given). |
 | **`add_model`** | `source_type`, `model_name`, `version`, `source_path`, `code_path`, `category`, `tags`, `external_ceph_*`, `model_folder_name` | Uploads a model. `version`: if `None`, generates `_v{n}` automatically. `category`: must be one of `get_allowed_categories()` (defaults to "Text Generation"/"Image Classification", overridable via `MODEL_ALLOWED_CATEGORIES`). `tags`: list of strings. `model_folder_name`: optional explicit folder name for the uploaded model; if omitted, it's derived automatically as before (backward compatible). |
 | **`get_model`** | `model_name`, `local_dest`, `version`, `owner` | Downloads model files. If `version` is omitted, downloads the version marked as 'latest'. `owner` (optional): resolve `model_name` from another user's project, e.g. to download a public model that isn't yours. |
 | **`get_model_info`** | `identifier` (Name or ID), `owner` | Prints detailed info including **Owner**, **Web Link**, Tags, Category, and Version History. `owner` (optional): resolve a name-based `identifier` from another user's project. |
 | **`get_allowed_categories`** | *(none)* | Returns the list of valid `category` values for `add_model`. |
 | **`list_models`** | `verbose`, `category`, `tag`, `public_only`, `private_only`, `include_public` | Lists your own models **plus** (by default, `include_public=True`) every other user's `public=True` model — never another user's private model. Each entry includes a real `owner` field. `verbose=True` shows full JSON metadata. `category`/`tag` filter by exact match; `public_only`/`private_only` filter by visibility, applied across the combined set. `include_public=False` restores the pre-existing own-project-only behavior. When ACL is enabled, results are additionally filtered to models you can access (plus public ones). See "Cross-User Visibility" above. |
-| **`get_model_id_by_name`** | `name`, `owner` | Resolves a model name to its ClearML ID. `owner` (optional): look it up in another user's project instead of your own. |
-| **`get_model_name_by_id`** | `model_id` | Resolves a ClearML ID back to its model name (works for any model, regardless of owner). |
+| **`get_model_id_by_name`** | `name`, `owner` | Resolves a model name to its Mlops-Core ID. `owner` (optional): look it up in another user's project instead of your own. |
+| **`get_model_name_by_id`** | `model_id` | Resolves a Mlops-Core ID back to its model name (works for any model, regardless of owner). |
 | **`get_latest_version`** | `model_identifier` (Name or ID) | Returns the version tag currently marked 'latest'. |
 | **`set_latest_version`** | `model_name`, `version` | Updates the 'latest' pointer to a specific version without modifying files. |
 | **`set_model_public`** | `model_name`, `public` | Moves a model's files between the private and public buckets (transactional, parallel server-side copy). |
@@ -348,12 +350,12 @@ The short version: run `uvicorn main:app --host 0.0.0.0 --port 8080`, pass `Auth
 
 ## Testing
 
-- `test_model_registry.py` / `test_ceph_s3_manager.py`: automated unit tests (mock the ClearML/S3 boundary, no live infra needed). Run with:
+- `test_model_registry.py` / `test_ceph_s3_manager.py`: automated unit tests (mock the Mlops-Core/S3 boundary, no live infra needed). Run with:
   ```bash
   pip install pytest
   pytest test_model_registry.py test_ceph_s3_manager.py -v
   ```
-- `test.py`: a manual/live-integration demo script covering every scenario above against **real** ClearML/Ceph/ACL services — everything is commented out by default; uncomment only what you intend to run.
+- `test.py`: a manual/live-integration demo script covering every scenario above against **real** Mlops-Core/Ceph/ACL services — everything is commented out by default; uncomment only what you intend to run.
 
 ---
 

{aipmodel-0.2.68 → aipmodel-0.2.69}/aipmodel/CephS3Manager.py

@@ -858,7 +858,9 @@ class CephS3Manager:
             with tqdm(total=len(keys), desc="Moving", unit="file", ascii=True, disable=not self.verbose) as pbar:
                 for src in keys:
                     dest = src.replace(src_prefix, dest_prefix, 1)
-                    self.s3.copy_object(Bucket=self.bucket_name, CopySource={"Bucket": self.bucket_name, "Key": src}, Key=dest)
+                    # s3.copy (not copy_object) — handles files >5GB via
+                    # automatic multipart copy, same as server_side_copy().
+                    self.s3.copy({"Bucket": self.bucket_name, "Key": src}, self.bucket_name, dest, Config=self.transfer_config)
                     pbar.update(1)
             
             self.delete_folder(src_prefix)

aipmodel-0.2.69/aipmodel/__init__.py

@@ -0,0 +1,15 @@
+import os as _os
+
+from .CephS3Manager import CephS3Manager
+from .model_registry import MLOpsManager
+from .update_checker import check_latest_version
+
+# The PyPI update check makes a network call; doing that on EVERY import stalls
+# air-gapped/cluster startups. Now opt-in: set AIPMODEL_UPDATE_CHECK=1 to enable.
+if _os.environ.get("AIPMODEL_UPDATE_CHECK", "").lower() in ("1", "true", "yes"):
+    check_latest_version("aipmodel")
+
+__all__ = ["MLOpsManager", "CephS3Manager"]
+
+__version__ = "0.2.69"
+__description__ = "SDK for model registration, versioning, and storage"

aipmodel-0.2.69/aipmodel/exceptions.py

@@ -0,0 +1,23 @@
+"""
+Typed SDK exceptions.
+
+All subclass ValueError so existing `except ValueError` callers keep working
+(fully backward compatible), while new callers — e.g. the HTTP API — can map
+each type to a precise status code instead of guessing from one generic error.
+"""
+
+
+class ModelRegistryError(ValueError):
+    """Base class for all aipmodel errors."""
+
+
+class ModelNotFoundError(ModelRegistryError):
+    """The requested model / version does not exist."""
+
+
+class MetadataError(ModelRegistryError):
+    """The registry (Mlops-Core) rejected the request or returned bad data."""
+
+
+class StorageError(ModelRegistryError):
+    """S3 / Ceph operation failed (upload, download, copy, delete)."""

{aipmodel-0.2.68 → aipmodel-0.2.69}/aipmodel/model_registry.py

@@ -18,6 +18,7 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
 
 from .CephS3Manager import CephS3Manager
 from .acl_manager import AclManager
+from .exceptions import ModelRegistryError, ModelNotFoundError, MetadataError, StorageError
 
 load_dotenv()
 
@@ -57,7 +58,7 @@ class ProjectsAPI:
         
         response = self._post("/projects.create", {"name": name, "description": description})
         if not response or "id" not in response:
-            error_msg = "Failed to create project in ClearML"
+            error_msg = "Failed to create project in Mlops-Core"
             print(f"[SDK_FAIL] {error_msg}")
             raise ValueError(error_msg)
         
@@ -71,7 +72,7 @@ class ProjectsAPI:
         
         response = self._post("/projects.get_all")
         if not response or "projects" not in response:
-            error_msg = "Failed to retrieve projects from ClearML"
+            error_msg = "Failed to retrieve projects from Mlops-Core"
             print(f"[SDK_FAIL] {error_msg}")
             raise ValueError(error_msg)
         
@@ -107,7 +108,7 @@ class ModelsAPI:
 
         error_msg = f"'models' not found in response"
         print(f"[SDK_ERROR] {error_msg}")
-        raise ValueError("Failed to retrieve models from ClearML")
+        raise ValueError("Failed to retrieve models from Mlops-Core")
 
     def create(self, name, project_id, metadata=None, uri="", tags=None):
         if self.verbose:
@@ -126,7 +127,7 @@ class ModelsAPI:
 
         response = self._post("/models.create", payload)
         if not response or "id" not in response:
-            error_msg = "Failed to create model in ClearML"
+            error_msg = "Failed to create model in Mlops-Core"
             print(f"[SDK_FAIL] {error_msg}")
             raise ValueError(error_msg)
         
@@ -213,9 +214,15 @@ def get_user_info_with_bearer(bearer_token: str, user_management_url, timeout: i
             headers={"Authorization": f"Bearer {bearer_token}"},
             timeout=timeout,
         )
+        if response.status_code in (401, 403):
+            # Invalid/expired token is an AUTH failure, not an outage — callers
+            # (e.g. the HTTP API) must be able to return 401/403, not 503.
+            raise PermissionError(
+                f"Authentication failed (HTTP {response.status_code}): {response.text}"
+            )
         if response.status_code != 200:
             raise ConnectionError(
-                f"Failed to authenticate (HTTP {response.status_code}): {response.text}"
+                f"User management error (HTTP {response.status_code}): {response.text}"
             )
         return response.json()
     except requests.RequestException as e:
@@ -353,14 +360,16 @@ class MLOpsManager:
     def __init__(
         self,
         user_token,
-        CLEARML_API_HOST=None,
-        CLEARML_WEB_HOST=None,
+        CLEARML_API_HOST=None,   # deprecated alias of MLOPS_CORE_API_HOST (legacy compat)
+        CLEARML_WEB_HOST=None,   # deprecated alias of MLOPS_CORE_WEB_HOST (legacy compat)
         CEPH_ENDPOINT_URL=None,
         USER_MANAGEMENT_API=None,
         verbose=False,
         skip_connection_check=False,
         ACL_API_HOST=None,
-        ACL_ENABLED=None
+        ACL_ENABLED=None,
+        MLOPS_CORE_API_HOST=None,
+        MLOPS_CORE_WEB_HOST=None
     ):
         self.verbose = verbose
         if self.verbose:
@@ -373,13 +382,27 @@ class MLOpsManager:
         self.ALLOWED_CATEGORIES = self.resolve_allowed_categories()
 
         self.USER_TOKEN = user_token
-        self.CLEARML_API_HOST = CLEARML_API_HOST or os.environ.get("CLEARML_API_HOST")
+        # Preferred configuration: MLOPS_CORE_* kwargs / env vars. The legacy
+        # CLEARML_* kwargs and env vars are still accepted as deprecated
+        # fallbacks for backward compatibility; when both are given, the new
+        # MLOPS_CORE_* name wins.
+        self.MLOPS_CORE_API_HOST = (
+            MLOPS_CORE_API_HOST
+            or CLEARML_API_HOST  # deprecated kwarg (legacy compat)
+            or os.environ.get("MLOPS_CORE_API_HOST")
+            or os.environ.get("CLEARML_API_HOST")  # legacy env var fallback
+        )
         # Ensure we prioritize specific env var for WEB HOST
-        self.CLEARML_WEB_HOST = CLEARML_WEB_HOST or os.environ.get("CLEARML_WEB_HOST")
-        
-        if not self.CLEARML_WEB_HOST:
-            self.CLEARML_WEB_HOST = self.CLEARML_API_HOST
-            print(f"[SDK_WARN] CLEARML_WEB_HOST not set. Defaulting to API host: {self.CLEARML_API_HOST}")
+        self.MLOPS_CORE_WEB_HOST = (
+            MLOPS_CORE_WEB_HOST
+            or CLEARML_WEB_HOST  # deprecated kwarg (legacy compat)
+            or os.environ.get("MLOPS_CORE_WEB_HOST")
+            or os.environ.get("CLEARML_WEB_HOST")  # legacy env var fallback
+        )
+
+        if not self.MLOPS_CORE_WEB_HOST:
+            self.MLOPS_CORE_WEB_HOST = self.MLOPS_CORE_API_HOST
+            print(f"[SDK_WARN] MLOPS_CORE_WEB_HOST not set. Defaulting to API host: {self.MLOPS_CORE_API_HOST}")
             print("[SDK_WARN] Web links generated might be incorrect if API and Web UI ports differ.")
         
         self.USER_MANAGEMENT_API = USER_MANAGEMENT_API or os.environ.get("USER_MANAGEMENT_API")
@@ -390,10 +413,10 @@ class MLOpsManager:
         self.PUBLIC_ACCESS_KEY = os.environ.get("CEPH_ADMIN_ACCESS_KEY")
         self.PUBLIC_SECRET_KEY = os.environ.get("CEPH_ADMIN_SECRET_KEY")
 
-        user_info, self.CLEARML_USERNAME = get_user_metadata(bearer_token=user_token, user_management_url=self.USER_MANAGEMENT_API)
+        user_info, self.MLOPS_CORE_USERNAME = get_user_metadata(bearer_token=user_token, user_management_url=self.USER_MANAGEMENT_API)
 
         missing_cfg = [k for k, v in {
-            "CLEARML_API_HOST": self.CLEARML_API_HOST,
+            "MLOPS_CORE_API_HOST": self.MLOPS_CORE_API_HOST,
             "USER_MANAGEMENT_API": self.USER_MANAGEMENT_API,
             "CEPH_ENDPOINT_URL": self.CEPH_ENDPOINT_URL,
         }.items() if not v]
@@ -407,18 +430,20 @@ class MLOpsManager:
         self.CEPH_ADMIN_SECRET_KEY = user_info["s3_secret_key"]
         self.CEPH_USER_BUCKET = user_info["s3_bucket"]
         
-        self.CLEARML_ACCESS_KEY = user_info["clearml_access_key"]
-        self.CLEARML_SECRET_KEY = user_info["clearml_secret_key"]
+        # NOTE: "clearml_access_key" / "clearml_secret_key" are legacy key names
+        # from the user-management wire format — do not rename them.
+        self.MLOPS_CORE_ACCESS_KEY = user_info["clearml_access_key"]
+        self.MLOPS_CORE_SECRET_KEY = user_info["clearml_secret_key"]
         
         if not skip_connection_check:
             if self.verbose:
                 print("[SDK_INFO] Performing health checks...")
             if not self.check_user_management_service():
                  raise ValueError("User Management API unreachable")
-            if not self.check_clearml_service():
-                raise ValueError("ClearML Server down")
-            if not self.check_clearml_auth():
-                raise ValueError("ClearML Auth incorrect")
+            if not self.check_mlops_core_service():
+                raise ValueError("Mlops-Core Server down")
+            if not self.check_mlops_core_auth():
+                raise ValueError("Mlops-Core Auth incorrect")
             if self.verbose:
                 print("[SDK_SUCCESS] Health checks passed")
 
@@ -470,29 +495,29 @@ class MLOpsManager:
                 print("[SDK_INFO] ACL enforcement disabled (bypass mode)")
 
         if self.verbose:
-            print("[SDK_INFO] Authenticating with ClearML...")
+            print("[SDK_INFO] Authenticating with Mlops-Core...")
 
-        creds = f"{self.CLEARML_ACCESS_KEY}:{self.CLEARML_SECRET_KEY}"
+        creds = f"{self.MLOPS_CORE_ACCESS_KEY}:{self.MLOPS_CORE_SECRET_KEY}"
         auth_header = b64encode(creds.encode("utf-8")).decode("utf-8")
         res = requests.post(
-            f"{self.CLEARML_API_HOST}/auth.login",
+            f"{self.MLOPS_CORE_API_HOST}/auth.login",
             headers={"Authorization": f"Basic {auth_header}"},
             timeout=self.REQUEST_TIMEOUT
         )
         if res.status_code != 200:
-            error_msg = "Failed to authenticate with ClearML"
+            error_msg = "Failed to authenticate with Mlops-Core"
             print(f"[SDK_FAIL] {error_msg}")
             raise ValueError(error_msg)
         self.token = res.json()["data"]["token"]
         
         if self.verbose:
-            print("[SDK_SUCCESS] Authenticated with ClearML")
+            print("[SDK_SUCCESS] Authenticated with Mlops-Core")
 
         self.projects = ProjectsAPI(self._post, verbose=self.verbose)
         self.models = ModelsAPI(self._post, verbose=self.verbose)
 
         projects = self.projects.get_all()
-        self.project_name = f"project_{self.CLEARML_USERNAME}"
+        self.project_name = f"project_{self.MLOPS_CORE_USERNAME}"
         exists = [p for p in projects if p["name"] == self.project_name]
         self.project_id = exists[0]["id"] if exists else self.projects.create(self.project_name)["id"]
         
@@ -505,12 +530,12 @@ class MLOpsManager:
 
         headers = {"Authorization": f"Bearer {self.token}"}
         try:
-            res = requests.post(f"{self.CLEARML_API_HOST}{path}", headers=headers, json=params, timeout=self.REQUEST_TIMEOUT)
+            res = requests.post(f"{self.MLOPS_CORE_API_HOST}{path}", headers=headers, json=params, timeout=self.REQUEST_TIMEOUT)
             res.raise_for_status()
 
             data = res.json()
             if "data" not in data:
-                error_msg = f"Unexpected response from ClearML {path}: missing 'data' field"
+                error_msg = f"Unexpected response from Mlops-Core {path}: missing 'data' field"
                 print(f"[SDK_ERROR] {error_msg}")
                 raise ValueError(error_msg)
 
@@ -518,13 +543,18 @@ class MLOpsManager:
 
         except requests.exceptions.HTTPError as e:
             status = e.response.status_code if (hasattr(e, "response") and e.response is not None) else 0
-            error_msg = f"ClearML returned HTTP {status} for {path}: {e}"
+            error_msg = f"Mlops-Core returned HTTP {status} for {path}: {e}"
             print(f"[SDK_ERROR] {error_msg}")
             if status in (401, 403):
-                raise PermissionError(f"ClearML access denied (HTTP {status}) on {path}: {e}") from e
+                raise PermissionError(f"Mlops-Core access denied (HTTP {status}) on {path}: {e}") from e
+            if 400 <= status < 500:
+                # A 4xx is a BAD REQUEST (e.g. nonexistent id) — not an outage.
+                # Raising MetadataError (a ValueError) lets callers/API report
+                # "not found / invalid" instead of a bogus 503.
+                raise MetadataError(error_msg) from e
             raise ConnectionError(error_msg) from e
         except requests.exceptions.RequestException as e:
-            error_msg = f"Network error calling ClearML {path}: {e}"
+            error_msg = f"Network error calling Mlops-Core {path}: {e}"
             print(f"[SDK_ERROR] {error_msg}")
             raise ConnectionError(error_msg) from e
         except ValueError:
@@ -546,49 +576,49 @@ class MLOpsManager:
                 print(f"[SDK_FAIL] User Management API Check Failed: {e}")
             return False
 
-    def check_clearml_service(self):
+    def check_mlops_core_service(self):
         try:
-            r = requests.get(self.CLEARML_API_HOST + "/auth.login", timeout=5)
+            r = requests.get(self.MLOPS_CORE_API_HOST + "/auth.login", timeout=5)
             if r.status_code in [200, 401]:
                 if self.verbose:
-                    print("[SDK_SUCCESS] ClearML Service Reachable")
+                    print("[SDK_SUCCESS] Mlops-Core Service Reachable")
                 return True
             raise ValueError(f"Status Code: {r.status_code}")
         except Exception as e:
             if self.verbose:
-                print(f"[SDK_FAIL] ClearML Service Check Failed: {e}")
+                print(f"[SDK_FAIL] Mlops-Core Service Check Failed: {e}")
             return False
 
-    def check_clearml_auth(self):
+    def check_mlops_core_auth(self):
         try:
-            creds = f"{self.CLEARML_ACCESS_KEY}:{self.CLEARML_SECRET_KEY}"
+            creds = f"{self.MLOPS_CORE_ACCESS_KEY}:{self.MLOPS_CORE_SECRET_KEY}"
             auth_header = b64encode(creds.encode("utf-8")).decode("utf-8")
             r = requests.post(
-                self.CLEARML_API_HOST + "/auth.login",
+                self.MLOPS_CORE_API_HOST + "/auth.login",
                 headers={"Authorization": f"Basic {auth_header}"},
                 timeout=5
             )
             if r.status_code == 200:
                 if self.verbose:
-                    print("[SDK_SUCCESS] ClearML Auth Valid")
+                    print("[SDK_SUCCESS] Mlops-Core Auth Valid")
                 return True
             raise ValueError(f"Status Code: {r.status_code}")
         except Exception as e:
             if self.verbose:
-                print(f"[SDK_FAIL] ClearML Auth Check Failed: {e}")
+                print(f"[SDK_FAIL] Mlops-Core Auth Check Failed: {e}")
             return False
 
     def get_model_id_by_name(self, name, owner=None):
         """
-        Resolves a model name to its ClearML ID.
+        Resolves a model name to its Mlops-Core ID.
 
         owner: if omitted, searches your own project only (original behavior).
-        If given (a CLEARML username), searches that user's project instead -
+        If given (a Mlops-Core username), searches that user's project instead -
         use this to resolve a model you don't own, e.g. a public model
         someone else created with the same kind of name.
         """
         project_id = self.project_id
-        if owner and owner != self.CLEARML_USERNAME:
+        if owner and owner != self.MLOPS_CORE_USERNAME:
             target_project_name = f"project_{owner}"
             match = next((p for p in self.projects.get_all() if p["name"] == target_project_name), None)
             if not match:
@@ -616,7 +646,7 @@ class MLOpsManager:
 
     def _resolve_owner_for_project(self, project_id):
         if project_id == self.project_id:
-            return self.CLEARML_USERNAME
+            return self.MLOPS_CORE_USERNAME
         return self._get_project_owner_map().get(project_id, "unknown")
 
     def get_model_link(self, model_id):
@@ -631,7 +661,7 @@ class MLOpsManager:
         except Exception:
             pass
 
-        host = self.CLEARML_WEB_HOST.rstrip('/')
+        host = self.MLOPS_CORE_WEB_HOST.rstrip('/')
         link = f"{host}/projects/{project_id}/models/{model_id}"
 
         if self.verbose:
@@ -897,6 +927,7 @@ class MLOpsManager:
         temp_local_path = None
         have_model_py = False
         size_mb = 0.0
+        moved_to_final = False
 
         try:
             if self.verbose:
@@ -921,11 +952,41 @@ class MLOpsManager:
                 current_manager.upload(code_path, dest_temp_path + "model.py")
                 have_model_py = True
 
+            # Recovery metadata: a small _meta.json travels WITH the files so a
+            # registry entry can be rebuilt from storage alone (see
+            # recover_model_from_storage) if the Mlops-Core metadata is ever lost.
+            recovery_meta = {
+                "model_id": model_id,
+                "model_name": model_name,
+                "version": version,
+                "version_number": next_version_number,
+                "category": category,
+                "tags": final_tags,
+                "owner": self.MLOPS_CORE_USERNAME,
+                "public": public,
+                "folderName": model_folder_name,
+                "haveModelPy": have_model_py,
+                "size": size_mb,
+                "created": time.strftime("%Y-%m-%d %H:%M:%S"),
+            }
+            try:
+                current_manager.s3.put_object(
+                    Bucket=current_manager.bucket_name,
+                    Key=dest_temp_path + "_meta.json",
+                    Body=json.dumps(recovery_meta).encode("utf-8"),
+                    ContentType="application/json",
+                )
+            except Exception as e:
+                # best-effort: never fail an add because of the recovery sidecar
+                if self.verbose:
+                    print(f"[SDK_WARN] Could not write recovery _meta.json: {e}")
+
             if current_manager.check_if_exists(dest_version_path):
                 print(f"[SDK_WARN] Version {version} exists. Overwriting...")
                 current_manager.delete_folder(dest_version_path)
-            
+
             current_manager.move_folder(dest_temp_path, dest_version_path)
+            moved_to_final = True  # from here on, rollback must clean dest_version_path too
 
             model_data = self.models.get_by_id(model_id)
             metadata = model_data.get("metadata", {})
@@ -985,11 +1046,29 @@ class MLOpsManager:
                             print("[SDK_INFO] Rolled back new model creation")
                 except Exception:
                     pass
+                # ACL was registered before upload — roll it back too so the ACL
+                # service doesn't keep a record of a model that no longer exists.
+                if is_new_model and self.acl:
+                    try:
+                        self.acl.delete_model(model_id)
+                    except Exception:
+                        pass
             if dest_temp_path and "current_manager" in locals():
                 try:
                     current_manager.delete_folder(dest_temp_path)
                 except Exception:
                     pass
+            # If files were already moved to the FINAL path but the metadata
+            # write failed, those files would be invisible zombies (metadata
+            # doesn't reference them and the *_tmp_* sweep won't find them).
+            # Delete them so registry and S3 stay consistent.
+            if moved_to_final and "current_manager" in locals():
+                try:
+                    current_manager.delete_folder(dest_version_path)
+                    if self.verbose:
+                        print("[SDK_INFO] Rolled back uploaded files from final path")
+                except Exception:
+                    pass
             raise
         finally:
             for path in [local_path, temp_local_path]:
@@ -1207,7 +1286,7 @@ class MLOpsManager:
         
         # 2. If not found by name, assume it might be an ID
         if not model_id:
-            # Simple regex check for ClearML ID format (32 hex chars) just to be safe, 
+            # Simple regex check for Mlops-Core ID format (32 hex chars) just to be safe, 
             # or directly attempt to use it as ID.
             if re.match(r'^[a-f0-9]{32}$', model_identifier):
                 model_id = model_identifier
@@ -1216,8 +1295,10 @@ class MLOpsManager:
 
         try:
             model_data = self.models.get_by_id(model_id)
+        except (PermissionError, ConnectionError):
+            raise  # outage/auth must NOT be reported as "not found"
         except Exception:
-             raise ValueError(f"Model data retrieval failed for ID: {model_id}")
+            raise ModelNotFoundError(f"Model data retrieval failed for ID: {model_id}")
 
         metadata = model_data.get("metadata", {})
         latest_version = self._extract_metadata_value(metadata, "latest_version")
@@ -1253,8 +1334,10 @@ class MLOpsManager:
 
         try:
             data = self.models.get_by_id(model_id)
+        except (PermissionError, ConnectionError):
+            raise  # outage/auth must NOT be reported as "not found"
         except Exception:
-             raise ValueError(f"No model found with identifier: '{identifier}'")
+            raise ModelNotFoundError(f"No model found with identifier: '{identifier}'")
 
         m = data
         metadata = m.get("metadata", {})
@@ -1440,7 +1523,7 @@ class MLOpsManager:
                 print(f"{name:<25} | {mid:<25} | {latest:<10} | {public_display:<6} | {str(total_size):<10} | {display_versions}")
                 
                 owner_username = (
-                    self.CLEARML_USERNAME if mid in own_ids
+                    self.MLOPS_CORE_USERNAME if mid in own_ids
                     else project_owner_map.get(m.get("project"), "unknown")
                 )
                 model_entry = {
@@ -1504,15 +1587,19 @@ class MLOpsManager:
             path_to_delete = versions_map[version].get("path")
             if not path_to_delete:
                 raise ValueError(f"Version '{version}' of model '{model_name or model_id}' has no stored path — metadata may be corrupted")
-            ceph_manager.delete_folder(path_to_delete)
+            # Metadata-FIRST ordering: update the registry before deleting S3
+            # files. If we crash in between, the worst case is an orphan S3
+            # folder (harmless, discoverable by verify tooling) — never a
+            # registry entry pointing at deleted files (a zombie the dashboard
+            # would render as a broken model).
             del versions_map[version]
-            
+
             if not versions_map:
                 print(f"[SDK_INFO] No versions left for model. Deleting entire model...")
-                ceph_manager.delete_folder(f"_models/{model_id}/")
                 self.models.delete(model_id)
                 if self.acl:
                     self.acl.delete_model(model_id)
+                ceph_manager.delete_folder(f"_models/{model_id}/")
                 print(f"[SDK_SUCCESS] Model '{model_name or model_id}' completely deleted.")
                 return
             else:
@@ -1546,12 +1633,216 @@ class MLOpsManager:
                     {"key": "haveModelPy", "type": "str", "value": versions_map[new_latest].get("haveModelPy", "false") if new_latest in versions_map else "false"}
                 ]
                 self.models.update_metadata(model_id, metadata=new_metadata)
-            
+
+            # S3 delete LAST (see metadata-first note above).
+            ceph_manager.delete_folder(path_to_delete)
+
             print(f"[SDK_SUCCESS] Version {version} deleted.")
 
         else:
-            ceph_manager.delete_folder(f"_models/{model_id}/")
+            # Metadata-first here too: registry entry goes before the files.
             self.models.delete(model_id)
             if self.acl:
                 self.acl.delete_model(model_id)
-            print(f"[SDK_SUCCESS] Model '{model_id}' deleted completely.")
+            ceph_manager.delete_folder(f"_models/{model_id}/")
+            print(f"[SDK_SUCCESS] Model '{model_id}' deleted completely.")
+
+    # ------------------------------------------------------------------
+    # Registry <-> Storage consistency (zombie/orphan detection & recovery)
+    # ------------------------------------------------------------------
+
+    def verify_model_integrity(self, model_name=None, model_id=None):
+        """
+        Cross-check one model between the registry (Mlops-Core) and S3.
+
+        Returns:
+            {
+              "model_id", "model_name", "status": "ok"|"missing_files"|"orphan_files",
+              "versions": [{"version","path","files_exist","status"}...],
+              "orphan_folders": [<S3 folders under the model not referenced by any version>],
+            }
+        Raises ModelNotFoundError if the model is not in the registry at all.
+        """
+        if not model_id:
+            model_id = self.get_model_id_by_name(model_name)
+            if not model_id:
+                raise ModelNotFoundError(f"Model not found: '{model_name}'")
+        data = self.models.get_by_id(model_id)
+        metadata = data.get("metadata", {})
+        versions_map = self._parse_versions_map(metadata)
+        is_public = self._extract_metadata_value(metadata, "public", "false").lower() == "true"
+        mgr = self._get_ceph_manager(is_public)
+
+        report = {
+            "model_id": model_id,
+            "model_name": data.get("name"),
+            "status": "ok",
+            "versions": [],
+            "orphan_folders": [],
+        }
+        referenced_dirs = set()
+        for ver, info in versions_map.items():
+            if not isinstance(info, dict):
+                continue
+            path = info.get("path") or ""
+            if path:
+                referenced_dirs.add(path.rstrip("/").split("/")[-1])
+            files_exist = bool(path) and mgr.check_if_exists(path)
+            report["versions"].append({
+                "version": ver,
+                "path": path,
+                "files_exist": files_exist,
+                "status": "ok" if files_exist else "missing_files",
+            })
+
+        # S3 folders under this model that no registry version references
+        base = f"_models/{model_id}/"
+        tops = set()
+        try:
+            keys = mgr.list_folder_contents(base) if mgr.check_if_exists(base) else []
+            for k in keys:
+                rel = k[len(base):] if k.startswith(base) else k
+                top = rel.split("/", 1)[0]
+                if top and top != "_meta.json":
+                    tops.add(top)
+        except Exception:
+            pass  # storage listing unavailable -> report only version checks
+        report["orphan_folders"] = sorted(t for t in tops if t not in referenced_dirs)
+
+        if any(v["status"] != "ok" for v in report["versions"]):
+            report["status"] = "missing_files"
+        elif report["orphan_folders"]:
+            report["status"] = "orphan_files"
+        return report
+
+    def verify_all_models(self):
+        """Run verify_model_integrity over every model in the user's project.
+        Returns a list of per-model reports (never raises per-model; a model
+        that fails to verify is reported with status "error")."""
+        reports = []
+        models = self.models.get_all(self.project_id) or []
+        for m in models:
+            mid = m.get("id")
+            try:
+                reports.append(self.verify_model_integrity(model_id=mid))
+            except (PermissionError, ConnectionError):
+                raise
+            except Exception as e:
+                reports.append({
+                    "model_id": mid, "model_name": m.get("name"),
+                    "status": "error", "error": str(e),
+                    "versions": [], "orphan_folders": [],
+                })
+        return reports
+
+    def find_orphan_models(self):
+        """
+        Find S3 model folders (under `_models/`) that the registry does NOT
+        know about — e.g. leftovers after a registry (Mlops-Core) data loss.
+        Returns [{"model_id", "recoverable" (has _meta.json), "meta": {...}|None}].
+        """
+        mgr = self._get_ceph_manager(False)
+        registry_ids = {m.get("id") for m in (self.models.get_all(self.project_id) or [])}
+        s3_ids = set()
+        try:
+            for k in mgr.list_folder_contents("_models/"):
+                rel = k[len("_models/"):] if k.startswith("_models/") else k
+                top = rel.split("/", 1)[0]
+                if top:
+                    s3_ids.add(top)
+        except Exception:
+            pass
+        orphans = []
+        for mid in sorted(s3_ids - registry_ids):
+            meta = self._read_first_meta_json(mgr, mid)
+            orphans.append({"model_id": mid, "recoverable": meta is not None, "meta": meta})
+        return orphans
+
+    def _read_first_meta_json(self, mgr, model_id):
+        """Read any one version's recovery _meta.json for a model, else None."""
+        try:
+            for k in mgr.list_folder_contents(f"_models/{model_id}/"):
+                if k.endswith("/_meta.json"):
+                    try:
+                        obj = mgr.s3.get_object(Bucket=mgr.bucket_name, Key=k)
+                        return json.loads(obj["Body"].read().decode("utf-8"))
+                    except Exception:
+                        continue
+        except Exception:
+            pass
+        return None
+
+    def recover_model_from_storage(self, model_id):
+        """
+        Rebuild a registry (Mlops-Core) entry for a model that exists in S3 but
+        not in the registry, using the per-version `_meta.json` sidecar files
+        written by add_model. This is the disaster-recovery path when the
+        registry's data is lost while S3 survives.
+
+        Returns the new registry model id.
+        """
+        mgr = self._get_ceph_manager(False)
+        base = f"_models/{model_id}/"
+        try:
+            keys = mgr.list_folder_contents(base)
+        except Exception as e:
+            raise StorageError(f"Cannot list storage for model {model_id}: {e}")
+        # collect every version folder + its _meta.json
+        metas = {}
+        for k in keys:
+            if k.endswith("/_meta.json"):
+                try:
+                    obj = mgr.s3.get_object(Bucket=mgr.bucket_name, Key=k)
+                    meta = json.loads(obj["Body"].read().decode("utf-8"))
+                    metas[meta.get("version")] = (k, meta)
+                except Exception:
+                    continue
+        if not metas:
+            raise StorageError(
+                f"No recovery _meta.json found under {base} — this model predates "
+                f"recovery sidecars and must be re-registered manually."
+            )
+        any_meta = next(iter(metas.values()))[1]
+        name = any_meta.get("model_name") or f"recovered_{model_id[:8]}"
+        if self.get_model_id_by_name(name):
+            raise ValueError(f"A model named '{name}' already exists in the registry — refusing to overwrite.")
+
+        tags = any_meta.get("tags") or []
+        category = any_meta.get("category")
+        created_model = self.models.create(name=name, project_id=self.project_id, uri="s3://placeholder", tags=tags)
+        new_id = created_model["id"]
+
+        versions_map, latest, latest_num = {}, None, -1
+        for ver, (key, meta) in metas.items():
+            old_prefix = key[: -len("_meta.json")]           # _models/<old_id>/<ver>/
+            versions_map[ver] = {
+                "path": old_prefix,
+                "size": meta.get("size", 0.0),
+                "created": meta.get("created", ""),
+                "haveModelPy": str(meta.get("haveModelPy", False)).lower(),
+                "folderName": meta.get("folderName", ""),
+                "version_number": meta.get("version_number", 0),
+            }
+            try:
+                vn = int(meta.get("version_number", 0))
+            except Exception:
+                vn = 0
+            if vn >= latest_num:
+                latest_num, latest = vn, ver
+
+        total_size = sum(float(v.get("size") or 0) for v in versions_map.values())
+        new_metadata = [
+            {"key": "versions_map", "type": "str", "value": json.dumps(versions_map)},
+            {"key": "latest_version", "type": "str", "value": latest or ""},
+            {"key": "modelSize", "type": "float", "value": f"{total_size:.2f}"},
+            {"key": "version_count", "type": "int", "value": str(len(versions_map))},
+            {"key": "public", "type": "bool", "value": str(any_meta.get("public", False)).lower()},
+            {"key": "recovered_from", "type": "str", "value": model_id},
+        ]
+        if category:
+            new_metadata.append({"key": "category", "type": "str", "value": category})
+        uri = f"s3://{mgr.bucket_name}/{versions_map[latest]['path']}" if latest else "s3://placeholder"
+        self.models.update_info(new_id, uri=uri, tags=tags)
+        self.models.update_metadata(new_id, metadata=new_metadata)
+        print(f"[SDK_SUCCESS] Recovered model '{name}' ({len(versions_map)} versions) as registry id {new_id} (files remain at {base}).")
+        return new_id

{aipmodel-0.2.68 → aipmodel-0.2.69}/aipmodel/template.py

@@ -11,7 +11,7 @@ print("\n[STEP 1] Initialize MLOps Manager")
 # verbose=True enables detailed logging (Transparency Mode)
 manager = MLOpsManager(
     user_token=os.getenv("USER_TOKEN"), 
-    CLEARML_API_HOST=os.getenv("CLEARML_API_HOST"),
+    MLOPS_CORE_API_HOST=os.getenv("MLOPS_CORE_API_HOST"),
     CEPH_ENDPOINT_URL=os.getenv("CEPH_ENDPOINT_URL"),
     USER_MANAGEMENT_API=os.getenv("USER_MANAGEMENT_API"),
     verbose=True 

{aipmodel-0.2.68 → aipmodel-0.2.69}/aipmodel.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aipmodel
-Version: 0.2.68
+Version: 0.2.69
 Summary: SDK for model registration, versioning, and storage
 Home-page: https://github.com/AIP-MLOPS/model-registry
 Author: AIP MLOPS Team

{aipmodel-0.2.68 → aipmodel-0.2.69}/aipmodel.egg-info/SOURCES.txt

@@ -5,6 +5,7 @@ setup.py
 aipmodel/CephS3Manager.py
 aipmodel/__init__.py
 aipmodel/acl_manager.py
+aipmodel/exceptions.py
 aipmodel/model_registry.py
 aipmodel/template.py
 aipmodel/update_checker.py

{aipmodel-0.2.68 → aipmodel-0.2.69}/setup.py

@@ -31,7 +31,7 @@ version, description = read_meta()
 
 setup(
     name="aipmodel",
-    version="0.2.68",
+    version="0.2.69",
     description=description,
     author="AIP MLOPS Team",
     author_email="mohmmadweb@gmail.com",

aipmodel-0.2.68/aipmodel/__init__.py

@@ -1,10 +0,0 @@
-from .CephS3Manager import CephS3Manager
-from .model_registry import MLOpsManager
-from .update_checker import check_latest_version
-
-check_latest_version("aipmodel")
-
-__all__ = ["MLOpsManager", "CephS3Manager"]
-
-__version__ = "0.2.68"
-__description__ = "SDK for model registration, versioning, and storage"