aipmodel 0.2.64__tar.gz → 0.2.66__tar.gz

{aipmodel-0.2.64 → aipmodel-0.2.66}/PKG-INFO

@@ -1,23 +1,16 @@
 Metadata-Version: 2.4
 Name: aipmodel
-Version: 0.2.64
+Version: 0.2.66
 Summary: SDK for model registration, versioning, and storage
 Home-page: https://github.com/AIP-MLOPS/model-registry
 Author: AIP MLOPS Team
 Author-email: mohmmadweb@gmail.com
 Requires-Python: >=3.6
 Requires-Dist: boto3==1.34.0
-Requires-Dist: clearml==2.0.2
-Requires-Dist: fastapi==0.128.2
-Requires-Dist: hf_xet==1.2.0
 Requires-Dist: huggingface-hub==0.36.2
-Requires-Dist: pydantic==2.12.5
 Requires-Dist: python-dotenv==1.2.1
 Requires-Dist: requests==2.32.5
-Requires-Dist: tabulate==0.9.0
 Requires-Dist: tqdm==4.67.1
-Requires-Dist: ruff==0.14.7
-Requires-Dist: uvicorn[standard]==0.38.0
 Dynamic: author
 Dynamic: author-email
 Dynamic: home-page

aipmodel-0.2.66/README.md

@@ -0,0 +1,371 @@
+# AIP Model SDK
+
+This SDK provides a robust interface for registering, uploading, downloading, listing, and deleting machine learning models using ClearML and S3 (Ceph) as storage. It supports **automatic versioning**, **categorization**, **tagging**, **metadata transparency**, **web link generation**, and fail-fast connectivity checks.
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install aipmodel
+
+```
+
+---
+
+## Configuration & Authentication
+
+To use the SDK, you must set up your environment variables. You can create a `.env` file in your project root:
+
+```env
+# ClearML API Credentials
+CLEARML_API_HOST="http://your-clearml-server:8008"
+CLEARML_WEB_HOST="http://your-clearml-web-ui:8080"  # Required for generating model web links
+CLEARML_ACCESS_KEY="your_access_key"
+CLEARML_SECRET_KEY="your_secret_key"
+
+# Ceph / S3 Storage
+CEPH_ENDPOINT_URL="https://your-ceph-endpoint.com"
+
+# User Management (Internal)
+USER_MANAGEMENT_API="http://your-user-manager:30009"
+USER_TOKEN="your_bearer_token" # Or pass explicitly in code
+
+# Public bucket (used by public=True / set_model_public)
+CEPH_PUBLIC_BUCKET_NAME="aip-public"
+CEPH_ADMIN_ACCESS_KEY="your_public_bucket_access_key"
+CEPH_ADMIN_SECRET_KEY="your_public_bucket_secret_key"
+
+# ACL (Access Control) - optional, see "Access Control (ACL)" below
+ACL_ENABLED="false"
+ACL_API_HOST="http://your-acl-server:30011"
+
+# Optional overrides (all have sane defaults if omitted)
+SDK_REQUEST_TIMEOUT="30"
+MODEL_ALLOWED_CATEGORIES="Text Generation,Image Classification"
+S3_MULTIPART_THRESHOLD_MB="64"
+S3_MULTIPART_CHUNKSIZE_MB="16"
+S3_MAX_CONCURRENCY="10"
+S3_UPLOAD_WORKERS="4"
+S3_DOWNLOAD_WORKERS="4"
+
+# Auto-retry for transient connection drops during large transfers
+S3_MAX_RETRIES="5"               # per-file retries before giving up
+S3_RETRY_BACKOFF_SECONDS="2"     # base backoff, grows exponentially per attempt
+S3_RETRY_MAX_DELAY_SECONDS="30"  # cap on a single backoff sleep
+S3_BOTO_MAX_ATTEMPTS="10"        # botocore request-level retry attempts (adaptive)
+
+```
+
+> **Note:** `CLEARML_WEB_HOST` is new. If not provided, the SDK will default to the API host, which might result in incorrect clickable links in the output.
+
+> Every constant the SDK relies on (timeouts, categories, transfer tuning, ACL) is overridable via `.env` — nothing important is hardcoded.
+
+---
+
+## Logging Verbosity
+
+`MLOpsManager(..., verbose=False)` — **`False` is the default.**
+
+- `verbose=False`: only essential output is printed — operation start/result, successes, failures, and warnings that affect state (e.g. overwriting a version, falling back to a new "latest" version).
+- `verbose=True`: every internal step is also printed (`[SDK_DEBUG]`/`[SDK_INFO]` narration), including the full raw metadata for transparency.
+
+Nothing is ever logged to a file or hidden — all output goes to stdout via `print()`, so it's visible wherever the process runs.
+
+## Resilient Transfers (Auto-Retry)
+
+Large multi-GB uploads/downloads over Ceph/RGW occasionally hit a **transient**
+connection drop mid-stream (`IncompleteRead` / `ProtocolError` / connection
+reset). The SDK now **retries these automatically**, so `add_model`,
+`get_model` and `download` complete without manual intervention:
+
+- Each **individual file** transfer is retried on a transient error with
+  exponential backoff (`S3_RETRY_BACKOFF_SECONDS`, capped by
+  `S3_RETRY_MAX_DELAY_SECONDS`), up to `S3_MAX_RETRIES` times. Downloads reset
+  the partial file before each retry so no corrupt bytes remain.
+- The underlying boto3 client also uses botocore's **adaptive** request-level
+  retries (`S3_BOTO_MAX_ATTEMPTS`).
+- **Non-transient** errors (bad input, auth, not-found) are **not** retried —
+  they fail immediately as before.
+- Fully backward compatible: method signatures, return values and the final
+  `ValueError` raised on ultimate failure are unchanged. All knobs are
+  `.env`-overridable with sane defaults.
+
+## Concurrent Adds
+
+`add_model` is **concurrency-safe** — you can run many adds at the same time:
+
+- Adds with **different model names** run **fully in parallel** (own ClearML id,
+  own S3 prefix — no interference).
+- Adds with the **same `(project, model_name)`** are **serialized** by a
+  process-wide lock. Otherwise two simultaneous same-name adds would race on the
+  existence check + auto version-number computation and could create a
+  **duplicate model** or **overwrite a version**. With the lock the second add
+  waits for the first, then correctly continues to the next version
+  (`_v2` → `_v3`, …) — no duplicates, no lost versions.
+- Uploads are **staged**: files go to a temp S3 path first and are only moved to
+  the final version path once the whole upload succeeds; any failure rolls back
+  (deletes the new model if it was new + the temp folder) and re-raises, so a
+  crashed/aborted add never leaves a half-written version.
+- The lock is **per-process**. One server process (e.g. a single `uvicorn`
+  worker) is fully protected; for multiple workers/replicas add a distributed
+  lock (ClearML/Redis).
+
+## Access Control (ACL)
+
+ACL is **off by default** and fully optional. Toggle it with one env var:
+
+```env
+ACL_ENABLED="true"
+ACL_API_HOST="http://your-acl-server:30011"
+```
+
+- `ACL_ENABLED=false` (default): identical behavior to a build with no ACL at all — every call bypasses permission checks.
+- `ACL_ENABLED=true`: `add_model`, `get_model`, `get_model_info`, `delete_model`, `set_model_public`, `set_latest_version`, and `list_models` all enforce permission checks against the ACL service, and `add_model`/`delete_model`/`set_model_public` keep the ACL service's record of each model in sync.
+
+### Sharing a model
+
+```python
+manager.share_model(model_name="demo_model_local", target_user="colleague@example.com", access_level="write")
+# access_level: "read" | "write" | "delete" | "admin" | "owner"
+```
+
+Requires `ACL_ENABLED=true` and that you have `admin`-level access on the model. **Note:** the underlying ACL endpoint contract for sharing was inferred from the existing ACL client conventions and hasn't been verified against the live ACL service — confirm the request shape before relying on it in production.
+
+---
+
+## Cross-User Visibility (Public Models)
+
+Every model lives in its owner's own ClearML project (`project_<owner_username>`) — that part hasn't changed. What changed is **how `list_models`/`get_model_id_by_name`/`get_model_info`/`get_model` look things up**:
+
+- **`list_models()`** (default `include_public=True`): returns your own models (private + public) **plus** every other user's model that is marked `public=True`. Other users' **private** models are never returned, under any setting. Each entry's `owner` field is resolved from the model's actual project, so models from other users show their real owner, not yours.
+  - Pass `include_public=False` to get the exact pre-existing behavior: your own project only, nothing from other users.
+  - `public_only=True` / `private_only=True` now filter across this combined set too.
+- **`get_model_id_by_name(name, owner=None)`**, **`get_model(model_name, local_dest, version=None, owner=None)`**, **`get_model_info(identifier, owner=None)`**: all gained an optional `owner` argument. Omit it and behavior is identical to before (your own project only). Pass `owner="some_username"` to resolve a model by name from *that* user's project — useful once `list_models()` has shown you a public model that isn't yours.
+- **`get_model_link`** now resolves the link using the model's *actual* owning project (previously it always used your own project ID, which silently produced a wrong link for any model you didn't own — this never came up before because nothing could return another user's model anyway).
+- Write operations (`add_model`, `delete_model`, `set_model_public`, `set_latest_version`) are **unchanged** — they still only ever operate within your own project by name. To act on a model you don't own (e.g. as an ACL-granted admin), pass its `model_id` directly (already supported) rather than its name.
+
+**Performance note:** when `include_public=True` (the default), `list_models()` makes one extra unscoped ClearML query (`models.get_all` with no project filter) to find other users' public models — on a very large ClearML deployment this returns every model company-wide before filtering down to public ones. If that ever becomes slow, set `include_public=False` and only fetch other users' models on demand via `owner=`.
+
+**Backward compatibility:** `get_model_id_by_name`, `get_model`, `get_model_info` keep their exact old signatures plus one new optional trailing argument — no existing call site breaks. `list_models()`'s default *output* now includes more rows than before (other users' public models) since that was the explicit point of this change; pass `include_public=False` if your code depends on the exact old row set.
+
+---
+
+## Example Usage
+
+This example demonstrates how to use the new features: **Auto-Versioning**, **Categories**, **Tags**, and **Web Links**.
+
+```python
+import os
+from dotenv import load_dotenv
+from aipmodel.model_registry import MLOpsManager
+
+load_dotenv()
+
+# --- STEP 1: Initialize MLOps Manager ---
+print("\n[STEP 1] Initialize MLOps Manager")
+# verbose=True enables detailed logging and transparency mode.
+manager = MLOpsManager(
+    user_token=os.getenv("USER_TOKEN"), 
+    CLEARML_API_HOST=os.getenv("CLEARML_API_HOST"),
+    CLEARML_WEB_HOST=os.getenv("CLEARML_WEB_HOST"), # For generating UI links
+    CEPH_ENDPOINT_URL=os.getenv("CEPH_ENDPOINT_URL"),
+    USER_MANAGEMENT_API=os.getenv("USER_MANAGEMENT_API"),
+    verbose=True 
+)
+
+# --- STEP 2: Add Private Local Model (Explicit Version) ---
+print("\n[STEP 2] Add Private Model (Explicit Version + Metadata)")
+local_model_id = manager.add_model(
+    source_type="local",
+    model_name="demo_model_local",
+    version="v1.0", 
+    source_path="./my_models/v1",       
+    code_path="./my_models/train.py",   
+    category="Text Generation",         
+    tags=["production", "base-model"],
+    public=False                        # Default is False (Private Bucket)
+)
+print(f" -> Registered ID: {local_model_id}")
+
+# --- STEP 3: Add Auto-Versioned Model ---
+print("\n[STEP 3] Add Local Model (Auto-Versioning)")
+# Passing version=None triggers auto-generation (e.g., _v2, _v3 based on history)
+auto_ver_id = manager.add_model(
+    source_type="local",
+    model_name="demo_model_local",
+    version=None, 
+    source_path="./my_models/v2_finetuned",
+    tags=["experimental", "auto-versioned"]
+)
+print(f" -> Registered ID: {auto_ver_id}")
+
+# --- STEP 4: Add Public Model (Direct to Public Bucket) ---
+print("\n[STEP 4] Add Public Model")
+# Setting public=True uploads directly to the shared/public bucket
+public_model_id = manager.add_model(
+    source_type="local",
+    model_name="demo_model_public",
+    version="v1.0", 
+    source_path="./my_models/public_release",
+    tags=["community", "public"],
+    public=True 
+)
+print(f" -> Registered ID: {public_model_id}")
+
+# --- STEP 5: Add Hugging Face Model ---
+print("\n[STEP 5] Add Model from Hugging Face")
+hf_model_id = manager.add_model(
+    source_type="hf",
+    model_name="demo_model_hf",
+    version="v1", 
+    source_path="facebook/wav2vec2-base-960h",
+    category="Image Classification", 
+    tags=["imported", "audio"]
+)
+print(f" -> Registered ID: {hf_model_id}")
+
+# --- STEP 6: Get Detailed Info ---
+print("\n[STEP 6] Get Detailed Model Info")
+# Displays ID, Public status, Total Versions, Size, and ClearML Web Link
+manager.get_model_info("demo_model_local")
+
+# --- STEP 7: List All Models ---
+print("\n[STEP 7] List All Models")
+# Check the 'Public' column (T/F) and 'Latest' version column.
+# By default this also includes OTHER users' public models (each tagged with
+# its real 'owner') - pass include_public=False to see only your own, like before.
+manager.list_models(verbose=False)
+manager.list_models(category="Text Generation")   # filter by category
+manager.list_models(tag="production")              # filter by tag
+manager.list_models(public_only=True)              # only public models (yours + others')
+manager.list_models(include_public=False)          # only your own project, old behavior
+
+# --- STEP 7B: Look Up a Model You Don't Own ---
+print("\n[STEP 7B] Inspect/Download a Public Model From Another User")
+# Suppose list_models() above showed a public model named "shared_model" owned by "colleague_username"
+# other_info = manager.get_model_info("shared_model", owner="colleague_username")
+# manager.get_model(model_name="shared_model", local_dest="./downloads/shared", owner="colleague_username")
+
+# --- STEP 8: Management & Visibility Operations ---
+print("\n[STEP 8] Management Operations")
+
+# A. Change Visibility (Private -> Public)
+# Moves files physically from User Bucket to Public Bucket
+manager.set_model_public("demo_model_local", public=True)
+
+# B. Get Latest Version Name
+# Returns the string tag of the latest version (e.g., "_v2")
+latest_ver = manager.get_latest_version("demo_model_local")
+print(f"Latest version is: {latest_ver}")
+
+# C. Set Manual Latest Version
+# Points the 'latest' tag to a specific version
+manager.set_latest_version(model_name="demo_model_local", version="v1.0")
+
+# --- STEP 9: Download Operations ---
+print("\n[STEP 9] Download Model")
+
+# Download latest version
+# NOTE: The SDK automatically appends the Model ID to the path.
+# Destination will be: ./downloads/latest/<MODEL_ID>/<VERSION_NAME>/
+manager.get_model(
+    model_name="demo_model_local", 
+    local_dest="./downloads/latest"
+)
+
+# --- STEP 10: Deletion ---
+print("\n[STEP 10] Deletion")
+
+# Delete a specific version (Removes files + Metadata entry)
+manager.delete_model(model_name="demo_model_local", version="v1.0")
+
+# Delete the entire model (Removes all versions + ClearML ID)
+manager.delete_model(model_name="demo_model_hf")
+
+```
+
+---
+
+## API Reference
+
+### `MLOpsManager` Methods
+
+| Function | Input Arguments | Description |
+| --- | --- | --- |
+| **`__init__`** | `user_token`, `CLEARML_API_HOST`, `CLEARML_WEB_HOST`, `CEPH_ENDPOINT_URL`, `USER_MANAGEMENT_API`, `verbose`, `ACL_API_HOST`, `ACL_ENABLED` | Initializes connections. Performs fail-fast health checks. `ACL_*` args (or env vars) toggle ACL enforcement (default: disabled). |
+| **`add_model`** | `source_type`, `model_name`, `version`, `source_path`, `code_path`, `category`, `tags`, `external_ceph_*`, `model_folder_name` | Uploads a model. `version`: if `None`, generates `_v{n}` automatically. `category`: must be one of `get_allowed_categories()` (defaults to "Text Generation"/"Image Classification", overridable via `MODEL_ALLOWED_CATEGORIES`). `tags`: list of strings. `model_folder_name`: optional explicit folder name for the uploaded model; if omitted, it's derived automatically as before (backward compatible). |
+| **`get_model`** | `model_name`, `local_dest`, `version`, `owner` | Downloads model files. If `version` is omitted, downloads the version marked as 'latest'. `owner` (optional): resolve `model_name` from another user's project, e.g. to download a public model that isn't yours. |
+| **`get_model_info`** | `identifier` (Name or ID), `owner` | Prints detailed info including **Owner**, **Web Link**, Tags, Category, and Version History. `owner` (optional): resolve a name-based `identifier` from another user's project. |
+| **`get_allowed_categories`** | *(none)* | Returns the list of valid `category` values for `add_model`. |
+| **`list_models`** | `verbose`, `category`, `tag`, `public_only`, `private_only`, `include_public` | Lists your own models **plus** (by default, `include_public=True`) every other user's `public=True` model — never another user's private model. Each entry includes a real `owner` field. `verbose=True` shows full JSON metadata. `category`/`tag` filter by exact match; `public_only`/`private_only` filter by visibility, applied across the combined set. `include_public=False` restores the pre-existing own-project-only behavior. When ACL is enabled, results are additionally filtered to models you can access (plus public ones). See "Cross-User Visibility" above. |
+| **`get_model_id_by_name`** | `name`, `owner` | Resolves a model name to its ClearML ID. `owner` (optional): look it up in another user's project instead of your own. |
+| **`get_model_name_by_id`** | `model_id` | Resolves a ClearML ID back to its model name (works for any model, regardless of owner). |
+| **`get_latest_version`** | `model_identifier` (Name or ID) | Returns the version tag currently marked 'latest'. |
+| **`set_latest_version`** | `model_name`, `version` | Updates the 'latest' pointer to a specific version without modifying files. |
+| **`set_model_public`** | `model_name`, `public` | Moves a model's files between the private and public buckets (transactional, parallel server-side copy). |
+| **`share_model`** | `model_name`, `target_user`, `access_level` | Grants another user access to a model. Requires `ACL_ENABLED=true`. |
+| **`delete_model`** | `model_name`, `version` | If `version` provided: deletes that version (and rolls "latest" back to the next one if it was the latest). If that was the *last* version, the model container is removed too. If `version` is omitted: deletes the entire model in one call — this already is "delete all versions", no need to delete one by one. |
+
+---
+
+## HTTP API
+
+The SDK ships a companion **HTTP API** service (`api/model-registry-api`) that exposes every SDK operation over HTTP for non-Python clients.
+
+Full documentation for that service — endpoints, auth, request/response shapes, ACL toggle, curl examples — lives in its own repo:
+
+> **`api/model-registry-api/README.md`**
+
+The short version: run `uvicorn main:app --host 0.0.0.0 --port 8080`, pass `Authorization: Bearer <token>` on every call, and toggle ACL with `ACL_ENABLED=true/false` in the API service's `.env`.
+
+---
+
+## Constraints & Validations
+
+1. **Model Names:** Cannot start with `tmp_`, `_tmp_`, or `#`.
+2. **Version Names:**
+* Manual versions cannot start with `_` (reserved for auto-generated versions).
+* Auto-generated versions follow the pattern `_v1`, `_v2`, etc.
+
+
+3. **Categories:** Only categories returned by `manager.get_allowed_categories()` are allowed. Defaults to:
+* `Text Generation`
+* `Image Classification`
+
+   Override the full list via `MODEL_ALLOWED_CATEGORIES` in `.env` (comma-separated).
+
+
+
+---
+
+## Testing
+
+- `test_model_registry.py` / `test_ceph_s3_manager.py`: automated unit tests (mock the ClearML/S3 boundary, no live infra needed). Run with:
+  ```bash
+  pip install pytest
+  pytest test_model_registry.py test_ceph_s3_manager.py -v
+  ```
+- `test.py`: a manual/live-integration demo script covering every scenario above against **real** ClearML/Ceph/ACL services — everything is commented out by default; uncomment only what you intend to run.
+
+---
+
+## Admin Instructions: Auto-Publishing to PyPI
+
+This SDK uses a GitHub Actions workflow (`.github/workflows/publish.yaml`) for automatic versioning and PyPI publishing.
+
+### Trigger Conditions
+
+* Push to `main` branch.
+* Commit message contains `pipy commit -push`.
+* GitHub variable `PUBLISH_TO_PYPI` is set to `true`.
+
+### Commit Message Format
+
+| Keyword | Action |
+| --- | --- |
+| `pipy commit -push major` | Increments **Major** version (e.g., 1.0.0 -> 2.0.0) |
+| `pipy commit -push minor` | Increments **Minor** version (e.g., 0.1.0 -> 0.2.0) |
+| `pipy commit -push` | Increments **Patch** version (e.g., 0.1.1 -> 0.1.2) |