ai-model-manager 0.1.0__tar.gz

ai_model_manager-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 wingsfly
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ai_model_manager-0.1.0/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.4
+Name: ai-model-manager
+Version: 0.1.0
+Summary: aim — a single-file, zero-dependency CLI to manage local AI models across HuggingFace, Ollama, ModelScope, PyTorch Hub & openai-whisper: store-as-source-of-truth, in-place load shims, portable backup/restore.
+Author-email: wingsfly <nemosapple@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/wingsfly/ai-model-manager
+Project-URL: Repository, https://github.com/wingsfly/ai-model-manager
+Project-URL: Issues, https://github.com/wingsfly/ai-model-manager/issues
+Keywords: ai,models,model-management,huggingface,ollama,modelscope,pytorch-hub,whisper,llm,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Archiving :: Mirroring
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# aim — AI Model Manager
+
+Unified CLI for managing AI models across multiple inference engines.
+
+## Features
+
+- **Multi-engine support**: Ollama, Hugging Face, MLX (omlx), ComfyUI, Whisper, Coqui TTS, SparkTTS, Piper, Fish Speech
+- **Canonical Store**: Models organized by category in `~/AI/store/{category}/{model_id}/`, with symlinks/hardlinks back to engine directories
+- **Deduplication**: Detect and deduplicate identical model files across engines via hardlinks
+- **Registry**: Track all models in a single JSON registry with metadata, provenance, and provision info
+
+## Docs
+
+- Download PRD: `docs/download-prd.md`
+- Download implementation plan: `docs/download-implementation-plan.md`
+- Download JSON contract: `docs/download-json-contract.md`
+
+## Install
+
+```bash
+# Symlink to PATH
+ln -sf "$(pwd)/aim.py" ~/.local/bin/aim
+chmod +x aim.py
+```
+
+## Usage
+
+```bash
+# Scan all engines and register models
+aim scan
+
+# List registered models
+aim list
+aim list --engine comfyui --sort size
+aim list --json                      # JSON array output
+
+# Show model details
+aim info <model_id>
+aim info <model_id> --json           # full metadata as JSON
+
+# Resolve model to absolute path (for inference)
+aim resolve <model_id>               # prints absolute path
+aim resolve <model_id> --json        # full metadata + path + resolved_file
+aim resolve <model_id> --engine comfyui  # prefer engine provision path
+
+# Organize models into canonical store (preview)
+aim organize
+
+# Organize all non-CAS models
+aim organize --all
+
+# Organize a single model
+aim organize <model_id>
+
+# Download a model
+aim download hf:org/repo
+aim download ollama:model:tag
+aim download hf:org/repo --json
+aim download status <job_id> --json
+aim download cancel <job_id>
+aim download hf:org/repo --category llm/chat
+aim download hf:org/repo --path /custom/path
+aim download url:https://example.com/model.bin --no-progress
+aim download url:https://example.com/model.bin --no-resume
+aim download hf:org/repo --proxy http://127.0.0.1:7890 --retry 3 --retry-backoff 1.5
+
+# Import/register existing local model path
+aim import /path/to/local/model-dir --id my-model --category llm/chat
+
+# Ingest a native-cache model (HF/Ollama/ModelScope) into the store + rebuild its load shim
+aim ingest <model_id>                 # one model: copy real files flat into store/, rebuild the tool's load shim
+aim ingest --all-native               # ingest all native_cas (HF/Ollama/MS) models
+aim ingest <model_id> --dry-run       # preview, change nothing
+aim ingest <model_id> --keep-native   # keep original native bytes (default reclaims them)
+aim convert <model_id>                # deprecated alias -> ingest
+aim verify --fix                      # also rebuilds storage shims from the recorded annotation
+
+# Ingestable sources now include single-file weights:
+#   PyTorch Hub    ($TORCH_HOME/hub/checkpoints/*.pth)              -> aim ingest <torch-id>
+#   openai-whisper (${XDG_CACHE_HOME:-~/.cache}/whisper/*.pt)       -> aim ingest <whisper-id>
+# aim scan discovers them; ingest copies into store and leaves a file symlink so the tool still loads.
+
+# Portable backup / restore (store/ + manifest; shims are regenerated on restore)
+aim backup /Volumes/Backup/aim       # mirror store/ + write aim-backup.json (idempotent; re-runnable)
+aim backup /Volumes/Backup/aim --verify
+aim restore /Volumes/Backup/aim      # recreate store, rebuild tool shims for THIS machine, print env to set
+aim restore /Volumes/Backup/aim --apply-env   # also write env to shell config
+
+# Provision a model for an engine
+aim provision <model_id> --engine comfyui
+
+# Verify link integrity
+aim verify
+aim verify --fix
+
+# Storage overview
+aim status
+aim status --by category
+
+# Detect / manage download-source env vars (HF, Ollama, ModelScope, PyTorch Hub, Civitai, Git)
+aim env show                          # detected vars + resolved cache dirs (read-only)
+aim env show --json                   # machine-readable; secret values are masked
+aim env path huggingface              # resolved cache dir for a source
+aim env apply --shell zsh             # write ~/.aim/env.{sh,fish} + wire rc (one guarded line)
+aim env apply --set HF_ENDPOINT=https://hf-mirror.com --set HF_HUB_ENABLE_HF_TRANSFER=1
+aim env apply --dry-run               # preview, write nothing
+aim env apply --service               # also print daemon-level (launchctl/systemd) env commands
+aim sources list                      # sources, tool install state, env summary
+aim sources install huggingface -y    # install a source's download tool
+
+# Find duplicates
+aim dedup
+aim dedup --apply
+
+# Find unregistered model files
+aim orphans
+```
+
+## JSON Output
+
+`aim list`, `aim info`, and `aim resolve` support `--json` for programmatic integration.
+`aim download --json` emits JSONL progress events and a final JSON summary.
+
+`aim resolve <id> --json` is the recommended single-call API — it returns the full model
+metadata (superset of `aim info --json`) plus two extra fields:
+
+```jsonc
+{
+  "id": "whisper-large-v3-turbo",
+  "name": "whisper-large-v3-turbo",
+  "source": { "type": "local", "repo_id": "openai/whisper-large-v3-turbo" },
+  "format": "pt",
+  "size_bytes": 1617941637,
+  "category": "asr/model",
+  "engines": ["whisper"],
+  // ... all other model fields ...
+  "path": "/Users/you/AI/store/asr/model/whisper-large-v3-turbo",
+  "resolved_file": "/Users/you/AI/store/asr/model/whisper-large-v3-turbo/large-v3-turbo.pt"
+}
+```
+
+| Field | Description |
+|-------|-------------|
+| `path` | Absolute directory (or file) path, resolved via provision or canonical store |
+| `resolved_file` | Primary weight file inside the directory, or `null` for sharded / complex models |
+
+Download placement policy:
+
+1. `--path` has highest priority
+2. Else auto place under `store/<category>/<model_id>/`
+3. Missing category falls back to inferred category, then `uncategorized`
+
+Download control flags:
+
+1. `--no-progress`: only final summary (especially useful with `--json`)
+2. `--resume` / `--no-resume`: toggle resume behavior
+3. `--proxy --timeout --connect-timeout --retry --retry-backoff --max-speed --concurrency`
+
+`resolved_file` detection: scans top-level weight files (`.safetensors`, `.pt`, `.pth`, `.gguf`,
+`.bin`, `.onnx`). Single file → returned directly. Multiple → picks largest matching format.
+Sharded models (`model-00001-of-00006`) → `null` (load from directory).
+
+## Store Layout
+
+```
+~/AI/store/
+├── asr/model/           — Whisper models
+├── image-gen/
+│   ├── checkpoint/      — FLUX, SDXL checkpoints
+│   ├── lora/            — LoRA weights
+│   ├── text-encoder/    — CLIP, T5 encoders
+│   └── vae/             — VAE models
+├── llm/chat/            — LLM chat models (MLX)
+└── tts/
+    ├── model/           — TTS models
+    └── vocoder/         — Vocoder models
+```
+
+## How It Works
+
+1. **`aim scan`** discovers models in engine directories, registers them
+2. **`aim organize`** moves models into `store/{category}/{id}/`, replaces originals with:
+   - **Directories** → symlink back to engine location
+   - **Files in shared dirs** → hardlink back
+3. **`aim verify`** checks all links are intact
+4. **`aim provision`** creates links for a model in any supported engine
+
+## Configuration
+
+Config is stored at `~/.aim/config.json`. Registry at `~/.aim/registry.json`.
+
+## Requirements
+
+- Python 3.10+
+- macOS / Linux
+- Optional: `gh` CLI, `hfd.sh` for downloads
+
+## Testing
+
+```bash
+# Run syntax check
+make lint
+
+# Run all tests (unit + e2e)
+make test
+
+# Run only unit tests
+make test-unit
+
+# Run only end-to-end tests
+make test-e2e
+```
+
+## License
+
+MIT

ai_model_manager-0.1.0/README.md

@@ -0,0 +1,216 @@
+# aim — AI Model Manager
+
+Unified CLI for managing AI models across multiple inference engines.
+
+## Features
+
+- **Multi-engine support**: Ollama, Hugging Face, MLX (omlx), ComfyUI, Whisper, Coqui TTS, SparkTTS, Piper, Fish Speech
+- **Canonical Store**: Models organized by category in `~/AI/store/{category}/{model_id}/`, with symlinks/hardlinks back to engine directories
+- **Deduplication**: Detect and deduplicate identical model files across engines via hardlinks
+- **Registry**: Track all models in a single JSON registry with metadata, provenance, and provision info
+
+## Docs
+
+- Download PRD: `docs/download-prd.md`
+- Download implementation plan: `docs/download-implementation-plan.md`
+- Download JSON contract: `docs/download-json-contract.md`
+
+## Install
+
+```bash
+# Symlink to PATH
+ln -sf "$(pwd)/aim.py" ~/.local/bin/aim
+chmod +x aim.py
+```
+
+## Usage
+
+```bash
+# Scan all engines and register models
+aim scan
+
+# List registered models
+aim list
+aim list --engine comfyui --sort size
+aim list --json                      # JSON array output
+
+# Show model details
+aim info <model_id>
+aim info <model_id> --json           # full metadata as JSON
+
+# Resolve model to absolute path (for inference)
+aim resolve <model_id>               # prints absolute path
+aim resolve <model_id> --json        # full metadata + path + resolved_file
+aim resolve <model_id> --engine comfyui  # prefer engine provision path
+
+# Organize models into canonical store (preview)
+aim organize
+
+# Organize all non-CAS models
+aim organize --all
+
+# Organize a single model
+aim organize <model_id>
+
+# Download a model
+aim download hf:org/repo
+aim download ollama:model:tag
+aim download hf:org/repo --json
+aim download status <job_id> --json
+aim download cancel <job_id>
+aim download hf:org/repo --category llm/chat
+aim download hf:org/repo --path /custom/path
+aim download url:https://example.com/model.bin --no-progress
+aim download url:https://example.com/model.bin --no-resume
+aim download hf:org/repo --proxy http://127.0.0.1:7890 --retry 3 --retry-backoff 1.5
+
+# Import/register existing local model path
+aim import /path/to/local/model-dir --id my-model --category llm/chat
+
+# Ingest a native-cache model (HF/Ollama/ModelScope) into the store + rebuild its load shim
+aim ingest <model_id>                 # one model: copy real files flat into store/, rebuild the tool's load shim
+aim ingest --all-native               # ingest all native_cas (HF/Ollama/MS) models
+aim ingest <model_id> --dry-run       # preview, change nothing
+aim ingest <model_id> --keep-native   # keep original native bytes (default reclaims them)
+aim convert <model_id>                # deprecated alias -> ingest
+aim verify --fix                      # also rebuilds storage shims from the recorded annotation
+
+# Ingestable sources now include single-file weights:
+#   PyTorch Hub    ($TORCH_HOME/hub/checkpoints/*.pth)              -> aim ingest <torch-id>
+#   openai-whisper (${XDG_CACHE_HOME:-~/.cache}/whisper/*.pt)       -> aim ingest <whisper-id>
+# aim scan discovers them; ingest copies into store and leaves a file symlink so the tool still loads.
+
+# Portable backup / restore (store/ + manifest; shims are regenerated on restore)
+aim backup /Volumes/Backup/aim       # mirror store/ + write aim-backup.json (idempotent; re-runnable)
+aim backup /Volumes/Backup/aim --verify
+aim restore /Volumes/Backup/aim      # recreate store, rebuild tool shims for THIS machine, print env to set
+aim restore /Volumes/Backup/aim --apply-env   # also write env to shell config
+
+# Provision a model for an engine
+aim provision <model_id> --engine comfyui
+
+# Verify link integrity
+aim verify
+aim verify --fix
+
+# Storage overview
+aim status
+aim status --by category
+
+# Detect / manage download-source env vars (HF, Ollama, ModelScope, PyTorch Hub, Civitai, Git)
+aim env show                          # detected vars + resolved cache dirs (read-only)
+aim env show --json                   # machine-readable; secret values are masked
+aim env path huggingface              # resolved cache dir for a source
+aim env apply --shell zsh             # write ~/.aim/env.{sh,fish} + wire rc (one guarded line)
+aim env apply --set HF_ENDPOINT=https://hf-mirror.com --set HF_HUB_ENABLE_HF_TRANSFER=1
+aim env apply --dry-run               # preview, write nothing
+aim env apply --service               # also print daemon-level (launchctl/systemd) env commands
+aim sources list                      # sources, tool install state, env summary
+aim sources install huggingface -y    # install a source's download tool
+
+# Find duplicates
+aim dedup
+aim dedup --apply
+
+# Find unregistered model files
+aim orphans
+```
+
+## JSON Output
+
+`aim list`, `aim info`, and `aim resolve` support `--json` for programmatic integration.
+`aim download --json` emits JSONL progress events and a final JSON summary.
+
+`aim resolve <id> --json` is the recommended single-call API — it returns the full model
+metadata (superset of `aim info --json`) plus two extra fields:
+
+```jsonc
+{
+  "id": "whisper-large-v3-turbo",
+  "name": "whisper-large-v3-turbo",
+  "source": { "type": "local", "repo_id": "openai/whisper-large-v3-turbo" },
+  "format": "pt",
+  "size_bytes": 1617941637,
+  "category": "asr/model",
+  "engines": ["whisper"],
+  // ... all other model fields ...
+  "path": "/Users/you/AI/store/asr/model/whisper-large-v3-turbo",
+  "resolved_file": "/Users/you/AI/store/asr/model/whisper-large-v3-turbo/large-v3-turbo.pt"
+}
+```
+
+| Field | Description |
+|-------|-------------|
+| `path` | Absolute directory (or file) path, resolved via provision or canonical store |
+| `resolved_file` | Primary weight file inside the directory, or `null` for sharded / complex models |
+
+Download placement policy:
+
+1. `--path` has highest priority
+2. Else auto place under `store/<category>/<model_id>/`
+3. Missing category falls back to inferred category, then `uncategorized`
+
+Download control flags:
+
+1. `--no-progress`: only final summary (especially useful with `--json`)
+2. `--resume` / `--no-resume`: toggle resume behavior
+3. `--proxy --timeout --connect-timeout --retry --retry-backoff --max-speed --concurrency`
+
+`resolved_file` detection: scans top-level weight files (`.safetensors`, `.pt`, `.pth`, `.gguf`,
+`.bin`, `.onnx`). Single file → returned directly. Multiple → picks largest matching format.
+Sharded models (`model-00001-of-00006`) → `null` (load from directory).
+
+## Store Layout
+
+```
+~/AI/store/
+├── asr/model/           — Whisper models
+├── image-gen/
+│   ├── checkpoint/      — FLUX, SDXL checkpoints
+│   ├── lora/            — LoRA weights
+│   ├── text-encoder/    — CLIP, T5 encoders
+│   └── vae/             — VAE models
+├── llm/chat/            — LLM chat models (MLX)
+└── tts/
+    ├── model/           — TTS models
+    └── vocoder/         — Vocoder models
+```
+
+## How It Works
+
+1. **`aim scan`** discovers models in engine directories, registers them
+2. **`aim organize`** moves models into `store/{category}/{id}/`, replaces originals with:
+   - **Directories** → symlink back to engine location
+   - **Files in shared dirs** → hardlink back
+3. **`aim verify`** checks all links are intact
+4. **`aim provision`** creates links for a model in any supported engine
+
+## Configuration
+
+Config is stored at `~/.aim/config.json`. Registry at `~/.aim/registry.json`.
+
+## Requirements
+
+- Python 3.10+
+- macOS / Linux
+- Optional: `gh` CLI, `hfd.sh` for downloads
+
+## Testing
+
+```bash
+# Run syntax check
+make lint
+
+# Run all tests (unit + e2e)
+make test
+
+# Run only unit tests
+make test-unit
+
+# Run only end-to-end tests
+make test-e2e
+```
+
+## License
+
+MIT