cortex-llm 1.0.0__py3-none-any.whl

cortex_llm-1.0.0.dist-info/METADATA

@@ -0,0 +1,275 @@
+Metadata-Version: 2.4
+Name: cortex-llm
+Version: 1.0.0
+Summary: GPU-Accelerated LLM Terminal for Apple Silicon
+Home-page: https://github.com/faisalmumtaz/Cortex
+Author: Cortex Development Team
+License: MIT
+Project-URL: Homepage, https://github.com/faisalmumtaz/Cortex
+Project-URL: Bug Tracker, https://github.com/faisalmumtaz/Cortex/issues
+Project-URL: Documentation, https://github.com/faisalmumtaz/Cortex/wiki
+Keywords: llm,gpu,metal,mps,apple-silicon,ai,machine-learning,terminal,mlx,pytorch
+Platform: darwin
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: MacOS
+Classifier: Environment :: Console
+Classifier: Environment :: GPU
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.1.0
+Requires-Dist: mlx>=0.10.0
+Requires-Dist: mlx-lm>=0.10.0
+Requires-Dist: transformers>=4.36.0
+Requires-Dist: safetensors>=0.4.0
+Requires-Dist: huggingface-hub>=0.19.0
+Requires-Dist: accelerate>=0.25.0
+Requires-Dist: llama-cpp-python>=0.2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: packaging>=23.0
+Requires-Dist: requests>=2.31.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.8.0; extra == "dev"
+Provides-Extra: optional
+Requires-Dist: sentencepiece>=0.1.99; extra == "optional"
+Requires-Dist: auto-gptq>=0.7.0; extra == "optional"
+Requires-Dist: autoawq>=0.2.0; extra == "optional"
+Requires-Dist: bitsandbytes>=0.41.0; extra == "optional"
+Requires-Dist: optimum>=1.16.0; extra == "optional"
+Requires-Dist: torchvision>=0.16.0; extra == "optional"
+Requires-Dist: torchaudio>=2.1.0; extra == "optional"
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: platform
+Dynamic: requires-python
+
+# Cortex - LLM Terminal Client for Apple Silicon
+
+Cortex is an LLM terminal interface designed for Apple Silicon, using MLX and PyTorch MPS frameworks for GPU-accelerated inference.
+
+## What It Does
+
+- **GPU-accelerated inference** via MLX (primary) and PyTorch MPS backends
+- **Apple Silicon required** - leverages unified memory architecture
+- **Multiple model formats** - MLX, GGUF, SafeTensors, PyTorch, GPTQ, AWQ
+- **Built-in fine-tuning** - LoRA-based model customization via interactive wizard
+- **Chat template auto-detection** - automatic format detection with confidence scoring
+- **Conversation persistence** - SQLite-backed chat history with branching
+
+## Features
+
+- **GPU-Accelerated Inference** - Delegates to MLX and PyTorch MPS for Metal-based execution
+- **Apple Silicon Only** - Requires Metal GPU; exits if GPU acceleration is unavailable
+- **Model Format Support**:
+  - MLX (Apple's format, loaded via `mlx_lm`)
+  - GGUF (via `llama-cpp-python` with Metal backend)
+  - SafeTensors (via HuggingFace `transformers`)
+  - PyTorch models (via HuggingFace `transformers` with MPS device)
+  - GPTQ quantized (via `auto-gptq`)
+  - AWQ quantized (via `awq`)
+- **Quantization** - 4-bit, 5-bit, 8-bit, and mixed-precision quantization via MLX conversion pipeline
+- **Model Conversion** - Convert HuggingFace models to MLX format with configurable quantization recipes
+- **Template Registry** - Automatic detection of chat templates (ChatML, Llama, Alpaca, Gemma, Reasoning) with confidence scoring and real-time token filtering for reasoning models
+- **Rotating KV Cache** - MLX-based KV cache for long context handling (default 4096 tokens)
+- **Fine-Tuning** - LoRA-based model customization with interactive 6-step wizard
+- **Terminal UI** - ANSI terminal interface with streaming output
+
+## Installation
+
+```bash
+# Clone and install
+git clone https://github.com/faisalmumtaz/Cortex.git
+cd Cortex
+./install.sh
+```
+
+The installer:
+- Checks for Apple Silicon (arm64) compatibility
+- Creates a Python virtual environment
+- Installs dependencies via `pip install -e .` (from `pyproject.toml`)
+- Sets up the `cortex` command in your PATH
+
+### Quick Install (pipx)
+
+If you just want the CLI without cloning the repo, use pipx:
+
+```bash
+pipx install cortex-llm
+```
+
+## Quick Start
+
+```bash
+# After installation, just run:
+cortex
+```
+
+### Downloading Models
+
+```bash
+# Inside Cortex, use the download command:
+cortex
+# Then type: /download
+```
+
+The download feature:
+- **HuggingFace integration** - download any model by repository ID
+- **Automatic loading** - option to load model immediately after download
+
+## Documentation
+
+### User Documentation
+- **[Installation Guide](docs/installation.md)** - Complete setup instructions
+- **[CLI Reference](docs/cli.md)** - Commands and user interface
+- **[Configuration](docs/configuration.md)** - System settings and optimization
+- **[Model Management](docs/model-management.md)** - Loading and managing models
+- **[Template Registry](docs/template-registry.md)** - Automatic chat template detection and management
+- **[Fine-Tuning Guide](docs/fine-tuning.md)** - Customize models with LoRA
+- **[Troubleshooting](docs/troubleshooting.md)** - Common issues and solutions
+
+### Technical Documentation
+- **[MLX Acceleration](docs/mlx-acceleration.md)** - MLX framework integration and optimization
+- **[GPU Validation](docs/gpu-validation.md)** - Hardware requirements and detection
+- **[Inference Engine](docs/inference-engine.md)** - Text generation architecture
+- **[Conversation Management](docs/conversation-management.md)** - Chat history and persistence
+- **[Development Guide](docs/development.md)** - Contributing and architecture
+
+## System Requirements
+
+- Apple Silicon Mac (M1/M2/M3/M4 - all variants supported)
+- macOS 13.3+ (required by MLX framework)
+- Python 3.11+
+- 16GB+ unified memory (24GB+ recommended for larger models)
+- Xcode Command Line Tools
+
+## Performance
+
+Performance depends on your Apple Silicon chip, model size, and quantization level. The inference engine measures tokens/second, first-token latency, and memory usage at runtime.
+
+To check that GPU acceleration is working:
+
+```bash
+source venv/bin/activate
+python tests/test_apple_silicon.py
+```
+
+You should see:
+- All validation checks passing
+- Measured GFLOPS from matrix operations
+- Confirmation of Metal and MLX availability
+
+## GPU Acceleration Architecture
+
+Cortex uses a multi-layer approach, delegating all GPU computation to established frameworks:
+
+1. **MLX Framework (Primary Backend)**
+   - Apple's ML framework with native Metal support
+   - Quantization support (4-bit, 5-bit, 8-bit, mixed-precision)
+   - Rotating KV cache for long contexts
+   - JIT compilation via `mx.compile`
+   - Operation fusion for reduced kernel launches
+
+2. **PyTorch MPS Backend**
+   - Metal Performance Shaders for PyTorch models
+   - FP16 optimization and channels-last tensor format
+
+3. **llama.cpp (GGUF Backend)**
+   - Metal-accelerated inference for GGUF models
+
+4. **Memory Management**
+   - Pre-allocated memory pools with best-fit/first-fit allocation strategies
+   - Automatic pool sizing (60% of available memory, capped at 75% of total)
+   - Defragmentation support
+
+### Understanding "Skipping Kernel" Messages
+
+When loading GGUF models, you may see messages like:
+```
+ggml_metal_init: skipping kernel_xxx_bf16 (not supported)
+```
+
+**These are NORMAL!** They indicate:
+- BF16 kernels being skipped (your GPU uses FP16 instead)
+- GPU acceleration is still fully active
+- The system automatically uses optimal alternatives
+
+## Troubleshooting
+
+If you suspect GPU isn't being used:
+
+1. **Run validation**: `python tests/test_apple_silicon.py`
+2. **Check output**: Should see passing checks and measured GFLOPS
+3. **Monitor tokens/sec**: Displayed during inference
+4. **Verify Metal**: Ensure Xcode Command Line Tools installed
+
+Common issues:
+- **Low performance**: Run `python tests/test_apple_silicon.py` to diagnose
+- **Memory errors**: Reduce `gpu_memory_fraction` in config.yaml
+
+## MLX Model Conversion
+
+Cortex includes an MLX model converter:
+
+```python
+from cortex.metal.mlx_converter import MLXConverter, ConversionConfig, QuantizationRecipe
+
+converter = MLXConverter()
+config = ConversionConfig(
+    quantization=QuantizationRecipe.SPEED_4BIT,  # 4-bit quantization
+    compile_model=True  # JIT compilation
+)
+
+success, message, output_path = converter.convert_model(
+    "microsoft/DialoGPT-medium",
+    config=config
+)
+```
+
+### Quantization Options
+
+- **4-bit**: Maximum speed, 75% size reduction
+- **5-bit**: Balanced speed and quality
+- **8-bit**: Higher quality, 50% size reduction
+- **Mixed Precision**: Custom per-layer quantization
+
+## MLX as Primary Backend
+
+Cortex uses MLX (Apple's machine learning framework) as the primary acceleration backend:
+- **Metal Support**: GPU execution via MLX's built-in Metal operations
+- **Quantization**: Support for 4-bit, 5-bit, 8-bit, and mixed-precision quantization
+- **Model Conversion**: Convert HuggingFace models to MLX format
+
+## Built With
+
+- [MLX](https://github.com/ml-explore/mlx) - Apple's machine learning framework
+- [mlx-lm](https://github.com/ml-explore/mlx-examples) - LLM utilities and LoRA fine-tuning for MLX
+- [PyTorch](https://pytorch.org/) - With Metal Performance Shaders backend
+- [llama.cpp](https://github.com/ggerganov/llama.cpp) - Metal-accelerated GGUF support
+- [Rich](https://github.com/Textualize/rich) - Terminal formatting
+- [HuggingFace](https://huggingface.co/) - Model hub and transformers
+
+## Contributing
+
+We welcome contributions! Please see the [Development Guide](docs/development.md) for contributing guidelines and setup instructions.
+
+## License
+
+MIT License - See [LICENSE](LICENSE) for details.
+
+---
+
+**Note**: Cortex requires Apple Silicon. Intel Macs are not supported.

cortex_llm-1.0.0.dist-info/RECORD

@@ -0,0 +1,48 @@
+cortex/__init__.py,sha256=cAZXKEqjOtK2P-kij00tDHwtM3asYNCW1386R4WHyTQ,2201
+cortex/__main__.py,sha256=9k1Cvz8ncltQzbrJlxz8JDv-jHAOLCZoCaKSlJ4YFHE,2664
+cortex/config.py,sha256=txmpJXy3kUEKULZyu1OWb_jkNQRHZClm5ovZfCTX_Zc,13444
+cortex/conversation_manager.py,sha256=aSTdGjVttsMKIiRPzztP0tOXlqZBkWtgZDNCZGyaR-c,17177
+cortex/gpu_validator.py,sha256=un6vMQ78MWMnKWIz8n-92v9Fb4g_YXqU_E1pUPinncY,16582
+cortex/inference_engine.py,sha256=WUi5YS_sTQFayc-UJbyixHEzZlMN0-ATNdoPNEhHbbo,28543
+cortex/model_downloader.py,sha256=VuPhvxq_66qKjsPjEWcLW-VmUHzOHik6LBMiGDk-cX8,4977
+cortex/model_manager.py,sha256=Blk-JA_kajJcDp-h2A4tplECijHPw8LZ8c_fbq0FGFg,100670
+cortex/fine_tuning/__init__.py,sha256=IXKQqNqN1C3mha3na35i7KI-hMnsqqrmUgV4NrPKHy0,269
+cortex/fine_tuning/dataset.py,sha256=hIz_dfFSaJoiFzWZ6vwlwqjpTfdsnFNIEmwhhTD2d9k,15414
+cortex/fine_tuning/mlx_lora_trainer.py,sha256=idNzKtVG8pObwsnSrP0N1rU1EanhrIRvHiNL1asdzr8,22438
+cortex/fine_tuning/trainer.py,sha256=Y9l7RQMGHEqVpVS0rcuCe0ZkNzE0Yvmgb00oCWH_b6Q,38775
+cortex/fine_tuning/wizard.py,sha256=eIRUM3zTqKKATJEbQrBsaOfFfRWfY9BV5FkSAzT82QM,31815
+cortex/metal/__init__.py,sha256=Ycs81qVOsaYV4UJocCFGW3rPPBySMPy7eOHKzfc4Q7o,8780
+cortex/metal/gpu_validator.py,sha256=1YHKJXqicXvTwKIdSj34n1DgKoluy9yho6S1jWt1UAs,5818
+cortex/metal/memory_pool.py,sha256=g5PFQAiouQe4TyX-SVi-Di1MLysb3YBF77uR4nAEomo,34698
+cortex/metal/mlx_accelerator.py,sha256=HknYFHwFOPzAtdrV5OqxBvENX3qwNoiKC6H8lXGmJPg,25637
+cortex/metal/mlx_converter.py,sha256=lMmIh6PqLwraWInZDLJjktBtumLHD9TYRnhiKCpilrc,26722
+cortex/metal/mps_optimizer.py,sha256=4r6dj-_KAr3vedCwwu7lR-nIaF4g4D4kkOoF2KiQ0FQ,15307
+cortex/metal/optimizer.py,sha256=9ixKj8ca1iovF-mFHYGa9_DUHcqgGyzLoP_lIRAzfMM,21996
+cortex/metal/performance_profiler.py,sha256=GMxxqwqE2kVJ4WePwVdUp2ADqhrV6wCCNrFnaMfBDpI,12274
+cortex/quantization/__init__.py,sha256=ElLP3ZO_XItddTl-PeoJ5GPb16RYIAk8m5sqwfAVE9s,184
+cortex/quantization/dynamic_quantizer.py,sha256=sAoHoQ6wfs6FvejG-iehB2Qij-0WC9qSTlBfj3D1pTI,31724
+cortex/template_registry/__init__.py,sha256=O5BWmHRmfMSK-Ukpu8UqFO_kaN0kum-d-Wsz0Ds-sC0,491
+cortex/template_registry/auto_detector.py,sha256=lqI19Ef_w6ClZvD5dzDw1i5gnf2AUN_L4WjCMvW99Yg,5432
+cortex/template_registry/config_manager.py,sha256=vh7cXAUTJ4dLY74u5EHTpTa46jXxj34BlMyWsC_ZIaM,8658
+cortex/template_registry/interactive.py,sha256=ldKc8Ilvr5RBrtBRJjue-hFgLSB5uh7dQmVW7rI1Fms,10986
+cortex/template_registry/registry.py,sha256=pVgyGhEMemCN8mdZXxg8rnWKr1FMEIK1JiEEu8rta4M,12949
+cortex/template_registry/template_profiles/__init__.py,sha256=bM_VkDzHJYIty-5Gr92mbUPEnwmMtMrIZZvkp2Nqxos,232
+cortex/template_registry/template_profiles/base.py,sha256=2dHsdv0MRrNEEd5G8Y4WWey_g9ylMZ8orUh5CMrSPYw,4854
+cortex/template_registry/template_profiles/complex/__init__.py,sha256=g1fcy4magj26j8-OuXJHH5gEyoIVpbWkJLEN8k_-RDI,182
+cortex/template_registry/template_profiles/complex/reasoning.py,sha256=RZ8-JSEz48bF99WyVwadAaPYOQk69iTGfuMXyAUeiGQ,11460
+cortex/template_registry/template_profiles/standard/__init__.py,sha256=9CfYhufq3fGwmX1J7IRa4RJT15Qnqwf1aj2tKbw--ls,550
+cortex/template_registry/template_profiles/standard/alpaca.py,sha256=vk3o27fSqXOJFhVhipxolTKd9hcafqE8SUo2UsIuSS4,2732
+cortex/template_registry/template_profiles/standard/chatml.py,sha256=_oQdqV90bGNPZy5d27qzyRCzM5ykSTSow0ygLyvHURA,3190
+cortex/template_registry/template_profiles/standard/gemma.py,sha256=D4wZN3_6QzUj-icfkX6o35USLbhXe08VaYGclHXvHSw,4074
+cortex/template_registry/template_profiles/standard/llama.py,sha256=jz4MyvmISSPtIAcffPE7LrTosHvlC0NoJhzTw1DCvpY,3209
+cortex/template_registry/template_profiles/standard/simple.py,sha256=dGOOcL6HRoJFxkixLrYC4w7c63h-QmOOWC2TsOihYog,2422
+cortex/ui/__init__.py,sha256=t3GrHJMHTVgBEKh2_qt4B9mS594V5jriTDqc3eZKMGc,3409
+cortex/ui/cli.py,sha256=oxHRIhLu_izy4ZfBWqUngnV060UD4P0J8bcvquKz2lU,74632
+cortex/ui/markdown_render.py,sha256=bXt60vkNYT_jbpKeIg_1OlcrxssmdbMO7RB2E1sWw3E,5759
+cortex/ui/terminal_app.py,sha256=SF3KqcGFyZ4hpTmgX21idPzOTJLdKGkt4QdA-wwUBNE,18317
+cortex_llm-1.0.0.dist-info/licenses/LICENSE,sha256=_frJ3VsZWQGhMznZw2Tgjk7xwfAfDZRcBl43uZh8_4E,1070
+cortex_llm-1.0.0.dist-info/METADATA,sha256=STGY7B8BthPSY29Slojmq1_HiXsQ5NxUtgwQoKPToMg,10087
+cortex_llm-1.0.0.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+cortex_llm-1.0.0.dist-info/entry_points.txt,sha256=g83Nuz3iFrNdMLHxGLR2LnscdM7rdQRchuL3WGobQC8,48
+cortex_llm-1.0.0.dist-info/top_level.txt,sha256=79LAeTJJ_pMIBy3mkF7uNaN0mdBRt5tGrnne5N_iAio,7
+cortex_llm-1.0.0.dist-info/RECORD,,

cortex_llm-1.0.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.10.2)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

cortex_llm-1.0.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+cortex = cortex.__main__:main

cortex_llm-1.0.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Faisal Mumtaz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cortex_llm-1.0.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+cortex