cortex-llm 1.0.5__tar.gz → 1.0.7__tar.gz

cortex_llm-1.0.7/PKG-INFO

@@ -0,0 +1,169 @@
+Metadata-Version: 2.4
+Name: cortex-llm
+Version: 1.0.7
+Summary: GPU-Accelerated LLM Terminal for Apple Silicon
+Home-page: https://github.com/faisalmumtaz/Cortex
+Author: Cortex Development Team
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/faisalmumtaz/Cortex
+Project-URL: Bug Tracker, https://github.com/faisalmumtaz/Cortex/issues
+Project-URL: Documentation, https://github.com/faisalmumtaz/Cortex/wiki
+Keywords: llm,gpu,metal,mps,apple-silicon,ai,machine-learning,terminal,mlx,pytorch
+Platform: darwin
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: MacOS
+Classifier: Environment :: Console
+Classifier: Environment :: GPU
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.1.0
+Requires-Dist: mlx>=0.30.4
+Requires-Dist: mlx-lm>=0.30.5
+Requires-Dist: transformers>=4.36.0
+Requires-Dist: safetensors>=0.4.0
+Requires-Dist: huggingface-hub>=0.19.0
+Requires-Dist: accelerate>=0.25.0
+Requires-Dist: llama-cpp-python>=0.2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: packaging>=23.0
+Requires-Dist: requests>=2.31.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.8.0; extra == "dev"
+Provides-Extra: optional
+Requires-Dist: sentencepiece>=0.1.99; extra == "optional"
+Requires-Dist: auto-gptq>=0.7.0; extra == "optional"
+Requires-Dist: autoawq>=0.2.0; extra == "optional"
+Requires-Dist: bitsandbytes>=0.41.0; extra == "optional"
+Requires-Dist: optimum>=1.16.0; extra == "optional"
+Requires-Dist: torchvision>=0.16.0; extra == "optional"
+Requires-Dist: torchaudio>=2.1.0; extra == "optional"
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: platform
+Dynamic: requires-python
+
+# Cortex
+
+GPU-accelerated local LLMs on Apple Silicon, built for the terminal.
+
+Cortex is a fast, native CLI for running and fine-tuning LLMs on Apple Silicon using MLX and Metal. It automatically detects chat templates, supports multiple model formats, and keeps your workflow inside the terminal.
+
+## Highlights
+
+- Apple Silicon GPU acceleration via MLX (primary) and PyTorch MPS
+- Multi-format model support: MLX, GGUF, SafeTensors, PyTorch, GPTQ, AWQ
+- Built-in LoRA fine-tuning wizard
+- Chat template auto-detection (ChatML, Llama, Alpaca, Gemma, Reasoning)
+- Conversation history with autosave and export
+
+## Quick Start
+
+```bash
+pipx install cortex-llm
+cortex
+```
+
+Inside Cortex:
+
+- `/download` to fetch a model from HuggingFace
+- `/model` to load or manage models
+- `/status` to confirm GPU acceleration and current settings
+
+## Installation
+
+### Option A: pipx (recommended)
+
+```bash
+pipx install cortex-llm
+```
+
+### Option B: from source
+
+```bash
+git clone https://github.com/faisalmumtaz/Cortex.git
+cd Cortex
+./install.sh
+```
+
+The installer checks Apple Silicon compatibility, creates a venv, installs dependencies from `pyproject.toml`, and sets up the `cortex` command.
+
+## Requirements
+
+- Apple Silicon Mac (M1/M2/M3/M4)
+- macOS 13.3+
+- Python 3.11+
+- 16GB+ unified memory (24GB+ recommended for larger models)
+- Xcode Command Line Tools
+
+## Model Support
+
+Cortex supports:
+
+- **MLX** (recommended)
+- **GGUF** (llama.cpp + Metal)
+- **SafeTensors**
+- **PyTorch** (Transformers + MPS)
+- **GPTQ** / **AWQ** quantized models
+
+## Advanced Features
+
+- **Dynamic quantization fallback** for PyTorch/SafeTensors models that do not fit GPU memory (INT8 preferred, INT4 fallback)
+  - `docs/dynamic-quantization.md`
+- **MLX conversion with quantization recipes** (4/5/8-bit, mixed precision) for speed vs quality control
+  - `docs/mlx-acceleration.md`
+- **LoRA fine-tuning wizard** for local adapters (`/finetune`)
+  - `docs/fine-tuning.md`
+- **Template registry and auto-detection** for chat formatting (ChatML, Llama, Alpaca, Gemma, Reasoning)
+  - `docs/template-registry.md`
+- **Inference engine details** and backend behavior
+  - `docs/inference-engine.md`
+
+## Configuration
+
+Cortex reads `config.yaml` from the current working directory. For tuning GPU memory limits, quantization defaults, and inference parameters, see:
+
+- `docs/configuration.md`
+
+## Documentation
+
+Start here:
+
+- `docs/installation.md`
+- `docs/cli.md`
+- `docs/model-management.md`
+- `docs/troubleshooting.md`
+
+Advanced topics:
+
+- `docs/mlx-acceleration.md`
+- `docs/inference-engine.md`
+- `docs/dynamic-quantization.md`
+- `docs/template-registry.md`
+- `docs/fine-tuning.md`
+- `docs/development.md`
+
+## Contributing
+
+Contributions are welcome. See `docs/development.md` for setup and workflow.
+
+## License
+
+MIT License. See `LICENSE`.
+
+---
+
+Note: Cortex requires Apple Silicon. Intel Macs are not supported.

cortex_llm-1.0.7/README.md

@@ -0,0 +1,111 @@
+# Cortex
+
+GPU-accelerated local LLMs on Apple Silicon, built for the terminal.
+
+Cortex is a fast, native CLI for running and fine-tuning LLMs on Apple Silicon using MLX and Metal. It automatically detects chat templates, supports multiple model formats, and keeps your workflow inside the terminal.
+
+## Highlights
+
+- Apple Silicon GPU acceleration via MLX (primary) and PyTorch MPS
+- Multi-format model support: MLX, GGUF, SafeTensors, PyTorch, GPTQ, AWQ
+- Built-in LoRA fine-tuning wizard
+- Chat template auto-detection (ChatML, Llama, Alpaca, Gemma, Reasoning)
+- Conversation history with autosave and export
+
+## Quick Start
+
+```bash
+pipx install cortex-llm
+cortex
+```
+
+Inside Cortex:
+
+- `/download` to fetch a model from HuggingFace
+- `/model` to load or manage models
+- `/status` to confirm GPU acceleration and current settings
+
+## Installation
+
+### Option A: pipx (recommended)
+
+```bash
+pipx install cortex-llm
+```
+
+### Option B: from source
+
+```bash
+git clone https://github.com/faisalmumtaz/Cortex.git
+cd Cortex
+./install.sh
+```
+
+The installer checks Apple Silicon compatibility, creates a venv, installs dependencies from `pyproject.toml`, and sets up the `cortex` command.
+
+## Requirements
+
+- Apple Silicon Mac (M1/M2/M3/M4)
+- macOS 13.3+
+- Python 3.11+
+- 16GB+ unified memory (24GB+ recommended for larger models)
+- Xcode Command Line Tools
+
+## Model Support
+
+Cortex supports:
+
+- **MLX** (recommended)
+- **GGUF** (llama.cpp + Metal)
+- **SafeTensors**
+- **PyTorch** (Transformers + MPS)
+- **GPTQ** / **AWQ** quantized models
+
+## Advanced Features
+
+- **Dynamic quantization fallback** for PyTorch/SafeTensors models that do not fit GPU memory (INT8 preferred, INT4 fallback)
+  - `docs/dynamic-quantization.md`
+- **MLX conversion with quantization recipes** (4/5/8-bit, mixed precision) for speed vs quality control
+  - `docs/mlx-acceleration.md`
+- **LoRA fine-tuning wizard** for local adapters (`/finetune`)
+  - `docs/fine-tuning.md`
+- **Template registry and auto-detection** for chat formatting (ChatML, Llama, Alpaca, Gemma, Reasoning)
+  - `docs/template-registry.md`
+- **Inference engine details** and backend behavior
+  - `docs/inference-engine.md`
+
+## Configuration
+
+Cortex reads `config.yaml` from the current working directory. For tuning GPU memory limits, quantization defaults, and inference parameters, see:
+
+- `docs/configuration.md`
+
+## Documentation
+
+Start here:
+
+- `docs/installation.md`
+- `docs/cli.md`
+- `docs/model-management.md`
+- `docs/troubleshooting.md`
+
+Advanced topics:
+
+- `docs/mlx-acceleration.md`
+- `docs/inference-engine.md`
+- `docs/dynamic-quantization.md`
+- `docs/template-registry.md`
+- `docs/fine-tuning.md`
+- `docs/development.md`
+
+## Contributing
+
+Contributions are welcome. See `docs/development.md` for setup and workflow.
+
+## License
+
+MIT License. See `LICENSE`.
+
+---
+
+Note: Cortex requires Apple Silicon. Intel Macs are not supported.

{cortex_llm-1.0.5 → cortex_llm-1.0.7}/cortex/__init__.py

@@ -5,7 +5,7 @@ A high-performance terminal interface for running Hugging Face LLMs locally
 with exclusive GPU acceleration via Metal Performance Shaders (MPS) and MLX.
 """
 
-__version__ = "1.0.5"
+__version__ = "1.0.7"
 __author__ = "Cortex Development Team"
 __license__ = "MIT"
 

{cortex_llm-1.0.5 → cortex_llm-1.0.7}/cortex/config.py

@@ -74,7 +74,7 @@ class InferenceConfig(BaseModel):
     top_p: float = Field(default=0.95, ge=0.0, le=1.0)
     top_k: int = Field(default=40, ge=0)
     repetition_penalty: float = Field(default=1.1, ge=0.0, le=2.0)
-    max_tokens: int = Field(default=2048, ge=1)
+    max_tokens: int = Field(default=4096, ge=1)
     stream_output: bool = True
     seed: int = Field(default=-1)
 

{cortex_llm-1.0.5 → cortex_llm-1.0.7}/cortex/inference_engine.py

@@ -138,7 +138,7 @@ class InferenceEngine:
                 use_fp16=True,
                 use_channels_last=True,
                 optimize_memory=True,
-                max_batch_size=self.config.performance.batch_size
+                max_batch_size=self.config.performance.max_batch_size
             )
             self.mps_optimizer = MPSOptimizer(mps_config)
         
@@ -153,7 +153,7 @@ class InferenceEngine:
                 fuse_operations=True,
                 lazy_evaluation=True,
                 rotating_kv_cache=True,
-                kv_cache_size=self.config.model.context_length if hasattr(self.config.model, 'context_length') else 4096,
+                kv_cache_size=self.config.performance.context_length,
                 quantization_bits=4
             )
             self.mlx_accelerator = MLXAccelerator(mlx_config)
@@ -204,6 +204,9 @@ class InferenceEngine:
                     yield from self._generate_pytorch(model, tokenizer, request)
                 elif model_info.format == ModelFormat.SAFETENSORS:
                     yield from self._generate_safetensors(model, tokenizer, request)
+                elif model_info.format == ModelFormat.QUANTIZED:
+                    # Quantized models are loaded as PyTorch-compatible modules
+                    yield from self._generate_pytorch(model, tokenizer, request)
                 elif model_info.format == ModelFormat.GGUF:
                     yield from self._generate_gguf(model, tokenizer, request)
                 else:
@@ -401,7 +404,18 @@ class InferenceEngine:
         last_metrics_update = time.time()
         
         try:
-            device = torch.device("mps")
+            # Use the model's device when available (quantized models may be CPU-only on macOS)
+            device = None
+            try:
+                first_param = next(model.parameters())
+                device = first_param.device
+            except Exception:
+                device = None
+
+            if device is None or str(device) == "meta":
+                device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
+            elif device.type == "mps" and not torch.backends.mps.is_available():
+                device = torch.device("cpu")
             
             inputs = tokenizer(request.prompt, return_tensors="pt").to(device)
             

{cortex_llm-1.0.5 → cortex_llm-1.0.7}/cortex/model_manager.py

@@ -133,7 +133,8 @@ class ModelManager:
         self.quantizer = DynamicQuantizer(QuantizationConfig(
             mode=QuantizationMode.DYNAMIC,
             per_channel=True,
-            cache_quantized=True
+            cache_quantized=True,
+            cache_dir=self.config.model.quantization_cache
         ))
         
         # Initialize MLX converter for native conversion
@@ -201,6 +202,39 @@ class ModelManager:
         level = getattr(self.config.gpu, "gpu_optimization_level", "maximum")
         level = str(level).lower().strip()
         return level in {"maximum", "max", "speed", "fast", "performance"}
+
+    def _get_default_quant_recipe(self) -> Optional[QuantizationRecipe]:
+        """Map configured default_quantization to an MLX quantization recipe."""
+        raw = getattr(self.config.model, "default_quantization", "") or ""
+        value = str(raw).strip().lower()
+        if not value or value == "auto":
+            return None
+
+        mapping = {
+            "q4_k_m": QuantizationRecipe.SPEED_4BIT,
+            "q5_k_m": QuantizationRecipe.BALANCED_5BIT,
+            "q6_k": QuantizationRecipe.QUALITY_8BIT,  # closest available MLX recipe
+            "q8_0": QuantizationRecipe.QUALITY_8BIT,
+            "4bit": QuantizationRecipe.SPEED_4BIT,
+            "5bit": QuantizationRecipe.BALANCED_5BIT,
+            "8bit": QuantizationRecipe.QUALITY_8BIT,
+            "mixed": QuantizationRecipe.MIXED_PRECISION,
+            "none": QuantizationRecipe.NONE,
+        }
+
+        recipe = mapping.get(value)
+        if recipe is None:
+            logger.warning("Unknown default_quantization value: %s", raw)
+            return None
+
+        supported = getattr(self.config.model, "supported_quantizations", None)
+        if supported:
+            supported_norm = {str(s).strip().lower() for s in supported}
+            if value.startswith("q") and value not in supported_norm:
+                logger.warning("default_quantization '%s' not in supported_quantizations", raw)
+                return None
+
+        return recipe
     
     def load_model(
         self,
@@ -374,6 +408,10 @@ class ModelManager:
                         except Exception as e:
                             logger.warning(f"Could not estimate model parameters: {e}, defaulting to 4-bit")
                             quant_recipe = QuantizationRecipe.SPEED_4BIT  # Fallback
+
+                    default_recipe = self._get_default_quant_recipe()
+                    if default_recipe is not None:
+                        quant_recipe = default_recipe
                     
                     if quantization:
                         quant_map = {
@@ -452,6 +490,10 @@ class ModelManager:
                 else:
                     quant_recipe = QuantizationRecipe.SPEED_4BIT  # Default for larger models
                 
+                default_recipe = self._get_default_quant_recipe()
+                if default_recipe is not None:
+                    quant_recipe = default_recipe
+                
                 if quantization:
                     quant_map = {
                         "4bit": QuantizationRecipe.SPEED_4BIT,
@@ -563,6 +605,8 @@ class ModelManager:
         )
         
         if not can_load and can_apply_quantization:
+            if not getattr(self.config.model, "auto_quantize", True):
+                return False, f"GPU incompatible: {message} (auto_quantize disabled)"
             # Check if quantization would help
             gpu_status = self.gpu_validator.get_gpu_memory_status()
             available_gb = gpu_status['available_gb']

{cortex_llm-1.0.5 → cortex_llm-1.0.7}/cortex/quantization/dynamic_quantizer.py

@@ -3,7 +3,7 @@
 import torch
 import torch.nn as nn
 from typing import Dict, Any, Optional, Tuple, Union
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from enum import Enum
 import gc
 from pathlib import Path
@@ -40,6 +40,7 @@ class QuantizationConfig:
     cache_quantized: bool = True  # Cache quantized models to disk
     compress_cache: bool = False  # Compress cached models (slower but smaller)
     validate_quantization: bool = True  # Validate quantized models work correctly
+    cache_dir: Path = field(default_factory=lambda: Path.home() / ".cortex" / "quantized_models")
     
     def to_dict(self) -> Dict[str, Any]:
         """Convert to dictionary for serialization."""
@@ -118,6 +119,8 @@ class DynamicQuantizer:
     def __init__(self, config: Optional[QuantizationConfig] = None):
         """Initialize quantizer with configuration."""
         self.config = config or QuantizationConfig()
+        self.config.cache_dir = Path(self.config.cache_dir).expanduser()
+        self.config.cache_dir.mkdir(parents=True, exist_ok=True)
         self.device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
         self._quantization_cache: Dict[str, Dict[str, Any]] = {}
     
@@ -681,10 +684,10 @@ class DynamicQuantizer:
         
         # Generate cache key including model metadata
         cache_key = hashlib.md5(
-            f"{model_path}_{model_mtime}_{model_size}_{json.dumps(quantization_info)}".encode()
+            f"{model_path}_{model_mtime}_{model_size}_{json.dumps(self.config.to_dict())}".encode()
         ).hexdigest()
         
-        cache_dir = Path.home() / ".cortex" / "quantized_cache"
+        cache_dir = self.config.cache_dir
         cache_dir.mkdir(parents=True, exist_ok=True)
         
         cache_path = cache_dir / f"{cache_key}.pt"
@@ -723,7 +726,7 @@ class DynamicQuantizer:
             f"{model_path}_{model_mtime}_{model_size}_{json.dumps(config.to_dict())}".encode()
         ).hexdigest()
         
-        cache_path = Path.home() / ".cortex" / "quantized_cache" / f"{cache_key}.pt"
+        cache_path = Path(self.config.cache_dir) / f"{cache_key}.pt"
         
         if cache_path.exists():
             try:
@@ -733,4 +736,4 @@ class DynamicQuantizer:
                 # Cache corrupted, will re-quantize
                 cache_path.unlink()
         
-        return None
+        return None

{cortex_llm-1.0.5 → cortex_llm-1.0.7}/cortex/ui/cli.py

@@ -30,7 +30,7 @@ from cortex.conversation_manager import ConversationManager, MessageRole
 from cortex.model_downloader import ModelDownloader
 from cortex.template_registry import TemplateRegistry
 from cortex.fine_tuning import FineTuneWizard
-from cortex.ui.markdown_render import ThinkMarkdown, PrefixedRenderable
+from cortex.ui.markdown_render import ThinkMarkdown, PrefixedRenderable, render_plain_with_think
 
 
 class CortexCLI:
@@ -1135,15 +1135,15 @@ class CortexCLI:
                 logger.debug(f"Could not get stop sequences: {e}")
         
         # Create generation request with formatted prompt
-        # Use lower temperature for more focused responses
         request = GenerationRequest(
             prompt=formatted_prompt,
             max_tokens=self.config.inference.max_tokens,
-            temperature=0.3,  # Lower temperature for less randomness
-            top_p=0.9,  # Slightly lower top_p
+            temperature=self.config.inference.temperature,
+            top_p=self.config.inference.top_p,
             top_k=self.config.inference.top_k,
             repetition_penalty=self.config.inference.repetition_penalty,
-            stream=True,
+            stream=self.config.inference.stream_output,
+            seed=self.config.inference.seed if self.config.inference.seed >= 0 else None,
             stop_sequences=stop_sequences
         )
         
@@ -1167,50 +1167,65 @@ class CortexCLI:
             prefix_style = Style(color="cyan")
 
             def build_renderable(text: str):
-                markdown = ThinkMarkdown(text, code_theme="monokai", use_line_numbers=False)
-                return PrefixedRenderable(markdown, prefix="⏺ ", prefix_style=prefix_style, indent="  ")
+                if getattr(self.config.ui, "markdown_rendering", True):
+                    markdown = ThinkMarkdown(
+                        text,
+                        code_theme="monokai",
+                        use_line_numbers=False,
+                        syntax_highlighting=getattr(self.config.ui, "syntax_highlighting", True),
+                    )
+                    renderable = markdown
+                else:
+                    renderable = render_plain_with_think(text)
 
-            with Live(
-                build_renderable(""),
-                console=self.console,
-                refresh_per_second=20,
-                transient=False,
-            ) as live:
-                for token in self.inference_engine.generate(request):
-                    if first_token_time is None:
-                        first_token_time = time.time()
+                return PrefixedRenderable(renderable, prefix="⏺ ", prefix_style=prefix_style, indent="  ")
 
-                    generated_text += token
-                    token_count += 1
+            original_console_width = self.console._width
+            target_width = max(40, int(self.get_terminal_width() * 0.75))
+            self.console.width = target_width
+            try:
+                with Live(
+                    build_renderable(""),
+                    console=self.console,
+                    auto_refresh=False,
+                    refresh_per_second=20,
+                    transient=False,
+                    vertical_overflow="visible",
+                ) as live:
+                    for token in self.inference_engine.generate(request):
+                        if first_token_time is None:
+                            first_token_time = time.time()
 
-                    display_token = token
-                    if uses_reasoning_template and template_profile and template_profile.supports_streaming():
-                        display_token, should_display = template_profile.process_streaming_response(
-                            token, accumulated_response
-                        )
-                        accumulated_response += token
-                        if not should_display:
-                            display_token = ""
+                        generated_text += token
+                        token_count += 1
+
+                        display_token = token
+                        if uses_reasoning_template and template_profile and template_profile.supports_streaming():
+                            display_token, should_display = template_profile.process_streaming_response(
+                                token, accumulated_response
+                            )
+                            accumulated_response += token
+                            if not should_display:
+                                display_token = ""
 
-                    if display_token:
-                        display_text += display_token
+                        if display_token:
+                            display_text += display_token
 
-                    now = time.time()
-                    if display_token and ("\n" in display_token or now - last_render_time >= render_interval):
-                        live.update(build_renderable(display_text))
-                        last_render_time = now
+                        now = time.time()
+                        if display_token and ("\n" in display_token or now - last_render_time >= render_interval):
+                            live.update(build_renderable(display_text), refresh=True)
+                            last_render_time = now
 
-                if uses_reasoning_template and template_profile:
-                    final_text = template_profile.process_response(generated_text)
-                    generated_text = final_text
-                    if not template_profile.config.show_reasoning:
-                        display_text = final_text
+                    if uses_reasoning_template and template_profile:
+                        final_text = template_profile.process_response(generated_text)
+                        generated_text = final_text
+                        if not template_profile.config.show_reasoning:
+                            display_text = final_text
 
-                live.update(build_renderable(display_text))
+                    live.update(build_renderable(display_text), refresh=True)
+            finally:
+                self.console._width = original_console_width
 
-            # Add blank line for spacing between response and metrics
-            print()
-            
             # Display final metrics in a clean, professional way
             elapsed = time.time() - start_time
             if token_count > 0 and elapsed > 0:
@@ -1238,6 +1253,9 @@ class CortexCLI:
                 metrics_line = " · ".join(metrics_parts)
                 print(f"  \033[2m{metrics_line}\033[0m")
             
+            if token_count >= request.max_tokens:
+                print(f"  \033[2m(output truncated at max_tokens={request.max_tokens}; increase in config.yaml)\033[0m")
+            
             # Add assistant message to conversation history
             self.conversation_manager.add_message(MessageRole.ASSISTANT, generated_text)