cortex-engine 0.1.0__tar.gz

cortex_engine-0.1.0/PKG-INFO

@@ -0,0 +1,352 @@
+Metadata-Version: 2.3
+Name: cortex-engine
+Version: 0.1.0
+Summary: AI Operating System for Coding — routes queries across 30-50+ LLMs with GPU scheduling, LRU caching, evaluation, and feedback-driven routing.
+Keywords: ai,llm,inference,routing,orchestration,gpu,fastapi,coding-assistant,model-serving
+Author: Cortex-Engine Contributors
+Author-email: Cortex-Engine Contributors <hello@cortex-engine.dev>
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
+Classifier: Framework :: FastAPI
+Classifier: Typing :: Typed
+Requires-Dist: fastapi>=0.115,<1.0
+Requires-Dist: uvicorn[standard]>=0.34,<1.0
+Requires-Dist: redis>=5.2,<6.0
+Requires-Dist: pydantic>=2.10,<3.0
+Requires-Dist: pydantic-settings>=2.7,<3.0
+Requires-Dist: httpx>=0.28,<1.0
+Requires-Dist: sentence-transformers>=3.4,<4.0 ; extra == 'embeddings'
+Requires-Dist: faiss-cpu>=1.10,<2.0 ; extra == 'embeddings'
+Requires-Dist: cortex-engine[embeddings,qdrant,postgres] ; extra == 'full'
+Requires-Dist: sqlalchemy[asyncio]>=2.0,<3.0 ; extra == 'postgres'
+Requires-Dist: asyncpg>=0.30,<1.0 ; extra == 'postgres'
+Requires-Dist: qdrant-client>=1.14,<2.0 ; extra == 'qdrant'
+Requires-Python: >=3.11
+Project-URL: Bug Tracker, https://github.com/imnotdev25/cortex-engine/issues
+Project-URL: Changelog, https://github.com/imnotdev25/cortex-engine/blob/main/CHANGELOG.md
+Project-URL: Homepage, https://github.com/imnotdev25/cortex-engine
+Project-URL: Repository, https://github.com/imnotdev25/cortex-engine
+Provides-Extra: embeddings
+Provides-Extra: full
+Provides-Extra: postgres
+Provides-Extra: qdrant
+Description-Content-Type: text/markdown
+
+# 🧠 Cortex-Engine
+
+> An AI Operating System for coding — routes queries across 30–50+ models, manages GPU scheduling, caching, evaluation, and continuous feedback.
+
+```
+models = processes   |   GPU = CPU   |   router = scheduler   |   kernel = control plane
+```
+
+---
+
+## Architecture
+
+```
+User Query
+    │
+    ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                        Cortex-Engine API  (FastAPI)                  │
+│                                                                   │
+│  POST /inference                                                  │
+│       │                                                           │
+│       ▼                                                           │
+│  ┌─────────────────────────────────────────────────────────┐     │
+│  │              Model Orchestrator (central brain)          │     │
+│  │                                                          │     │
+│  │  ① Cache Check ──────────► Redis LRU Cache              │     │
+│  │       │ miss                                             │     │
+│  │       ▼                                                  │     │
+│  │  ② Router Engine ─────────► Cluster Detection           │     │
+│  │       │                     Model Selection              │     │
+│  │       ▼                     Confidence Score            │     │
+│  │  ③ Scheduler ──────────────► GPU Assignment             │     │
+│  │       │                     Priority Queue              │     │
+│  │       ▼                                                  │     │
+│  │  ④ Model Worker ──────────► vLLM / Triton (Phase 2)     │     │
+│  │       │                                                  │     │
+│  │       ▼                                                  │     │
+│  │  ⑤ Evaluator ─────────────► Static Analysis             │     │
+│  │       │                     LLM Grading (Phase 2)       │     │
+│  │       ▼                                                  │     │
+│  │  ⑥ Feedback Log ──────────► Redis (rolling 10k)         │     │
+│  └─────────────────────────────────────────────────────────┘     │
+└─────────────────────────────────────────────────────────────────┘
+         │
+         ▼
+    Model Registry (Redis Hash)
+    7 seed models → 50+ in Phase 3
+```
+
+---
+
+## Latency Targets
+
+| Component   | Target  |
+|-------------|---------|
+| Router      | < 50 ms |
+| Cache check | < 5 ms  |
+| Scheduling  | < 10 ms |
+| Total overhead | < 100 ms |
+
+---
+
+## Project Structure
+
+```
+cortex_engine/
+├── main.py                  ← FastAPI app, lifespan, middleware, health/metrics
+├── config.py                ← Pydantic Settings (env-based)
+├── dependencies.py          ← All Depends() providers
+│
+├── models/
+│   └── schemas.py           ← All Pydantic request/response models + enums
+│
+├── services/
+│   ├── registry.py          ← Redis-backed model registry (7 seed models)
+│   ├── router.py            ← Keyword/heuristic router with tiebreak scoring
+│   ├── cache_manager.py     ← LRU response cache + warm pool + eviction log
+│   ├── scheduler.py         ← Priority queue scheduler (Redis sorted sets)
+│   ├── evaluator.py         ← Static analysis + LLM-grade stub
+│   ├── feedback.py          ← Rolling feedback log + accuracy stats
+│   └── orchestrator.py      ← Central brain: routes→schedules→infers→evals
+│
+├── routers/
+│   ├── inference.py         ← POST /inference, GET /inference/route-preview
+│   └── api.py               ← /registry CRUD, /admin (queue/cache/feedback)
+│
+├── tests/
+│   ├── conftest.py          ← Async fixtures with fakeredis (no real Redis needed)
+│   ├── test_registry.py     ← 11 tests
+│   ├── test_router.py       ← 11 tests
+│   ├── test_cache.py        ← 7 tests
+│   ├── test_evaluator.py    ← 6 tests
+│   ├── test_scheduler.py    ← 7 tests
+│   ├── test_orchestrator.py ← 11 tests (integration)
+│   └── test_feedback.py     ← 4 tests  (57 total → all pass)
+│
+├── Dockerfile
+├── docker-compose.yml
+├── requirements.txt
+├── pytest.ini
+└── .env.example
+```
+
+---
+
+## Quickstart
+
+### Option A — Local (needs Redis)
+
+```bash
+# 1. Clone & install
+git clone <repo>
+cd cortex_engine
+pip install -r requirements.txt
+
+# 2. Start Redis
+docker run -d -p 6379:6379 redis:7.4-alpine
+
+# 3. Configure
+cp .env.example .env
+
+# 4. Run
+uvicorn main:app --reload --port 8000
+```
+
+### Option B — Docker Compose (recommended)
+
+```bash
+docker compose up --build
+# Optional: include Redis Commander UI
+docker compose --profile dev up --build
+```
+
+### Run Tests (no Redis needed)
+
+```bash
+pip install -r requirements.txt
+pytest tests/ -v
+```
+
+---
+
+## API Reference
+
+### `POST /inference`
+
+Route a query to the best model and get a response.
+
+```json
+// Request
+{
+  "query": "Write a pytest test for a function that reverses a string",
+  "preferred_type": null,       // optional: debugging|testing|explanation|...
+  "preferred_model": null,      // optional: override routing entirely
+  "max_tokens": 2048,
+  "temperature": 0.2,
+  "evaluate": true
+}
+
+// Response
+{
+  "request_id": "a1b2c3d4e5f6",
+  "output": "...",
+  "model_used": "starcoder2-7b",
+  "route": {
+    "cluster": "testing",
+    "selected_model": "starcoder2-7b",
+    "confidence": 0.72,
+    "fallback_models": ["phi-3-mini"],
+    "routing_latency_ms": 1.4
+  },
+  "evaluation": {
+    "success": true,
+    "score": 0.8,
+    "method": "static_analysis",
+    "details": "has_content=✓; has_code=✓; no_apology=✓; syntax_ok=✓; keyword_coverage=✓"
+  },
+  "total_latency_ms": 14.2,
+  "cached": false,
+  "tokens_used": 87
+}
+```
+
+### `GET /inference/route-preview?query=...`
+
+Dry-run: see which model would be selected without running inference.
+
+### `GET /registry/`
+
+List all registered models with status and metadata.
+
+### `POST /registry/`
+
+Register a new model.
+
+### `PATCH /registry/{model_name}/status`
+
+Set model status: `available | loading | busy | offline`
+
+### `GET /admin/queue`
+
+Current GPU queue depth, running jobs, per-GPU load counts.
+
+### `GET /admin/cache`
+
+Cache hit rate, warm pool, recent evictions.
+
+### `GET /admin/feedback`
+
+Routing accuracy stats and recent feedback log.
+
+### `GET /health`
+
+Liveness check — Redis connectivity + model counts.
+
+### `GET /metrics`
+
+Full system metrics: routing accuracy, cache hit rate, GPU loads, queue depth.
+
+---
+
+## Cluster → Model Routing
+
+| Cluster | Trigger Keywords | Models |
+|---------|-----------------|--------|
+| `debugging` | error, exception, traceback, crash, fix bug | codellama-7b |
+| `testing` | pytest, jest, unit test, mock, assert | starcoder2-7b |
+| `explanation` | explain, what is, how does, document | mistral-7b-instruct |
+| `refactoring` | refactor, optimize, clean up, simplify | qwen-coder-14b |
+| `python` | python, django, fastapi, flask, .py | qwen-coder-7b, qwen-coder-14b |
+| `general_code` | code, class, implement, build | deepseek-coder-6.7b |
+| `fallback` | everything else | phi-3-mini |
+
+---
+
+## Seed Models (Phase 1 MVP)
+
+| Model | Cluster | Size | Latency | GPU |
+|-------|---------|------|---------|-----|
+| `qwen-coder-7b` | python | 7B | 250ms | gpu-0 |
+| `qwen-coder-14b` | python | 14B | 400ms | gpu-1 |
+| `deepseek-coder-6.7b` | general_code | 6.7B | 220ms | gpu-0 |
+| `codellama-7b` | debugging | 7B | 270ms | gpu-1 |
+| `mistral-7b-instruct` | explanation | 7B | 300ms | gpu-2 |
+| `starcoder2-7b` | testing | 7B | 260ms | gpu-2 |
+| `phi-3-mini` | fallback | 3.8B | 150ms | gpu-3 |
+
+---
+
+## Roadmap
+
+### Phase 1 (now — MVP)
+- [x] FastAPI kernel with all core services
+- [x] Redis-backed model registry (7 models)
+- [x] Keyword/heuristic router with cluster detection
+- [x] LRU response cache + warm pool + eviction log
+- [x] Priority queue GPU scheduler (Redis sorted sets)
+- [x] Static analysis evaluator
+- [x] Rolling feedback system + accuracy tracking
+- [x] 59 tests (all passing, no real Redis needed)
+- [x] Docker Compose stack
+
+### Phase 2
+- [ ] Swap inference stub → real vLLM HTTP calls
+- [ ] Embedding-based routing (BGE/E5 + FAISS/Qdrant)
+- [ ] 15 models across more language clusters
+- [ ] Ray Serve for distributed model workers
+- [ ] LLM judge model for evaluation (phi-3-mini)
+
+### Phase 3
+- [ ] 30–50+ models
+- [ ] Self-improving router (daily retraining from feedback)
+- [ ] Cost-aware routing (balance quality vs. $/token)
+- [ ] Multi-model collaboration (chain models)
+- [ ] PostgreSQL for persistent metadata
+- [ ] Reinforcement learning router
+
+---
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
+| `REDIS_MAX_CONNECTIONS` | `50` | Connection pool size |
+| `PORT` | `8000` | API server port |
+| `WORKERS` | `1` | Uvicorn worker count |
+| `CACHE_TTL_SECONDS` | `3600` | Response cache TTL |
+| `ENABLE_EVALUATION` | `true` | Run evaluator on outputs |
+| `ENABLE_FEEDBACK` | `true` | Log routing feedback |
+
+---
+
+## Tech Stack
+
+| Layer | Tech |
+|-------|------|
+| API | FastAPI + Uvicorn |
+| Cache / State | Redis 7 (asyncio) |
+| Scheduling | Redis sorted sets |
+| Config | Pydantic Settings |
+| Testing | pytest-asyncio + fakeredis |
+| Containers | Docker + Docker Compose |
+| Phase 2 Serving | vLLM + Triton |
+| Phase 2 Orchestration | Ray Serve + Kubernetes |
+| Phase 2 Vector DB | FAISS / Qdrant |
+| Phase 3 Metadata | PostgreSQL + SQLAlchemy |

cortex_engine-0.1.0/README.md

@@ -0,0 +1,307 @@
+# 🧠 Cortex-Engine
+
+> An AI Operating System for coding — routes queries across 30–50+ models, manages GPU scheduling, caching, evaluation, and continuous feedback.
+
+```
+models = processes   |   GPU = CPU   |   router = scheduler   |   kernel = control plane
+```
+
+---
+
+## Architecture
+
+```
+User Query
+    │
+    ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                        Cortex-Engine API  (FastAPI)                  │
+│                                                                   │
+│  POST /inference                                                  │
+│       │                                                           │
+│       ▼                                                           │
+│  ┌─────────────────────────────────────────────────────────┐     │
+│  │              Model Orchestrator (central brain)          │     │
+│  │                                                          │     │
+│  │  ① Cache Check ──────────► Redis LRU Cache              │     │
+│  │       │ miss                                             │     │
+│  │       ▼                                                  │     │
+│  │  ② Router Engine ─────────► Cluster Detection           │     │
+│  │       │                     Model Selection              │     │
+│  │       ▼                     Confidence Score            │     │
+│  │  ③ Scheduler ──────────────► GPU Assignment             │     │
+│  │       │                     Priority Queue              │     │
+│  │       ▼                                                  │     │
+│  │  ④ Model Worker ──────────► vLLM / Triton (Phase 2)     │     │
+│  │       │                                                  │     │
+│  │       ▼                                                  │     │
+│  │  ⑤ Evaluator ─────────────► Static Analysis             │     │
+│  │       │                     LLM Grading (Phase 2)       │     │
+│  │       ▼                                                  │     │
+│  │  ⑥ Feedback Log ──────────► Redis (rolling 10k)         │     │
+│  └─────────────────────────────────────────────────────────┘     │
+└─────────────────────────────────────────────────────────────────┘
+         │
+         ▼
+    Model Registry (Redis Hash)
+    7 seed models → 50+ in Phase 3
+```
+
+---
+
+## Latency Targets
+
+| Component   | Target  |
+|-------------|---------|
+| Router      | < 50 ms |
+| Cache check | < 5 ms  |
+| Scheduling  | < 10 ms |
+| Total overhead | < 100 ms |
+
+---
+
+## Project Structure
+
+```
+cortex_engine/
+├── main.py                  ← FastAPI app, lifespan, middleware, health/metrics
+├── config.py                ← Pydantic Settings (env-based)
+├── dependencies.py          ← All Depends() providers
+│
+├── models/
+│   └── schemas.py           ← All Pydantic request/response models + enums
+│
+├── services/
+│   ├── registry.py          ← Redis-backed model registry (7 seed models)
+│   ├── router.py            ← Keyword/heuristic router with tiebreak scoring
+│   ├── cache_manager.py     ← LRU response cache + warm pool + eviction log
+│   ├── scheduler.py         ← Priority queue scheduler (Redis sorted sets)
+│   ├── evaluator.py         ← Static analysis + LLM-grade stub
+│   ├── feedback.py          ← Rolling feedback log + accuracy stats
+│   └── orchestrator.py      ← Central brain: routes→schedules→infers→evals
+│
+├── routers/
+│   ├── inference.py         ← POST /inference, GET /inference/route-preview
+│   └── api.py               ← /registry CRUD, /admin (queue/cache/feedback)
+│
+├── tests/
+│   ├── conftest.py          ← Async fixtures with fakeredis (no real Redis needed)
+│   ├── test_registry.py     ← 11 tests
+│   ├── test_router.py       ← 11 tests
+│   ├── test_cache.py        ← 7 tests
+│   ├── test_evaluator.py    ← 6 tests
+│   ├── test_scheduler.py    ← 7 tests
+│   ├── test_orchestrator.py ← 11 tests (integration)
+│   └── test_feedback.py     ← 4 tests  (57 total → all pass)
+│
+├── Dockerfile
+├── docker-compose.yml
+├── requirements.txt
+├── pytest.ini
+└── .env.example
+```
+
+---
+
+## Quickstart
+
+### Option A — Local (needs Redis)
+
+```bash
+# 1. Clone & install
+git clone <repo>
+cd cortex_engine
+pip install -r requirements.txt
+
+# 2. Start Redis
+docker run -d -p 6379:6379 redis:7.4-alpine
+
+# 3. Configure
+cp .env.example .env
+
+# 4. Run
+uvicorn main:app --reload --port 8000
+```
+
+### Option B — Docker Compose (recommended)
+
+```bash
+docker compose up --build
+# Optional: include Redis Commander UI
+docker compose --profile dev up --build
+```
+
+### Run Tests (no Redis needed)
+
+```bash
+pip install -r requirements.txt
+pytest tests/ -v
+```
+
+---
+
+## API Reference
+
+### `POST /inference`
+
+Route a query to the best model and get a response.
+
+```json
+// Request
+{
+  "query": "Write a pytest test for a function that reverses a string",
+  "preferred_type": null,       // optional: debugging|testing|explanation|...
+  "preferred_model": null,      // optional: override routing entirely
+  "max_tokens": 2048,
+  "temperature": 0.2,
+  "evaluate": true
+}
+
+// Response
+{
+  "request_id": "a1b2c3d4e5f6",
+  "output": "...",
+  "model_used": "starcoder2-7b",
+  "route": {
+    "cluster": "testing",
+    "selected_model": "starcoder2-7b",
+    "confidence": 0.72,
+    "fallback_models": ["phi-3-mini"],
+    "routing_latency_ms": 1.4
+  },
+  "evaluation": {
+    "success": true,
+    "score": 0.8,
+    "method": "static_analysis",
+    "details": "has_content=✓; has_code=✓; no_apology=✓; syntax_ok=✓; keyword_coverage=✓"
+  },
+  "total_latency_ms": 14.2,
+  "cached": false,
+  "tokens_used": 87
+}
+```
+
+### `GET /inference/route-preview?query=...`
+
+Dry-run: see which model would be selected without running inference.
+
+### `GET /registry/`
+
+List all registered models with status and metadata.
+
+### `POST /registry/`
+
+Register a new model.
+
+### `PATCH /registry/{model_name}/status`
+
+Set model status: `available | loading | busy | offline`
+
+### `GET /admin/queue`
+
+Current GPU queue depth, running jobs, per-GPU load counts.
+
+### `GET /admin/cache`
+
+Cache hit rate, warm pool, recent evictions.
+
+### `GET /admin/feedback`
+
+Routing accuracy stats and recent feedback log.
+
+### `GET /health`
+
+Liveness check — Redis connectivity + model counts.
+
+### `GET /metrics`
+
+Full system metrics: routing accuracy, cache hit rate, GPU loads, queue depth.
+
+---
+
+## Cluster → Model Routing
+
+| Cluster | Trigger Keywords | Models |
+|---------|-----------------|--------|
+| `debugging` | error, exception, traceback, crash, fix bug | codellama-7b |
+| `testing` | pytest, jest, unit test, mock, assert | starcoder2-7b |
+| `explanation` | explain, what is, how does, document | mistral-7b-instruct |
+| `refactoring` | refactor, optimize, clean up, simplify | qwen-coder-14b |
+| `python` | python, django, fastapi, flask, .py | qwen-coder-7b, qwen-coder-14b |
+| `general_code` | code, class, implement, build | deepseek-coder-6.7b |
+| `fallback` | everything else | phi-3-mini |
+
+---
+
+## Seed Models (Phase 1 MVP)
+
+| Model | Cluster | Size | Latency | GPU |
+|-------|---------|------|---------|-----|
+| `qwen-coder-7b` | python | 7B | 250ms | gpu-0 |
+| `qwen-coder-14b` | python | 14B | 400ms | gpu-1 |
+| `deepseek-coder-6.7b` | general_code | 6.7B | 220ms | gpu-0 |
+| `codellama-7b` | debugging | 7B | 270ms | gpu-1 |
+| `mistral-7b-instruct` | explanation | 7B | 300ms | gpu-2 |
+| `starcoder2-7b` | testing | 7B | 260ms | gpu-2 |
+| `phi-3-mini` | fallback | 3.8B | 150ms | gpu-3 |
+
+---
+
+## Roadmap
+
+### Phase 1 (now — MVP)
+- [x] FastAPI kernel with all core services
+- [x] Redis-backed model registry (7 models)
+- [x] Keyword/heuristic router with cluster detection
+- [x] LRU response cache + warm pool + eviction log
+- [x] Priority queue GPU scheduler (Redis sorted sets)
+- [x] Static analysis evaluator
+- [x] Rolling feedback system + accuracy tracking
+- [x] 59 tests (all passing, no real Redis needed)
+- [x] Docker Compose stack
+
+### Phase 2
+- [ ] Swap inference stub → real vLLM HTTP calls
+- [ ] Embedding-based routing (BGE/E5 + FAISS/Qdrant)
+- [ ] 15 models across more language clusters
+- [ ] Ray Serve for distributed model workers
+- [ ] LLM judge model for evaluation (phi-3-mini)
+
+### Phase 3
+- [ ] 30–50+ models
+- [ ] Self-improving router (daily retraining from feedback)
+- [ ] Cost-aware routing (balance quality vs. $/token)
+- [ ] Multi-model collaboration (chain models)
+- [ ] PostgreSQL for persistent metadata
+- [ ] Reinforcement learning router
+
+---
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
+| `REDIS_MAX_CONNECTIONS` | `50` | Connection pool size |
+| `PORT` | `8000` | API server port |
+| `WORKERS` | `1` | Uvicorn worker count |
+| `CACHE_TTL_SECONDS` | `3600` | Response cache TTL |
+| `ENABLE_EVALUATION` | `true` | Run evaluator on outputs |
+| `ENABLE_FEEDBACK` | `true` | Log routing feedback |
+
+---
+
+## Tech Stack
+
+| Layer | Tech |
+|-------|------|
+| API | FastAPI + Uvicorn |
+| Cache / State | Redis 7 (asyncio) |
+| Scheduling | Redis sorted sets |
+| Config | Pydantic Settings |
+| Testing | pytest-asyncio + fakeredis |
+| Containers | Docker + Docker Compose |
+| Phase 2 Serving | vLLM + Triton |
+| Phase 2 Orchestration | Ray Serve + Kubernetes |
+| Phase 2 Vector DB | FAISS / Qdrant |
+| Phase 3 Metadata | PostgreSQL + SQLAlchemy |