modelab 0.1.0__tar.gz

modelab-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,43 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Run tests
+        run: pytest -m "not docker"
+
+  publish:
+    needs: test
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

modelab-0.1.0/.gitignore

@@ -0,0 +1,39 @@
+# Python
+.venv
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+.pytest_cache
+.ruff_cache
+*.egg-info
+dist/
+build/
+
+# Node.js
+node_modules/
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# IDEs
+.vscode
+.idea
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Environment
+.env
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+
+# Dashboard build output
+dashboard/dist/

modelab-0.1.0/ARD/001-record-response-api.md

@@ -0,0 +1,58 @@
+# ARD-001: Replace `track()` with `record(response)`
+
+**Status:** Accepted
+**Date:** 2026-02-16
+
+## Context
+
+The `track()` context manager forced users to nest their LLM calls inside modelab:
+
+```python
+with assignment.track() as t:
+    response = client.chat.completions.create(...)
+    t.set_tokens(input=usage.prompt_tokens, output=usage.completion_tokens)
+    t.set_cost(0.013)
+```
+
+This couples user code to the SDK. Users with existing wrappers (LangSmith, Sentry, custom middleware) cannot compose them with `track()` — context managers don't compose cleanly. modelab should observe LLM calls, not own their execution.
+
+## Decision
+
+Remove `track()` and `_Tracker` entirely. Extend `record()` to accept a provider response object as its first positional argument and auto-extract token usage via duck-typing:
+
+```python
+response = client.chat.completions.create(...)
+assignment.record(response, cost=0.013)
+```
+
+Duck-typing order:
+1. **OpenAI**: `response.usage.prompt_tokens` / `response.usage.completion_tokens`
+2. **Anthropic**: `response.usage.input_tokens` / `response.usage.output_tokens`
+
+Explicit keyword arguments (`input_tokens=`, `output_tokens=`) always override extracted values. The backward-compatible kwargs-only call still works:
+
+```python
+assignment.record(input_tokens=50, output_tokens=100, cost=0.01)
+```
+
+## Consequences
+
+### Positive
+
+- **No coupling**: Users call their LLM however they want, then pass the response to `record()`.
+- **Composable**: Works alongside any other middleware, wrappers, or observability tools.
+- **Zero dependencies**: Duck-typing via `getattr` — no provider imports needed.
+- **Simpler API surface**: One method (`record`) instead of a context manager + 3 setter methods.
+
+### Negative
+
+- **No auto-latency**: `track()` measured wall-clock time automatically. Users who want latency now measure it externally and pass `latency_ms=`.
+- **No auto-error capture**: `track()` caught exceptions and recorded them. Users now use try/except and pass `error=` + call `mark_failure()`.
+- **Breaking change**: All code using `track()`, `set_tokens()`, `set_cost()`, `set_metadata()` must migrate.
+
+### Design decisions within this change
+
+- **Cost stays manual**: Provider pricing changes too frequently for reliable auto-calculation.
+- **No auto-latency**: Acceptable tradeoff — users who need it measure externally. Keeps the SDK stateless.
+- **No auto-error capture**: Users call `mark_failure()` for explicit error handling. Cleaner separation of concerns.
+- **Duck-typing order**: OpenAI first (larger market share), then Anthropic. Both are tried via `getattr` with no short-circuiting penalty.

modelab-0.1.0/Dockerfile

@@ -0,0 +1,26 @@
+# Stage 1: Build dashboard
+FROM node:22-slim AS frontend
+WORKDIR /app/dashboard
+COPY dashboard/package.json dashboard/package-lock.json ./
+RUN npm ci
+COPY dashboard/ ./
+RUN npm run build
+
+# Stage 2: Python runtime
+FROM python:3.13-slim
+WORKDIR /app
+
+# Copy Python source + metadata needed for install
+COPY pyproject.toml README.md ./
+COPY modelab/ modelab/
+COPY server/ server/
+
+# Install Python dependencies
+RUN pip install --no-cache-dir ".[server]"
+
+# Copy built dashboard
+COPY --from=frontend /app/dashboard/dist dashboard/dist
+
+EXPOSE 8100
+
+CMD ["python", "-m", "uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8100"]

modelab-0.1.0/IMPLEMENTATION.md

@@ -0,0 +1,252 @@
+# modelab — Implementation Plan
+
+## Overview
+
+**modelab** is a provider-agnostic Python library + self-hosted server for A/B testing LLM systems in production.
+
+Two components:
+
+1. **Python SDK** — zero-dep library developers install in their app (assignment, tracking, events)
+2. **Server + Dashboard** — self-hosted FastAPI + React app for visualization (Docker Compose)
+
+---
+
+## API
+
+```python
+import modelab
+from modelab import Flag, Variant, EvalContext
+
+# Initialize
+modelab.init(
+    server="http://localhost:8100",
+    flags=[
+        Flag(
+            name="summarizer_v2",
+            variants=[
+                Variant("control", weight=50, config={"model": "gpt-3.5-turbo", "prompt": "Summarize: {input}"}),
+                Variant("treatment", weight=50, config={"model": "gpt-4", "prompt": "Concisely summarize: {input}"}),
+            ],
+            rollout_pct=100,
+        ),
+    ],
+)
+
+# Assign
+ctx = EvalContext(user_id="123", session_id="abc")
+assignment = modelab.assign("summarizer_v2", ctx)
+
+if assignment is None:
+    # Outside rollout — default behavior
+    response = call_llm(model="gpt-3.5-turbo", prompt=text)
+else:
+    # In experiment — use assigned variant
+    response = call_llm(
+        model=assignment.config["model"],
+        prompt=assignment.config["prompt"].format(input=text),
+    )
+    assignment.record(response, cost=0.013)
+
+    assignment.mark_success()
+    assignment.mark_custom_event("copied")
+
+# Evaluate
+results = modelab.evaluate("summarizer_v2")
+```
+
+---
+
+## System Architecture
+
+```
+Developer's App
+│
+├── modelab SDK (pip install modelab)
+│   └── ServerStorage ──HTTP POST──▶ modelab-server
+│
+modelab-server (docker compose up)
+├── FastAPI backend
+├── React dashboard (served as static files)
+└── PostgreSQL
+```
+
+---
+
+## File Structure
+
+```
+modelab/                         # Python SDK
+    __init__.py                  # Public API: init(), assign(), evaluate()
+    _types.py                    # Flag, Variant, EvalContext, records, Storage Protocol
+    _engine.py                   # Deterministic hashing + bucketing
+    _assignment.py               # Assignment class: record(), mark_*()
+    _storage.py                  # Storage Protocol + SQLiteStorage
+    _server_storage.py           # ServerStorage (HTTP + buffering)
+    _aggregator.py               # Metrics evaluation queries
+    _state.py                    # Module-level singleton state
+    _errors.py                   # Exception hierarchy
+    py.typed                     # PEP 561 marker
+
+server/                          # FastAPI backend
+    __init__.py
+    app.py                       # FastAPI app + static file mount
+    config.py                    # Settings via env vars
+    database.py                  # Postgres connection, schema, queries
+    models.py                    # Pydantic request/response schemas
+    routes/
+        __init__.py
+        ingest.py                # POST /api/v1/ingest/*
+        api.py                   # GET /api/v1/flags/*
+
+dashboard/                       # React SPA
+    package.json
+    vite.config.ts
+    tsconfig.json
+    src/
+        main.tsx
+        App.tsx
+        lib/
+            utils.ts             # shadcn utils
+            api.ts               # Fetch wrapper for server API
+        components/
+            ui/                  # shadcn components
+            flags-table.tsx
+            flag-detail.tsx
+            variant-card.tsx
+            metrics-chart.tsx
+        pages/
+            overview.tsx         # /
+            flag.tsx             # /flags/:name
+
+tests/
+    conftest.py
+    test_engine.py
+    test_storage.py
+    test_assignment.py
+    test_aggregator.py
+    test_integration.py
+
+pyproject.toml
+Dockerfile
+docker-compose.yml
+README.md
+```
+
+---
+
+## Data Model
+
+### assignments
+
+| Column        | Type        | Description             |
+| ------------- | ----------- | ----------------------- |
+| assignment_id | UUID PK     | Unique ID               |
+| flag_name     | TEXT        | Flag identifier         |
+| variant_name  | TEXT        | Assigned variant        |
+| user_id       | TEXT        | User identifier         |
+| session_id    | TEXT        | Optional session        |
+| config_json   | JSONB       | Variant config snapshot |
+| assigned_at   | TIMESTAMPTZ | When assigned           |
+
+### executions (1:1 with assignments)
+
+| Column        | Type        | Description         |
+| ------------- | ----------- | ------------------- |
+| assignment_id | UUID PK FK  | Links to assignment |
+| latency_ms    | FLOAT       | Wall-clock latency  |
+| input_tokens  | INT         | Input token count   |
+| output_tokens | INT         | Output token count  |
+| cost          | FLOAT       | Dollar cost         |
+| error         | TEXT        | Exception message   |
+| metadata_json | JSONB       | Arbitrary metadata  |
+| recorded_at   | TIMESTAMPTZ | When recorded       |
+
+### events (1:N with assignments)
+
+| Column        | Type        | Description                |
+| ------------- | ----------- | -------------------------- |
+| event_id      | UUID PK     | Unique ID                  |
+| assignment_id | UUID FK     | Links to assignment        |
+| event_type    | TEXT        | success / failure / custom |
+| event_name    | TEXT        | For custom events          |
+| payload_json  | JSONB       | Optional payload           |
+| created_at    | TIMESTAMPTZ | When created               |
+
+### Indexes
+
+- `(flag_name, variant_name)` on assignments
+- `(flag_name, user_id)` on assignments
+- `(assigned_at)` on assignments
+- `(assignment_id)` on events
+- `(assignment_id, event_type)` on events
+
+---
+
+## Server API
+
+### Ingestion (from SDK)
+
+```
+POST /api/v1/ingest/assignments    body: AssignmentRecord[]
+POST /api/v1/ingest/executions     body: ExecutionRecord[]
+POST /api/v1/ingest/events         body: EventRecord[]
+```
+
+All accept batches. Authenticated via `X-API-Key` header.
+
+### Dashboard API
+
+```
+GET /api/v1/flags                  → list of flags with summary stats
+GET /api/v1/flags/{name}           → detailed per-variant evaluation
+GET /api/v1/flags/{name}/timeline  → time-series metrics for charts
+```
+
+---
+
+## Dashboard Pages
+
+### Flags Overview (`/`)
+
+- Table: flag name, variants, total assignments, success rate, status
+- Click row → detail page
+
+### Flag Detail (`/flags/:name`)
+
+- Variant comparison cards (assignments, success rate, avg latency, avg cost)
+- Charts (recharts):
+  - Success rate bar chart
+  - Latency comparison
+  - Cost comparison
+  - Assignments over time
+- Events breakdown table
+
+---
+
+## Key Design Decisions
+
+| Decision         | Choice                                                    | Why                                        |
+| ---------------- | --------------------------------------------------------- | ------------------------------------------ |
+| LLM coupling     | None — `assign()` returns config, dev calls their own LLM | Works with any stack                       |
+| Events target    | Assignment object only                                    | Explicit, no ambiguity with multi-flag     |
+| Hashing          | md5 (stdlib)                                              | Zero deps, excellent distribution          |
+| Bucket count     | 10,000                                                    | 0.01% rollout granularity                  |
+| Storage errors   | Log warning, don't crash                                  | Feature flags are auxiliary                |
+| Python version   | 3.10+                                                     | Modern type hints                          |
+| SDK dependencies | Zero                                                      | Maximum adoptability                       |
+| Server DB        | PostgreSQL                                                | Production-grade, JSONB, proper timestamps |
+| Dashboard        | React + shadcn/ui                                         | Polished components, good DX               |
+| Deployment       | Single Dockerfile, docker-compose                         | One command to self-host                   |
+
+---
+
+## Excluded from v1
+
+- Async SDK
+- LLM provider integrations
+- Statistical significance / p-values
+- Bayesian testing
+- Multi-armed bandits
+- Remote flag management (flags in code, dashboard is read-only)
+- User auth in dashboard (API key only)
+- Real-time streaming

modelab-0.1.0/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: modelab
+Version: 0.1.0
+Summary: Provider-agnostic A/B testing for LLM systems
+License-Expression: MIT
+Keywords: ab-testing,experiments,feature-flags,llm
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110; extra == 'server'
+Requires-Dist: psycopg[binary,pool]>=3.1; extra == 'server'
+Requires-Dist: uvicorn[standard]>=0.29; extra == 'server'
+Description-Content-Type: text/markdown
+
+# modelab
+
+Provider-agnostic A/B testing for LLM systems in production.
+
+**Two components:**
+1. **Python SDK** — zero-dependency library for assignment, tracking, and evaluation
+2. **Server + Dashboard** — self-hosted FastAPI + React app for visualization (Docker Compose)
+
+## Quick Start
+
+### SDK (local development)
+
+```bash
+pip install modelab
+```
+
+```python
+import modelab
+from modelab import Flag, Variant, EvalContext
+
+# Initialize — point to the modelab server
+modelab.init(
+    server="http://localhost:8100",
+    flags=[
+        Flag(
+            name="summarizer_v2",
+            variants=[
+                Variant("control", weight=50, config={"model": "gpt-3.5-turbo", "prompt": "Summarize: {input}"}),
+                Variant("treatment", weight=50, config={"model": "gpt-4", "prompt": "Concisely summarize: {input}"}),
+            ],
+            rollout_pct=100,
+        ),
+    ],
+)
+
+# Assign a variant
+ctx = EvalContext(user_id="user_123", session_id="abc")
+assignment = modelab.assign("summarizer_v2", ctx)
+
+if assignment is None:
+    # Outside rollout — use default behavior
+    response = call_llm(model="gpt-3.5-turbo", prompt=text)
+else:
+    # In experiment — use assigned variant config
+    response = call_llm(
+        model=assignment.config["model"],
+        prompt=assignment.config["prompt"].format(input=text),
+    )
+    assignment.record(response, cost=0.013)
+    assignment.mark_success()
+
+# Evaluate results
+results = modelab.evaluate("summarizer_v2")
+print(results)
+```
+
+### Self-Hosted Server + Dashboard
+
+```bash
+docker compose up
+```
+
+This starts:
+- **PostgreSQL** on port 5432
+- **modelab server + dashboard** on port 8100
+
+## Concepts
+
+### Flags
+An experiment with one or more variants and a rollout percentage (0-100%).
+
+### Variants
+Each variant has a name, weight (for traffic splitting), and a config dict you use to parameterize your LLM calls.
+
+### Assignment
+Deterministic — the same `(flag_name, user_id)` always maps to the same variant. Uses MD5 hashing into 10,000 buckets for 0.01% rollout granularity.
+
+### Recording
+
+Use `assignment.record(response)` to capture execution metrics. Token counts are automatically extracted from the response object via duck-typing (supports OpenAI and Anthropic response formats). Cost, latency, error, and arbitrary metadata can be passed as keyword arguments:
+
+```python
+assignment.record(response, cost=0.013, latency_ms=250.0, model="gpt-4o")
+```
+
+You can also record without a response object:
+
+```python
+assignment.record(input_tokens=50, output_tokens=100, cost=0.01)
+```
+
+### Events
+Mark assignments as success/failure or record custom events (e.g., "copied", "thumbs_up").
+
+### Evaluation
+`modelab.evaluate(flag_name)` returns per-variant metrics: success rate, avg latency, avg cost, token usage, and custom event counts.
+
+## Server API
+
+### Ingestion (from SDK)
+```
+POST /api/v1/ingest/assignments    (batch)
+POST /api/v1/ingest/executions     (batch)
+POST /api/v1/ingest/events         (batch)
+```
+
+### Dashboard API
+```
+GET /api/v1/flags                  — list flags with summary stats
+GET /api/v1/flags/{name}           — detailed per-variant evaluation
+GET /api/v1/flags/{name}/timeline  — time-series metrics
+```
+
+## Development
+
+```bash
+# Install in dev mode
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run dashboard dev server
+cd dashboard && npm install && npm run dev
+
+# Run API server (requires Postgres)
+uvicorn server.app:app --reload --port 8100
+```
+
+## Architecture
+
+```
+Developer's App
+│
+├── modelab SDK (pip install modelab)
+│   └── ServerStorage ──HTTP POST──▶ modelab-server
+│
+modelab-server (docker compose up)
+├── FastAPI backend
+├── React dashboard (served as static files)
+└── PostgreSQL
+```
+
+## License
+
+MIT