ctx-cc 3.5.0 → 4.1.0

package/skills/ctx-ml-pipeline/SKILL.md

@@ -0,0 +1,437 @@
+---
+name: ctx-ml-pipeline
+description: |
+  WHEN: Building ML training pipelines, data processing pipelines, inference services, model deployment, CI/CT/CD for ML, or adding production reliability patterns (circuit breaker, drift detection, conformal prediction).
+  WHEN NOT: One-off analysis, EDA, non-ML infrastructure, experiment hypothesis formation.
+---
+
+# CTX ML Pipeline — Production ML Engineering Patterns
+
+You build production-grade ML pipelines. Speed matters less than correctness and reliability. Every pipeline component must be testable, observable, and reproducible.
+
+## Core Principle
+
+A model is not done when training converges. It is done when it can be deployed, monitored, degraded gracefully, and retrained without manual intervention.
+
+## Pipeline Architecture
+
+```
+                    ┌─────────────┐
+                    │   Raw Data  │
+                    └──────┬──────┘
+                           │
+                    ┌──────▼──────┐
+                    │  Validation │  Pandera schema — fail fast
+                    └──────┬──────┘
+                           │
+                    ┌──────▼──────┐
+                    │  Features   │  Deterministic transforms
+                    └──────┬──────┘
+                           │
+              ┌────────────┼────────────┐
+              │            │            │
+       ┌──────▼──────┐ ┌──▼──────┐ ┌──▼──────────┐
+       │  Training   │ │   HPO   │ │  Evaluation  │
+       └──────┬──────┘ └──┬──────┘ └──────┬───────┘
+              │            │               │
+              └────────────┴───────────────┘
+                           │
+                    ┌──────▼──────┐
+                    │  Registry   │  Version + lineage + promotion
+                    └──────┬──────┘
+                           │
+                    ┌──────▼──────┐
+                    │  Inference  │  Envelope + circuit breaker
+                    └──────┬──────┘
+                           │
+                    ┌──────▼──────┐
+                    │  Monitoring │  Drift + calibration alerts
+                    └─────────────┘
+```
+
+## Stage 1: Data Validation
+
+Use Pandera for schema enforcement. Fail at ingestion, not at training time.
+
+```python
+import pandera as pa
+from pandera import Column, DataFrameSchema, Check
+
+schema = DataFrameSchema({
+    "age": Column(float, Check.between(0, 120), nullable=False),
+    "cholesterol": Column(float, Check.greater_than(0), nullable=True),
+    "label": Column(int, Check.isin([0, 1]), nullable=False),
+})
+
+def validate(df: pd.DataFrame) -> pd.DataFrame:
+    try:
+        return schema.validate(df, lazy=True)
+    except pa.errors.SchemaErrors as e:
+        raise ValueError(f"Schema validation failed:\n{e.failure_cases}")
+```
+
+Validation failures are hard errors. Never train on unvalidated data.
+
+## Stage 2: Feature Pipeline
+
+Transforms must be:
+- Deterministic given the same input
+- Serializable (fit statistics saved alongside model)
+- Versioned in the feature registry
+
+```python
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+import joblib
+
+def build_feature_pipeline(feature_registry: dict) -> Pipeline:
+    steps = []
+    for feature, spec in feature_registry.items():
+        if spec["type"] == "numeric":
+            steps.append((f"scale_{feature}", StandardScaler()))
+    return Pipeline(steps)
+
+def fit_and_save(pipeline, X_train, path):
+    pipeline.fit(X_train)
+    joblib.dump(pipeline, path)
+    return pipeline
+```
+
+Save fit statistics (mean, std, bin edges) alongside model artifacts. Inference must use the same fitted pipeline as training.
+
+## Stage 3: Training
+
+```python
+import xgboost as xgb
+from sklearn.model_selection import cross_val_score
+import numpy as np
+
+def train(X_train, y_train, config: dict):
+    model = xgb.XGBClassifier(
+        max_depth=config["max_depth"],
+        n_estimators=config["n_estimators"],
+        learning_rate=config["learning_rate"],
+        subsample=config["subsample"],
+        colsample_bytree=config["colsample_bytree"],
+        random_state=config["seed"],
+        eval_metric="auc",
+        early_stopping_rounds=20,
+    )
+    model.fit(
+        X_train, y_train,
+        eval_set=[(X_val, y_val)],
+        verbose=50,
+    )
+    return model
+```
+
+Always use early stopping. Log training curves to artifacts.
+
+## Stage 4: Hyperparameter Optimization
+
+Use Optuna for HPO. Never tune manually.
+
+```python
+import optuna
+
+def objective(trial, X_train, y_train, X_val, y_val):
+    params = {
+        "max_depth": trial.suggest_int("max_depth", 3, 8),
+        "n_estimators": trial.suggest_int("n_estimators", 100, 500),
+        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
+        "subsample": trial.suggest_float("subsample", 0.6, 1.0),
+        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.6, 1.0),
+    }
+    model = xgb.XGBClassifier(**params, random_state=42)
+    model.fit(X_train, y_train)
+    return roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
+
+study = optuna.create_study(direction="maximize")
+study.optimize(objective, n_trials=100, timeout=3600)
+best_params = study.best_params
+```
+
+Save study results to `artifacts/hpo_study.pkl` for reproducibility.
+
+## Stage 5: Evaluation
+
+```python
+from sklearn.metrics import roc_auc_score, precision_score, recall_score
+from sklearn.calibration import calibration_curve
+import matplotlib.pyplot as plt
+
+def evaluate(model, X_test, y_test, threshold=0.5) -> dict:
+    proba = model.predict_proba(X_test)[:, 1]
+    pred = (proba >= threshold).astype(int)
+
+    metrics = {
+        "auc": roc_auc_score(y_test, proba),
+        "precision": precision_score(y_test, pred),
+        "recall": recall_score(y_test, pred),
+        "threshold": threshold,
+    }
+
+    # Calibration check
+    fraction_pos, mean_pred = calibration_curve(y_test, proba, n_bins=10)
+    metrics["calibration_error"] = float(np.mean(np.abs(fraction_pos - mean_pred)))
+
+    return metrics
+```
+
+Write metrics to `RESULTS.md` and update `models/registry.yaml` if promotion criteria are met.
+
+## Stage 6: Model Registry
+
+Registry enforces promotion criteria before writing a new production version.
+
+```python
+import yaml
+from pathlib import Path
+
+def promote_model(name: str, version: str, metrics: dict, experiment_id: str):
+    registry_path = Path(".ctx/ml/models/registry.yaml")
+    registry = yaml.safe_load(registry_path.read_text())
+
+    model = registry["models"][name]
+    current_version = model["current"]
+    current_metrics = model["versions"][current_version]["metrics"]
+    criteria = model["promotion_criteria"]
+
+    primary_field, primary_op, primary_threshold = parse_criterion(criteria["primary"])
+    guard_field, guard_op, guard_threshold = parse_criterion(criteria["guard"])
+
+    primary_delta = metrics[primary_field] - current_metrics[primary_field]
+    guard_delta = abs(metrics[guard_field] - current_metrics[guard_field])
+
+    if not eval(f"{primary_delta} {primary_op} {primary_threshold}"):
+        raise ValueError(f"Promotion rejected: {primary_field} delta={primary_delta:.4f} < {primary_threshold}")
+    if not eval(f"{guard_delta} {guard_op} {guard_threshold}"):
+        raise ValueError(f"Promotion rejected: {guard_field} regression={guard_delta:.4f} > {guard_threshold}")
+
+    model["versions"][version] = {
+        "metrics": metrics,
+        "experiment": experiment_id,
+        "date": date.today().isoformat(),
+        "status": "production",
+        "artifacts": f".ctx/ml/experiments/{experiment_id}/artifacts/",
+    }
+    model["versions"][current_version]["status"] = "retired"
+    model["current"] = version
+
+    registry_path.write_text(yaml.dump(registry, default_flow_style=False))
+```
+
+Never promote by hand-editing registry.yaml. Always go through this gate.
+
+## Stage 7: Inference Envelope
+
+Every prediction must return a full envelope. Callers receive lineage and uncertainty, not just a score.
+
+```python
+from datetime import datetime, timezone
+from dataclasses import dataclass, asdict
+from typing import List, Optional
+
+@dataclass
+class PredictionEnvelope:
+    prediction: float
+    confidence: float
+    prediction_set: List[str]         # Conformal prediction set
+    lineage: dict
+
+def predict(model, pipeline, mapie, X, model_meta: dict) -> PredictionEnvelope:
+    X_transformed = pipeline.transform(X)
+    proba = model.predict_proba(X_transformed)[:, 1]
+
+    # Conformal prediction set at 90% coverage
+    _, y_set = mapie.predict(X_transformed, alpha=0.1)
+
+    return PredictionEnvelope(
+        prediction=float(proba[0]),
+        confidence=float(max(proba[0], 1 - proba[0])),
+        prediction_set=[str(c) for c in y_set[0]],
+        lineage={
+            "model_name": model_meta["name"],
+            "model_version": model_meta["version"],
+            "build_hash": model_meta["build_hash"],
+            "timestamp": datetime.now(timezone.utc).isoformat(),
+        }
+    )
+```
+
+Never return a bare float from inference. The envelope is non-negotiable.
+
+## Stage 8: Circuit Breaker
+
+Wrap all inference calls in a circuit breaker. When the model degrades, return safe defaults — not errors.
+
+```python
+import time
+from enum import Enum
+from collections import deque
+
+class CircuitState(Enum):
+    CLOSED = "closed"       # Normal operation
+    OPEN = "open"           # Failing, return defaults
+    HALF_OPEN = "half_open" # Testing recovery
+
+class CircuitBreaker:
+    def __init__(self, error_threshold=0.05, latency_p95_ms=500, cooldown_s=60):
+        self.state = CircuitState.CLOSED
+        self.error_threshold = error_threshold
+        self.latency_p95_ms = latency_p95_ms
+        self.cooldown_s = cooldown_s
+        self.errors = deque(maxlen=100)
+        self.latencies = deque(maxlen=100)
+        self.opened_at: Optional[float] = None
+
+    def call(self, fn, *args, safe_default=None, **kwargs):
+        if self.state == CircuitState.OPEN:
+            if time.time() - self.opened_at > self.cooldown_s:
+                self.state = CircuitState.HALF_OPEN
+            else:
+                return safe_default
+
+        start = time.time()
+        try:
+            result = fn(*args, **kwargs)
+            self.errors.append(0)
+            self.latencies.append((time.time() - start) * 1000)
+            if self.state == CircuitState.HALF_OPEN:
+                self.state = CircuitState.CLOSED
+            self._check_thresholds()
+            return result
+        except Exception as e:
+            self.errors.append(1)
+            if self.state == CircuitState.HALF_OPEN:
+                self._open()
+            self._check_thresholds()
+            raise
+
+    def _check_thresholds(self):
+        if len(self.errors) < 20:
+            return
+        error_rate = sum(self.errors) / len(self.errors)
+        p95_lat = sorted(self.latencies)[int(len(self.latencies) * 0.95)]
+        if error_rate > self.error_threshold or p95_lat > self.latency_p95_ms:
+            self._open()
+
+    def _open(self):
+        self.state = CircuitState.OPEN
+        self.opened_at = time.time()
+```
+
+```
+State transitions:
+CLOSED → (error_rate > 5% OR p95_latency > 500ms) → OPEN
+OPEN → (cooldown 60s elapsed) → HALF_OPEN
+HALF_OPEN → (success) → CLOSED
+HALF_OPEN → (failure) → OPEN
+
+When OPEN: return safe_default, log alert, never raise to caller
+```
+
+## Stage 9: Drift Detection
+
+Run drift checks on a schedule or on each inference batch. Alert immediately, do not silently degrade.
+
+```python
+from scipy.stats import ks_2samp
+import pandas as pd
+
+def check_drift(train_df: pd.DataFrame, prod_df: pd.DataFrame, features: list, alpha=0.05) -> list:
+    alerts = []
+    for feature in features:
+        if feature not in train_df.columns or feature not in prod_df.columns:
+            continue
+        stat, pvalue = ks_2samp(
+            train_df[feature].dropna(),
+            prod_df[feature].dropna()
+        )
+        if pvalue < alpha:
+            alerts.append({
+                "feature": feature,
+                "ks_stat": round(stat, 4),
+                "pvalue": round(pvalue, 6),
+                "severity": "high" if pvalue < 0.001 else "medium",
+            })
+    return alerts
+
+def log_drift_alerts(alerts: list, experiment_id: str):
+    if not alerts:
+        return
+    path = Path(f".ctx/ml/experiments/{experiment_id}/artifacts/drift_alerts.json")
+    path.write_text(json.dumps(alerts, indent=2))
+    print(f"[DRIFT] {len(alerts)} feature(s) drifted — see {path}")
+```
+
+## Stage 10: Conformal Prediction
+
+All classifiers must support conformal prediction sets. Use MAPIE.
+
+```python
+from mapie.classification import MapieClassifier
+
+def fit_conformal(model, X_train, y_train) -> MapieClassifier:
+    mapie = MapieClassifier(estimator=model, method="score", cv=5)
+    mapie.fit(X_train, y_train)
+    return mapie
+
+def predict_with_coverage(mapie, X, alpha=0.1):
+    # alpha=0.1 → 90% marginal coverage guaranteed
+    y_pred, y_set = mapie.predict(X, alpha=alpha)
+    return y_pred, y_set
+```
+
+Coverage is a contract, not a suggestion. If the prediction set is too wide, the model needs retraining — not a tighter alpha.
+
+## CI/CT/CD for ML
+
+```
+On pull request:
+  1. Validate schema (pandera)
+  2. Run feature pipeline (determinism check)
+  3. Train on sample data (smoke test)
+  4. Evaluate on holdout (metric regression gate)
+  5. Drift check on validation set
+
+On merge to main:
+  1. Full training run
+  2. Full evaluation
+  3. Registry promotion check
+  4. Deploy if criteria met
+
+On scheduled cadence (weekly):
+  1. Drift check on production traffic
+  2. Calibration check
+  3. Trigger retraining if drift thresholds exceeded
+```
+
+## Reproducibility Requirements
+
+Every artifact directory must contain:
+
+```
+artifacts/
+├── model.pkl               # Serialized model
+├── pipeline.pkl            # Fitted feature pipeline
+├── mapie.pkl               # Fitted conformal wrapper
+├── config.yaml             # Exact config used for this run
+├── metrics.json            # All evaluation metrics
+├── train.log               # Full training log
+├── plots/
+│   ├── roc_curve.png
+│   ├── calibration.png
+│   └── feature_importance.png
+└── hpo_study.pkl           # If HPO was run
+```
+
+No artifact is complete without all of these files.
+
+## Pipeline Wiring Rules
+
+1. Feature pipeline is always fit on training data only. Validation and test data are transformed, never used to fit.
+2. Conformal calibration uses a dedicated calibration split, not val or test.
+3. Circuit breaker wraps the full inference stack, not just the model call.
+4. Drift detection compares production distribution against training distribution. Never compare to previous week.
+5. Promotion criteria are code, not conversation. No manual overrides.

package/skills/ctx-orchestrator/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: ctx-orchestrator
+description: |
+  WHEN: User wants to run a multi-step workflow like plan→execute→verify, asks for "ctx pipeline", "ctx next", "ctx auto", or wants autonomous story execution.
+  WHEN NOT: Single agent invocation, simple questions, or non-CTX work.
+---
+
+# CTX Orchestrator — Pipeline & Lifecycle Execution
+
+You orchestrate multi-agent workflows by spawning CTX agents via the Agent tool.
+
+## Phase Lifecycle
+
+CTX follows a strict phase progression:
+
+```
+init → plan → execute → verify → complete
+                ↑          ↓
+                ←── (fix failures)
+```
+
+## How to Advance Phases
+
+Read `.ctx/STATE.json` to determine current phase, then spawn the right agent:
+
+| Current Phase | Action | Agent to Spawn |
+|---------------|--------|----------------|
+| init | Plan the story | `subagent_type: "ctx-planner"` |
+| plan | Execute the plan | `subagent_type: "ctx-executor"` |
+| execute | Verify acceptance criteria | `subagent_type: "ctx-verifier"` |
+| verify (pass) | Mark complete | Update STATE.json + PRD.json |
+| verify (fail) | Fix and retry | `subagent_type: "ctx-executor"` with failure context |
+
+## Pipeline Execution
+
+When the user requests a pipeline (e.g., "plan, execute, verify"):
+
+1. Read `.ctx/STATE.json` for current state
+2. For each step in the pipeline:
+   a. Spawn the agent using the Agent tool with the appropriate `subagent_type`
+   b. Wait for completion
+   c. Update `.ctx/STATE.json` with the result
+   d. Pass output summary as context to the next agent
+3. If any step fails, halt and report
+
+Example agent spawn:
+```
+Agent({
+  subagent_type: "ctx-planner",
+  prompt: "Plan story S001: <title>. Acceptance criteria: <list>. Write plan to .ctx/phases/S001/PLAN.md",
+  description: "Plan story S001"
+})
+```
+
+## Autonomous Mode
+
+When user requests "ctx auto" or autonomous execution:
+
+1. Read `.ctx/PRD.json` for pending stories (sorted by priority)
+2. For each story:
+   a. Set as active in STATE.json
+   b. Run pipeline: plan → execute → verify
+   c. If verify passes: mark story passed, commit, continue to next
+   d. If verify fails: retry up to 3 times, then skip and log failure
+3. Check for `.ctx/STOP` file before each story (graceful halt)
+4. Write summary to `.ctx/AUTO-LOG.md`
+
+## State Management
+
+Always update `.ctx/STATE.json` after phase transitions:
+
+```json
+{
+  "version": "4.0",
+  "phase": "<current phase>",
+  "activeStory": "<story ID>",
+  "storyTitle": "<title>",
+  "completedTasks": [],
+  "agentHistory": [
+    { "agent": "<name>", "invokedAt": "<ISO>", "completedAt": "<ISO>", "taskSummary": "<text>" }
+  ]
+}
+```
+
+## Rules
+
+- NEVER skip phases. init→plan→execute→verify is mandatory.
+- ALWAYS spawn agents via `Agent` tool with `subagent_type` — never shell out to `claude` CLI.
+- ALWAYS update STATE.json after each phase transition.
+- Max 3 retries per story before skipping.
+- Check `.ctx/STOP` file before starting each story in auto mode.

package/skills/ctx-review-gate/SKILL.md

@@ -0,0 +1,147 @@
+---
+name: ctx-review-gate
+description: |
+  WHEN: Code has been implemented and needs quality verification before marking a story complete. Runs three-stage review: spec compliance, code quality, and optional cross-model adversarial review via OpenAI Codex.
+  WHEN NOT: During planning, research, or when review gate is disabled in config.
+---
+
+# CTX Three-Stage Review Gate
+
+Automated quality gate that runs after execution and before verification.
+
+## Three Stages
+
+### Stage 1: Spec Compliance (ctx-reviewer)
+Checks whether the code satisfies the story's acceptance criteria.
+
+Spawn:
+```
+Agent({
+  subagent_type: "ctx-reviewer",
+  prompt: "Review recent changes for SPEC COMPLIANCE against story <ID> acceptance criteria: <list>. Output VERDICT: PASS or FAIL with ISSUES list.",
+  description: "Spec compliance review"
+})
+```
+
+### Stage 2: Code Quality (ctx-reviewer)
+Reuses ctx-reviewer with a quality-focused prompt: security, performance, error handling, style. **Only runs if Stage 1 passes.**
+
+(Note: earlier versions of this skill called `ctx-auditor` here. That was a miscast — `ctx-auditor` is an audit-trail/compliance agent, not a code-quality reviewer. `ctx-reviewer` already covers type checks, imports, security scans, and best-practice enforcement, so it handles both stages with different framings.)
+
+Spawn:
+```
+Agent({
+  subagent_type: "ctx-reviewer",
+  prompt: "Review recent changes for CODE QUALITY. Check: security vulnerabilities, performance, error handling, style. Output VERDICT: PASS or FAIL with ISSUES list.",
+  description: "Code quality review"
+})
+```
+
+### Stage 3: Cross-Model Review (ctx-codex-reviewer) — optional
+Sends the diff to OpenAI Codex via MCP for a second-pair-of-eyes review with different model priors. **Only runs if Stage 2 passes AND `config.codexReview !== false`.**
+
+Short-circuits on docs-only, test-only, or trivial (<20 LOC) diffs. Fails soft — if the Codex MCP is unavailable, rate-limited, or unauthenticated, returns `SKIP` rather than `FAIL` so infrastructure problems never block the gate.
+
+Spawn:
+```
+Agent({
+  subagent_type: "ctx-codex-reviewer",
+  prompt: "Cross-model review story <ID>. Dispatch the current diff to Codex via mcp__codex__codex with sandbox=read-only. Acceptance criteria: <list>. Output VERDICT: PASS, FAIL, or SKIP.",
+  description: "Codex adversarial review"
+})
+```
+
+Prerequisites (user-side, not automated by CTX):
+- Codex CLI installed (`npm i -g @openai/codex`)
+- Signed in via ChatGPT subscription (`codex login` — no `--api-key` flag)
+- MCP registered (`claude mcp add codex -- codex mcp-server`)
+
+## Flow
+
+```
+executor completes
+    │
+    ▼
+Stage 1: ctx-reviewer (spec compliance)
+    │
+    ├── FAIL → Feed issues back to executor, increment cycle
+    │
+    ▼ PASS
+Stage 2: ctx-auditor (code quality)
+    │
+    ├── FAIL → Feed issues back to executor, increment cycle
+    │
+    ▼ PASS
+Stage 3: ctx-codex-reviewer (cross-model, if enabled)
+    │
+    ├── FAIL → Feed issues back to executor, increment cycle
+    ├── SKIP → Treat as pass (infra problem, not code problem)
+    │
+    ▼ PASS
+Mark story for verification
+```
+
+## Retry Logic
+
+- Max 3 review cycles per story
+- Each cycle: execute → review → audit
+- If cycle 3 fails: **ESCALATE** — halt and ask human for review
+- Record each cycle in `.ctx/STATE.json` under `reviewGate.history`
+
+## Review Result Format
+
+Expect agents to output:
+```
+VERDICT: PASS
+```
+or:
+```
+VERDICT: FAIL
+ISSUES:
+- Missing error handling in auth middleware
+- No test for edge case X
+```
+
+Parse the VERDICT line to determine pass/fail. Extract ISSUES for feedback.
+
+## State Tracking
+
+Update `.ctx/STATE.json`:
+```json
+{
+  "reviewGate": {
+    "cycle": 2,
+    "history": [
+      { "cycle": 1, "timestamp": "ISO", "stage1": { "passed": true }, "stage2": { "passed": false, "issues": "..." }, "stage3": null, "result": "fail" },
+      { "cycle": 2, "timestamp": "ISO", "stage1": { "passed": true }, "stage2": { "passed": true }, "stage3": { "passed": true, "threadId": "thr_...", "skipped": false }, "result": "pass" }
+    ]
+  }
+}
+```
+
+`stage3` is `null` when Stage 2 fails (not reached) or when `codexReview` is disabled. When Stage 3 runs, record `threadId` so follow-ups reuse the same Codex session.
+
+## Save Review Artifacts
+
+Write review results to `.ctx/reviews/<story-id>-<timestamp>.json`.
+
+## Configuration
+
+Review gate can be disabled entirely:
+- Check `.ctx/config.json` for `"reviewGate": false`
+- If disabled, skip directly to verification
+
+Stage 3 (Codex cross-review) can be disabled independently:
+- Check `.ctx/config.json` for `"codexReview": false`
+- Useful when offline, when the ChatGPT rate-limit budget is depleted, or when the change is trivial
+- Stages 1 and 2 continue to run normally
+
+## Rules
+
+- ALWAYS run Stage 1 before Stage 2, Stage 2 before Stage 3 (fail-fast ordering)
+- NEVER run Stage 2 if Stage 1 fails
+- NEVER run Stage 3 if Stage 2 fails, or if `codexReview === false`
+- Stage 3 SKIP (infrastructure failure) is NOT a gate failure — treat as pass
+- ALWAYS feed review issues back to executor as context on retry
+- Max 3 cycles — then escalate to human
+- Record every cycle in state, including `stage3: null` when not reached