@zigrivers/scaffold 3.14.0 → 3.16.0

package/content/knowledge/research/research-architecture.md

@@ -0,0 +1,385 @@
+---
+name: research-architecture
+description: Experiment runner architecture including pluggable experiment and evaluation interfaces, state management patterns, and result persistence
+topics: [research, architecture, experiment-runner, state-management, interfaces, persistence]
+---
+
+The experiment runner is the central architectural component of a research project. It orchestrates the loop of loading configuration, executing experiments, evaluating results, and deciding whether to keep or discard each run. The runner must be completely decoupled from the specific experiment logic (strategies, models, parameter spaces) so that it can drive any experiment without modification. This separation is what makes autonomous iteration possible -- the agent modifies experiment code while the runner infrastructure remains stable.
+
+## Summary
+
+Build the experiment runner around three pluggable interfaces: Strategy (executes an experiment given config), Evaluator (computes metrics from raw results), and Tracker (records results for comparison). Use a state manager to track the current best result, iteration history, and budget consumption. Persist all state to disk so that the runner can resume after crashes. The runner never imports specific strategy code -- it discovers strategies via a registry or config-specified entry point.
+
+## Deep Guidance
+
+### Core Architecture
+
+```
+                    ┌──────────────────────┐
+                    │   ExperimentRunner    │
+                    │  ┌────────────────┐  │
+  Config ──────────►│  │ State Manager  │  │
+                    │  │ (best, history)│  │
+                    │  └───────┬────────┘  │
+                    │          │            │
+                    │  ┌───────▼────────┐  │
+                    │  │ Budget Checker │  │
+                    │  └───────┬────────┘  │
+                    │          │            │
+                    │  ┌───────▼────────┐  │
+                    │  │   Strategy     │◄─┼── Registry lookup
+                    │  │  (pluggable)   │  │
+                    │  └───────┬────────┘  │
+                    │          │            │
+                    │  ┌───────▼────────┐  │
+                    │  │   Evaluator    │  │
+                    │  │  (pluggable)   │  │
+                    │  └───────┬────────┘  │
+                    │          │            │
+                    │  ┌───────▼────────┐  │
+                    │  │   Tracker      │  │
+                    │  │  (pluggable)   │  │
+                    │  └────────────────┘  │
+                    └──────────────────────┘
+```
+
+### Pluggable Interface Design
+
+The three core interfaces use Python's Protocol type for structural subtyping. This means strategies do not need to inherit from a base class -- they only need to implement the required methods:
+
+```python
+# src/interfaces.py
+from typing import Protocol, Any, runtime_checkable
+
+@runtime_checkable
+class Strategy(Protocol):
+    """Interface for experiment execution strategies."""
+
+    @property
+    def name(self) -> str:
+        """Unique identifier for this strategy."""
+        ...
+
+    def execute(self, config: dict[str, Any]) -> dict[str, Any]:
+        """
+        Execute the experiment and return raw results.
+
+        Args:
+            config: Experiment configuration dict.
+
+        Returns:
+            Raw results dict. Structure is strategy-specific but must
+            contain enough information for the Evaluator to compute metrics.
+        """
+        ...
+
+@runtime_checkable
+class Evaluator(Protocol):
+    """Interface for result evaluation."""
+
+    def evaluate(self, raw_results: dict[str, Any]) -> dict[str, float]:
+        """
+        Compute metrics from raw experiment results.
+
+        Args:
+            raw_results: Output from Strategy.execute().
+
+        Returns:
+            Dict mapping metric names to float values.
+        """
+        ...
+
+    def is_improvement(self, current: dict[str, float],
+                        best: dict[str, float]) -> bool:
+        """
+        Determine if current results improve on the best so far.
+
+        Args:
+            current: Metrics from the current run.
+            best: Metrics from the best run so far.
+
+        Returns:
+            True if current should replace best.
+        """
+        ...
+
+@runtime_checkable
+class Tracker(Protocol):
+    """Interface for experiment result tracking."""
+
+    def log_run(self, run_id: str, config: dict, metrics: dict[str, float],
+                artifacts: dict[str, Any] | None = None) -> None:
+        """Record a single experiment run."""
+        ...
+
+    def get_history(self) -> list[dict]:
+        """Return all recorded runs."""
+        ...
+```
+
+### Strategy Registry
+
+The registry pattern allows the runner to instantiate strategies by name without importing them directly:
+
+```python
+# src/strategies/registry.py
+from typing import Type
+from src.interfaces import Strategy
+
+class StrategyRegistry:
+    """Registry for experiment strategy classes."""
+
+    _registry: dict[str, Type[Strategy]] = {}
+
+    @classmethod
+    def register(cls, name: str):
+        """Decorator to register a strategy class."""
+        def decorator(strategy_cls: Type[Strategy]):
+            if name in cls._registry:
+                raise ValueError(f"Strategy '{name}' already registered")
+            cls._registry[name] = strategy_cls
+            return strategy_cls
+        return decorator
+
+    @classmethod
+    def get(cls, name: str) -> Type[Strategy]:
+        """Look up a strategy by name."""
+        if name not in cls._registry:
+            available = ", ".join(sorted(cls._registry.keys()))
+            raise KeyError(
+                f"Strategy '{name}' not found. Available: {available}"
+            )
+        return cls._registry[name]
+
+    @classmethod
+    def list_strategies(cls) -> list[str]:
+        return sorted(cls._registry.keys())
+
+
+# Usage in a strategy file:
+# src/strategies/momentum.py
+from src.strategies.registry import StrategyRegistry
+
+@StrategyRegistry.register("momentum_crossover")
+class MomentumCrossover:
+    name = "momentum_crossover"
+
+    def __init__(self, lookback: int = 20, **kwargs):
+        self.lookback = lookback
+
+    def execute(self, config: dict) -> dict:
+        # ... run the momentum crossover strategy ...
+        return {"trades": trades, "equity_curve": equity}
+```
+
+### State Management
+
+The state manager tracks the experiment loop's progress and enables resume-after-crash:
+
+```python
+# src/runner/state.py
+import json
+from pathlib import Path
+from dataclasses import dataclass, field, asdict
+from typing import Any
+
+@dataclass
+class RunRecord:
+    """Record of a single experiment run."""
+    run_id: str
+    config: dict[str, Any]
+    metrics: dict[str, float]
+    is_best: bool = False
+    decision: str = ""  # "keep" or "discard"
+    reason: str = ""
+
+@dataclass
+class ExperimentState:
+    """Persistent state for the experiment loop."""
+    experiment_id: str
+    total_runs: int = 0
+    best_run: RunRecord | None = None
+    history: list[RunRecord] = field(default_factory=list)
+    runs_since_improvement: int = 0
+
+    def record_run(self, run: RunRecord) -> None:
+        """Record a completed run and update state."""
+        self.total_runs += 1
+        self.history.append(run)
+
+        if run.is_best:
+            self.best_run = run
+            self.runs_since_improvement = 0
+        else:
+            self.runs_since_improvement += 1
+
+    def save(self, path: Path) -> None:
+        """Persist state to disk for crash recovery."""
+        path.parent.mkdir(parents=True, exist_ok=True)
+        with open(path, "w") as f:
+            json.dump(asdict(self), f, indent=2, default=str)
+
+    @classmethod
+    def load(cls, path: Path) -> "ExperimentState":
+        """Load state from disk. Returns empty state if file missing."""
+        if not path.exists():
+            return cls(experiment_id="unknown")
+        with open(path) as f:
+            data = json.load(f)
+        state = cls(experiment_id=data["experiment_id"])
+        state.total_runs = data["total_runs"]
+        state.runs_since_improvement = data["runs_since_improvement"]
+        state.history = [RunRecord(**r) for r in data["history"]]
+        if data["best_run"]:
+            state.best_run = RunRecord(**data["best_run"])
+        return state
+```
+
+### The Experiment Runner
+
+The runner ties the interfaces together:
+
+```python
+# src/runner/experiment_runner.py
+import logging
+from pathlib import Path
+from src.interfaces import Strategy, Evaluator, Tracker
+from src.runner.state import ExperimentState, RunRecord
+from src.runner.budget import IterationBudget
+from src.config import load_config
+from src.seed import set_seed, capture_environment
+from src.strategies.registry import StrategyRegistry
+
+logger = logging.getLogger(__name__)
+
+class ExperimentRunner:
+    def __init__(self, config_path: str):
+        self.config = load_config(config_path)
+        self.experiment_id = Path(config_path).stem
+        self.results_dir = Path(self.config["logging"]["results_dir"]) / self.experiment_id
+
+        # Load pluggable components
+        strategy_cls = StrategyRegistry.get(self.config["strategy"]["type"])
+        self.strategy: Strategy = strategy_cls(**self.config["strategy"].get("params", {}))
+        self.evaluator: Evaluator = self._build_evaluator()
+        self.tracker: Tracker = self._build_tracker()
+        self.budget = IterationBudget(**self.config.get("budget", {}))
+
+        # Load or initialize state
+        self.state_path = self.results_dir / "state.json"
+        self.state = ExperimentState.load(self.state_path)
+        self.state.experiment_id = self.experiment_id
+
+    def run_loop(self) -> ExperimentState:
+        """Run the full experiment loop until budget exhaustion or convergence."""
+        logger.info("Starting experiment %s (resuming from run %d)",
+                     self.experiment_id, self.state.total_runs)
+
+        while True:
+            # Check budget
+            exhausted, reason = self.budget.is_exhausted(
+                runs=self.state.total_runs,
+                runs_since_improvement=self.state.runs_since_improvement,
+            )
+            if exhausted:
+                logger.info("Stopping: %s", reason)
+                break
+
+            # Execute one iteration
+            run_id = f"run-{self.state.total_runs + 1:04d}"
+            set_seed(self.config["experiment"]["seed"] + self.state.total_runs)
+
+            try:
+                raw_results = self.strategy.execute(self.config)
+                metrics = self.evaluator.evaluate(raw_results)
+            except Exception as e:
+                logger.error("Run %s failed: %s", run_id, e)
+                continue
+
+            # Evaluate improvement
+            is_best = (
+                self.state.best_run is None
+                or self.evaluator.is_improvement(metrics, self.state.best_run.metrics)
+            )
+            decision = "keep" if is_best else "discard"
+
+            run = RunRecord(
+                run_id=run_id,
+                config=self.config,
+                metrics=metrics,
+                is_best=is_best,
+                decision=decision,
+                reason=f"{'New best' if is_best else 'No improvement'}",
+            )
+
+            # Record and persist
+            self.state.record_run(run)
+            self.tracker.log_run(run_id, self.config, metrics)
+            self.state.save(self.state_path)
+
+            logger.info(
+                "Run %s: %s (metrics: %s)",
+                run_id, decision,
+                {k: f"{v:.4f}" for k, v in metrics.items()},
+            )
+
+        return self.state
+```
+
+### Result Persistence
+
+Results are persisted at two levels:
+
+1. **Per-run**: Each run's config, metrics, and artifacts are saved to `results/{experiment_id}/{run_id}/`.
+2. **Experiment state**: The full experiment state (history, best run, budget consumption) is saved to `results/{experiment_id}/state.json` after every run.
+
+```python
+# src/tracking/file_tracker.py
+import json
+from pathlib import Path
+from src.interfaces import Tracker
+
+class FileTracker:
+    """Simple file-based experiment tracker."""
+
+    def __init__(self, results_dir: str):
+        self.results_dir = Path(results_dir)
+        self.results_dir.mkdir(parents=True, exist_ok=True)
+
+    def log_run(self, run_id: str, config: dict, metrics: dict[str, float],
+                artifacts: dict | None = None) -> None:
+        run_dir = self.results_dir / run_id
+        run_dir.mkdir(parents=True, exist_ok=True)
+
+        with open(run_dir / "config.json", "w") as f:
+            json.dump(config, f, indent=2, default=str)
+        with open(run_dir / "metrics.json", "w") as f:
+            json.dump(metrics, f, indent=2)
+
+        if artifacts:
+            artifact_dir = run_dir / "artifacts"
+            artifact_dir.mkdir(exist_ok=True)
+            for name, data in artifacts.items():
+                with open(artifact_dir / name, "w") as f:
+                    json.dump(data, f, indent=2, default=str)
+
+    def get_history(self) -> list[dict]:
+        runs = []
+        for run_dir in sorted(self.results_dir.iterdir()):
+            if run_dir.is_dir() and (run_dir / "metrics.json").exists():
+                with open(run_dir / "metrics.json") as f:
+                    metrics = json.load(f)
+                runs.append({"run_id": run_dir.name, "metrics": metrics})
+        return runs
+```
+
+### Architecture Decision: When to Use Each Driver
+
+| Driver | Architecture Pattern | Use When |
+|--------|---------------------|----------|
+| Code-driven | Git state machine, agent modifies source | Exploring algorithmic variations, strategy development |
+| Config-driven | Fixed runner, parameterised configs | Hyperparameter sweeps, systematic parameter search |
+| API-driven | Client wrapper, parameter serialization | External backtest engines, cloud simulation APIs |
+| Notebook-driven | Papermill execution, cell-level tracking | Exploratory research, visualization-heavy analysis |
+
+The runner architecture remains the same across all drivers. What changes is the Strategy implementation: code-driven strategies contain the algorithm directly, config-driven strategies delegate to a parameterised engine, API-driven strategies wrap HTTP calls, and notebook-driven strategies use papermill to execute notebooks.

package/content/knowledge/research/research-conventions.md

@@ -0,0 +1,248 @@
+---
+name: research-conventions
+description: Coding conventions for research projects including experiment branching, result naming, config management, and reproducibility standards
+topics: [research, conventions, git, branching, reproducibility, config-management]
+---
+
+Research code has a unique lifecycle: most code is written to be tried and discarded. A trading strategy that underperforms is reverted. A hyperparameter sweep that converges to a local minimum is abandoned. The conventions must make this try-and-discard cycle fast and safe while preserving a complete audit trail of what was tried and why it was kept or discarded.
+
+## Summary
+
+Use git branches as the state machine for experiment lifecycle (try, evaluate, keep/revert). Name branches, results, and configs with a consistent scheme that encodes the experiment ID, hypothesis, and timestamp. Pin every dependency and seed every random source for reproducibility. Separate experiment code (disposable) from infrastructure code (durable) in the repository structure. Use structured config files (YAML/TOML) instead of command-line argument sprawl.
+
+## Deep Guidance
+
+### Git as Experiment State Machine
+
+The experiment loop uses git as its state management layer. Each experiment run is a branch. The decision to keep or discard is a merge or branch deletion:
+
+```
+main (stable baseline)
+  |
+  +-- exp/001-momentum-lookback-20  (try → evaluate → keep → merge)
+  |
+  +-- exp/002-momentum-lookback-10  (try → evaluate → discard → delete)
+  |
+  +-- exp/003-mean-revert-rsi       (try → evaluate → keep → merge)
+```
+
+**Branch naming convention**: `exp/{NNN}-{short-description}`
+- `NNN`: Zero-padded sequential experiment number
+- `short-description`: Kebab-case summary of what is being tested
+- Examples: `exp/001-adaptive-lookback`, `exp/042-ensemble-top3`
+
+**Workflow**:
+```bash
+# Start a new experiment
+git checkout main
+git checkout -b exp/015-rsi-threshold-sweep
+
+# ... agent modifies code, runs experiment ...
+
+# Experiment succeeded — merge to main
+git checkout main
+git merge --no-ff exp/015-rsi-threshold-sweep -m "exp/015: RSI threshold 30/70 Sharpe=1.6"
+
+# Experiment failed — discard
+git branch -D exp/015-rsi-threshold-sweep
+# Or keep for reference:
+git tag archive/exp/015-rsi-threshold-sweep exp/015-rsi-threshold-sweep
+git branch -D exp/015-rsi-threshold-sweep
+```
+
+**Commit message convention for experiments**:
+```
+exp/015: RSI threshold sweep
+
+Hypothesis: RSI overbought/oversold thresholds of 30/70 will outperform
+the default 20/80 on 2020-2023 equity data.
+
+Result: Sharpe=1.6, MaxDD=11%, 247 trades
+Decision: KEEP — new best by Sharpe, DD within guardrail
+```
+
+### Result Naming
+
+Every experiment run produces artifacts. Use a consistent naming scheme:
+
+```
+results/
+  exp-001/
+    config.yml          # Exact config used for this run
+    metrics.json        # Final metrics
+    metrics_history.csv # Per-iteration metrics
+    artifacts/          # Model checkpoints, plots, etc.
+    log.txt             # Full stdout/stderr
+  exp-002/
+    ...
+```
+
+**File naming rules**:
+- Directories: `exp-{NNN}` matching the git branch number
+- Timestamps in filenames when multiple runs share an experiment: `exp-001-20240315T143022`
+- Never use spaces or special characters in result paths
+- Metrics files are always JSON (machine-readable) or CSV (tabular)
+
+### Config Management
+
+Research projects accumulate dozens of configuration parameters. Manage them with structured config files, not argument sprawl:
+
+```yaml
+# configs/base.yml — shared defaults
+experiment:
+  seed: 42
+  num_runs: 100
+  patience: 20
+
+data:
+  source: "data/prices.parquet"
+  train_start: "2015-01-01"
+  train_end: "2019-12-31"
+  test_start: "2020-01-01"
+  test_end: "2023-12-31"
+
+logging:
+  level: INFO
+  results_dir: "results"
+```
+
+```yaml
+# configs/exp-015-rsi-sweep.yml — experiment-specific overrides
+_base_: base.yml
+
+strategy:
+  type: "rsi_threshold"
+  params:
+    overbought: 70
+    oversold: 30
+    lookback: 14
+
+experiment:
+  num_runs: 200  # Override base
+```
+
+**Config loading pattern** (merge base + override):
+
+```python
+# src/config.py
+import yaml
+from pathlib import Path
+from typing import Any
+
+def load_config(config_path: str) -> dict[str, Any]:
+    """Load config with base inheritance."""
+    with open(config_path) as f:
+        config = yaml.safe_load(f)
+
+    # Resolve base config inheritance
+    if "_base_" in config:
+        base_path = Path(config_path).parent / config.pop("_base_")
+        base = load_config(str(base_path))
+        base = deep_merge(base, config)
+        return base
+
+    return config
+
+def deep_merge(base: dict, override: dict) -> dict:
+    """Recursively merge override into base."""
+    result = base.copy()
+    for key, value in override.items():
+        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
+            result[key] = deep_merge(result[key], value)
+        else:
+            result[key] = value
+    return result
+```
+
+### Reproducibility Standards
+
+Every experiment must be reproducible. This means another researcher (or the same agent in a future session) can re-run the experiment and get the same result:
+
+**Mandatory reproducibility checklist**:
+
+1. **Seed everything**: Random number generators, data shuffling, model initialization.
+   ```python
+   import random
+   import numpy as np
+
+   def set_seed(seed: int) -> None:
+       random.seed(seed)
+       np.random.seed(seed)
+       # Framework-specific seeding
+       try:
+           import torch
+           torch.manual_seed(seed)
+           torch.cuda.manual_seed_all(seed)
+           torch.backends.cudnn.deterministic = True
+           torch.backends.cudnn.benchmark = False
+       except ImportError:
+           pass
+   ```
+
+2. **Pin dependencies**: Use exact versions, not ranges.
+   ```
+   # requirements.txt — pinned
+   numpy==1.26.4
+   pandas==2.2.1
+   scikit-learn==1.4.1
+   optuna==3.5.0
+   ```
+
+3. **Record environment**: Capture the full environment at experiment start.
+   ```python
+   import subprocess
+   import platform
+   import json
+
+   def capture_environment() -> dict:
+       return {
+           "python": platform.python_version(),
+           "platform": platform.platform(),
+           "pip_freeze": subprocess.check_output(
+               ["pip", "freeze"], text=True
+           ).strip().split("\n"),
+           "git_sha": subprocess.check_output(
+               ["git", "rev-parse", "HEAD"], text=True
+           ).strip(),
+           "git_dirty": bool(subprocess.check_output(
+               ["git", "status", "--porcelain"], text=True
+           ).strip()),
+       }
+   ```
+
+4. **Never modify data in place**: Raw data is immutable. Processed data is derived and can be regenerated from raw data + processing code.
+
+5. **Config-as-code**: The experiment config file (committed to git) must fully define the experiment. No "I changed that parameter manually."
+
+### Code Organization Conventions
+
+Separate durable infrastructure code from disposable experiment code:
+
+| Category | Location | Lifecycle |
+|----------|----------|-----------|
+| Experiment runner | `src/runner/` | Durable — rarely changes |
+| Evaluation framework | `src/evaluation/` | Durable — rarely changes |
+| Data loading | `src/data/` | Durable — rarely changes |
+| Strategy/model code | `src/strategies/` or `src/models/` | Disposable — changes every experiment |
+| Config files | `configs/` | Per-experiment |
+| Results | `results/` | Per-experiment output |
+
+**Import hygiene**: Experiment code imports from infrastructure code, never the reverse. The runner does not import specific strategies -- it discovers them via a registry or config-specified entry point.
+
+### Code Style for Research
+
+- **Type hints everywhere**: Even in experiment code. Catches bugs early in a fast-iteration cycle.
+- **Docstrings on public functions**: Especially for metric computation (document the formula).
+- **No notebooks in git**: Notebooks are for interactive exploration. Convert to scripts before committing. If notebook-driven experiments are required, use `nbstripout` to strip outputs before committing.
+- **Linting**: Use `ruff` for fast linting. Research code skips some style rules (unused imports during exploration) but enforces correctness rules (undefined variables, type errors).
+
+```toml
+# pyproject.toml
+[tool.ruff]
+line-length = 100
+select = ["E", "F", "W", "I"]  # Errors, pyflakes, warnings, isort
+ignore = ["E501"]  # Allow long lines in research code
+
+[tool.ruff.lint.isort]
+known-first-party = ["src"]
+```