@zigrivers/scaffold 3.13.0 → 3.15.0

package/content/knowledge/research/research-conventions.md

@@ -0,0 +1,248 @@
+---
+name: research-conventions
+description: Coding conventions for research projects including experiment branching, result naming, config management, and reproducibility standards
+topics: [research, conventions, git, branching, reproducibility, config-management]
+---
+
+Research code has a unique lifecycle: most code is written to be tried and discarded. A trading strategy that underperforms is reverted. A hyperparameter sweep that converges to a local minimum is abandoned. The conventions must make this try-and-discard cycle fast and safe while preserving a complete audit trail of what was tried and why it was kept or discarded.
+
+## Summary
+
+Use git branches as the state machine for experiment lifecycle (try, evaluate, keep/revert). Name branches, results, and configs with a consistent scheme that encodes the experiment ID, hypothesis, and timestamp. Pin every dependency and seed every random source for reproducibility. Separate experiment code (disposable) from infrastructure code (durable) in the repository structure. Use structured config files (YAML/TOML) instead of command-line argument sprawl.
+
+## Deep Guidance
+
+### Git as Experiment State Machine
+
+The experiment loop uses git as its state management layer. Each experiment run is a branch. The decision to keep or discard is a merge or branch deletion:
+
+```
+main (stable baseline)
+  |
+  +-- exp/001-momentum-lookback-20  (try → evaluate → keep → merge)
+  |
+  +-- exp/002-momentum-lookback-10  (try → evaluate → discard → delete)
+  |
+  +-- exp/003-mean-revert-rsi       (try → evaluate → keep → merge)
+```
+
+**Branch naming convention**: `exp/{NNN}-{short-description}`
+- `NNN`: Zero-padded sequential experiment number
+- `short-description`: Kebab-case summary of what is being tested
+- Examples: `exp/001-adaptive-lookback`, `exp/042-ensemble-top3`
+
+**Workflow**:
+```bash
+# Start a new experiment
+git checkout main
+git checkout -b exp/015-rsi-threshold-sweep
+
+# ... agent modifies code, runs experiment ...
+
+# Experiment succeeded — merge to main
+git checkout main
+git merge --no-ff exp/015-rsi-threshold-sweep -m "exp/015: RSI threshold 30/70 Sharpe=1.6"
+
+# Experiment failed — discard
+git branch -D exp/015-rsi-threshold-sweep
+# Or keep for reference:
+git tag archive/exp/015-rsi-threshold-sweep exp/015-rsi-threshold-sweep
+git branch -D exp/015-rsi-threshold-sweep
+```
+
+**Commit message convention for experiments**:
+```
+exp/015: RSI threshold sweep
+
+Hypothesis: RSI overbought/oversold thresholds of 30/70 will outperform
+the default 20/80 on 2020-2023 equity data.
+
+Result: Sharpe=1.6, MaxDD=11%, 247 trades
+Decision: KEEP — new best by Sharpe, DD within guardrail
+```
+
+### Result Naming
+
+Every experiment run produces artifacts. Use a consistent naming scheme:
+
+```
+results/
+  exp-001/
+    config.yml          # Exact config used for this run
+    metrics.json        # Final metrics
+    metrics_history.csv # Per-iteration metrics
+    artifacts/          # Model checkpoints, plots, etc.
+    log.txt             # Full stdout/stderr
+  exp-002/
+    ...
+```
+
+**File naming rules**:
+- Directories: `exp-{NNN}` matching the git branch number
+- Timestamps in filenames when multiple runs share an experiment: `exp-001-20240315T143022`
+- Never use spaces or special characters in result paths
+- Metrics files are always JSON (machine-readable) or CSV (tabular)
+
+### Config Management
+
+Research projects accumulate dozens of configuration parameters. Manage them with structured config files, not argument sprawl:
+
+```yaml
+# configs/base.yml — shared defaults
+experiment:
+  seed: 42
+  num_runs: 100
+  patience: 20
+
+data:
+  source: "data/prices.parquet"
+  train_start: "2015-01-01"
+  train_end: "2019-12-31"
+  test_start: "2020-01-01"
+  test_end: "2023-12-31"
+
+logging:
+  level: INFO
+  results_dir: "results"
+```
+
+```yaml
+# configs/exp-015-rsi-sweep.yml — experiment-specific overrides
+_base_: base.yml
+
+strategy:
+  type: "rsi_threshold"
+  params:
+    overbought: 70
+    oversold: 30
+    lookback: 14
+
+experiment:
+  num_runs: 200  # Override base
+```
+
+**Config loading pattern** (merge base + override):
+
+```python
+# src/config.py
+import yaml
+from pathlib import Path
+from typing import Any
+
+def load_config(config_path: str) -> dict[str, Any]:
+    """Load config with base inheritance."""
+    with open(config_path) as f:
+        config = yaml.safe_load(f)
+
+    # Resolve base config inheritance
+    if "_base_" in config:
+        base_path = Path(config_path).parent / config.pop("_base_")
+        base = load_config(str(base_path))
+        base = deep_merge(base, config)
+        return base
+
+    return config
+
+def deep_merge(base: dict, override: dict) -> dict:
+    """Recursively merge override into base."""
+    result = base.copy()
+    for key, value in override.items():
+        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
+            result[key] = deep_merge(result[key], value)
+        else:
+            result[key] = value
+    return result
+```
+
+### Reproducibility Standards
+
+Every experiment must be reproducible. This means another researcher (or the same agent in a future session) can re-run the experiment and get the same result:
+
+**Mandatory reproducibility checklist**:
+
+1. **Seed everything**: Random number generators, data shuffling, model initialization.
+   ```python
+   import random
+   import numpy as np
+
+   def set_seed(seed: int) -> None:
+       random.seed(seed)
+       np.random.seed(seed)
+       # Framework-specific seeding
+       try:
+           import torch
+           torch.manual_seed(seed)
+           torch.cuda.manual_seed_all(seed)
+           torch.backends.cudnn.deterministic = True
+           torch.backends.cudnn.benchmark = False
+       except ImportError:
+           pass
+   ```
+
+2. **Pin dependencies**: Use exact versions, not ranges.
+   ```
+   # requirements.txt — pinned
+   numpy==1.26.4
+   pandas==2.2.1
+   scikit-learn==1.4.1
+   optuna==3.5.0
+   ```
+
+3. **Record environment**: Capture the full environment at experiment start.
+   ```python
+   import subprocess
+   import platform
+   import json
+
+   def capture_environment() -> dict:
+       return {
+           "python": platform.python_version(),
+           "platform": platform.platform(),
+           "pip_freeze": subprocess.check_output(
+               ["pip", "freeze"], text=True
+           ).strip().split("\n"),
+           "git_sha": subprocess.check_output(
+               ["git", "rev-parse", "HEAD"], text=True
+           ).strip(),
+           "git_dirty": bool(subprocess.check_output(
+               ["git", "status", "--porcelain"], text=True
+           ).strip()),
+       }
+   ```
+
+4. **Never modify data in place**: Raw data is immutable. Processed data is derived and can be regenerated from raw data + processing code.
+
+5. **Config-as-code**: The experiment config file (committed to git) must fully define the experiment. No "I changed that parameter manually."
+
+### Code Organization Conventions
+
+Separate durable infrastructure code from disposable experiment code:
+
+| Category | Location | Lifecycle |
+|----------|----------|-----------|
+| Experiment runner | `src/runner/` | Durable — rarely changes |
+| Evaluation framework | `src/evaluation/` | Durable — rarely changes |
+| Data loading | `src/data/` | Durable — rarely changes |
+| Strategy/model code | `src/strategies/` or `src/models/` | Disposable — changes every experiment |
+| Config files | `configs/` | Per-experiment |
+| Results | `results/` | Per-experiment output |
+
+**Import hygiene**: Experiment code imports from infrastructure code, never the reverse. The runner does not import specific strategies -- it discovers them via a registry or config-specified entry point.
+
+### Code Style for Research
+
+- **Type hints everywhere**: Even in experiment code. Catches bugs early in a fast-iteration cycle.
+- **Docstrings on public functions**: Especially for metric computation (document the formula).
+- **No notebooks in git**: Notebooks are for interactive exploration. Convert to scripts before committing. If notebook-driven experiments are required, use `nbstripout` to strip outputs before committing.
+- **Linting**: Use `ruff` for fast linting. Research code skips some style rules (unused imports during exploration) but enforces correctness rules (undefined variables, type errors).
+
+```toml
+# pyproject.toml
+[tool.ruff]
+line-length = 100
+select = ["E", "F", "W", "I"]  # Errors, pyflakes, warnings, isort
+ignore = ["E501"]  # Allow long lines in research code
+
+[tool.ruff.lint.isort]
+known-first-party = ["src"]
+```

package/content/knowledge/research/research-dev-environment.md

@@ -0,0 +1,303 @@
+---
+name: research-dev-environment
+description: Development tooling for research projects including virtual environments, dependency management, GPU setup, and data access configuration
+topics: [research, dev-environment, dependencies, virtual-env, gpu, data-access, tooling]
+---
+
+Research dev environments have stricter reproducibility requirements than typical application development. A trading strategy that produces different results on a different machine is useless -- the environment itself is a variable that must be controlled. At the same time, research environments need flexibility for rapid iteration: installing new packages, switching between CPU and GPU, and accessing large datasets must be frictionless.
+
+## Summary
+
+Use `uv` (preferred) or `pip` with pinned dependencies in a virtual environment for reproducible Python dependency management. Lock the full dependency tree (not just direct dependencies). Configure GPU access and CUDA versions explicitly when applicable. Set up data access credentials via environment variables (never in code or config files). Use a Makefile with standard targets (`setup`, `run`, `test`) so that both humans and agents can operate the environment identically.
+
+## Deep Guidance
+
+### Python Environment Setup
+
+**Recommended: `uv` for dependency management**
+
+`uv` is the fastest Python package manager and provides deterministic resolution with a lockfile:
+
+```bash
+# Install uv
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a new project
+uv init research-project
+cd research-project
+
+# Add dependencies
+uv add numpy pandas scikit-learn optuna
+uv add --dev pytest ruff mypy
+
+# The lockfile (uv.lock) is auto-generated and pinned
+# Commit both pyproject.toml and uv.lock
+```
+
+```toml
+# pyproject.toml
+[project]
+name = "research-project"
+version = "0.1.0"
+requires-python = ">=3.11"
+dependencies = [
+    "numpy>=1.26",
+    "pandas>=2.2",
+    "scikit-learn>=1.4",
+    "optuna>=3.5",
+    "pyyaml>=6.0",
+    "structlog>=24.1",
+]
+
+[project.optional-dependencies]
+gpu = ["torch>=2.2"]
+tracking = ["mlflow>=2.11"]
+notebooks = ["jupyter>=1.0", "papermill>=2.5"]
+
+[tool.uv]
+dev-dependencies = [
+    "pytest>=8.0",
+    "ruff>=0.3",
+    "mypy>=1.9",
+]
+```
+
+**Alternative: `pip` with `requirements.txt`**
+
+If `uv` is not available, use `pip` with fully pinned requirements:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+```
+# requirements.txt — fully pinned (generated by pip freeze)
+numpy==1.26.4
+pandas==2.2.1
+scikit-learn==1.4.1.post1
+optuna==3.5.0
+PyYAML==6.0.1
+structlog==24.1.0
+```
+
+**Alternative: `conda` for complex native dependencies**
+
+Use conda when the project requires system-level native libraries (CUDA toolkit, MKL, OpenBLAS) that pip cannot manage:
+
+```yaml
+# environment.yml
+name: research
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.11
+  - numpy=1.26
+  - pandas=2.2
+  - scikit-learn=1.4
+  - cudatoolkit=12.1  # Native dependency
+  - pip:
+    - optuna==3.5.0   # pip packages within conda env
+```
+
+### GPU Configuration
+
+For research projects that use GPU acceleration (ML model training, simulation, etc.):
+
+```python
+# src/gpu.py
+import os
+import logging
+
+logger = logging.getLogger(__name__)
+
+def configure_gpu(config: dict) -> str:
+    """Configure GPU access. Returns device string."""
+    if not config.get("gpu", {}).get("enabled", False):
+        logger.info("GPU disabled by config, using CPU")
+        return "cpu"
+
+    # Restrict visible GPUs (useful for multi-GPU machines)
+    gpu_ids = config.get("gpu", {}).get("device_ids", [0])
+    os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(str(i) for i in gpu_ids)
+
+    try:
+        import torch
+        if torch.cuda.is_available():
+            device = f"cuda:{gpu_ids[0]}"
+            logger.info(
+                "GPU configured: %s (%s, %.1f GB)",
+                device,
+                torch.cuda.get_device_name(0),
+                torch.cuda.get_device_properties(0).total_mem / 1e9,
+            )
+            return device
+        else:
+            logger.warning("CUDA not available, falling back to CPU")
+            return "cpu"
+    except ImportError:
+        logger.warning("PyTorch not installed, using CPU")
+        return "cpu"
+```
+
+**GPU config in YAML**:
+```yaml
+gpu:
+  enabled: true
+  device_ids: [0]
+  memory_fraction: 0.8  # Limit GPU memory usage
+```
+
+### Data Access Configuration
+
+Data credentials are managed via environment variables, never committed to git:
+
+```bash
+# .env (gitignored)
+DATA_SOURCE_PATH=/mnt/data/research
+DATABASE_URL=postgresql://user:pass@host:5432/research
+AWS_PROFILE=research-data
+POLYGON_API_KEY=pk_xxx  # Market data API
+```
+
+```python
+# src/data/credentials.py
+import os
+from dataclasses import dataclass
+
+@dataclass
+class DataCredentials:
+    """Data access credentials loaded from environment."""
+    data_path: str
+    database_url: str | None = None
+    api_key: str | None = None
+
+    @classmethod
+    def from_env(cls) -> "DataCredentials":
+        data_path = os.environ.get("DATA_SOURCE_PATH", "data/raw")
+        if not os.path.exists(data_path):
+            raise EnvironmentError(
+                f"DATA_SOURCE_PATH={data_path} does not exist. "
+                "Set DATA_SOURCE_PATH to your data directory."
+            )
+        return cls(
+            data_path=data_path,
+            database_url=os.environ.get("DATABASE_URL"),
+            api_key=os.environ.get("POLYGON_API_KEY"),
+        )
+```
+
+### Makefile for Environment Management
+
+```makefile
+.PHONY: setup run test lint clean
+
+PYTHON ?= python3
+UV := $(shell command -v uv 2>/dev/null)
+
+setup: ## Set up development environment
+ifdef UV
+	uv sync
+	uv sync --group dev
+else
+	$(PYTHON) -m venv .venv
+	.venv/bin/pip install -r requirements.txt
+	.venv/bin/pip install -r requirements-dev.txt
+endif
+	@echo "Environment ready. Activate with: source .venv/bin/activate"
+
+setup-gpu: setup ## Set up with GPU dependencies
+ifdef UV
+	uv sync --extra gpu
+else
+	.venv/bin/pip install -r requirements-gpu.txt
+endif
+
+run: ## Run experiment (usage: make run CONFIG=configs/exp-001.yml)
+	$(PYTHON) -m src.runner.experiment_runner --config $(CONFIG)
+
+test: ## Run test suite
+	$(PYTHON) -m pytest tests/ -v --tb=short
+
+lint: ## Lint and type-check
+	ruff check src/ tests/
+	mypy src/ --ignore-missing-imports
+
+clean: ## Clean generated artifacts
+	rm -rf .venv/ __pycache__/ .mypy_cache/ .pytest_cache/
+	find . -name '*.pyc' -delete
+```
+
+### IDE Configuration
+
+**VS Code** (`.vscode/settings.json`):
+```json
+{
+    "python.defaultInterpreterPath": ".venv/bin/python",
+    "python.analysis.typeCheckingMode": "basic",
+    "editor.formatOnSave": true,
+    "[python]": {
+        "editor.defaultFormatter": "charliermarsh.ruff"
+    },
+    "python.testing.pytestEnabled": true,
+    "python.testing.pytestArgs": ["tests/"]
+}
+```
+
+### Environment Verification Script
+
+Run this at the start of every experiment to verify the environment:
+
+```python
+# scripts/verify_env.py
+"""Verify that the research environment is correctly configured."""
+import sys
+import importlib
+
+REQUIRED_PACKAGES = [
+    "numpy", "pandas", "sklearn", "optuna", "yaml", "structlog",
+]
+
+def verify():
+    errors = []
+
+    # Python version
+    if sys.version_info < (3, 11):
+        errors.append(f"Python >= 3.11 required, got {sys.version}")
+
+    # Required packages
+    for pkg in REQUIRED_PACKAGES:
+        try:
+            importlib.import_module(pkg)
+        except ImportError:
+            errors.append(f"Missing required package: {pkg}")
+
+    # Data access
+    import os
+    data_path = os.environ.get("DATA_SOURCE_PATH", "data/raw")
+    if not os.path.exists(data_path):
+        errors.append(f"Data path not found: {data_path}")
+
+    if errors:
+        print("Environment verification FAILED:")
+        for e in errors:
+            print(f"  - {e}")
+        sys.exit(1)
+    else:
+        print("Environment verification passed.")
+
+if __name__ == "__main__":
+    verify()
+```
+
+### Dependency Update Strategy
+
+Research projects should update dependencies cautiously:
+
+1. **Lock everything**: Both direct and transitive dependencies are pinned.
+2. **Update on a schedule**: Not every commit. Weekly or per-milestone.
+3. **Test after update**: Run the full test suite after any dependency change.
+4. **Document breaking changes**: If a dependency update changes experiment results, document which runs were affected.
+5. **Separate experiment deps from infra deps**: Changing the plotting library should not affect experiment reproducibility. Use optional dependency groups.