@zigrivers/scaffold 3.14.0 → 3.16.0

package/content/knowledge/research/research-dev-environment.md

@@ -0,0 +1,303 @@
+---
+name: research-dev-environment
+description: Development tooling for research projects including virtual environments, dependency management, GPU setup, and data access configuration
+topics: [research, dev-environment, dependencies, virtual-env, gpu, data-access, tooling]
+---
+
+Research dev environments have stricter reproducibility requirements than typical application development. A trading strategy that produces different results on a different machine is useless -- the environment itself is a variable that must be controlled. At the same time, research environments need flexibility for rapid iteration: installing new packages, switching between CPU and GPU, and accessing large datasets must be frictionless.
+
+## Summary
+
+Use `uv` (preferred) or `pip` with pinned dependencies in a virtual environment for reproducible Python dependency management. Lock the full dependency tree (not just direct dependencies). Configure GPU access and CUDA versions explicitly when applicable. Set up data access credentials via environment variables (never in code or config files). Use a Makefile with standard targets (`setup`, `run`, `test`) so that both humans and agents can operate the environment identically.
+
+## Deep Guidance
+
+### Python Environment Setup
+
+**Recommended: `uv` for dependency management**
+
+`uv` is the fastest Python package manager and provides deterministic resolution with a lockfile:
+
+```bash
+# Install uv
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a new project
+uv init research-project
+cd research-project
+
+# Add dependencies
+uv add numpy pandas scikit-learn optuna
+uv add --dev pytest ruff mypy
+
+# The lockfile (uv.lock) is auto-generated and pinned
+# Commit both pyproject.toml and uv.lock
+```
+
+```toml
+# pyproject.toml
+[project]
+name = "research-project"
+version = "0.1.0"
+requires-python = ">=3.11"
+dependencies = [
+    "numpy>=1.26",
+    "pandas>=2.2",
+    "scikit-learn>=1.4",
+    "optuna>=3.5",
+    "pyyaml>=6.0",
+    "structlog>=24.1",
+]
+
+[project.optional-dependencies]
+gpu = ["torch>=2.2"]
+tracking = ["mlflow>=2.11"]
+notebooks = ["jupyter>=1.0", "papermill>=2.5"]
+
+[tool.uv]
+dev-dependencies = [
+    "pytest>=8.0",
+    "ruff>=0.3",
+    "mypy>=1.9",
+]
+```
+
+**Alternative: `pip` with `requirements.txt`**
+
+If `uv` is not available, use `pip` with fully pinned requirements:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+```
+# requirements.txt — fully pinned (generated by pip freeze)
+numpy==1.26.4
+pandas==2.2.1
+scikit-learn==1.4.1.post1
+optuna==3.5.0
+PyYAML==6.0.1
+structlog==24.1.0
+```
+
+**Alternative: `conda` for complex native dependencies**
+
+Use conda when the project requires system-level native libraries (CUDA toolkit, MKL, OpenBLAS) that pip cannot manage:
+
+```yaml
+# environment.yml
+name: research
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.11
+  - numpy=1.26
+  - pandas=2.2
+  - scikit-learn=1.4
+  - cudatoolkit=12.1  # Native dependency
+  - pip:
+    - optuna==3.5.0   # pip packages within conda env
+```
+
+### GPU Configuration
+
+For research projects that use GPU acceleration (ML model training, simulation, etc.):
+
+```python
+# src/gpu.py
+import os
+import logging
+
+logger = logging.getLogger(__name__)
+
+def configure_gpu(config: dict) -> str:
+    """Configure GPU access. Returns device string."""
+    if not config.get("gpu", {}).get("enabled", False):
+        logger.info("GPU disabled by config, using CPU")
+        return "cpu"
+
+    # Restrict visible GPUs (useful for multi-GPU machines)
+    gpu_ids = config.get("gpu", {}).get("device_ids", [0])
+    os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(str(i) for i in gpu_ids)
+
+    try:
+        import torch
+        if torch.cuda.is_available():
+            device = f"cuda:{gpu_ids[0]}"
+            logger.info(
+                "GPU configured: %s (%s, %.1f GB)",
+                device,
+                torch.cuda.get_device_name(0),
+                torch.cuda.get_device_properties(0).total_mem / 1e9,
+            )
+            return device
+        else:
+            logger.warning("CUDA not available, falling back to CPU")
+            return "cpu"
+    except ImportError:
+        logger.warning("PyTorch not installed, using CPU")
+        return "cpu"
+```
+
+**GPU config in YAML**:
+```yaml
+gpu:
+  enabled: true
+  device_ids: [0]
+  memory_fraction: 0.8  # Limit GPU memory usage
+```
+
+### Data Access Configuration
+
+Data credentials are managed via environment variables, never committed to git:
+
+```bash
+# .env (gitignored)
+DATA_SOURCE_PATH=/mnt/data/research
+DATABASE_URL=postgresql://user:pass@host:5432/research
+AWS_PROFILE=research-data
+POLYGON_API_KEY=pk_xxx  # Market data API
+```
+
+```python
+# src/data/credentials.py
+import os
+from dataclasses import dataclass
+
+@dataclass
+class DataCredentials:
+    """Data access credentials loaded from environment."""
+    data_path: str
+    database_url: str | None = None
+    api_key: str | None = None
+
+    @classmethod
+    def from_env(cls) -> "DataCredentials":
+        data_path = os.environ.get("DATA_SOURCE_PATH", "data/raw")
+        if not os.path.exists(data_path):
+            raise EnvironmentError(
+                f"DATA_SOURCE_PATH={data_path} does not exist. "
+                "Set DATA_SOURCE_PATH to your data directory."
+            )
+        return cls(
+            data_path=data_path,
+            database_url=os.environ.get("DATABASE_URL"),
+            api_key=os.environ.get("POLYGON_API_KEY"),
+        )
+```
+
+### Makefile for Environment Management
+
+```makefile
+.PHONY: setup run test lint clean
+
+PYTHON ?= python3
+UV := $(shell command -v uv 2>/dev/null)
+
+setup: ## Set up development environment
+ifdef UV
+	uv sync
+	uv sync --group dev
+else
+	$(PYTHON) -m venv .venv
+	.venv/bin/pip install -r requirements.txt
+	.venv/bin/pip install -r requirements-dev.txt
+endif
+	@echo "Environment ready. Activate with: source .venv/bin/activate"
+
+setup-gpu: setup ## Set up with GPU dependencies
+ifdef UV
+	uv sync --extra gpu
+else
+	.venv/bin/pip install -r requirements-gpu.txt
+endif
+
+run: ## Run experiment (usage: make run CONFIG=configs/exp-001.yml)
+	$(PYTHON) -m src.runner.experiment_runner --config $(CONFIG)
+
+test: ## Run test suite
+	$(PYTHON) -m pytest tests/ -v --tb=short
+
+lint: ## Lint and type-check
+	ruff check src/ tests/
+	mypy src/ --ignore-missing-imports
+
+clean: ## Clean generated artifacts
+	rm -rf .venv/ __pycache__/ .mypy_cache/ .pytest_cache/
+	find . -name '*.pyc' -delete
+```
+
+### IDE Configuration
+
+**VS Code** (`.vscode/settings.json`):
+```json
+{
+    "python.defaultInterpreterPath": ".venv/bin/python",
+    "python.analysis.typeCheckingMode": "basic",
+    "editor.formatOnSave": true,
+    "[python]": {
+        "editor.defaultFormatter": "charliermarsh.ruff"
+    },
+    "python.testing.pytestEnabled": true,
+    "python.testing.pytestArgs": ["tests/"]
+}
+```
+
+### Environment Verification Script
+
+Run this at the start of every experiment to verify the environment:
+
+```python
+# scripts/verify_env.py
+"""Verify that the research environment is correctly configured."""
+import sys
+import importlib
+
+REQUIRED_PACKAGES = [
+    "numpy", "pandas", "sklearn", "optuna", "yaml", "structlog",
+]
+
+def verify():
+    errors = []
+
+    # Python version
+    if sys.version_info < (3, 11):
+        errors.append(f"Python >= 3.11 required, got {sys.version}")
+
+    # Required packages
+    for pkg in REQUIRED_PACKAGES:
+        try:
+            importlib.import_module(pkg)
+        except ImportError:
+            errors.append(f"Missing required package: {pkg}")
+
+    # Data access
+    import os
+    data_path = os.environ.get("DATA_SOURCE_PATH", "data/raw")
+    if not os.path.exists(data_path):
+        errors.append(f"Data path not found: {data_path}")
+
+    if errors:
+        print("Environment verification FAILED:")
+        for e in errors:
+            print(f"  - {e}")
+        sys.exit(1)
+    else:
+        print("Environment verification passed.")
+
+if __name__ == "__main__":
+    verify()
+```
+
+### Dependency Update Strategy
+
+Research projects should update dependencies cautiously:
+
+1. **Lock everything**: Both direct and transitive dependencies are pinned.
+2. **Update on a schedule**: Not every commit. Weekly or per-milestone.
+3. **Test after update**: Run the full test suite after any dependency change.
+4. **Document breaking changes**: If a dependency update changes experiment results, document which runs were affected.
+5. **Separate experiment deps from infra deps**: Changing the plotting library should not affect experiment reproducibility. Use optional dependency groups.