@synsci/cli-darwin-x64 1.1.49

package/bin/skills/nemo-evaluator/references/custom-benchmarks.md

@@ -0,0 +1,315 @@
+# Custom Benchmark Integration
+
+NeMo Evaluator supports adding custom benchmarks through Framework Definition Files (FDFs) and custom containers.
+
+## Overview
+
+Custom benchmarks are added by:
+
+1. **Framework Definition Files (FDFs)**: YAML files that define evaluation tasks, commands, and output parsing
+2. **Custom Containers**: Package your framework with nemo-evaluator for reproducible execution
+
+> **Note**: NeMo Evaluator does not currently support programmatic harness APIs or custom metric implementations via Python classes. Customization is done through FDFs and containers.
+
+## Framework Definition Files (FDFs)
+
+FDFs are the primary way to add custom evaluations. An FDF declares framework metadata, default commands, and evaluation tasks.
+
+### FDF Structure
+
+```yaml
+# framework_def.yaml
+framework:
+  name: my-custom-framework
+  package_name: my_custom_eval
+
+defaults:
+  command: "python -m my_custom_eval.run --model-id {model_id} --task {task} --output-dir {output_dir}"
+
+evaluations:
+  - name: custom_task_1
+    defaults:
+      temperature: 0.0
+      max_new_tokens: 512
+      extra:
+        custom_param: value
+
+  - name: custom_task_2
+    defaults:
+      temperature: 0.7
+      max_new_tokens: 1024
+```
+
+### Key FDF Components
+
+**Framework section**:
+- `name`: Human-readable name for your framework
+- `package_name`: Python package name
+
+**Defaults section**:
+- `command`: The command template to execute your evaluation
+- Placeholders: `{model_id}`, `{task}`, `{output_dir}` are substituted at runtime
+
+**Evaluations section**:
+- List of tasks with their default parameters
+- Each task can override the framework defaults
+
+### Output Parser
+
+When creating a custom FDF, you need an output parser function that translates your framework's results into NeMo Evaluator's standard schema:
+
+```python
+# my_custom_eval/parser.py
+def parse_output(output_dir: str) -> dict:
+    """
+    Parse evaluation results from output_dir.
+
+    Returns dict with metrics in NeMo Evaluator format.
+    """
+    # Read your framework's output files
+    results_file = Path(output_dir) / "results.json"
+    with open(results_file) as f:
+        raw_results = json.load(f)
+
+    # Transform to standard schema
+    return {
+        "metrics": {
+            "accuracy": raw_results["score"],
+            "total_samples": raw_results["num_samples"]
+        }
+    }
+```
+
+## Custom Container Creation
+
+Package your custom framework as a container for reproducibility.
+
+### Dockerfile Example
+
+```dockerfile
+# Dockerfile
+FROM python:3.10-slim
+
+# Install nemo-evaluator
+RUN pip install nemo-evaluator
+
+# Install your custom framework
+COPY my_custom_eval/ /opt/my_custom_eval/
+RUN pip install /opt/my_custom_eval/
+
+# Copy framework definition
+COPY framework_def.yaml /opt/framework_def.yaml
+
+# Set working directory
+WORKDIR /opt
+
+ENTRYPOINT ["python", "-m", "nemo_evaluator"]
+```
+
+### Build and Push
+
+```bash
+docker build -t my-registry/custom-eval:1.0 .
+docker push my-registry/custom-eval:1.0
+```
+
+### Register in mapping.toml
+
+Add your custom container to the task registry:
+
+```toml
+# Add to mapping.toml
+[my-custom-framework]
+container = "my-registry/custom-eval:1.0"
+
+[my-custom-framework.tasks.chat.custom_task_1]
+required_env_vars = []
+
+[my-custom-framework.tasks.chat.custom_task_2]
+required_env_vars = ["CUSTOM_API_KEY"]
+```
+
+## Using Custom Datasets
+
+### Dataset Mounting
+
+Mount proprietary datasets at runtime rather than baking them into containers:
+
+```yaml
+# config.yaml
+defaults:
+  - execution: local
+  - deployment: none
+  - _self_
+
+execution:
+  output_dir: ./results
+
+evaluation:
+  tasks:
+    - name: custom_task_1
+      dataset_dir: /path/to/local/data
+      dataset_mount_path: /data  # Optional, defaults to /datasets
+```
+
+The launcher will mount the dataset directory into the container and set `NEMO_EVALUATOR_DATASET_DIR` environment variable.
+
+### Task-Specific Environment Variables
+
+Pass environment variables to specific tasks:
+
+```yaml
+evaluation:
+  tasks:
+    - name: gpqa_diamond
+      env_vars:
+        HF_TOKEN: HF_TOKEN  # Maps to $HF_TOKEN from host
+
+    - name: custom_task
+      env_vars:
+        CUSTOM_API_KEY: MY_CUSTOM_KEY
+        DATA_PATH: /data/custom.jsonl
+```
+
+## Parameter Overrides
+
+Override evaluation parameters at multiple levels:
+
+### Global Overrides
+
+Apply to all tasks:
+
+```yaml
+evaluation:
+  nemo_evaluator_config:
+    config:
+      params:
+        temperature: 0.0
+        max_new_tokens: 512
+        parallelism: 4
+        request_timeout: 300
+```
+
+### Task-Specific Overrides
+
+Override for individual tasks:
+
+```yaml
+evaluation:
+  tasks:
+    - name: humaneval
+      nemo_evaluator_config:
+        config:
+          params:
+            temperature: 0.8
+            max_new_tokens: 1024
+            n_samples: 200  # Task-specific parameter
+```
+
+### CLI Overrides
+
+Override at runtime:
+
+```bash
+nemo-evaluator-launcher run \
+  --config-dir . \
+  --config-name config \
+  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=10
+```
+
+## Testing Custom Benchmarks
+
+### Dry Run
+
+Validate configuration without execution:
+
+```bash
+nemo-evaluator-launcher run \
+  --config-dir . \
+  --config-name custom_config \
+  --dry-run
+```
+
+### Limited Sample Testing
+
+Test with a small subset first:
+
+```bash
+nemo-evaluator-launcher run \
+  --config-dir . \
+  --config-name custom_config \
+  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=5
+```
+
+### Check Results
+
+```bash
+# View results
+cat results/<invocation_id>/<task>/artifacts/results.json
+
+# Check logs
+cat results/<invocation_id>/<task>/artifacts/logs/eval.log
+```
+
+## Best Practices
+
+1. **Use FDFs**: Define custom benchmarks via Framework Definition Files
+2. **Containerize**: Package frameworks as containers for reproducibility
+3. **Mount data**: Use volume mounts for datasets instead of baking into images
+4. **Test incrementally**: Use `limit_samples` for quick validation
+5. **Version containers**: Tag containers with semantic versions
+6. **Document parameters**: Include clear documentation in your FDF
+
+## Limitations
+
+Currently **not supported**:
+- Custom Python metric classes via plugin system
+- Programmatic harness registration via Python API
+- Runtime metric injection via configuration
+
+Custom scoring logic must be implemented within your evaluation framework and exposed through the FDF's output parser.
+
+## Example: Complete Custom Setup
+
+```yaml
+# custom_eval_config.yaml
+defaults:
+  - execution: local
+  - deployment: none
+  - _self_
+
+execution:
+  output_dir: ./custom_results
+
+target:
+  api_endpoint:
+    model_id: my-model
+    url: http://localhost:8000/v1/chat/completions
+    api_key_name: ""
+
+evaluation:
+  nemo_evaluator_config:
+    config:
+      params:
+        parallelism: 4
+        request_timeout: 300
+
+  tasks:
+    - name: custom_task_1
+      dataset_dir: /data/benchmarks
+      env_vars:
+        DATA_VERSION: v2
+      nemo_evaluator_config:
+        config:
+          params:
+            temperature: 0.0
+            max_new_tokens: 256
+```
+
+Run with:
+
+```bash
+nemo-evaluator-launcher run \
+  --config-dir . \
+  --config-name custom_eval_config
+```

package/bin/skills/nemo-evaluator/references/execution-backends.md

@@ -0,0 +1,361 @@
+# Execution Backends
+
+NeMo Evaluator supports three execution backends: Local (Docker), Slurm (HPC), and Lepton (Cloud). Each backend implements the same interface but has different configuration requirements.
+
+## Backend Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    nemo-evaluator-launcher                   │
+│                                                              │
+│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
+│  │ LocalExecutor │  │ SlurmExecutor │  │ LeptonExecutor│     │
+│  │   (Docker)    │  │   (SSH+sbatch)│  │  (Cloud API)  │     │
+│  └──────────────┘  └──────────────┘  └──────────────┘       │
+│           │                │                 │               │
+└───────────┼────────────────┼─────────────────┼───────────────┘
+            │                │                 │
+            ▼                ▼                 ▼
+       ┌─────────┐    ┌───────────┐    ┌────────────┐
+       │ Docker  │    │  Slurm    │    │  Lepton AI │
+       │ Engine  │    │  Cluster  │    │  Platform  │
+       └─────────┘    └───────────┘    └────────────┘
+```
+
+## Local Executor (Docker)
+
+The local executor runs evaluation containers on your local machine using Docker.
+
+### Prerequisites
+
+- Docker installed and running
+- `docker` command available in PATH
+- GPU drivers and nvidia-container-toolkit for GPU tasks
+
+### Configuration
+
+```yaml
+defaults:
+  - execution: local
+  - deployment: none
+  - _self_
+
+execution:
+  output_dir: ./results
+  mode: sequential  # or parallel
+
+  # Docker-specific options
+  docker_args:
+    - "--gpus=all"
+    - "--shm-size=16g"
+
+  # Container resource limits
+  memory_limit: "64g"
+  cpus: 8
+```
+
+### How It Works
+
+1. Launcher reads `mapping.toml` to find container image for task
+2. Creates run configuration and mounts volumes
+3. Executes `docker run` via subprocess
+4. Monitors stage files (`stage.pre-start`, `stage.running`, `stage.exit`)
+5. Collects results from mounted output directory
+
+### Example Usage
+
+```bash
+# Simple local evaluation
+nemo-evaluator-launcher run \
+  --config-dir . \
+  --config-name local_config
+
+# With GPU allocation
+nemo-evaluator-launcher run \
+  --config-dir . \
+  --config-name local_config \
+  -o 'execution.docker_args=["--gpus=all"]'
+```
+
+### Status Tracking
+
+Status is tracked via file markers in the output directory:
+
+| File | Meaning |
+|------|---------|
+| `stage.pre-start` | Container starting |
+| `stage.running` | Evaluation in progress |
+| `stage.exit` | Evaluation complete |
+
+## Slurm Executor
+
+The Slurm executor submits evaluation jobs to HPC clusters via SSH.
+
+### Prerequisites
+
+- SSH access to cluster head node
+- Slurm commands available (`sbatch`, `squeue`, `sacct`)
+- NGC containers accessible from compute nodes
+- Shared filesystem for results
+
+### Configuration
+
+```yaml
+defaults:
+  - execution: slurm
+  - deployment: vllm  # or sglang, nim, none
+  - _self_
+
+execution:
+  # SSH connection settings
+  hostname: cluster.example.com
+  username: myuser  # Optional, uses SSH config
+  ssh_key_path: ~/.ssh/id_rsa
+
+  # Slurm job settings
+  account: my_account
+  partition: gpu
+  qos: normal
+  nodes: 1
+  gpus_per_node: 8
+  cpus_per_task: 32
+  memory: "256G"
+  walltime: "04:00:00"
+
+  # Output settings
+  output_dir: /shared/nfs/results
+
+  # Container settings
+  container_mounts:
+    - "/shared/data:/data:ro"
+    - "/shared/models:/models:ro"
+```
+
+### Deployment Options
+
+When running on Slurm, you can deploy models alongside evaluation:
+
+```yaml
+# vLLM deployment
+deployment:
+  type: vllm
+  checkpoint_path: /models/llama-3.1-8b
+  tensor_parallel_size: 4
+  max_model_len: 8192
+  gpu_memory_utilization: 0.9
+
+# SGLang deployment
+deployment:
+  type: sglang
+  checkpoint_path: /models/llama-3.1-8b
+  tensor_parallel_size: 4
+
+# NVIDIA NIM deployment
+deployment:
+  type: nim
+  nim_model_name: meta/llama-3.1-8b-instruct
+```
+
+### Job Submission Flow
+
+```
+┌─────────────────┐
+│ Launcher CLI    │
+└────────┬────────┘
+         │ SSH
+         ▼
+┌─────────────────┐
+│ Cluster Head    │
+│    Node         │
+└────────┬────────┘
+         │ sbatch
+         ▼
+┌─────────────────┐
+│ Compute Node    │
+│                 │
+│ ┌─────────────┐ │
+│ │ Deployment  │ │
+│ │ Container   │ │
+│ └─────────────┘ │
+│        │        │
+│        ▼        │
+│ ┌─────────────┐ │
+│ │ Evaluation  │ │
+│ │ Container   │ │
+│ └─────────────┘ │
+└─────────────────┘
+```
+
+### Status Queries
+
+The Slurm executor queries job status via `sacct`:
+
+```bash
+# Status command checks these Slurm states
+sacct -j <job_id> --format=JobID,State,ExitCode
+
+# Mapped to ExecutionState:
+# PENDING -> pending
+# RUNNING -> running
+# COMPLETED -> completed
+# FAILED -> failed
+# CANCELLED -> cancelled
+```
+
+### Long-Running Jobs
+
+For long-running evaluations on Slurm, consider:
+
+```yaml
+execution:
+  walltime: "24:00:00"  # Extended walltime
+  # Use caching to resume from interruptions
+
+target:
+  api_endpoint:
+    adapter_config:
+      interceptors:
+        - name: caching
+          config:
+            cache_dir: "/shared/cache"
+            reuse_cached_responses: true
+```
+
+The caching interceptor helps resume interrupted evaluations by reusing previous API responses.
+
+## Lepton Executor
+
+The Lepton executor runs evaluations on Lepton AI's cloud platform.
+
+### Prerequisites
+
+- Lepton AI account
+- `LEPTON_API_TOKEN` environment variable set
+- `leptonai` Python package (auto-installed)
+
+### Configuration
+
+```yaml
+defaults:
+  - execution: lepton
+  - deployment: none
+  - _self_
+
+execution:
+  # Lepton job settings
+  resource_shape: gpu.a100-80g
+  num_replicas: 1
+
+  # Environment
+  env_vars:
+    NGC_API_KEY: NGC_API_KEY
+    HF_TOKEN: HF_TOKEN
+```
+
+### How It Works
+
+1. Launcher creates Lepton job specification
+2. Submits job via Lepton API
+3. Optionally creates endpoint for model serving
+4. Polls job status via API
+5. Retrieves results when complete
+
+### Endpoint Management
+
+For evaluating Lepton-hosted models:
+
+```yaml
+target:
+  api_endpoint:
+    type: lepton
+    deployment_name: my-llama-deployment
+    # URL auto-generated from deployment
+```
+
+## Backend Selection Guide
+
+| Use Case | Recommended Backend |
+|----------|-------------------|
+| Quick local testing | Local |
+| Large-scale batch evaluation | Slurm |
+| CI/CD pipeline | Local or Lepton |
+| Multi-model comparison | Slurm (parallel jobs) |
+| Cloud-native workflow | Lepton |
+| Self-hosted model evaluation | Local or Slurm |
+
+## Execution Database
+
+All backends share the `ExecutionDB` for tracking jobs:
+
+```
+┌─────────────────────────────────────────────┐
+│               ExecutionDB (SQLite)           │
+│                                              │
+│  invocation_id │ job_id │ status │ backend  │
+│  ─────────────────────────────────────────  │
+│  inv_abc123    │ 12345  │ running │ slurm   │
+│  inv_def456    │ cont_1 │ done    │ local   │
+└─────────────────────────────────────────────┘
+```
+
+Query via CLI:
+
+```bash
+# List all invocations
+nemo-evaluator-launcher ls runs
+
+# Get specific invocation
+nemo-evaluator-launcher info <invocation_id>
+```
+
+## Troubleshooting
+
+### Local Executor
+
+**Issue: Docker permission denied**
+```bash
+sudo usermod -aG docker $USER
+newgrp docker
+```
+
+**Issue: GPU not available in container**
+```bash
+# Install nvidia-container-toolkit
+sudo apt-get install nvidia-container-toolkit
+sudo systemctl restart docker
+```
+
+### Slurm Executor
+
+**Issue: SSH connection fails**
+```bash
+# Test SSH connection
+ssh -v cluster.example.com
+
+# Check SSH key permissions
+chmod 600 ~/.ssh/id_rsa
+```
+
+**Issue: Job stuck in pending**
+```bash
+# Check queue status
+squeue -u $USER
+
+# Check account limits
+sacctmgr show associations user=$USER
+```
+
+### Lepton Executor
+
+**Issue: API token invalid**
+```bash
+# Verify token
+curl -H "Authorization: Bearer $LEPTON_API_TOKEN" \
+  https://api.lepton.ai/v1/jobs
+```
+
+**Issue: Resource shape unavailable**
+```bash
+# List available shapes
+lepton shape list
+```