swegen 0.1.0__py3-none-any.whl

swegen/analyze/verdict_prompt.txt

@@ -0,0 +1,126 @@
+You are synthesizing multiple independent trial analyses into a final task quality verdict.
+
+## Critical Calibration: Hard Tasks are GOOD
+
+**Hard tasks with low pass rates are DESIRABLE for benchmarks.** A 20-40% pass rate indicates a well-calibrated task. Do NOT penalize tasks for being difficult.
+
+- 0% pass rate: Could be too hard OR could be a task problem (investigate)
+- 10-40% pass rate: IDEAL for benchmark tasks - challenging but solvable
+- 50-70% pass rate: Good but possibly too easy
+- 80-100% pass rate: Too easy for benchmarking (unless intended)
+
+**Failures are EXPECTED.** The default classification for failures should be GOOD_FAILURE (agent's fault) unless there's strong evidence the task itself is broken.
+
+## Context
+
+You have {num_trials} trial runs of the same Harbor task. Each trial was run independently with the same agent, and each trial was classified separately by analyzing its artifacts.
+
+**Baseline Validation:**
+{baseline_summary}
+
+**Static Quality Check:**
+{quality_check_summary}
+
+## Individual Trial Classifications
+
+{trial_classifications}
+
+## Your Goal
+
+Synthesize these independent analyses into a **final verdict** on the task quality.
+
+Consider:
+1. **Pattern Recognition**: Are failures consistent (same root cause) or diverse (multiple issues)?
+2. **Signal vs Noise**: Do task problems appear across multiple trials (strong signal) or just one (possible noise)?
+3. **Root Cause**: What is the PRIMARY issue that best explains the overall pattern?
+4. **Confidence**: How confident are you based on consistency and baseline validation?
+5. **Actionability**: What are the most important fixes needed (if any)?
+6. **Solvability**: Did ANY trial succeed? If yes, the task IS solvable - failures are agent variance.
+7. **Over-classification check**: Is "Underspecified Instruction" just meaning "task is hard"? That's not a problem.
+
+## Classification Categories
+
+**Task problems** (need fixing):
+- BAD_FAILURE: Task specification issues (underspecified instruction, brittle tests, ambiguous requirements, etc.)
+- BAD_SUCCESS: Cheating/gaming (hardcoding, test inspection, oracle copying, tests too permissive)
+
+**Normal outcomes** (task is fine):
+- GOOD_SUCCESS: Agent solved it legitimately
+- GOOD_FAILURE: Agent couldn't solve it due to agent limitations (timeout, wrong approach, complexity, etc.)
+- HARNESS_ERROR: Infrastructure issues (not task's fault)
+
+## Confidence Levels
+
+- **high**: All trials agree OR baseline validation critical failure OR clear consistent pattern
+- **medium**: Majority of trials agree (>50%) OR mixed signals but leaning one way OR no successes but failures look legitimate
+- **low**: Trials disagree OR unclear pattern OR only harness errors
+
+## Decision Logic
+
+**Default assumption: Task is GOOD.** Only mark as bad with strong evidence.
+
+1. **Critical baseline failures** → is_good=false, high confidence
+   - If nop passed: "task may be pre-solved"
+   - If oracle failed: "reference solution doesn't work"
+
+2. **Consistent STRONG task problems** → is_good=false
+   - Majority (>50%) show the SAME BAD_FAILURE subtype with clear evidence → medium confidence
+   - All trials show task problems AND evidence is compelling → high confidence
+   - NOTE: "Underspecified Instruction" alone is often OVER-diagnosed. Scrutinize this carefully.
+
+3. **Some successes, mixed failures** → is_good=true, medium-high confidence
+   - Having ANY success means the task IS solvable
+   - Mixed failures are normal for hard tasks
+
+4. **All failures but mostly GOOD_FAILURE** → is_good=true, medium confidence
+   - Task is hard but not broken
+   - This is the IDEAL outcome for a challenging benchmark task
+
+5. **Mix of GOOD_FAILURE and scattered BAD_FAILURE** → is_good=true, medium confidence
+   - Isolated BAD_FAILURE classifications may be noise/over-classification
+   - Default to trusting the task unless pattern is clear
+
+6. **Any successes** → is_good=true, medium-high confidence
+   - Task is clearly solvable
+
+## Primary Issue Format
+
+If is_good=false, write a clear primary issue:
+- For consistent problem: "{{count}}/{{total}} trials show: {{most_common_subtype}} - {{brief_explanation}}"
+- For diverse problems: "{{count}}/{{total}} trials show task issues: {{list_of_problems}}"
+- For baseline: "CRITICAL: {{baseline_issue}}"
+
+If is_good=true:
+- null (or omit)
+
+## Recommendations
+
+For BAD tasks, provide 3-5 **specific, actionable** recommendations:
+- Prioritize issues that appeared in multiple trials
+- Focus on fixes that address the root cause
+- Be concrete (e.g., "Add file path to instruction.md line 15" not "improve instructions")
+- Deduplicate similar recommendations across trials
+
+For GOOD tasks, recommendations should be empty or minor suggestions.
+
+## Output Format
+
+Return ONLY valid JSON (no markdown, no code blocks, no explanation):
+
+{{
+  "is_good": true/false,
+  "confidence": "high" | "medium" | "low",
+  "primary_issue": "clear description of main problem" or null,
+  "recommendations": ["specific fix 1", "specific fix 2", ...],
+  "reasoning": "1-2 sentence explanation of your verdict based on the trial pattern"
+}}
+
+## Important Notes
+
+- **Default to GOOD**: Tasks that passed baseline validation are presumed good. Failures are expected.
+- **Consistency matters**: 1/4 trials showing issue is noise, 3/4 with SAME issue is a signal
+- **Baseline validation overrides**: Critical baseline failures always mean is_good=false
+- **Be skeptical of BAD_FAILURE**: "Underspecified Instruction" is often over-diagnosed. Ask: could the agent have figured this out by exploring the codebase?
+- **Low pass rates are GOOD**: 25% pass rate on a hard task is ideal for benchmarking
+- **Successes prove solvability**: Any success means the task CAN be solved with the given instruction
+

swegen/cli.py

@@ -0,0 +1,411 @@
+from __future__ import annotations
+
+from importlib.metadata import PackageNotFoundError as _PkgNotFound
+from importlib.metadata import version as _pkg_version
+from pathlib import Path
+
+import typer
+import json
+from dotenv import load_dotenv
+from harbor.models.environment_type import EnvironmentType
+from rich.console import Console
+
+from swegen.config import CreateConfig, FarmConfig
+from swegen.create import MissingIssueError, TrivialPRError
+from swegen.create.create import run_reversal
+from swegen.farm import StreamFarmer
+from swegen.analyze import AnalyzeArgs, run_analyze, TrialClassifier, write_trial_analysis_files
+from swegen.tools.validate import ValidateArgs, run_validate
+from swegen.tools.validate_utils import ValidationError
+
+load_dotenv()
+
+app = typer.Typer(no_args_is_help=True, add_completion=False, help="Task generation CLI")
+
+
+@app.callback(invoke_without_command=True)
+def _root(
+    version: bool = typer.Option(
+        False,
+        "--version",
+        "-V",
+        help="Show swegen version and exit",
+        is_eager=True,
+    ),
+) -> None:
+    if version:
+        try:
+            typer.echo(f"swegen {_pkg_version('swe-gen')}")
+        except _PkgNotFound:
+            typer.echo("swegen (version unknown)")
+        raise typer.Exit()
+
+
+create_app = typer.Typer(
+    no_args_is_help=True,
+    invoke_without_command=True,
+    add_completion=False,
+    help="Create a Harbor task from a merged PR and validate",
+)
+
+
+@create_app.callback()
+def create_cmd(
+    repo: str = typer.Option(..., help="GitHub repository (owner/repo or URL)"),
+    pr: int = typer.Option(..., help="PR number"),
+    output: Path = typer.Option(Path("tasks"), help="Output root", show_default=True),
+    cc_timeout: int = typer.Option(
+        3200, help="Timeout for CC session in seconds (~53 min default)", show_default=True
+    ),
+    validate: bool = typer.Option(
+        True, help="Run Harbor validations; --no-validate skips validation"
+    ),
+    force: bool = typer.Option(False, help="Bypass local dedupe and regenerate"),
+    state_dir: Path = typer.Option(
+        Path(".state"), help="Local dedupe state dir", show_default=True
+    ),
+    no_cache: bool = typer.Option(
+        False, "--no-cache", help="Disable reusing cached Dockerfiles/test.sh from previous tasks"
+    ),
+    require_minimum_difficulty: bool = typer.Option(
+        True,
+        help="Require minimum difficulty (3+ source files); --no-require-minimum-difficulty to skip this check",
+    ),
+    min_source_files: int = typer.Option(
+        3, help="Minimum number of source files required (tests excluded)", show_default=True
+    ),
+    max_source_files: int = typer.Option(
+        10,
+        help="Maximum number of source files to avoid large refactors (tests excluded)",
+        show_default=True,
+    ),
+    require_issue: bool = typer.Option(
+        True,
+        help="Require PR to have a linked issue (higher quality instructions); --no-require-issue uses PR body/title instead",
+    ),
+    allow_unmerged: bool = typer.Option(
+        False,
+        help="Allow processing unmerged PRs (for testing/preview); --allow-unmerged to enable",
+    ),
+    environment: str = typer.Option(
+        "docker",
+        "-e",
+        "--env",
+        help="Environment type for Harbor runs (docker|daytona|e2b|modal|runloop|gke)",
+        show_default=True,
+    ),
+    verbose: bool = typer.Option(False, "-v", "--verbose", help="Increase output verbosity"),
+    quiet: bool = typer.Option(False, "-q", "--quiet", help="Reduce output verbosity"),
+) -> None:
+    config = CreateConfig(
+        repo=repo,
+        pr=pr,
+        output=output,
+        cc_timeout=cc_timeout,
+        validate=validate,
+        force=force,
+        state_dir=state_dir,
+        use_cache=not no_cache,
+        require_minimum_difficulty=require_minimum_difficulty,
+        min_source_files=min_source_files,
+        max_source_files=max_source_files,
+        require_issue=require_issue,
+        allow_unmerged=allow_unmerged,
+        environment=EnvironmentType(environment),
+        verbose=verbose,
+        quiet=quiet,
+    )
+    try:
+        run_reversal(config)
+    except (TrivialPRError, MissingIssueError, ValidationError, FileExistsError) as err:
+        # These exceptions have already displayed user-friendly messages
+        # Exit with error code but don't show traceback
+        raise SystemExit(1) from err
+
+
+app.add_typer(create_app, name="create")
+
+
+@app.command(help="Validate an existing Harbor task by running NOP and ORACLE")
+def validate(
+    path: Path = typer.Argument(
+        ...,
+        help="Path to Harbor dataset root, specific task directory, or task ID when used with dataset root",
+    ),
+    task: str
+    | None = typer.Option(None, "--task", "-t", help="Task ID when --path points to dataset root"),
+    agent: str = typer.Option("both", help="Agent to run: both|nop|oracle", show_default=True),
+    jobs_dir: Path = typer.Option(
+        Path(".state/harbor-jobs"),
+        help="Directory to store Harbor job artifacts",
+        show_default=True,
+    ),
+    timeout_multiplier: float
+    | None = typer.Option(None, help="Multiply default timeouts (e.g., 3.0)"),
+    environment: str = typer.Option(
+        "docker",
+        "-e",
+        "--env",
+        help="Environment type for Harbor runs (docker|daytona|e2b|modal|runloop|gke)",
+        show_default=True,
+    ),
+    verbose: bool = typer.Option(False, "-v", "--verbose", help="Increase output verbosity"),
+    quiet: bool = typer.Option(False, "-q", "--quiet", help="Reduce output verbosity"),
+    max_parallel: int = typer.Option(
+        8, help="Maximum number of parallel validations (batch mode only)", show_default=True
+    ),
+    show_passed: bool = typer.Option(
+        False,
+        "--show-passed",
+        help="Show passed tasks in output (batch mode: default shows only failures)",
+    ),
+    output: Path
+    | None = typer.Option(
+        None, "-o", "--output", help="Write results to file as they complete (batch mode only)"
+    ),
+    docker_prune_batch: int = typer.Option(
+        5,
+        help="Run docker cleanup after every N tasks (0 to disable, local docker only)",
+        show_default=True,
+    ),
+) -> None:
+    if agent not in ("both", "nop", "oracle"):
+        raise typer.BadParameter("agent must be one of: both, nop, oracle")
+    run_validate(
+        ValidateArgs(
+            path=path,
+            task=task,
+            jobs_dir=jobs_dir,
+            agent=agent,
+            timeout_multiplier=timeout_multiplier,
+            verbose=verbose,
+            quiet=quiet,
+            environment=EnvironmentType(environment),
+            max_parallel=max_parallel,
+            show_passed=show_passed,
+            output_file=output,
+            docker_prune_batch=docker_prune_batch,
+        )
+    )
+
+
+@app.command(help="Analyze a task by running agent trials and classifying outcomes")
+def analyze(
+    path: Path = typer.Argument(..., help="Path to the task directory to analyze"),
+    agent: str = typer.Option(
+        "claude-code", "-a", "--agent", help="Agent to run trials with", show_default=True
+    ),
+    model: str = typer.Option(
+        "anthropic/claude-sonnet-4-5",
+        "-m",
+        "--model",
+        help="Model to use for agent trials",
+        show_default=True,
+    ),
+    n_trials: int = typer.Option(
+        3, "-k", "--n-trials", help="Number of trials to run", show_default=True
+    ),
+    n_concurrent: int = typer.Option(
+        3, "-n", "--n-concurrent", help="Number of concurrent trials (1=sequential, 3-5 recommended)", show_default=True
+    ),
+    jobs_dir: Path = typer.Option(
+        Path(".state/analyze-jobs"),
+        "--jobs-dir",
+        help="Directory to store job artifacts",
+        show_default=True,
+    ),
+    skip_quality_check: bool = typer.Option(
+        False, "--skip-quality-check", help="Skip static quality check"
+    ),
+    skip_baseline: bool = typer.Option(
+        False, "--skip-baseline", help="Skip baseline validation (nop/oracle)"
+    ),
+    skip_classify: bool = typer.Option(
+        False, "--skip-classify", help="Skip LLM classification of trial outcomes"
+    ),
+    analysis_model: str = typer.Option(
+        "claude-sonnet-4-5",
+        "--analysis-model",
+        help="Model for Claude Code classification",
+        show_default=True,
+    ),
+    timeout_multiplier: float = typer.Option(
+        1.0, "--timeout-multiplier", help="Multiply default timeouts", show_default=True
+    ),
+    environment: str = typer.Option(
+        "docker",
+        "-e",
+        "--env",
+        help="Environment type for Harbor runs (docker|daytona|e2b|modal|runloop|gke)",
+        show_default=True,
+    ),
+    verbose: bool = typer.Option(False, "-v", "--verbose", help="Increase output verbosity"),
+    classification_timeout: int = typer.Option(
+        300,
+        "--classification-timeout",
+        help="Timeout per trial classification in seconds",
+        show_default=True,
+    ),
+    verdict_timeout: int = typer.Option(
+        180,
+        "--verdict-timeout",
+        help="Timeout for verdict synthesis in seconds",
+        show_default=True,
+    ),
+    save_to_dir: bool = typer.Option(
+        False,
+        "--save-to-dir",
+        help="Write trajectory-analysis.{md,json} to each trial directory",
+    ),
+) -> None:
+    """
+    Analyze a Harbor task to determine if it's well-specified.
+
+    This command classifies trial outcomes to identify TASK PROBLEMS vs AGENT PROBLEMS:
+
+    1. Static quality check (Harbor's tasks check)
+    2. Baseline validation (nop should fail, oracle should pass)
+    3. Run N agent trials (default: 3 with Claude Code)
+    4. Classify each trial outcome:
+       - GOOD_SUCCESS: Agent solved it correctly
+       - BAD_SUCCESS: Agent cheated or tests too permissive
+       - GOOD_FAILURE: Agent failed due to its own limitations
+       - BAD_FAILURE: Agent failed due to task issues
+       - HARNESS_ERROR: Infrastructure problem
+    5. Compute task verdict with recommendations
+
+    The goal is to identify tasks that need fixing before release.
+
+    Flags match Harbor CLI conventions:
+        -k / --n-trials: Total number of trials to run
+        -n / --n-concurrent: Number of trials to run concurrently (parallelism)
+
+    Examples:
+        # Sequential (default)
+        swegen analyze tasks/my-task -k 5
+
+        # Parallel (3 trials at once)
+        swegen analyze tasks/my-task -k 10 -n 3
+    """
+    run_analyze(
+        AnalyzeArgs(
+            task_path=path,
+            agent=agent,
+            model=model,
+            n_trials=n_trials,
+            n_concurrent=n_concurrent,
+            jobs_dir=jobs_dir,
+            skip_quality_check=skip_quality_check,
+            skip_baseline=skip_baseline,
+            skip_classify=skip_classify,
+            analysis_model=analysis_model,
+            environment=environment,
+            timeout_multiplier=timeout_multiplier,
+            verbose=verbose,
+            classification_timeout=classification_timeout,
+            verdict_timeout=verdict_timeout,
+            save_to_dir=save_to_dir,
+        )
+    )
+
+
+
+
+@app.command(help="Continuous PR farming - stream through entire PR history")
+def farm(
+    repo: str = typer.Argument(
+        ..., help="GitHub repository in owner/name format (e.g., fastapi/fastapi)"
+    ),
+    output: Path = typer.Option(
+        Path("tasks"), help="Output directory for generated tasks", show_default=True
+    ),
+    state_dir: Path = typer.Option(
+        Path(".state"), help="State directory for cache/logs", show_default=True
+    ),
+    force: bool = typer.Option(True, help="Regenerate even if task already exists"),
+    timeout: int = typer.Option(300, help="Timeout per PR in seconds", show_default=True),
+    cc_timeout: int = typer.Option(
+        3200, help="Timeout for Claude Code session in seconds (~53 min default)", show_default=True
+    ),
+    api_delay: float = typer.Option(
+        0.5, help="Delay between GitHub API calls in seconds", show_default=True
+    ),
+    task_delay: int = typer.Option(60, help="Delay between tasks in seconds", show_default=True),
+    reset: bool = typer.Option(False, "--reset", help="Reset state and start from beginning"),
+    resume_from: str
+    | None = typer.Option(
+        None, help="Resume from date (e.g., '2024-01-15' or '2024-01-15T10:30:00Z')"
+    ),
+    dry_run: bool = typer.Option(
+        False, "--dry-run", help="Only show what would run (no task generation)"
+    ),
+    docker_prune_batch: int = typer.Option(
+        5, help="Run docker cleanup after every N PRs (0 to disable)", show_default=True
+    ),
+    skip_list: str
+    | None = typer.Option(None, help="Path to file with task IDs to skip (one per line)"),
+    no_cache: bool = typer.Option(
+        False, "--no-cache", help="Disable reusing cached Dockerfiles/test.sh"
+    ),
+    require_minimum_difficulty: bool = typer.Option(
+        True,
+        help="Require minimum difficulty (3+ source files); --no-require-minimum-difficulty to skip this check",
+    ),
+    min_source_files: int = typer.Option(
+        3, help="Minimum number of source files required (tests excluded)", show_default=True
+    ),
+    max_source_files: int = typer.Option(
+        10,
+        help="Maximum number of source files to avoid large refactors (tests excluded)",
+        show_default=True,
+    ),
+    environment: str = typer.Option(
+        "docker",
+        "-e",
+        "--env",
+        help="Environment type for Harbor runs (docker|daytona|e2b|modal|runloop|gke)",
+        show_default=True,
+    ),
+    verbose: bool = typer.Option(False, "-v", "--verbose", help="Enable verbose output"),
+    issue_only: bool = typer.Option(
+        True,
+        "--issue-only",
+        help="Only process PRs with linked issues (higher quality instructions)",
+    ),
+    validate: bool = typer.Option(
+        True, help="Run Harbor validation after CC; --no-validate to skip"
+    ),
+) -> None:
+    """
+    Continuously process merged GitHub PRs and convert them to Harbor tasks.
+    Streams PRs page-by-page, processes them immediately, and maintains state for resumable operation.
+    Uses a language-agnostic pipeline that works for any repository.
+    """
+    config = FarmConfig(
+        repo=repo,
+        output=output,
+        state_dir=state_dir,
+        force=force,
+        timeout=timeout,
+        cc_timeout=cc_timeout,
+        api_delay=api_delay,
+        task_delay=task_delay,
+        reset=reset,
+        resume_from=resume_from,
+        dry_run=dry_run,
+        docker_prune_batch=docker_prune_batch,
+        skip_list=skip_list,
+        no_cache=no_cache,
+        require_minimum_difficulty=require_minimum_difficulty,
+        min_source_files=min_source_files,
+        max_source_files=max_source_files,
+        environment=EnvironmentType(environment),
+        verbose=verbose,
+        issue_only=issue_only,
+        validate=validate,
+    )
+
+    console = Console()
+    farmer = StreamFarmer(config.repo, config, console)
+    exit_code = farmer.run()
+    raise typer.Exit(code=exit_code)

swegen/config.py

@@ -0,0 +1,142 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Literal
+
+from harbor.models.environment_type import EnvironmentType
+
+
+@dataclass(frozen=True)
+class CreateConfig:
+    """Configuration for the create command (PR → Harbor task).
+
+    The create command uses a language-agnostic pipeline that works
+    for any repository. Claude Code analyzes the repo to detect language, runtime,
+    build system, and test framework automatically.
+
+    Attributes:
+        repo: GitHub repository in "owner/repo" format or full URL
+        pr: Pull request number
+        output: Output directory for generated tasks (default: tasks/)
+        cc_timeout: Timeout for Claude Code session in seconds
+        validate: Run Harbor validations (NOP + Oracle)
+        force: Bypass local dedupe and regenerate existing tasks
+        state_dir: Directory for local state/cache
+        use_cache: Reuse cached Dockerfiles/test.sh from previous tasks
+        require_minimum_difficulty: Require 3+ source files for task
+        min_source_files: Minimum number of source files required (default: 3)
+        max_source_files: Maximum number of source files allowed to avoid large refactors (default: 10)
+        require_issue: Require PR to have a linked issue (higher quality instructions)
+        allow_unmerged: Allow processing unmerged PRs (for testing/preview, default: False)
+        environment: Environment type for Harbor runs (docker, daytona, e2b, modal, runloop, gke)
+        verbose: Increase output verbosity
+        quiet: Reduce output verbosity
+    """
+
+    repo: str
+    pr: int
+    output: Path = field(default_factory=lambda: Path("tasks"))
+    cc_timeout: int = 3200
+    validate: bool = True
+    force: bool = False
+    state_dir: Path = field(default_factory=lambda: Path(".state"))
+    use_cache: bool = True
+    require_minimum_difficulty: bool = True
+    min_source_files: int = 3
+    max_source_files: int = 10
+    require_issue: bool = True
+    allow_unmerged: bool = False
+    environment: EnvironmentType = EnvironmentType.DOCKER
+    verbose: bool = False
+    quiet: bool = False
+
+    # Computed property for backward compatibility with old code
+    @property
+    def no_validate(self) -> bool:
+        """Inverse of validate for backward compatibility."""
+        return not self.validate
+
+
+@dataclass(frozen=True)
+class FarmConfig:
+    """Configuration for the farm command (continuous PR processing).
+
+    The farm command uses a language-agnostic pipeline that works
+    for any repository. Claude Code analyzes the repo to detect language, runtime,
+    build system, and test framework automatically.
+
+    Attributes:
+        repo: GitHub repository in "owner/repo" format
+        output: Output directory for generated tasks (default: tasks/)
+        state_dir: Directory for local state/cache
+        force: Regenerate even if task already exists
+        timeout: Timeout per PR in seconds
+        cc_timeout: Timeout for Claude Code session in seconds
+        api_delay: Delay between GitHub API calls in seconds
+        task_delay: Delay between tasks in seconds
+        reset: Reset state and start from beginning
+        resume_from: Resume from date (ISO format or YYYY-MM-DD)
+        dry_run: Only show what would run (no task generation)
+        docker_prune_batch: Run docker cleanup after every N PRs (0 to disable)
+        skip_list: Path to file with task IDs to skip
+        no_cache: Disable reusing cached Dockerfiles/test.sh
+        require_minimum_difficulty: Require 3+ source files for task
+        min_source_files: Minimum number of source files required (default: 3)
+        max_source_files: Maximum number of source files allowed to avoid large refactors (default: 10)
+        environment: Environment type for Harbor runs (docker, daytona, e2b, modal, runloop, gke)
+        verbose: Enable verbose output
+        issue_only: Only process PRs that have linked issues (higher quality instructions)
+        validate: Run Harbor validation after CC (useful when CC times out but task may be valid)
+    """
+
+    repo: str
+    output: Path = field(default_factory=lambda: Path("tasks"))
+    state_dir: Path = field(default_factory=lambda: Path(".state"))
+    force: bool = True
+    timeout: int = 300
+    cc_timeout: int = 900
+    api_delay: float = 0.5
+    task_delay: int = 60
+    reset: bool = False
+    resume_from: str | None = None
+    dry_run: bool = False
+    docker_prune_batch: int = 5
+    skip_list: str | None = None
+    no_cache: bool = False
+    require_minimum_difficulty: bool = True
+    min_source_files: int = 3
+    max_source_files: int = 10
+    environment: EnvironmentType = EnvironmentType.DOCKER
+    verbose: bool = False
+    issue_only: bool = False
+    validate: bool = True
+
+
+@dataclass(frozen=True)
+class ValidateConfig:
+    """Configuration for the validate command.
+
+    Attributes:
+        path: Path to Harbor dataset root or specific task directory
+        task: Task ID when path points to dataset root
+        agent: Agent to run: both, nop, or oracle
+        jobs_dir: Directory to store Harbor job artifacts
+        timeout_multiplier: Multiply default timeouts
+        environment: Environment type for Harbor runs (docker, daytona, e2b, modal, runloop, gke)
+        verbose: Increase output verbosity
+        quiet: Reduce output verbosity
+        max_parallel: Maximum number of parallel validations (batch mode)
+        show_passed: Show passed tasks in output (batch mode)
+    """
+
+    path: Path
+    task: str | None = None
+    agent: Literal["both", "nop", "oracle"] = "both"
+    jobs_dir: Path = field(default_factory=lambda: Path(".state/harbor-jobs"))
+    timeout_multiplier: float | None = None
+    environment: EnvironmentType = EnvironmentType.DOCKER
+    verbose: bool = False
+    quiet: bool = False
+    max_parallel: int = 8
+    show_passed: bool = False