swegen 0.1.0__py3-none-any.whl

swegen/analyze/classify_prompt.txt

@@ -0,0 +1,241 @@
+You are analyzing a Harbor task trial to determine if the task is well-specified.
+
+## Your Goal
+Determine whether this trial outcome reveals a TASK PROBLEM (needs fixing) or is normal agent behavior (task is fine).
+
+**Critical Context:** This task has already passed baseline validation (oracle passes, nop fails). Your job is to detect problems that baseline validation CANNOT catch:
+- Underspecified instructions (agent lacks critical details)
+- Overspecified/brittle tests (tests coupled to specific implementation)
+- Ambiguous requirements (multiple valid interpretations)
+- Tests checking for details not mentioned in instructions
+
+## CRITICAL: Calibration for Hard Tasks
+
+**Hard tasks are SUPPOSED to be hard.** A 20-40% pass rate is EXPECTED and DESIRABLE for good benchmark tasks. Do NOT classify a failure as a task problem just because:
+- The agent had to explore the codebase to understand what to change
+- The instruction doesn't explicitly list every file that needs modification
+- The agent tried a reasonable approach that turned out to be wrong
+- The task requires significant investigation or domain expertise
+
+**The bar for BAD_FAILURE is HIGH.** Only classify as BAD_FAILURE if:
+- Information is GENUINELY IMPOSSIBLE to derive from instruction + codebase combined
+- Tests check for something that contradicts the instruction
+- Multiple valid solutions exist but tests only accept one specific approach
+- Tests are flaky or depend on non-deterministic behavior
+
+**Default to GOOD_FAILURE** when the agent fails. Agent failures are the norm for hard tasks.
+
+## CRITICAL: What the Agent Can and Cannot See
+
+**During the trial, the agent ONLY has access to:**
+- The `instruction.md` file describing the bug/task
+- The buggy codebase (repository code with the bug present)
+- Standard development tools (editor, terminal, etc.)
+
+**The agent CANNOT see and has NO knowledge of:**
+- `solution/` directory - contains fix.patch and solve.sh (used ONLY for oracle validation)
+- `tests/` directory - test files are copied in AFTER the agent finishes (for verification only)
+- Any patches, diffs, or reference solutions
+
+**This means:**
+- The agent must figure out the fix from scratch using only instruction.md and the buggy code
+- The agent has NO access to any "solution patch" - do NOT fault the agent for not using it
+- The agent cannot see how tests verify the solution - they work blind
+
+## The Verified Result
+**Test outcome: {result}** (pass = reward 1.0, fail = reward 0.0)
+
+This result is FINAL and has been verified by running the tests. Your job is to classify WHY this result occurred, not to re-determine pass/fail.
+
+**Classification constraints based on verified result:**
+- If result = 'pass' → classify as GOOD_SUCCESS or BAD_SUCCESS
+- If result = 'fail' → classify as GOOD_FAILURE, BAD_FAILURE, or HARNESS_ERROR
+
+## Where to Look (For YOUR Analysis - NOT What Agent Saw)
+
+**Task Definition ({task_dir}):** (for your analysis)
+- instruction.md - What the agent was told (ONLY thing agent sees from task)
+- solution/solve.sh - Reference solution (agent CANNOT see this)
+- tests/ - Test files that verify (agent CANNOT see these)
+
+**Trial Execution ({trial_dir}):**
+- agent/ - Agent execution logs and trajectory
+- verifier/test-stdout.txt - Test output
+- result.json - Contains verifier_result.rewards.reward
+
+Read the relevant files to understand WHY the result occurred, then classify accordingly.
+
+**Task directory structure:**
+
+```
+<task-dir>
+├── instruction.md
+├── task.toml
+├── environment
+│   ├── Dockerfile
+│   └── bug.patch
+├── solution
+│   ├── solve.sh
+│   └── fix.patch
+└── tests
+    ├── test.sh
+    └── # test files (e.g., test_*.py, *.test.ts, *_test.go, etc.)
+```
+
+## Classification Taxonomy
+
+### HARNESS_ERROR (Infrastructure Issue)
+The agent never ran properly:
+- Agent binary not found (e.g., 'bash: claude: command not found')
+- Docker/container setup failures
+- Missing dependencies in test environment
+- Empty trajectory files
+
+### GOOD_FAILURE (Agent's Fault - Task is Fine) ✓ DEFAULT FOR FAILURES
+Agent ran but couldn't solve it due to its own limitations. **This is the expected outcome for hard tasks.**
+- **Timeout**: Task requires many steps, agent ran out of time
+- **Wrong Approach**: Agent tried reasonable approaches but couldn't find the right solution
+- **Implementation Bugs**: Agent understood task but made coding errors
+- **Context Loss**: Agent forgot earlier context or requirements
+- **Premature Stop**: Agent gave up early or declared success incorrectly
+- **Complexity Overwhelm**: Task is genuinely difficult and agent couldn't handle it
+- **Insufficient Exploration**: Agent didn't explore the codebase enough to understand what to change
+- **Incomplete Understanding**: Agent misunderstood the problem or solution space
+
+**Key insight**: If the agent COULD have solved it with more effort, better exploration, or smarter reasoning, it's GOOD_FAILURE even if the task is hard.
+
+### BAD_FAILURE (Task's Fault - Needs Fix) ⚠️
+Agent failed due to task specification issues.
+
+**⚠️ IMPORTANT: The bar for BAD_FAILURE is VERY HIGH. Default to GOOD_FAILURE.**
+
+**Underspecified Instruction** - Information is IMPOSSIBLE to derive:
+- Tests require behavior that is NOT mentioned in instruction AND NOT discoverable from codebase
+- The instruction is actively misleading or contradicts what tests expect
+- Example: Instruction says "validate cookies" but tests ONLY check "authorization" header (completely different requirement)
+
+**NOT underspecified** (classify as GOOD_FAILURE instead):
+- Instruction describes the problem but agent must explore to find which files to change
+- Tests check specific files that a competent developer could identify by investigation
+- Agent needs to understand the codebase structure to implement the fix
+- Example: Instruction says "fix version references" - agent must explore to find go.mod files
+
+**Rigid/Brittle Tests** - Tests reject CORRECT solutions:
+- Tests check exact string matches instead of behavior (e.g., `assert "duplicate" in msg` rejects valid "conflicts with")
+- Tests require specific variable/function names not specified in instruction
+- Agent's solution is FUNCTIONALLY CORRECT but fails due to superficial differences
+- Example: Agent fixes the bug correctly but test fails because it expects specific error message format
+
+**NOT brittle** (classify as GOOD_FAILURE instead):
+- Tests check for the correct behavior and agent's solution doesn't implement it
+- Agent's approach was reasonable but wrong (this is expected for hard tasks)
+
+**Non-deterministic Tests** - Flaky/unpredictable:
+- Tests fail or pass inconsistently due to timing, race conditions, or randomness
+- Tests depend on external state or network conditions
+- Tests have order dependencies
+
+**Environment Issues** - Task-specific setup problems:
+- Missing dependencies, wrong package versions, docker/container problems specific to this task
+- Not general infrastructure failures (those are HARNESS_ERROR)
+
+**Missing File Reference** - Tests expect files not mentioned:
+- Tests check for outputs to files that instruction doesn't specify
+- Tests import modules that instruction doesn't mention needing
+
+**Ambiguous Requirements** - Multiple valid interpretations:
+- Instruction wording allows different correct implementations, but tests only accept one
+- Example: "Handle duplicates" could mean reject, skip, or merge - instruction doesn't specify
+
+⚠️ **KEY SIGNAL FOR BAD_FAILURE:** The agent's solution is FUNCTIONALLY CORRECT but tests reject it for superficial reasons, OR the task requires information that doesn't exist anywhere.
+
+⚠️ **KEY SIGNAL FOR GOOD_FAILURE:** The agent failed to find the right solution, but the information WAS available in the codebase if they had explored more thoroughly.
+
+### GOOD_SUCCESS (Legitimate Solution)
+Agent correctly solved the task:
+- Solution matches expected behavior from instruction
+- Tests verify actual functionality
+- Agent demonstrated understanding of the problem
+
+### BAD_SUCCESS (Cheating/Gaming) ⚠️
+Agent "passed" but through illegitimate means:
+
+**Hardcoding** - Agent hardcodes expected outputs:
+- Returns specific values instead of computing them
+- Pattern-matched test cases and hardcoded responses
+- Example: Returns "success" for every input instead of actual validation logic
+
+**Test Inspection** - Agent somehow accessed test files:
+- NOTE: Tests should NOT be visible to agent - if they accessed them, that's an infrastructure issue
+- Look for evidence in trajectory of agent reading test files from unexpected locations
+- Based solution on what tests check rather than instruction requirements
+
+**Oracle Copying** - Agent somehow accessed solution:
+- NOTE: solution/ should NOT be visible to agent - if they accessed it, that's an infrastructure issue
+- Look for evidence in trajectory of agent finding/reading solution files
+- This is rare and indicates a task setup problem, not normal agent cheating
+
+**Minimal Compliance** - Bare minimum to pass:
+- Technically passes tests but doesn't solve the actual problem
+- Only handles the specific test cases, would fail on similar inputs
+- Example: Hardcodes 3 expected values instead of implementing the algorithm
+
+**Tests Too Permissive** - Tests accept bad solutions:
+- Tests don't actually verify the requirement from instruction
+- Tests pass for trivial/wrong implementations
+- Example: Test checks function exists but doesn't verify behavior
+
+**Task Pre-solved** - Solution already present:
+- Repository already contained working code, agent just had to find it
+- Tests pass without any meaningful changes
+
+⚠️ **KEY SIGNAL:** If agent passed but their implementation is suspiciously minimal or hardcodes specific values, classify as BAD_SUCCESS. If they somehow accessed solution/ or tests/ (which should be hidden), note this as an infrastructure concern.
+
+## How to Analyze
+
+1. **Remember agent visibility** - Agent only saw instruction.md + buggy code. No tests, no solution, no patches.
+2. **Read the test output** (verifier/test-stdout.txt) - What specifically failed or passed?
+3. **Compare instruction vs tests** - Are tests checking for things NOT in instructions?
+4. **Examine agent trajectory** (agent/) - Did the agent try reasonable approaches given what they could see?
+5. **Check for cheating patterns** - Did agent hardcode values? (Accessing tests/solution should be impossible)
+6. **Consider consistency** - Would other agents likely have the same outcome?
+7. **Alternative solution test** - Would a different valid approach (that matches instruction) pass the tests?
+
+## Key Questions for Task Quality
+
+**For BAD_FAILURE (instruction/test problems) - ALL must be true:**
+- Is the required information IMPOSSIBLE to derive from instruction + codebase?
+- Did the agent implement something that is FUNCTIONALLY CORRECT but tests reject it?
+- Would ANY competent developer struggle because the spec is genuinely ambiguous or contradictory?
+
+**For GOOD_FAILURE (task is fine, agent failed) - ANY is sufficient:**
+- Could a skilled developer solve this by exploring the codebase carefully?
+- Is the information technically available but just requires investigation?
+- Did the agent fail to explore enough or make reasoning errors?
+- Is this just a hard problem that requires expertise?
+
+**For BAD_SUCCESS (cheating/too easy):**
+- Did the agent hardcode outputs instead of implementing logic?
+- Could an agent pass by pattern-matching without understanding the problem?
+- Do tests actually verify the requirement or just check superficial things?
+- Is there evidence the agent somehow accessed hidden files? (This shouldn't be possible normally)
+
+**Critical distinction (GOOD vs BAD):**
+- **GOOD_FAILURE**: Agent tried reasonable approaches but couldn't solve it (agent's limitation)
+- **BAD_FAILURE**: Agent tried reasonable approaches but tests rejected valid solutions (task's fault)
+- **GOOD_SUCCESS**: Agent solved it properly by understanding and implementing requirements
+- **BAD_SUCCESS**: Agent "solved" it by cheating, hardcoding, or tests are too permissive
+
+## Output Format
+
+REMEMBER: Your classification MUST match the verified result!
+- Result '{result}' means you must choose a matching classification (SUCCESS for pass, FAILURE for fail)
+
+Output ONLY valid JSON with this exact structure (no markdown, no code blocks, no explanation):
+{{
+  "classification": "HARNESS_ERROR | GOOD_FAILURE | BAD_FAILURE | GOOD_SUCCESS | BAD_SUCCESS",
+  "subtype": "specific subtype from the taxonomy above",
+  "evidence": "Quote specific test names, error messages, or code snippets that support your classification",
+  "root_cause": "1-2 sentence explanation of what specifically caused this outcome",
+  "recommendation": "If BAD_FAILURE or BAD_SUCCESS, explain how to fix the task. Otherwise write 'N/A - task is fine'"
+}}

swegen/analyze/models.py

@@ -0,0 +1,253 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+
+class Classification(str, Enum):
+    """Top-level classification of a trial outcome.
+    
+    The classification indicates whether the outcome reveals a problem
+    with the task (BAD_*) or is expected behavior (GOOD_*, HARNESS_ERROR).
+    """
+    
+    # Infrastructure problem - agent never ran
+    HARNESS_ERROR = "HARNESS_ERROR"
+    
+    # Agent ran but failed - task is fine, agent couldn't solve it
+    GOOD_FAILURE = "GOOD_FAILURE"
+    
+    # Agent failed due to task issues - task needs fixing
+    BAD_FAILURE = "BAD_FAILURE"
+    
+    # Agent solved it legitimately - task is working
+    GOOD_SUCCESS = "GOOD_SUCCESS"
+    
+    # Agent "solved" it by cheating or task is broken - task needs fixing
+    BAD_SUCCESS = "BAD_SUCCESS"
+    
+    @property
+    def is_task_problem(self) -> bool:
+        """Returns True if this classification indicates a task issue."""
+        return self in (Classification.BAD_FAILURE, Classification.BAD_SUCCESS)
+    
+    @property
+    def is_success(self) -> bool:
+        """Returns True if tests passed."""
+        return self in (Classification.GOOD_SUCCESS, Classification.BAD_SUCCESS)
+
+
+class Subtype(str, Enum):
+    """Detailed subtype explaining the classification.
+    
+    These provide actionable information about what specifically
+    caused the outcome.
+    """
+    
+    # HARNESS_ERROR subtypes
+    AGENT_NOT_FOUND = "Agent Not Found"
+    CONTAINER_FAILURE = "Container/Docker Failure"
+    MISSING_DEPENDENCIES = "Missing Dependencies"
+    EMPTY_TRAJECTORY = "Empty Trajectory"
+    INFRASTRUCTURE_ERROR = "Infrastructure Error"
+    
+    # GOOD_FAILURE subtypes (agent's fault)
+    TIMEOUT = "Timeout"
+    WRONG_APPROACH = "Wrong Approach"
+    IMPLEMENTATION_BUGS = "Implementation Bugs"
+    CONTEXT_LOSS = "Context Loss"
+    PREMATURE_STOP = "Premature Stop"
+    COMPLEXITY_OVERWHELM = "Complexity Overwhelm"
+    INCOMPLETE_SOLUTION = "Incomplete Solution"
+    LOGIC_ERROR = "Logic Error"
+    
+    # BAD_FAILURE subtypes (task's fault)
+    UNDERSPECIFIED_INSTRUCTION = "Underspecified Instruction"
+    RIGID_BRITTLE_TESTS = "Rigid/Brittle Tests"
+    NONDETERMINISTIC_TESTS = "Non-deterministic Tests"
+    ENVIRONMENT_ISSUES = "Environment Issues"
+    MISSING_FILE_REFERENCE = "Missing File Reference"
+    AMBIGUOUS_REQUIREMENTS = "Ambiguous Requirements"
+    IMPLEMENTATION_DETAILS_REQUIRED = "Implementation Details Required"
+    EDGE_CASES_NOT_SPECIFIED = "Edge Cases Not Specified"
+    TEST_EXPECTS_SPECIFIC_FORMAT = "Test Expects Specific Format"
+    
+    # GOOD_SUCCESS subtypes
+    CORRECT_SOLUTION = "Correct Solution"
+    ALTERNATIVE_VALID_SOLUTION = "Alternative Valid Solution"
+    
+    # BAD_SUCCESS subtypes (cheating/gaming)
+    HARDCODING = "Hardcoding"
+    TEST_INSPECTION = "Test Inspection"
+    ORACLE_COPYING = "Oracle Copying"
+    MINIMAL_COMPLIANCE = "Minimal Compliance"
+    TESTS_TOO_PERMISSIVE = "Tests Too Permissive"
+    TASK_PRE_SOLVED = "Task Pre-solved"
+
+
+class TrialClassificationModel(BaseModel):
+    """Pydantic model for LLM structured output."""
+    
+    classification: Literal[
+        "HARNESS_ERROR", "GOOD_FAILURE", "BAD_FAILURE", "GOOD_SUCCESS", "BAD_SUCCESS"
+    ] = Field(description="Top-level classification")
+    
+    subtype: str = Field(
+        description="Specific subtype from the taxonomy (e.g., 'Timeout', 'Underspecified Instruction')"
+    )
+    
+    evidence: str = Field(
+        description="Specific evidence from files: test names, error messages, code snippets"
+    )
+    
+    root_cause: str = Field(
+        description="1-2 sentence explanation of what caused this outcome"
+    )
+    
+    recommendation: str = Field(
+        description="How to fix the task (if BAD_FAILURE or BAD_SUCCESS), or 'N/A' if task is fine"
+    )
+
+
+class TaskVerdictModel(BaseModel):
+    """Pydantic model for LLM structured output for the overall task verdict."""
+
+    is_good: bool = Field(description="Whether the task is good (true) or needs review (false)")
+    confidence: Literal["high", "medium", "low"] = Field(description="Confidence level")
+    primary_issue: str | None = Field(
+        default=None, description="Primary issue if task needs review, else null"
+    )
+    recommendations: list[str] = Field(
+        default_factory=list, description="Actionable recommendations (3-5 for bad tasks)"
+    )
+    reasoning: str | None = Field(
+        default=None, description="1-2 sentence explanation of the verdict (optional)"
+    )
+
+
+@dataclass
+class TrialClassification:
+    """Classification result for a single trial.
+    
+    This captures why a trial succeeded or failed, and whether
+    the outcome indicates a task problem that needs fixing.
+    """
+    
+    trial_name: str
+    classification: Classification
+    subtype: str
+    evidence: str
+    root_cause: str
+    recommendation: str
+    
+    # Derived from verifier
+    reward: float | None = None
+    
+    @property
+    def is_task_problem(self) -> bool:
+        """Returns True if this trial reveals a task issue."""
+        return self.classification.is_task_problem
+    
+    @classmethod
+    def from_model(cls, trial_name: str, model: TrialClassificationModel, reward: float | None = None) -> "TrialClassification":
+        """Create from Pydantic model response."""
+        return cls(
+            trial_name=trial_name,
+            classification=Classification(model.classification),
+            subtype=model.subtype,
+            evidence=model.evidence,
+            root_cause=model.root_cause,
+            recommendation=model.recommendation,
+            reward=reward,
+        )
+
+
+@dataclass
+class BaselineResult:
+    """Result from running a baseline agent (nop or oracle)."""
+    
+    agent: Literal["nop", "oracle"]
+    passed: bool  # reward == 1
+    reward: float | None
+    error: str | None = None
+    
+    @property
+    def is_expected(self) -> bool:
+        """Returns True if the result is what we expect for a good task."""
+        if self.agent == "nop":
+            # nop should FAIL (reward=0) - tests should require changes
+            return not self.passed
+        else:
+            # oracle should PASS (reward=1) - reference solution should work
+            return self.passed
+
+
+@dataclass
+class BaselineValidation:
+    """Results from baseline validation (nop and oracle runs).
+    
+    For a well-formed task:
+    - nop should FAIL (tests require actual work)
+    - oracle should PASS (reference solution works)
+    """
+    
+    nop: BaselineResult | None = None
+    oracle: BaselineResult | None = None
+    
+    @property
+    def is_valid(self) -> bool:
+        """Returns True if baseline validation passes."""
+        nop_ok = self.nop is None or self.nop.is_expected
+        oracle_ok = self.oracle is None or self.oracle.is_expected
+        return nop_ok and oracle_ok
+    
+    @property
+    def issues(self) -> list[str]:
+        """Returns list of baseline validation issues."""
+        issues = []
+        if self.nop and not self.nop.is_expected:
+            issues.append(
+                "CRITICAL: nop agent passed - task may be pre-solved or tests are broken"
+            )
+        if self.oracle and not self.oracle.is_expected:
+            issues.append(
+                "CRITICAL: oracle agent failed - reference solution doesn't work"
+            )
+        return issues
+
+
+@dataclass
+class TaskVerdict:
+    """Final verdict on task quality based on all analysis.
+    
+    This aggregates results from:
+    - Static quality checks
+    - Baseline validation (nop/oracle)
+    - Agent trial classifications
+    """
+    
+    is_good: bool
+    confidence: Literal["high", "medium", "low"]
+    primary_issue: str | None
+    recommendations: list[str] = field(default_factory=list)
+    
+    # Breakdown
+    task_problem_count: int = 0
+    agent_problem_count: int = 0
+    success_count: int = 0
+    harness_error_count: int = 0
+    
+    # From classifications
+    classifications: list[TrialClassification] = field(default_factory=list)
+    baseline: BaselineValidation | None = None
+    
+    def summary(self) -> str:
+        """Return a one-line summary of the verdict."""
+        if self.is_good:
+            return f"✅ GOOD TASK (confidence: {self.confidence})"
+        else:
+            return f"❌ NEEDS REVIEW: {self.primary_issue}"
+