@openhands/extensions 0.0.1-alpha → 0.2.0

package/plugins/pr-review/scripts/prompt.py

@@ -0,0 +1,260 @@
+"""
+PR Review Prompt Template
+
+This module contains the prompt template used by the OpenHands agent
+for conducting pull request reviews. The template uses skill triggers:
+- {skill_trigger} will be replaced with '/codereview'
+- /github-pr-review provides instructions for posting review comments via GitHub API
+
+The template includes:
+- {diff} - The complete git diff for the PR (may be truncated for large files)
+- {pr_number} - The PR number
+- {commit_id} - The HEAD commit SHA
+- {review_context} - Previous review comments and thread resolution status
+
+When sub-agent delegation is enabled (``use_sub_agents=True``), a short
+delegation suffix is appended to the base prompt giving the agent the
+option to delegate file-level reviews via the TaskToolSet.
+"""
+
+# Template for when there is review context available
+_REVIEW_CONTEXT_SECTION = """
+## Previous Review History
+
+The following shows previous reviews and review threads on this PR.
+Pay attention to:
+- **Unresolved threads**: These issues may still need to be addressed
+- **Resolved threads**: These provide context on what was already discussed
+- **Previous review decisions**: See what other reviewers have said
+
+{review_context}
+
+When reviewing, consider:
+1. Don't repeat comments that have already been made and are still relevant
+2. If an issue is still unresolved in the code, you may reference it
+3. If resolved, don't bring it up unless the fix introduced new problems
+4. Focus on NEW issues in the current diff that haven't been discussed yet
+"""
+
+_EVIDENCE_REQUIREMENT_SECTION = """
+## PR Description Evidence Requirement
+
+Require the PR description to include an `Evidence` section (or similarly labeled section) showing that the code actually works.
+
+When checking the PR description:
+- For frontend or UI changes, require a screenshot or video that demonstrates the implemented behavior in the actual product.
+- For backend, API, CLI, or script changes, require the command(s) used to run or exercise the real code path end-to-end and the resulting output.
+- Unit tests alone do **not** count as evidence. Do not accept `pytest`, unit test output, or similar test runs as the only proof that the change works.
+- If the change appears to come from an agent conversation or AI-assisted workflow, prefer a conversation link such as `https://app.all-hands.dev/conversations/{conversation_id}` so reviewers can trace the work.
+- Do not accept vague claims like "tested locally" without concrete runtime artifacts, commands, or output.
+
+If the change is substantive and this evidence is missing or weak, call it out as a must-fix issue in your review. Do not invent evidence that is not present in the PR description.
+"""
+
+FEEDBACK_COMMENT_MARKER = "<!-- openhands-pr-review-feedback -->"
+
+_FEEDBACK_FOOTER_SECTION = """
+## Review Feedback Footer
+
+When you submit the top-level GitHub review body, append this exact footer at the end of that same review body so maintainers can react without creating a separate PR comment:
+
+```md
+---
+Was this automated review useful? React with 👍 or 👎 to this review to help us measure review quality.
+Workflow run: {review_run_url}
+{feedback_comment_marker}
+```
+
+Requirements:
+- Put this footer in the main review body, not in a separate issue comment.
+- Keep the rest of the review body concise.
+- If you would otherwise post only inline comments, still include a short top-level review body so this footer has somewhere to live.
+"""
+
+PROMPT = """{skill_trigger}
+/github-pr-review
+
+When posting a review, keep the review body brief unless your active review instructions require a longer structured format.
+
+For dependency update PRs, do **NOT** approve a target version that was published less than 7 days ago.
+
+Review the PR changes below and identify issues that need to be addressed.
+
+## Pull Request Information
+
+- **Title**: {title}
+- **Description**: {body}
+- **Repository**: {repo_name}
+- **Base Branch**: {base_branch}
+- **Head Branch**: {head_branch}
+- **PR Number**: {pr_number}
+- **Commit ID**: {commit_id}
+
+{review_context_section}{evidence_requirements_section}{feedback_footer_section}
+{files_manifest}
+## Patches
+
+The fenced block below contains the per-file patches. Individual patches may be **abbreviated** (look for `[patch abbreviated: ...]`) or **omitted** (look for `[patch omitted: ...]`) when they exceed the per-file or total budget. Files that appear in the manifest above but whose patch is missing or short here are still present in the PR — read the file from the workspace to inspect them. Do not flag them as missing from the PR.
+
+```diff
+{diff}
+```
+
+Analyze the changes and post your review using the GitHub API.
+"""
+
+# Appended to PROMPT when use_sub_agents=True.  Gives the main agent the
+# option to delegate via the TaskToolSet without duplicating the base prompt.
+_DELEGATION_SUFFIX = """
+## Sub-agent Delegation
+
+You have access to the **task** tool for delegating file-level reviews to
+`file_reviewer` sub-agents. Use it when the diff is large — roughly 4+ files
+or 500+ changed lines. For smaller diffs, just review directly.
+
+When delegating, split the diff by file (or small group of related files) and
+call the task tool with `subagent_type: "file_reviewer"`. Each sub-agent will
+return a JSON array of findings. Merge them, de-duplicate, drop noise, and
+post a single consolidated review via the GitHub API.
+"""
+
+# Skill content injected into each file_reviewer sub-agent.
+# Defines the review persona, available tools, and — most importantly — the
+# exact JSON schema the sub-agent must return.
+FILE_REVIEWER_SKILL = """\
+You are a **file-level code reviewer** sub-agent.
+
+## Your Task
+
+You will receive a diff for one or more files from a pull request.
+Review the changes and return structured findings.
+
+## Tools
+
+You have `terminal` and `file_editor` so you can inspect the full source
+files for surrounding context — use `cat`, `grep`, or `file_editor view`
+when the diff alone is not enough to judge an issue.
+
+## Review Style
+
+Be direct, pragmatic, and thorough. Focus on correctness, security,
+simplicity, and maintainability. Call out real problems; skip trivial noise.
+
+## Output Format
+
+Return a JSON array wrapped in a ```json fenced code block.
+Each element must have exactly these fields:
+
+| Field      | Type   | Description |
+|------------|--------|-------------|
+| `path`     | string | File path exactly as shown in the diff header (e.g. `src/utils.py`) |
+| `line`     | int    | Line number in the **new** file where the issue occurs |
+| `severity` | string | One of: `"critical"`, `"major"`, `"minor"`, `"nit"` |
+| `body`     | string | Concise description of the issue, including a suggested fix |
+
+### Severity guide
+- **critical** — bug, security vulnerability, or data loss
+- **major** — incorrect logic, missing error handling, performance issue
+- **minor** — style, readability, or minor correctness concern
+- **nit** — cosmetic or trivial preference
+
+### Example
+
+```json
+[
+  {{"path": "src/utils.py", "line": 42, "severity": "major", "body": "Unchecked `None` return — add a guard before accessing `.value`."}},
+  {{"path": "src/utils.py", "line": 78, "severity": "nit", "body": "Unused import `os`."}}
+]
+```
+
+If you find no issues, return:
+```json
+[]
+```
+
+When you are done, call the `finish` tool with the JSON array as the message.
+"""
+
+
+def format_prompt(
+    skill_trigger: str,
+    title: str,
+    body: str,
+    repo_name: str,
+    base_branch: str,
+    head_branch: str,
+    pr_number: str,
+    commit_id: str,
+    diff: str,
+    files_manifest: str = "",
+    review_context: str = "",
+    require_evidence: bool = False,
+    collect_feedback: bool = False,
+    review_run_url: str = "",
+    use_sub_agents: bool = False,
+) -> str:
+    """Format the PR review prompt with all parameters.
+
+    Args:
+        skill_trigger: The skill trigger (e.g., '/codereview')
+        title: PR title
+        body: PR description
+        repo_name: Repository name (owner/repo)
+        base_branch: Base branch name
+        head_branch: Head branch name
+        pr_number: PR number
+        commit_id: HEAD commit SHA
+        diff: Git diff content
+        review_context: Formatted previous review context. If empty or whitespace-only,
+                        the review context section is omitted from the prompt.
+        require_evidence: Whether to instruct the reviewer to enforce PR description
+                          evidence showing the code works.
+        collect_feedback: Whether to instruct the reviewer to append the feedback
+                          footer to the main review body.
+        review_run_url: Workflow run URL to embed in the feedback footer.
+        use_sub_agents: When True, the agent gets the TaskToolSet and decides
+                        at runtime whether to delegate file-level reviews to
+                        sub-agents based on diff size and complexity.
+
+    Returns:
+        Formatted prompt string
+    """
+    # Only include the review context section if there is actual context
+    if review_context and review_context.strip():
+        review_context_section = _REVIEW_CONTEXT_SECTION.format(
+            review_context=review_context
+        )
+    else:
+        review_context_section = ""
+
+    evidence_requirements_section = (
+        _EVIDENCE_REQUIREMENT_SECTION if require_evidence else ""
+    )
+
+    feedback_footer_section = ""
+    if collect_feedback:
+        feedback_footer_section = _FEEDBACK_FOOTER_SECTION.format(
+            review_run_url=review_run_url or "unavailable",
+            feedback_comment_marker=FEEDBACK_COMMENT_MARKER,
+        )
+
+    prompt = PROMPT.format(
+        skill_trigger=skill_trigger,
+        title=title,
+        body=body,
+        repo_name=repo_name,
+        base_branch=base_branch,
+        head_branch=head_branch,
+        pr_number=pr_number,
+        commit_id=commit_id,
+        review_context_section=review_context_section,
+        evidence_requirements_section=evidence_requirements_section,
+        feedback_footer_section=feedback_footer_section,
+        files_manifest=files_manifest,
+        diff=diff,
+    )
+
+    if use_sub_agents:
+        prompt += _DELEGATION_SUFFIX
+
+    return prompt

package/plugins/pr-review/workflows/pr-review-by-openhands.yml

@@ -0,0 +1,51 @@
+---
+name: PR Review by OpenHands
+
+on:
+    # Use pull_request_target to allow fork PRs to access secrets when triggered by maintainers
+    # Security: This workflow runs when:
+    #   1. A new PR is opened (non-draft), OR
+    #   2. A draft PR is marked as ready for review, OR
+    #   3. A maintainer adds the 'review-this' label, OR
+    #   4. A maintainer requests openhands-agent or all-hands-bot as a reviewer
+    # Adding labels and requesting reviewers requires write access.
+    # The PR code is explicitly checked out for review, but secrets are only accessible
+    # because the workflow runs in the base repository context.
+    pull_request_target:
+        types: [opened, ready_for_review, labeled, review_requested]
+
+permissions:
+    contents: read
+    pull-requests: write
+    issues: write
+
+jobs:
+    pr-review:
+        # Run when one of the following conditions is met:
+        #   1. A new non-draft PR is opened by a non-first-time contributor, OR
+        #   2. A draft PR is converted to ready for review by a non-first-time contributor, OR
+        #   3. 'review-this' label is added, OR
+        #   4. openhands-agent or all-hands-bot is requested as a reviewer
+        # Note: FIRST_TIME_CONTRIBUTOR and NONE PRs require manual trigger via label/reviewer request.
+        if: |
+            (github.event.action == 'opened' && github.event.pull_request.draft == false && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' && github.event.pull_request.author_association != 'NONE') ||
+            (github.event.action == 'ready_for_review' && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' && github.event.pull_request.author_association != 'NONE') ||
+            github.event.label.name == 'review-this' ||
+            github.event.requested_reviewer.login == 'openhands-agent' ||
+            github.event.requested_reviewer.login == 'all-hands-bot'
+        concurrency:
+            group: pr-review-${{ github.event.pull_request.number }}
+            cancel-in-progress: true
+        runs-on: ubuntu-24.04
+        steps:
+            - name: Run PR Review
+              uses: OpenHands/extensions/plugins/pr-review@main
+              with:
+                  llm-model: litellm_proxy/claude-sonnet-4-5-20250929
+                  llm-base-url: https://llm-proxy.app.all-hands.dev
+                  # [DEPRECATED] review-style is no longer used; standard and roasted are merged
+                  # review-style: roasted
+                  use-sub-agents: 'false' # Disabled by default (issue #208: high cost / timeouts)
+                  llm-api-key: ${{ secrets.LLM_API_KEY }}
+                  github-token: ${{ secrets.OPENHANDS_BOT_GITHUB_PAT_PUBLIC }}
+                  lmnr-api-key: ${{ secrets.LMNR_SKILLS_API_KEY }}

package/plugins/pr-review/workflows/pr-review-evaluation.yml

@@ -0,0 +1,85 @@
+---
+name: PR Review Evaluation
+
+# This workflow evaluates how well PR review comments were addressed.
+# It runs when a PR is closed to assess review effectiveness.
+#
+# Security note: pull_request_target is safe here because:
+# 1. Only triggers on PR close (not on code changes)
+# 2. Does not checkout PR code - only downloads artifacts from trusted workflow runs
+# 3. Runs evaluation scripts from the extensions repo, not from the PR
+
+on:
+    pull_request_target:
+        types: [closed]
+
+permissions:
+    contents: read
+    pull-requests: read
+
+jobs:
+    evaluate:
+        runs-on: ubuntu-24.04
+        env:
+            PR_NUMBER: ${{ github.event.pull_request.number }}
+            REPO_NAME: ${{ github.repository }}
+            PR_MERGED: ${{ github.event.pull_request.merged }}
+
+        steps:
+            - name: Download review trace artifact
+              id: download-trace
+              uses: dawidd6/action-download-artifact@v6
+              continue-on-error: true
+              with:
+                  workflow: pr-review-by-openhands.yml
+                  name: pr-review-trace-${{ github.event.pull_request.number }}
+                  path: trace-info
+                  search_artifacts: true
+                  if_no_artifact_found: warn
+
+            - name: Check if trace file exists
+              id: check-trace
+              run: |
+                  if [ -f "trace-info/laminar_trace_info.json" ]; then
+                    echo "trace_exists=true" >> $GITHUB_OUTPUT
+                    echo "Found trace file for PR #$PR_NUMBER"
+                  else
+                    echo "trace_exists=false" >> $GITHUB_OUTPUT
+                    echo "No trace file found for PR #$PR_NUMBER - skipping evaluation"
+                  fi
+
+            # Always checkout main branch for security - cannot test script changes in PRs
+            - name: Checkout extensions repository
+              if: steps.check-trace.outputs.trace_exists == 'true'
+              uses: actions/checkout@v5
+              with:
+                  repository: OpenHands/extensions
+                  path: extensions
+
+            - name: Set up Python
+              if: steps.check-trace.outputs.trace_exists == 'true'
+              uses: actions/setup-python@v6
+              with:
+                  python-version: '3.12'
+
+            - name: Install dependencies
+              if: steps.check-trace.outputs.trace_exists == 'true'
+              run: pip install lmnr
+
+            - name: Run evaluation
+              if: steps.check-trace.outputs.trace_exists == 'true'
+              env:
+                  # Script expects LMNR_PROJECT_API_KEY; org secret is named LMNR_SKILLS_API_KEY
+                  LMNR_PROJECT_API_KEY: ${{ secrets.LMNR_SKILLS_API_KEY }}
+                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+              run: |
+                  python extensions/plugins/pr-review/scripts/evaluate_review.py \
+                      --trace-file trace-info/laminar_trace_info.json
+
+            - name: Upload evaluation logs
+              uses: actions/upload-artifact@v5
+              if: always() && steps.check-trace.outputs.trace_exists == 'true'
+              with:
+                  name: pr-review-evaluation-${{ github.event.pull_request.number }}
+                  path: '*.log'
+                  retention-days: 30

package/plugins/qa-changes/.plugin/plugin.json

@@ -0,0 +1,11 @@
+{
+  "name": "qa-changes",
+  "version": "0.1.0",
+  "description": "Automated QA validation of PR changes — sets up the environment, runs tests, exercises changed behavior, and reports results",
+  "author": {
+    "name": "OpenHands",
+    "email": "contact@all-hands.dev"
+  },
+  "license": "MIT",
+  "repository": "https://github.com/OpenHands/extensions"
+}

package/plugins/qa-changes/README.md

@@ -0,0 +1,185 @@
+# QA Changes Plugin
+
+Automated pull request QA validation using OpenHands agents. Unlike the [PR Review plugin](../pr-review/) which reads diffs and posts code review comments, this plugin **actually runs the software** — setting up the environment, exercising changed behavior as a real user would, and posting a structured QA report. It does not re-run the test suite (that's CI's job) or analyze code style/logic (that's code review's job).
+
+## Quick Start
+
+Copy the workflow file to your repository:
+
+```bash
+cp plugins/qa-changes/workflows/qa-changes-by-openhands.yml \
+   .github/workflows/qa-changes-by-openhands.yml
+```
+
+Then configure the required secrets (see [Installation](#installation) below).
+
+## How It Works
+
+The QA agent follows a four-phase methodology:
+
+1. **Understand** — Reads the PR diff, title, and description. Classifies changes and identifies entry points (CLI commands, API endpoints, UI pages).
+2. **Setup** — Bootstraps the repo: installs dependencies, builds the project. Notes CI status but does not re-run tests.
+3. **Exercise** — The core phase. Actually uses the software the way a human would: spins up servers, opens browsers, runs CLI commands, makes HTTP requests. Focuses on functional verification that CI and code review cannot do.
+4. **Report** — Posts a structured QA report as a PR review with evidence (commands, outputs, screenshots) and a verdict.
+
+The agent knows when to give up: if a verification approach fails after three materially different attempts, it switches to a different approach. If two fundamentally different approaches fail, it reports honestly what could not be verified and suggests `AGENTS.md` guidance for future runs.
+
+## Plugin Contents
+
+```
+plugins/qa-changes/
+├── README.md              # This file
+├── action.yml             # Composite GitHub Action
+├── skills/                # Symbolic links to QA skills
+│   └── qa-changes -> ../../../skills/qa-changes
+├── workflows/             # Example GitHub workflow files
+│   └── qa-changes-by-openhands.yml
+└── scripts/               # Python scripts for QA execution
+    ├── agent_script.py    # Main QA agent script
+    └── prompt.py          # Prompt template
+```
+
+## Installation
+
+### 1. Copy the Workflow File
+
+Copy the workflow file to your repository's `.github/workflows/` directory:
+
+```bash
+mkdir -p .github/workflows
+cp plugins/qa-changes/workflows/qa-changes-by-openhands.yml \
+   .github/workflows/qa-changes-by-openhands.yml
+```
+
+### 2. Configure Secrets
+
+Add the following secrets in your repository settings (**Settings → Secrets and variables → Actions**):
+
+| Secret | Required | Description |
+|--------|----------|-------------|
+| `LLM_API_KEY` | Yes | API key for your LLM provider |
+| `GITHUB_TOKEN` | Auto | Provided automatically by GitHub Actions |
+
+### 3. Create the QA Label (Optional)
+
+Create a `qa-this` label for manual QA triggers:
+
+1. Go to **Issues → Labels**
+2. Click **New label**
+3. Name: `qa-this`
+4. Description: `Trigger OpenHands QA validation`
+
+## Usage
+
+### Automatic Triggers
+
+QA validation is automatically triggered when:
+- A new non-draft PR is opened (by trusted contributors)
+- A draft PR is marked as ready for review
+- The `qa-this` label is added
+- `openhands-agent` is requested as a reviewer
+
+### Requesting QA
+
+**Option 1: Add Label**
+
+Add the `qa-this` label to any PR.
+
+**Option 2: Request as Reviewer**
+
+Request `openhands-agent` as a reviewer on the PR.
+
+## Action Inputs
+
+| Input | Required | Default | Description |
+|-------|----------|---------|-------------|
+| `llm-model` | No | `anthropic/claude-sonnet-4-5-20250929` | LLM model to use |
+| `llm-base-url` | No | `''` | Custom LLM endpoint URL |
+| `extensions-repo` | No | `OpenHands/extensions` | Extensions repository |
+| `extensions-version` | No | `main` | Git ref (tag, branch, or SHA) |
+| `max-budget` | No | `10.0` | Maximum LLM cost in dollars — agent stops when exceeded |
+| `timeout-minutes` | No | `30` | Wall-clock timeout for the QA step |
+| `max-iterations` | No | `500` | Maximum agent iterations (each is one LLM call + action) |
+| `llm-api-key` | Yes | - | LLM API key |
+| `github-token` | Yes | - | GitHub token for API access |
+
+## QA Report Format
+
+The agent posts a PR comment with this structure:
+
+```
+## QA Report
+
+**Summary**: [One-sentence verdict]
+
+### Environment Setup
+[Build/install results]
+
+### CI Status
+[Note whether CI checks pass or fail — do not re-run tests]
+
+### Functional Verification
+[Commands run, outputs observed, screenshots, behavior verified]
+
+### Unable to Verify (if applicable)
+[What could not be verified, what was attempted, suggested AGENTS.md guidance]
+
+### Issues Found
+- 🔴 **Blocker**: [Description]
+- 🟠 **Issue**: [Description]
+- 🟡 **Minor**: [Description]
+
+### Verdict
+✅ PASS / ⚠️ PASS WITH ISSUES / ❌ FAIL / 🟡 PARTIAL
+```
+
+## Customizing QA Guidelines
+
+Add project-specific QA guidelines to your repository:
+
+### Option 1: Custom QA Skill
+
+Create `.agents/skills/qa-guide.md`:
+
+```markdown
+---
+name: qa-guide
+description: Project-specific QA guidelines
+triggers:
+- /qa-changes
+---
+
+# Project QA Guidelines
+
+## Setup Commands
+- `make install` to install dependencies
+- `make build` to build the project
+
+## How to Run the App
+- `make serve` to start the dev server on port 8080
+- `python -m myapp --help` for CLI usage
+
+## Key Behaviors to Verify
+- [List critical user flows]
+- [List known fragile areas]
+```
+
+### Option 2: Repository AGENTS.md
+
+Add setup and test commands to `AGENTS.md` at your repository root. The agent reads this file automatically.
+
+## Security
+
+The workflow uses `pull_request` (not `pull_request_target`) so that fork PRs do **not** get access to the base repository's secrets. Since the QA agent *executes* code from the PR (unlike a code-review agent which only reads diffs), using `pull_request_target` would allow untrusted fork code to run with the repo's `GITHUB_TOKEN` and `LLM_API_KEY`.
+
+The trade-off is that fork PRs won't have access to repository secrets. In the example `pull_request` workflow, the action now detects that case and exits successfully with a clear skip notice instead of failing. Maintainers can run QA locally or set up a separate trusted workflow for those cases.
+
+**Note**: The `FIRST_TIME_CONTRIBUTOR` and `NONE` author associations are excluded from automatic triggers as an additional safety layer.
+
+## Contributing
+
+See the main [extensions repository](https://github.com/OpenHands/extensions) for contribution guidelines.
+
+## License
+
+This plugin is part of the OpenHands extensions repository. See [LICENSE](/LICENSE) for details.