opencode-agent-skill 7.7.0

package/evals/live/tasks.json

@@ -0,0 +1,126 @@
+{
+  "version": 2,
+  "description": "Executable multi-domain coding-agent benchmark tasks. Each task runs in an isolated workspace and is graded outside the workspace.",
+  "tasks": [
+    {
+      "id": "js-discount-regression",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix calculateDiscount(price, percent) in src/discount.mjs. Percent is 0..100, so 20 means a 20% discount. Reject non-finite/non-number price or percent with TypeError and percent outside 0..100 with RangeError. Preserve the export and do not weaken the requested semantics.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "auth-resource-ownership",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix canEditResource(user, resource) in src/authz.mjs. Missing inputs deny. Suspended users always deny. Admin users may edit any non-missing resource. Non-admin users may edit only when user.id equals resource.ownerId AND user.tenantId equals resource.tenantId. Return a boolean and do not mutate inputs.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "api-pagination-contract",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix paginate(items, options) in src/pagination.mjs. pages are 1-indexed; defaults are page=1 and pageSize=10. items must be an array. page/pageSize must be positive integers. Return {items,total,page,pageSize,totalPages}; out-of-range pages return an empty items array. Use TypeError for wrong types and RangeError for invalid numeric ranges.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "inventory-reservation",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix reserveStock(stock, quantity) in src/inventory.mjs. Both values must be finite integers; stock must be >=0 and quantity must be >0. Throw TypeError for non-number/non-finite/non-integer values and RangeError for invalid ranges or insufficient stock. Return remaining stock without mutating external state.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "payment-idempotency",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix applyPaymentEvent(order, event) in src/payment.mjs. Do not mutate order. Duplicate event.id values already in order.processedEvents must be idempotent. A new event must have a non-empty id. For status='succeeded', amountCents and currency must exactly match order.totalCents/order.currency or throw RangeError, then mark status paid. Failed events must not mark paid. Record each new event id exactly once.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "webhook-ordering",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix advancePaymentState(state, event) in src/webhook.mjs. Ignore duplicate/stale events whose integer sequence <= state.lastSequence. Newer events must follow legal transitions pending->authorized|failed, authorized->paid|failed, paid->refunded; same-status newer events are allowed. Illegal transitions throw RangeError. Return a new state and never mutate inputs.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "secure-file-upload",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix validateUpload(file, options) in src/upload.mjs. file must have name/type/size. Reject names containing path separators, '..', or NUL. size must be a finite non-negative integer no larger than options.maxBytes. type must be in options.allowedTypes. Throw TypeError for malformed inputs and RangeError for size violations; return true when valid.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "path-traversal-defense",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix safeJoin(root, userPath) in src/path-safe.mjs. Return the absolute resolved path only when userPath is a non-empty relative path that stays inside root. Reject absolute paths, traversal outside root, and NUL with RangeError; wrong argument types use TypeError. The check must be cross-platform.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "stable-api-errors",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix toHttpError(error) in src/api-errors.mjs. Map codes VALIDATION->400, FORBIDDEN->403, NOT_FOUND->404, CONFLICT->409 and unknown/missing->500. Return {status,body:{error:{code,message}}}. Unknown errors must expose code INTERNAL and message 'Internal server error'; never expose stack or arbitrary extra fields.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "http-retry-policy",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix shouldRetry({status,attempt,maxAttempts}) in src/retry.mjs. attempt and maxAttempts are positive integers and status is an integer HTTP status. Retry only while attempt < maxAttempts and status is 408, 429, or 500..599. Return false for other statuses. Invalid input types/ranges should throw TypeError or RangeError appropriately.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "dedupe-stable-order",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix dedupeById(items) in src/dedupe.mjs. items must be an array of objects with non-null id. Preserve the first occurrence of each id and original order, do not mutate inputs, and treat numeric 1 and string '1' as distinct ids. Malformed items throw TypeError.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "legacy-user-migration",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix migrateUsers(rows) in src/migration.mjs. Return new objects without mutating rows. If firstName/lastName already exist, preserve them and remove legacy fullName. Otherwise split trimmed fullName on whitespace: first token -> firstName, remaining tokens -> lastName; an empty/missing fullName becomes empty strings. Preserve unrelated fields.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "money-integer-invariants",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix orderTotal(lines) in src/money.mjs. Each line uses integer unitPriceCents>=0, integer quantity>0, optional integer discountCents>=0. Line total is unitPriceCents*quantity-discountCents and may not go below zero. Return an integer cent total. Malformed numeric types use TypeError; invalid ranges use RangeError.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "sql-sort-allowlist",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix buildSort(sort) in src/query-sort.mjs. Accept an object {field,direction}. field allowlist: createdAt->created_at, name->name, price->price_cents. direction accepts asc/desc case-insensitively. Return exactly 'ORDER BY <mapped-column> <ASC|DESC>'. Reject all other values with RangeError and malformed input with TypeError; never interpolate arbitrary field text.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "react-stale-request",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix reducer in src/react-state.mjs for request races. start sets requestId, loading=true, error=null. success/error actions apply only when action.requestId equals current state.requestId; stale completions return the unchanged state object. Matching success sets data and loading=false/error=null; matching error sets error and loading=false. Unknown actions return state.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "react-native-keyboard-offset",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix keyboardOffset(platform, safeAreaTop, headerHeight) in src/rn-platform.mjs. platform must be 'ios' or 'android'; offsets must be finite non-negative numbers. iOS returns safeAreaTop+headerHeight; Android returns headerHeight. Invalid types use TypeError and invalid ranges/platform use RangeError.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "dependency-caret-range",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix satisfiesCaret(version, range) in src/dependency.mjs for stable numeric semver x.y.z and caret ranges ^x.y.z. ^1.2.3 allows >=1.2.3 and <2.0.0; ^0.2.3 allows >=0.2.3 and <0.3.0; ^0.0.3 allows only 0.0.3. Return false for malformed versions/ranges instead of throwing.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "strict-env-boolean",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix readBooleanEnv(value, fallback=false) in src/config.mjs. undefined/null/empty-string use Boolean(fallback). Boolean inputs return themselves. Case-insensitive trimmed strings true/1/yes/on => true and false/0/no/off => false. Any other value throws TypeError.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "cache-invalidation-tags",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Fix tagsForProductMutation(product) in src/cache-tags.mjs. product must contain a non-empty id. Return unique tags in this order: 'products', 'product:<id>', then optional 'category:<categoryId>' and 'seller:<sellerId>' when those values are non-null/non-empty. Do not emit undefined/empty tags and do not mutate product.",
+      "grader": "graders/engineering-bench.mjs"
+    },
+    {
+      "id": "multi-file-contract-compatibility",
+      "fixture": "fixtures/engineering-bench",
+      "prompt": "Repair the user serialization contract across src/contract-producer.mjs and src/contract-consumer.mjs. serializeUser must preserve id and legacy name while also exposing displayName, using user.displayName when non-empty otherwise falling back to name. userLabel must prefer the serialized displayName but remain compatible with name. Do not remove the legacy name field.",
+      "grader": "graders/engineering-bench.mjs"
+    }
+  ]
+}

package/evals/long/fixtures/long-horizon/package.json

@@ -0,0 +1 @@
+{"name":"ues-long-horizon-bench","private":true,"type":"module"}

package/evals/long/fixtures/long-horizon/src/auth.mjs

@@ -0,0 +1,5 @@
+export function canEditProject(user, project) {
+  if (!user || !project) return false
+  if (user.role === "admin") return true
+  return user.id === project.ownerId
+}

package/evals/long/fixtures/long-horizon/src/checkout.mjs

@@ -0,0 +1,9 @@
+import { reserveAll } from "./inventory.mjs"
+import { applyCharge } from "./payment.mjs"
+
+export function checkout(order, stock, event) {
+  return {
+    order: applyCharge(order, event),
+    stock: reserveAll(stock, order.lines || []),
+  }
+}

package/evals/long/fixtures/long-horizon/src/inventory.mjs

@@ -0,0 +1,5 @@
+export function reserveAll(stock, lines) {
+  const next = stock
+  for (const line of lines) next[line.sku] = (next[line.sku] || 0) - line.quantity
+  return next
+}

package/evals/long/fixtures/long-horizon/src/money.mjs

@@ -0,0 +1,3 @@
+export function calculateTotal(lines) {
+  return lines.reduce((sum, line) => sum + line.unitPriceCents, 0)
+}

package/evals/long/fixtures/long-horizon/src/payment.mjs

@@ -0,0 +1,7 @@
+export function applyCharge(order, event) {
+  return {
+    ...order,
+    status: event.status === "succeeded" ? "paid" : order.status,
+    processedEvents: [...(order.processedEvents || []), event.id],
+  }
+}

package/evals/long/fixtures/long-horizon/src/product-api.mjs

@@ -0,0 +1,10 @@
+import { updateProduct } from "./product-service.mjs"
+
+export function handleProductUpdate(input) {
+  try {
+    const result = updateProduct(input.product, input.patch)
+    return { status: 200, body: { product: result.product }, revalidate: result.tags }
+  } catch (error) {
+    return { status: 500, body: { error: { message: error.message, stack: error.stack } } }
+  }
+}

package/evals/long/fixtures/long-horizon/src/product-cache.mjs

@@ -0,0 +1,3 @@
+export function tagsForProduct(product) {
+  return ["products", "product:" + product.id, "category:" + product.categoryId, "seller:" + product.sellerId]
+}

package/evals/long/fixtures/long-horizon/src/product-service.mjs

@@ -0,0 +1,6 @@
+import { tagsForProduct } from "./product-cache.mjs"
+
+export function updateProduct(product, patch) {
+  Object.assign(product, patch)
+  return { product, tags: tagsForProduct(product) }
+}

package/evals/long/fixtures/long-horizon/src/product-state.mjs

@@ -0,0 +1,8 @@
+export const initialProductState = { requestId: null, loading: false, data: null, error: null }
+
+export function productReducer(state = initialProductState, action) {
+  if (action.type === "start") return { ...state, requestId: action.requestId, loading: true, error: null }
+  if (action.type === "success") return { ...state, data: action.data, loading: false, error: null }
+  if (action.type === "error") return { ...state, error: action.error, loading: false }
+  return state
+}

package/evals/long/fixtures/long-horizon/src/project-api.mjs

@@ -0,0 +1,9 @@
+import { updateProject } from "./project-service.mjs"
+
+export function handleProjectUpdate(input) {
+  try {
+    return { status: 200, body: { project: updateProject(input.user, input.project, input.patch) } }
+  } catch (error) {
+    return { status: 500, body: { error } }
+  }
+}

package/evals/long/fixtures/long-horizon/src/project-service.mjs

@@ -0,0 +1,7 @@
+import { canEditProject } from "./auth.mjs"
+
+export function updateProject(user, project, patch) {
+  if (!canEditProject(user, project)) throw new Error("no")
+  Object.assign(project, patch)
+  return project
+}

package/evals/long/fixtures/long-horizon/src/user-consumer.mjs

@@ -0,0 +1,3 @@
+export function userLabel(user) {
+  return user.name
+}

package/evals/long/fixtures/long-horizon/src/user-migration.mjs

@@ -0,0 +1,7 @@
+export function migrateUser(user) {
+  const [firstName = "", ...rest] = String(user.fullName || "").trim().split(/\s+/)
+  user.firstName = firstName
+  user.lastName = rest.join(" ")
+  delete user.fullName
+  return user
+}

package/evals/long/fixtures/long-horizon/src/user-serializer.mjs

@@ -0,0 +1,3 @@
+export function serializeUser(user) {
+  return { id: user.id, displayName: user.displayName || user.name }
+}

package/evals/long/fixtures/long-horizon/src/user-validation.mjs

@@ -0,0 +1,3 @@
+export function validateSerializedUser(user) {
+  return Boolean(user && user.id && user.displayName)
+}

package/evals/long/graders/long-horizon.mjs

@@ -0,0 +1,209 @@
+import assert from "node:assert/strict"
+import path from "node:path"
+import { spawnSync } from "node:child_process"
+import { fileURLToPath, pathToFileURL } from "node:url"
+
+const workspace = process.env.UES_EVAL_WORKSPACE
+const task = process.env.UES_EVAL_TASK
+if (!workspace || !task) {
+  console.error("UES_EVAL_WORKSPACE and UES_EVAL_TASK are required")
+  process.exit(2)
+}
+
+const load = async (file) =>
+  import(pathToFileURL(path.join(workspace, "src", file)).href + "?eval=" + Date.now() + "-" + Math.random())
+
+const throws = (fn, Type) => assert.throws(fn, Type)
+
+switch (task) {
+  case "long-checkout-integration": {
+    const { calculateTotal } = await load("money.mjs")
+    const { reserveAll } = await load("inventory.mjs")
+    const { applyCharge } = await load("payment.mjs")
+    const { checkout } = await load("checkout.mjs")
+
+    const lines = [
+      { sku: "a", unitPriceCents: 250, quantity: 2 },
+      { sku: "a", unitPriceCents: 100, quantity: 1 },
+      { sku: "b", unitPriceCents: 50, quantity: 3 },
+    ]
+    assert.equal(calculateTotal(lines), 750)
+    throws(() => calculateTotal("x"), TypeError)
+    throws(() => calculateTotal([{ sku: "a", unitPriceCents: 1.5, quantity: 1 }]), TypeError)
+    throws(() => calculateTotal([{ sku: "a", unitPriceCents: -1, quantity: 1 }]), RangeError)
+    throws(() => calculateTotal([{ sku: "a", unitPriceCents: 100, quantity: 0 }]), RangeError)
+
+    const stock = { a: 5, b: 4 }
+    const reserved = reserveAll(stock, lines)
+    assert.deepEqual(reserved, { a: 2, b: 1 })
+    assert.deepEqual(stock, { a: 5, b: 4 })
+    throws(() => reserveAll(stock, [...lines, { sku: "b", unitPriceCents: 1, quantity: 3 }]), RangeError)
+    assert.deepEqual(stock, { a: 5, b: 4 })
+    throws(() => reserveAll(stock, [{ sku: "missing", unitPriceCents: 1, quantity: 1 }]), RangeError)
+
+    const order = {
+      id: "o1",
+      status: "pending",
+      totalCents: 750,
+      currency: "USD",
+      processedEvents: [],
+      lines,
+    }
+    const paid = applyCharge(order, { id: "evt1", status: "succeeded", amountCents: 750, currency: "USD" })
+    assert.equal(order.status, "pending")
+    assert.deepEqual(order.processedEvents, [])
+    assert.equal(paid.status, "paid")
+    assert.deepEqual(paid.processedEvents, ["evt1"])
+    assert.deepEqual(applyCharge(paid, { id: "evt1", status: "succeeded", amountCents: 750, currency: "USD" }), paid)
+    throws(() => applyCharge(order, { id: "evt2", status: "succeeded", amountCents: 749, currency: "USD" }), RangeError)
+
+    const result = checkout(order, stock, { id: "evt3", status: "succeeded", amountCents: 750, currency: "USD" })
+    assert.equal(result.order.status, "paid")
+    assert.deepEqual(result.stock, { a: 2, b: 1 })
+    assert.deepEqual(stock, { a: 5, b: 4 })
+    assert.equal(order.status, "pending")
+    throws(() => checkout({ ...order, totalCents: 999 }, stock, { id: "evt4", status: "succeeded", amountCents: 750, currency: "USD" }), RangeError)
+    break
+  }
+
+  case "long-tenant-security-integration": {
+    const { canEditProject } = await load("auth.mjs")
+    const { updateProject } = await load("project-service.mjs")
+    const { handleProjectUpdate } = await load("project-api.mjs")
+
+    const project = { id: "p1", tenantId: "t1", ownerId: "u1", title: "Old", description: "D" }
+    const member = { id: "u1", tenantId: "t1", role: "member" }
+    assert.equal(canEditProject(member, project), true)
+    assert.equal(canEditProject({ ...member, tenantId: "t2" }, project), false)
+    assert.equal(canEditProject({ ...member, id: "u2" }, project), false)
+    assert.equal(canEditProject({ id: "a", tenantId: "t1", role: "admin" }, project), true)
+    assert.equal(canEditProject({ id: "a", tenantId: "t2", role: "admin" }, project), false)
+    assert.equal(canEditProject({ ...member, suspended: true }, project), false)
+
+    const updated = updateProject(member, project, { title: "New" })
+    assert.deepEqual(project, { id: "p1", tenantId: "t1", ownerId: "u1", title: "Old", description: "D" })
+    assert.deepEqual(updated, { id: "p1", tenantId: "t1", ownerId: "u1", title: "New", description: "D" })
+
+    assert.throws(
+      () => updateProject({ ...member, tenantId: "t2" }, project, { title: "X" }),
+      (error) => error && error.code === "FORBIDDEN",
+    )
+
+    const forbidden = handleProjectUpdate({
+      user: { ...member, tenantId: "t2" },
+      project,
+      patch: { title: "X" },
+    })
+    assert.deepEqual(forbidden, { status: 403, body: { error: { code: "FORBIDDEN", message: "Forbidden" } } })
+
+    const invalid = handleProjectUpdate({ user: member, project, patch: { tenantId: "evil" } })
+    assert.deepEqual(invalid, { status: 400, body: { error: { code: "VALIDATION", message: "Invalid request" } } })
+
+    const success = handleProjectUpdate({ user: member, project, patch: { description: "New D" } })
+    assert.equal(success.status, 200)
+    assert.equal(success.body.project.description, "New D")
+    assert.equal(success.body.project.tenantId, "t1")
+    break
+  }
+
+  case "long-user-contract-migration": {
+    const { migrateUser } = await load("user-migration.mjs")
+    const { serializeUser } = await load("user-serializer.mjs")
+    const { validateSerializedUser } = await load("user-validation.mjs")
+    const { userLabel } = await load("user-consumer.mjs")
+
+    const legacy = { id: 1, fullName: "  Ada   Lovelace  ", name: "Legacy Ada", extra: true }
+    const migrated = migrateUser(legacy)
+    assert.deepEqual(legacy, { id: 1, fullName: "  Ada   Lovelace  ", name: "Legacy Ada", extra: true })
+    assert.deepEqual(migrated, { id: 1, name: "Legacy Ada", extra: true, firstName: "Ada", lastName: "Lovelace" })
+
+    const existing = migrateUser({ id: 2, fullName: "Ignore Me", firstName: "Existing", lastName: "Name", x: 1 })
+    assert.deepEqual(existing, { id: 2, firstName: "Existing", lastName: "Name", x: 1 })
+
+    const serialized = serializeUser(migrated)
+    assert.deepEqual(serialized, { id: 1, name: "Legacy Ada", displayName: "Ada Lovelace" })
+    assert.deepEqual(
+      serializeUser({ id: 3, name: "Legacy", displayName: "Preferred", firstName: "A", lastName: "B" }),
+      { id: 3, name: "Legacy", displayName: "Preferred" },
+    )
+    assert.deepEqual(serializeUser({ id: 4, name: "Only Legacy" }), { id: 4, name: "Only Legacy", displayName: "Only Legacy" })
+
+    assert.equal(validateSerializedUser(serialized), true)
+    assert.equal(validateSerializedUser({ id: 1, displayName: "" }), false)
+    assert.equal(validateSerializedUser({ id: null, displayName: "X" }), false)
+    assert.equal(validateSerializedUser({ id: 1, displayName: "X", name: "" }), false)
+
+    assert.equal(userLabel({ id: 1, displayName: "Preferred", name: "Legacy" }), "Preferred")
+    assert.equal(userLabel({ id: 1, name: "Legacy" }), "Legacy")
+    break
+  }
+
+  case "long-product-race-cache": {
+    const { initialProductState, productReducer } = await load("product-state.mjs")
+    const { tagsForProduct } = await load("product-cache.mjs")
+    const { updateProduct } = await load("product-service.mjs")
+    const { handleProductUpdate } = await load("product-api.mjs")
+
+    const started = productReducer(initialProductState, { type: "start", requestId: 2 })
+    const stale = productReducer(started, { type: "success", requestId: 1, data: { id: "old" } })
+    assert.equal(stale, started)
+    assert.deepEqual(
+      productReducer(started, { type: "success", requestId: 2, data: { id: "new" } }),
+      { requestId: 2, loading: false, data: { id: "new" }, error: null },
+    )
+    const staleError = productReducer(started, { type: "error", requestId: 1, error: "old" })
+    assert.equal(staleError, started)
+
+    assert.deepEqual(
+      tagsForProduct({ id: "p1", categoryId: "c1", sellerId: "s1" }),
+      ["products", "product:p1", "category:c1", "seller:s1"],
+    )
+    assert.deepEqual(tagsForProduct({ id: "p1", categoryId: "", sellerId: null }), ["products", "product:p1"])
+    throws(() => tagsForProduct({ id: "" }), TypeError)
+
+    const product = { id: "p1", title: "Old", categoryId: "c1", sellerId: "s1", immutable: 7 }
+    const result = updateProduct(product, { title: "New", categoryId: "c2" })
+    assert.equal(product.title, "Old")
+    assert.deepEqual(result.product, { id: "p1", title: "New", categoryId: "c2", sellerId: "s1", immutable: 7 })
+    assert.deepEqual(result.tags, ["products", "product:p1", "category:c2", "seller:s1"])
+    throws(() => updateProduct(product, { id: "evil" }), TypeError)
+
+    const response = handleProductUpdate({ product, patch: { title: "API" } })
+    assert.equal(response.status, 200)
+    assert.equal(response.body.product.title, "API")
+    assert.deepEqual(response.revalidate, ["products", "product:p1", "category:c1", "seller:s1"])
+
+    const invalid = handleProductUpdate({ product, patch: { unknown: true } })
+    assert.deepEqual(invalid, { status: 400, body: { error: { code: "VALIDATION", message: "Invalid request" } } })
+    break
+  }
+
+  case "long-full-system-integration": {
+    const graderFile = fileURLToPath(import.meta.url)
+    const subtasks = [
+      "long-checkout-integration",
+      "long-tenant-security-integration",
+      "long-user-contract-migration",
+      "long-product-race-cache",
+    ]
+    for (const subtask of subtasks) {
+      const run = spawnSync(process.execPath, [graderFile], {
+        cwd: workspace,
+        env: { ...process.env, UES_EVAL_WORKSPACE: workspace, UES_EVAL_TASK: subtask },
+        encoding: "utf8",
+        maxBuffer: 1024 * 1024,
+      })
+      assert.equal(
+        run.status,
+        0,
+        subtask + " failed inside full-system grader:\n" + (run.stderr || run.stdout || ""),
+      )
+    }
+    break
+  }
+
+  default:
+    throw new Error("Unknown long-horizon task: " + task)
+}
+
+console.log("long-horizon hidden grader passed:", task)

package/evals/long/tasks.json

@@ -0,0 +1,36 @@
+{
+  "version": 2,
+  "description": "Long-horizon weak-model baseline vs UES benchmark. UES-mode passes require both hidden behavior graders and a completed durable multi-task orchestration state.",
+  "tasks": [
+    {
+      "id": "long-checkout-integration",
+      "fixture": "fixtures/long-horizon",
+      "prompt": "Treat this as a long-horizon multi-file task and use durable/fresh-task orchestration if the environment provides it. Repair the checkout flow across src/money.mjs, src/inventory.mjs, src/payment.mjs, and src/checkout.mjs. Money is integer cents. calculateTotal(lines) must validate integer unitPriceCents>=0 and quantity>0, and sum exactly. reserveAll(stock, lines) must be immutable and all-or-nothing: validate requested integer quantities, aggregate duplicate sku demand, reject insufficient/missing stock with RangeError, and return a new stock object. applyCharge(order,event) must be immutable/idempotent by event.id; successful events must exactly match order.totalCents and currency before marking paid; failed events do not mark paid; each new event id is recorded once. checkout(order,stock,event) must recompute the trusted total from order.lines, reject a client/order total mismatch, reserve stock, then apply the event, returning {order,stock} without mutating inputs.",
+      "grader": "graders/long-horizon.mjs"
+    },
+    {
+      "id": "long-tenant-security-integration",
+      "fixture": "fixtures/long-horizon",
+      "prompt": "Treat this as a long-horizon multi-file task and use durable/fresh-task orchestration if the environment provides it. Repair tenant authorization across src/auth.mjs, src/project-service.mjs, and src/project-api.mjs. Missing inputs deny. Suspended users always deny. Admin may edit a project only inside the same tenant. Members may edit only projects in the same tenant where project.ownerId equals user.id. updateProject must reject unauthorized access with an error whose code is FORBIDDEN, must allow only title and description patch fields, must not mutate project/patch, and must preserve id/tenantId/ownerId. handleProjectUpdate must return {status:200,body:{project}} on success, map FORBIDDEN to 403 with body {error:{code:'FORBIDDEN',message:'Forbidden'}}, map malformed patch input to 400 VALIDATION, and never expose stack/arbitrary error fields.",
+      "grader": "graders/long-horizon.mjs"
+    },
+    {
+      "id": "long-user-contract-migration",
+      "fixture": "fixtures/long-horizon",
+      "prompt": "Treat this as a long-horizon multi-file task and use durable/fresh-task orchestration if the environment provides it. Repair the legacy user migration contract across src/user-migration.mjs, src/user-serializer.mjs, src/user-consumer.mjs, and src/user-validation.mjs. migrateUser must return a new object, preserve unrelated fields, preserve existing firstName/lastName, otherwise split trimmed legacy fullName into first token/rest, remove fullName, and keep legacy name when present. serializeUser must preserve id and legacy name while exposing displayName from non-empty user.displayName, otherwise firstName+lastName, otherwise name. validateSerializedUser must require non-null id, non-empty displayName, and if name is present require it to be non-empty. userLabel must prefer displayName but remain backward-compatible with name. Do not mutate inputs.",
+      "grader": "graders/long-horizon.mjs"
+    },
+    {
+      "id": "long-product-race-cache",
+      "fixture": "fixtures/long-horizon",
+      "prompt": "Treat this as a long-horizon multi-file task and use durable/fresh-task orchestration if the environment provides it. Repair product mutation state across src/product-state.mjs, src/product-cache.mjs, src/product-service.mjs, and src/product-api.mjs. The reducer must ignore stale success/error actions whose requestId differs from current state.requestId, preserving object identity for stale actions. tagsForProduct must return unique tags in order products, product:<id>, optional category:<categoryId>, optional seller:<sellerId>, rejecting missing id. updateProduct must allow only title/categoryId/sellerId patch fields, be immutable, preserve id, and return {product,tags}. handleProductUpdate must use updateProduct and return {status:200,body:{product},revalidate:tags}; malformed patches return 400 VALIDATION without leaking stack.",
+      "grader": "graders/long-horizon.mjs"
+    },
+    {
+      "id": "long-full-system-integration",
+      "fixture": "fixtures/long-horizon",
+      "prompt": "Treat this as a true long-horizon task. You must use the available durable UES workflow with an approved multi-task PLAN, fresh executors, task evidence, integration verification and finalization rather than solving everything in one context. Repair all four broken domains across 15 modules: src/money.mjs, src/inventory.mjs, src/payment.mjs, src/checkout.mjs, src/auth.mjs, src/project-service.mjs, src/project-api.mjs, src/user-migration.mjs, src/user-serializer.mjs, src/user-consumer.mjs, src/user-validation.mjs, src/product-state.mjs, src/product-cache.mjs, src/product-service.mjs, and src/product-api.mjs. Preserve immutability and strict validation throughout. Checkout must use trusted integer-cent totals, all-or-nothing aggregate stock reservation and idempotent exact-currency payment events. Tenant project edits must deny suspended/cross-tenant access, limit patch fields and return sanitized API errors. User migration/serialization must preserve backward compatibility while producing a valid displayName contract without mutating inputs. Product mutation must ignore stale reducer actions, generate stable unique cache tags, restrict patch fields and sanitize API validation errors. All existing hidden requirements of the four domain tasks apply together.",
+      "grader": "graders/long-horizon.mjs"
+    }
+  ]
+}