@tangle-network/agent-eval 0.136.0 → 0.138.0

package/docs/feedback-trajectories.md

@@ -55,7 +55,7 @@ const optimizerRows = feedbackTrajectoriesToOptimizerRows([trajectory])
 
 ## What Belongs In Core
 
-`agent-eval` owns the substrate:
+`agent-eval` owns the shared data model:
 
 - trajectory and label schemas
 - in-memory and JSONL-backed stores
@@ -74,6 +74,105 @@ Downstream repos own domain adapters:
 - which budgets and metrics matter
 - where task-local data is stored
 
+## Trace Analyst Bridge
+
+Trace findings are hypotheses until another signal confirms them.
+Use `analystRunToReviewRequests()` to populate a review queue.
+Use `analystRunToFeedbackTrajectory()` immediately to archive the run, even when review has not started.
+The archived trajectory cannot become optimizer input until every finding has one independent decision and the run has one independent completeness assessment.
+
+```ts
+import {
+  analystRunDigest,
+  analystRunToFeedbackTrajectory,
+  analystRunToReviewRequests,
+  feedbackTrajectoryToOptimizerRow,
+} from '@tangle-network/agent-eval'
+
+const runDigest = analystRunDigest(analystRun)
+const requests = analystRunToReviewRequests(analystRun)
+await reviewQueue.add(requests)
+
+const archived = analystRunToFeedbackTrajectory(analystRun, {
+  task: { intent: 'Explain the failed agent run.' },
+  reviewRequests: requests,
+})
+
+await trajectoryStore.save(archived)
+
+declare const reviewedFindingIds: ReadonlySet<string>
+declare const reviewReasons: ReadonlyMap<string, string>
+declare const missedIssues: Array<{
+  id: string
+  reason: string
+  evidence?: Array<{ kind: 'span'; uri: string }>
+}>
+const requestsByFindingId = new Map(requests.map((request) => [request.findingId, request]))
+
+const reviewed = analystRunToFeedbackTrajectory(analystRun, {
+  task: { intent: 'Explain the failed agent run.' },
+  reviewRequests: requests,
+  reviewDecisions: [
+    ...analystRun.findings.map((finding) => ({
+      runDigest,
+      findingId: finding.finding_id,
+      findingDigest: requestsByFindingId.get(finding.finding_id)!.findingDigest,
+      verdict: reviewedFindingIds.has(finding.finding_id) ? 'confirmed' as const : 'rejected' as const,
+      source: 'user' as const,
+      reviewerId: 'reviewer-42',
+      reviewId: 'trace-review-918',
+      reason: reviewReasons.get(finding.finding_id) ?? 'Reviewed against the cited evidence.',
+      decidedAt: new Date().toISOString(),
+    })),
+    {
+      runDigest,
+      verdict: 'completeness_assessed',
+      missedIssues,
+      source: 'user',
+      reviewerId: 'reviewer-42',
+      reviewId: 'trace-review-918',
+      reason: 'Reviewed the full run for findings the analyst omitted.',
+      decidedAt: new Date().toISOString(),
+    },
+  ],
+})
+
+const optimizerRow = feedbackTrajectoryToOptimizerRow(reviewed)
+```
+
+Review requests, generic labels, and run-level outcomes do not validate findings.
+Every request id includes the digest of the complete analyst run, so the same finding in two runs produces two request ids.
+Every decision must echo that run digest and cannot be replayed onto another run.
+Each finding decision must name a known finding and echo its full content digest.
+The reviewer identity must differ from every analyst that produced the run.
+The completeness assessment records unique missed issue ids, reasons, and optional evidence.
+Duplicate, unknown, incomplete, mutated, and self-reviewed data throws instead of being filtered.
+The optimizer score is F1 over confirmed findings and independently identified misses.
+A run with no findings and no missed issues scores `1`.
+The run-level outcome remains context and does not replace that score.
+
+Every run needs one `completeness_assessed` decision from an independent reviewer or environment check:
+
+```ts
+const cleanRunDigest = analystRunDigest(cleanRun)
+const reviewedCleanRun = analystRunToFeedbackTrajectory(cleanRun, {
+  task: { intent: 'Check whether the run has a reportable failure.' },
+  reviewDecisions: [{
+    runDigest: cleanRunDigest,
+    verdict: 'completeness_assessed',
+    missedIssues: [],
+    source: 'environment',
+    reviewerId: 'release-test-suite',
+    reviewId: 'release-check-284',
+    reason: 'The independent release checks found no omitted reportable failure.',
+    decidedAt: new Date().toISOString(),
+  }],
+})
+```
+
+A raw zero-finding run is still valid archival data.
+It is not learning data until completeness is independently assessed.
+
 ## Label Sources
 
 Labels can come from multiple places: