@tangle-network/agent-eval 0.53.0 → 0.55.0

package/docs/pilot/integration-foreign-stack.md

@@ -0,0 +1,296 @@
+# Integration — Tangle Intelligence on any stack (OpenRouter, OpenAI, Anthropic, LangChain, LlamaIndex, custom)
+
+Companion to `integration-tangle-stack.md`. This doc is the path for customers NOT on `@tangle-network/sandbox` + tcloud — you bring your own agent, your own LLM provider, your own trace format. We meet you where you are.
+
+## Zero-setup demo first
+
+```sh
+npx @tangle-network/intelligence demo
+```
+
+Synthetic agent + scenarios + selfImprove end-to-end with zero setup. Prints the `InsightReport` shape — same output you'll get against your real data. Hosted equivalent: **[staging-intelligence.tangle.tools](https://staging-intelligence.tangle.tools)**.
+
+## Decision tree — pick your starting point
+
+```
+What's your trace source?
+├── OTel-compatible (Datadog APM, Honeycomb, NewRelic, raw OTLP) → fromOtelSpans
+├── LangChain (LangSmith traces, LCEL run traces)               → fromLangChain      (#104, queued)
+├── LlamaIndex (callback traces)                                → fromLlamaIndex     (#104, queued)
+├── Anthropic SDK direct (Messages API call logs)               → fromAnthropicSDK   (#104, queued)
+├── OpenAI Assistants API (run + step events)                   → fromOpenAIAssistants (#104, queued)
+├── Multi-rater human approval/reject corpus                    → fromFeedbackTable
+├── Custom (your own logs / DB rows)                            → 20-line mapper to RunRecord
+└── @tangle-network/sandbox                                     → fromTangleSandbox (see Tangle-stack doc)
+
+What's your LLM provider for the closed loop?
+├── OpenAI                          → gepaDriver({ llm: { apiKey, baseUrl: 'https://api.openai.com/v1' } })
+├── Anthropic                       → gepaDriver({ llm: { apiKey, baseUrl: 'https://api.anthropic.com/v1' } })
+├── OpenRouter                      → gepaDriver({ llm: { apiKey, baseUrl: 'https://openrouter.ai/api/v1' } })
+├── Tangle tcloud                   → gepaDriver({ llm: { apiKey, baseUrl: 'https://router.tangle.tools/v1' } })
+├── Azure OpenAI                    → gepaDriver({ llm: { apiKey, baseUrl: 'https://<resource>.openai.azure.com/...' } })
+├── Bedrock / Vertex                → custom client wrapper (we help)
+└── Self-hosted (vLLM, Ollama, etc.) → OpenAI-compat endpoint works directly
+```
+
+## OTel → InsightReport (5 minutes)
+
+```ts
+import { fromOtelSpans } from '@tangle-network/agent-eval'
+import { analyzeRuns } from '@tangle-network/agent-eval/contract'
+
+// You probably already export OTel via OTLP. Pull a JSONL dump of spans
+// for your last week of agent activity.
+const spans = JSON.parse(fs.readFileSync('./agent-traces.jsonl', 'utf8').split('\n').filter(Boolean).map(JSON.parse))
+
+const runs = fromOtelSpans({ spans })  // RunRecord[]
+const report = await analyzeRuns({ runs })
+
+console.log(report.composite.mean)
+console.log(report.recommendations)
+```
+
+What `fromOtelSpans` expects in spans (it's flexible — it tries multiple attribute keys):
+- `trace_id` (groups spans into a run)
+- `attributes['llm.tokens.in' | 'llm.input_tokens']` (optional)
+- `attributes['llm.tokens.out' | 'llm.output_tokens']` (optional)
+- `attributes['tool.name']` (optional, for tool-failure-rate analysis)
+- `status.code: 'ERROR' | 'OK'` (for failure detection)
+- `start_unix_nano` + `end_unix_nano` (for duration)
+
+If your spans use different attribute keys, pass `{ attributeMap }` to override.
+
+## OpenRouter as your closed-loop LLM provider
+
+OpenRouter speaks OpenAI-compat. Plug it in directly:
+
+```ts
+import { selfImprove, gepaDriver } from '@tangle-network/agent-eval/contract'
+
+const result = await selfImprove({
+  scenarios: yourScenarios,
+  agent: yourAgent,             // your existing dispatch — any framework
+  judge: yourJudge,
+  baselineSurface: currentPrompt,
+  driver: gepaDriver({
+    llm: {
+      apiKey: process.env.OPENROUTER_KEY!,
+      baseUrl: 'https://openrouter.ai/api/v1',
+    },
+    model: 'anthropic/claude-sonnet-4.6',     // any OpenRouter-supported model
+    target: 'agent system prompt',
+  }),
+  budget: { generations: 3, populationSize: 4, holdoutFraction: 0.3, maxUsd: 25 },
+})
+```
+
+Same shape works for: OpenAI direct, Anthropic direct, Azure OpenAI, Vertex (with their OpenAI-compat layer), Bedrock (via LiteLLM proxy), self-hosted vLLM / Ollama / LMStudio.
+
+## LangChain customer — three minute integration
+
+While the dedicated `fromLangChain` adapter is queued (#104), the universal path:
+
+```ts
+// Step 1: configure LangSmith to also export OTel
+// (LangSmith → Project Settings → Trace exports → enable OTLP)
+
+// Step 2: ingest as OTel
+import { fromOtelSpans } from '@tangle-network/agent-eval'
+const runs = fromOtelSpans({ spans: yourLangSmithOtelDump })
+
+// Step 3: analyze
+import { analyzeRuns } from '@tangle-network/agent-eval/contract'
+const report = await analyzeRuns({ runs })
+```
+
+When `fromLangChain` lands, it'll be a one-liner:
+
+```ts
+// Coming in 0.55.0
+import { fromLangChain } from '@tangle-network/agent-eval/adapters/langchain'
+const runs = fromLangChain({ traces: yourLangSmithExport })
+```
+
+Same for LlamaIndex / OpenAI Assistants / Anthropic SDK — direct adapters queued in #104.
+
+## LlamaIndex customer
+
+LlamaIndex's callback manager emits OTel spans natively. Wire it once:
+
+```python
+# Python side — your LlamaIndex setup
+from llama_index.callbacks import OpenInferenceCallbackHandler
+
+callback_handler = OpenInferenceCallbackHandler()
+# This emits to your OTel exporter
+
+# Then on the agent-eval side (TypeScript or Python):
+from agent_eval_rpc import Client
+client = Client(base_url='https://api.tangle.tools/v1', api_key=YOUR_KEY)
+report = client.analyze_runs(spans=your_otel_spans)
+```
+
+The Python client (`agent-eval-rpc@0.53.0`) speaks the same wire protocol — no functional difference between TS and Python customers.
+
+## Anthropic SDK direct (no framework)
+
+If you're calling `@anthropic-ai/sdk` directly without an agent framework:
+
+```ts
+import Anthropic from '@anthropic-ai/sdk'
+import { fromOtelSpans } from '@tangle-network/agent-eval'
+
+// Step 1: wrap your Anthropic calls to emit OTel
+import { trace } from '@opentelemetry/api'
+const tracer = trace.getTracer('your-agent')
+
+async function callAnthropic(scenario: Scenario) {
+  return tracer.startActiveSpan('agent.turn', async (span) => {
+    const result = await anthropic.messages.create({...})
+    span.setAttribute('llm.input_tokens', result.usage.input_tokens)
+    span.setAttribute('llm.output_tokens', result.usage.output_tokens)
+    span.setAttribute('tangle.runId', scenario.id)
+    span.end()
+    return result
+  })
+}
+
+// Step 2: same pipeline
+const runs = fromOtelSpans({ spans: yourOtelExport })
+const report = await analyzeRuns({ runs })
+```
+
+20 lines of OTel wrapping; the rest is pure substrate.
+
+## OpenAI Assistants API
+
+The Assistants API emits `runs.steps` events natively. Map them to RunRecord:
+
+```ts
+// Custom mapper while fromOpenAIAssistants (queued #104) lands:
+function mapAssistantRunToRunRecord(threadId: string, runId: string): RunRecord {
+  const run = await openai.beta.threads.runs.retrieve(threadId, runId)
+  const steps = await openai.beta.threads.runs.steps.list(threadId, runId)
+
+  return {
+    runId: run.id,
+    experimentId: 'default',
+    candidateId: run.assistant_id,
+    seed: 0,
+    model: run.model,
+    promptHash: hashOf(run.instructions),
+    configHash: hashOf({ tools: run.tools, model: run.model }),
+    commitSha: process.env.GIT_SHA ?? 'unknown',
+    wallMs: (run.completed_at - run.created_at) * 1000,
+    costUsd: estimateCostFromUsage(run.usage),
+    tokenUsage: {
+      input: run.usage?.prompt_tokens ?? 0,
+      output: run.usage?.completion_tokens ?? 0,
+    },
+    outcome: {
+      holdoutScore: yourScoring(run),
+      raw: { stepCount: steps.data.length, status: run.status },
+    },
+    splitTag: 'holdout',
+  }
+}
+```
+
+Once the dedicated adapter ships in 0.55.0 this becomes one line.
+
+## Custom trace format
+
+Your logs / DB rows / proprietary schema → `RunRecord`:
+
+```ts
+import type { RunRecord } from '@tangle-network/agent-eval'
+
+function mapMyRowToRunRecord(row: MyAgentLog): RunRecord {
+  return {
+    runId: row.id,
+    experimentId: row.experiment_name ?? 'default',
+    candidateId: row.model_version,
+    seed: row.random_seed ?? 0,
+    model: row.model,
+    promptHash: row.prompt_hash,
+    configHash: row.config_hash,
+    commitSha: row.git_sha,
+    wallMs: row.duration_ms,
+    costUsd: row.cost,
+    tokenUsage: {
+      input: row.input_tokens,
+      output: row.output_tokens,
+    },
+    outcome: {
+      holdoutScore: row.score,
+      raw: row.raw_output,    // free-form bag for fields the substrate doesn't standardize
+    },
+    splitTag: row.is_holdout ? 'holdout' : 'search',
+  }
+}
+
+const runs = myLogs.map(mapMyRowToRunRecord)
+const report = await analyzeRuns({ runs })
+```
+
+That's the worst case. ~20 lines of mapping, and you're in.
+
+## Multi-rater human feedback (no LLM-as-judge yet)
+
+If you don't have an automated judge but you DO have human raters approving/rejecting agent outputs:
+
+```ts
+import { fromFeedbackTable } from '@tangle-network/agent-eval'
+import { analyzeRuns } from '@tangle-network/agent-eval/contract'
+
+// Your data shape:
+const ratings = [
+  { runId: 'r-001', rater: 'alice', score: 1 },   // approved
+  { runId: 'r-001', rater: 'bob', score: 0 },     // rejected — disagreement!
+  { runId: 'r-002', rater: 'alice', score: 1 },
+  { runId: 'r-002', rater: 'bob', score: 1 },
+  // ...
+]
+
+const { runs, raterScores } = fromFeedbackTable({ ratings })
+const report = await analyzeRuns({ runs, raterScores })
+
+// report.interRater.kappa → how much your raters agree
+// report.interRater.disagreementCases → which runs raters split on
+// → use these to iterate the rubric until kappa > 0.7
+// → then build an LLM-as-judge against that aligned rubric
+```
+
+This is the warm-up path for customers who don't have a judge yet.
+
+## Hosted vs self-hosted — what's the difference?
+
+| | Self-hosted | Hosted (Tangle Intelligence) |
+|---|---|---|
+| Cost | Your LLM bills + your compute | Same LLM bills + hosted-tier subscription |
+| Dashboard | You build it (or use the OSS examples) | Renders InsightReport out of the box |
+| Cron / scheduling | Your CI / cron / GitHub Action | Managed scheduler runs weekly |
+| Slack / email digest | You wire it | Included |
+| Multi-week trends | You persist | Persisted for you |
+| Decision packet generation | Local (free) | API call (same code; we run it) |
+| Closed-loop campaigns | Local (you pay LLM directly) | Pass-through pricing on LLM, plus per-campaign fee |
+| Auto-PR | Your GitHub token | Your GitHub token via OAuth |
+
+Both work end-to-end. Hosted tier is convenience; self-hosted is fine for engineering-heavy teams who want full control.
+
+## Common foreign-stack questions
+
+**Q: We use vLLM / Ollama / a custom self-hosted LLM. Does the closed-loop driver work?**
+A: Yes if your server speaks OpenAI-compat (most do). Pass `baseUrl: 'http://localhost:8000/v1'` (or wherever) and your dummy `apiKey`. We've shipped customers running selfImprove against local LMStudio + Ollama.
+
+**Q: We're a Python shop, not TypeScript. Does anything change?**
+A: `agent-eval-rpc@0.53.0` on PyPI speaks the same wire protocol. The Python client is a thin wrapper around the hosted endpoints — same `analyzeRuns()` / `selfImprove()` calls, same `InsightReport` shape, same `gateDecision` values.
+
+**Q: We have an extremely custom agent (not LLM-call-shaped). Can we still use this?**
+A: Yes. The substrate doesn't care what your agent IS — it only cares that you can express your runs as `RunRecord[]` and your judge as `(artifact) → JudgeScore`. RL-trained agents, multi-step plan-and-execute, browser-driving agents, code-generating agents — all map cleanly.
+
+**Q: What's the minimum cost to try it?**
+A: Free. `analyzeRuns()` is deterministic, runs locally, $0 LLM cost. You can ingest your last week of traces and get a real decision packet without spending a cent. The LLM-cost-incurring step is `selfImprove()` and you set the ceiling.
+
+**Q: We don't want to send traces to your hosted tier. Self-hosted only — works?**
+A: Yes. Every primitive in this doc runs locally. The package is MIT-licensed, no SaaS lock-in, no required network call.

package/docs/pilot/integration-tangle-stack.md

@@ -0,0 +1,248 @@
+# Integration — Tangle Intelligence on the Tangle stack (sandbox + tcloud)
+
+Step-by-step. This is what we run with you on the onboarding call.
+
+## Zero-setup demo first (30 seconds, no install)
+
+```sh
+npx @tangle-network/intelligence demo
+```
+
+End-to-end loop against synthetic data — agent + judge + scenarios + selfImprove. Prints the `InsightReport` shape you'll get on your real data. Useful to confirm the output is what you want before any integration. Hosted equivalent: open **[staging-intelligence.tangle.tools](https://staging-intelligence.tangle.tools)**.
+
+## Prerequisites you already have
+
+- `@tangle-network/sandbox` running your agent in a session
+- `@tangle-network/tcloud` for LLM routing (or any OpenAI-compat router)
+- Your scenarios (the inputs your agent handles) listed somewhere — even as YAML or a TS array
+- A judge function for scoring outputs — LLM-as-judge is fine for v1
+
+## Install
+
+The CLI scaffolds and runs everything; you only add the substrate package if your code calls primitives directly:
+
+```sh
+# CLI (zero-install via npx, or add to your repo as a dev-dep)
+npx @tangle-network/intelligence init
+
+# Optional — only if your code imports analyzeRuns / selfImprove directly
+pnpm add @tangle-network/agent-eval
+# or for Python customers:
+pip install agent-eval-rpc
+```
+
+`@tangle-network/intelligence` is the customer-facing CLI + hosted product (binary `tangle-intel`). `@tangle-network/agent-eval` is the substrate it wraps — install only if you want to script directly against the primitives.
+
+## Step 1 — Ingest your trace stream
+
+You already emit traces via sandbox sessions. Pull them into canonical `RunRecord[]`:
+
+```ts
+import { fromTangleSandbox } from '@tangle-network/agent-eval/adapters/sandbox'
+
+const runs = await fromTangleSandbox({
+  sessionIds: ['session_abc', 'session_def'],   // your current week
+  fromMs: lastReportTime,
+  toMs: Date.now(),
+})
+// runs is RunRecord[] — canonical wire shape, ready for any downstream substrate primitive
+```
+
+If your agent emits OTel directly instead of going through `@tangle-network/sandbox`:
+
+```ts
+import { fromOtelSpans } from '@tangle-network/agent-eval'
+
+const runs = fromOtelSpans({ spans: yourOtelSpans })
+```
+
+## Step 2 — Get the decision packet (no LLM cost)
+
+```ts
+import { analyzeRuns } from '@tangle-network/agent-eval/contract'
+
+const report = await analyzeRuns({
+  runs: thisWeek,
+  baselineRuns: lastWeek,          // optional — gives you the "did my change help?" answer
+  baselineLabel: 'vs prior 7 days',
+})
+
+console.log(report.composite.mean)                       // overall score
+console.log(report.composite.tailRuns)                   // worst 5 runs by name
+console.log(report.priorPeriodComparison?.improvedMetrics)  // ['composite'] if significantly better
+console.log(report.priorPeriodComparison?.regressedMetrics) // ['cost'] if cost went up significantly
+console.log(report.recommendations)                      // priority-ranked actions
+```
+
+That's the **full deterministic flow** — no LLM, $0 cost, runs in ms.
+
+Render in your dashboard or pipe to Slack:
+
+```ts
+for (const rec of report.recommendations) {
+  if (rec.priority === 'critical') {
+    await slack.post(`🔴 ${rec.title}\n${rec.detail}`)
+  }
+}
+```
+
+## Step 3 — Wire the closed loop (real LLM cost — opt-in)
+
+Pick the surface you want to optimize. For most customers this is the agent's system-prompt addendum:
+
+```ts
+import { selfImprove, gepaDriver } from '@tangle-network/agent-eval/contract'
+
+const result = await selfImprove({
+  scenarios: yourScenarios,         // 20-50 representative inputs
+  agent: async (surface, scenario) => {
+    // Your existing agent invocation, with the substrate-proposed surface
+    // injected as the system-prompt addendum.
+    return await runYourAgent({
+      ...scenario,
+      systemPromptAddendum: surface as string,
+    })
+  },
+  judge: yourJudge,                  // function (artifact) → { composite, dimensions }
+  baselineSurface: currentAddendum,  // the production string today
+  driver: gepaDriver({
+    llm: { apiKey: tcloudKey, baseUrl: 'https://router.tangle.tools/v1' },
+    model: 'anthropic/claude-sonnet-4.6',
+    target: 'agent system-prompt addendum',
+  }),
+  budget: {
+    generations: 3,
+    populationSize: 4,
+    holdoutFraction: 0.3,
+    maxUsd: 25,                      // hard ceiling — refuses to overspend
+  },
+})
+
+console.log(`gate: ${result.gateDecision.kind}`)
+console.log(`lift: ${result.lift.delta.toFixed(3)} CI=[${result.lift.ci95.join(', ')}]`)
+console.log(`cost spent: $${result.totalCostUsd.toFixed(2)}`)
+```
+
+`result.gateDecision` is one of:
+- `ship-substrate` — winner statistically beats baseline; safe to deploy
+- `inconclusive` — CI straddles zero; either run more rollouts or expand corpus
+- `ship-harness` / `merge` — only when `driftPolicy: 'benchmark-branches'` is on (advanced)
+
+## Step 4 — Auto-PR the winner
+
+```ts
+if (result.gateDecision.kind === 'ship-substrate') {
+  await openAutoPr({
+    title: `eval: auto-improve ${target} (composite +${result.lift.delta.toFixed(3)})`,
+    body: `${result.gateDecision.reason}\n\n${formatInsight(result.insight)}`,
+    filePath: 'src/lib/.server/production-loop/prompt-addendum.ts',
+    newContent: result.diff.kind === 'replace' ? result.diff.content : applyDiff(currentAddendum, result.diff),
+  })
+}
+```
+
+We ship `openAutoPr` from `@tangle-network/agent-eval/contract`. It wraps the GitHub PR flow with your existing token.
+
+## The full canonical flow (script you copy and run)
+
+```ts
+// scripts/weekly-improvement.ts — run from a cron / GitHub Action
+
+import { fromTangleSandbox } from '@tangle-network/agent-eval/adapters/sandbox'
+import {
+  analyzeRuns,
+  gepaDriver,
+  openAutoPr,
+  selfImprove,
+} from '@tangle-network/agent-eval/contract'
+import { scenarios } from './eval/scenarios'
+import { judge } from './eval/judges'
+import { runYourAgent } from './src/agent'
+import { PRODUCTION_ADDENDUM } from './src/lib/.server/production-loop/prompt-addendum'
+
+const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
+const twoWeeksAgo = lastWeek - 7 * 24 * 60 * 60 * 1000
+
+const thisWeekRuns = await fromTangleSandbox({ fromMs: lastWeek, toMs: Date.now() })
+const lastWeekRuns = await fromTangleSandbox({ fromMs: twoWeeksAgo, toMs: lastWeek })
+
+// 1. Deterministic packet — always
+const report = await analyzeRuns({
+  runs: thisWeekRuns,
+  baselineRuns: lastWeekRuns,
+  baselineLabel: 'vs prior 7 days',
+})
+
+// 2. Closed loop — only if composite regressed OR we haven't tried in a while
+const shouldRun =
+  report.priorPeriodComparison?.regressedMetrics.includes('composite') ||
+  daysSinceLastImprovement() > 7
+
+if (!shouldRun) {
+  console.log('No regression + recent run; skipping.')
+  process.exit(0)
+}
+
+const result = await selfImprove({
+  scenarios,
+  agent: (surface, scenario) =>
+    runYourAgent({ ...scenario, systemPromptAddendum: surface as string }),
+  judge,
+  baselineSurface: PRODUCTION_ADDENDUM,
+  driver: gepaDriver({
+    llm: { apiKey: process.env.TANGLE_KEY!, baseUrl: 'https://router.tangle.tools/v1' },
+    model: 'anthropic/claude-sonnet-4.6',
+    target: 'production agent system-prompt addendum',
+  }),
+  budget: { generations: 3, populationSize: 4, holdoutFraction: 0.3, maxUsd: 50 },
+})
+
+if (result.gateDecision.kind === 'ship-substrate') {
+  await openAutoPr({
+    title: `eval: auto-improve addendum (composite +${result.lift.delta.toFixed(3)})`,
+    body: renderInsightAsPrBody(result.insight),
+    filePath: 'src/lib/.server/production-loop/prompt-addendum.ts',
+    newContent: result.diff.kind === 'replace' ? result.diff.content : '...',
+  })
+}
+```
+
+## What we'll do together on the onboarding call
+
+1. **Map your existing setup** — where do your traces emit? which sandbox sessions? which scenarios exist already?
+2. **Stub the judge** — even a single dimension is enough to start
+3. **Run a deterministic `analyzeRuns()` against your live data** — first decision packet rendered live
+4. **Wire one selfImprove cycle** — small budget, single generation, see the loop fire
+5. **Schedule the cron + auto-PR target** — the loop runs autonomously thereafter
+
+Time budget: ~90 minutes. By the end you have a working pilot.
+
+## What our hosted tier adds on top
+
+- Decision packet rendered weekly in the Intelligence dashboard — no code changes needed
+- Slack / email digest on `regressedMetrics`
+- Pareto chart, judge calibration, failure-cluster drilldown in the UI
+- Multi-week trend lines
+- Stripe-billed usage tracking
+
+If you want self-hosted only, every primitive above works locally. The hosted tier is a convenience.
+
+## FAQ
+
+**Q: What's the smallest scenario corpus that gives useful results?**
+A: ~15 scenarios for the deterministic packet (you get distributional stats + recommendations). For `selfImprove`'s held-out gate you want ≥20 since `holdoutFraction: 0.3` reserves 6 for the gate. Below that, the gate often returns `inconclusive`.
+
+**Q: What if my judge isn't reliable yet?**
+A: That's normal. Use multi-rater intake (`fromFeedbackTable`) to get inter-rater agreement (κ) first, then iterate on the judge until raters agree. Substrate has an `interRater` block in InsightReport showing exactly which scenarios raters disagree on.
+
+**Q: What if a `selfImprove` campaign returns `inconclusive`?**
+A: It refused to claim improvement because the CI straddles zero. Either expand the corpus, raise `holdoutFraction`, or run more generations. Better than shipping noise.
+
+**Q: Can I use a non-tcloud LLM provider?**
+A: Yes — `gepaDriver` accepts any `LlmClientOptions` (any OpenAI-compatible endpoint). We default to tcloud because we already have your auth.
+
+**Q: How do I see what changed when the gate ships?**
+A: `result.diff` is a structured patch. We also ship `diffRuns()` separately if you want to compare two campaign outputs.
+
+**Q: What if my agent self-modifies (Hermes / Claude Code skills)?**
+A: This is the offline/online drift case. We have the architecture spec ready (`docs/specs/profile-versioning.md`) but the implementation is gated on a forcing-function experiment. For v0.5x pilots we assume the substrate is the only writer to your agent's optimizable surface.