reeval 0.20.0 → 0.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +841 -0
- package/README.md +15 -15
- package/assets/gateway-models.mjs +65 -31
- package/dist/{adapter-C60z3wLq.js → adapter-Di9H2etg.js} +8 -49
- package/dist/adapter-Di9H2etg.js.map +1 -0
- package/dist/{agent-BmsMXZXx.js → agent-B-6GFDqA.js} +185 -38
- package/dist/agent-B-6GFDqA.js.map +1 -0
- package/dist/{agents-DXuFxtGC.js → agents-KBTYhSSh.js} +385 -116
- package/dist/agents-KBTYhSSh.js.map +1 -0
- package/dist/{app-files-DT5mJyV4.js → app-files-Cmh9FGr5.js} +3 -3
- package/dist/app-files-Cmh9FGr5.js.map +1 -0
- package/dist/{assemble-CE4pHKIa.js → assemble-CBMRIhso.js} +1 -1
- package/dist/{assemble-DM2PNU4R.js → assemble-DNVBTBp_.js} +172 -18
- package/dist/assemble-DNVBTBp_.js.map +1 -0
- package/dist/attach-EbDGKGTO.js +152 -0
- package/dist/attach-EbDGKGTO.js.map +1 -0
- package/dist/{attempt-CAdlwlNu.js → attempt-_dUsblVx.js} +2 -2
- package/dist/{attempt-CAdlwlNu.js.map → attempt-_dUsblVx.js.map} +1 -1
- package/dist/{attempt-host-7VviAJrl.js → attempt-host-DQvynDHk.js} +17 -8
- package/dist/attempt-host-DQvynDHk.js.map +1 -0
- package/dist/boat-Bzz8L-KQ.js +701 -0
- package/dist/boat-Bzz8L-KQ.js.map +1 -0
- package/dist/{boat-BjLqFJIZ.js → boat-Dek8ftIt.js} +20 -8
- package/dist/boat-Dek8ftIt.js.map +1 -0
- package/dist/bridge/index.mjs +6 -1
- package/dist/bundle-DZstHDeW.js +718 -0
- package/dist/bundle-DZstHDeW.js.map +1 -0
- package/dist/{capture-Ccf0Hmyx.js → capture-BkbPsfej.js} +61 -32
- package/dist/capture-BkbPsfej.js.map +1 -0
- package/dist/capture-CUAvk9KM.js +1 -0
- package/dist/{cases-z9E6DCd5.js → cases-CM-MQRQp.js} +133 -19
- package/dist/cases-CM-MQRQp.js.map +1 -0
- package/dist/cases-FvhbM8M5.js +1 -0
- package/dist/{cli-DTuLSlmW.js → cli-BgBBuxxJ.js} +412 -1044
- package/dist/cli-BgBBuxxJ.js.map +1 -0
- package/dist/{cli-spec-Dsr79Qy1.js → cli-spec-B5L9n-DV.js} +199 -35
- package/dist/cli-spec-B5L9n-DV.js.map +1 -0
- package/dist/cli.js +2 -2
- package/dist/client-DMEYR3MA.js +807 -0
- package/dist/client-DMEYR3MA.js.map +1 -0
- package/dist/{command-C2C7P5qo.js → command-BKb8eN_X.js} +10 -10
- package/dist/{command-C2C7P5qo.js.map → command-BKb8eN_X.js.map} +1 -1
- package/dist/{command-DSL7I-SI.js → command-D_w_kDpI.js} +25 -21
- package/dist/command-D_w_kDpI.js.map +1 -0
- package/dist/{command-Ff0T-1Hi.js → command-Dwlh0Vhu.js} +17 -16
- package/dist/command-Dwlh0Vhu.js.map +1 -0
- package/dist/{command-D8oMB2fa.js → command-loXH2jHr.js} +16 -16
- package/dist/{command-D8oMB2fa.js.map → command-loXH2jHr.js.map} +1 -1
- package/dist/{commands-xAOi_VEk.js → commands-Cuf57OYb.js} +174 -40
- package/dist/commands-Cuf57OYb.js.map +1 -0
- package/dist/{commands-Bk9NG1s_.js → commands-Dqli45sR.js} +1 -1
- package/dist/{config-SvK8pjEK.js → config-C0RKt00M.js} +288 -106
- package/dist/config-C0RKt00M.js.map +1 -0
- package/dist/{context-BdgCao9Q.js → context-DrEAtUyg.js} +2 -2
- package/dist/{context-BdgCao9Q.js.map → context-DrEAtUyg.js.map} +1 -1
- package/dist/{cost-BdRgySQl.js → cost-C21gGrju.js} +19 -9
- package/dist/cost-C21gGrju.js.map +1 -0
- package/dist/cost-IM80LyGD.js +1 -0
- package/dist/{cursor-catalog-DEzan4IT.js → cursor-catalog-CRsFxorF.js} +4 -4
- package/dist/{cursor-catalog-DEzan4IT.js.map → cursor-catalog-CRsFxorF.js.map} +1 -1
- package/dist/db-3o8s2EyD.js +2 -0
- package/dist/{db-BzOJ4Js4.js → db-ErnJ-gMD.js} +168 -248
- package/dist/db-ErnJ-gMD.js.map +1 -0
- package/dist/debug-CIXzB4wK.js.map +1 -1
- package/dist/{default-effort-BDKIxJyC.js → default-effort-D70cTG00.js} +4 -4
- package/dist/{default-effort-BDKIxJyC.js.map → default-effort-D70cTG00.js.map} +1 -1
- package/dist/{default-effort-BBDPbJ4D.js → default-effort-DHzj84dD.js} +1 -1
- package/dist/{dist-Db7ifchA.js → dist-DeqeCUZe.js} +5 -5
- package/dist/{dist-Db7ifchA.js.map → dist-DeqeCUZe.js.map} +1 -1
- package/dist/{docker-DFrjoNs-.js → docker-CFWzZOEk.js} +4 -4
- package/dist/{docker-DFrjoNs-.js.map → docker-CFWzZOEk.js.map} +1 -1
- package/dist/docker-DrMnpL84.js +1 -0
- package/dist/{doctor-DWK4ttex.js → doctor-B7utCSdh.js} +121 -100
- package/dist/doctor-B7utCSdh.js.map +1 -0
- package/dist/{environment-cPMCOk-h.js → environment-C0Gg8kyN.js} +49 -11
- package/dist/environment-C0Gg8kyN.js.map +1 -0
- package/dist/{errors-BZgLSZtV.js → errors-C1GBcWE2.js} +2 -2
- package/dist/{errors-BZgLSZtV.js.map → errors-C1GBcWE2.js.map} +1 -1
- package/dist/{eval-source-C9b1h8L6.js → eval-source-CsW5VIXS.js} +24 -12
- package/dist/eval-source-CsW5VIXS.js.map +1 -0
- package/dist/{evaluate-B12Tosvq.js → evaluate-BHQyAszQ.js} +679 -142
- package/dist/evaluate-BHQyAszQ.js.map +1 -0
- package/dist/{evaluate-COdzf7Hl.d.ts → evaluate-DzHq4EsU.d.ts} +2 -2
- package/dist/fake-bootstrap-VpisF1WE.js +15 -0
- package/dist/fake-bootstrap-VpisF1WE.js.map +1 -0
- package/dist/{files-tiNWe8Ih.js → files-BCraBodb.js} +3 -3
- package/dist/{files-tiNWe8Ih.js.map → files-BCraBodb.js.map} +1 -1
- package/dist/files-BjOJw4sn.js +1 -0
- package/dist/{files-DgXyaAg6.js → files-BrGVs4gY.js} +2 -2
- package/dist/{files-DgXyaAg6.js.map → files-BrGVs4gY.js.map} +1 -1
- package/dist/{files-BOBdC0uI.js → files-BuKWZCzY.js} +6 -6
- package/dist/{files-BOBdC0uI.js.map → files-BuKWZCzY.js.map} +1 -1
- package/dist/files-CAzVG3Rr.js +1 -0
- package/dist/{files-F5EG0WPT.js → files-Cv856jLN.js} +5 -5
- package/dist/{files-F5EG0WPT.js.map → files-Cv856jLN.js.map} +1 -1
- package/dist/files-D-Lir4aE.js +1 -0
- package/dist/files-DRlJ0Xr2.js +1 -0
- package/dist/{files-DlQshsks.js → files-g1watgRC.js} +2 -2
- package/dist/{files-DlQshsks.js.map → files-g1watgRC.js.map} +1 -1
- package/dist/{files-s28a6czq.js → files-qjeqXnfE.js} +143 -20
- package/dist/files-qjeqXnfE.js.map +1 -0
- package/dist/gateway.d.ts +1 -1
- package/dist/{git-BpoqEgDV.js → git-DOrOtAiD.js} +3 -3
- package/dist/{git-BpoqEgDV.js.map → git-DOrOtAiD.js.map} +1 -1
- package/dist/{harness-DXv2gnaT.js → harness-FcTyyPco.js} +17 -8
- package/dist/harness-FcTyyPco.js.map +1 -0
- package/dist/harness-install-Dxho1yaU.js +53 -0
- package/dist/harness-install-Dxho1yaU.js.map +1 -0
- package/dist/{harness-C2berMiM.js → harness-nwnyghJZ.js} +14 -11
- package/dist/{harness-C2berMiM.js.map → harness-nwnyghJZ.js.map} +1 -1
- package/dist/harness.d.ts +11 -8
- package/dist/harness.js +1 -1
- package/dist/{ids-C_advdJA.js → ids-B-JKTCP8.js} +2 -2
- package/dist/{ids-C_advdJA.js.map → ids-B-JKTCP8.js.map} +1 -1
- package/dist/{images-D2jFze2v.js → images-Ch1y1t6_.js} +122 -73
- package/dist/images-Ch1y1t6_.js.map +1 -0
- package/dist/index.d.ts +46 -7
- package/dist/index.js +6 -6
- package/dist/{init-Bi1Q3lRi.js → init-C2HqCwnt.js} +33 -14
- package/dist/init-C2HqCwnt.js.map +1 -0
- package/dist/init-CM9xwyre.js +1 -0
- package/dist/{ink-list-BpfidQ1h.js → ink-list-BbKXElcQ.js} +54 -15
- package/dist/ink-list-BbKXElcQ.js.map +1 -0
- package/dist/inputs-5UIDpjz-.js +190 -0
- package/dist/inputs-5UIDpjz-.js.map +1 -0
- package/dist/inputs-load-C82HOKnb.js +1 -0
- package/dist/inputs-load-CGPyigMC.js +95 -0
- package/dist/inputs-load-CGPyigMC.js.map +1 -0
- package/dist/{invocations-CxGNOlh0.js → invocations-BrZAMuTb.js} +5 -5
- package/dist/{invocations-CxGNOlh0.js.map → invocations-BrZAMuTb.js.map} +1 -1
- package/dist/jiti-radm1Soi.js +52 -0
- package/dist/jiti-radm1Soi.js.map +1 -0
- package/dist/{judge-COMXT1Hx.js → judge-DwzqucLS.js} +17 -12
- package/dist/{judge-COMXT1Hx.js.map → judge-DwzqucLS.js.map} +1 -1
- package/dist/judge.d.ts +3 -1
- package/dist/judge.js +1 -1
- package/dist/{judgments-CzgpsTqv.js → judgments-Cj8bFaDG.js} +2 -2
- package/dist/{judgments-CzgpsTqv.js.map → judgments-Cj8bFaDG.js.map} +1 -1
- package/dist/{layout-Cm85tQYk.js → layout-Czh5wOpz.js} +14 -11
- package/dist/layout-Czh5wOpz.js.map +1 -0
- package/dist/{list-DuThsE28.js → list-B_C0k5ex.js} +33 -19
- package/dist/{list-DuThsE28.js.map → list-B_C0k5ex.js.map} +1 -1
- package/dist/live-C9Et_BVT.js +1 -0
- package/dist/{live-CgYbO4Te.js → live-DGgIrVw0.js} +7 -5
- package/dist/live-DGgIrVw0.js.map +1 -0
- package/dist/{live-DL0se4Zd.js → live-DgFm0H90.js} +9 -9
- package/dist/{live-DL0se4Zd.js.map → live-DgFm0H90.js.map} +1 -1
- package/dist/machine-service-2lIv5UWF.js +1 -0
- package/dist/machine-service-Ct_1XMMi.js +520 -0
- package/dist/machine-service-Ct_1XMMi.js.map +1 -0
- package/dist/machines-BnquHcBZ.js +120 -0
- package/dist/machines-BnquHcBZ.js.map +1 -0
- package/dist/{model-CkNVAxob.js → model-B1tdv9-d.js} +1 -1
- package/dist/model-Bcy4Qjmm.js +1 -0
- package/dist/{model-V3mMXp6x.js → model-DF77SgZv.js} +6 -6
- package/dist/{model-V3mMXp6x.js.map → model-DF77SgZv.js.map} +1 -1
- package/dist/{model-CwwkrCb7.js → model-DRa7yv6K.js} +9 -6
- package/dist/model-DRa7yv6K.js.map +1 -0
- package/dist/{model-cgt5uvu8.js → model-E5X8YBJM.js} +7 -6
- package/dist/model-E5X8YBJM.js.map +1 -0
- package/dist/model-calls-B0BoQfJ4.js +95 -0
- package/dist/model-calls-B0BoQfJ4.js.map +1 -0
- package/dist/{node-D7pAU--0.js → node-BjuhqQwa.js} +21 -10
- package/dist/node-BjuhqQwa.js.map +1 -0
- package/dist/{opencode-providers-BGiXt8_B.js → opencode-providers-ChJTQRmz.js} +5 -2
- package/dist/{opencode-providers-BGiXt8_B.js.map → opencode-providers-ChJTQRmz.js.map} +1 -1
- package/dist/{openrouter-models-Cw1z8-sq.js → openrouter-models-B1sIOTD2.js} +2 -2
- package/dist/{openrouter-models-Cw1z8-sq.js.map → openrouter-models-B1sIOTD2.js.map} +1 -1
- package/dist/options-C989kuVE.js +1 -0
- package/dist/own-reads-CYmBlrHK.js +15 -0
- package/dist/own-reads-CYmBlrHK.js.map +1 -0
- package/dist/{patch-DAi1nCuH.js → patch-BR8rdl7r.js} +2 -2
- package/dist/{patch-DAi1nCuH.js.map → patch-BR8rdl7r.js.map} +1 -1
- package/dist/pi-providers-HygmL1y-.js +552 -0
- package/dist/pi-providers-HygmL1y-.js.map +1 -0
- package/dist/{plain-BYUf_fUV.js → plain-Ctn8oUdp.js} +44 -10
- package/dist/plain-Ctn8oUdp.js.map +1 -0
- package/dist/{plan-CsPilzpk.js → plan-BcT4G5jj.js} +104 -15
- package/dist/plan-BcT4G5jj.js.map +1 -0
- package/dist/{plugins-BfnhazOv.js → plugins-cC0sqoDA.js} +608 -12
- package/dist/plugins-cC0sqoDA.js.map +1 -0
- package/dist/preflight-BMVDBcux.js +1 -0
- package/dist/{preflight-BixaE4BA.js → preflight-C9OtTb7k.js} +4 -4
- package/dist/{preflight-BixaE4BA.js.map → preflight-C9OtTb7k.js.map} +1 -1
- package/dist/{preflight-GkpgrBxP.js → preflight-DySHXeFJ.js} +9 -10
- package/dist/preflight-DySHXeFJ.js.map +1 -0
- package/dist/{preflight-D1C-iH0U.js → preflight-zsAn3heN.js} +1 -1
- package/dist/{prepare-C--rM_wO.js → prepare-MaRWRhNT.js} +2 -2
- package/dist/prepare-nbX1CXXg.js +409 -0
- package/dist/prepare-nbX1CXXg.js.map +1 -0
- package/dist/prices-B1vm_lPk.js +1 -0
- package/dist/{prices-5m3SzMZ3.js → prices-BrY_ihLs.js} +80 -22
- package/dist/prices-BrY_ihLs.js.map +1 -0
- package/dist/{provenance-BXyYwUC8.js → provenance-CkykifD9.js} +5 -5
- package/dist/{provenance-BXyYwUC8.js.map → provenance-CkykifD9.js.map} +1 -1
- package/dist/provider-DHT20Gl8.js +1 -0
- package/dist/{provider-BxzZq3Th.js → provider-ieWOn4YS.js} +2 -2
- package/dist/{provider-BxzZq3Th.js.map → provider-ieWOn4YS.js.map} +1 -1
- package/dist/provider-stop-CxzWW7Ft.js +69 -0
- package/dist/provider-stop-CxzWW7Ft.js.map +1 -0
- package/dist/providers-BkLGzaSY.js +1 -0
- package/dist/{providers-1p4HKYIp.js → providers-bNgfKa6m.js} +224 -49
- package/dist/providers-bNgfKa6m.js.map +1 -0
- package/dist/query-B2SQddoV.js +1 -0
- package/dist/{query-DhhU9ADI.js → query-DhV3a8lM.js} +2 -2
- package/dist/{query-DhhU9ADI.js.map → query-DhV3a8lM.js.map} +1 -1
- package/dist/{questions-CMAq4pr8.js → questions-Csv64WCb.js} +7 -7
- package/dist/{questions-CMAq4pr8.js.map → questions-Csv64WCb.js.map} +1 -1
- package/dist/{record-6IUzwP9K.js → record--vHreTLk.js} +11 -5
- package/dist/record--vHreTLk.js.map +1 -0
- package/dist/{redact-BhjtnjrU.js → redact-t-L3VGy8.js} +106 -2
- package/dist/redact-t-L3VGy8.js.map +1 -0
- package/dist/{remarks-D9Lcd1fF.js → remarks-BpdM_2RC.js} +2 -1
- package/dist/remarks-BpdM_2RC.js.map +1 -0
- package/dist/{remarks-BNABzemt.js → remarks-CYFOsnz5.js} +46 -11
- package/dist/remarks-CYFOsnz5.js.map +1 -0
- package/dist/remarks-CleoDlYw.js +1 -0
- package/dist/{remarks-DPCONFxZ.js → remarks-DqriPKxm.js} +3 -2
- package/dist/{remarks-DPCONFxZ.js.map → remarks-DqriPKxm.js.map} +1 -1
- package/dist/{remote-U7fNt0UP.js → remote-0CHog7Oq.js} +6 -6
- package/dist/{remote-U7fNt0UP.js.map → remote-0CHog7Oq.js.map} +1 -1
- package/dist/resolve-Bm9ZyFR-.js +1 -0
- package/dist/resolve-Bul8lHaa.js +205 -0
- package/dist/resolve-Bul8lHaa.js.map +1 -0
- package/dist/{resolve-Cx3a9UQ2.js → resolve-DorSHDBs.js} +14 -11
- package/dist/resolve-DorSHDBs.js.map +1 -0
- package/dist/resources-DqdwihhT.js +62 -0
- package/dist/resources-DqdwihhT.js.map +1 -0
- package/dist/{results-CQKXmHym.js → results-DhMgRkR0.js} +9 -22
- package/dist/results-DhMgRkR0.js.map +1 -0
- package/dist/{resume-POpZa1ge.js → resume-C8Ihfy-C.js} +166 -17
- package/dist/resume-C8Ihfy-C.js.map +1 -0
- package/dist/{retry-CmG809Mp.js → retry-Bfk7pDti.js} +4 -4
- package/dist/{retry-CmG809Mp.js.map → retry-Bfk7pDti.js.map} +1 -1
- package/dist/{root-Co0jhOT5.js → root-BzeoyfGd.js} +2 -2
- package/dist/{root-Co0jhOT5.js.map → root-BzeoyfGd.js.map} +1 -1
- package/dist/routes-Cp4gAjBZ.js +427 -0
- package/dist/routes-Cp4gAjBZ.js.map +1 -0
- package/dist/{runtime-Xm9oeC7o.js → runtime-CPUH0Vk5.js} +46 -5
- package/dist/runtime-CPUH0Vk5.js.map +1 -0
- package/dist/sandbox/boat.d.ts +1 -1
- package/dist/sandbox/boat.js +1 -1
- package/dist/{sandbox-DrlL4yqb.js → sandbox-z8q_4Ywn.js} +3 -3
- package/dist/{sandbox-DrlL4yqb.js.map → sandbox-z8q_4Ywn.js.map} +1 -1
- package/dist/sandbox.d.ts +1 -1
- package/dist/sandbox.js +1 -1
- package/dist/{server-C00zd6zl.js → server-BsQZtuF9.js} +9 -9
- package/dist/{server-C00zd6zl.js.map → server-BsQZtuF9.js.map} +1 -1
- package/dist/server-ZaMzUNmV.js +1 -0
- package/dist/services-BG8xuU82.js +39 -0
- package/dist/services-BG8xuU82.js.map +1 -0
- package/dist/{shared-Cfi45Bvy.js → shared-DtZwIAfy.js} +7 -1
- package/dist/shared-DtZwIAfy.js.map +1 -0
- package/dist/{source-DDgr48Gn.js → source-9HWaTEgV.js} +2 -2
- package/dist/{source-DDgr48Gn.js.map → source-9HWaTEgV.js.map} +1 -1
- package/dist/source-BcBDj-R0.js +1 -0
- package/dist/src-B2eHTCnp.js +3 -0
- package/dist/{src-DAzaCTcN.js → src-B5r1d23Y.js} +176 -18
- package/dist/src-B5r1d23Y.js.map +1 -0
- package/dist/{src-Dw79VLa3.js → src-Bb3k9Kn_.js} +208 -59
- package/dist/src-Bb3k9Kn_.js.map +1 -0
- package/dist/src-BfOlH_Bu.js +2 -0
- package/dist/src-BuGc40IF.js +2 -0
- package/dist/{src-CV_7DkPa.js → src-C-ChdnG6.js} +102 -13
- package/dist/src-C-ChdnG6.js.map +1 -0
- package/dist/src-C2K5_afU.js +2 -0
- package/dist/{src-YvblOrBU.js → src-C3OhA5pw.js} +87 -13
- package/dist/src-C3OhA5pw.js.map +1 -0
- package/dist/src-CSVJjghq.js +4 -0
- package/dist/{src-y0nOeRRE.js → src-CWfy8ct8.js} +1572 -487
- package/dist/src-CWfy8ct8.js.map +1 -0
- package/dist/src-CcrQQB0_.js +2 -0
- package/dist/src-CotUxxQV.js +2 -0
- package/dist/src-D-PwBaBB.js +2 -0
- package/dist/{src-Cb2MWozp.js → src-DNWiU__5.js} +276 -41
- package/dist/src-DNWiU__5.js.map +1 -0
- package/dist/{src-KxS91P7L.js → src-DwcTn4JY.js} +79 -11
- package/dist/src-DwcTn4JY.js.map +1 -0
- package/dist/{src-BNQznj4z.js → src-boB6D-5T.js} +124 -45
- package/dist/src-boB6D-5T.js.map +1 -0
- package/dist/{src-DvGns8cz.js → src-c3ZQu7Uu.js} +299 -17
- package/dist/src-c3ZQu7Uu.js.map +1 -0
- package/dist/src-jYqNkT-F.js +1 -0
- package/dist/start-Db5VRcRZ.js +660 -0
- package/dist/start-Db5VRcRZ.js.map +1 -0
- package/dist/state-CKIjZZRK.js +1 -0
- package/dist/{state-328JpnHA.js → state-DUYdzNA7.js} +21 -21
- package/dist/{state-328JpnHA.js.map → state-DUYdzNA7.js.map} +1 -1
- package/dist/store/local.d.ts +8 -0
- package/dist/store/local.js +16 -0
- package/dist/store/local.js.map +1 -0
- package/dist/store/s3.d.ts +10 -0
- package/dist/store/s3.js +18 -0
- package/dist/store/s3.js.map +1 -0
- package/dist/suite-CH1aaAbs.js +2 -0
- package/dist/{suite-COglUhPd.js → suite-DjGxmDSY.js} +6 -5
- package/dist/{suite-COglUhPd.js.map → suite-DjGxmDSY.js.map} +1 -1
- package/dist/{summary-DDy9DfyJ.js → summary-BK5Sa_N0.js} +28 -7
- package/dist/summary-BK5Sa_N0.js.map +1 -0
- package/dist/sync-DUi_P2UJ.js +559 -0
- package/dist/sync-DUi_P2UJ.js.map +1 -0
- package/dist/{tarball-CESxtK4X.js → tarball-C8BSTAMk.js} +2 -2
- package/dist/{tarball-CESxtK4X.js.map → tarball-C8BSTAMk.js.map} +1 -1
- package/dist/tart-BQ_Ap0l4.js +508 -0
- package/dist/tart-BQ_Ap0l4.js.map +1 -0
- package/dist/tart-CzViW3WZ.js +186 -0
- package/dist/tart-CzViW3WZ.js.map +1 -0
- package/dist/tart-DmVuVsmX.js +1 -0
- package/dist/tart-bake-D1RttYfM.js +148 -0
- package/dist/tart-bake-D1RttYfM.js.map +1 -0
- package/dist/tart-bake-DjFcsG3N.js +1 -0
- package/dist/tart-cli-BOcluha1.js +1 -0
- package/dist/tart-cli-gA9xEaQL.js +89 -0
- package/dist/tart-cli-gA9xEaQL.js.map +1 -0
- package/dist/tart-hosts-CHNrfV6u.js +313 -0
- package/dist/tart-hosts-CHNrfV6u.js.map +1 -0
- package/dist/tart-hosts-CWse-ASo.js +1 -0
- package/dist/{tart-images-DZ_Mbh-g.js → tart-images-CLBjfF9u.js} +64 -6
- package/dist/tart-images-CLBjfF9u.js.map +1 -0
- package/dist/testing.d.ts +36 -2
- package/dist/testing.js +4 -4
- package/dist/tool-class-krsmEo44.js +433 -0
- package/dist/tool-class-krsmEo44.js.map +1 -0
- package/dist/{trace-Dx2NBE1O.js → trace-BpXByD7W.js} +13 -2
- package/dist/{trace-Dx2NBE1O.js.map → trace-BpXByD7W.js.map} +1 -1
- package/dist/{traits-DKnYkqeK.js → traits-CCD8XsmI.js} +2 -2
- package/dist/{traits-DKnYkqeK.js.map → traits-CCD8XsmI.js.map} +1 -1
- package/dist/{transcript-CCf07UmU.js → transcript-BXWZiMuC.js} +3 -3
- package/dist/{transcript-CCf07UmU.js.map → transcript-BXWZiMuC.js.map} +1 -1
- package/dist/{types-C2mLpph3.js → types-CbcnJxCy.js} +7 -1
- package/dist/types-CbcnJxCy.js.map +1 -0
- package/dist/{types-DBwFsGem.d.ts → types-DKK8jj--.d.ts} +517 -45
- package/dist/ui-DKYjTVpn.js +1455 -0
- package/dist/ui-DKYjTVpn.js.map +1 -0
- package/dist/{usage-DJiDnh2D.js → usage-CwIRPPZ9.js} +16 -3
- package/dist/usage-CwIRPPZ9.js.map +1 -0
- package/dist/vercel-models-Ci33llXz.js +21 -0
- package/dist/vercel-models-Ci33llXz.js.map +1 -0
- package/dist/{vertex-Eb2wqahk.js → vertex-BEWxHoxH.js} +22 -6
- package/dist/{vertex-Eb2wqahk.js.map → vertex-BEWxHoxH.js.map} +1 -1
- package/dist/vertex-DXfcXYKn.js +1 -0
- package/dist/view/index.d.ts +32 -3
- package/dist/view/index.js +8 -8
- package/dist/{views-BROacjBv.js → views-B1zVLJsF.js} +6 -6
- package/dist/{views-BROacjBv.js.map → views-B1zVLJsF.js.map} +1 -1
- package/dist/views-DCjtw1jF.js +2 -0
- package/dist/{worker-entry-Smb4bNDc.js → worker-entry-DrAYYt0M.js} +37 -14
- package/dist/worker-entry-DrAYYt0M.js.map +1 -0
- package/dist/workspace-CnRbV8cM.js +1 -0
- package/dist/{workspace-DNDTEk8v.js → workspace-DwTe1zo6.js} +7 -7
- package/dist/{workspace-DNDTEk8v.js.map → workspace-DwTe1zo6.js.map} +1 -1
- package/docker/Dockerfile +1 -1
- package/package.json +15 -1
- package/skills/eval-rubric/SKILL.md +10 -0
- package/skills/reeval/SKILL.md +370 -0
- package/skills/reeval/references/capture.md +48 -0
- package/skills/reeval/references/cli.md +162 -0
- package/skills/reeval/references/compare.md +180 -0
- package/skills/reeval/references/config.md +147 -0
- package/skills/reeval/references/environments.md +218 -0
- package/skills/reeval/references/evals.md +241 -0
- package/skills/reeval/references/mistakes.md +39 -0
- package/skills/reeval/references/vocabulary.md +19 -0
- package/skills/setup/SKILL.md +219 -0
- package/skills/setup/references/templates.md +258 -0
- package/dist/adapter-C60z3wLq.js.map +0 -1
- package/dist/agent-BmsMXZXx.js.map +0 -1
- package/dist/agents-DXuFxtGC.js.map +0 -1
- package/dist/app-files-DT5mJyV4.js.map +0 -1
- package/dist/assemble-DM2PNU4R.js.map +0 -1
- package/dist/attempt-host-7VviAJrl.js.map +0 -1
- package/dist/boat-BjLqFJIZ.js.map +0 -1
- package/dist/boat-DSNgZt-t.js +0 -322
- package/dist/boat-DSNgZt-t.js.map +0 -1
- package/dist/capture-BzotD2hs.js +0 -1
- package/dist/capture-Ccf0Hmyx.js.map +0 -1
- package/dist/cases-Dy0mifal.js +0 -1
- package/dist/cases-z9E6DCd5.js.map +0 -1
- package/dist/cli-DTuLSlmW.js.map +0 -1
- package/dist/cli-spec-Dsr79Qy1.js.map +0 -1
- package/dist/client-KM4C5r5_.js +0 -387
- package/dist/client-KM4C5r5_.js.map +0 -1
- package/dist/command-DSL7I-SI.js.map +0 -1
- package/dist/command-Ff0T-1Hi.js.map +0 -1
- package/dist/commands-xAOi_VEk.js.map +0 -1
- package/dist/config-SvK8pjEK.js.map +0 -1
- package/dist/cost-BdRgySQl.js.map +0 -1
- package/dist/cost-Cr26Alhq.js +0 -1
- package/dist/db-BzOJ4Js4.js.map +0 -1
- package/dist/db-CQRZ3URX.js +0 -2
- package/dist/docker--mukta3L.js +0 -1
- package/dist/docker-DG_uFgn2.js +0 -246
- package/dist/docker-DG_uFgn2.js.map +0 -1
- package/dist/doctor-DWK4ttex.js.map +0 -1
- package/dist/environment-cPMCOk-h.js.map +0 -1
- package/dist/eval-source-C9b1h8L6.js.map +0 -1
- package/dist/evaluate-B12Tosvq.js.map +0 -1
- package/dist/files-C1Rk9FYA.js +0 -1
- package/dist/files-DMQ9-Vv5.js +0 -1
- package/dist/files-Di0xDbCc.js +0 -1
- package/dist/files-G5jf8wBp.js +0 -1
- package/dist/files-s28a6czq.js.map +0 -1
- package/dist/harness-DXv2gnaT.js.map +0 -1
- package/dist/images-D2jFze2v.js.map +0 -1
- package/dist/init-Bi1Q3lRi.js.map +0 -1
- package/dist/init-CUw65o36.js +0 -1
- package/dist/ink-list-BpfidQ1h.js.map +0 -1
- package/dist/kinds-Bh2I35id.js +0 -698
- package/dist/kinds-Bh2I35id.js.map +0 -1
- package/dist/layout-Cm85tQYk.js.map +0 -1
- package/dist/live-CgYbO4Te.js.map +0 -1
- package/dist/live-DZLmj0iG.js +0 -1
- package/dist/matchers-CXLn-1jm.js +0 -148
- package/dist/matchers-CXLn-1jm.js.map +0 -1
- package/dist/model-Cv4GuTsZ.js +0 -1
- package/dist/model-CwwkrCb7.js.map +0 -1
- package/dist/model-cgt5uvu8.js.map +0 -1
- package/dist/node-D7pAU--0.js.map +0 -1
- package/dist/options-DPUmk7v_.js +0 -1
- package/dist/plain-BYUf_fUV.js.map +0 -1
- package/dist/plan-CsPilzpk.js.map +0 -1
- package/dist/plugins-BfnhazOv.js.map +0 -1
- package/dist/preflight-Bn3ianye.js +0 -1
- package/dist/preflight-GkpgrBxP.js.map +0 -1
- package/dist/prepare-DQEDXoDp.js +0 -268
- package/dist/prepare-DQEDXoDp.js.map +0 -1
- package/dist/prices-5m3SzMZ3.js.map +0 -1
- package/dist/provider-BnUoz4KB.js +0 -1
- package/dist/providers-1p4HKYIp.js.map +0 -1
- package/dist/providers-B-ijCktf.js +0 -1
- package/dist/query-BYhyf0LR.js +0 -1
- package/dist/record-6IUzwP9K.js.map +0 -1
- package/dist/redact-BhjtnjrU.js.map +0 -1
- package/dist/remarks-BNABzemt.js.map +0 -1
- package/dist/remarks-CAvTSq16.js +0 -1
- package/dist/remarks-D9Lcd1fF.js.map +0 -1
- package/dist/resolve-C14Pmtej.js +0 -1
- package/dist/resolve-Cx3a9UQ2.js.map +0 -1
- package/dist/results-CQKXmHym.js.map +0 -1
- package/dist/resume-POpZa1ge.js.map +0 -1
- package/dist/runtime-Xm9oeC7o.js.map +0 -1
- package/dist/server-C1UJcpUh.js +0 -1
- package/dist/shared-Cfi45Bvy.js.map +0 -1
- package/dist/source-DJHmQv_j.js +0 -1
- package/dist/src-1dBuXzs4.js +0 -3
- package/dist/src-BKpioayC.js +0 -2
- package/dist/src-BNQznj4z.js.map +0 -1
- package/dist/src-BY1cCaoJ.js +0 -2
- package/dist/src-BbZYaQ8b.js +0 -2
- package/dist/src-CDFH2vVN.js +0 -2
- package/dist/src-CMW5R9Q0.js +0 -1
- package/dist/src-CV_7DkPa.js.map +0 -1
- package/dist/src-Cb2MWozp.js.map +0 -1
- package/dist/src-CrV2WA2i.js +0 -2
- package/dist/src-CsFaawcZ.js +0 -2
- package/dist/src-DAzaCTcN.js.map +0 -1
- package/dist/src-DvGns8cz.js.map +0 -1
- package/dist/src-Dw79VLa3.js.map +0 -1
- package/dist/src-KxS91P7L.js.map +0 -1
- package/dist/src-YvblOrBU.js.map +0 -1
- package/dist/src-wKwznPgt.js +0 -2
- package/dist/src-y0nOeRRE.js.map +0 -1
- package/dist/state-Ca55Gk_O.js +0 -1
- package/dist/suite-CKBa6MKt.js +0 -2
- package/dist/summary-DDy9DfyJ.js.map +0 -1
- package/dist/tart-cli-C-GsbVsQ.js +0 -1
- package/dist/tart-cli-CU0nFS-L.js +0 -55
- package/dist/tart-cli-CU0nFS-L.js.map +0 -1
- package/dist/tart-dWoJvWHc.js +0 -586
- package/dist/tart-dWoJvWHc.js.map +0 -1
- package/dist/tart-images-DZ_Mbh-g.js.map +0 -1
- package/dist/tool-target-DS5d72eH.js +0 -40
- package/dist/tool-target-DS5d72eH.js.map +0 -1
- package/dist/types-C2mLpph3.js.map +0 -1
- package/dist/usage-DJiDnh2D.js.map +0 -1
- package/dist/vercel-CGlSBv63.js +0 -344
- package/dist/vercel-CGlSBv63.js.map +0 -1
- package/dist/vertex-O88TCwgY.js +0 -1
- package/dist/views-B0Yt_Q1S.js +0 -2
- package/dist/worker-entry-Smb4bNDc.js.map +0 -1
- package/dist/workspace-C0dDP75l.js +0 -1
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
# The CLI, results and analysis
|
|
2
|
+
|
|
3
|
+
## Contents
|
|
4
|
+
|
|
5
|
+
- Commands
|
|
6
|
+
- From a failure to what the agent did
|
|
7
|
+
- `reeval analyze`
|
|
8
|
+
- Config discovery, filters and exit codes
|
|
9
|
+
- Recording runs
|
|
10
|
+
|
|
11
|
+
## Commands
|
|
12
|
+
|
|
13
|
+
```text
|
|
14
|
+
reeval run [filter...] --dry-run [--json] --runs <n> --tag <t> --trait <k=v> --profile <p>
|
|
15
|
+
--record --ui plain -v
|
|
16
|
+
reeval record reeval history reeval show [run]
|
|
17
|
+
reeval compare [base] [head] [--md] [--fail-on-regression] [-v]
|
|
18
|
+
reeval query lift reeval query effort
|
|
19
|
+
reeval query "SELECT … FROM failing_attempts"
|
|
20
|
+
reeval analyze one checked LLM reading of a run; --vs none for this
|
|
21
|
+
run only; --context for your own agent
|
|
22
|
+
reeval studio reeval studio build --out site
|
|
23
|
+
reeval image build [--dry-run] [--profile <p>] reeval clean
|
|
24
|
+
reeval mcp the history as read-only MCP tools
|
|
25
|
+
reeval doctor Docker, keys and images, with fixes (--json, --online)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## From a failure to what the agent did
|
|
29
|
+
|
|
30
|
+
- `reeval run`'s status list names each failed check with its message and line, and `reeval show` prints the result blocks. Then `.reeval/runs/<run>/summary.md` and each attempt's
|
|
31
|
+
directory (`transcript.jsonl`, `diff.patch`, `checks/<n>.log`, `judge-input.json`,
|
|
32
|
+
`setup.log`). `reeval run -v` prints a timestamped trace per attempt; `.reeval/debug/` keeps it.
|
|
33
|
+
- For a slow run, `reeval show <run> --trace` writes `.reeval/traces/<run>.json`, a Perfetto
|
|
34
|
+
timeline: attempts in lanes with their setup, turns, tool calls, checks and judge.
|
|
35
|
+
- Each attempt's directory in `.reeval/runs/<run>/` has `snapshots/final/`: its whole final
|
|
36
|
+
workspace after the checks (the app the agent built, not only its diff), to read or run. A failed
|
|
37
|
+
attempt's result block in `reeval show` prints its path. `snapshot: 'turns'` on an eval also keeps
|
|
38
|
+
`snapshots/turn-<n>/` after each `c.send`; `snapshot: { compress: true }` writes
|
|
39
|
+
`<name>.tar.gz` instead; `snapshot: false` turns it off. Snapshots leave out `node_modules/`,
|
|
40
|
+
`.git/` and caches, are capped (50 MB as a directory, 10 MB compressed), never go into the
|
|
41
|
+
history folder, and are kept for the last `debug.keep` runs (20 by default) until `reeval clean`.
|
|
42
|
+
An attempt that timed out or was canceled has none.
|
|
43
|
+
- `reeval query "SELECT … FROM failing_attempts"`: each failed attempt and infra error with its
|
|
44
|
+
cell, skill and `first_failed_check` (the label of the first check that failed).
|
|
45
|
+
- `reeval query lift` and `reeval query effort` list every variant cell's lift and effort from
|
|
46
|
+
recorded runs. `reeval query "SELECT eval_key, label, verdict FROM check_discrimination WHERE
|
|
47
|
+
verdict LIKE 'both-%'"` finds checks that don't tell a variant from its control, and
|
|
48
|
+
`reeval query "SELECT cell_key, role, rate FROM check_rates WHERE label = 'flow works'"` is one
|
|
49
|
+
check's pass rate per cell (`pass / (pass + fail)`; errors and not-run left out).
|
|
50
|
+
- `reeval query flaky_cells` lists cells that both passed and failed over comparable runs:
|
|
51
|
+
`verdict` is `flaky` from 6 counted attempts, else `mixed` (too few to call it flaky; raise
|
|
52
|
+
`runs` or record more runs before treating it as flaky).
|
|
53
|
+
- `reeval query skill_files` says which declared files (`SKILL.md`, references, rules) each
|
|
54
|
+
attempt opened, and `context_use` counts its tool calls as `context`, `dependency`
|
|
55
|
+
(`node_modules`, `--help`), `web`, `verify` or `workspace`: a heuristic, read it as counts.
|
|
56
|
+
- `reeval query --schema` lists every view and its columns; `--json` on any query for a program.
|
|
57
|
+
`--run last` (or a run id) binds `:run`: `reeval query failing_attempts --run last`, or
|
|
58
|
+
`reeval query "SELECT … FROM cells WHERE run_id = :run" --run last`. Without it a query spans
|
|
59
|
+
every run.
|
|
60
|
+
- If your client has a `reeval` MCP server (`reeval mcp`), prefer its tools for reading results:
|
|
61
|
+
`failures` is the quickest way from "it failed" to why (failed checks and judge scores, check
|
|
62
|
+
logs, the transcript's tail, and whether the cell failed in earlier runs), far smaller than
|
|
63
|
+
`show` or summary.md; then `transcript` or `compare`. The rest return the same JSON as `--json`.
|
|
64
|
+
|
|
65
|
+
## `reeval analyze`
|
|
66
|
+
|
|
67
|
+
`reeval analyze` gives a one-sentence TL;DR. Its confidence is capped after the citation and
|
|
68
|
+
command checks: not high if any check failed, low if a claim was dropped or a command rejected.
|
|
69
|
+
Anomalies are number-checked, and a cited judge score that disagrees is dropped.
|
|
70
|
+
|
|
71
|
+
When every cell is ungated (`gate: 'observe'`), it reads the run as a study: how the arms differ, check by
|
|
72
|
+
check, so label your expects. `analysis.instructions` is the question it answers.
|
|
73
|
+
|
|
74
|
+
`analysis.model` is the model `reeval analyze` uses (else the judge's, when it's a language
|
|
75
|
+
model), and the one questions without their own model use. `reeval init` sets it to `google/gemini-3.8-flash` (`GOOGLE_GENERATIVE_AI_API_KEY`;
|
|
76
|
+
`AI_GATEWAY_API_KEY` only with `gateway: [vercel()]`).
|
|
77
|
+
|
|
78
|
+
`reeval run` lists remarks under `Worth knowing:` beneath an eval that itself needs work, with no model: `flaky-check`,
|
|
79
|
+
`slow-mcp-start`, `proxy-only-egress`, `setup-process-secret`, `setup-tunnel-closed`, `snapshot-too-large`, `too-few-tries`,
|
|
80
|
+
`cant-tell-apart` and `observe-throws` (concerns, at most three), and
|
|
81
|
+
with `-v` `check-never-fails` (a suggestion). They go in `remarks.json` beside
|
|
82
|
+
the run, which `reeval record` copies into the history folder. Each names the source line to change (`Fix at <file:line>: …`).
|
|
83
|
+
Act on them: raise `runs`, make a task both arms always pass harder, tighten or remove the check, bake the MCP server's dependencies into
|
|
84
|
+
the image, or set `egress: 'allow-all'` on an eval that needs raw TCP. Each prints once per eval
|
|
85
|
+
version (`too-few-tries`, `slow-mcp-start`, `proxy-only-egress`, `setup-process-secret`, `setup-tunnel-closed` and `snapshot-too-large`
|
|
86
|
+
every run); an edit
|
|
87
|
+
re-arms it.
|
|
88
|
+
One the cap or a run without `-v` hid isn't spent: the next `reeval run -v` prints it.
|
|
89
|
+
|
|
90
|
+
## Config discovery, filters and exit codes
|
|
91
|
+
|
|
92
|
+
Filters: a part that is a path on disk (`evals/a.eval.ts`, `evals/slow/`) picks only the files it
|
|
93
|
+
names; one that names a file but isn't on disk (`skill`) also matches by name and cell key; any
|
|
94
|
+
other part picks evals by name or cell key (`reeval run skill antigravity`).
|
|
95
|
+
|
|
96
|
+
Every command uses the nearest `reeval.config.*` (or `reeval.yaml`) in the current directory or
|
|
97
|
+
above (paths resolve from its directory), or `--config <path>`. `reeval run` and `reeval image
|
|
98
|
+
build` fail with no config (`No reeval.config.ts … here or above. Run reeval init.`); there is
|
|
99
|
+
no hidden default harness or include.
|
|
100
|
+
|
|
101
|
+
| Exit code | Means |
|
|
102
|
+
|---|---|
|
|
103
|
+
| 0 | Pass, and `--dry-run` |
|
|
104
|
+
| 1 | A gated failure |
|
|
105
|
+
| 2 | A usage or config error, including a filter that matches nothing |
|
|
106
|
+
| 3 | An infra error, such as a sandbox that failed to start, a harness key the API rejected or a turn the provider stopped |
|
|
107
|
+
| 130, 143 | Ctrl-C, SIGTERM |
|
|
108
|
+
|
|
109
|
+
One Ctrl-C (or SIGTERM) cancels the run and still writes its record. A second one quits without a
|
|
110
|
+
record: `reeval run --resume <id>` keeps the finished attempts and writes it. pnpm passes Ctrl-C on
|
|
111
|
+
as a SIGTERM, which counts as the same press, and gives the terminal back: the run prints the
|
|
112
|
+
`kill <pid>` that quits it.
|
|
113
|
+
|
|
114
|
+
`--resume` plans with the run's own `--runs`, profile, filters, `--tag`, `--trait` and
|
|
115
|
+
`--concurrency`, so leave them out. Another `--runs` or profile exits 2; other filters, tags or
|
|
116
|
+
traits replace the run's for that resume. Attempts the run planned that they leave out stay in
|
|
117
|
+
the record as not run (or as they finished), the run stays `interrupted` (or `budget`), and a
|
|
118
|
+
later `--resume` runs them. A `.skip` or `.todo` never runs, so it counts as done. A resume
|
|
119
|
+
keeps passes and fails and runs infra errors again.
|
|
120
|
+
|
|
121
|
+
After an outage (a provider down, Docker out of disk), `reeval run --retry-errors [<id>]` runs a
|
|
122
|
+
finished run's infra-errored attempts again into the same run, with the same attempt ids; passes
|
|
123
|
+
and fails stay. It exits 2 if an eval's source, harness, judge, environment, the reeval version
|
|
124
|
+
or the plan changed since the run (start a new run instead). A run with no infra errors prints a
|
|
125
|
+
note and exits with its own outcome, so `reeval run || reeval run --retry-errors` stays red
|
|
126
|
+
after a real fail. Each retried attempt keeps its earlier errors in `retries`, and the run's exit
|
|
127
|
+
code is its outcome after the retry. A Ctrl-C or a kill stops the retry; `--resume` finishes
|
|
128
|
+
the retry, with the same refusals.
|
|
129
|
+
|
|
130
|
+
## Recording runs
|
|
131
|
+
|
|
132
|
+
Runs land in `.reeval/runs/`; `--record` (or `record: 'ci'` in CI) copies them to
|
|
133
|
+
`evals/.history/` to commit. That folder gets a `.gitignore` that un-ignores recorded `*.log`
|
|
134
|
+
files and a `.gitattributes` that marks recorded runs `linguist-generated`; `reeval record` warns if git still ignores a file it wrote.
|
|
135
|
+
|
|
136
|
+
`reeval compare` diffs head (default `latest`) against its nearest recorded ancestor. It prints a
|
|
137
|
+
block per eval that changed, then a tally; unchanged evals collapse to `+N same`:
|
|
138
|
+
|
|
139
|
+
```text
|
|
140
|
+
regressed greets in the house style
|
|
141
|
+
base 20261020T231407Z_ea2a649_ada-lovelace_8h1n
|
|
142
|
+
head 20261020T231408Z_b2c98b4_ada-lovelace_fypq
|
|
143
|
+
house greeting pass → fail
|
|
144
|
+
expected command to succeed, exit code 1
|
|
145
|
+
sign-off not run
|
|
146
|
+
+1 more passed
|
|
147
|
+
|
|
148
|
+
+2 same
|
|
149
|
+
…
|
|
150
|
+
1 regressed · 1 fixed · 1 can't compare · 1 new · 1 removed · 4 same
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
A check head never reached reads `not run`, not a failure. `can't compare` names what changed
|
|
154
|
+
(`changed environment`, `evals/helpers.ts`): that flip isn't a regression. A changed gate
|
|
155
|
+
(eval, config or profile `defaults`) keeps the eval comparable: base's attempts are re-scored
|
|
156
|
+
under head's gate, so a flip the gate alone explains is `gate changed`, not a regression, and
|
|
157
|
+
one base would still have passed under the new gate is `regressed`. An eval Ctrl-C
|
|
158
|
+
canceled is `can't compare` with that side `canceled`. An eval with `cases` is one row
|
|
159
|
+
with its cases' rows inside it (`cases` in `--json`); its pooled `minLift` or `passRate` gate
|
|
160
|
+
decides the row, so it can regress while every case passes. `--fail-on-regression` exits 1 on a
|
|
161
|
+
regressed eval, or on a new or can't-compare one that fails, unless head was canceled. `-v` adds each cell's
|
|
162
|
+
lift and effort, base → head.
|
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
# compare, cells, lift and questions
|
|
2
|
+
|
|
3
|
+
## Contents
|
|
4
|
+
|
|
5
|
+
- What `compare` runs
|
|
6
|
+
- Gates and lift
|
|
7
|
+
- Effort and contrasts
|
|
8
|
+
- Labeled checks that discriminate
|
|
9
|
+
- `questions`
|
|
10
|
+
|
|
11
|
+
## What `compare` runs
|
|
12
|
+
|
|
13
|
+
`compare` answers "does this environment help?". `compare: { control, variants }` names
|
|
14
|
+
both sides: the control is what each variant is measured against, and `control` is required.
|
|
15
|
+
|
|
16
|
+
Each variant runs as that environment on its own, with the same prompt and checks. `compare`
|
|
17
|
+
does not layer, inherit or extend. Share setup first (the example is in SKILL.md).
|
|
18
|
+
|
|
19
|
+
- Attempts = evals × harnesses × (1 control + variants) × runs. The example is 1 eval, 2 cells
|
|
20
|
+
(`<harness>/plain/<sandbox>` control, `<harness>/house-style/<sandbox>` variant), 8 attempts.
|
|
21
|
+
- With `runs` unset at every level, a compare with variants runs 4 a cell, the fewest tries a
|
|
22
|
+
side a lift verdict needs (the plan notes `runs 4 (compare default)`); anything else runs once,
|
|
23
|
+
including a compare with two or more `cases`: its lift pairs cases, so add cases rather than
|
|
24
|
+
runs (the plan notes `runs 1 a case`).
|
|
25
|
+
Any level that sets `runs`, `runs: 1` and `--runs` included, replaces it, and a profile's
|
|
26
|
+
`limit.runs` caps it. A compare with `cases` runs each case once: its lift pools over the cases.
|
|
27
|
+
Don't set `runs: 1` in the config unless you mean every compare to run once.
|
|
28
|
+
- `parent.extendEnvironment(name, spec)` appends the child's spec as a new layer. The child wins
|
|
29
|
+
on shared keys; `setup` concatenates. Call it before `compare`. Two stacks that should not
|
|
30
|
+
share are two `defineEnvironment` calls, then `{ control, variants }`.
|
|
31
|
+
- `environments` beside `compare` (on the same object) is exit 2: move it to `compare.control`.
|
|
32
|
+
An inherited `environments` is not applied as files, skills or variables. A side that omits
|
|
33
|
+
`sandbox` uses the cascaded `sandbox` option, else implicit Docker; `.extend()` keeps the
|
|
34
|
+
parent env's sandbox when the child omits it.
|
|
35
|
+
- There is no `strategy` field. A profile's `limit.compare: false` strips variants and runs
|
|
36
|
+
the control, which then gates like an eval without compare (as does `maxVariants: 0`), whatever
|
|
37
|
+
`compare.gate` says. `compare: false` on `evaluate` or `group` is an error.
|
|
38
|
+
|
|
39
|
+
## Gates and lift
|
|
40
|
+
|
|
41
|
+
- Only variant cells gate by default (`gate?: 'variants' | 'all' | 'observe'`). A control's misses
|
|
42
|
+
show in its arm's count (`plain 0 of 2`) and never fail the run. `compare.harnesses: [cursor({ model: 'composer-2.5' })]` compares on some harnesses only.
|
|
43
|
+
- With `runs > 1`, a cell passes only if every counted attempt passes (pass^k), unless
|
|
44
|
+
`gate: { passRate: 0.8 }` (a pass@1 threshold; pass@1 is passes over passes and fails). Infra errors don't count; a cell is `infra_error`
|
|
45
|
+
only when nothing counted.
|
|
46
|
+
- **Lift** is variant pass@1 minus control pass@1, with a 95% interval. The result block
|
|
47
|
+
prints it as an effect word after the variant's count: `helped`, `made it worse`, or `can't
|
|
48
|
+
tell yet` below four tries a side or when the interval crosses 0; `-v` adds the difference
|
|
49
|
+
and interval. `reeval query factor_effects` and `reeval analyze` use the same four: below
|
|
50
|
+
it the lift is `insufficient` (not measured), never a results line. When the eval's sources changed across the compared runs, that pass@1
|
|
51
|
+
difference is `not comparable: the eval changed`, not an effect, and mixed counts such as
|
|
52
|
+
`6/10 vs 8/9` are not a results line.
|
|
53
|
+
- `compare: { control, variants: [x], minLift: 0 }` fails a variant whose interval's lower
|
|
54
|
+
bound is below 0.
|
|
55
|
+
|
|
56
|
+
## Effort and contrasts
|
|
57
|
+
|
|
58
|
+
- From four tries a side the result block adds an **effort** table: each arm's median tool
|
|
59
|
+
calls, agent time and, when both sides reported usage, tokens, which often shows a skill's
|
|
60
|
+
effect before pass@1 does:
|
|
61
|
+
`effort tools time tokens` / `plain 54 1m 39s 2,890k` / `with-house-greeting 3 6.6s 57k`.
|
|
62
|
+
A cell without usage is left out of the token sample only; if either side then
|
|
63
|
+
has none, the token metric is omitted, not `insufficient`. Effort's rank test needs
|
|
64
|
+
`runs: 4` a cell, as lift does (below it lift shows only a direction and effort can't be tested; the plan hints so, at `runs: 1` too); `reeval query effort` has
|
|
65
|
+
its p-values, verdicts and whether the ranges are `separated`.
|
|
66
|
+
- **One comparison per variant** against its baseline covers every measure: pass rate, each
|
|
67
|
+
check's pass rate, each judge's scores, each compare question's votes, tool calls, agent time
|
|
68
|
+
and tokens. Each is `better`, `worse`, `same` or `leans` (a difference its own test doesn't hold
|
|
69
|
+
yet, at p below 0.1 and of at least 10 points, 0.05 of a judge's score, 60/40 of 4 or more
|
|
70
|
+
decisive pairs for a question at p below 0.2, or 10% of an effort median, with the runs that would hold it;
|
|
71
|
+
weaker is `same`, unless the sample is too small for its test to show anything: "12× faster,
|
|
72
|
+
but 3 runs a side can't confirm it; 4 is the least that can"). A question counts votes, both
|
|
73
|
+
orders of each pair, as its standings do ("in 8 of 8 votes (4 of 4 decisive pairs)"). Each measure keeps its own 0.05, so with identical arms some measure reads
|
|
74
|
+
better or worse by chance in about 1 run in 14 at 4 a side, and 1 in 7 at 8 or more. The status list prints the ones that hold under the
|
|
75
|
+
variant (`Same pass rate · 5× faster.`), `reeval compare -v` the leans too, and
|
|
76
|
+
`reeval query measure_effects` has a row per measure. When both arms pass every run, a measure
|
|
77
|
+
that holds is the result (no remark), one that leans raises `too-few-tries` with the runs, and
|
|
78
|
+
all the same raises `cant-tell-apart`.
|
|
79
|
+
- With two variants or more, `reeval query contrasts` sets every pair of arms side by side,
|
|
80
|
+
variant against variant too.
|
|
81
|
+
- `reeval query skill_files` says which declared files (`SKILL.md`, references, rules) each
|
|
82
|
+
attempt opened, and `context_use` counts its tool calls as `context`, `dependency`
|
|
83
|
+
(`node_modules`, `--help`), `web`, `verify` or `workspace`: a heuristic, read it as counts.
|
|
84
|
+
|
|
85
|
+
## Labeled checks that discriminate
|
|
86
|
+
|
|
87
|
+
- **Label each requirement** with `expect`'s second argument:
|
|
88
|
+
`await expect(c, 'exports greet').toHaveFile('greet.js', /export/)`. Every matcher call
|
|
89
|
+
is recorded (`evals[].attempts[].assertions`; unlabeled ones get `<matcher> <first arg>`).
|
|
90
|
+
In a compare, keep independent requirements as `expect`, not `expect.require`: a failed
|
|
91
|
+
`expect.require` ends the attempt, and the checks after it count as not run: a side that never reached a check reads
|
|
92
|
+
`not run` and the check gets no verdict, so it can't say which arm met it. The result block's
|
|
93
|
+
check table puts a row per check under each arm (`house greeting 0 of 2 2 of 2`) with the
|
|
94
|
+
first failure's message, and folds the checks that passed everywhere into `+2 more passed`.
|
|
95
|
+
After a run, `reeval query "SELECT eval_key, label, verdict FROM check_discrimination WHERE verdict
|
|
96
|
+
LIKE 'both-%'"` and rework or drop those checks: they measure nothing about the variant.
|
|
97
|
+
Changing a label changes the eval's fingerprint. Label judges too
|
|
98
|
+
(`expect(await c.judge(q, { diff: true }), 'Social OAuth').toPass()`): the label replaces the clipped
|
|
99
|
+
question in judge rates, `reeval show`, `summary.md` and the digest.
|
|
100
|
+
|
|
101
|
+
## `questions`
|
|
102
|
+
|
|
103
|
+
```ts
|
|
104
|
+
evaluate('greets in the house style', {
|
|
105
|
+
compare: { control: plain, variants: [houseStyle] },
|
|
106
|
+
questions: [
|
|
107
|
+
// asked of each attempt as it ends: free text, or a pick from choices
|
|
108
|
+
{ question: 'How did the agent pick its greeting?', transcript: true },
|
|
109
|
+
{
|
|
110
|
+
question: 'Does greet.js handle a missing name?',
|
|
111
|
+
diff: true,
|
|
112
|
+
choices: ['yes', 'no'],
|
|
113
|
+
},
|
|
114
|
+
// pairwise: which side did better, after the run
|
|
115
|
+
{
|
|
116
|
+
question: 'Which greet.js reads more like our house style?',
|
|
117
|
+
across: 'environment',
|
|
118
|
+
diff: true,
|
|
119
|
+
},
|
|
120
|
+
{
|
|
121
|
+
question: 'Which sign-in page matches the mock?',
|
|
122
|
+
across: 'environment',
|
|
123
|
+
artifacts: ['sign-in'],
|
|
124
|
+
},
|
|
125
|
+
],
|
|
126
|
+
}, async (c) => { … })
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
- `questions` sits beside `compare`, on an eval or a group (never the config or a
|
|
130
|
+
profile), and works without a `compare` too. It's always a list of objects:
|
|
131
|
+
`{ question, across?, choices?, model?, turns?, …inputs }`, with at least one input
|
|
132
|
+
turned on (`task`, `output`, `files`, `diff`, `transcript`, `checks`, `artifacts`;
|
|
133
|
+
no `contents`, the workspaces are gone). There are no default inputs: an entry with
|
|
134
|
+
none exits 2 at plan time. An eval's list replaces its group's.
|
|
135
|
+
- **No `across`: asked of each attempt** as it ends with a pass or a fail (infra
|
|
136
|
+
errors and skipped attempts aren't asked), while the run goes on. `choices`
|
|
137
|
+
(two or more) makes the model pick one, with a reason; the result block prints
|
|
138
|
+
the tally (`yes 8 · no 4`), else the first few answers. Answers land on
|
|
139
|
+
`evals[].attempts[].answers` and in `reeval query attempt_answers` (with the
|
|
140
|
+
attempt's outcome, `harness`, `environment` and `role`, so you can split
|
|
141
|
+
answers by harness, arm or pass/fail).
|
|
142
|
+
Ask these where a control is expected to fail: every attempt is asked.
|
|
143
|
+
- **`across`: pairwise.** `'environment'` pairs two environments on one harness;
|
|
144
|
+
`'harness'` two harnesses on one environment; a list asks both pair kinds.
|
|
145
|
+
The factor must vary in the eval (else exit 2); a profile that narrows it away
|
|
146
|
+
drops the question (`1 pairwise question off (profile …)`). `choices` with
|
|
147
|
+
`across` exits 2. `--dry-run` shows the pair count. The result block prints
|
|
148
|
+
each side's wins (`with-house-greeting won 4 of 4`). Each pair is voted in
|
|
149
|
+
both orders; one whose orders disagree is a split (the position decided). With
|
|
150
|
+
three arms or more, read `reeval query question_pairs` (head to head, with
|
|
151
|
+
splits), not the pooled wins, which count a split as a win for each side. A
|
|
152
|
+
vote may carry a short `reason` (capped); do not gate on its wording.
|
|
153
|
+
- Questions annotate and never gate: pass, fail and the exit code stay the
|
|
154
|
+
checks'. `reeval analyze` puts each answer in the digest as annotation, not a
|
|
155
|
+
results line. Their model is the entry's `model`, else the cascaded
|
|
156
|
+
`question.model`, else `analysis.model`, else the run's judge (a profile's
|
|
157
|
+
included) when it's a language model (not `typesafe-ai/*`); with none, `reeval run` exits 2 at plan
|
|
158
|
+
time while `--dry-run` still lists them. Keep deterministic checks for pass or
|
|
159
|
+
fail.
|
|
160
|
+
- Each side shows only the entry's inputs (`diff: ['src/**']`,
|
|
161
|
+
`transcript: 'tools', checks: ['npm test']`); `turns` defaults to `'all'`. Keep
|
|
162
|
+
`output` off for pairwise questions, since the more confident summary wins on
|
|
163
|
+
wording. For UI evals, `artifacts: ['sign-in']` (or `true`) sends each attempt's
|
|
164
|
+
own capture of that name as an image. The `question` option takes only `model`
|
|
165
|
+
(config → profile → group → eval → entry). A pairwise question showing
|
|
166
|
+
`transcript` across harnesses warns (tool names unblind them): ask it with
|
|
167
|
+
`across: 'environment'`.
|
|
168
|
+
- Pairwise questions are answered on the host once all of an eval's attempts
|
|
169
|
+
end. Attempts that failed their checks (a judge failure doesn't count) are
|
|
170
|
+
left out of the pairs: a pairwise question on a compare whose baseline is
|
|
171
|
+
expected to fail gets no pairs, so ask it of each attempt instead. Each pair
|
|
172
|
+
goes to the model twice, order swapped, labeled A and B; the record's
|
|
173
|
+
`evals[].answers` has every vote and each side's `wins`/`n` over decided
|
|
174
|
+
votes, per pair kind. A model error is stored on the answer as `error` and
|
|
175
|
+
warned about; it never changes an outcome. To see what a question was shown,
|
|
176
|
+
open its `input`: `<eval-slug>/questions/<n>.json` for a pairwise one (each
|
|
177
|
+
pair and order's masked prompt, which side was A, screenshot refs), or
|
|
178
|
+
`questions/<n>.json` in the attempt's directory. Test with
|
|
179
|
+
`fakeAnalysis({ vote: 'A' | 'B' | 'tie' | (pair) => … | { error } })` and
|
|
180
|
+
`fakeAnalysis({ answer: 'text' | { choice, text } | (q) => … | { error } })`.
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# reeval.config.ts, harnesses and options
|
|
2
|
+
|
|
3
|
+
## Contents
|
|
4
|
+
|
|
5
|
+
- reeval.yaml
|
|
6
|
+
- Path rule and where the sandbox comes from
|
|
7
|
+
- Harnesses, keys and effort
|
|
8
|
+
- Judge models and imports
|
|
9
|
+
- Options and the cascade
|
|
10
|
+
|
|
11
|
+
## reeval.yaml
|
|
12
|
+
|
|
13
|
+
A project without TypeScript can write `reeval.yaml` (or `.yml`) instead, with no callbacks:
|
|
14
|
+
|
|
15
|
+
```yaml
|
|
16
|
+
harnesses: { kind: cursor, model: composer-2.5 }
|
|
17
|
+
judge: { model: typesafe-ai/jev-1.14.0 } # or { kind: fake-judge, score: 0.9 }
|
|
18
|
+
env:
|
|
19
|
+
NPM_TOKEN: { kind: secret, hosts: [registry.npmjs.org] }
|
|
20
|
+
environments:
|
|
21
|
+
kind: environment
|
|
22
|
+
name: base
|
|
23
|
+
layers: [{ name: base, spec: { setup: [npm ci] } }]
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
A `reeval.yaml` beside a `reeval.config.*` is an error.
|
|
27
|
+
|
|
28
|
+
## Path rule and where the sandbox comes from
|
|
29
|
+
|
|
30
|
+
One path rule for both: a relative host path resolves against the directory of the file it's
|
|
31
|
+
written in (the config's, an eval file's, or the module calling `defineEnvironment` or a helper),
|
|
32
|
+
whether it comes from a helper (`file()`, `skill.local()`, `skill.npm()`, `git()`,
|
|
33
|
+
`docker({ dockerfile })`, `fakeSandbox({ dir })`) or its data form (`{ kind: file, path }`,
|
|
34
|
+
`{ kind: fake, dir }`, …). Absolute paths stay as written.
|
|
35
|
+
|
|
36
|
+
| Where the sandbox comes from | When |
|
|
37
|
+
|---|---|
|
|
38
|
+
| The environment's `sandbox` | It sets one; `extendEnvironment` keeps the parent's when the child omits it |
|
|
39
|
+
| The cascaded `sandbox` option (`defineConfig({ sandbox })`) | The resolved environment omits it; it's the default machine, not an environment |
|
|
40
|
+
| Implicit Docker (reeval's image: Node 26, git, pnpm, the agent CLI) | Neither is set |
|
|
41
|
+
|
|
42
|
+
A named env is a total override: no skills, files or sandbox inherit from the config
|
|
43
|
+
environment. A fake profile or config puts `sandbox: fakeSandbox()` on `defaults` or
|
|
44
|
+
`defineConfig` so `plain` and skill variants run without Docker; that's for trying configs, not
|
|
45
|
+
for validating checks (an environment's own `docker()` still wins). Don't put `fakeSandbox()`
|
|
46
|
+
on `plain` itself: real Cursor runs need Docker. An oracle or canary profile sets only
|
|
47
|
+
`defaults.harnesses`, so each eval keeps its sandbox: see
|
|
48
|
+
[evals.md](evals.md#trustworthy-evals).
|
|
49
|
+
|
|
50
|
+
## Harnesses, keys and effort
|
|
51
|
+
|
|
52
|
+
Harnesses (`reeval/harness`), each with a required `model`; a list runs each as its own cell:
|
|
53
|
+
|
|
54
|
+
| Factory | Key | Notes |
|
|
55
|
+
|---|---|---|
|
|
56
|
+
| `cursor({ model, params?, effort? })` | `CURSOR_API_KEY` | Cursor's agent through the Cursor SDK. `params` such as `{ fast: 'true' }` pick the variant; `effort` maps onto the parameter `Cursor.models.list()` names for that model |
|
|
57
|
+
| `claudeCode({ model, effort? })` | `ANTHROPIC_API_KEY`, else a subscription's `CLAUDE_CODE_OAUTH_TOKEN` (`claude setup-token`) | `effort` is the Agent SDK's `/effort` |
|
|
58
|
+
| `antigravity({ model, effort? })` | `GEMINI_API_KEY` (a Gemini API key) | e.g. `gemini-3.8-flash-medium`; `effort` is `agy --effort` |
|
|
59
|
+
| `opencode({ model, effort? })` | That provider's key, e.g. `ANTHROPIC_API_KEY` | OpenCode 2; the model is `<provider>/<model>`, e.g. `google/gemini-3.8-flash`. `amazon-bedrock/…` takes `AWS_BEARER_TOKEN_BEDROCK` and an optional `AWS_REGION`; Vertex (on `direct()`; it runs through `vertex()`), Azure, Copilot and local providers are refused at plan time. Web search via Exa, whose host it allows; it also reads `.claude/skills` and `.agents/skills`. `effort` is refused where the model's API answers it with a 400 (any value on `anthropic/claude-haiku-4-5`, `openai/gpt-4.1` or `google/gemini-2.5-flash`, any but `high` on `openai/gpt-5-pro`), and such a model runs with none by default; on `openrouter()` nothing is refused |
|
|
60
|
+
| `goose({ model, effort? })` | The provider's key: `GOOGLE_GENERATIVE_AI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `XAI_API_KEY` | `<provider>/<model>` on Goose's `google`, `anthropic`, `openai` or `xai` provider, e.g. `google/gemini-3.8-flash` (any gateway id on `vercel()` or `openrouter()`); `effort` is `GOOSE_THINKING_EFFORT` |
|
|
61
|
+
| `pi({ model, effort? })` | The provider's key, e.g. `GOOGLE_GENERATIVE_AI_API_KEY`, `ANTHROPIC_API_KEY`, `XAI_API_KEY`, `MISTRAL_API_KEY` | `<provider>/<model>` on any of pi's providers that sends its key in a header on a fixed host, e.g. `google/gemini-3.8-flash` (any gateway id on `vercel()` or `openrouter()`); `effort` is `--thinking` |
|
|
62
|
+
| `codex({ model, effort? })` | `OPENAI_API_KEY` | An OpenAI model id, e.g. `gpt-5.6-luna`; `effort` is `model_reasoning_effort` |
|
|
63
|
+
| `fx({ model, effort? })` | `OPENAI_API_KEY` for a bare id, `AI_GATEWAY_API_KEY` on `vercel()`, `OPENROUTER_API_KEY` on `openrouter()`, or none on `vertex()` | Vercel's fx over ACP: a gateway id (`anthropic/claude-sonnet-5`) runs only on `vercel()` or `openrouter()` in `gateway`, a bare OpenAI id (`gpt-5.5`) on OpenAI, which must accept function tools without `reasoning_effort` (not `gpt-5.6` or later), a Gemini id (`google/gemini-3.8-flash`) on `vertex()`. `effort` (`reasoning_effort`) only on `vertex()`. Its own web tools bypass the egress gateway, so `deny-all` blocks them |
|
|
64
|
+
| `grokBuild({ model, effort? })` | `XAI_API_KEY` | xAI's Grok Build CLI; an xAI model id, e.g. `grok-4.6`; `effort` is `default_reasoning_effort`. It finds MCP tools with its `search_tool` before calling them |
|
|
65
|
+
| `fakeHarness(…)` | None | See the fake trio in SKILL.md |
|
|
66
|
+
|
|
67
|
+
OpenCode, Goose, pi and Codex refuse `mcp.sse`. Cursor's `fast` defaults to `'false'` (standard
|
|
68
|
+
price; `{ fast: 'true' }` costs several times more). Other unset Cursor params take Cursor's
|
|
69
|
+
default, which the record shows (`composer-2.5[fast=false]`), so pin them for reproducible evals.
|
|
70
|
+
|
|
71
|
+
`effort` is the same field on every factory (`fx()` takes it only on `vertex()`), with whatever values that CLI takes (no shared
|
|
72
|
+
enum); Fast is not effort. The default harness id includes `[effort=<value>]` when it's set, and
|
|
73
|
+
the record stores the requested effort as `effort` and the served one as `servedEffort`.
|
|
74
|
+
|
|
75
|
+
In `reeval query` those are `reasoning_effort` (`cells`, `attempts`) and
|
|
76
|
+
`served_reasoning_effort` (`attempts`). **effort** on its own (`reeval query effort`, the
|
|
77
|
+
result block's effort table) is always the measured tool-call, time and token metric.
|
|
78
|
+
|
|
79
|
+
Keys never enter the sandbox: the agent sees a placeholder that reeval's gateway swaps on the
|
|
80
|
+
harness's own endpoints. Every harness reports tokens, which reeval records per turn,
|
|
81
|
+
attempt and run with their cost (reported, else estimated at list prices).
|
|
82
|
+
|
|
83
|
+
`gateway` (`reeval/gateway`) lists the routes model calls may take, in order: each harness, the
|
|
84
|
+
judge and analysis take the first that can serve them. Every list ends with `direct()` (each
|
|
85
|
+
provider's own key), so don't write it at the end: list it only to move it (`[direct(), vercel()]`
|
|
86
|
+
tries own keys first), or write `direct({ enabled: false })` for strict routing, where a cell no
|
|
87
|
+
entry serves stops the plan (never both). `gateway: [vercel()]` runs Claude
|
|
88
|
+
Code, Codex, Goose, pi, OpenCode, fx and Grok Build on Vercel AI Gateway with one `AI_GATEWAY_API_KEY` (else `VERCEL_OIDC_TOKEN`) (`effort`
|
|
89
|
+
on Goose, or on OpenCode with a model other than Gemini, falls back to `direct()`), and
|
|
90
|
+
everything else on its own key; the plan's hint says which cells fell back. Pin one with
|
|
91
|
+
the factory's or the judge's `gateway: 'vercel'`; a pin never falls back. `vertex({ project })`
|
|
92
|
+
(else `GOOGLE_CLOUD_PROJECT`) runs Claude Code, Antigravity, OpenCode, Goose, pi, Grok Build and fx (Gemini only) on Vertex AI
|
|
93
|
+
with no key: reeval mints a token from Application Default Credentials on the host and the egress
|
|
94
|
+
gateway adds it (OpenCode's Gemini model runs as `google-vertex/<id>`, Grok Build's xAI model as `xai/<id>` on Vertex's Chat Completions API).
|
|
95
|
+
There Claude Code sends `effort` only to the models its own catalog lists, so `effort` on one it
|
|
96
|
+
doesn't, such as `claude-haiku-5-5`, falls through, and with none set that model runs with no effort. The `sonnet`
|
|
97
|
+
and `opusplan` aliases fall through too: Claude Code resolves them on Vertex AI to Sonnet 4.5, which Anthropic retires on 2026-11-30, so name
|
|
98
|
+
`claude-sonnet-5-5` instead. Claude ids work in
|
|
99
|
+
either form (`claude-opus-4-5-20251101` is sent as Vertex's `claude-opus-4-5@20251101`), so one
|
|
100
|
+
string works through `[vertex()]` and its `direct()` fallback; `google/*` and `anthropic/*` judge and analysis models run there
|
|
101
|
+
too. `reeval doctor --online` checks the token, the project's Vertex AI API and that Vertex serves each model the config sends there (Claude needs enabling in Model Garden). Default `[direct()]`:
|
|
102
|
+
no gateway, so the judge, analysis and captures take only their provider's own key.
|
|
103
|
+
`gateway: [openrouter()]` runs Claude Code, Codex, Goose, OpenCode, fx, Grok Build and pi on
|
|
104
|
+
OpenRouter with one `OPENROUTER_API_KEY` (Cursor, Antigravity, and `effort` on fx, on Claude Code or
|
|
105
|
+
on Goose or OpenCode with a model other than Gemini, fall through). Every OpenRouter call records its upstream provider and cost
|
|
106
|
+
(`attempts[].upstreams`, `reportedCostUsd`; a judgment's `upstream`, `costUsd`).
|
|
107
|
+
`openrouter({ fallbacks: true })` lets reeval's own calls (judge, analysis, captures) fail over
|
|
108
|
+
between upstreams; the agents' calls take OpenRouter's routing whatever it says. Write OpenCode
|
|
109
|
+
ids as `anthropic/…`, never `openrouter/…` or `vercel/…`.
|
|
110
|
+
|
|
111
|
+
## Judge models and imports
|
|
112
|
+
|
|
113
|
+
- Judge models: `'typesafe-ai/jev-1.14.0'` (`TYPESAFE_AI_API_KEY`, recommended),
|
|
114
|
+
`'google/gemini-3.8-flash'` (`GOOGLE_GENERATIVE_AI_API_KEY`), `anthropic/*` (`ANTHROPIC_API_KEY`)
|
|
115
|
+
and `'openai/gpt-6-luna'` (`OPENAI_API_KEY`; OpenAI's Decisions API takes only its decision
|
|
116
|
+
model, so not `openai/gpt-5.5`), each on its provider's own key. reeval uses no gateway by
|
|
117
|
+
default: `AI_GATEWAY_API_KEY` is read only with `gateway: [vercel()]` in the config (or a judge
|
|
118
|
+
`gateway: 'vercel'` pin, listed there), `OPENROUTER_API_KEY` only with `openrouter()`. As a
|
|
119
|
+
judge, `vercel()` serves decision models only (Jev, `openai/gpt-6-luna-decisions`), and
|
|
120
|
+
`openrouter()` only those on its Decisions API (`typesafe/jev-1.14`), never a language model:
|
|
121
|
+
pin a language-model judge to `gateway: 'direct'`. Always a provider prefix.
|
|
122
|
+
With a `gateway` list set, a judge model takes the first listed gateway that serves it, and only
|
|
123
|
+
that gateway's key.
|
|
124
|
+
- Imports: `reeval` (`defineConfig`, `defineEnvironment`, `group`, `evaluate`,
|
|
125
|
+
`expect`, `DEFAULT_EGRESS_ALLOW`, `DEFAULT_SKILL_EXCLUDE`, types), `/harness`, `/gateway`
|
|
126
|
+
(`direct`, `vercel`), `/sandbox`
|
|
127
|
+
(`docker`, `fakeSandbox`), `/judge` (`fakeJudge`, `RECOMMENDED_JUDGE_MODEL`,
|
|
128
|
+
`DEFAULT_THRESHOLD`). There is no `test`, `describe`, `baseline`, `secret` or `skill` export.
|
|
129
|
+
|
|
130
|
+
## Options and the cascade
|
|
131
|
+
|
|
132
|
+
`defineConfig`, `group` and `evaluate` take the same options, and inner levels win:
|
|
133
|
+
|
|
134
|
+
| Option | Cascades as |
|
|
135
|
+
|---|---|
|
|
136
|
+
| `harness` | One or a list, or an edit of the inherited list: `{ add: h }`, `{ only: 'cursor:composer-2.5' }` or a function |
|
|
137
|
+
| `sandbox` | The default machine when the resolved env omits one |
|
|
138
|
+
| `environment` | One; a named env is a total override of the config environment |
|
|
139
|
+
| `env` | Merged by name |
|
|
140
|
+
| `judge` | Merged by field |
|
|
141
|
+
| `capture` | Shallow-merged camera defaults (`viewport`, `timeout`, `browser`, `model`, the vision model); `viewport` replaces the value, and the port is not an option |
|
|
142
|
+
| `runs`, `gate`, `egress`, `timeout` | Replaced (`runs` unset everywhere: 4 for a compare with variants, else 1) |
|
|
143
|
+
| `tags`, `pin` | Unioned |
|
|
144
|
+
| `compare` | Groups and evals only |
|
|
145
|
+
|
|
146
|
+
Options are plain objects, never functions. Unknown keys are errors with a hint (`repeat` → "Did
|
|
147
|
+
you mean runs?").
|