@grok/agent-sdk 0.2.28 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +29 -29
- package/dist/bin/agent-serve.d.ts +2 -2
- package/dist/bin/agent-serve.js +3 -3
- package/dist/channels/pagerduty/api.js +1 -1
- package/dist/channels/pagerduty/pagerduty-channel.d.ts +1 -1
- package/dist/channels/pagerduty/pagerduty-channel.d.ts.map +1 -1
- package/dist/channels/pagerduty/pagerduty-channel.js +1 -1
- package/dist/channels/pagerduty/replay.d.ts +1 -1
- package/dist/channels/pagerduty/replay.js +4 -4
- package/dist/channels/slack/cli-project.js +1 -1
- package/dist/docs/404.html +2 -2
- package/dist/docs/assets/{app.DF9vlcdn.js → app.BzTUBOAD.js} +4 -4
- package/dist/docs/assets/building-with-agents.md.BqJ7AAjG.js +6 -0
- package/dist/docs/assets/{building-with-agents.md.Bf1yH2tG.lean.js → building-with-agents.md.BqJ7AAjG.lean.js} +1 -1
- package/dist/docs/assets/changelog.md.BWhC6DET.js +1 -0
- package/dist/docs/assets/changelog.md.BWhC6DET.lean.js +1 -0
- package/dist/docs/assets/chunks/@localSearchIndexroot.CWxwVoCG.js +1 -0
- package/dist/docs/assets/chunks/{VPLocalSearchBox.D8OuwKO0.js → VPLocalSearchBox.DsnRlUT5.js} +1 -1
- package/dist/docs/assets/chunks/{arc.DCq_TCYV.js → arc.DH8GOhiv.js} +1 -1
- package/dist/docs/assets/chunks/{architectureDiagram-Q4EWVU46.PWRNTPbk.js → architectureDiagram-Q4EWVU46.9m9ObyuE.js} +1 -1
- package/dist/docs/assets/chunks/{baseUniq.BeRPvRUL.js → baseUniq.BXHpx1lE.js} +1 -1
- package/dist/docs/assets/chunks/{blockDiagram-DXYQGD6D.BuRmZVSo.js → blockDiagram-DXYQGD6D.xY3EX59q.js} +1 -1
- package/dist/docs/assets/chunks/{c4Diagram-AHTNJAMY.7DKSMwuv.js → c4Diagram-AHTNJAMY.DnqgMVgc.js} +1 -1
- package/dist/docs/assets/chunks/channel.Cctv0f7X.js +1 -0
- package/dist/docs/assets/chunks/{chunk-4BX2VUAB.7VrO9-9i.js → chunk-4BX2VUAB.BjFxK2A3.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-4TB4RGXK.CHystNE2.js → chunk-4TB4RGXK.BujS_Spt.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-55IACEB6.Nuh9TbTq.js → chunk-55IACEB6.YbTjhdaM.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-EDXVE4YY.CA0EKugm.js → chunk-EDXVE4YY.DnQ8t4GC.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-FMBD7UC4.F9_mp7U8.js → chunk-FMBD7UC4.B67AwAmj.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-OYMX7WX6.CEU4QfE1.js → chunk-OYMX7WX6.DrZIAFOm.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-QZHKN3VN.B8h_x9_i.js → chunk-QZHKN3VN.BLmmydkn.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-YZCP3GAM.Btfb-rKg.js → chunk-YZCP3GAM.DaoOfw0O.js} +1 -1
- package/dist/docs/assets/chunks/classDiagram-6PBFFD2Q.DU5sqchW.js +1 -0
- package/dist/docs/assets/chunks/classDiagram-v2-HSJHXN6E.DU5sqchW.js +1 -0
- package/dist/docs/assets/chunks/clone.B9rdNt6j.js +1 -0
- package/dist/docs/assets/chunks/{cose-bilkent-S5V4N54A.DKvc96Wa.js → cose-bilkent-S5V4N54A.DMQuNhM5.js} +1 -1
- package/dist/docs/assets/chunks/{dagre-KV5264BT.CMo0cy1b.js → dagre-KV5264BT.CZoGVuUf.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-5BDNPKRD.DDgMOQhB.js → diagram-5BDNPKRD.CLt9uOyL.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-G4DWMVQ6.BdJnoIsU.js → diagram-G4DWMVQ6.DpczEm1G.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-MMDJMWI5.oSvhAZaL.js → diagram-MMDJMWI5.D4r13DvJ.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-TYMM5635.C9zZLIE-.js → diagram-TYMM5635.CYU9eGFu.js} +1 -1
- package/dist/docs/assets/chunks/{erDiagram-SMLLAGMA.BNhVyccy.js → erDiagram-SMLLAGMA.QA9naWEw.js} +1 -1
- package/dist/docs/assets/chunks/{flowDiagram-DWJPFMVM.BEFLLBiU.js → flowDiagram-DWJPFMVM.BbNoVZYd.js} +1 -1
- package/dist/docs/assets/chunks/{ganttDiagram-T4ZO3ILL.BdUzXBgY.js → ganttDiagram-T4ZO3ILL.IHjDoG7r.js} +1 -1
- package/dist/docs/assets/chunks/{gitGraphDiagram-UUTBAWPF.Rd3LlOIJ.js → gitGraphDiagram-UUTBAWPF.B21VMdf5.js} +1 -1
- package/dist/docs/assets/chunks/{graph.DnJC1GpG.js → graph.DgJ4L-8V.js} +1 -1
- package/dist/docs/assets/chunks/{infoDiagram-42DDH7IO.xv0xntDJ.js → infoDiagram-42DDH7IO.CQjrcthc.js} +1 -1
- package/dist/docs/assets/chunks/{ishikawaDiagram-UXIWVN3A.BaGIlviU.js → ishikawaDiagram-UXIWVN3A.D4tPGtVo.js} +1 -1
- package/dist/docs/assets/chunks/{journeyDiagram-VCZTEJTY.Cof_h0fp.js → journeyDiagram-VCZTEJTY.BD3WaNbQ.js} +1 -1
- package/dist/docs/assets/chunks/{kanban-definition-6JOO6SKY.C5GozFuU.js → kanban-definition-6JOO6SKY.Ca0o4Tqz.js} +1 -1
- package/dist/docs/assets/chunks/{layout.zc9UShLc.js → layout.DXXqX4MQ.js} +1 -1
- package/dist/docs/assets/chunks/{linear.BoqrFVcJ.js → linear.5c61gNw4.js} +1 -1
- package/dist/docs/assets/chunks/{min.DwIfr8hG.js → min.B1-NAjHX.js} +1 -1
- package/dist/docs/assets/chunks/{mindmap-definition-QFDTVHPH.Dkq2yTtp.js → mindmap-definition-QFDTVHPH.BELznv3-.js} +1 -1
- package/dist/docs/assets/chunks/{pieDiagram-DEJITSTG.B4CpTmHK.js → pieDiagram-DEJITSTG.qPzO1mq5.js} +1 -1
- package/dist/docs/assets/chunks/{quadrantDiagram-34T5L4WZ.BqXnFC9C.js → quadrantDiagram-34T5L4WZ.CxIvzbjX.js} +1 -1
- package/dist/docs/assets/chunks/{requirementDiagram-MS252O5E.DhIN2z-E.js → requirementDiagram-MS252O5E.CqA0iJx8.js} +1 -1
- package/dist/docs/assets/chunks/{sankeyDiagram-XADWPNL6.BhxUAmg2.js → sankeyDiagram-XADWPNL6.XRTutBnb.js} +1 -1
- package/dist/docs/assets/chunks/{sequenceDiagram-FGHM5R23.BqgU7HU-.js → sequenceDiagram-FGHM5R23.E--7bRLQ.js} +1 -1
- package/dist/docs/assets/chunks/{stateDiagram-FHFEXIEX.B1BCO18N.js → stateDiagram-FHFEXIEX.C4vGu8zq.js} +1 -1
- package/dist/docs/assets/chunks/stateDiagram-v2-QKLJ7IA2.ECMu2BdJ.js +1 -0
- package/dist/docs/assets/chunks/{theme.oV7rddEz.js → theme.DMj_7q3B.js} +2 -2
- package/dist/docs/assets/chunks/{timeline-definition-GMOUNBTQ.oRI8UwMC.js → timeline-definition-GMOUNBTQ.C7LgNVAL.js} +1 -1
- package/dist/docs/assets/chunks/{vennDiagram-DHZGUBPP.DmuWrIxN.js → vennDiagram-DHZGUBPP.HUwoZcy_.js} +1 -1
- package/dist/docs/assets/chunks/{wardley-RL74JXVD.CEBjM5ov.js → wardley-RL74JXVD.BxFEFYZS.js} +1 -1
- package/dist/docs/assets/chunks/{wardleyDiagram-NUSXRM2D.m5zsCOF9.js → wardleyDiagram-NUSXRM2D.B2pSo5rB.js} +1 -1
- package/dist/docs/assets/chunks/{xychartDiagram-5P7HB3ND.cPN-587n.js → xychartDiagram-5P7HB3ND.NQhMiOOL.js} +1 -1
- package/dist/docs/assets/deployment.md.BtqJrHhp.js +31 -0
- package/dist/docs/assets/{deployment.md.DDmjAVbg.lean.js → deployment.md.BtqJrHhp.lean.js} +1 -1
- package/dist/docs/assets/evals.md.Dne6VRmZ.js +49 -0
- package/dist/docs/assets/evals.md.Dne6VRmZ.lean.js +1 -0
- package/dist/docs/assets/{guides_agent-to-agent.md.D1D5f6Jc.js → guides_agent-to-agent.md.BcmSqL2z.js} +8 -8
- package/dist/docs/assets/guides_agent-to-agent.md.BcmSqL2z.lean.js +1 -0
- package/dist/docs/assets/{guides_bitbucket.md.DkjkBRU8.js → guides_bitbucket.md.BiA6ZMFW.js} +27 -27
- package/dist/docs/assets/{guides_bitbucket.md.DkjkBRU8.lean.js → guides_bitbucket.md.BiA6ZMFW.lean.js} +1 -1
- package/dist/docs/assets/guides_cloud-agents.md.CiXAW6on.js +9 -0
- package/dist/docs/assets/{guides_cloud-agents.md.Ceyjv0ul.lean.js → guides_cloud-agents.md.CiXAW6on.lean.js} +1 -1
- package/dist/docs/assets/guides_convert-automation.md.BvTxsJ8J.js +26 -0
- package/dist/docs/assets/{guides_convert-automation.md.BjraCvok.lean.js → guides_convert-automation.md.BvTxsJ8J.lean.js} +1 -1
- package/dist/docs/assets/{guides_github.md.CgN5sNbW.js → guides_github.md.6jiEH7dg.js} +11 -19
- package/dist/docs/assets/{guides_github.md.CgN5sNbW.lean.js → guides_github.md.6jiEH7dg.lean.js} +1 -1
- package/dist/docs/assets/{guides_gitlab.md.DCE0p7_w.js → guides_gitlab.md.DcKusOcU.js} +63 -73
- package/dist/docs/assets/{guides_gitlab.md.DCE0p7_w.lean.js → guides_gitlab.md.DcKusOcU.lean.js} +1 -1
- package/dist/docs/assets/guides_improve.md.o3WOGR7n.js +16 -0
- package/dist/docs/assets/{guides_improve.md.BFYtrtLE.lean.js → guides_improve.md.o3WOGR7n.lean.js} +1 -1
- package/dist/docs/assets/{guides_jev.md.Ccw4pedT.js → guides_jev.md.DGCzkMUN.js} +20 -60
- package/dist/docs/assets/{guides_jev.md.Ccw4pedT.lean.js → guides_jev.md.DGCzkMUN.lean.js} +1 -1
- package/dist/docs/assets/{guides_mcp-oauth.md.ekpWOZCC.js → guides_mcp-oauth.md.C-Sx_IPa.js} +3 -5
- package/dist/docs/assets/{guides_mcp-oauth.md.ekpWOZCC.lean.js → guides_mcp-oauth.md.C-Sx_IPa.lean.js} +1 -1
- package/dist/docs/assets/guides_migrate.md.BbhHYeCr.js +3 -0
- package/dist/docs/assets/{guides_migrate.md.RS9Eu-Xh.lean.js → guides_migrate.md.BbhHYeCr.lean.js} +1 -1
- package/dist/docs/assets/guides_opentelemetry.md.BRSktBHt.js +26 -0
- package/dist/docs/assets/guides_opentelemetry.md.BRSktBHt.lean.js +1 -0
- package/dist/docs/assets/guides_pagerduty.md.pMXUkZZc.js +61 -0
- package/dist/docs/assets/guides_pagerduty.md.pMXUkZZc.lean.js +1 -0
- package/dist/docs/assets/guides_slack.md.BgIGG9ur.js +82 -0
- package/dist/docs/assets/{guides_slack.md.BqiGSYGX.lean.js → guides_slack.md.BgIGG9ur.lean.js} +1 -1
- package/dist/docs/assets/{guides_webhooks.md.BJk63ZTs.js → guides_webhooks.md.B5DXt2v1.js} +7 -7
- package/dist/docs/assets/{guides_webhooks.md.BJk63ZTs.lean.js → guides_webhooks.md.B5DXt2v1.lean.js} +1 -1
- package/dist/docs/assets/hillclimbing.md.3TkdF79A.js +5 -0
- package/dist/docs/assets/hillclimbing.md.3TkdF79A.lean.js +1 -0
- package/dist/docs/assets/index.md.DRfaoRCZ.js +20 -0
- package/dist/docs/assets/index.md.DRfaoRCZ.lean.js +1 -0
- package/dist/docs/assets/quickstart.md.r-oE8nkX.js +113 -0
- package/dist/docs/assets/{quickstart.md.DaLk4ZIF.lean.js → quickstart.md.r-oE8nkX.lean.js} +1 -1
- package/dist/docs/assets/reference_access.md.C5lblfUC.js +1 -0
- package/dist/docs/assets/{reference_access.md.CExlH_mR.lean.js → reference_access.md.C5lblfUC.lean.js} +1 -1
- package/dist/docs/assets/reference_agent-config.md.C0Tp_Fk9.js +27 -0
- package/dist/docs/assets/{reference_agent-config.md.11E_xe--.lean.js → reference_agent-config.md.C0Tp_Fk9.lean.js} +1 -1
- package/dist/docs/assets/reference_artifacts.md.BJpAI7h9.js +17 -0
- package/dist/docs/assets/{reference_artifacts.md.C-hKR2hp.lean.js → reference_artifacts.md.BJpAI7h9.lean.js} +1 -1
- package/dist/docs/assets/reference_channels.md.CPWOpdsC.js +21 -0
- package/dist/docs/assets/{reference_channels.md.DsJfGSBJ.lean.js → reference_channels.md.CPWOpdsC.lean.js} +1 -1
- package/dist/docs/assets/reference_cli.md.B8GdpaRj.js +83 -0
- package/dist/docs/assets/reference_cli.md.B8GdpaRj.lean.js +1 -0
- package/dist/docs/assets/reference_connections.md.DoY4vrDH.js +54 -0
- package/dist/docs/assets/{reference_connections.md.BoAotr7_.lean.js → reference_connections.md.DoY4vrDH.lean.js} +1 -1
- package/dist/docs/assets/reference_evals.md.DgfUtfIc.js +69 -0
- package/dist/docs/assets/{reference_evals.md.DYKjGFi-.lean.js → reference_evals.md.DgfUtfIc.lean.js} +1 -1
- package/dist/docs/assets/{reference_extensions.md.pEVj72Xo.js → reference_extensions.md.DHbW6eOf.js} +6 -8
- package/dist/docs/assets/{reference_extensions.md.pEVj72Xo.lean.js → reference_extensions.md.DHbW6eOf.lean.js} +1 -1
- package/dist/docs/assets/reference_hooks.md.CSWmOwMM.js +39 -0
- package/dist/docs/assets/reference_hooks.md.CSWmOwMM.lean.js +1 -0
- package/dist/docs/assets/reference_http-api.md.C7Cb0r2J.js +7 -0
- package/dist/docs/assets/{reference_http-api.md._RWELr3d.lean.js → reference_http-api.md.C7Cb0r2J.lean.js} +1 -1
- package/dist/docs/assets/reference_instructions.md.CV6Qq0UX.js +14 -0
- package/dist/docs/assets/reference_instructions.md.CV6Qq0UX.lean.js +1 -0
- package/dist/docs/assets/reference_limits.md.CbdtFr4T.js +8 -0
- package/dist/docs/assets/reference_limits.md.CbdtFr4T.lean.js +1 -0
- package/dist/docs/assets/reference_memory.md.CdoJISM-.js +3 -0
- package/dist/docs/assets/{reference_memory.md.Dd7cyV9V.lean.js → reference_memory.md.CdoJISM-.lean.js} +1 -1
- package/dist/docs/assets/reference_playground.md.DIUsRAcy.js +1 -0
- package/dist/docs/assets/{reference_playground.md.CQiJk9CM.lean.js → reference_playground.md.DIUsRAcy.lean.js} +1 -1
- package/dist/docs/assets/reference_project-layout.md.9DiKoE16.js +26 -0
- package/dist/docs/assets/reference_project-layout.md.9DiKoE16.lean.js +1 -0
- package/dist/docs/assets/reference_schedules.md.RhQ9ZLo1.js +45 -0
- package/dist/docs/assets/{reference_schedules.md.RC47aFm4.lean.js → reference_schedules.md.RhQ9ZLo1.lean.js} +1 -1
- package/dist/docs/assets/reference_sessions.md.NXVQt9mP.js +22 -0
- package/dist/docs/assets/{reference_sessions.md.C7KFhWXV.lean.js → reference_sessions.md.NXVQt9mP.lean.js} +1 -1
- package/dist/docs/assets/{reference_skills.md.Dej2zQxi.js → reference_skills.md.DzCPNPkG.js} +4 -4
- package/dist/docs/assets/{reference_skills.md.Dej2zQxi.lean.js → reference_skills.md.DzCPNPkG.lean.js} +1 -1
- package/dist/docs/assets/reference_subagents.md.Dvpq2sUu.js +10 -0
- package/dist/docs/assets/{reference_subagents.md.CAhvW4yS.lean.js → reference_subagents.md.Dvpq2sUu.lean.js} +1 -1
- package/dist/docs/assets/reference_tools.md.B2OaR1Nb.js +62 -0
- package/dist/docs/assets/{reference_tools.md.DQr7Fnhu.lean.js → reference_tools.md.B2OaR1Nb.lean.js} +1 -1
- package/dist/docs/assets/templates_agentic-owners.md.B6ZmKAzT.js +13 -0
- package/dist/docs/assets/{templates_agentic-owners.md.Dh2nDSqg.lean.js → templates_agentic-owners.md.B6ZmKAzT.lean.js} +1 -1
- package/dist/docs/assets/templates_reviews.md.dr9Xv_KM.js +10 -0
- package/dist/docs/assets/templates_reviews.md.dr9Xv_KM.lean.js +1 -0
- package/dist/docs/assets/troubleshooting.md.DfKKvW1R.js +3 -0
- package/dist/docs/assets/{troubleshooting.md.AGJ3jDzj.lean.js → troubleshooting.md.DfKKvW1R.lean.js} +1 -1
- package/dist/docs/building-with-agents.html +41 -44
- package/dist/docs/building-with-agents.md +11 -104
- package/dist/docs/changelog.html +37 -37
- package/dist/docs/changelog.md +5 -145
- package/dist/docs/deployment.html +54 -54
- package/dist/docs/deployment.md +86 -265
- package/dist/docs/evals.html +52 -71
- package/dist/docs/evals.md +33 -127
- package/dist/docs/guides/agent-to-agent.html +43 -43
- package/dist/docs/guides/agent-to-agent.md +27 -64
- package/dist/docs/guides/bitbucket.html +62 -62
- package/dist/docs/guides/bitbucket.md +73 -134
- package/dist/docs/guides/cloud-agents.html +37 -37
- package/dist/docs/guides/cloud-agents.md +23 -53
- package/dist/docs/guides/convert-automation.html +39 -56
- package/dist/docs/guides/convert-automation.md +31 -115
- package/dist/docs/guides/github.html +46 -54
- package/dist/docs/guides/github.md +49 -131
- package/dist/docs/guides/gitlab.html +98 -108
- package/dist/docs/guides/gitlab.md +104 -184
- package/dist/docs/guides/improve.html +38 -43
- package/dist/docs/guides/improve.md +24 -90
- package/dist/docs/guides/jev.html +55 -95
- package/dist/docs/guides/jev.md +57 -159
- package/dist/docs/guides/mcp-oauth.html +38 -40
- package/dist/docs/guides/mcp-oauth.md +34 -69
- package/dist/docs/guides/migrate.html +38 -43
- package/dist/docs/guides/migrate.md +56 -130
- package/dist/docs/guides/opentelemetry.html +41 -43
- package/dist/docs/guides/opentelemetry.md +32 -27
- package/dist/docs/guides/pagerduty.html +50 -49
- package/dist/docs/guides/pagerduty.md +65 -199
- package/dist/docs/guides/slack.html +51 -57
- package/dist/docs/guides/slack.md +76 -205
- package/dist/docs/guides/webhooks.html +42 -42
- package/dist/docs/guides/webhooks.md +64 -146
- package/dist/docs/hashmap.json +1 -1
- package/dist/docs/hillclimbing.html +40 -42
- package/dist/docs/hillclimbing.md +40 -102
- package/dist/docs/index.html +55 -62
- package/dist/docs/index.md +49 -82
- package/dist/docs/llms-full.txt +2292 -5830
- package/dist/docs/llms.txt +9 -15
- package/dist/docs/quickstart.html +96 -153
- package/dist/docs/quickstart.md +93 -178
- package/dist/docs/reference/access.html +36 -36
- package/dist/docs/reference/access.md +7 -12
- package/dist/docs/reference/agent-config.html +43 -56
- package/dist/docs/reference/agent-config.md +93 -176
- package/dist/docs/reference/artifacts.html +38 -39
- package/dist/docs/reference/artifacts.md +38 -62
- package/dist/docs/reference/channels.html +38 -39
- package/dist/docs/reference/channels.md +102 -151
- package/dist/docs/reference/cli.html +68 -84
- package/dist/docs/reference/cli.md +233 -698
- package/dist/docs/reference/connections.html +45 -57
- package/dist/docs/reference/connections.md +83 -164
- package/dist/docs/reference/evals.html +66 -53
- package/dist/docs/reference/evals.md +97 -128
- package/dist/docs/reference/extensions.html +41 -43
- package/dist/docs/reference/extensions.md +37 -70
- package/dist/docs/reference/hooks.html +40 -52
- package/dist/docs/reference/hooks.md +62 -133
- package/dist/docs/reference/http-api.html +38 -42
- package/dist/docs/reference/http-api.md +66 -118
- package/dist/docs/reference/instructions.html +39 -39
- package/dist/docs/reference/instructions.md +19 -29
- package/dist/docs/reference/limits.html +38 -38
- package/dist/docs/reference/limits.md +9 -21
- package/dist/docs/reference/memory.html +37 -38
- package/dist/docs/reference/memory.md +38 -75
- package/dist/docs/reference/playground.html +36 -36
- package/dist/docs/reference/playground.md +14 -22
- package/dist/docs/reference/project-layout.html +49 -48
- package/dist/docs/reference/project-layout.md +38 -84
- package/dist/docs/reference/schedules.html +46 -48
- package/dist/docs/reference/schedules.md +59 -90
- package/dist/docs/reference/sessions.html +37 -37
- package/dist/docs/reference/sessions.md +66 -113
- package/dist/docs/reference/skills.html +39 -39
- package/dist/docs/reference/skills.md +11 -31
- package/dist/docs/reference/subagents.html +37 -37
- package/dist/docs/reference/subagents.md +17 -31
- package/dist/docs/reference/tools.html +47 -64
- package/dist/docs/reference/tools.md +48 -130
- package/dist/docs/templates/agentic-owners.html +38 -38
- package/dist/docs/templates/agentic-owners.md +50 -74
- package/dist/docs/templates/reviews.html +69 -0
- package/dist/docs/templates/reviews.md +71 -0
- package/dist/docs/troubleshooting.html +38 -36
- package/dist/docs/troubleshooting.md +48 -101
- package/dist/evals.d.ts +2 -2
- package/dist/extensions/jev/extension.d.ts +12 -2
- package/dist/extensions/jev/extension.d.ts.map +1 -1
- package/dist/extensions/jev/extension.js +4 -0
- package/dist/extensions/jev/lib/evaluate.d.ts +31 -4
- package/dist/extensions/jev/lib/evaluate.d.ts.map +1 -1
- package/dist/extensions/jev/lib/evaluate.js +18 -4
- package/dist/internal/cli/command.d.ts +1 -1
- package/dist/internal/cli/command.js +1 -1
- package/dist/internal/cli/commands/channels.js +6 -6
- package/dist/internal/cli/commands/develop.d.ts.map +1 -1
- package/dist/internal/cli/commands/develop.js +23 -14
- package/dist/internal/cli/commands/hosted.js +4 -4
- package/dist/internal/cli/commands/index.d.ts +1 -1
- package/dist/internal/cli/commands/index.d.ts.map +1 -1
- package/dist/internal/cli/commands/index.js +1 -1
- package/dist/internal/cli/commands/mcp.js +1 -1
- package/dist/internal/cli/commands/project.js +4 -4
- package/dist/internal/cli/commands/scaffold.d.ts.map +1 -1
- package/dist/internal/cli/commands/scaffold.js +9 -6
- package/dist/internal/cli/commands/serve.d.ts +1 -1
- package/dist/internal/cli/commands/serve.js +5 -5
- package/dist/internal/cli/commands/session.d.ts.map +1 -1
- package/dist/internal/cli/commands/session.js +12 -7
- package/dist/internal/cli/dispatch.d.ts +3 -3
- package/dist/internal/cli/dispatch.js +3 -3
- package/dist/internal/cli/errors.d.ts +1 -1
- package/dist/internal/cli/errors.d.ts.map +1 -1
- package/dist/internal/cli/errors.js +1 -1
- package/dist/internal/cli/flags.d.ts +2 -2
- package/dist/internal/cli/flags.d.ts.map +1 -1
- package/dist/internal/cli/flags.js +2 -2
- package/dist/internal/cli/help-command.d.ts +1 -1
- package/dist/internal/cli/help-command.js +2 -2
- package/dist/internal/cli/help.d.ts +7 -7
- package/dist/internal/cli/help.d.ts.map +1 -1
- package/dist/internal/cli/help.js +8 -8
- package/dist/internal/cli-ax.d.ts.map +1 -1
- package/dist/internal/cli-ax.js +3 -0
- package/dist/internal/cli-convert-automation.d.ts +1 -1
- package/dist/internal/cli-convert-automation.js +1 -1
- package/dist/internal/cli-deploy.js +1 -1
- package/dist/internal/cli-docs.js +1 -1
- package/dist/internal/cli-images.js +1 -1
- package/dist/internal/cli-migrate.d.ts +1 -1
- package/dist/internal/cli-migrate.js +1 -1
- package/dist/internal/cli-pagerduty.d.ts +1 -1
- package/dist/internal/cli-pagerduty.js +1 -1
- package/dist/internal/cli-skills.js +1 -1
- package/dist/internal/convert-automation/types.d.ts +1 -1
- package/dist/internal/convert-automation/types.js +1 -1
- package/dist/internal/cursor/credentials.js +1 -1
- package/dist/internal/deploy-client.d.ts +2 -2
- package/dist/internal/deploy-client.d.ts.map +1 -1
- package/dist/internal/deploy-manifest.d.ts +1 -0
- package/dist/internal/deploy-manifest.d.ts.map +1 -1
- package/dist/internal/deploy-manifest.js +0 -1
- package/dist/internal/discovery/agent-config.d.ts +2 -6
- package/dist/internal/discovery/agent-config.d.ts.map +1 -1
- package/dist/internal/discovery/agent-config.js +8 -25
- package/dist/internal/discovery/agent.d.ts.map +1 -1
- package/dist/internal/discovery/agent.js +2 -3
- package/dist/internal/discovery/project.d.ts.map +1 -1
- package/dist/internal/discovery/project.js +15 -0
- package/dist/internal/init-project.d.ts +7 -3
- package/dist/internal/init-project.d.ts.map +1 -1
- package/dist/internal/init-project.js +60 -12
- package/dist/internal/init-scaffold.d.ts +2 -1
- package/dist/internal/init-scaffold.d.ts.map +1 -1
- package/dist/internal/init-scaffold.js +20 -7
- package/dist/internal/peer-connections.js +1 -1
- package/dist/internal/reminder-runner.js +1 -1
- package/dist/internal/server.d.ts +1 -1
- package/dist/internal/session-cost.d.ts +4 -4
- package/dist/internal/session-cost.d.ts.map +1 -1
- package/dist/internal/session-cost.js +4 -4
- package/dist/internal/session-engine.d.ts +1 -1
- package/dist/internal/session-engine.js +2 -2
- package/dist/internal/shallow-clone.js +1 -1
- package/dist/playground/assets/index-D6mVn9og.js +62 -0
- package/dist/playground/assets/index-DlALM6gr.css +1 -0
- package/dist/playground/index.html +2 -2
- package/dist/types.d.ts +9 -22
- package/dist/types.d.ts.map +1 -1
- package/docs/README.md +49 -82
- package/docs/building-with-agents.md +11 -104
- package/docs/changelog.md +6 -146
- package/docs/deployment.md +86 -265
- package/docs/evals.md +34 -128
- package/docs/guides/agent-to-agent.md +27 -64
- package/docs/guides/bitbucket.md +73 -134
- package/docs/guides/cloud-agents.md +23 -53
- package/docs/guides/convert-automation.md +31 -115
- package/docs/guides/github.md +49 -131
- package/docs/guides/gitlab.md +104 -184
- package/docs/guides/improve.md +24 -90
- package/docs/guides/jev.md +57 -159
- package/docs/guides/mcp-oauth.md +34 -69
- package/docs/guides/migrate.md +56 -130
- package/docs/guides/opentelemetry.md +37 -27
- package/docs/guides/pagerduty.md +68 -200
- package/docs/guides/slack.md +76 -205
- package/docs/guides/webhooks.md +64 -146
- package/docs/hillclimbing.md +41 -103
- package/docs/quickstart.md +93 -178
- package/docs/reference/access.md +7 -12
- package/docs/reference/agent-config.md +93 -176
- package/docs/reference/artifacts.md +38 -62
- package/docs/reference/channels.md +102 -151
- package/docs/reference/cli.md +233 -698
- package/docs/reference/connections.md +83 -164
- package/docs/reference/evals.md +97 -128
- package/docs/reference/extensions.md +37 -70
- package/docs/reference/hooks.md +61 -132
- package/docs/reference/http-api.md +66 -118
- package/docs/reference/instructions.md +20 -30
- package/docs/reference/limits.md +10 -22
- package/docs/reference/memory.md +38 -75
- package/docs/reference/playground.md +14 -22
- package/docs/reference/project-layout.md +38 -84
- package/docs/reference/schedules.md +59 -90
- package/docs/reference/sessions.md +66 -113
- package/docs/reference/skills.md +12 -32
- package/docs/reference/subagents.md +17 -31
- package/docs/reference/tools.md +48 -130
- package/docs/templates/agentic-owners.md +50 -74
- package/docs/templates/reviews.md +76 -0
- package/docs/troubleshooting.md +48 -101
- package/package.json +3 -9
- package/skills/cli/SKILL.md +15 -15
- package/skills/create-agent/SKILL.md +20 -19
- package/skills/debug/SKILL.md +4 -4
- package/skills/deploy/SKILL.md +2 -2
- package/skills/evals/SKILL.md +8 -8
- package/skills/framework-map/SKILL.md +3 -3
- package/skills/github/SKILL.md +7 -7
- package/skills/hillclimb/SKILL.md +4 -4
- package/skills/mcp-auth/SKILL.md +3 -3
- package/skills/migrate-v2/SKILL.md +40 -41
- package/skills/otel/SKILL.md +2 -2
- package/skills/setup-slack/SKILL.md +8 -8
- package/templates/agentic-owners/README.md +5 -5
- package/templates/agentic-owners/bot/tools/record_review.ts +1 -1
- package/templates/agentic-owners/package.json +1 -1
- package/templates/agents-md/README.md +4 -4
- package/templates/agents-md/package.json +1 -1
- package/templates/code-wiki/README.md +3 -3
- package/templates/code-wiki/package.json +1 -1
- package/templates/demo/README.md +3 -3
- package/templates/demo/package.json +1 -1
- package/templates/pr-autofixer/package.json +1 -1
- package/templates/security-help/README.md +5 -5
- package/templates/security-help/package.json +1 -1
- package/templates/security-reviewer/README.md +3 -3
- package/templates/security-reviewer/package.json +1 -1
- package/templates/thermo-quality-review/README.md +4 -4
- package/templates/thermo-quality-review/package.json +1 -1
- package/templates/thermo-review/README.md +4 -4
- package/templates/thermo-review/package.json +1 -1
- package/templates/triage/README.md +3 -3
- package/templates/triage/overlays/jira/README.md +3 -3
- package/templates/triage/overlays/linear/README.md +3 -3
- package/templates/triage/package.json +1 -1
- package/dist/docs/assets/building-with-agents.md.Bf1yH2tG.js +0 -9
- package/dist/docs/assets/changelog.md.fZ-NNFeJ.js +0 -1
- package/dist/docs/assets/changelog.md.fZ-NNFeJ.lean.js +0 -1
- package/dist/docs/assets/chunks/@localSearchIndexroot.mfzjiOsL.js +0 -1
- package/dist/docs/assets/chunks/channel.goFR5lbe.js +0 -1
- package/dist/docs/assets/chunks/classDiagram-6PBFFD2Q.B2yLtFja.js +0 -1
- package/dist/docs/assets/chunks/classDiagram-v2-HSJHXN6E.B2yLtFja.js +0 -1
- package/dist/docs/assets/chunks/clone.CztV97Wu.js +0 -1
- package/dist/docs/assets/chunks/stateDiagram-v2-QKLJ7IA2.CZlVHdDZ.js +0 -1
- package/dist/docs/assets/deployment.md.DDmjAVbg.js +0 -31
- package/dist/docs/assets/evals.md.OjKb7tNA.js +0 -68
- package/dist/docs/assets/evals.md.OjKb7tNA.lean.js +0 -1
- package/dist/docs/assets/guides_agent-to-agent.md.D1D5f6Jc.lean.js +0 -1
- package/dist/docs/assets/guides_cloud-agents.md.Ceyjv0ul.js +0 -9
- package/dist/docs/assets/guides_convert-automation.md.BjraCvok.js +0 -43
- package/dist/docs/assets/guides_grokbot-agents.md.D0PpM_Gf.js +0 -15
- package/dist/docs/assets/guides_grokbot-agents.md.D0PpM_Gf.lean.js +0 -1
- package/dist/docs/assets/guides_hooks.md.BjeEiM7s.js +0 -54
- package/dist/docs/assets/guides_hooks.md.BjeEiM7s.lean.js +0 -1
- package/dist/docs/assets/guides_improve.md.BFYtrtLE.js +0 -21
- package/dist/docs/assets/guides_migrate.md.RS9Eu-Xh.js +0 -8
- package/dist/docs/assets/guides_opentelemetry.md.CoZ1kNyd.js +0 -28
- package/dist/docs/assets/guides_opentelemetry.md.CoZ1kNyd.lean.js +0 -1
- package/dist/docs/assets/guides_pagerduty.md.mnKXe9U0.js +0 -60
- package/dist/docs/assets/guides_pagerduty.md.mnKXe9U0.lean.js +0 -1
- package/dist/docs/assets/guides_slack.md.BqiGSYGX.js +0 -88
- package/dist/docs/assets/hillclimbing.md.By61f3uq.js +0 -7
- package/dist/docs/assets/hillclimbing.md.By61f3uq.lean.js +0 -1
- package/dist/docs/assets/index.md.CVRjO09Y.js +0 -27
- package/dist/docs/assets/index.md.CVRjO09Y.lean.js +0 -1
- package/dist/docs/assets/quickstart.md.DaLk4ZIF.js +0 -170
- package/dist/docs/assets/reference_access.md.CExlH_mR.js +0 -1
- package/dist/docs/assets/reference_agent-config.md.11E_xe--.js +0 -40
- package/dist/docs/assets/reference_artifacts.md.C-hKR2hp.js +0 -18
- package/dist/docs/assets/reference_channels.md.DsJfGSBJ.js +0 -22
- package/dist/docs/assets/reference_cli.md.DQ3dCgmk.js +0 -99
- package/dist/docs/assets/reference_cli.md.DQ3dCgmk.lean.js +0 -1
- package/dist/docs/assets/reference_connections.md.BoAotr7_.js +0 -66
- package/dist/docs/assets/reference_evals.md.DYKjGFi-.js +0 -56
- package/dist/docs/assets/reference_hooks.md.B-bwJuiG.js +0 -51
- package/dist/docs/assets/reference_hooks.md.B-bwJuiG.lean.js +0 -1
- package/dist/docs/assets/reference_http-api.md._RWELr3d.js +0 -11
- package/dist/docs/assets/reference_instructions.md.yL2hXLnl.js +0 -14
- package/dist/docs/assets/reference_instructions.md.yL2hXLnl.lean.js +0 -1
- package/dist/docs/assets/reference_limits.md.Da1HegjB.js +0 -8
- package/dist/docs/assets/reference_limits.md.Da1HegjB.lean.js +0 -1
- package/dist/docs/assets/reference_memory.md.Dd7cyV9V.js +0 -4
- package/dist/docs/assets/reference_playground.md.CQiJk9CM.js +0 -1
- package/dist/docs/assets/reference_project-layout.md.BamcPDUn.js +0 -25
- package/dist/docs/assets/reference_project-layout.md.BamcPDUn.lean.js +0 -1
- package/dist/docs/assets/reference_prompt.md.BMjm1Jp3.js +0 -9
- package/dist/docs/assets/reference_prompt.md.BMjm1Jp3.lean.js +0 -1
- package/dist/docs/assets/reference_schedules.md.RC47aFm4.js +0 -47
- package/dist/docs/assets/reference_sessions.md.C7KFhWXV.js +0 -22
- package/dist/docs/assets/reference_subagents.md.CAhvW4yS.js +0 -10
- package/dist/docs/assets/reference_tools.md.DQr7Fnhu.js +0 -79
- package/dist/docs/assets/templates_agentic-owners.md.Dh2nDSqg.js +0 -13
- package/dist/docs/assets/templates_security-reviewer.md.DnwXdnB5.js +0 -3
- package/dist/docs/assets/templates_security-reviewer.md.DnwXdnB5.lean.js +0 -1
- package/dist/docs/assets/templates_thermo-quality-review.md.MJIdHkzE.js +0 -3
- package/dist/docs/assets/templates_thermo-quality-review.md.MJIdHkzE.lean.js +0 -1
- package/dist/docs/assets/templates_thermo-review.md.DfN1OFOl.js +0 -3
- package/dist/docs/assets/templates_thermo-review.md.DfN1OFOl.lean.js +0 -1
- package/dist/docs/assets/templates_triage.md.BYWMloV4.js +0 -9
- package/dist/docs/assets/templates_triage.md.BYWMloV4.lean.js +0 -1
- package/dist/docs/assets/troubleshooting.md.AGJ3jDzj.js +0 -1
- package/dist/docs/guides/grokbot-agents.html +0 -74
- package/dist/docs/guides/grokbot-agents.md +0 -129
- package/dist/docs/guides/hooks.html +0 -113
- package/dist/docs/guides/hooks.md +0 -116
- package/dist/docs/reference/prompt.html +0 -68
- package/dist/docs/reference/prompt.md +0 -47
- package/dist/docs/templates/security-reviewer.html +0 -62
- package/dist/docs/templates/security-reviewer.md +0 -81
- package/dist/docs/templates/thermo-quality-review.html +0 -62
- package/dist/docs/templates/thermo-quality-review.md +0 -75
- package/dist/docs/templates/thermo-review.html +0 -62
- package/dist/docs/templates/thermo-review.md +0 -75
- package/dist/docs/templates/triage.html +0 -68
- package/dist/docs/templates/triage.md +0 -101
- package/dist/extensions/cursor-grokbot-agents/extension.d.ts +0 -71
- package/dist/extensions/cursor-grokbot-agents/extension.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/extension.js +0 -64
- package/dist/extensions/cursor-grokbot-agents/lib/agents.d.ts +0 -11
- package/dist/extensions/cursor-grokbot-agents/lib/agents.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/agents.js +0 -31
- package/dist/extensions/cursor-grokbot-agents/lib/backend.d.ts +0 -56
- package/dist/extensions/cursor-grokbot-agents/lib/backend.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/backend.js +0 -139
- package/dist/extensions/cursor-grokbot-agents/lib/contacts.d.ts +0 -28
- package/dist/extensions/cursor-grokbot-agents/lib/contacts.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/contacts.js +0 -46
- package/dist/extensions/cursor-grokbot-agents/lib/conversation.d.ts +0 -53
- package/dist/extensions/cursor-grokbot-agents/lib/conversation.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/conversation.js +0 -186
- package/dist/extensions/cursor-grokbot-agents/lib/inputs.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/lib/inputs.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/inputs.js +0 -15
- package/dist/extensions/cursor-grokbot-agents/lib/lock.d.ts +0 -12
- package/dist/extensions/cursor-grokbot-agents/lib/lock.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/lock.js +0 -24
- package/dist/extensions/cursor-grokbot-agents/skills/consult.md +0 -40
- package/dist/extensions/cursor-grokbot-agents/tools/ask.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/ask.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/ask.js +0 -26
- package/dist/extensions/cursor-grokbot-agents/tools/check.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/check.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/check.js +0 -21
- package/dist/extensions/cursor-grokbot-agents/tools/interrupt.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/interrupt.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/interrupt.js +0 -13
- package/dist/extensions/cursor-grokbot-agents/tools/list.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/list.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/list.js +0 -22
- package/dist/playground/assets/index-BMpVzZzC.css +0 -1
- package/dist/playground/assets/index-_3zz3yh8.js +0 -62
- package/docs/guides/grokbot-agents.md +0 -134
- package/docs/guides/hooks.md +0 -121
- package/docs/reference/prompt.md +0 -52
- package/docs/templates/security-reviewer.md +0 -86
- package/docs/templates/thermo-quality-review.md +0 -80
- package/docs/templates/thermo-review.md +0 -80
- package/docs/templates/triage.md +0 -106
package/dist/docs/evals.md
CHANGED
|
@@ -1,21 +1,13 @@
|
|
|
1
1
|
# Evals
|
|
2
2
|
|
|
3
|
-
An eval sends a fixed input to the real agent and checks the
|
|
4
|
-
|
|
5
|
-
the answer took. Use evals to protect behavior you already understand,
|
|
6
|
-
not to discover what the prompt should do.
|
|
3
|
+
An eval sends a fixed input to the real agent and checks which tools ran and what the answer looks
|
|
4
|
+
like. To improve an agent and lock each win with an eval, follow [Hillclimbing](/docs/hillclimbing.md).
|
|
7
5
|
|
|
8
|
-
Evals run real
|
|
9
|
-
running a suite against an agent that can post, merge, or deploy.
|
|
6
|
+
Evals run real tools, so [guard any writes](#keep-side-effects-out-of-eval-sessions) first.
|
|
10
7
|
|
|
11
8
|
## Protect a tool decision
|
|
12
9
|
|
|
13
|
-
|
|
14
|
-
tool, and fails if the agent tries to approve. The CLI reports all three
|
|
15
|
-
decisions together instead of stopping at the first miss.
|
|
16
|
-
|
|
17
|
-
Use the [Jev extension](/docs/guides/jev.md) when host code needs a typed
|
|
18
|
-
choice, score, or boolean before it writes.
|
|
10
|
+
The case below passes only if the agent calls the inspection tool and doesn't approve.
|
|
19
11
|
|
|
20
12
|
```ts
|
|
21
13
|
// evals/readiness.eval.ts
|
|
@@ -24,9 +16,7 @@ import { defineEval, includes } from "@grok/agent-sdk/evals";
|
|
|
24
16
|
export default defineEval({
|
|
25
17
|
tags: ["smoke"],
|
|
26
18
|
async test(t) {
|
|
27
|
-
await t.send(
|
|
28
|
-
"Is https://github.com/acme/checkout/pull/42 ready to approve?"
|
|
29
|
-
);
|
|
19
|
+
await t.send("Is https://github.com/acme/checkout/pull/42 ready to approve?");
|
|
30
20
|
|
|
31
21
|
t.succeeded();
|
|
32
22
|
t.calledTool("inspect_pr");
|
|
@@ -36,8 +26,7 @@ export default defineEval({
|
|
|
36
26
|
});
|
|
37
27
|
```
|
|
38
28
|
|
|
39
|
-
Evals live in
|
|
40
|
-
Add the required concurrency config once:
|
|
29
|
+
Evals live in `evals/` beside `bot/`. Set how many cases run at once:
|
|
41
30
|
|
|
42
31
|
```ts
|
|
43
32
|
// evals/evals.config.ts
|
|
@@ -48,52 +37,34 @@ export default defineEvalConfig({
|
|
|
48
37
|
});
|
|
49
38
|
```
|
|
50
39
|
|
|
51
|
-
```
|
|
40
|
+
```bash
|
|
52
41
|
agent-sdk eval --dir . --list
|
|
53
42
|
agent-sdk eval --dir . --tag smoke
|
|
54
43
|
```
|
|
55
44
|
|
|
56
45
|
## Check answer shape or quality
|
|
57
46
|
|
|
58
|
-
|
|
59
|
-
failure tells you which field was wrong, and the case remains stable
|
|
60
|
-
when the model changes its wording.
|
|
47
|
+
When the output has a contract, check its shape:
|
|
61
48
|
|
|
62
49
|
```ts
|
|
63
50
|
import { matches } from "@grok/agent-sdk/evals";
|
|
64
51
|
import { z } from "zod";
|
|
65
52
|
|
|
66
53
|
const verdict = JSON.parse(t.reply ?? "{}");
|
|
67
|
-
t.check(
|
|
68
|
-
verdict,
|
|
69
|
-
matches(
|
|
70
|
-
z.object({
|
|
71
|
-
ready: z.boolean(),
|
|
72
|
-
blockers: z.array(z.string()),
|
|
73
|
-
})
|
|
74
|
-
)
|
|
75
|
-
);
|
|
54
|
+
t.check(verdict, matches(z.object({ ready: z.boolean(), blockers: z.array(z.string()) })));
|
|
76
55
|
```
|
|
77
56
|
|
|
78
|
-
|
|
79
|
-
|
|
57
|
+
When correctness depends on meaning, use a judge. [Judges](/docs/reference/evals.md#judges) covers
|
|
58
|
+
models and grader options.
|
|
80
59
|
|
|
81
60
|
```ts
|
|
82
|
-
t.judge
|
|
83
|
-
.factuality("The lint step failed on src/sidebar.ts.")
|
|
84
|
-
.atLeast(0.8);
|
|
61
|
+
t.judge.factuality("The lint step failed on src/sidebar.ts.").atLeast(0.8);
|
|
85
62
|
```
|
|
86
63
|
|
|
87
|
-
Judge checks are tracked scores until you give them a hard gate.
|
|
88
|
-
Configure the judge model in `evals.config.ts`. The
|
|
89
|
-
[Evals reference](/docs/reference/evals.md#judges) owns grader options and
|
|
90
|
-
severity rules.
|
|
91
|
-
|
|
92
64
|
## Test a conversation
|
|
93
65
|
|
|
94
|
-
Several `t.send` calls in one case share a session.
|
|
95
|
-
|
|
96
|
-
the whole conversation.
|
|
66
|
+
Several `t.send` calls in one case share a session. An assertion on a returned turn checks only that
|
|
67
|
+
turn, and an assertion on `t` checks the whole conversation.
|
|
97
68
|
|
|
98
69
|
```ts
|
|
99
70
|
const first = await t.send("Inspect PR 42.");
|
|
@@ -107,9 +78,7 @@ t.succeeded();
|
|
|
107
78
|
|
|
108
79
|
## Pin realistic inputs
|
|
109
80
|
|
|
110
|
-
|
|
111
|
-
prompts, seed workspace evidence directly, and save webhook payloads
|
|
112
|
-
instead of depending on a developer's checkout or live external state.
|
|
81
|
+
Seed the files the agent reads instead of depending on your checkout:
|
|
113
82
|
|
|
114
83
|
```ts
|
|
115
84
|
await t.send("Review pr/diff.patch and report blockers.", {
|
|
@@ -122,47 +91,32 @@ await t.send("Review pr/diff.patch and report blockers.", {
|
|
|
122
91
|
});
|
|
123
92
|
```
|
|
124
93
|
|
|
125
|
-
|
|
94
|
+
Save a GitHub pull request's webhook payloads as fixtures:
|
|
126
95
|
|
|
127
|
-
```
|
|
128
|
-
agent-sdk github replay acme/checkout
|
|
129
|
-
--events '*'
|
|
130
|
-
--dry-run \
|
|
131
|
-
--out fixtures/github
|
|
96
|
+
```bash
|
|
97
|
+
agent-sdk github replay https://github.com/acme/checkout/pull/42 \
|
|
98
|
+
--events '*' --dry-run --fixtures-dir fixtures/github
|
|
132
99
|
```
|
|
133
100
|
|
|
134
|
-
|
|
135
|
-
drift-prone live cases behind an explicit selection. See
|
|
136
|
-
[Datasets and fixtures](/docs/reference/evals.md#datasets-and-fixtures) for
|
|
137
|
-
loaders and case expansion.
|
|
101
|
+
[Datasets and fixtures](/docs/reference/evals.md#datasets-and-fixtures) covers loaders for larger sets.
|
|
138
102
|
|
|
139
103
|
## Run and debug a regression suite
|
|
140
104
|
|
|
141
|
-
Run one case while
|
|
142
|
-
failure artifact records the input, assertions, tool calls, and final
|
|
143
|
-
text, making it the first place to look before changing the prompt.
|
|
105
|
+
Run one case while you iterate, then the smoke set before review:
|
|
144
106
|
|
|
145
|
-
```
|
|
107
|
+
```bash
|
|
146
108
|
agent-sdk eval --dir . readiness
|
|
147
109
|
agent-sdk eval --dir . --tag smoke
|
|
148
|
-
agent-sdk eval --dir . --json
|
|
149
110
|
```
|
|
150
111
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
Model turns need a credential. Resolution checks `CURSOR_API_KEY`,
|
|
157
|
-
`CURSOR_API_KEY_FILE`, `CURSOR_SERVICE_ACCOUNT_KEY`, then a saved
|
|
158
|
-
`agent-sdk login`. `eval --list` only discovers cases and needs none.
|
|
112
|
+
Each run writes a folder under `evals/` in the project state directory, or under `--artifacts <dir>`
|
|
113
|
+
if you pass it. Open `evals/<case-id>.json` in that folder to see a failed case's input, assertions,
|
|
114
|
+
tool calls, and final text. Model turns need a
|
|
115
|
+
[Cursor credential](/docs/reference/cli.md#environment-variables).
|
|
159
116
|
|
|
160
117
|
## Run evals in CI
|
|
161
118
|
|
|
162
|
-
|
|
163
|
-
gate fail the job:
|
|
164
|
-
|
|
165
|
-
```text
|
|
119
|
+
```bash
|
|
166
120
|
# CURSOR_API_KEY comes from the CI secret store.
|
|
167
121
|
agent-sdk eval --dir . --json --no-stream \
|
|
168
122
|
--junit reports/evals.xml \
|
|
@@ -170,70 +124,22 @@ agent-sdk eval --dir . --json --no-stream \
|
|
|
170
124
|
> reports/evals.json
|
|
171
125
|
```
|
|
172
126
|
|
|
173
|
-
Run `--tag smoke` on pull requests and the full suite on a schedule
|
|
174
|
-
|
|
175
|
-
for a new probabilistic benchmark until its threshold is trustworthy.
|
|
176
|
-
|
|
177
|
-
The [CLI reference](/docs/reference/cli.md#eval) covers selectors,
|
|
178
|
-
server-side batches, exit codes, and JSON output.
|
|
127
|
+
Run `--tag smoke` on pull requests and the full suite on a schedule. The
|
|
128
|
+
[CLI reference](/docs/reference/cli.md#eval) covers every flag.
|
|
179
129
|
|
|
180
130
|
## Keep side effects out of eval sessions
|
|
181
131
|
|
|
182
|
-
Eval sessions carry `purpose: "eval"`. Check it
|
|
183
|
-
|
|
184
|
-
action:
|
|
132
|
+
Eval sessions carry `purpose: "eval"`. Check it where the tool writes, and return the decision
|
|
133
|
+
without the external action:
|
|
185
134
|
|
|
186
135
|
```ts
|
|
187
136
|
async execute({ findings }, ctx) {
|
|
188
137
|
if (ctx.session.purpose === "eval") {
|
|
189
|
-
return {
|
|
190
|
-
posted: false,
|
|
191
|
-
reason: "eval",
|
|
192
|
-
count: findings.length,
|
|
193
|
-
};
|
|
138
|
+
return { posted: false, reason: "eval", count: findings.length };
|
|
194
139
|
}
|
|
195
140
|
|
|
196
141
|
return await postFindings(findings);
|
|
197
142
|
}
|
|
198
143
|
```
|
|
199
144
|
|
|
200
|
-
|
|
201
|
-
assert that the agent chose to post.
|
|
202
|
-
|
|
203
|
-
## Evals, hooks, or hillclimbing?
|
|
204
|
-
|
|
205
|
-
Use an eval to gate one fixed input. Use a
|
|
206
|
-
[hook](/docs/reference/hooks.md) to observe every live session for metrics,
|
|
207
|
-
auditing, or alerts. Use [hillclimbing](/docs/hillclimbing.md) to decide
|
|
208
|
-
which source change improves a fixed set of inputs; every kept
|
|
209
|
-
hillclimb change should add an eval.
|
|
210
|
-
|
|
211
|
-
## Gates and scores
|
|
212
|
-
|
|
213
|
-
`t.succeeded()`, tool assertions, and deterministic checks are hard
|
|
214
|
-
gates by default. A failed gate fails the case. `.soft()` records a
|
|
215
|
-
measurement without blocking, while `.atLeast(threshold)` records a
|
|
216
|
-
score and marks a miss as `scored`.
|
|
217
|
-
|
|
218
|
-
Start with decisions and output shape: the intended tool, the tempting
|
|
219
|
-
wrong tool, and one structural check. Exact prose is rarely a stable
|
|
220
|
-
contract.
|
|
221
|
-
|
|
222
|
-
## Keep improvements with regression evals
|
|
223
|
-
|
|
224
|
-
Every kept improvement needs an eval that would have failed before the
|
|
225
|
-
change. A tool-choice gate, bounded `maxToolCalls`, or output-shape check
|
|
226
|
-
turns the improvement into a durable contract.
|
|
227
|
-
|
|
228
|
-
Never weaken an existing gate to make a new implementation pass. That
|
|
229
|
-
moves the freeze line instead of proving the change.
|
|
230
|
-
|
|
231
|
-
## Related
|
|
232
|
-
|
|
233
|
-
- [Evals reference](/docs/reference/evals.md): complete authoring and runner
|
|
234
|
-
contracts
|
|
235
|
-
- [`skills/evals/SKILL.md`](https://github.com/cursor/cursor/blob/main/packages/agent-serve/skills/evals/SKILL.md): author and seed
|
|
236
|
-
cases with a coding agent
|
|
237
|
-
- [Hillclimbing](/docs/hillclimbing.md): measure a change and lock the win
|
|
238
|
-
- [Sessions](/docs/reference/sessions.md): events available to trajectory
|
|
239
|
-
assertions
|
|
145
|
+
The tool call stays in the trajectory, so the eval can still assert that the agent chose to post.
|