@grok/agent-sdk 0.2.28 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +29 -29
- package/dist/bin/agent-serve.d.ts +2 -2
- package/dist/bin/agent-serve.js +3 -3
- package/dist/channels/pagerduty/api.js +1 -1
- package/dist/channels/pagerduty/pagerduty-channel.d.ts +1 -1
- package/dist/channels/pagerduty/pagerduty-channel.d.ts.map +1 -1
- package/dist/channels/pagerduty/pagerduty-channel.js +1 -1
- package/dist/channels/pagerduty/replay.d.ts +1 -1
- package/dist/channels/pagerduty/replay.js +4 -4
- package/dist/channels/slack/cli-project.js +1 -1
- package/dist/docs/404.html +2 -2
- package/dist/docs/assets/{app.DF9vlcdn.js → app.BzTUBOAD.js} +4 -4
- package/dist/docs/assets/building-with-agents.md.BqJ7AAjG.js +6 -0
- package/dist/docs/assets/{building-with-agents.md.Bf1yH2tG.lean.js → building-with-agents.md.BqJ7AAjG.lean.js} +1 -1
- package/dist/docs/assets/changelog.md.BWhC6DET.js +1 -0
- package/dist/docs/assets/changelog.md.BWhC6DET.lean.js +1 -0
- package/dist/docs/assets/chunks/@localSearchIndexroot.CWxwVoCG.js +1 -0
- package/dist/docs/assets/chunks/{VPLocalSearchBox.D8OuwKO0.js → VPLocalSearchBox.DsnRlUT5.js} +1 -1
- package/dist/docs/assets/chunks/{arc.DCq_TCYV.js → arc.DH8GOhiv.js} +1 -1
- package/dist/docs/assets/chunks/{architectureDiagram-Q4EWVU46.PWRNTPbk.js → architectureDiagram-Q4EWVU46.9m9ObyuE.js} +1 -1
- package/dist/docs/assets/chunks/{baseUniq.BeRPvRUL.js → baseUniq.BXHpx1lE.js} +1 -1
- package/dist/docs/assets/chunks/{blockDiagram-DXYQGD6D.BuRmZVSo.js → blockDiagram-DXYQGD6D.xY3EX59q.js} +1 -1
- package/dist/docs/assets/chunks/{c4Diagram-AHTNJAMY.7DKSMwuv.js → c4Diagram-AHTNJAMY.DnqgMVgc.js} +1 -1
- package/dist/docs/assets/chunks/channel.Cctv0f7X.js +1 -0
- package/dist/docs/assets/chunks/{chunk-4BX2VUAB.7VrO9-9i.js → chunk-4BX2VUAB.BjFxK2A3.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-4TB4RGXK.CHystNE2.js → chunk-4TB4RGXK.BujS_Spt.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-55IACEB6.Nuh9TbTq.js → chunk-55IACEB6.YbTjhdaM.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-EDXVE4YY.CA0EKugm.js → chunk-EDXVE4YY.DnQ8t4GC.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-FMBD7UC4.F9_mp7U8.js → chunk-FMBD7UC4.B67AwAmj.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-OYMX7WX6.CEU4QfE1.js → chunk-OYMX7WX6.DrZIAFOm.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-QZHKN3VN.B8h_x9_i.js → chunk-QZHKN3VN.BLmmydkn.js} +1 -1
- package/dist/docs/assets/chunks/{chunk-YZCP3GAM.Btfb-rKg.js → chunk-YZCP3GAM.DaoOfw0O.js} +1 -1
- package/dist/docs/assets/chunks/classDiagram-6PBFFD2Q.DU5sqchW.js +1 -0
- package/dist/docs/assets/chunks/classDiagram-v2-HSJHXN6E.DU5sqchW.js +1 -0
- package/dist/docs/assets/chunks/clone.B9rdNt6j.js +1 -0
- package/dist/docs/assets/chunks/{cose-bilkent-S5V4N54A.DKvc96Wa.js → cose-bilkent-S5V4N54A.DMQuNhM5.js} +1 -1
- package/dist/docs/assets/chunks/{dagre-KV5264BT.CMo0cy1b.js → dagre-KV5264BT.CZoGVuUf.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-5BDNPKRD.DDgMOQhB.js → diagram-5BDNPKRD.CLt9uOyL.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-G4DWMVQ6.BdJnoIsU.js → diagram-G4DWMVQ6.DpczEm1G.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-MMDJMWI5.oSvhAZaL.js → diagram-MMDJMWI5.D4r13DvJ.js} +1 -1
- package/dist/docs/assets/chunks/{diagram-TYMM5635.C9zZLIE-.js → diagram-TYMM5635.CYU9eGFu.js} +1 -1
- package/dist/docs/assets/chunks/{erDiagram-SMLLAGMA.BNhVyccy.js → erDiagram-SMLLAGMA.QA9naWEw.js} +1 -1
- package/dist/docs/assets/chunks/{flowDiagram-DWJPFMVM.BEFLLBiU.js → flowDiagram-DWJPFMVM.BbNoVZYd.js} +1 -1
- package/dist/docs/assets/chunks/{ganttDiagram-T4ZO3ILL.BdUzXBgY.js → ganttDiagram-T4ZO3ILL.IHjDoG7r.js} +1 -1
- package/dist/docs/assets/chunks/{gitGraphDiagram-UUTBAWPF.Rd3LlOIJ.js → gitGraphDiagram-UUTBAWPF.B21VMdf5.js} +1 -1
- package/dist/docs/assets/chunks/{graph.DnJC1GpG.js → graph.DgJ4L-8V.js} +1 -1
- package/dist/docs/assets/chunks/{infoDiagram-42DDH7IO.xv0xntDJ.js → infoDiagram-42DDH7IO.CQjrcthc.js} +1 -1
- package/dist/docs/assets/chunks/{ishikawaDiagram-UXIWVN3A.BaGIlviU.js → ishikawaDiagram-UXIWVN3A.D4tPGtVo.js} +1 -1
- package/dist/docs/assets/chunks/{journeyDiagram-VCZTEJTY.Cof_h0fp.js → journeyDiagram-VCZTEJTY.BD3WaNbQ.js} +1 -1
- package/dist/docs/assets/chunks/{kanban-definition-6JOO6SKY.C5GozFuU.js → kanban-definition-6JOO6SKY.Ca0o4Tqz.js} +1 -1
- package/dist/docs/assets/chunks/{layout.zc9UShLc.js → layout.DXXqX4MQ.js} +1 -1
- package/dist/docs/assets/chunks/{linear.BoqrFVcJ.js → linear.5c61gNw4.js} +1 -1
- package/dist/docs/assets/chunks/{min.DwIfr8hG.js → min.B1-NAjHX.js} +1 -1
- package/dist/docs/assets/chunks/{mindmap-definition-QFDTVHPH.Dkq2yTtp.js → mindmap-definition-QFDTVHPH.BELznv3-.js} +1 -1
- package/dist/docs/assets/chunks/{pieDiagram-DEJITSTG.B4CpTmHK.js → pieDiagram-DEJITSTG.qPzO1mq5.js} +1 -1
- package/dist/docs/assets/chunks/{quadrantDiagram-34T5L4WZ.BqXnFC9C.js → quadrantDiagram-34T5L4WZ.CxIvzbjX.js} +1 -1
- package/dist/docs/assets/chunks/{requirementDiagram-MS252O5E.DhIN2z-E.js → requirementDiagram-MS252O5E.CqA0iJx8.js} +1 -1
- package/dist/docs/assets/chunks/{sankeyDiagram-XADWPNL6.BhxUAmg2.js → sankeyDiagram-XADWPNL6.XRTutBnb.js} +1 -1
- package/dist/docs/assets/chunks/{sequenceDiagram-FGHM5R23.BqgU7HU-.js → sequenceDiagram-FGHM5R23.E--7bRLQ.js} +1 -1
- package/dist/docs/assets/chunks/{stateDiagram-FHFEXIEX.B1BCO18N.js → stateDiagram-FHFEXIEX.C4vGu8zq.js} +1 -1
- package/dist/docs/assets/chunks/stateDiagram-v2-QKLJ7IA2.ECMu2BdJ.js +1 -0
- package/dist/docs/assets/chunks/{theme.oV7rddEz.js → theme.DMj_7q3B.js} +2 -2
- package/dist/docs/assets/chunks/{timeline-definition-GMOUNBTQ.oRI8UwMC.js → timeline-definition-GMOUNBTQ.C7LgNVAL.js} +1 -1
- package/dist/docs/assets/chunks/{vennDiagram-DHZGUBPP.DmuWrIxN.js → vennDiagram-DHZGUBPP.HUwoZcy_.js} +1 -1
- package/dist/docs/assets/chunks/{wardley-RL74JXVD.CEBjM5ov.js → wardley-RL74JXVD.BxFEFYZS.js} +1 -1
- package/dist/docs/assets/chunks/{wardleyDiagram-NUSXRM2D.m5zsCOF9.js → wardleyDiagram-NUSXRM2D.B2pSo5rB.js} +1 -1
- package/dist/docs/assets/chunks/{xychartDiagram-5P7HB3ND.cPN-587n.js → xychartDiagram-5P7HB3ND.NQhMiOOL.js} +1 -1
- package/dist/docs/assets/deployment.md.BtqJrHhp.js +31 -0
- package/dist/docs/assets/{deployment.md.DDmjAVbg.lean.js → deployment.md.BtqJrHhp.lean.js} +1 -1
- package/dist/docs/assets/evals.md.Dne6VRmZ.js +49 -0
- package/dist/docs/assets/evals.md.Dne6VRmZ.lean.js +1 -0
- package/dist/docs/assets/{guides_agent-to-agent.md.D1D5f6Jc.js → guides_agent-to-agent.md.BcmSqL2z.js} +8 -8
- package/dist/docs/assets/guides_agent-to-agent.md.BcmSqL2z.lean.js +1 -0
- package/dist/docs/assets/{guides_bitbucket.md.DkjkBRU8.js → guides_bitbucket.md.BiA6ZMFW.js} +27 -27
- package/dist/docs/assets/{guides_bitbucket.md.DkjkBRU8.lean.js → guides_bitbucket.md.BiA6ZMFW.lean.js} +1 -1
- package/dist/docs/assets/guides_cloud-agents.md.CiXAW6on.js +9 -0
- package/dist/docs/assets/{guides_cloud-agents.md.Ceyjv0ul.lean.js → guides_cloud-agents.md.CiXAW6on.lean.js} +1 -1
- package/dist/docs/assets/guides_convert-automation.md.BvTxsJ8J.js +26 -0
- package/dist/docs/assets/{guides_convert-automation.md.BjraCvok.lean.js → guides_convert-automation.md.BvTxsJ8J.lean.js} +1 -1
- package/dist/docs/assets/{guides_github.md.CgN5sNbW.js → guides_github.md.6jiEH7dg.js} +11 -19
- package/dist/docs/assets/{guides_github.md.CgN5sNbW.lean.js → guides_github.md.6jiEH7dg.lean.js} +1 -1
- package/dist/docs/assets/{guides_gitlab.md.DCE0p7_w.js → guides_gitlab.md.DcKusOcU.js} +63 -73
- package/dist/docs/assets/{guides_gitlab.md.DCE0p7_w.lean.js → guides_gitlab.md.DcKusOcU.lean.js} +1 -1
- package/dist/docs/assets/guides_improve.md.o3WOGR7n.js +16 -0
- package/dist/docs/assets/{guides_improve.md.BFYtrtLE.lean.js → guides_improve.md.o3WOGR7n.lean.js} +1 -1
- package/dist/docs/assets/{guides_jev.md.Ccw4pedT.js → guides_jev.md.DGCzkMUN.js} +20 -60
- package/dist/docs/assets/{guides_jev.md.Ccw4pedT.lean.js → guides_jev.md.DGCzkMUN.lean.js} +1 -1
- package/dist/docs/assets/{guides_mcp-oauth.md.ekpWOZCC.js → guides_mcp-oauth.md.C-Sx_IPa.js} +3 -5
- package/dist/docs/assets/{guides_mcp-oauth.md.ekpWOZCC.lean.js → guides_mcp-oauth.md.C-Sx_IPa.lean.js} +1 -1
- package/dist/docs/assets/guides_migrate.md.BbhHYeCr.js +3 -0
- package/dist/docs/assets/{guides_migrate.md.RS9Eu-Xh.lean.js → guides_migrate.md.BbhHYeCr.lean.js} +1 -1
- package/dist/docs/assets/guides_opentelemetry.md.BRSktBHt.js +26 -0
- package/dist/docs/assets/guides_opentelemetry.md.BRSktBHt.lean.js +1 -0
- package/dist/docs/assets/guides_pagerduty.md.pMXUkZZc.js +61 -0
- package/dist/docs/assets/guides_pagerduty.md.pMXUkZZc.lean.js +1 -0
- package/dist/docs/assets/guides_slack.md.BgIGG9ur.js +82 -0
- package/dist/docs/assets/{guides_slack.md.BqiGSYGX.lean.js → guides_slack.md.BgIGG9ur.lean.js} +1 -1
- package/dist/docs/assets/{guides_webhooks.md.BJk63ZTs.js → guides_webhooks.md.B5DXt2v1.js} +7 -7
- package/dist/docs/assets/{guides_webhooks.md.BJk63ZTs.lean.js → guides_webhooks.md.B5DXt2v1.lean.js} +1 -1
- package/dist/docs/assets/hillclimbing.md.3TkdF79A.js +5 -0
- package/dist/docs/assets/hillclimbing.md.3TkdF79A.lean.js +1 -0
- package/dist/docs/assets/index.md.DRfaoRCZ.js +20 -0
- package/dist/docs/assets/index.md.DRfaoRCZ.lean.js +1 -0
- package/dist/docs/assets/quickstart.md.r-oE8nkX.js +113 -0
- package/dist/docs/assets/{quickstart.md.DaLk4ZIF.lean.js → quickstart.md.r-oE8nkX.lean.js} +1 -1
- package/dist/docs/assets/reference_access.md.C5lblfUC.js +1 -0
- package/dist/docs/assets/{reference_access.md.CExlH_mR.lean.js → reference_access.md.C5lblfUC.lean.js} +1 -1
- package/dist/docs/assets/reference_agent-config.md.C0Tp_Fk9.js +27 -0
- package/dist/docs/assets/{reference_agent-config.md.11E_xe--.lean.js → reference_agent-config.md.C0Tp_Fk9.lean.js} +1 -1
- package/dist/docs/assets/reference_artifacts.md.BJpAI7h9.js +17 -0
- package/dist/docs/assets/{reference_artifacts.md.C-hKR2hp.lean.js → reference_artifacts.md.BJpAI7h9.lean.js} +1 -1
- package/dist/docs/assets/reference_channels.md.CPWOpdsC.js +21 -0
- package/dist/docs/assets/{reference_channels.md.DsJfGSBJ.lean.js → reference_channels.md.CPWOpdsC.lean.js} +1 -1
- package/dist/docs/assets/reference_cli.md.B8GdpaRj.js +83 -0
- package/dist/docs/assets/reference_cli.md.B8GdpaRj.lean.js +1 -0
- package/dist/docs/assets/reference_connections.md.DoY4vrDH.js +54 -0
- package/dist/docs/assets/{reference_connections.md.BoAotr7_.lean.js → reference_connections.md.DoY4vrDH.lean.js} +1 -1
- package/dist/docs/assets/reference_evals.md.DgfUtfIc.js +69 -0
- package/dist/docs/assets/{reference_evals.md.DYKjGFi-.lean.js → reference_evals.md.DgfUtfIc.lean.js} +1 -1
- package/dist/docs/assets/{reference_extensions.md.pEVj72Xo.js → reference_extensions.md.DHbW6eOf.js} +6 -8
- package/dist/docs/assets/{reference_extensions.md.pEVj72Xo.lean.js → reference_extensions.md.DHbW6eOf.lean.js} +1 -1
- package/dist/docs/assets/reference_hooks.md.CSWmOwMM.js +39 -0
- package/dist/docs/assets/reference_hooks.md.CSWmOwMM.lean.js +1 -0
- package/dist/docs/assets/reference_http-api.md.C7Cb0r2J.js +7 -0
- package/dist/docs/assets/{reference_http-api.md._RWELr3d.lean.js → reference_http-api.md.C7Cb0r2J.lean.js} +1 -1
- package/dist/docs/assets/reference_instructions.md.CV6Qq0UX.js +14 -0
- package/dist/docs/assets/reference_instructions.md.CV6Qq0UX.lean.js +1 -0
- package/dist/docs/assets/reference_limits.md.CbdtFr4T.js +8 -0
- package/dist/docs/assets/reference_limits.md.CbdtFr4T.lean.js +1 -0
- package/dist/docs/assets/reference_memory.md.CdoJISM-.js +3 -0
- package/dist/docs/assets/{reference_memory.md.Dd7cyV9V.lean.js → reference_memory.md.CdoJISM-.lean.js} +1 -1
- package/dist/docs/assets/reference_playground.md.DIUsRAcy.js +1 -0
- package/dist/docs/assets/{reference_playground.md.CQiJk9CM.lean.js → reference_playground.md.DIUsRAcy.lean.js} +1 -1
- package/dist/docs/assets/reference_project-layout.md.9DiKoE16.js +26 -0
- package/dist/docs/assets/reference_project-layout.md.9DiKoE16.lean.js +1 -0
- package/dist/docs/assets/reference_schedules.md.RhQ9ZLo1.js +45 -0
- package/dist/docs/assets/{reference_schedules.md.RC47aFm4.lean.js → reference_schedules.md.RhQ9ZLo1.lean.js} +1 -1
- package/dist/docs/assets/reference_sessions.md.NXVQt9mP.js +22 -0
- package/dist/docs/assets/{reference_sessions.md.C7KFhWXV.lean.js → reference_sessions.md.NXVQt9mP.lean.js} +1 -1
- package/dist/docs/assets/{reference_skills.md.Dej2zQxi.js → reference_skills.md.DzCPNPkG.js} +4 -4
- package/dist/docs/assets/{reference_skills.md.Dej2zQxi.lean.js → reference_skills.md.DzCPNPkG.lean.js} +1 -1
- package/dist/docs/assets/reference_subagents.md.Dvpq2sUu.js +10 -0
- package/dist/docs/assets/{reference_subagents.md.CAhvW4yS.lean.js → reference_subagents.md.Dvpq2sUu.lean.js} +1 -1
- package/dist/docs/assets/reference_tools.md.B2OaR1Nb.js +62 -0
- package/dist/docs/assets/{reference_tools.md.DQr7Fnhu.lean.js → reference_tools.md.B2OaR1Nb.lean.js} +1 -1
- package/dist/docs/assets/templates_agentic-owners.md.B6ZmKAzT.js +13 -0
- package/dist/docs/assets/{templates_agentic-owners.md.Dh2nDSqg.lean.js → templates_agentic-owners.md.B6ZmKAzT.lean.js} +1 -1
- package/dist/docs/assets/templates_reviews.md.dr9Xv_KM.js +10 -0
- package/dist/docs/assets/templates_reviews.md.dr9Xv_KM.lean.js +1 -0
- package/dist/docs/assets/troubleshooting.md.DfKKvW1R.js +3 -0
- package/dist/docs/assets/{troubleshooting.md.AGJ3jDzj.lean.js → troubleshooting.md.DfKKvW1R.lean.js} +1 -1
- package/dist/docs/building-with-agents.html +41 -44
- package/dist/docs/building-with-agents.md +11 -104
- package/dist/docs/changelog.html +37 -37
- package/dist/docs/changelog.md +5 -145
- package/dist/docs/deployment.html +54 -54
- package/dist/docs/deployment.md +86 -265
- package/dist/docs/evals.html +52 -71
- package/dist/docs/evals.md +33 -127
- package/dist/docs/guides/agent-to-agent.html +43 -43
- package/dist/docs/guides/agent-to-agent.md +27 -64
- package/dist/docs/guides/bitbucket.html +62 -62
- package/dist/docs/guides/bitbucket.md +73 -134
- package/dist/docs/guides/cloud-agents.html +37 -37
- package/dist/docs/guides/cloud-agents.md +23 -53
- package/dist/docs/guides/convert-automation.html +39 -56
- package/dist/docs/guides/convert-automation.md +31 -115
- package/dist/docs/guides/github.html +46 -54
- package/dist/docs/guides/github.md +49 -131
- package/dist/docs/guides/gitlab.html +98 -108
- package/dist/docs/guides/gitlab.md +104 -184
- package/dist/docs/guides/improve.html +38 -43
- package/dist/docs/guides/improve.md +24 -90
- package/dist/docs/guides/jev.html +55 -95
- package/dist/docs/guides/jev.md +57 -159
- package/dist/docs/guides/mcp-oauth.html +38 -40
- package/dist/docs/guides/mcp-oauth.md +34 -69
- package/dist/docs/guides/migrate.html +38 -43
- package/dist/docs/guides/migrate.md +56 -130
- package/dist/docs/guides/opentelemetry.html +41 -43
- package/dist/docs/guides/opentelemetry.md +32 -27
- package/dist/docs/guides/pagerduty.html +50 -49
- package/dist/docs/guides/pagerduty.md +65 -199
- package/dist/docs/guides/slack.html +51 -57
- package/dist/docs/guides/slack.md +76 -205
- package/dist/docs/guides/webhooks.html +42 -42
- package/dist/docs/guides/webhooks.md +64 -146
- package/dist/docs/hashmap.json +1 -1
- package/dist/docs/hillclimbing.html +40 -42
- package/dist/docs/hillclimbing.md +40 -102
- package/dist/docs/index.html +55 -62
- package/dist/docs/index.md +49 -82
- package/dist/docs/llms-full.txt +2292 -5830
- package/dist/docs/llms.txt +9 -15
- package/dist/docs/quickstart.html +96 -153
- package/dist/docs/quickstart.md +93 -178
- package/dist/docs/reference/access.html +36 -36
- package/dist/docs/reference/access.md +7 -12
- package/dist/docs/reference/agent-config.html +43 -56
- package/dist/docs/reference/agent-config.md +93 -176
- package/dist/docs/reference/artifacts.html +38 -39
- package/dist/docs/reference/artifacts.md +38 -62
- package/dist/docs/reference/channels.html +38 -39
- package/dist/docs/reference/channels.md +102 -151
- package/dist/docs/reference/cli.html +68 -84
- package/dist/docs/reference/cli.md +233 -698
- package/dist/docs/reference/connections.html +45 -57
- package/dist/docs/reference/connections.md +83 -164
- package/dist/docs/reference/evals.html +66 -53
- package/dist/docs/reference/evals.md +97 -128
- package/dist/docs/reference/extensions.html +41 -43
- package/dist/docs/reference/extensions.md +37 -70
- package/dist/docs/reference/hooks.html +40 -52
- package/dist/docs/reference/hooks.md +62 -133
- package/dist/docs/reference/http-api.html +38 -42
- package/dist/docs/reference/http-api.md +66 -118
- package/dist/docs/reference/instructions.html +39 -39
- package/dist/docs/reference/instructions.md +19 -29
- package/dist/docs/reference/limits.html +38 -38
- package/dist/docs/reference/limits.md +9 -21
- package/dist/docs/reference/memory.html +37 -38
- package/dist/docs/reference/memory.md +38 -75
- package/dist/docs/reference/playground.html +36 -36
- package/dist/docs/reference/playground.md +14 -22
- package/dist/docs/reference/project-layout.html +49 -48
- package/dist/docs/reference/project-layout.md +38 -84
- package/dist/docs/reference/schedules.html +46 -48
- package/dist/docs/reference/schedules.md +59 -90
- package/dist/docs/reference/sessions.html +37 -37
- package/dist/docs/reference/sessions.md +66 -113
- package/dist/docs/reference/skills.html +39 -39
- package/dist/docs/reference/skills.md +11 -31
- package/dist/docs/reference/subagents.html +37 -37
- package/dist/docs/reference/subagents.md +17 -31
- package/dist/docs/reference/tools.html +47 -64
- package/dist/docs/reference/tools.md +48 -130
- package/dist/docs/templates/agentic-owners.html +38 -38
- package/dist/docs/templates/agentic-owners.md +50 -74
- package/dist/docs/templates/reviews.html +69 -0
- package/dist/docs/templates/reviews.md +71 -0
- package/dist/docs/troubleshooting.html +38 -36
- package/dist/docs/troubleshooting.md +48 -101
- package/dist/evals.d.ts +2 -2
- package/dist/extensions/jev/extension.d.ts +12 -2
- package/dist/extensions/jev/extension.d.ts.map +1 -1
- package/dist/extensions/jev/extension.js +4 -0
- package/dist/extensions/jev/lib/evaluate.d.ts +31 -4
- package/dist/extensions/jev/lib/evaluate.d.ts.map +1 -1
- package/dist/extensions/jev/lib/evaluate.js +18 -4
- package/dist/internal/cli/command.d.ts +1 -1
- package/dist/internal/cli/command.js +1 -1
- package/dist/internal/cli/commands/channels.js +6 -6
- package/dist/internal/cli/commands/develop.d.ts.map +1 -1
- package/dist/internal/cli/commands/develop.js +23 -14
- package/dist/internal/cli/commands/hosted.js +4 -4
- package/dist/internal/cli/commands/index.d.ts +1 -1
- package/dist/internal/cli/commands/index.d.ts.map +1 -1
- package/dist/internal/cli/commands/index.js +1 -1
- package/dist/internal/cli/commands/mcp.js +1 -1
- package/dist/internal/cli/commands/project.js +4 -4
- package/dist/internal/cli/commands/scaffold.d.ts.map +1 -1
- package/dist/internal/cli/commands/scaffold.js +9 -6
- package/dist/internal/cli/commands/serve.d.ts +1 -1
- package/dist/internal/cli/commands/serve.js +5 -5
- package/dist/internal/cli/commands/session.d.ts.map +1 -1
- package/dist/internal/cli/commands/session.js +12 -7
- package/dist/internal/cli/dispatch.d.ts +3 -3
- package/dist/internal/cli/dispatch.js +3 -3
- package/dist/internal/cli/errors.d.ts +1 -1
- package/dist/internal/cli/errors.d.ts.map +1 -1
- package/dist/internal/cli/errors.js +1 -1
- package/dist/internal/cli/flags.d.ts +2 -2
- package/dist/internal/cli/flags.d.ts.map +1 -1
- package/dist/internal/cli/flags.js +2 -2
- package/dist/internal/cli/help-command.d.ts +1 -1
- package/dist/internal/cli/help-command.js +2 -2
- package/dist/internal/cli/help.d.ts +7 -7
- package/dist/internal/cli/help.d.ts.map +1 -1
- package/dist/internal/cli/help.js +8 -8
- package/dist/internal/cli-ax.d.ts.map +1 -1
- package/dist/internal/cli-ax.js +3 -0
- package/dist/internal/cli-convert-automation.d.ts +1 -1
- package/dist/internal/cli-convert-automation.js +1 -1
- package/dist/internal/cli-deploy.js +1 -1
- package/dist/internal/cli-docs.js +1 -1
- package/dist/internal/cli-images.js +1 -1
- package/dist/internal/cli-migrate.d.ts +1 -1
- package/dist/internal/cli-migrate.js +1 -1
- package/dist/internal/cli-pagerduty.d.ts +1 -1
- package/dist/internal/cli-pagerduty.js +1 -1
- package/dist/internal/cli-skills.js +1 -1
- package/dist/internal/convert-automation/types.d.ts +1 -1
- package/dist/internal/convert-automation/types.js +1 -1
- package/dist/internal/cursor/credentials.js +1 -1
- package/dist/internal/deploy-client.d.ts +2 -2
- package/dist/internal/deploy-client.d.ts.map +1 -1
- package/dist/internal/deploy-manifest.d.ts +1 -0
- package/dist/internal/deploy-manifest.d.ts.map +1 -1
- package/dist/internal/deploy-manifest.js +0 -1
- package/dist/internal/discovery/agent-config.d.ts +2 -6
- package/dist/internal/discovery/agent-config.d.ts.map +1 -1
- package/dist/internal/discovery/agent-config.js +8 -25
- package/dist/internal/discovery/agent.d.ts.map +1 -1
- package/dist/internal/discovery/agent.js +2 -3
- package/dist/internal/discovery/project.d.ts.map +1 -1
- package/dist/internal/discovery/project.js +15 -0
- package/dist/internal/init-project.d.ts +7 -3
- package/dist/internal/init-project.d.ts.map +1 -1
- package/dist/internal/init-project.js +60 -12
- package/dist/internal/init-scaffold.d.ts +2 -1
- package/dist/internal/init-scaffold.d.ts.map +1 -1
- package/dist/internal/init-scaffold.js +20 -7
- package/dist/internal/peer-connections.js +1 -1
- package/dist/internal/reminder-runner.js +1 -1
- package/dist/internal/server.d.ts +1 -1
- package/dist/internal/session-cost.d.ts +4 -4
- package/dist/internal/session-cost.d.ts.map +1 -1
- package/dist/internal/session-cost.js +4 -4
- package/dist/internal/session-engine.d.ts +1 -1
- package/dist/internal/session-engine.js +2 -2
- package/dist/internal/shallow-clone.js +1 -1
- package/dist/playground/assets/index-D6mVn9og.js +62 -0
- package/dist/playground/assets/index-DlALM6gr.css +1 -0
- package/dist/playground/index.html +2 -2
- package/dist/types.d.ts +9 -22
- package/dist/types.d.ts.map +1 -1
- package/docs/README.md +49 -82
- package/docs/building-with-agents.md +11 -104
- package/docs/changelog.md +6 -146
- package/docs/deployment.md +86 -265
- package/docs/evals.md +34 -128
- package/docs/guides/agent-to-agent.md +27 -64
- package/docs/guides/bitbucket.md +73 -134
- package/docs/guides/cloud-agents.md +23 -53
- package/docs/guides/convert-automation.md +31 -115
- package/docs/guides/github.md +49 -131
- package/docs/guides/gitlab.md +104 -184
- package/docs/guides/improve.md +24 -90
- package/docs/guides/jev.md +57 -159
- package/docs/guides/mcp-oauth.md +34 -69
- package/docs/guides/migrate.md +56 -130
- package/docs/guides/opentelemetry.md +37 -27
- package/docs/guides/pagerduty.md +68 -200
- package/docs/guides/slack.md +76 -205
- package/docs/guides/webhooks.md +64 -146
- package/docs/hillclimbing.md +41 -103
- package/docs/quickstart.md +93 -178
- package/docs/reference/access.md +7 -12
- package/docs/reference/agent-config.md +93 -176
- package/docs/reference/artifacts.md +38 -62
- package/docs/reference/channels.md +102 -151
- package/docs/reference/cli.md +233 -698
- package/docs/reference/connections.md +83 -164
- package/docs/reference/evals.md +97 -128
- package/docs/reference/extensions.md +37 -70
- package/docs/reference/hooks.md +61 -132
- package/docs/reference/http-api.md +66 -118
- package/docs/reference/instructions.md +20 -30
- package/docs/reference/limits.md +10 -22
- package/docs/reference/memory.md +38 -75
- package/docs/reference/playground.md +14 -22
- package/docs/reference/project-layout.md +38 -84
- package/docs/reference/schedules.md +59 -90
- package/docs/reference/sessions.md +66 -113
- package/docs/reference/skills.md +12 -32
- package/docs/reference/subagents.md +17 -31
- package/docs/reference/tools.md +48 -130
- package/docs/templates/agentic-owners.md +50 -74
- package/docs/templates/reviews.md +76 -0
- package/docs/troubleshooting.md +48 -101
- package/package.json +3 -9
- package/skills/cli/SKILL.md +15 -15
- package/skills/create-agent/SKILL.md +20 -19
- package/skills/debug/SKILL.md +4 -4
- package/skills/deploy/SKILL.md +2 -2
- package/skills/evals/SKILL.md +8 -8
- package/skills/framework-map/SKILL.md +3 -3
- package/skills/github/SKILL.md +7 -7
- package/skills/hillclimb/SKILL.md +4 -4
- package/skills/mcp-auth/SKILL.md +3 -3
- package/skills/migrate-v2/SKILL.md +40 -41
- package/skills/otel/SKILL.md +2 -2
- package/skills/setup-slack/SKILL.md +8 -8
- package/templates/agentic-owners/README.md +5 -5
- package/templates/agentic-owners/bot/tools/record_review.ts +1 -1
- package/templates/agentic-owners/package.json +1 -1
- package/templates/agents-md/README.md +4 -4
- package/templates/agents-md/package.json +1 -1
- package/templates/code-wiki/README.md +3 -3
- package/templates/code-wiki/package.json +1 -1
- package/templates/demo/README.md +3 -3
- package/templates/demo/package.json +1 -1
- package/templates/pr-autofixer/package.json +1 -1
- package/templates/security-help/README.md +5 -5
- package/templates/security-help/package.json +1 -1
- package/templates/security-reviewer/README.md +3 -3
- package/templates/security-reviewer/package.json +1 -1
- package/templates/thermo-quality-review/README.md +4 -4
- package/templates/thermo-quality-review/package.json +1 -1
- package/templates/thermo-review/README.md +4 -4
- package/templates/thermo-review/package.json +1 -1
- package/templates/triage/README.md +3 -3
- package/templates/triage/overlays/jira/README.md +3 -3
- package/templates/triage/overlays/linear/README.md +3 -3
- package/templates/triage/package.json +1 -1
- package/dist/docs/assets/building-with-agents.md.Bf1yH2tG.js +0 -9
- package/dist/docs/assets/changelog.md.fZ-NNFeJ.js +0 -1
- package/dist/docs/assets/changelog.md.fZ-NNFeJ.lean.js +0 -1
- package/dist/docs/assets/chunks/@localSearchIndexroot.mfzjiOsL.js +0 -1
- package/dist/docs/assets/chunks/channel.goFR5lbe.js +0 -1
- package/dist/docs/assets/chunks/classDiagram-6PBFFD2Q.B2yLtFja.js +0 -1
- package/dist/docs/assets/chunks/classDiagram-v2-HSJHXN6E.B2yLtFja.js +0 -1
- package/dist/docs/assets/chunks/clone.CztV97Wu.js +0 -1
- package/dist/docs/assets/chunks/stateDiagram-v2-QKLJ7IA2.CZlVHdDZ.js +0 -1
- package/dist/docs/assets/deployment.md.DDmjAVbg.js +0 -31
- package/dist/docs/assets/evals.md.OjKb7tNA.js +0 -68
- package/dist/docs/assets/evals.md.OjKb7tNA.lean.js +0 -1
- package/dist/docs/assets/guides_agent-to-agent.md.D1D5f6Jc.lean.js +0 -1
- package/dist/docs/assets/guides_cloud-agents.md.Ceyjv0ul.js +0 -9
- package/dist/docs/assets/guides_convert-automation.md.BjraCvok.js +0 -43
- package/dist/docs/assets/guides_grokbot-agents.md.D0PpM_Gf.js +0 -15
- package/dist/docs/assets/guides_grokbot-agents.md.D0PpM_Gf.lean.js +0 -1
- package/dist/docs/assets/guides_hooks.md.BjeEiM7s.js +0 -54
- package/dist/docs/assets/guides_hooks.md.BjeEiM7s.lean.js +0 -1
- package/dist/docs/assets/guides_improve.md.BFYtrtLE.js +0 -21
- package/dist/docs/assets/guides_migrate.md.RS9Eu-Xh.js +0 -8
- package/dist/docs/assets/guides_opentelemetry.md.CoZ1kNyd.js +0 -28
- package/dist/docs/assets/guides_opentelemetry.md.CoZ1kNyd.lean.js +0 -1
- package/dist/docs/assets/guides_pagerduty.md.mnKXe9U0.js +0 -60
- package/dist/docs/assets/guides_pagerduty.md.mnKXe9U0.lean.js +0 -1
- package/dist/docs/assets/guides_slack.md.BqiGSYGX.js +0 -88
- package/dist/docs/assets/hillclimbing.md.By61f3uq.js +0 -7
- package/dist/docs/assets/hillclimbing.md.By61f3uq.lean.js +0 -1
- package/dist/docs/assets/index.md.CVRjO09Y.js +0 -27
- package/dist/docs/assets/index.md.CVRjO09Y.lean.js +0 -1
- package/dist/docs/assets/quickstart.md.DaLk4ZIF.js +0 -170
- package/dist/docs/assets/reference_access.md.CExlH_mR.js +0 -1
- package/dist/docs/assets/reference_agent-config.md.11E_xe--.js +0 -40
- package/dist/docs/assets/reference_artifacts.md.C-hKR2hp.js +0 -18
- package/dist/docs/assets/reference_channels.md.DsJfGSBJ.js +0 -22
- package/dist/docs/assets/reference_cli.md.DQ3dCgmk.js +0 -99
- package/dist/docs/assets/reference_cli.md.DQ3dCgmk.lean.js +0 -1
- package/dist/docs/assets/reference_connections.md.BoAotr7_.js +0 -66
- package/dist/docs/assets/reference_evals.md.DYKjGFi-.js +0 -56
- package/dist/docs/assets/reference_hooks.md.B-bwJuiG.js +0 -51
- package/dist/docs/assets/reference_hooks.md.B-bwJuiG.lean.js +0 -1
- package/dist/docs/assets/reference_http-api.md._RWELr3d.js +0 -11
- package/dist/docs/assets/reference_instructions.md.yL2hXLnl.js +0 -14
- package/dist/docs/assets/reference_instructions.md.yL2hXLnl.lean.js +0 -1
- package/dist/docs/assets/reference_limits.md.Da1HegjB.js +0 -8
- package/dist/docs/assets/reference_limits.md.Da1HegjB.lean.js +0 -1
- package/dist/docs/assets/reference_memory.md.Dd7cyV9V.js +0 -4
- package/dist/docs/assets/reference_playground.md.CQiJk9CM.js +0 -1
- package/dist/docs/assets/reference_project-layout.md.BamcPDUn.js +0 -25
- package/dist/docs/assets/reference_project-layout.md.BamcPDUn.lean.js +0 -1
- package/dist/docs/assets/reference_prompt.md.BMjm1Jp3.js +0 -9
- package/dist/docs/assets/reference_prompt.md.BMjm1Jp3.lean.js +0 -1
- package/dist/docs/assets/reference_schedules.md.RC47aFm4.js +0 -47
- package/dist/docs/assets/reference_sessions.md.C7KFhWXV.js +0 -22
- package/dist/docs/assets/reference_subagents.md.CAhvW4yS.js +0 -10
- package/dist/docs/assets/reference_tools.md.DQr7Fnhu.js +0 -79
- package/dist/docs/assets/templates_agentic-owners.md.Dh2nDSqg.js +0 -13
- package/dist/docs/assets/templates_security-reviewer.md.DnwXdnB5.js +0 -3
- package/dist/docs/assets/templates_security-reviewer.md.DnwXdnB5.lean.js +0 -1
- package/dist/docs/assets/templates_thermo-quality-review.md.MJIdHkzE.js +0 -3
- package/dist/docs/assets/templates_thermo-quality-review.md.MJIdHkzE.lean.js +0 -1
- package/dist/docs/assets/templates_thermo-review.md.DfN1OFOl.js +0 -3
- package/dist/docs/assets/templates_thermo-review.md.DfN1OFOl.lean.js +0 -1
- package/dist/docs/assets/templates_triage.md.BYWMloV4.js +0 -9
- package/dist/docs/assets/templates_triage.md.BYWMloV4.lean.js +0 -1
- package/dist/docs/assets/troubleshooting.md.AGJ3jDzj.js +0 -1
- package/dist/docs/guides/grokbot-agents.html +0 -74
- package/dist/docs/guides/grokbot-agents.md +0 -129
- package/dist/docs/guides/hooks.html +0 -113
- package/dist/docs/guides/hooks.md +0 -116
- package/dist/docs/reference/prompt.html +0 -68
- package/dist/docs/reference/prompt.md +0 -47
- package/dist/docs/templates/security-reviewer.html +0 -62
- package/dist/docs/templates/security-reviewer.md +0 -81
- package/dist/docs/templates/thermo-quality-review.html +0 -62
- package/dist/docs/templates/thermo-quality-review.md +0 -75
- package/dist/docs/templates/thermo-review.html +0 -62
- package/dist/docs/templates/thermo-review.md +0 -75
- package/dist/docs/templates/triage.html +0 -68
- package/dist/docs/templates/triage.md +0 -101
- package/dist/extensions/cursor-grokbot-agents/extension.d.ts +0 -71
- package/dist/extensions/cursor-grokbot-agents/extension.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/extension.js +0 -64
- package/dist/extensions/cursor-grokbot-agents/lib/agents.d.ts +0 -11
- package/dist/extensions/cursor-grokbot-agents/lib/agents.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/agents.js +0 -31
- package/dist/extensions/cursor-grokbot-agents/lib/backend.d.ts +0 -56
- package/dist/extensions/cursor-grokbot-agents/lib/backend.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/backend.js +0 -139
- package/dist/extensions/cursor-grokbot-agents/lib/contacts.d.ts +0 -28
- package/dist/extensions/cursor-grokbot-agents/lib/contacts.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/contacts.js +0 -46
- package/dist/extensions/cursor-grokbot-agents/lib/conversation.d.ts +0 -53
- package/dist/extensions/cursor-grokbot-agents/lib/conversation.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/conversation.js +0 -186
- package/dist/extensions/cursor-grokbot-agents/lib/inputs.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/lib/inputs.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/inputs.js +0 -15
- package/dist/extensions/cursor-grokbot-agents/lib/lock.d.ts +0 -12
- package/dist/extensions/cursor-grokbot-agents/lib/lock.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/lib/lock.js +0 -24
- package/dist/extensions/cursor-grokbot-agents/skills/consult.md +0 -40
- package/dist/extensions/cursor-grokbot-agents/tools/ask.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/ask.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/ask.js +0 -26
- package/dist/extensions/cursor-grokbot-agents/tools/check.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/check.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/check.js +0 -21
- package/dist/extensions/cursor-grokbot-agents/tools/interrupt.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/interrupt.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/interrupt.js +0 -13
- package/dist/extensions/cursor-grokbot-agents/tools/list.d.ts +0 -4
- package/dist/extensions/cursor-grokbot-agents/tools/list.d.ts.map +0 -1
- package/dist/extensions/cursor-grokbot-agents/tools/list.js +0 -22
- package/dist/playground/assets/index-BMpVzZzC.css +0 -1
- package/dist/playground/assets/index-_3zz3yh8.js +0 -62
- package/docs/guides/grokbot-agents.md +0 -134
- package/docs/guides/hooks.md +0 -121
- package/docs/reference/prompt.md +0 -52
- package/docs/templates/security-reviewer.md +0 -86
- package/docs/templates/thermo-quality-review.md +0 -80
- package/docs/templates/thermo-review.md +0 -80
- package/docs/templates/triage.md +0 -106
package/docs/evals.md
CHANGED
|
@@ -1,26 +1,18 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "Evals"
|
|
3
|
-
description: "
|
|
3
|
+
description: "Measure and Improve agent behavior with Evals"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Evals
|
|
7
7
|
|
|
8
|
-
An eval sends a fixed input to the real agent and checks the
|
|
9
|
-
|
|
10
|
-
the answer took. Use evals to protect behavior you already understand,
|
|
11
|
-
not to discover what the prompt should do.
|
|
8
|
+
An eval sends a fixed input to the real agent and checks which tools ran and what the answer looks
|
|
9
|
+
like. To improve an agent and lock each win with an eval, follow [Hillclimbing](./hillclimbing.md).
|
|
12
10
|
|
|
13
|
-
Evals run real
|
|
14
|
-
running a suite against an agent that can post, merge, or deploy.
|
|
11
|
+
Evals run real tools, so [guard any writes](#keep-side-effects-out-of-eval-sessions) first.
|
|
15
12
|
|
|
16
13
|
## Protect a tool decision
|
|
17
14
|
|
|
18
|
-
|
|
19
|
-
tool, and fails if the agent tries to approve. The CLI reports all three
|
|
20
|
-
decisions together instead of stopping at the first miss.
|
|
21
|
-
|
|
22
|
-
Use the [Jev extension](./guides/jev.md) when host code needs a typed
|
|
23
|
-
choice, score, or boolean before it writes.
|
|
15
|
+
The case below passes only if the agent calls the inspection tool and doesn't approve.
|
|
24
16
|
|
|
25
17
|
```ts
|
|
26
18
|
// evals/readiness.eval.ts
|
|
@@ -29,9 +21,7 @@ import { defineEval, includes } from "@grok/agent-sdk/evals";
|
|
|
29
21
|
export default defineEval({
|
|
30
22
|
tags: ["smoke"],
|
|
31
23
|
async test(t) {
|
|
32
|
-
await t.send(
|
|
33
|
-
"Is https://github.com/acme/checkout/pull/42 ready to approve?"
|
|
34
|
-
);
|
|
24
|
+
await t.send("Is https://github.com/acme/checkout/pull/42 ready to approve?");
|
|
35
25
|
|
|
36
26
|
t.succeeded();
|
|
37
27
|
t.calledTool("inspect_pr");
|
|
@@ -41,8 +31,7 @@ export default defineEval({
|
|
|
41
31
|
});
|
|
42
32
|
```
|
|
43
33
|
|
|
44
|
-
Evals live in
|
|
45
|
-
Add the required concurrency config once:
|
|
34
|
+
Evals live in `evals/` beside `bot/`. Set how many cases run at once:
|
|
46
35
|
|
|
47
36
|
```ts
|
|
48
37
|
// evals/evals.config.ts
|
|
@@ -53,52 +42,34 @@ export default defineEvalConfig({
|
|
|
53
42
|
});
|
|
54
43
|
```
|
|
55
44
|
|
|
56
|
-
```
|
|
45
|
+
```bash
|
|
57
46
|
agent-sdk eval --dir . --list
|
|
58
47
|
agent-sdk eval --dir . --tag smoke
|
|
59
48
|
```
|
|
60
49
|
|
|
61
50
|
## Check answer shape or quality
|
|
62
51
|
|
|
63
|
-
|
|
64
|
-
failure tells you which field was wrong, and the case remains stable
|
|
65
|
-
when the model changes its wording.
|
|
52
|
+
When the output has a contract, check its shape:
|
|
66
53
|
|
|
67
54
|
```ts
|
|
68
55
|
import { matches } from "@grok/agent-sdk/evals";
|
|
69
56
|
import { z } from "zod";
|
|
70
57
|
|
|
71
58
|
const verdict = JSON.parse(t.reply ?? "{}");
|
|
72
|
-
t.check(
|
|
73
|
-
verdict,
|
|
74
|
-
matches(
|
|
75
|
-
z.object({
|
|
76
|
-
ready: z.boolean(),
|
|
77
|
-
blockers: z.array(z.string()),
|
|
78
|
-
})
|
|
79
|
-
)
|
|
80
|
-
);
|
|
59
|
+
t.check(verdict, matches(z.object({ ready: z.boolean(), blockers: z.array(z.string()) })));
|
|
81
60
|
```
|
|
82
61
|
|
|
83
|
-
|
|
84
|
-
|
|
62
|
+
When correctness depends on meaning, use a judge. [Judges](./reference/evals.md#judges) covers
|
|
63
|
+
models and grader options.
|
|
85
64
|
|
|
86
65
|
```ts
|
|
87
|
-
t.judge
|
|
88
|
-
.factuality("The lint step failed on src/sidebar.ts.")
|
|
89
|
-
.atLeast(0.8);
|
|
66
|
+
t.judge.factuality("The lint step failed on src/sidebar.ts.").atLeast(0.8);
|
|
90
67
|
```
|
|
91
68
|
|
|
92
|
-
Judge checks are tracked scores until you give them a hard gate.
|
|
93
|
-
Configure the judge model in `evals.config.ts`. The
|
|
94
|
-
[Evals reference](./reference/evals.md#judges) owns grader options and
|
|
95
|
-
severity rules.
|
|
96
|
-
|
|
97
69
|
## Test a conversation
|
|
98
70
|
|
|
99
|
-
Several `t.send` calls in one case share a session.
|
|
100
|
-
|
|
101
|
-
the whole conversation.
|
|
71
|
+
Several `t.send` calls in one case share a session. An assertion on a returned turn checks only that
|
|
72
|
+
turn, and an assertion on `t` checks the whole conversation.
|
|
102
73
|
|
|
103
74
|
```ts
|
|
104
75
|
const first = await t.send("Inspect PR 42.");
|
|
@@ -112,9 +83,7 @@ t.succeeded();
|
|
|
112
83
|
|
|
113
84
|
## Pin realistic inputs
|
|
114
85
|
|
|
115
|
-
|
|
116
|
-
prompts, seed workspace evidence directly, and save webhook payloads
|
|
117
|
-
instead of depending on a developer's checkout or live external state.
|
|
86
|
+
Seed the files the agent reads instead of depending on your checkout:
|
|
118
87
|
|
|
119
88
|
```ts
|
|
120
89
|
await t.send("Review pr/diff.patch and report blockers.", {
|
|
@@ -127,47 +96,32 @@ await t.send("Review pr/diff.patch and report blockers.", {
|
|
|
127
96
|
});
|
|
128
97
|
```
|
|
129
98
|
|
|
130
|
-
|
|
99
|
+
Save a GitHub pull request's webhook payloads as fixtures:
|
|
131
100
|
|
|
132
|
-
```
|
|
133
|
-
agent-sdk github replay acme/checkout
|
|
134
|
-
--events '*'
|
|
135
|
-
--dry-run \
|
|
136
|
-
--out fixtures/github
|
|
101
|
+
```bash
|
|
102
|
+
agent-sdk github replay https://github.com/acme/checkout/pull/42 \
|
|
103
|
+
--events '*' --dry-run --fixtures-dir fixtures/github
|
|
137
104
|
```
|
|
138
105
|
|
|
139
|
-
|
|
140
|
-
drift-prone live cases behind an explicit selection. See
|
|
141
|
-
[Datasets and fixtures](./reference/evals.md#datasets-and-fixtures) for
|
|
142
|
-
loaders and case expansion.
|
|
106
|
+
[Datasets and fixtures](./reference/evals.md#datasets-and-fixtures) covers loaders for larger sets.
|
|
143
107
|
|
|
144
108
|
## Run and debug a regression suite
|
|
145
109
|
|
|
146
|
-
Run one case while
|
|
147
|
-
failure artifact records the input, assertions, tool calls, and final
|
|
148
|
-
text, making it the first place to look before changing the prompt.
|
|
110
|
+
Run one case while you iterate, then the smoke set before review:
|
|
149
111
|
|
|
150
|
-
```
|
|
112
|
+
```bash
|
|
151
113
|
agent-sdk eval --dir . readiness
|
|
152
114
|
agent-sdk eval --dir . --tag smoke
|
|
153
|
-
agent-sdk eval --dir . --json
|
|
154
115
|
```
|
|
155
116
|
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
Model turns need a credential. Resolution checks `CURSOR_API_KEY`,
|
|
162
|
-
`CURSOR_API_KEY_FILE`, `CURSOR_SERVICE_ACCOUNT_KEY`, then a saved
|
|
163
|
-
`agent-sdk login`. `eval --list` only discovers cases and needs none.
|
|
117
|
+
Each run writes a folder under `evals/` in the project state directory, or under `--artifacts <dir>`
|
|
118
|
+
if you pass it. Open `evals/<case-id>.json` in that folder to see a failed case's input, assertions,
|
|
119
|
+
tool calls, and final text. Model turns need a
|
|
120
|
+
[Cursor credential](./reference/cli.md#environment-variables).
|
|
164
121
|
|
|
165
122
|
## Run evals in CI
|
|
166
123
|
|
|
167
|
-
|
|
168
|
-
gate fail the job:
|
|
169
|
-
|
|
170
|
-
```text
|
|
124
|
+
```bash
|
|
171
125
|
# CURSOR_API_KEY comes from the CI secret store.
|
|
172
126
|
agent-sdk eval --dir . --json --no-stream \
|
|
173
127
|
--junit reports/evals.xml \
|
|
@@ -175,70 +129,22 @@ agent-sdk eval --dir . --json --no-stream \
|
|
|
175
129
|
> reports/evals.json
|
|
176
130
|
```
|
|
177
131
|
|
|
178
|
-
Run `--tag smoke` on pull requests and the full suite on a schedule
|
|
179
|
-
|
|
180
|
-
for a new probabilistic benchmark until its threshold is trustworthy.
|
|
181
|
-
|
|
182
|
-
The [CLI reference](./reference/cli.md#eval) covers selectors,
|
|
183
|
-
server-side batches, exit codes, and JSON output.
|
|
132
|
+
Run `--tag smoke` on pull requests and the full suite on a schedule. The
|
|
133
|
+
[CLI reference](./reference/cli.md#eval) covers every flag.
|
|
184
134
|
|
|
185
135
|
## Keep side effects out of eval sessions
|
|
186
136
|
|
|
187
|
-
Eval sessions carry `purpose: "eval"`. Check it
|
|
188
|
-
|
|
189
|
-
action:
|
|
137
|
+
Eval sessions carry `purpose: "eval"`. Check it where the tool writes, and return the decision
|
|
138
|
+
without the external action:
|
|
190
139
|
|
|
191
140
|
```ts
|
|
192
141
|
async execute({ findings }, ctx) {
|
|
193
142
|
if (ctx.session.purpose === "eval") {
|
|
194
|
-
return {
|
|
195
|
-
posted: false,
|
|
196
|
-
reason: "eval",
|
|
197
|
-
count: findings.length,
|
|
198
|
-
};
|
|
143
|
+
return { posted: false, reason: "eval", count: findings.length };
|
|
199
144
|
}
|
|
200
145
|
|
|
201
146
|
return await postFindings(findings);
|
|
202
147
|
}
|
|
203
148
|
```
|
|
204
149
|
|
|
205
|
-
|
|
206
|
-
assert that the agent chose to post.
|
|
207
|
-
|
|
208
|
-
## Evals, hooks, or hillclimbing?
|
|
209
|
-
|
|
210
|
-
Use an eval to gate one fixed input. Use a
|
|
211
|
-
[hook](./reference/hooks.md) to observe every live session for metrics,
|
|
212
|
-
auditing, or alerts. Use [hillclimbing](./hillclimbing.md) to decide
|
|
213
|
-
which source change improves a fixed set of inputs; every kept
|
|
214
|
-
hillclimb change should add an eval.
|
|
215
|
-
|
|
216
|
-
## Gates and scores
|
|
217
|
-
|
|
218
|
-
`t.succeeded()`, tool assertions, and deterministic checks are hard
|
|
219
|
-
gates by default. A failed gate fails the case. `.soft()` records a
|
|
220
|
-
measurement without blocking, while `.atLeast(threshold)` records a
|
|
221
|
-
score and marks a miss as `scored`.
|
|
222
|
-
|
|
223
|
-
Start with decisions and output shape: the intended tool, the tempting
|
|
224
|
-
wrong tool, and one structural check. Exact prose is rarely a stable
|
|
225
|
-
contract.
|
|
226
|
-
|
|
227
|
-
## Keep improvements with regression evals
|
|
228
|
-
|
|
229
|
-
Every kept improvement needs an eval that would have failed before the
|
|
230
|
-
change. A tool-choice gate, bounded `maxToolCalls`, or output-shape check
|
|
231
|
-
turns the improvement into a durable contract.
|
|
232
|
-
|
|
233
|
-
Never weaken an existing gate to make a new implementation pass. That
|
|
234
|
-
moves the freeze line instead of proving the change.
|
|
235
|
-
|
|
236
|
-
## Related
|
|
237
|
-
|
|
238
|
-
- [Evals reference](./reference/evals.md): complete authoring and runner
|
|
239
|
-
contracts
|
|
240
|
-
- [`skills/evals/SKILL.md`](../skills/evals/SKILL.md): author and seed
|
|
241
|
-
cases with a coding agent
|
|
242
|
-
- [Hillclimbing](./hillclimbing.md): measure a change and lock the win
|
|
243
|
-
- [Sessions](./reference/sessions.md): events available to trajectory
|
|
244
|
-
assertions
|
|
150
|
+
The tool call stays in the trajectory, so the eval can still assert that the agent chose to post.
|
|
@@ -5,18 +5,15 @@ description: "Let one Agent SDK agent consult an independent specialist, continu
|
|
|
5
5
|
|
|
6
6
|
# Peer agents
|
|
7
7
|
|
|
8
|
-
A peer connection lets one agent ask an independent specialist for help.
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
instructions, tools, and session while the caller decides when to delegate
|
|
13
|
-
and returns the answer to the user.
|
|
8
|
+
A peer connection lets one agent ask an independent specialist for help. The
|
|
9
|
+
specialist keeps its own instructions, tools, and sessions. On Cursor hosting,
|
|
10
|
+
team-owned agents reach each other by deployment slug, and Agent SDK handles
|
|
11
|
+
routing and authentication for you.
|
|
14
12
|
|
|
15
13
|
## Delegate a question to a specialist
|
|
16
14
|
|
|
17
|
-
Ask the concierge about the weather and it hands the question to the
|
|
18
|
-
|
|
19
|
-
concierge returns that answer to you.
|
|
15
|
+
Ask the concierge about the weather, and it hands the question to the weather
|
|
16
|
+
agent and returns the answer.
|
|
20
17
|
|
|
21
18
|
```ts
|
|
22
19
|
// bot/mcp-connections/weather.ts
|
|
@@ -28,30 +25,27 @@ export default defineConnection({
|
|
|
28
25
|
});
|
|
29
26
|
```
|
|
30
27
|
|
|
31
|
-
|
|
28
|
+
The filename, `weather.ts`, becomes the
|
|
29
|
+
[MCP server name](../reference/connections.md#peer-mcp-connection), and `agent`
|
|
30
|
+
is the specialist's lowercase deployment slug. Give the concierge a one-way
|
|
31
|
+
routing rule, so the agents never pass a request back and forth:
|
|
32
32
|
|
|
33
33
|
```md
|
|
34
34
|
When a request needs current weather, ask the `weather` peer. Return its
|
|
35
35
|
answer.
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
-
Deploy
|
|
38
|
+
Deploy both agents from pushed Git refs:
|
|
39
39
|
|
|
40
40
|
```bash
|
|
41
41
|
agent-sdk deploy --dir ./agents/weather-agent --slug weather-agent
|
|
42
42
|
agent-sdk deploy --dir ./agents/concierge --slug concierge
|
|
43
43
|
```
|
|
44
44
|
|
|
45
|
-
The connection filename, `weather.ts`, becomes the MCP server name the
|
|
46
|
-
concierge calls. Set `agent` to the specialist's lowercase deployment slug.
|
|
47
|
-
|
|
48
45
|
## Continue the specialist's session
|
|
49
46
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
place. The first `ask` returns `{ status, sessionId, reply }` as text on
|
|
53
|
-
the `callTool` result. Parse `sessionId` and pass it on the next `ask`
|
|
54
|
-
so the peer remembers its earlier work instead of starting fresh.
|
|
47
|
+
The first `ask` returns `{ status, sessionId, reply }` as text. Pass that
|
|
48
|
+
`sessionId` on the next `ask` to continue the specialist's earlier work.
|
|
55
49
|
|
|
56
50
|
```ts
|
|
57
51
|
const first = await ctx.host.mcp.callTool("weather", "ask", {
|
|
@@ -69,8 +63,8 @@ await ctx.host.mcp.callTool("weather", "ask", {
|
|
|
69
63
|
});
|
|
70
64
|
```
|
|
71
65
|
|
|
72
|
-
If the specialist is still working, `ask` returns `status: "running"
|
|
73
|
-
|
|
66
|
+
If the specialist is still working, `ask` returns `status: "running"`, and
|
|
67
|
+
`check` waits for the answer.
|
|
74
68
|
|
|
75
69
|
```ts
|
|
76
70
|
await ctx.host.mcp.callTool("weather", "check", {
|
|
@@ -79,14 +73,12 @@ await ctx.host.mcp.callTool("weather", "check", {
|
|
|
79
73
|
});
|
|
80
74
|
```
|
|
81
75
|
|
|
82
|
-
|
|
83
|
-
context.
|
|
84
|
-
|
|
85
|
-
## Call a deterministic peer tool
|
|
76
|
+
## Call a peer tool directly
|
|
86
77
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
and the
|
|
78
|
+
Use `call_tool` when you want to run a tool on a peer agent directly, without
|
|
79
|
+
starting a model turn on that agent. The peer needs at least one server tool,
|
|
80
|
+
and the call follows the
|
|
81
|
+
[direct-call rules](../reference/tools.md#call-a-tool-without-a-model-turn).
|
|
90
82
|
|
|
91
83
|
```ts
|
|
92
84
|
const result = await ctx.host.mcp.callTool("weather", "call_tool", {
|
|
@@ -108,12 +100,9 @@ return {
|
|
|
108
100
|
};
|
|
109
101
|
```
|
|
110
102
|
|
|
111
|
-
`call_tool` appears only when the peer declares at least one
|
|
112
|
-
deterministic server tool.
|
|
113
|
-
|
|
114
103
|
## Try it locally
|
|
115
104
|
|
|
116
|
-
|
|
105
|
+
To try it locally, run both projects in one process:
|
|
117
106
|
|
|
118
107
|
```bash
|
|
119
108
|
agent-sdk dev ./agents
|
|
@@ -121,14 +110,12 @@ agent-sdk chat --url http://127.0.0.1:3000/concierge \
|
|
|
121
110
|
--message "What's the weather in Paris right now?"
|
|
122
111
|
```
|
|
123
112
|
|
|
124
|
-
|
|
125
|
-
normally its directory name.
|
|
113
|
+
Locally, `agent` is the specialist's mount slug, usually its directory name.
|
|
126
114
|
|
|
127
115
|
## Peer or subagent?
|
|
128
116
|
|
|
129
|
-
Use a [subagent](../reference/subagents.md) to split one agent's job
|
|
130
|
-
|
|
131
|
-
complete agent you would also run, inspect, or expose on its own.
|
|
117
|
+
Use a [subagent](../reference/subagents.md) to split one agent's job into roles.
|
|
118
|
+
Use a peer when the specialist is a complete agent you also run on its own.
|
|
132
119
|
|
|
133
120
|
| | Subagent | Peer |
|
|
134
121
|
| --- | --- | --- |
|
|
@@ -136,32 +123,8 @@ complete agent you would also run, inspect, or expose on its own.
|
|
|
136
123
|
| Tools and connections | Inherits the parent project | Owns its project surface |
|
|
137
124
|
| Reachability | Parent only | Local co-hosted agents, or declared deployments on the same team |
|
|
138
125
|
|
|
139
|
-
## Affinity
|
|
140
|
-
|
|
141
|
-
Peer sessions appear in the specialist's playground. Only the calling
|
|
142
|
-
deployment can continue a hosted peer session.
|
|
143
|
-
|
|
144
|
-
## Practices
|
|
145
|
-
|
|
146
|
-
- Give each caller a one-way routing rule, and don't configure reciprocal
|
|
147
|
-
routes for the same request.
|
|
148
|
-
|
|
149
|
-
- Prefer `ask` when the specialist should reason. Use `call_tool` when
|
|
150
|
-
host code already knows which server tool to run.
|
|
151
|
-
|
|
152
126
|
## Other hosts
|
|
153
127
|
|
|
154
|
-
|
|
155
|
-
workers, protect
|
|
156
|
-
|
|
157
|
-
`--peers-url`; Cursor-managed hosting configures it automatically. See
|
|
158
|
-
[Deployment](../deployment.md) for the secure hosting setup.
|
|
159
|
-
|
|
160
|
-
## Related
|
|
161
|
-
|
|
162
|
-
- [MCP connections](../reference/connections.md#peer-mcp-connection):
|
|
163
|
-
peer configuration and connection naming
|
|
164
|
-
- [MCP endpoint](../reference/http-api.md#mcp-endpoint): exact `ask`,
|
|
165
|
-
`check`, and `call_tool` schemas
|
|
166
|
-
- [Subagents](../reference/subagents.md): specialists inside one agent
|
|
167
|
-
- [Deployment](../deployment.md): secure shared-host setup
|
|
128
|
+
To self-host, run the peers under one `agent-sdk` process, and if turns run on
|
|
129
|
+
remote workers, protect it with `--bearer-token` and set `--public-url` as
|
|
130
|
+
[Deployment](../deployment.md) describes.
|