@cursor/july 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/ab.d.ts +8 -95
- package/dist/ab.d.ts.map +1 -1
- package/dist/ab.js +9 -150
- package/dist/bin/agent-serve.js +14 -6
- package/dist/docs/404.html +2 -2
- package/dist/docs/ab.html +4 -4
- package/dist/docs/assets/{app.DF-Yx4OP.js → app.DMboGKUD.js} +1 -1
- package/dist/docs/assets/chunks/@localSearchIndexroot.O6eP7QZr.js +1 -0
- package/dist/docs/assets/chunks/{VPLocalSearchBox.DSeJSudL.js → VPLocalSearchBox.DkEqEnJU.js} +1 -1
- package/dist/docs/assets/chunks/{theme.DZVLTXOC.js → theme.ASugdMNm.js} +2 -2
- package/dist/docs/assets/deployment.md.BtfEsc9S.js +55 -0
- package/dist/docs/assets/deployment.md.BtfEsc9S.lean.js +1 -0
- package/dist/docs/assets/example-agents_approval-buddy.md.8R5phXb5.js +10 -0
- package/dist/docs/assets/example-agents_approval-buddy.md.8R5phXb5.lean.js +1 -0
- package/dist/docs/assets/example-agents_benny.md.B0gjhI-p.js +7 -0
- package/dist/docs/assets/example-agents_benny.md.B0gjhI-p.lean.js +1 -0
- package/dist/docs/assets/example-agents_bugbot.md.DelIdhxB.js +11 -0
- package/dist/docs/assets/example-agents_bugbot.md.DelIdhxB.lean.js +1 -0
- package/dist/docs/assets/example-agents_codebase-wiki.md.DC6sgwn0.js +8 -0
- package/dist/docs/assets/example-agents_codebase-wiki.md.DC6sgwn0.lean.js +1 -0
- package/dist/docs/assets/example-agents_codeowners-review.md.Ku_tG2RY.js +8 -0
- package/dist/docs/assets/example-agents_codeowners-review.md.Ku_tG2RY.lean.js +1 -0
- package/dist/docs/assets/example-agents_concierge.md.4rQTSMXt.js +23 -0
- package/dist/docs/assets/example-agents_concierge.md.4rQTSMXt.lean.js +1 -0
- package/dist/docs/assets/example-agents_fsd.md.CzgUrDfi.js +15 -0
- package/dist/docs/assets/example-agents_fsd.md.CzgUrDfi.lean.js +1 -0
- package/dist/docs/assets/example-agents_index.md.CRqJlnIf.js +2 -0
- package/dist/docs/assets/example-agents_index.md.CRqJlnIf.lean.js +1 -0
- package/dist/docs/assets/example-agents_knowledge-base.md.BPJiVueF.js +11 -0
- package/dist/docs/assets/example-agents_knowledge-base.md.BPJiVueF.lean.js +1 -0
- package/dist/docs/assets/example-agents_security-reviewer.md.D2rtwDTO.js +19 -0
- package/dist/docs/assets/example-agents_security-reviewer.md.D2rtwDTO.lean.js +1 -0
- package/dist/docs/assets/example-agents_slack-agent.md.buLbgvBf.js +5 -0
- package/dist/docs/assets/example-agents_slack-agent.md.buLbgvBf.lean.js +1 -0
- package/dist/docs/assets/example-agents_weather-agent.md.C9Qv-W0o.js +24 -0
- package/dist/docs/assets/example-agents_weather-agent.md.C9Qv-W0o.lean.js +1 -0
- package/dist/docs/assets/index.md.COiu-1jL.js +20 -0
- package/dist/docs/assets/{index.md.Cylk70gg.lean.js → index.md.COiu-1jL.lean.js} +1 -1
- package/dist/docs/assets/reference_cli.md.D189RBCH.js +60 -0
- package/dist/docs/assets/reference_cli.md.D189RBCH.lean.js +1 -0
- package/dist/docs/building-with-agents.html +4 -4
- package/dist/docs/concepts.html +4 -4
- package/dist/docs/deployment.html +58 -17
- package/dist/docs/evals.html +4 -4
- package/dist/docs/example-agents/approval-buddy.html +34 -0
- package/dist/docs/example-agents/benny.html +31 -0
- package/dist/docs/example-agents/bugbot.html +35 -0
- package/dist/docs/example-agents/codebase-wiki.html +32 -0
- package/dist/docs/example-agents/codeowners-review.html +32 -0
- package/dist/docs/example-agents/concierge.html +47 -0
- package/dist/docs/example-agents/fsd.html +39 -0
- package/dist/docs/example-agents/index.html +26 -0
- package/dist/docs/example-agents/knowledge-base.html +35 -0
- package/dist/docs/example-agents/security-reviewer.html +43 -0
- package/dist/docs/example-agents/slack-agent.html +29 -0
- package/dist/docs/example-agents/weather-agent.html +48 -0
- package/dist/docs/guides/agent-to-agent.html +4 -4
- package/dist/docs/guides/cloud-runtime.html +5 -5
- package/dist/docs/guides/github.html +4 -4
- package/dist/docs/guides/human-in-the-loop.html +4 -4
- package/dist/docs/guides/slack.html +4 -4
- package/dist/docs/guides/webhooks.html +4 -4
- package/dist/docs/hashmap.json +1 -1
- package/dist/docs/hillclimbing.html +4 -4
- package/dist/docs/index.html +7 -7
- package/dist/docs/quickstart.html +4 -4
- package/dist/docs/reference/agent-config.html +4 -4
- package/dist/docs/reference/channels.html +4 -4
- package/dist/docs/reference/cli.html +52 -30
- package/dist/docs/reference/connections.html +4 -4
- package/dist/docs/reference/hooks.html +4 -4
- package/dist/docs/reference/http-api.html +4 -4
- package/dist/docs/reference/instructions.html +4 -4
- package/dist/docs/reference/playground.html +4 -4
- package/dist/docs/reference/project-layout.html +4 -4
- package/dist/docs/reference/schedules.html +4 -4
- package/dist/docs/reference/sessions.html +4 -4
- package/dist/docs/reference/skills.html +4 -4
- package/dist/docs/reference/subagents.html +4 -4
- package/dist/docs/reference/tools.html +4 -4
- package/dist/docs/scaffolding-agents.html +4 -4
- package/dist/docs/storage.html +4 -4
- package/dist/docs/troubleshooting.html +4 -4
- package/dist/evals.d.ts +4 -61
- package/dist/evals.d.ts.map +1 -1
- package/dist/evals.js +3 -66
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/internal/ab-collector.d.ts +7 -5
- package/dist/internal/ab-collector.d.ts.map +1 -1
- package/dist/internal/ab-collector.js +3 -14
- package/dist/internal/ab-snapshot.d.ts +2 -4
- package/dist/internal/ab-snapshot.d.ts.map +1 -1
- package/dist/internal/cli-ax.d.ts +18 -3
- package/dist/internal/cli-ax.d.ts.map +1 -1
- package/dist/internal/cli-ax.js +80 -42
- package/dist/internal/cli-deploy.d.ts.map +1 -1
- package/dist/internal/cli-deploy.js +5 -2
- package/dist/internal/cursor-event-relay.d.ts +4 -1
- package/dist/internal/cursor-event-relay.d.ts.map +1 -1
- package/dist/internal/cursor-event-relay.js +6 -0
- package/dist/internal/cursor-relay-core.d.ts +36 -1
- package/dist/internal/cursor-relay-core.d.ts.map +1 -1
- package/dist/internal/cursor-relay-core.js +89 -8
- package/dist/internal/cursor-slack-relay.d.ts +2 -2
- package/dist/internal/cursor-slack-relay.d.ts.map +1 -1
- package/dist/internal/cursor-slack-relay.js +4 -0
- package/dist/internal/discovery.js +3 -3
- package/dist/internal/eval-run-store.d.ts +33 -30
- package/dist/internal/eval-run-store.d.ts.map +1 -1
- package/dist/internal/eval-run-store.js +84 -96
- package/dist/internal/init-project.d.ts.map +1 -1
- package/dist/internal/init-project.js +3 -2
- package/dist/internal/playground-proxy.d.ts +5 -5
- package/dist/internal/playground-proxy.js +3 -3
- package/dist/internal/resolve-prod-target.d.ts +30 -0
- package/dist/internal/resolve-prod-target.d.ts.map +1 -1
- package/dist/internal/resolve-prod-target.js +74 -2
- package/dist/internal/server.d.ts.map +1 -1
- package/dist/internal/server.js +5 -3
- package/dist/internal/session-engine.d.ts +1 -2
- package/dist/internal/session-engine.d.ts.map +1 -1
- package/dist/internal/session-engine.js +14 -31
- package/dist/internal/storage-coordinator.d.ts +16 -15
- package/dist/internal/storage-coordinator.d.ts.map +1 -1
- package/dist/internal/storage-coordinator.js +73 -80
- package/dist/playground/assets/cursor-icons-16-CQ50JpfO.woff2 +0 -0
- package/dist/playground/assets/index-7GJIUFQj.css +1 -0
- package/dist/playground/assets/index-mux9aj5P.js +319 -0
- package/dist/playground/index.html +2 -2
- package/dist/storage.d.ts +51 -10
- package/dist/storage.d.ts.map +1 -1
- package/dist/storage.js +27 -10
- package/docs/README.md +34 -5
- package/docs/deployment.md +352 -149
- package/docs/example-agents/approval-buddy.md +270 -0
- package/docs/example-agents/benny.md +186 -0
- package/docs/example-agents/bugbot.md +231 -0
- package/docs/example-agents/codebase-wiki.md +174 -0
- package/docs/example-agents/codeowners-review.md +195 -0
- package/docs/example-agents/concierge.md +205 -0
- package/docs/example-agents/fsd.md +330 -0
- package/docs/example-agents/index.md +102 -0
- package/docs/example-agents/knowledge-base.md +171 -0
- package/docs/example-agents/security-reviewer.md +296 -0
- package/docs/example-agents/slack-agent.md +146 -0
- package/docs/example-agents/weather-agent.md +302 -0
- package/docs/reference/cli.md +546 -147
- package/package.json +1 -1
- package/src/ab.ts +9 -261
- package/src/bin/agent-serve.ts +14 -6
- package/src/evals.ts +4 -118
- package/src/index.ts +2 -0
- package/src/internal/ab-collector.ts +12 -22
- package/src/internal/ab-snapshot.ts +2 -4
- package/src/internal/cli-ax.ts +101 -45
- package/src/internal/cli-deploy.ts +5 -2
- package/src/internal/cursor-event-relay.ts +10 -1
- package/src/internal/cursor-relay-core.ts +115 -9
- package/src/internal/cursor-slack-relay.ts +6 -2
- package/src/internal/discovery.ts +2 -2
- package/src/internal/eval-run-store.ts +86 -100
- package/src/internal/init-project.ts +3 -2
- package/src/internal/playground-proxy.ts +5 -5
- package/src/internal/resolve-prod-target.ts +101 -3
- package/src/internal/server.ts +5 -3
- package/src/internal/session-engine.ts +9 -29
- package/src/internal/storage-coordinator.ts +109 -101
- package/src/storage.ts +79 -14
- package/dist/docs/assets/chunks/@localSearchIndexroot.0XrheBv2.js +0 -1
- package/dist/docs/assets/deployment.md.DTKwE15Z.js +0 -14
- package/dist/docs/assets/deployment.md.DTKwE15Z.lean.js +0 -1
- package/dist/docs/assets/index.md.Cylk70gg.js +0 -20
- package/dist/docs/assets/reference_cli.md.Bv6pOxcF.js +0 -38
- package/dist/docs/assets/reference_cli.md.Bv6pOxcF.lean.js +0 -1
- package/dist/internal/json-dir-store.d.ts +0 -32
- package/dist/internal/json-dir-store.d.ts.map +0 -1
- package/dist/internal/json-dir-store.js +0 -100
- package/dist/playground/assets/cursor-icons-outline-BxTT_FVJ.woff2 +0 -0
- package/dist/playground/assets/index-Ct_Usdm8.css +0 -1
- package/dist/playground/assets/index-Dsyb-Vaj.js +0 -319
- package/src/internal/json-dir-store.ts +0 -109
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Keep PR approval policy deterministic with Approval Buddy"
|
|
3
|
+
description: "Separate code-owned eligibility from model-owned review, then connect GitHub, Slack, subagents, durable storage, and evals."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Keep PR approval policy deterministic with Approval Buddy
|
|
7
|
+
|
|
8
|
+
Approval Buddy approves eligible pull requests from a fixed roster and
|
|
9
|
+
declines every other request. GitHub still blocks self-approval when the stamp
|
|
10
|
+
identity authored the PR. Code decides eligibility. The model prepares
|
|
11
|
+
evidence, runs two specialist reviews, and passes their findings to the
|
|
12
|
+
approval tool without changing the policy decision.
|
|
13
|
+
|
|
14
|
+
Use this example when an agent can make a judgment inside a workflow, but
|
|
15
|
+
authorization and the final side effect must stay in deterministic code.
|
|
16
|
+
|
|
17
|
+
[Browse the Approval Buddy source.](../../examples/approval-buddy/)
|
|
18
|
+
|
|
19
|
+
## Keep approval policy in code
|
|
20
|
+
|
|
21
|
+
Approval Buddy draws three hard boundaries:
|
|
22
|
+
|
|
23
|
+
- `prepare_review` and `approve_pr` re-read the live PR and apply the same
|
|
24
|
+
eligibility rules.
|
|
25
|
+
- Two subagents inspect prepared evidence, but their findings never grant or
|
|
26
|
+
block approval.
|
|
27
|
+
- Only `approve_pr` posts the GitHub review.
|
|
28
|
+
|
|
29
|
+
A spoofed webhook, Slack message, or model claim can't add someone to the
|
|
30
|
+
buddy roster. The mutating tool checks the source of truth immediately before it
|
|
31
|
+
acts.
|
|
32
|
+
|
|
33
|
+
## Follow the intended stamp flow
|
|
34
|
+
|
|
35
|
+
The root instructions ask the model to run this sequence for a qualifying PR:
|
|
36
|
+
|
|
37
|
+
1. A non-draft `pull_request` event arrives with action `opened`, `reopened`,
|
|
38
|
+
or `ready_for_review`.
|
|
39
|
+
2. The GitHub channel checks its repository allowlist and starts a session.
|
|
40
|
+
3. `turn.started` posts a pending commit status.
|
|
41
|
+
4. The model calls `prepare_review`.
|
|
42
|
+
5. Host code fetches the live PR. It checks the author, open state, merged
|
|
43
|
+
state, and draft state.
|
|
44
|
+
6. A qualifying PR gets `pr/MANIFEST.md`, `pr/meta.json`, and
|
|
45
|
+
`pr/diff.patch` in the session workspace. Diffs above 2,000,000
|
|
46
|
+
characters are truncated and marked in metadata.
|
|
47
|
+
7. The model calls both review subagents through the built-in `task` tool.
|
|
48
|
+
8. It concatenates their contracted replies and calls `approve_pr`.
|
|
49
|
+
9. `approve_pr` re-runs eligibility, posts an `APPROVE` review, and returns
|
|
50
|
+
the outcome.
|
|
51
|
+
10. The channel posts a final commit status. A self-approval block also gets
|
|
52
|
+
a short timeline comment because no approval review can appear.
|
|
53
|
+
|
|
54
|
+
Ineligible PRs skip evidence and subagents. The model still calls
|
|
55
|
+
`approve_pr` so the deterministic tool returns the formal decline reason.
|
|
56
|
+
|
|
57
|
+
Steps 4 through 9 are prompt-driven. The channel doesn't enforce tool order
|
|
58
|
+
or prove both subagents ran, and `approve_pr` accepts missing findings. A
|
|
59
|
+
failed turn clears the pending status with a green non-blocking result without
|
|
60
|
+
approving the PR.
|
|
61
|
+
|
|
62
|
+
## Map the framework features
|
|
63
|
+
|
|
64
|
+
| Capability | Source | Role |
|
|
65
|
+
| --- | --- | --- |
|
|
66
|
+
| Root agent and policy prompt | [`agent/agent.ts`](../../examples/approval-buddy/agent/agent.ts), [`agent/instructions.md`](../../examples/approval-buddy/agent/instructions.md) | Configure the local agent and describe orchestration order. |
|
|
67
|
+
| GitHub channel | [`agent/channels/github.ts`](../../examples/approval-buddy/agent/channels/github.ts) | Filter wakes, lease GitHub access, and publish status events. |
|
|
68
|
+
| Slack channel | [`agent/channels/slack.ts`](../../examples/approval-buddy/agent/channels/slack.ts) | Accept approval-bot stamp and qualification requests. |
|
|
69
|
+
| Server tools | [`agent/tools/`](../../examples/approval-buddy/agent/tools/) | Prepare evidence, approve, list buddies, and search GIFs. |
|
|
70
|
+
| Deterministic policy | [`agent/lib/approve.ts`](../../examples/approval-buddy/agent/lib/approve.ts), [`agent/lib/buddies.ts`](../../examples/approval-buddy/agent/lib/buddies.ts) | Own the roster and live eligibility checks. |
|
|
71
|
+
| Review subagents | [`agent/subagents/`](../../examples/approval-buddy/agent/subagents/) | Run deep audit and code-quality passes over the same evidence. |
|
|
72
|
+
| Storage | [`agent/storage.ts`](../../examples/approval-buddy/agent/storage.ts) | Persist sessions and events through a custom durable adapter. |
|
|
73
|
+
| Evals and unit tests | [`evals/`](../../examples/approval-buddy/evals/), [`agent/lib/`](../../examples/approval-buddy/agent/lib/) | Protect routing, output contracts, policy, and GitHub behavior. |
|
|
74
|
+
|
|
75
|
+
There are no authored skills, MCP connections, schedules, reminders, hooks,
|
|
76
|
+
A/B experiments, sandbox seeds, or tool approvals.
|
|
77
|
+
|
|
78
|
+
## Prepare credentials
|
|
79
|
+
|
|
80
|
+
You need:
|
|
81
|
+
|
|
82
|
+
- Node 22.13 or newer.
|
|
83
|
+
- An agent-runtime credential.
|
|
84
|
+
- GitHub access to read PRs, post reviews, create commit statuses, and
|
|
85
|
+
post the self-approval visibility comment.
|
|
86
|
+
|
|
87
|
+
Optional GIF selection uses:
|
|
88
|
+
|
|
89
|
+
- `GIPHY_API_KEY` or `APPROVAL_BUDDY_GIPHY_API_KEY`,
|
|
90
|
+
- `APPROVAL_BUDDY_STAMP_GIF`, or
|
|
91
|
+
- severity-specific `APPROVAL_BUDDY_STAMP_GIF_<LEVEL>` variables.
|
|
92
|
+
|
|
93
|
+
If you enable Giphy in a hosted copy, declare its secret and
|
|
94
|
+
`api.giphy.com` egress.
|
|
95
|
+
|
|
96
|
+
## Validate without approving a PR
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
agentkit validate --dir examples/approval-buddy
|
|
100
|
+
agentkit info --dir examples/approval-buddy --json
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
List the deterministic roster:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
agentkit call list_buddies \
|
|
107
|
+
--dir examples/approval-buddy \
|
|
108
|
+
--input '{}'
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Set a known merged PR, then run the read-only precheck:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
MERGED_PR_URL=https://github.com/your-org/your-repo/pull/123
|
|
115
|
+
agentkit call prepare_review \
|
|
116
|
+
--dir examples/approval-buddy \
|
|
117
|
+
--input "{\"prUrl\":\"$MERGED_PR_URL\"}"
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The result should decline because the PR is no longer open. `prepare_review`
|
|
121
|
+
never posts an approval.
|
|
122
|
+
|
|
123
|
+
> [!CAUTION]
|
|
124
|
+
> Don't use `agentkit call approve_pr` as a smoke test. The tool has no
|
|
125
|
+
> `needsApproval` gate and posts a real GitHub review when the PR qualifies.
|
|
126
|
+
|
|
127
|
+
## See why preparation is separate
|
|
128
|
+
|
|
129
|
+
`prepare_review` is read-only. It checks policy before fetching a large diff,
|
|
130
|
+
so declined requests don't spend review-agent work.
|
|
131
|
+
|
|
132
|
+
Direct calls return the evidence file map because their scratch workspace is
|
|
133
|
+
deleted after the call. In-session calls write the tree to
|
|
134
|
+
`ctx.workspaceDir`, where both subagents can read it.
|
|
135
|
+
|
|
136
|
+
`approve_pr` repeats the live check instead of trusting preparation. A PR can
|
|
137
|
+
close, merge, become a draft, or change author-related context between the two
|
|
138
|
+
steps. Revalidation keeps the final write bound to current state.
|
|
139
|
+
|
|
140
|
+
This is a reusable two-tool pattern:
|
|
141
|
+
|
|
142
|
+
- a read-only tool prepares and explains the decision,
|
|
143
|
+
- a mutating tool repeats policy at the side-effect boundary.
|
|
144
|
+
|
|
145
|
+
## Fan out two review contracts
|
|
146
|
+
|
|
147
|
+
The two discovered subagents have different contracts:
|
|
148
|
+
|
|
149
|
+
- The security reviewer reports bugs, breaking changes, and security findings
|
|
150
|
+
with `High`, `Medium`, or `Low` tags.
|
|
151
|
+
- The code-quality reviewer reports maintainability and structure concerns
|
|
152
|
+
with `Blocker`, `Major`, or `Minor` tags.
|
|
153
|
+
|
|
154
|
+
The parent calls both through the harness `task` tool. They inherit the root
|
|
155
|
+
agent's execution surface and read the same `pr/` workspace. The prompt asks
|
|
156
|
+
the parent not to rewrite either reply. The review body trims the combined
|
|
157
|
+
text and caps it at 16,000 characters.
|
|
158
|
+
|
|
159
|
+
Findings are informational. A high-severity finding doesn't veto the stamp.
|
|
160
|
+
That policy is explicit in the root instructions and approval code.
|
|
161
|
+
|
|
162
|
+
## Trace GitHub channel behavior
|
|
163
|
+
|
|
164
|
+
The channel uses `githubChannel` with:
|
|
165
|
+
|
|
166
|
+
- a configured repository allowlist on the account-linked GitHub transport,
|
|
167
|
+
- a second optional `APPROVAL_BUDDY_REPOS` wake filter,
|
|
168
|
+
- `deliverReplies: false`,
|
|
169
|
+
- progress reactions disabled, and
|
|
170
|
+
- event handlers for turn start, `approve_pr` results, and failed turns.
|
|
171
|
+
|
|
172
|
+
The source requests `contents-write`, even though the documented workflow
|
|
173
|
+
posts reviews, statuses, and comments. When adapting the example, start with
|
|
174
|
+
`pr-write` and opt up only if a tool must push code.
|
|
175
|
+
|
|
176
|
+
Every terminal status is green by design. Declines and crashed turns are
|
|
177
|
+
informational, not merge-blocking. This is a product decision in the example,
|
|
178
|
+
not an agentkit default.
|
|
179
|
+
|
|
180
|
+
A successful turn that never calls `approve_pr` leaves the pending status in
|
|
181
|
+
place. The channel clears it on `approve_pr` results and `turn.failed`, but
|
|
182
|
+
has no `turn.completed` fallback.
|
|
183
|
+
|
|
184
|
+
`github replay` reaches the same channel and can post a real approval, status,
|
|
185
|
+
or comment. Use replay only against a repository and PR created for this
|
|
186
|
+
test.
|
|
187
|
+
|
|
188
|
+
## Use Slack for explicit requests
|
|
189
|
+
|
|
190
|
+
Start the dev server:
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
agentkit dev examples/approval-buddy
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Then ask through the signed-in account-linked Slack connection:
|
|
197
|
+
|
|
198
|
+
> Would this PR qualify for a stamp?
|
|
199
|
+
|
|
200
|
+
The instructions route qualification questions to `prepare_review` only. A
|
|
201
|
+
stamp request runs the complete flow and may approve the PR.
|
|
202
|
+
|
|
203
|
+
This channel uses the account-linked transport instead of a dedicated Socket
|
|
204
|
+
Mode app.
|
|
205
|
+
|
|
206
|
+
## See how durable storage fits
|
|
207
|
+
|
|
208
|
+
`defineStorage` replaces the default local session store with a shared,
|
|
209
|
+
durable key-value adapter. Approval Buddy chooses:
|
|
210
|
+
|
|
211
|
+
- a 15-second write debounce,
|
|
212
|
+
- startup restoration for up to 200 sessions, and
|
|
213
|
+
- a 14-day restore window.
|
|
214
|
+
|
|
215
|
+
That policy fits long-lived Slack threads and a small webhook fleet. The
|
|
216
|
+
security reviewer uses the same adapter with lazy restore, which fits its
|
|
217
|
+
shorter sessions.
|
|
218
|
+
|
|
219
|
+
## Run the regression suite
|
|
220
|
+
|
|
221
|
+
List the four eval cases:
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
agentkit eval --dir examples/approval-buddy --list
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
The suite covers:
|
|
228
|
+
|
|
229
|
+
- buddy-list routing,
|
|
230
|
+
- declining a merged PR,
|
|
231
|
+
- using only `prepare_review` for a qualification question, and
|
|
232
|
+
- the combined findings headings and severity format over seeded evidence.
|
|
233
|
+
|
|
234
|
+
Run the safe qualification case:
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
agentkit eval \
|
|
238
|
+
--dir examples/approval-buddy \
|
|
239
|
+
qualify/merged-pr-question \
|
|
240
|
+
--json
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
The qualification case reads a live merged PR. The seeded format case shown
|
|
244
|
+
by `--list` uses a planted auth-bypass diff
|
|
245
|
+
and checks for a `task` call, both headings, and severity tags. It doesn't
|
|
246
|
+
prove both named subagents ran or whether their output reached `approve_pr`.
|
|
247
|
+
Unit tests under `agent/lib/` cover policy, self-approval handling, evidence
|
|
248
|
+
limits, status mapping, GIF selection, and severity parsing.
|
|
249
|
+
|
|
250
|
+
## Reuse the policy boundary
|
|
251
|
+
|
|
252
|
+
Keep these properties when you replace the buddy policy:
|
|
253
|
+
|
|
254
|
+
1. Put authorization in typed code.
|
|
255
|
+
2. Fetch the source of truth inside both prepare and mutate steps.
|
|
256
|
+
3. Give the model evidence only after the request qualifies.
|
|
257
|
+
4. Treat specialist findings as data, not authority.
|
|
258
|
+
5. Keep the core domain mutation in one named tool. Treat channel status and
|
|
259
|
+
visibility writes as separate, audited effects.
|
|
260
|
+
6. Add a human approval gate if your policy still needs operator consent.
|
|
261
|
+
7. Test read-only routing separately from mutation.
|
|
262
|
+
|
|
263
|
+
## Where to go next
|
|
264
|
+
|
|
265
|
+
- [GitHub](../guides/github.md)
|
|
266
|
+
- [Tools](../reference/tools.md)
|
|
267
|
+
- [Subagents](../reference/subagents.md)
|
|
268
|
+
- [Storage](../storage.md)
|
|
269
|
+
- [Slack](../guides/slack.md)
|
|
270
|
+
- [Evals](../evals.md)
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Route Slack work through repository playbooks"
|
|
3
|
+
description: "Combine account-linked chat, allowlisted Socket Mode channel watching, inherited repository skills, and a custom local workspace."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Route Slack work through repository playbooks
|
|
7
|
+
|
|
8
|
+
This agent is a Slack teammate for a product team. Mentions and direct
|
|
9
|
+
messages reach it through an account-linked transport. New top-level posts in
|
|
10
|
+
an allowlisted issue channel reach it through a dedicated Slack app, even
|
|
11
|
+
without a mention. The agent then selects a repository playbook for triage,
|
|
12
|
+
reproduction, fixes, reviews, on-call work, or design critique.
|
|
13
|
+
|
|
14
|
+
Use this example when Slack is the intake surface and your durable procedures
|
|
15
|
+
already live as repository skills.
|
|
16
|
+
|
|
17
|
+
[Browse the current playbook-router source.](../../examples/benny/)
|
|
18
|
+
|
|
19
|
+
## Combine two Slack transports with repo skills
|
|
20
|
+
|
|
21
|
+
The playbook router uniquely combines three decisions:
|
|
22
|
+
|
|
23
|
+
- Two Slack transports serve different engagement modes.
|
|
24
|
+
- `local.cwd` keeps session workspaces inside the monorepo.
|
|
25
|
+
- Instructions route work to inherited repository playbooks
|
|
26
|
+
instead of authored `agent/skills/`.
|
|
27
|
+
|
|
28
|
+
The result is a thin agent project over a mature procedure library.
|
|
29
|
+
|
|
30
|
+
## Follow an issue report
|
|
31
|
+
|
|
32
|
+
1. A teammate creates a top-level post in the allowlisted issue channel.
|
|
33
|
+
2. The dedicated Socket Mode channel accepts the allowlisted channel.
|
|
34
|
+
3. A 15-second debounce lets edits settle. Deleting the post during that
|
|
35
|
+
window cancels the dispatch.
|
|
36
|
+
4. Agentkit creates a thread-scoped session and sends the report to the
|
|
37
|
+
playbook router.
|
|
38
|
+
5. The instructions select the matching triage playbook.
|
|
39
|
+
6. The harness finds the repository root, opens the inherited playbook, and
|
|
40
|
+
follows its procedure.
|
|
41
|
+
7. The agent posts only in the source thread and reports the evidence it
|
|
42
|
+
gathered.
|
|
43
|
+
|
|
44
|
+
Mentions and direct messages follow the same agent instructions. They don't
|
|
45
|
+
need the watched-channel path.
|
|
46
|
+
|
|
47
|
+
## Map the playbook router files
|
|
48
|
+
|
|
49
|
+
| File | Purpose |
|
|
50
|
+
| --- | --- |
|
|
51
|
+
| [`agent/agent.ts`](../../examples/benny/agent/agent.ts) | Names the agent, selects its model, and keeps the harness under `.agent-serve/harness`. |
|
|
52
|
+
| [`agent/instructions.md`](../../examples/benny/agent/instructions.md) | Defines engagement rules, evidence policy, and the playbook routing map. |
|
|
53
|
+
| [`agent/channels/slack.ts`](../../examples/benny/agent/channels/slack.ts) | Handles account-linked mentions and direct messages. |
|
|
54
|
+
| [`agent/channels/slack-app.ts`](../../examples/benny/agent/channels/slack-app.ts) | Runs the dedicated app and watches one allowlisted channel. |
|
|
55
|
+
| [`evals/smoke.eval.ts`](../../examples/benny/evals/smoke.eval.ts) | Checks the agent identity and expected triage route. |
|
|
56
|
+
|
|
57
|
+
The playbook router authors no tools, MCP connections, subagents, schedules, hooks, A/B
|
|
58
|
+
experiments, or sandbox seeds.
|
|
59
|
+
|
|
60
|
+
## See why `local.cwd` matters
|
|
61
|
+
|
|
62
|
+
Agentkit normally keeps an ephemeral `run` or `eval` workspace outside a
|
|
63
|
+
large monorepo. This prevents ancestor instruction and repository-rule files
|
|
64
|
+
from leaking into an unrelated agent.
|
|
65
|
+
|
|
66
|
+
The playbook router needs the opposite. Its procedures live at the repository
|
|
67
|
+
root, so
|
|
68
|
+
`agent.ts` sets:
|
|
69
|
+
|
|
70
|
+
```ts
|
|
71
|
+
local: {
|
|
72
|
+
cwd: ".agent-serve/harness",
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Each harness workspace lands under
|
|
77
|
+
`examples/benny/.agent-serve/harness/<sessionId>`. Walking up the directory
|
|
78
|
+
tree reaches the host repository and its inherited playbook directory.
|
|
79
|
+
|
|
80
|
+
Those playbooks are inherited context. `agentkit info` reports zero authored
|
|
81
|
+
skills for the agent. Copying this project into another repository removes
|
|
82
|
+
its main procedures unless you copy or replace the skill library too.
|
|
83
|
+
|
|
84
|
+
## Connect both Slack paths
|
|
85
|
+
|
|
86
|
+
The account-linked path needs an agent-runtime login and a connected Slack
|
|
87
|
+
account:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
agentkit login
|
|
91
|
+
agentkit whoami
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
It routes explicit mentions without a dedicated Slack token on the host.
|
|
95
|
+
|
|
96
|
+
For the watched-channel path, configure a dedicated Socket Mode app with:
|
|
97
|
+
|
|
98
|
+
- subscribe to `message.channels` and `message.groups`,
|
|
99
|
+
- have an App-Level Token with `connections:write`, and
|
|
100
|
+
- be a member of the watched channel.
|
|
101
|
+
|
|
102
|
+
Run `agentkit slack setup` for the guided app workflow. Generate the project
|
|
103
|
+
manifest with `--channel-posts` when you create a new copy, then validate the
|
|
104
|
+
configured channel prefix with `agentkit slack doctor`.
|
|
105
|
+
|
|
106
|
+
Missing dedicated-app tokens leave that channel idle. They don't stop the
|
|
107
|
+
account-linked channel.
|
|
108
|
+
|
|
109
|
+
## Validate and start the server
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
agentkit validate --dir examples/benny
|
|
113
|
+
agentkit info --dir examples/benny --json
|
|
114
|
+
agentkit dev examples/benny
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
The info output should show two Slack channels and no authored skill. That
|
|
118
|
+
combination confirms the example is using inherited playbooks.
|
|
119
|
+
|
|
120
|
+
## Exercise each engagement mode
|
|
121
|
+
|
|
122
|
+
Test the explicit account-linked path by asking:
|
|
123
|
+
|
|
124
|
+
> Which playbook would you use to triage a product UI bug?
|
|
125
|
+
|
|
126
|
+
Test the dedicated app:
|
|
127
|
+
|
|
128
|
+
1. Create a top-level post in the allowlisted issue channel.
|
|
129
|
+
2. Don't mention the bot.
|
|
130
|
+
3. Wait for the debounce window.
|
|
131
|
+
4. Confirm the agent replies in the post's thread.
|
|
132
|
+
|
|
133
|
+
Thread replies don't trigger the proactive watch. Mentions still use Slack's
|
|
134
|
+
normal mention path. Bot-authored posts are ignored to prevent loops.
|
|
135
|
+
|
|
136
|
+
The channel uses the default handler after filtering. It doesn't apply a
|
|
137
|
+
second code-level classifier, so every accepted top-level post spends a model
|
|
138
|
+
turn and reaches the prompt.
|
|
139
|
+
|
|
140
|
+
## Inspect thread continuity
|
|
141
|
+
|
|
142
|
+
Agentkit keys Slack sessions by channel and thread timestamp. A follow-up in
|
|
143
|
+
the same thread resumes the conversation and workspace. A new top-level issue
|
|
144
|
+
gets a new session.
|
|
145
|
+
|
|
146
|
+
This lets a playbook gather evidence over several turns without mixing two
|
|
147
|
+
reports. The playground shows both the account-linked and dedicated-app
|
|
148
|
+
sessions while the dev server runs.
|
|
149
|
+
|
|
150
|
+
## Run the smoke eval
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
agentkit eval --dir examples/benny --list
|
|
154
|
+
agentkit eval --dir examples/benny smoke --json
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
The case asks for the agent identity and the playbook used for issue triage.
|
|
158
|
+
It checks the configured identity and route label.
|
|
159
|
+
|
|
160
|
+
This is a lexical smoke test. It doesn't prove Slack delivery, skill
|
|
161
|
+
selection, skill loading, procedure execution, or thread-only behavior. Add
|
|
162
|
+
fixture-backed evals around the playbooks when you reuse this design.
|
|
163
|
+
|
|
164
|
+
## Build a playbook-routed teammate
|
|
165
|
+
|
|
166
|
+
Use this structure when your organization already has tested skills:
|
|
167
|
+
|
|
168
|
+
1. Put the playbooks under a stable repository path.
|
|
169
|
+
2. Set `local.cwd` so harness workspaces can inherit that path.
|
|
170
|
+
3. Write a short routing table in `instructions.md`.
|
|
171
|
+
4. Use account-linked Slack for explicit requests.
|
|
172
|
+
5. Add a dedicated app only for allowlisted proactive intake.
|
|
173
|
+
6. Keep the channel allowlist narrow and debounce edited posts.
|
|
174
|
+
7. Add an eval for every important request-to-playbook route.
|
|
175
|
+
|
|
176
|
+
If the procedures should ship with the agent, put them under
|
|
177
|
+
`agent/skills/` instead. Authored skills appear in the manifest and travel
|
|
178
|
+
with the project.
|
|
179
|
+
|
|
180
|
+
## Where to go next
|
|
181
|
+
|
|
182
|
+
- [Slack](../guides/slack.md)
|
|
183
|
+
- [Agent config](../reference/agent-config.md)
|
|
184
|
+
- [Skills](../reference/skills.md)
|
|
185
|
+
- [Sessions and streaming](../reference/sessions.md)
|
|
186
|
+
- [Evals](../evals.md)
|
|
@@ -0,0 +1,231 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Review prepared pull-request evidence"
|
|
3
|
+
description: "Fetch a PR on the host, seed a trimmed diff-first workspace, and run a GitHub-read-only review through HTTP, Slack, or an eval."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Review prepared pull-request evidence
|
|
7
|
+
|
|
8
|
+
This GitHub-read-only reviewer uses host code to fetch the PR
|
|
9
|
+
with `gh` and `git`, builds a trimmed `pr/` evidence tree, then hands that tree
|
|
10
|
+
to the model. The model reads the diff, loads a review skill, and returns at
|
|
11
|
+
most three high-confidence findings.
|
|
12
|
+
|
|
13
|
+
Use this example when the host should control evidence collection and the
|
|
14
|
+
model shouldn't browse or mutate the source repository.
|
|
15
|
+
|
|
16
|
+
[Browse the current reviewer source.](../../examples/bugbot/)
|
|
17
|
+
|
|
18
|
+
## Separate evidence preparation from review
|
|
19
|
+
|
|
20
|
+
The reviewer separates preparation from judgment:
|
|
21
|
+
|
|
22
|
+
- Host code owns GitHub and Git access.
|
|
23
|
+
- A server tool turns untrusted PR input into bounded workspace files.
|
|
24
|
+
- A custom channel seeds those files before the model starts.
|
|
25
|
+
- An on-demand skill defines the review procedure and output contract.
|
|
26
|
+
- The model returns chat text. No path posts a GitHub review.
|
|
27
|
+
|
|
28
|
+
This architecture gives the model a purpose-built evidence package instead of
|
|
29
|
+
a checkout.
|
|
30
|
+
|
|
31
|
+
## Follow a review
|
|
32
|
+
|
|
33
|
+
The custom HTTP path runs this sequence:
|
|
34
|
+
|
|
35
|
+
1. `POST /v1/channels/review/` receives a PR reference.
|
|
36
|
+
2. The handler calls `prepare_pr` without a model turn.
|
|
37
|
+
3. Host code reads PR metadata and the unified diff.
|
|
38
|
+
4. It reuses a matching checkout, force-fetching the PR ref there when the
|
|
39
|
+
commit is missing. Without a matching checkout, it uses a temporary bare
|
|
40
|
+
cache.
|
|
41
|
+
5. It creates `pr/MANIFEST.md`, `pr/meta.json`, `pr/diff.patch`, and selected
|
|
42
|
+
small files and rules.
|
|
43
|
+
6. `send({ workspaceFiles })` creates the model session with that evidence.
|
|
44
|
+
7. The model reads the manifest and diff, then loads `pr-review`.
|
|
45
|
+
8. The channel returns session and playground URLs while the review streams.
|
|
46
|
+
|
|
47
|
+
If a normal chat starts without evidence, the model can call `prepare_pr`
|
|
48
|
+
mid-turn. That form writes the same files into the active session workspace.
|
|
49
|
+
|
|
50
|
+
## Map the evidence-review files
|
|
51
|
+
|
|
52
|
+
| File | Purpose |
|
|
53
|
+
| --- | --- |
|
|
54
|
+
| [`agent/agent.ts`](../../examples/bugbot/agent/agent.ts) | Selects the local runtime and model. |
|
|
55
|
+
| [`agent/instructions.md`](../../examples/bugbot/agent/instructions.md) | Requires diff-first review and confines model work to `pr/`. |
|
|
56
|
+
| [`agent/tools/prepare_pr.ts`](../../examples/bugbot/agent/tools/prepare_pr.ts) | Exposes host preparation as a typed server tool. |
|
|
57
|
+
| [`agent/lib/prepare-pr.ts`](../../examples/bugbot/agent/lib/prepare-pr.ts) | Parses PR references, runs `gh` and `git`, and builds the evidence map. |
|
|
58
|
+
| [`agent/channels/review.ts`](../../examples/bugbot/agent/channels/review.ts) | Provides the loopback-only prepare-and-send HTTP route. |
|
|
59
|
+
| [`agent/channels/slack.ts`](../../examples/bugbot/agent/channels/slack.ts) | Extracts PR references and prepares evidence for mentions and direct messages. |
|
|
60
|
+
| [`agent/skills/pr-review.md`](../../examples/bugbot/agent/skills/pr-review.md) | Sets finding limits, severities, and the machine-readable review format. |
|
|
61
|
+
| [`evals/review/smoke.eval.ts`](../../examples/bugbot/evals/review/smoke.eval.ts) | Seeds fake evidence and checks the review path without GitHub. |
|
|
62
|
+
|
|
63
|
+
There is no authored GitHub channel, MCP connection, subagent, schedule,
|
|
64
|
+
hook, A/B experiment, approval, or custom storage.
|
|
65
|
+
|
|
66
|
+
## Prepare the host
|
|
67
|
+
|
|
68
|
+
You need:
|
|
69
|
+
|
|
70
|
+
- Node 22.13 or newer.
|
|
71
|
+
- An agent-runtime credential for model turns and account-linked Slack.
|
|
72
|
+
- `gh` and `git` on `PATH`.
|
|
73
|
+
- `gh` access to the target PR.
|
|
74
|
+
- Network access to GitHub and a writable temporary directory.
|
|
75
|
+
|
|
76
|
+
The preparer can prefer a configured local checkout. Its `origin` must match
|
|
77
|
+
the target repository. Otherwise the reviewer uses its bare cache. It never
|
|
78
|
+
checks out the PR into the serve host's working tree.
|
|
79
|
+
|
|
80
|
+
## Validate the surface
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
agentkit validate --dir examples/bugbot
|
|
84
|
+
agentkit info --dir examples/bugbot --json
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
The manifest should show one server tool, one skill, and two authored
|
|
88
|
+
channels.
|
|
89
|
+
|
|
90
|
+
## Inspect evidence without a model turn
|
|
91
|
+
|
|
92
|
+
Call the preparation tool directly:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
agentkit call prepare_pr \
|
|
96
|
+
--dir examples/bugbot \
|
|
97
|
+
--input '{"pr":"https://github.com/owner/repo/pull/123"}'
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Direct tool calls use a scratch workspace removed after the call.
|
|
101
|
+
`prepare_pr` detects this path and returns the complete file map in its
|
|
102
|
+
result. In a model session, it writes the files and returns a smaller summary.
|
|
103
|
+
|
|
104
|
+
The evidence builder applies explicit limits:
|
|
105
|
+
|
|
106
|
+
| Evidence | Limit |
|
|
107
|
+
| --- | --- |
|
|
108
|
+
| Post-change file | 12,000 characters |
|
|
109
|
+
| One rule file | 8,000 characters |
|
|
110
|
+
| Combined rules | 12,000 characters |
|
|
111
|
+
| PR body in metadata | 2,000 characters |
|
|
112
|
+
|
|
113
|
+
Large files remain visible in `diff.patch`. The manifest records which full
|
|
114
|
+
files or rules were omitted.
|
|
115
|
+
|
|
116
|
+
The per-file limits aren't an aggregate context cap. Every changed file below
|
|
117
|
+
12,000 characters can be included. The diff command has a 12 MiB output
|
|
118
|
+
buffer; a larger diff fails preparation instead of being truncated.
|
|
119
|
+
|
|
120
|
+
## Run the HTTP review path
|
|
121
|
+
|
|
122
|
+
Start the server:
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
agentkit dev examples/bugbot
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
From another terminal:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
curl -s -X POST \
|
|
132
|
+
http://127.0.0.1:3000/bugbot/v1/channels/review/ \
|
|
133
|
+
-H 'content-type: application/json' \
|
|
134
|
+
-d '{"pr":"https://github.com/owner/repo/pull/123"}'
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
The route returns `status: "started"`, a continuation token, and session and
|
|
138
|
+
playground URLs. Open the session URL to watch the model read the evidence and
|
|
139
|
+
produce findings.
|
|
140
|
+
|
|
141
|
+
The channel declares `localDevStrict()`. Direct loopback callers can use it.
|
|
142
|
+
Proxy-forwarding headers and non-loopback hosts are rejected.
|
|
143
|
+
|
|
144
|
+
Send a follow-up by passing the returned key:
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
curl -s -X POST \
|
|
148
|
+
http://127.0.0.1:3000/bugbot/v1/channels/review/ \
|
|
149
|
+
-H 'content-type: application/json' \
|
|
150
|
+
-d '{"pr":"owner/repo#123","key":"<continuation-token>"}'
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
The follow-up resumes the session without fetching a new evidence tree.
|
|
154
|
+
|
|
155
|
+
## Run the Slack path
|
|
156
|
+
|
|
157
|
+
The account-linked Slack channel handles review-bot mentions and direct
|
|
158
|
+
messages:
|
|
159
|
+
|
|
160
|
+
> Review https://github.com/owner/repo/pull/123
|
|
161
|
+
|
|
162
|
+
Slack handlers don't receive the channel `callTool` helper. This example calls
|
|
163
|
+
the shared `preparePrReview` host function, then returns `workspaceFiles` in
|
|
164
|
+
the Slack message preparation result. The model sees the same evidence and
|
|
165
|
+
prompt as the HTTP path.
|
|
166
|
+
|
|
167
|
+
If a message contains no PR reference, the handler asks for one. Thread
|
|
168
|
+
follow-ups keep the same session.
|
|
169
|
+
|
|
170
|
+
## See how the skill constrains review
|
|
171
|
+
|
|
172
|
+
`pr-review.md` tells the model to:
|
|
173
|
+
|
|
174
|
+
- read the manifest and unified diff first,
|
|
175
|
+
- open at most one supporting file or rules file when a hunk is ambiguous,
|
|
176
|
+
- avoid shell, network, `gh`, and `git`,
|
|
177
|
+
- report no more than three findings,
|
|
178
|
+
- keep each description under 120 words, and
|
|
179
|
+
- emit the machine-readable review contract.
|
|
180
|
+
|
|
181
|
+
The root instructions set the evidence boundary. The skill holds the reusable
|
|
182
|
+
review procedure. Keeping those roles separate lets another agent reuse the
|
|
183
|
+
same skill with different intake channels.
|
|
184
|
+
|
|
185
|
+
## Run the fixture-backed eval
|
|
186
|
+
|
|
187
|
+
```bash
|
|
188
|
+
agentkit eval --dir examples/bugbot --list
|
|
189
|
+
agentkit eval --dir examples/bugbot review/smoke --json
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
The eval constructs a `PreparedPrReview`, seeds its file map through
|
|
193
|
+
`workspaceFiles`, and checks for at least one read call with no shell call. It
|
|
194
|
+
doesn't assert which evidence file was read or whether the skill loaded. It
|
|
195
|
+
accepts either a formatted review or a clean result.
|
|
196
|
+
|
|
197
|
+
This case tests review behavior without GitHub credentials or network data.
|
|
198
|
+
Add fixtures with reachable bugs when you need stricter location and severity
|
|
199
|
+
checks.
|
|
200
|
+
|
|
201
|
+
## Keep the side-effect boundary clear
|
|
202
|
+
|
|
203
|
+
The reviewer makes no remote GitHub writes. It doesn't author a GitHub channel and
|
|
204
|
+
doesn't call a review API. Host preparation does write session evidence and
|
|
205
|
+
force-update `refs/pull/<N>/head` in either its bare cache or a matching local
|
|
206
|
+
checkout when the commit is missing. Every result ends with a note saying no
|
|
207
|
+
GitHub review was posted.
|
|
208
|
+
|
|
209
|
+
If you add publishing later, keep it in a separate tool. This preserves a
|
|
210
|
+
read-only preparation and review path safe to run in evals.
|
|
211
|
+
|
|
212
|
+
## Reuse the evidence handoff
|
|
213
|
+
|
|
214
|
+
Use host-prepared workspaces when:
|
|
215
|
+
|
|
216
|
+
- external APIs should stay off the model's tool surface,
|
|
217
|
+
- context needs hard size limits,
|
|
218
|
+
- the model should inspect a snapshot instead of a live checkout, or
|
|
219
|
+
- several channels need the same preparation.
|
|
220
|
+
|
|
221
|
+
Return `workspaceFiles` from direct host preparation, write into
|
|
222
|
+
`ctx.workspaceDir` for mid-turn recovery, and encode the reading order in both
|
|
223
|
+
the manifest and a skill.
|
|
224
|
+
|
|
225
|
+
## Where to go next
|
|
226
|
+
|
|
227
|
+
- [Webhooks and custom channels](../guides/webhooks.md)
|
|
228
|
+
- [Tools](../reference/tools.md)
|
|
229
|
+
- [Skills](../reference/skills.md)
|
|
230
|
+
- [Slack](../guides/slack.md)
|
|
231
|
+
- [Evals](../evals.md)
|