@cursor/july 0.1.5 → 0.1.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (177) hide show
  1. package/dist/ab.d.ts +8 -95
  2. package/dist/ab.d.ts.map +1 -1
  3. package/dist/ab.js +9 -150
  4. package/dist/bin/agent-serve.js +41 -8
  5. package/dist/channels/slack/post-update-delivery.d.ts +85 -0
  6. package/dist/channels/slack/post-update-delivery.d.ts.map +1 -0
  7. package/dist/docs/404.html +2 -2
  8. package/dist/docs/ab.html +4 -4
  9. package/dist/docs/assets/{app.DabPG-io.js → app.COTN7wgo.js} +1 -1
  10. package/dist/docs/assets/chunks/@localSearchIndexroot.B7UcKvIn.js +1 -0
  11. package/dist/docs/assets/chunks/{VPLocalSearchBox.jmyr0bU0.js → VPLocalSearchBox.BW3TBdT0.js} +1 -1
  12. package/dist/docs/assets/chunks/{theme.DysN9-VN.js → theme.BEJW0vE7.js} +2 -2
  13. package/dist/docs/assets/deployment.md.BtfEsc9S.js +55 -0
  14. package/dist/docs/assets/deployment.md.BtfEsc9S.lean.js +1 -0
  15. package/dist/docs/assets/example-agents_approval-buddy.md.8R5phXb5.js +10 -0
  16. package/dist/docs/assets/example-agents_approval-buddy.md.8R5phXb5.lean.js +1 -0
  17. package/dist/docs/assets/example-agents_benny.md.B0gjhI-p.js +7 -0
  18. package/dist/docs/assets/example-agents_benny.md.B0gjhI-p.lean.js +1 -0
  19. package/dist/docs/assets/example-agents_bugbot.md.DelIdhxB.js +11 -0
  20. package/dist/docs/assets/example-agents_bugbot.md.DelIdhxB.lean.js +1 -0
  21. package/dist/docs/assets/example-agents_codebase-wiki.md.DC6sgwn0.js +8 -0
  22. package/dist/docs/assets/example-agents_codebase-wiki.md.DC6sgwn0.lean.js +1 -0
  23. package/dist/docs/assets/example-agents_codeowners-review.md.Ku_tG2RY.js +8 -0
  24. package/dist/docs/assets/example-agents_codeowners-review.md.Ku_tG2RY.lean.js +1 -0
  25. package/dist/docs/assets/example-agents_concierge.md.4rQTSMXt.js +23 -0
  26. package/dist/docs/assets/example-agents_concierge.md.4rQTSMXt.lean.js +1 -0
  27. package/dist/docs/assets/example-agents_fsd.md.CzgUrDfi.js +15 -0
  28. package/dist/docs/assets/example-agents_fsd.md.CzgUrDfi.lean.js +1 -0
  29. package/dist/docs/assets/example-agents_index.md.CRqJlnIf.js +2 -0
  30. package/dist/docs/assets/example-agents_index.md.CRqJlnIf.lean.js +1 -0
  31. package/dist/docs/assets/example-agents_knowledge-base.md.BPJiVueF.js +11 -0
  32. package/dist/docs/assets/example-agents_knowledge-base.md.BPJiVueF.lean.js +1 -0
  33. package/dist/docs/assets/example-agents_security-reviewer.md.D2rtwDTO.js +19 -0
  34. package/dist/docs/assets/example-agents_security-reviewer.md.D2rtwDTO.lean.js +1 -0
  35. package/dist/docs/assets/example-agents_slack-agent.md.buLbgvBf.js +5 -0
  36. package/dist/docs/assets/example-agents_slack-agent.md.buLbgvBf.lean.js +1 -0
  37. package/dist/docs/assets/example-agents_weather-agent.md.C9Qv-W0o.js +24 -0
  38. package/dist/docs/assets/example-agents_weather-agent.md.C9Qv-W0o.lean.js +1 -0
  39. package/dist/docs/assets/index.md.COiu-1jL.js +20 -0
  40. package/dist/docs/assets/{index.md.Cylk70gg.lean.js → index.md.COiu-1jL.lean.js} +1 -1
  41. package/dist/docs/assets/reference_cli.md.D189RBCH.js +60 -0
  42. package/dist/docs/assets/reference_cli.md.D189RBCH.lean.js +1 -0
  43. package/dist/docs/building-with-agents.html +4 -4
  44. package/dist/docs/concepts.html +4 -4
  45. package/dist/docs/deployment.html +58 -17
  46. package/dist/docs/evals.html +4 -4
  47. package/dist/docs/example-agents/approval-buddy.html +34 -0
  48. package/dist/docs/example-agents/benny.html +31 -0
  49. package/dist/docs/example-agents/bugbot.html +35 -0
  50. package/dist/docs/example-agents/codebase-wiki.html +32 -0
  51. package/dist/docs/example-agents/codeowners-review.html +32 -0
  52. package/dist/docs/example-agents/concierge.html +47 -0
  53. package/dist/docs/example-agents/fsd.html +39 -0
  54. package/dist/docs/example-agents/index.html +26 -0
  55. package/dist/docs/example-agents/knowledge-base.html +35 -0
  56. package/dist/docs/example-agents/security-reviewer.html +43 -0
  57. package/dist/docs/example-agents/slack-agent.html +29 -0
  58. package/dist/docs/example-agents/weather-agent.html +48 -0
  59. package/dist/docs/guides/agent-to-agent.html +4 -4
  60. package/dist/docs/guides/cloud-runtime.html +5 -5
  61. package/dist/docs/guides/github.html +4 -4
  62. package/dist/docs/guides/human-in-the-loop.html +4 -4
  63. package/dist/docs/guides/slack.html +4 -4
  64. package/dist/docs/guides/webhooks.html +4 -4
  65. package/dist/docs/hashmap.json +1 -1
  66. package/dist/docs/hillclimbing.html +4 -4
  67. package/dist/docs/index.html +7 -7
  68. package/dist/docs/quickstart.html +4 -4
  69. package/dist/docs/reference/agent-config.html +4 -4
  70. package/dist/docs/reference/channels.html +4 -4
  71. package/dist/docs/reference/cli.html +52 -30
  72. package/dist/docs/reference/connections.html +4 -4
  73. package/dist/docs/reference/hooks.html +4 -4
  74. package/dist/docs/reference/http-api.html +4 -4
  75. package/dist/docs/reference/instructions.html +4 -4
  76. package/dist/docs/reference/playground.html +4 -4
  77. package/dist/docs/reference/project-layout.html +4 -4
  78. package/dist/docs/reference/schedules.html +4 -4
  79. package/dist/docs/reference/sessions.html +4 -4
  80. package/dist/docs/reference/skills.html +4 -4
  81. package/dist/docs/reference/subagents.html +4 -4
  82. package/dist/docs/reference/tools.html +4 -4
  83. package/dist/docs/scaffolding-agents.html +4 -4
  84. package/dist/docs/storage.html +4 -4
  85. package/dist/docs/troubleshooting.html +4 -4
  86. package/dist/evals.d.ts +5 -62
  87. package/dist/evals.d.ts.map +1 -1
  88. package/dist/evals.js +3 -66
  89. package/dist/index.d.ts +1 -1
  90. package/dist/index.d.ts.map +1 -1
  91. package/dist/internal/ab-collector.d.ts +7 -5
  92. package/dist/internal/ab-collector.d.ts.map +1 -1
  93. package/dist/internal/ab-collector.js +3 -14
  94. package/dist/internal/ab-snapshot.d.ts +2 -4
  95. package/dist/internal/ab-snapshot.d.ts.map +1 -1
  96. package/dist/internal/cli-ax.d.ts +33 -5
  97. package/dist/internal/cli-ax.d.ts.map +1 -1
  98. package/dist/internal/cli-ax.js +428 -87
  99. package/dist/internal/cli-deploy.js +1 -1
  100. package/dist/internal/discovery.js +3 -3
  101. package/dist/internal/eval-run-store.d.ts +35 -30
  102. package/dist/internal/eval-run-store.d.ts.map +1 -1
  103. package/dist/internal/eval-run-store.js +88 -100
  104. package/dist/internal/evals-client.d.ts +96 -0
  105. package/dist/internal/evals-client.d.ts.map +1 -0
  106. package/dist/internal/evals-client.js +262 -0
  107. package/dist/internal/init-project.d.ts.map +1 -1
  108. package/dist/internal/init-project.js +1 -0
  109. package/dist/internal/persistence-coordinator.d.ts +127 -0
  110. package/dist/internal/persistence-coordinator.d.ts.map +1 -0
  111. package/dist/internal/playground-proxy.d.ts +5 -5
  112. package/dist/internal/playground-proxy.js +3 -3
  113. package/dist/internal/resolve-prod-target.d.ts +30 -0
  114. package/dist/internal/resolve-prod-target.d.ts.map +1 -1
  115. package/dist/internal/resolve-prod-target.js +74 -2
  116. package/dist/internal/server.d.ts.map +1 -1
  117. package/dist/internal/server.js +16 -5
  118. package/dist/internal/session-engine.d.ts +1 -2
  119. package/dist/internal/session-engine.d.ts.map +1 -1
  120. package/dist/internal/session-engine.js +14 -31
  121. package/dist/internal/storage-coordinator.d.ts +16 -15
  122. package/dist/internal/storage-coordinator.d.ts.map +1 -1
  123. package/dist/internal/storage-coordinator.js +73 -80
  124. package/dist/persistence.d.ts +184 -0
  125. package/dist/persistence.d.ts.map +1 -0
  126. package/dist/playground/assets/cursor-icons-16-CQ50JpfO.woff2 +0 -0
  127. package/dist/playground/assets/index-72vCOBWO.js +86 -0
  128. package/dist/playground/assets/index-BjnMwYoR.css +1 -0
  129. package/dist/playground/index.html +2 -2
  130. package/dist/storage.d.ts +51 -10
  131. package/dist/storage.d.ts.map +1 -1
  132. package/dist/storage.js +27 -10
  133. package/docs/README.md +34 -5
  134. package/docs/deployment.md +352 -149
  135. package/docs/example-agents/approval-buddy.md +270 -0
  136. package/docs/example-agents/benny.md +186 -0
  137. package/docs/example-agents/bugbot.md +231 -0
  138. package/docs/example-agents/codebase-wiki.md +174 -0
  139. package/docs/example-agents/codeowners-review.md +195 -0
  140. package/docs/example-agents/concierge.md +205 -0
  141. package/docs/example-agents/fsd.md +330 -0
  142. package/docs/example-agents/index.md +102 -0
  143. package/docs/example-agents/knowledge-base.md +171 -0
  144. package/docs/example-agents/security-reviewer.md +296 -0
  145. package/docs/example-agents/slack-agent.md +146 -0
  146. package/docs/example-agents/weather-agent.md +302 -0
  147. package/docs/reference/cli.md +546 -147
  148. package/package.json +1 -1
  149. package/src/ab.ts +9 -261
  150. package/src/bin/agent-serve.ts +46 -7
  151. package/src/evals.ts +5 -119
  152. package/src/index.ts +2 -0
  153. package/src/internal/ab-collector.ts +12 -22
  154. package/src/internal/ab-snapshot.ts +2 -4
  155. package/src/internal/cli-ax.ts +551 -104
  156. package/src/internal/cli-deploy.ts +1 -1
  157. package/src/internal/discovery.ts +2 -2
  158. package/src/internal/eval-run-store.ts +91 -100
  159. package/src/internal/evals-client.ts +431 -0
  160. package/src/internal/init-project.ts +1 -0
  161. package/src/internal/playground-proxy.ts +5 -5
  162. package/src/internal/resolve-prod-target.ts +101 -3
  163. package/src/internal/server.ts +17 -3
  164. package/src/internal/session-engine.ts +9 -29
  165. package/src/internal/storage-coordinator.ts +109 -101
  166. package/src/storage.ts +79 -14
  167. package/dist/docs/assets/chunks/@localSearchIndexroot.QwK5BtEH.js +0 -1
  168. package/dist/docs/assets/deployment.md.DTKwE15Z.js +0 -14
  169. package/dist/docs/assets/deployment.md.DTKwE15Z.lean.js +0 -1
  170. package/dist/docs/assets/index.md.Cylk70gg.js +0 -20
  171. package/dist/docs/assets/reference_cli.md.Bv6pOxcF.js +0 -38
  172. package/dist/docs/assets/reference_cli.md.Bv6pOxcF.lean.js +0 -1
  173. package/dist/internal/json-dir-store.js +0 -100
  174. package/dist/playground/assets/cursor-icons-outline-BxTT_FVJ.woff2 +0 -0
  175. package/dist/playground/assets/index-BEauYlII.css +0 -1
  176. package/dist/playground/assets/index-BtM0wEGg.js +0 -319
  177. package/src/internal/json-dir-store.ts +0 -109
@@ -0,0 +1,270 @@
1
+ ---
2
+ title: "Keep PR approval policy deterministic with Approval Buddy"
3
+ description: "Separate code-owned eligibility from model-owned review, then connect GitHub, Slack, subagents, durable storage, and evals."
4
+ ---
5
+
6
+ # Keep PR approval policy deterministic with Approval Buddy
7
+
8
+ Approval Buddy approves eligible pull requests from a fixed roster and
9
+ declines every other request. GitHub still blocks self-approval when the stamp
10
+ identity authored the PR. Code decides eligibility. The model prepares
11
+ evidence, runs two specialist reviews, and passes their findings to the
12
+ approval tool without changing the policy decision.
13
+
14
+ Use this example when an agent can make a judgment inside a workflow, but
15
+ authorization and the final side effect must stay in deterministic code.
16
+
17
+ [Browse the Approval Buddy source.](../../examples/approval-buddy/)
18
+
19
+ ## Keep approval policy in code
20
+
21
+ Approval Buddy draws three hard boundaries:
22
+
23
+ - `prepare_review` and `approve_pr` re-read the live PR and apply the same
24
+ eligibility rules.
25
+ - Two subagents inspect prepared evidence, but their findings never grant or
26
+ block approval.
27
+ - Only `approve_pr` posts the GitHub review.
28
+
29
+ A spoofed webhook, Slack message, or model claim can't add someone to the
30
+ buddy roster. The mutating tool checks the source of truth immediately before it
31
+ acts.
32
+
33
+ ## Follow the intended stamp flow
34
+
35
+ The root instructions ask the model to run this sequence for a qualifying PR:
36
+
37
+ 1. A non-draft `pull_request` event arrives with action `opened`, `reopened`,
38
+ or `ready_for_review`.
39
+ 2. The GitHub channel checks its repository allowlist and starts a session.
40
+ 3. `turn.started` posts a pending commit status.
41
+ 4. The model calls `prepare_review`.
42
+ 5. Host code fetches the live PR. It checks the author, open state, merged
43
+ state, and draft state.
44
+ 6. A qualifying PR gets `pr/MANIFEST.md`, `pr/meta.json`, and
45
+ `pr/diff.patch` in the session workspace. Diffs above 2,000,000
46
+ characters are truncated and marked in metadata.
47
+ 7. The model calls both review subagents through the built-in `task` tool.
48
+ 8. It concatenates their contracted replies and calls `approve_pr`.
49
+ 9. `approve_pr` re-runs eligibility, posts an `APPROVE` review, and returns
50
+ the outcome.
51
+ 10. The channel posts a final commit status. A self-approval block also gets
52
+ a short timeline comment because no approval review can appear.
53
+
54
+ Ineligible PRs skip evidence and subagents. The model still calls
55
+ `approve_pr` so the deterministic tool returns the formal decline reason.
56
+
57
+ Steps 4 through 9 are prompt-driven. The channel doesn't enforce tool order
58
+ or prove both subagents ran, and `approve_pr` accepts missing findings. A
59
+ failed turn clears the pending status with a green non-blocking result without
60
+ approving the PR.
61
+
62
+ ## Map the framework features
63
+
64
+ | Capability | Source | Role |
65
+ | --- | --- | --- |
66
+ | Root agent and policy prompt | [`agent/agent.ts`](../../examples/approval-buddy/agent/agent.ts), [`agent/instructions.md`](../../examples/approval-buddy/agent/instructions.md) | Configure the local agent and describe orchestration order. |
67
+ | GitHub channel | [`agent/channels/github.ts`](../../examples/approval-buddy/agent/channels/github.ts) | Filter wakes, lease GitHub access, and publish status events. |
68
+ | Slack channel | [`agent/channels/slack.ts`](../../examples/approval-buddy/agent/channels/slack.ts) | Accept approval-bot stamp and qualification requests. |
69
+ | Server tools | [`agent/tools/`](../../examples/approval-buddy/agent/tools/) | Prepare evidence, approve, list buddies, and search GIFs. |
70
+ | Deterministic policy | [`agent/lib/approve.ts`](../../examples/approval-buddy/agent/lib/approve.ts), [`agent/lib/buddies.ts`](../../examples/approval-buddy/agent/lib/buddies.ts) | Own the roster and live eligibility checks. |
71
+ | Review subagents | [`agent/subagents/`](../../examples/approval-buddy/agent/subagents/) | Run deep audit and code-quality passes over the same evidence. |
72
+ | Storage | [`agent/storage.ts`](../../examples/approval-buddy/agent/storage.ts) | Persist sessions and events through a custom durable adapter. |
73
+ | Evals and unit tests | [`evals/`](../../examples/approval-buddy/evals/), [`agent/lib/`](../../examples/approval-buddy/agent/lib/) | Protect routing, output contracts, policy, and GitHub behavior. |
74
+
75
+ There are no authored skills, MCP connections, schedules, reminders, hooks,
76
+ A/B experiments, sandbox seeds, or tool approvals.
77
+
78
+ ## Prepare credentials
79
+
80
+ You need:
81
+
82
+ - Node 22.13 or newer.
83
+ - An agent-runtime credential.
84
+ - GitHub access to read PRs, post reviews, create commit statuses, and
85
+ post the self-approval visibility comment.
86
+
87
+ Optional GIF selection uses:
88
+
89
+ - `GIPHY_API_KEY` or `APPROVAL_BUDDY_GIPHY_API_KEY`,
90
+ - `APPROVAL_BUDDY_STAMP_GIF`, or
91
+ - severity-specific `APPROVAL_BUDDY_STAMP_GIF_<LEVEL>` variables.
92
+
93
+ If you enable Giphy in a hosted copy, declare its secret and
94
+ `api.giphy.com` egress.
95
+
96
+ ## Validate without approving a PR
97
+
98
+ ```bash
99
+ agentkit validate --dir examples/approval-buddy
100
+ agentkit info --dir examples/approval-buddy --json
101
+ ```
102
+
103
+ List the deterministic roster:
104
+
105
+ ```bash
106
+ agentkit call list_buddies \
107
+ --dir examples/approval-buddy \
108
+ --input '{}'
109
+ ```
110
+
111
+ Set a known merged PR, then run the read-only precheck:
112
+
113
+ ```bash
114
+ MERGED_PR_URL=https://github.com/your-org/your-repo/pull/123
115
+ agentkit call prepare_review \
116
+ --dir examples/approval-buddy \
117
+ --input "{\"prUrl\":\"$MERGED_PR_URL\"}"
118
+ ```
119
+
120
+ The result should decline because the PR is no longer open. `prepare_review`
121
+ never posts an approval.
122
+
123
+ > [!CAUTION]
124
+ > Don't use `agentkit call approve_pr` as a smoke test. The tool has no
125
+ > `needsApproval` gate and posts a real GitHub review when the PR qualifies.
126
+
127
+ ## See why preparation is separate
128
+
129
+ `prepare_review` is read-only. It checks policy before fetching a large diff,
130
+ so declined requests don't spend review-agent work.
131
+
132
+ Direct calls return the evidence file map because their scratch workspace is
133
+ deleted after the call. In-session calls write the tree to
134
+ `ctx.workspaceDir`, where both subagents can read it.
135
+
136
+ `approve_pr` repeats the live check instead of trusting preparation. A PR can
137
+ close, merge, become a draft, or change author-related context between the two
138
+ steps. Revalidation keeps the final write bound to current state.
139
+
140
+ This is a reusable two-tool pattern:
141
+
142
+ - a read-only tool prepares and explains the decision,
143
+ - a mutating tool repeats policy at the side-effect boundary.
144
+
145
+ ## Fan out two review contracts
146
+
147
+ The two discovered subagents have different contracts:
148
+
149
+ - The security reviewer reports bugs, breaking changes, and security findings
150
+ with `High`, `Medium`, or `Low` tags.
151
+ - The code-quality reviewer reports maintainability and structure concerns
152
+ with `Blocker`, `Major`, or `Minor` tags.
153
+
154
+ The parent calls both through the harness `task` tool. They inherit the root
155
+ agent's execution surface and read the same `pr/` workspace. The prompt asks
156
+ the parent not to rewrite either reply. The review body trims the combined
157
+ text and caps it at 16,000 characters.
158
+
159
+ Findings are informational. A high-severity finding doesn't veto the stamp.
160
+ That policy is explicit in the root instructions and approval code.
161
+
162
+ ## Trace GitHub channel behavior
163
+
164
+ The channel uses `githubChannel` with:
165
+
166
+ - a configured repository allowlist on the account-linked GitHub transport,
167
+ - a second optional `APPROVAL_BUDDY_REPOS` wake filter,
168
+ - `deliverReplies: false`,
169
+ - progress reactions disabled, and
170
+ - event handlers for turn start, `approve_pr` results, and failed turns.
171
+
172
+ The source requests `contents-write`, even though the documented workflow
173
+ posts reviews, statuses, and comments. When adapting the example, start with
174
+ `pr-write` and opt up only if a tool must push code.
175
+
176
+ Every terminal status is green by design. Declines and crashed turns are
177
+ informational, not merge-blocking. This is a product decision in the example,
178
+ not an agentkit default.
179
+
180
+ A successful turn that never calls `approve_pr` leaves the pending status in
181
+ place. The channel clears it on `approve_pr` results and `turn.failed`, but
182
+ has no `turn.completed` fallback.
183
+
184
+ `github replay` reaches the same channel and can post a real approval, status,
185
+ or comment. Use replay only against a repository and PR created for this
186
+ test.
187
+
188
+ ## Use Slack for explicit requests
189
+
190
+ Start the dev server:
191
+
192
+ ```bash
193
+ agentkit dev examples/approval-buddy
194
+ ```
195
+
196
+ Then ask through the signed-in account-linked Slack connection:
197
+
198
+ > Would this PR qualify for a stamp?
199
+
200
+ The instructions route qualification questions to `prepare_review` only. A
201
+ stamp request runs the complete flow and may approve the PR.
202
+
203
+ This channel uses the account-linked transport instead of a dedicated Socket
204
+ Mode app.
205
+
206
+ ## See how durable storage fits
207
+
208
+ `defineStorage` replaces the default local session store with a shared,
209
+ durable key-value adapter. Approval Buddy chooses:
210
+
211
+ - a 15-second write debounce,
212
+ - startup restoration for up to 200 sessions, and
213
+ - a 14-day restore window.
214
+
215
+ That policy fits long-lived Slack threads and a small webhook fleet. The
216
+ security reviewer uses the same adapter with lazy restore, which fits its
217
+ shorter sessions.
218
+
219
+ ## Run the regression suite
220
+
221
+ List the four eval cases:
222
+
223
+ ```bash
224
+ agentkit eval --dir examples/approval-buddy --list
225
+ ```
226
+
227
+ The suite covers:
228
+
229
+ - buddy-list routing,
230
+ - declining a merged PR,
231
+ - using only `prepare_review` for a qualification question, and
232
+ - the combined findings headings and severity format over seeded evidence.
233
+
234
+ Run the safe qualification case:
235
+
236
+ ```bash
237
+ agentkit eval \
238
+ --dir examples/approval-buddy \
239
+ qualify/merged-pr-question \
240
+ --json
241
+ ```
242
+
243
+ The qualification case reads a live merged PR. The seeded format case shown
244
+ by `--list` uses a planted auth-bypass diff
245
+ and checks for a `task` call, both headings, and severity tags. It doesn't
246
+ prove both named subagents ran or whether their output reached `approve_pr`.
247
+ Unit tests under `agent/lib/` cover policy, self-approval handling, evidence
248
+ limits, status mapping, GIF selection, and severity parsing.
249
+
250
+ ## Reuse the policy boundary
251
+
252
+ Keep these properties when you replace the buddy policy:
253
+
254
+ 1. Put authorization in typed code.
255
+ 2. Fetch the source of truth inside both prepare and mutate steps.
256
+ 3. Give the model evidence only after the request qualifies.
257
+ 4. Treat specialist findings as data, not authority.
258
+ 5. Keep the core domain mutation in one named tool. Treat channel status and
259
+ visibility writes as separate, audited effects.
260
+ 6. Add a human approval gate if your policy still needs operator consent.
261
+ 7. Test read-only routing separately from mutation.
262
+
263
+ ## Where to go next
264
+
265
+ - [GitHub](../guides/github.md)
266
+ - [Tools](../reference/tools.md)
267
+ - [Subagents](../reference/subagents.md)
268
+ - [Storage](../storage.md)
269
+ - [Slack](../guides/slack.md)
270
+ - [Evals](../evals.md)
@@ -0,0 +1,186 @@
1
+ ---
2
+ title: "Route Slack work through repository playbooks"
3
+ description: "Combine account-linked chat, allowlisted Socket Mode channel watching, inherited repository skills, and a custom local workspace."
4
+ ---
5
+
6
+ # Route Slack work through repository playbooks
7
+
8
+ This agent is a Slack teammate for a product team. Mentions and direct
9
+ messages reach it through an account-linked transport. New top-level posts in
10
+ an allowlisted issue channel reach it through a dedicated Slack app, even
11
+ without a mention. The agent then selects a repository playbook for triage,
12
+ reproduction, fixes, reviews, on-call work, or design critique.
13
+
14
+ Use this example when Slack is the intake surface and your durable procedures
15
+ already live as repository skills.
16
+
17
+ [Browse the current playbook-router source.](../../examples/benny/)
18
+
19
+ ## Combine two Slack transports with repo skills
20
+
21
+ The playbook router uniquely combines three decisions:
22
+
23
+ - Two Slack transports serve different engagement modes.
24
+ - `local.cwd` keeps session workspaces inside the monorepo.
25
+ - Instructions route work to inherited repository playbooks
26
+ instead of authored `agent/skills/`.
27
+
28
+ The result is a thin agent project over a mature procedure library.
29
+
30
+ ## Follow an issue report
31
+
32
+ 1. A teammate creates a top-level post in the allowlisted issue channel.
33
+ 2. The dedicated Socket Mode channel accepts the allowlisted channel.
34
+ 3. A 15-second debounce lets edits settle. Deleting the post during that
35
+ window cancels the dispatch.
36
+ 4. Agentkit creates a thread-scoped session and sends the report to the
37
+ playbook router.
38
+ 5. The instructions select the matching triage playbook.
39
+ 6. The harness finds the repository root, opens the inherited playbook, and
40
+ follows its procedure.
41
+ 7. The agent posts only in the source thread and reports the evidence it
42
+ gathered.
43
+
44
+ Mentions and direct messages follow the same agent instructions. They don't
45
+ need the watched-channel path.
46
+
47
+ ## Map the playbook router files
48
+
49
+ | File | Purpose |
50
+ | --- | --- |
51
+ | [`agent/agent.ts`](../../examples/benny/agent/agent.ts) | Names the agent, selects its model, and keeps the harness under `.agent-serve/harness`. |
52
+ | [`agent/instructions.md`](../../examples/benny/agent/instructions.md) | Defines engagement rules, evidence policy, and the playbook routing map. |
53
+ | [`agent/channels/slack.ts`](../../examples/benny/agent/channels/slack.ts) | Handles account-linked mentions and direct messages. |
54
+ | [`agent/channels/slack-app.ts`](../../examples/benny/agent/channels/slack-app.ts) | Runs the dedicated app and watches one allowlisted channel. |
55
+ | [`evals/smoke.eval.ts`](../../examples/benny/evals/smoke.eval.ts) | Checks the agent identity and expected triage route. |
56
+
57
+ The playbook router authors no tools, MCP connections, subagents, schedules, hooks, A/B
58
+ experiments, or sandbox seeds.
59
+
60
+ ## See why `local.cwd` matters
61
+
62
+ Agentkit normally keeps an ephemeral `run` or `eval` workspace outside a
63
+ large monorepo. This prevents ancestor instruction and repository-rule files
64
+ from leaking into an unrelated agent.
65
+
66
+ The playbook router needs the opposite. Its procedures live at the repository
67
+ root, so
68
+ `agent.ts` sets:
69
+
70
+ ```ts
71
+ local: {
72
+ cwd: ".agent-serve/harness",
73
+ }
74
+ ```
75
+
76
+ Each harness workspace lands under
77
+ `examples/benny/.agent-serve/harness/<sessionId>`. Walking up the directory
78
+ tree reaches the host repository and its inherited playbook directory.
79
+
80
+ Those playbooks are inherited context. `agentkit info` reports zero authored
81
+ skills for the agent. Copying this project into another repository removes
82
+ its main procedures unless you copy or replace the skill library too.
83
+
84
+ ## Connect both Slack paths
85
+
86
+ The account-linked path needs an agent-runtime login and a connected Slack
87
+ account:
88
+
89
+ ```bash
90
+ agentkit login
91
+ agentkit whoami
92
+ ```
93
+
94
+ It routes explicit mentions without a dedicated Slack token on the host.
95
+
96
+ For the watched-channel path, configure a dedicated Socket Mode app with:
97
+
98
+ - subscribe to `message.channels` and `message.groups`,
99
+ - have an App-Level Token with `connections:write`, and
100
+ - be a member of the watched channel.
101
+
102
+ Run `agentkit slack setup` for the guided app workflow. Generate the project
103
+ manifest with `--channel-posts` when you create a new copy, then validate the
104
+ configured channel prefix with `agentkit slack doctor`.
105
+
106
+ Missing dedicated-app tokens leave that channel idle. They don't stop the
107
+ account-linked channel.
108
+
109
+ ## Validate and start the server
110
+
111
+ ```bash
112
+ agentkit validate --dir examples/benny
113
+ agentkit info --dir examples/benny --json
114
+ agentkit dev examples/benny
115
+ ```
116
+
117
+ The info output should show two Slack channels and no authored skill. That
118
+ combination confirms the example is using inherited playbooks.
119
+
120
+ ## Exercise each engagement mode
121
+
122
+ Test the explicit account-linked path by asking:
123
+
124
+ > Which playbook would you use to triage a product UI bug?
125
+
126
+ Test the dedicated app:
127
+
128
+ 1. Create a top-level post in the allowlisted issue channel.
129
+ 2. Don't mention the bot.
130
+ 3. Wait for the debounce window.
131
+ 4. Confirm the agent replies in the post's thread.
132
+
133
+ Thread replies don't trigger the proactive watch. Mentions still use Slack's
134
+ normal mention path. Bot-authored posts are ignored to prevent loops.
135
+
136
+ The channel uses the default handler after filtering. It doesn't apply a
137
+ second code-level classifier, so every accepted top-level post spends a model
138
+ turn and reaches the prompt.
139
+
140
+ ## Inspect thread continuity
141
+
142
+ Agentkit keys Slack sessions by channel and thread timestamp. A follow-up in
143
+ the same thread resumes the conversation and workspace. A new top-level issue
144
+ gets a new session.
145
+
146
+ This lets a playbook gather evidence over several turns without mixing two
147
+ reports. The playground shows both the account-linked and dedicated-app
148
+ sessions while the dev server runs.
149
+
150
+ ## Run the smoke eval
151
+
152
+ ```bash
153
+ agentkit eval --dir examples/benny --list
154
+ agentkit eval --dir examples/benny smoke --json
155
+ ```
156
+
157
+ The case asks for the agent identity and the playbook used for issue triage.
158
+ It checks the configured identity and route label.
159
+
160
+ This is a lexical smoke test. It doesn't prove Slack delivery, skill
161
+ selection, skill loading, procedure execution, or thread-only behavior. Add
162
+ fixture-backed evals around the playbooks when you reuse this design.
163
+
164
+ ## Build a playbook-routed teammate
165
+
166
+ Use this structure when your organization already has tested skills:
167
+
168
+ 1. Put the playbooks under a stable repository path.
169
+ 2. Set `local.cwd` so harness workspaces can inherit that path.
170
+ 3. Write a short routing table in `instructions.md`.
171
+ 4. Use account-linked Slack for explicit requests.
172
+ 5. Add a dedicated app only for allowlisted proactive intake.
173
+ 6. Keep the channel allowlist narrow and debounce edited posts.
174
+ 7. Add an eval for every important request-to-playbook route.
175
+
176
+ If the procedures should ship with the agent, put them under
177
+ `agent/skills/` instead. Authored skills appear in the manifest and travel
178
+ with the project.
179
+
180
+ ## Where to go next
181
+
182
+ - [Slack](../guides/slack.md)
183
+ - [Agent config](../reference/agent-config.md)
184
+ - [Skills](../reference/skills.md)
185
+ - [Sessions and streaming](../reference/sessions.md)
186
+ - [Evals](../evals.md)
@@ -0,0 +1,231 @@
1
+ ---
2
+ title: "Review prepared pull-request evidence"
3
+ description: "Fetch a PR on the host, seed a trimmed diff-first workspace, and run a GitHub-read-only review through HTTP, Slack, or an eval."
4
+ ---
5
+
6
+ # Review prepared pull-request evidence
7
+
8
+ This GitHub-read-only reviewer uses host code to fetch the PR
9
+ with `gh` and `git`, builds a trimmed `pr/` evidence tree, then hands that tree
10
+ to the model. The model reads the diff, loads a review skill, and returns at
11
+ most three high-confidence findings.
12
+
13
+ Use this example when the host should control evidence collection and the
14
+ model shouldn't browse or mutate the source repository.
15
+
16
+ [Browse the current reviewer source.](../../examples/bugbot/)
17
+
18
+ ## Separate evidence preparation from review
19
+
20
+ The reviewer separates preparation from judgment:
21
+
22
+ - Host code owns GitHub and Git access.
23
+ - A server tool turns untrusted PR input into bounded workspace files.
24
+ - A custom channel seeds those files before the model starts.
25
+ - An on-demand skill defines the review procedure and output contract.
26
+ - The model returns chat text. No path posts a GitHub review.
27
+
28
+ This architecture gives the model a purpose-built evidence package instead of
29
+ a checkout.
30
+
31
+ ## Follow a review
32
+
33
+ The custom HTTP path runs this sequence:
34
+
35
+ 1. `POST /v1/channels/review/` receives a PR reference.
36
+ 2. The handler calls `prepare_pr` without a model turn.
37
+ 3. Host code reads PR metadata and the unified diff.
38
+ 4. It reuses a matching checkout, force-fetching the PR ref there when the
39
+ commit is missing. Without a matching checkout, it uses a temporary bare
40
+ cache.
41
+ 5. It creates `pr/MANIFEST.md`, `pr/meta.json`, `pr/diff.patch`, and selected
42
+ small files and rules.
43
+ 6. `send({ workspaceFiles })` creates the model session with that evidence.
44
+ 7. The model reads the manifest and diff, then loads `pr-review`.
45
+ 8. The channel returns session and playground URLs while the review streams.
46
+
47
+ If a normal chat starts without evidence, the model can call `prepare_pr`
48
+ mid-turn. That form writes the same files into the active session workspace.
49
+
50
+ ## Map the evidence-review files
51
+
52
+ | File | Purpose |
53
+ | --- | --- |
54
+ | [`agent/agent.ts`](../../examples/bugbot/agent/agent.ts) | Selects the local runtime and model. |
55
+ | [`agent/instructions.md`](../../examples/bugbot/agent/instructions.md) | Requires diff-first review and confines model work to `pr/`. |
56
+ | [`agent/tools/prepare_pr.ts`](../../examples/bugbot/agent/tools/prepare_pr.ts) | Exposes host preparation as a typed server tool. |
57
+ | [`agent/lib/prepare-pr.ts`](../../examples/bugbot/agent/lib/prepare-pr.ts) | Parses PR references, runs `gh` and `git`, and builds the evidence map. |
58
+ | [`agent/channels/review.ts`](../../examples/bugbot/agent/channels/review.ts) | Provides the loopback-only prepare-and-send HTTP route. |
59
+ | [`agent/channels/slack.ts`](../../examples/bugbot/agent/channels/slack.ts) | Extracts PR references and prepares evidence for mentions and direct messages. |
60
+ | [`agent/skills/pr-review.md`](../../examples/bugbot/agent/skills/pr-review.md) | Sets finding limits, severities, and the machine-readable review format. |
61
+ | [`evals/review/smoke.eval.ts`](../../examples/bugbot/evals/review/smoke.eval.ts) | Seeds fake evidence and checks the review path without GitHub. |
62
+
63
+ There is no authored GitHub channel, MCP connection, subagent, schedule,
64
+ hook, A/B experiment, approval, or custom storage.
65
+
66
+ ## Prepare the host
67
+
68
+ You need:
69
+
70
+ - Node 22.13 or newer.
71
+ - An agent-runtime credential for model turns and account-linked Slack.
72
+ - `gh` and `git` on `PATH`.
73
+ - `gh` access to the target PR.
74
+ - Network access to GitHub and a writable temporary directory.
75
+
76
+ The preparer can prefer a configured local checkout. Its `origin` must match
77
+ the target repository. Otherwise the reviewer uses its bare cache. It never
78
+ checks out the PR into the serve host's working tree.
79
+
80
+ ## Validate the surface
81
+
82
+ ```bash
83
+ agentkit validate --dir examples/bugbot
84
+ agentkit info --dir examples/bugbot --json
85
+ ```
86
+
87
+ The manifest should show one server tool, one skill, and two authored
88
+ channels.
89
+
90
+ ## Inspect evidence without a model turn
91
+
92
+ Call the preparation tool directly:
93
+
94
+ ```bash
95
+ agentkit call prepare_pr \
96
+ --dir examples/bugbot \
97
+ --input '{"pr":"https://github.com/owner/repo/pull/123"}'
98
+ ```
99
+
100
+ Direct tool calls use a scratch workspace removed after the call.
101
+ `prepare_pr` detects this path and returns the complete file map in its
102
+ result. In a model session, it writes the files and returns a smaller summary.
103
+
104
+ The evidence builder applies explicit limits:
105
+
106
+ | Evidence | Limit |
107
+ | --- | --- |
108
+ | Post-change file | 12,000 characters |
109
+ | One rule file | 8,000 characters |
110
+ | Combined rules | 12,000 characters |
111
+ | PR body in metadata | 2,000 characters |
112
+
113
+ Large files remain visible in `diff.patch`. The manifest records which full
114
+ files or rules were omitted.
115
+
116
+ The per-file limits aren't an aggregate context cap. Every changed file below
117
+ 12,000 characters can be included. The diff command has a 12 MiB output
118
+ buffer; a larger diff fails preparation instead of being truncated.
119
+
120
+ ## Run the HTTP review path
121
+
122
+ Start the server:
123
+
124
+ ```bash
125
+ agentkit dev examples/bugbot
126
+ ```
127
+
128
+ From another terminal:
129
+
130
+ ```bash
131
+ curl -s -X POST \
132
+ http://127.0.0.1:3000/bugbot/v1/channels/review/ \
133
+ -H 'content-type: application/json' \
134
+ -d '{"pr":"https://github.com/owner/repo/pull/123"}'
135
+ ```
136
+
137
+ The route returns `status: "started"`, a continuation token, and session and
138
+ playground URLs. Open the session URL to watch the model read the evidence and
139
+ produce findings.
140
+
141
+ The channel declares `localDevStrict()`. Direct loopback callers can use it.
142
+ Proxy-forwarding headers and non-loopback hosts are rejected.
143
+
144
+ Send a follow-up by passing the returned key:
145
+
146
+ ```bash
147
+ curl -s -X POST \
148
+ http://127.0.0.1:3000/bugbot/v1/channels/review/ \
149
+ -H 'content-type: application/json' \
150
+ -d '{"pr":"owner/repo#123","key":"<continuation-token>"}'
151
+ ```
152
+
153
+ The follow-up resumes the session without fetching a new evidence tree.
154
+
155
+ ## Run the Slack path
156
+
157
+ The account-linked Slack channel handles review-bot mentions and direct
158
+ messages:
159
+
160
+ > Review https://github.com/owner/repo/pull/123
161
+
162
+ Slack handlers don't receive the channel `callTool` helper. This example calls
163
+ the shared `preparePrReview` host function, then returns `workspaceFiles` in
164
+ the Slack message preparation result. The model sees the same evidence and
165
+ prompt as the HTTP path.
166
+
167
+ If a message contains no PR reference, the handler asks for one. Thread
168
+ follow-ups keep the same session.
169
+
170
+ ## See how the skill constrains review
171
+
172
+ `pr-review.md` tells the model to:
173
+
174
+ - read the manifest and unified diff first,
175
+ - open at most one supporting file or rules file when a hunk is ambiguous,
176
+ - avoid shell, network, `gh`, and `git`,
177
+ - report no more than three findings,
178
+ - keep each description under 120 words, and
179
+ - emit the machine-readable review contract.
180
+
181
+ The root instructions set the evidence boundary. The skill holds the reusable
182
+ review procedure. Keeping those roles separate lets another agent reuse the
183
+ same skill with different intake channels.
184
+
185
+ ## Run the fixture-backed eval
186
+
187
+ ```bash
188
+ agentkit eval --dir examples/bugbot --list
189
+ agentkit eval --dir examples/bugbot review/smoke --json
190
+ ```
191
+
192
+ The eval constructs a `PreparedPrReview`, seeds its file map through
193
+ `workspaceFiles`, and checks for at least one read call with no shell call. It
194
+ doesn't assert which evidence file was read or whether the skill loaded. It
195
+ accepts either a formatted review or a clean result.
196
+
197
+ This case tests review behavior without GitHub credentials or network data.
198
+ Add fixtures with reachable bugs when you need stricter location and severity
199
+ checks.
200
+
201
+ ## Keep the side-effect boundary clear
202
+
203
+ The reviewer makes no remote GitHub writes. It doesn't author a GitHub channel and
204
+ doesn't call a review API. Host preparation does write session evidence and
205
+ force-update `refs/pull/<N>/head` in either its bare cache or a matching local
206
+ checkout when the commit is missing. Every result ends with a note saying no
207
+ GitHub review was posted.
208
+
209
+ If you add publishing later, keep it in a separate tool. This preserves a
210
+ read-only preparation and review path safe to run in evals.
211
+
212
+ ## Reuse the evidence handoff
213
+
214
+ Use host-prepared workspaces when:
215
+
216
+ - external APIs should stay off the model's tool surface,
217
+ - context needs hard size limits,
218
+ - the model should inspect a snapshot instead of a live checkout, or
219
+ - several channels need the same preparation.
220
+
221
+ Return `workspaceFiles` from direct host preparation, write into
222
+ `ctx.workspaceDir` for mid-turn recovery, and encode the reading order in both
223
+ the manifest and a skill.
224
+
225
+ ## Where to go next
226
+
227
+ - [Webhooks and custom channels](../guides/webhooks.md)
228
+ - [Tools](../reference/tools.md)
229
+ - [Skills](../reference/skills.md)
230
+ - [Slack](../guides/slack.md)
231
+ - [Evals](../evals.md)