@vellumai/assistant 0.11.2-dev.202608051957.9bcef49 → 0.11.2-dev.202608052129.f237b1e
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,195 @@
|
|
|
1
|
+
# Vellum Doctor
|
|
2
|
+
|
|
3
|
+
Vellum Doctor is a platform-hosted diagnostic tool for investigating problems with a Vellum assistant.
|
|
4
|
+
|
|
5
|
+
> **Status:** Beta. Vellum Doctor is available in the web client for platform-hosted assistants; it is not available for self-hosted assistants.
|
|
6
|
+
|
|
7
|
+
## What it does
|
|
8
|
+
|
|
9
|
+
Doctor provides a separate diagnostic session where a guardian can describe an issue and receive an investigation that may include:
|
|
10
|
+
|
|
11
|
+
- Plain-language explanations of what the Doctor found.
|
|
12
|
+
- Tool calls with expandable technical details and outputs.
|
|
13
|
+
- Requests for approval before an operation runs.
|
|
14
|
+
- A backup prompt before an operation that may modify the assistant.
|
|
15
|
+
- Feedback prompts when an issue may need attention from the Vellum team.
|
|
16
|
+
- A session transcript that can be copied for follow-up or support.
|
|
17
|
+
|
|
18
|
+
Doctor is separate from the assistant's normal conversation. It is a support and diagnosis surface, not another assistant that builds a long-term relationship with the guardian.
|
|
19
|
+
|
|
20
|
+
## When to use it
|
|
21
|
+
|
|
22
|
+
Open Doctor when an assistant is not behaving as expected, especially when the cause could be a runtime problem, an integration or credential problem, a missing capability, a failed scheduled action, or a permission boundary.
|
|
23
|
+
|
|
24
|
+
The web client may show a **Go to Doctor** action in an operational error notice. Doctor can also be opened from the **Doctor** tab under the assistant's debug settings.
|
|
25
|
+
|
|
26
|
+
### Open Doctor from chat
|
|
27
|
+
|
|
28
|
+
The web client supports a slash command that opens the Doctor panel:
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
/doctor
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
You can include the first message after the command:
|
|
35
|
+
|
|
36
|
+
```text
|
|
37
|
+
/doctor My assistant stopped responding after I connected Slack
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The command navigates to Doctor instead of sending the text as a normal assistant turn. On a self-hosted assistant, Doctor is unavailable and the command does not start a Doctor session.
|
|
41
|
+
|
|
42
|
+
## Session lifecycle
|
|
43
|
+
|
|
44
|
+
A Doctor session follows this flow:
|
|
45
|
+
|
|
46
|
+
1. The web client creates a diagnostic session for the active assistant.
|
|
47
|
+
2. Doctor sends a greeting and opens a live event stream.
|
|
48
|
+
3. The guardian describes the issue and can send follow-up messages while the session is active.
|
|
49
|
+
4. Doctor streams assistant messages, tool activity, approval requests, backup prompts, and errors into the panel.
|
|
50
|
+
5. The session ends with either `completed` or `error` status.
|
|
51
|
+
6. The guardian can start a new session after a terminal state.
|
|
52
|
+
|
|
53
|
+
The web client can load the most recent persisted session when the Doctor panel opens. An active session can be resumed, including pending approval or backup prompts, and completed sessions remain available as history.
|
|
54
|
+
|
|
55
|
+
## Approval and backup prompts
|
|
56
|
+
|
|
57
|
+
Doctor can ask for confirmation before running an operation.
|
|
58
|
+
|
|
59
|
+
- **Allow once** approves the requested operation for the current prompt.
|
|
60
|
+
- **Always Allow** is available for `exec_command` requests and approves future execution requests in that session.
|
|
61
|
+
- **Deny** rejects the requested operation.
|
|
62
|
+
- **Show details** reveals the tool name, description, and input that Doctor received.
|
|
63
|
+
|
|
64
|
+
Before an operation that may modify the assistant, Doctor can ask whether to create a backup first:
|
|
65
|
+
|
|
66
|
+
- **Back up** creates the backup before continuing.
|
|
67
|
+
- **Skip** continues without that backup.
|
|
68
|
+
|
|
69
|
+
Approval and backup prompts are part of the session transcript, so the guardian can see which operations were proposed and how they were handled.
|
|
70
|
+
|
|
71
|
+
## Session history and transcripts
|
|
72
|
+
|
|
73
|
+
Doctor sessions are persisted against the assistant that they diagnose. The web client can retrieve the newest sessions and the ordered message ledger for a selected session.
|
|
74
|
+
|
|
75
|
+
A persisted session includes:
|
|
76
|
+
|
|
77
|
+
- Lifecycle status: `active`, `completed`, or `error`.
|
|
78
|
+
- Ordered user and Doctor messages.
|
|
79
|
+
- Tool calls and tool results.
|
|
80
|
+
- Approval, backup, feedback, status, and error entries.
|
|
81
|
+
- Message timestamps and counts.
|
|
82
|
+
- Token and estimated-cost totals returned by the platform API.
|
|
83
|
+
|
|
84
|
+
The **Copy Session** action serializes the visible session into text. Doctor's idle panel also warns that Doctor logs may be temporarily stored, so guardians should avoid entering secrets that are not necessary for diagnosis.
|
|
85
|
+
|
|
86
|
+
## Feedback and support handoff
|
|
87
|
+
|
|
88
|
+
Doctor can show a feedback prompt when the investigation identifies an issue that may be useful to the Vellum team. The feedback flow can include the Doctor session ID and a transcript file, subject to the diagnostics choice in the feedback form.
|
|
89
|
+
|
|
90
|
+
Sharing feedback is separate from approving a Doctor operation. A guardian can continue describing the issue while the feedback prompt is present.
|
|
91
|
+
|
|
92
|
+
## API reference
|
|
93
|
+
|
|
94
|
+
The platform API exposes Doctor under the assistant-scoped routes below. Requests use the same authentication options as the surrounding platform API.
|
|
95
|
+
|
|
96
|
+
| Operation | Route | Purpose |
|
|
97
|
+
| ---------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------- |
|
|
98
|
+
| List history | `GET /v1/assistants/{assistant_id}/doctor/history/` | List persisted sessions, newest first. |
|
|
99
|
+
| Retrieve history | `GET /v1/assistants/{assistant_id}/doctor/history/{doctor_session_id}/` | Retrieve one session and its ordered message ledger. |
|
|
100
|
+
| Create session | `POST /v1/assistants/{assistant_id}/doctor/sessions/` | Create a new diagnostic session and return its session ID. |
|
|
101
|
+
| Stream events | `GET /v1/assistants/{assistant_id}/doctor/sessions/{session_id}/events/` | Receive Doctor events over Server-Sent Events. |
|
|
102
|
+
| Send message | `POST /v1/assistants/{assistant_id}/doctor/sessions/{session_id}/messages/` | Send a message to an active session. |
|
|
103
|
+
| Record outcome | `POST /v1/assistants/{assistant_id}/doctor/sessions/{session_id}/user-outcome/` | Record whether Doctor solved the problem. |
|
|
104
|
+
| Delete session | `DELETE /v1/assistants/{assistant_id}/doctor/sessions/{session_id}/` | End and clean up a session. |
|
|
105
|
+
|
|
106
|
+
### Create a session
|
|
107
|
+
|
|
108
|
+
A successful create request returns:
|
|
109
|
+
|
|
110
|
+
```json
|
|
111
|
+
{
|
|
112
|
+
"session_id": "doctor-session-id"
|
|
113
|
+
}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### Send a message
|
|
117
|
+
|
|
118
|
+
The message body requires non-empty `content` and can include an optional `source_event_id` for replay-safe delivery:
|
|
119
|
+
|
|
120
|
+
```json
|
|
121
|
+
{
|
|
122
|
+
"content": "My assistant cannot send messages to Slack"
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
A successful request is accepted with HTTP `202` while the response arrives on the event stream.
|
|
127
|
+
|
|
128
|
+
### Record the user outcome
|
|
129
|
+
|
|
130
|
+
When Doctor asks whether it solved the problem, submit the answer to the session's outcome route:
|
|
131
|
+
|
|
132
|
+
```json
|
|
133
|
+
{
|
|
134
|
+
"resolved": true
|
|
135
|
+
}
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Use `false` when the problem was not resolved. The endpoint returns `200`, records the latest answer for the session, and works for both active and completed sessions.
|
|
139
|
+
|
|
140
|
+
### Event types
|
|
141
|
+
|
|
142
|
+
The event stream validates and handles these event types:
|
|
143
|
+
|
|
144
|
+
| Event | Meaning |
|
|
145
|
+
| --------------------- | --------------------------------------------------------------- |
|
|
146
|
+
| `message` | A complete Doctor message. |
|
|
147
|
+
| `message_delta` | A streamed part of a Doctor message. |
|
|
148
|
+
| `tool_call` | Doctor started a tool operation. |
|
|
149
|
+
| `tool_result` | A tool operation returned output, including whether it failed. |
|
|
150
|
+
| `approval_required` | Doctor is waiting for an approval response. |
|
|
151
|
+
| `backup_prompt` | Doctor is asking whether to create a backup before continuing. |
|
|
152
|
+
| `feedback_prompt` | Doctor is offering a feedback handoff. |
|
|
153
|
+
| `user_outcome_prompt` | Doctor is asking whether it solved the guardian's problem. |
|
|
154
|
+
| `status` | The session is active, completed, or in an error state. |
|
|
155
|
+
| `error` | The session encountered an error with a human-readable message. |
|
|
156
|
+
|
|
157
|
+
Malformed events and unknown event types are ignored by the web client rather than being treated as trusted Doctor output.
|
|
158
|
+
|
|
159
|
+
## Failure states and recovery
|
|
160
|
+
|
|
161
|
+
### No assistant is selected
|
|
162
|
+
|
|
163
|
+
Doctor cannot start without an active assistant. Hatch or select an assistant, then reopen the Doctor panel.
|
|
164
|
+
|
|
165
|
+
### Doctor is unavailable
|
|
166
|
+
|
|
167
|
+
Doctor is platform-hosted. Self-hosted assistants do not expose the Doctor tab or the `/doctor` command.
|
|
168
|
+
|
|
169
|
+
### Monthly session limit
|
|
170
|
+
|
|
171
|
+
The platform can reject session creation when the available Doctor sessions for the month have been used. The web client tells the guardian to try again next month.
|
|
172
|
+
|
|
173
|
+
### Service or connection failure
|
|
174
|
+
|
|
175
|
+
Session creation, message delivery, and event streaming can fail when the Doctor service or platform proxy is unavailable. The panel surfaces the returned error and ends the session when it cannot recover.
|
|
176
|
+
|
|
177
|
+
The event stream retries recoverable interruptions with a bounded reconnect policy. If the session has expired, its event history cannot be replayed, or the stream remains idle, the panel asks the guardian to start a new session.
|
|
178
|
+
|
|
179
|
+
### A tool operation fails
|
|
180
|
+
|
|
181
|
+
A failed tool result is shown in the transcript with its technical output. Doctor can continue investigating, ask for another action, or end the session with an error. A failed operation does not become a successful diagnosis merely because the session is still open.
|
|
182
|
+
|
|
183
|
+
## Source map
|
|
184
|
+
|
|
185
|
+
The implementation currently lives in the web client and platform API contract:
|
|
186
|
+
|
|
187
|
+
- `clients/web/src/domains/settings/pages/debug-page.tsx` exposes the Doctor tab.
|
|
188
|
+
- `clients/web/src/domains/settings/components/panels/doctor-panel.tsx` owns session creation, messaging, history, cleanup, and the panel UI.
|
|
189
|
+
- `clients/web/src/domains/settings/components/panels/use-doctor-sse.ts` owns the event stream, validation boundary, reconnect behavior, and replay handling.
|
|
190
|
+
- `clients/web/src/domains/settings/components/panels/doctor-event-schema.ts` defines the validated event contract.
|
|
191
|
+
- `clients/web/src/domains/settings/components/panels/doctor-event-handlers.ts` maps events into visible session entries.
|
|
192
|
+
- `clients/web/src/domains/settings/components/panels/doctor-history.ts` maps persisted messages and serializes copied transcripts.
|
|
193
|
+
- `clients/web/openapi-schemas/platform.yaml` defines the assistant-scoped Doctor routes and persisted session schemas.
|
|
194
|
+
|
|
195
|
+
The Vellum Assistant runtime does not provide a local Doctor CLI command. Doctor is a web and platform capability, not a command run inside a self-hosted assistant.
|
package/package.json
CHANGED
|
@@ -135,7 +135,11 @@ const VELLUM_PROFILE_IMPLS: ProfileImpls = {
|
|
|
135
135
|
label: "Speed",
|
|
136
136
|
description: "Fastest responses, with reasoning turned off",
|
|
137
137
|
maxTokens: 8192,
|
|
138
|
-
|
|
138
|
+
// Explicit reasoning opt-out, matching `cost-optimized` above: this
|
|
139
|
+
// profile advertises reasoning as off, and OpenAI-compat APIs default
|
|
140
|
+
// reasoning to "medium" when the field is omitted, so the opt-out has to
|
|
141
|
+
// be stated rather than implied.
|
|
142
|
+
effort: "none",
|
|
139
143
|
thinking: { enabled: false, streamThinking: false },
|
|
140
144
|
contextWindow: {
|
|
141
145
|
maxInputTokens: DEFAULT_CONTEXT_WINDOW_MAX_INPUT_TOKENS,
|
|
@@ -194,7 +198,7 @@ const BYOK_PROFILE_IMPLS: Record<
|
|
|
194
198
|
label: "Speed",
|
|
195
199
|
description: "Fastest responses, with reasoning turned off",
|
|
196
200
|
maxTokens: 8192,
|
|
197
|
-
effort: "
|
|
201
|
+
effort: "none",
|
|
198
202
|
thinking: { enabled: false, streamThinking: false },
|
|
199
203
|
contextWindow: { maxInputTokens: DEFAULT_CONTEXT_WINDOW_MAX_INPUT_TOKENS },
|
|
200
204
|
},
|
|
@@ -726,7 +726,15 @@ export const PROVIDER_SEED_DATA: Record<
|
|
|
726
726
|
],
|
|
727
727
|
availableScopes:
|
|
728
728
|
"https://learn.microsoft.com/en-us/graph/permissions-reference",
|
|
729
|
-
|
|
729
|
+
// `select_account`, not `consent`: the Microsoft identity platform accepts
|
|
730
|
+
// a single prompt value, and `consent` honours an existing session cookie —
|
|
731
|
+
// it re-asks for consent but never for *which* account, so a user who
|
|
732
|
+
// already signed in cannot connect a second mailbox. `select_account`
|
|
733
|
+
// always shows the account picker with its "Use another account" option.
|
|
734
|
+
// Consent is still collected for an account that has not granted it, and
|
|
735
|
+
// refresh tokens come from the `offline_access` scope above, so nothing is
|
|
736
|
+
// lost by dropping `consent`.
|
|
737
|
+
authorizeParams: { prompt: "select_account" },
|
|
730
738
|
tokenEndpointAuthMethod: "client_secret_post",
|
|
731
739
|
loopbackPort: 17334,
|
|
732
740
|
managedServiceConfigKey: "outlook-oauth",
|