agent-scaffold-cli 0.1.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agent_scaffold/__init__.py +8 -0
- agent_scaffold/__main__.py +6 -0
- agent_scaffold/_bundled_deployments/__init__.py +15 -0
- agent_scaffold/_bundled_deployments/docs/cross-cutting/README.md +15 -0
- agent_scaffold/_bundled_deployments/docs/cross-cutting/auth-jwt.md +235 -0
- agent_scaffold/_bundled_deployments/docs/cross-cutting/logging-structured.md +196 -0
- agent_scaffold/_bundled_deployments/docs/cross-cutting/observability.md +259 -0
- agent_scaffold/_bundled_deployments/docs/cross-cutting/rate-limiting.md +171 -0
- agent_scaffold/_bundled_deployments/docs/cross-cutting/testing-strategy.md +261 -0
- agent_scaffold/_bundled_deployments/docs/frameworks/README.md +22 -0
- agent_scaffold/_bundled_deployments/docs/frameworks/crewai.md +91 -0
- agent_scaffold/_bundled_deployments/docs/frameworks/langgraph.md +79 -0
- agent_scaffold/_bundled_deployments/docs/frameworks/mastra.md +74 -0
- agent_scaffold/_bundled_deployments/docs/frameworks/pydantic-ai.md +77 -0
- agent_scaffold/_bundled_deployments/docs/frameworks/vercel-ai-sdk.md +83 -0
- agent_scaffold/_bundled_deployments/docs/patterns/README.md +26 -0
- agent_scaffold/_bundled_deployments/docs/patterns/memory.md +82 -0
- agent_scaffold/_bundled_deployments/docs/patterns/multi-agent-flat.md +72 -0
- agent_scaffold/_bundled_deployments/docs/patterns/multi-agent-hierarchical.md +83 -0
- agent_scaffold/_bundled_deployments/docs/patterns/parallel-calls.md +73 -0
- agent_scaffold/_bundled_deployments/docs/patterns/plan-execute-reflect.md +77 -0
- agent_scaffold/_bundled_deployments/docs/patterns/prompt-chaining.md +73 -0
- agent_scaffold/_bundled_deployments/docs/patterns/rag.md +84 -0
- agent_scaffold/_bundled_deployments/docs/patterns/react.md +77 -0
- agent_scaffold/_bundled_deployments/docs/patterns/routing-tool-use.md +69 -0
- agent_scaffold/_bundled_deployments/docs/recipes/README.md +39 -0
- agent_scaffold/_bundled_deployments/docs/recipes/code-review-agent.md +518 -0
- agent_scaffold/_bundled_deployments/docs/recipes/content-pipeline.md +525 -0
- agent_scaffold/_bundled_deployments/docs/recipes/customer-support-triage.md +1679 -0
- agent_scaffold/_bundled_deployments/docs/recipes/docs-rag-qa.md +1254 -0
- agent_scaffold/_bundled_deployments/docs/recipes/hierarchical-agent.md +554 -0
- agent_scaffold/_bundled_deployments/docs/recipes/memory-assistant.md +499 -0
- agent_scaffold/_bundled_deployments/docs/recipes/ops-crew.md +457 -0
- agent_scaffold/_bundled_deployments/docs/recipes/parallel-enricher.md +457 -0
- agent_scaffold/_bundled_deployments/docs/recipes/research-assistant.md +1096 -0
- agent_scaffold/_bundled_deployments/docs/stack/README.md +19 -0
- agent_scaffold/_bundled_deployments/docs/stack/api-fastapi.md +112 -0
- agent_scaffold/_bundled_deployments/docs/stack/api-hono.md +108 -0
- agent_scaffold/_bundled_deployments/docs/stack/cache-redis.md +85 -0
- agent_scaffold/_bundled_deployments/docs/stack/eval-deepeval-ragas-promptfoo.md +164 -0
- agent_scaffold/_bundled_deployments/docs/stack/llm-claude.md +105 -0
- agent_scaffold/_bundled_deployments/docs/stack/relational-postgres.md +122 -0
- agent_scaffold/_bundled_deployments/docs/stack/tool-protocol-mcp.md +275 -0
- agent_scaffold/_bundled_deployments/docs/stack/tracing-langfuse.md +108 -0
- agent_scaffold/_bundled_deployments/docs/stack/vector-qdrant.md +121 -0
- agent_scaffold/cache.py +32 -0
- agent_scaffold/cli.py +512 -0
- agent_scaffold/config.py +117 -0
- agent_scaffold/context.py +253 -0
- agent_scaffold/contract.py +141 -0
- agent_scaffold/discovery.py +112 -0
- agent_scaffold/generator.py +213 -0
- agent_scaffold/languages/__init__.py +0 -0
- agent_scaffold/languages/python.yaml +28 -0
- agent_scaffold/languages/typescript.yaml +25 -0
- agent_scaffold/prompts/__init__.py +0 -0
- agent_scaffold/prompts/repair.md +9 -0
- agent_scaffold/prompts/system.md +21 -0
- agent_scaffold/prompts/user_template.md +43 -0
- agent_scaffold/validator.py +133 -0
- agent_scaffold/writer.py +171 -0
- agent_scaffold_cli-0.1.1.dist-info/METADATA +147 -0
- agent_scaffold_cli-0.1.1.dist-info/RECORD +66 -0
- agent_scaffold_cli-0.1.1.dist-info/WHEEL +4 -0
- agent_scaffold_cli-0.1.1.dist-info/entry_points.txt +2 -0
- agent_scaffold_cli-0.1.1.dist-info/licenses/LICENSE +21 -0
|
@@ -0,0 +1,1679 @@
|
|
|
1
|
+
# Recipe: Customer Support Triage
|
|
2
|
+
|
|
3
|
+
**Status:** Blueprint (validated)
|
|
4
|
+
|
|
5
|
+
**Composes:**
|
|
6
|
+
|
|
7
|
+
- Pattern: [Routing + Tool Use](../patterns/routing-tool-use.md)
|
|
8
|
+
- Framework (Py): [Pydantic AI](../frameworks/pydantic-ai.md) (structured classification + specialist agents)
|
|
9
|
+
- Framework (TS): [Vercel AI SDK](../frameworks/vercel-ai-sdk.md) (`generateObject` for classification, `generateText` for specialists)
|
|
10
|
+
- Stack: [FastAPI](../stack/api-fastapi.md) / [Hono](../stack/api-hono.md), [Postgres](../stack/relational-postgres.md), [Redis](../stack/cache-redis.md), [Langfuse](../stack/tracing-langfuse.md)
|
|
11
|
+
- Cross-cutting: [Auth](../cross-cutting/auth-jwt.md), [Logging](../cross-cutting/logging-structured.md), [Observability](../cross-cutting/observability.md), [Rate limiting](../cross-cutting/rate-limiting.md)
|
|
12
|
+
|
|
13
|
+
## Load as Context
|
|
14
|
+
|
|
15
|
+
Feed these files to your AI coding assistant to build this agent:
|
|
16
|
+
|
|
17
|
+
**Core (always load):**
|
|
18
|
+
- `docs/recipes/customer-support-triage.md` — this blueprint
|
|
19
|
+
- `docs/patterns/routing-tool-use.md` — the routing + tool use pattern
|
|
20
|
+
- `docs/frameworks/pydantic-ai.md` (Python) or `docs/frameworks/vercel-ai-sdk.md` (TypeScript)
|
|
21
|
+
- `docs/stack/llm-claude.md` — LLM integration and model selection
|
|
22
|
+
|
|
23
|
+
**Stack (load for Tier 2 — API-ready):**
|
|
24
|
+
- `docs/stack/api-fastapi.md` or `docs/stack/api-hono.md` — API layer
|
|
25
|
+
- `docs/stack/relational-postgres.md` — conversation logging
|
|
26
|
+
- `docs/stack/cache-redis.md` — rate limiting backend
|
|
27
|
+
- `docs/stack/vector-qdrant.md` — knowledge base search (if using vector retrieval)
|
|
28
|
+
|
|
29
|
+
**Production concerns (load for Tier 3):**
|
|
30
|
+
- `docs/cross-cutting/auth-jwt.md` · `docs/cross-cutting/rate-limiting.md` · `docs/cross-cutting/logging-structured.md` · `docs/cross-cutting/observability.md` · `docs/cross-cutting/testing-strategy.md`
|
|
31
|
+
|
|
32
|
+
**Scaffolding:** `docs/reference/docker-templates.md` · `docs/reference/docker-compose-template.md`
|
|
33
|
+
|
|
34
|
+
## What it does
|
|
35
|
+
|
|
36
|
+
A customer support triage agent. Users send a message, the agent classifies the intent (billing, technical, account, or general), then routes to a specialized agent with the right tools for that intent. The billing specialist can look up Stripe data; the technical and account specialists can search a knowledge base.
|
|
37
|
+
|
|
38
|
+
This implements **single-hop routing** — classify once, route once. The classifier returns structured output (intent enum + confidence + reasoning), and a simple match/switch dispatches to the correct specialist.
|
|
39
|
+
|
|
40
|
+
## Architecture
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
┌─────────────┐
|
|
44
|
+
│ Client │
|
|
45
|
+
└──────┬──────┘
|
|
46
|
+
│
|
|
47
|
+
POST /triage
|
|
48
|
+
│
|
|
49
|
+
┌──────▼──────┐
|
|
50
|
+
│ FastAPI │ (or Hono)
|
|
51
|
+
│ + Auth │
|
|
52
|
+
│ + Rate │
|
|
53
|
+
└──────┬──────┘
|
|
54
|
+
│
|
|
55
|
+
┌──────▼──────┐
|
|
56
|
+
│ Classifier │ (cheap model)
|
|
57
|
+
│ → intent │
|
|
58
|
+
│ + confidence│
|
|
59
|
+
└──────┬──────┘
|
|
60
|
+
│
|
|
61
|
+
┌────────┬───────┼───────┬────────┐
|
|
62
|
+
v v v v │
|
|
63
|
+
[Billing] [Technical] [Account] [General]
|
|
64
|
+
(Stripe (KB search) (KB search) (no tools)
|
|
65
|
+
lookup)
|
|
66
|
+
│ │ │ │
|
|
67
|
+
└────────┴───────┴───────┘
|
|
68
|
+
│
|
|
69
|
+
┌──────▼──────┐
|
|
70
|
+
│ Response │
|
|
71
|
+
└─────────────┘
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### Triage flow
|
|
75
|
+
|
|
76
|
+
1. Client POSTs a message to `/triage`.
|
|
77
|
+
2. Classifier agent produces structured output: `{intent, confidence, reasoning}`.
|
|
78
|
+
3. Router dispatches to the specialist agent for that intent.
|
|
79
|
+
4. Specialist processes the message using its tools (Stripe lookup, KB search, or none).
|
|
80
|
+
5. Response includes the specialist's answer, the classification, and tool calls made.
|
|
81
|
+
|
|
82
|
+
## Key files
|
|
83
|
+
|
|
84
|
+
### Python track
|
|
85
|
+
|
|
86
|
+
| File | Role |
|
|
87
|
+
|------|------|
|
|
88
|
+
| `app/main.py` | FastAPI app with lifespan (DB init, logging config) |
|
|
89
|
+
| `app/settings.py` | Pydantic-settings config (classifier model, specialist model) |
|
|
90
|
+
| `app/agent/classifier.py` | Pydantic AI agent with `result_type=ClassificationResult` |
|
|
91
|
+
| `app/agent/specialists.py` | Factory for specialist agents — one per intent, each with its own tools |
|
|
92
|
+
| `app/api/triage.py` | `/triage` endpoint — classify, route, respond |
|
|
93
|
+
| `app/tools/stripe.py` | Stripe billing lookup tool |
|
|
94
|
+
| `app/tools/kb.py` | Knowledge base search tool |
|
|
95
|
+
| `app/models/schemas.py` | Pydantic schemas: `ClassificationResult`, `Intent` enum, request/response |
|
|
96
|
+
| `app/db/models.py` | SQLAlchemy models for conversation logging |
|
|
97
|
+
|
|
98
|
+
### TypeScript track
|
|
99
|
+
|
|
100
|
+
| File | Role |
|
|
101
|
+
|------|------|
|
|
102
|
+
| `src/index.ts` | Hono app entry point |
|
|
103
|
+
| `src/config.ts` | Zod-validated config from env |
|
|
104
|
+
| `src/agent/classifier.ts` | Vercel AI SDK `generateObject()` for intent classification |
|
|
105
|
+
| `src/agent/specialists.ts` | Specialist agents per intent with `generateText()` + tools |
|
|
106
|
+
| `src/api/triage.ts` | `/triage` route handler |
|
|
107
|
+
| `src/tools/stripe.ts` | Stripe billing lookup tool |
|
|
108
|
+
| `src/tools/kb.ts` | Knowledge base search tool |
|
|
109
|
+
| `src/schemas/index.ts` | Zod schemas for classification and request/response |
|
|
110
|
+
|
|
111
|
+
## Example interaction
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
curl -X POST http://localhost:8000/triage \
|
|
115
|
+
-H "Content-Type: application/json" \
|
|
116
|
+
-d '{"message": "I was charged twice for my subscription last month"}'
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Response:
|
|
120
|
+
|
|
121
|
+
```json
|
|
122
|
+
{
|
|
123
|
+
"classification": {
|
|
124
|
+
"intent": "billing",
|
|
125
|
+
"confidence": 0.95,
|
|
126
|
+
"reasoning": "Customer is reporting a duplicate charge on their subscription"
|
|
127
|
+
},
|
|
128
|
+
"response": "I understand you're concerned about a duplicate charge. Let me look into your billing history...",
|
|
129
|
+
"specialist": "billing",
|
|
130
|
+
"tool_calls": [
|
|
131
|
+
{"tool_name": "lookup_billing", "args": {"query": "duplicate charge subscription"}}
|
|
132
|
+
]
|
|
133
|
+
}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## Data Models
|
|
137
|
+
|
|
138
|
+
### Python (Pydantic)
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from enum import Enum
|
|
142
|
+
from pydantic import BaseModel, Field
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
class Intent(str, Enum):
|
|
146
|
+
billing = "billing"
|
|
147
|
+
technical = "technical"
|
|
148
|
+
account = "account"
|
|
149
|
+
general = "general"
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
class ClassificationResult(BaseModel):
|
|
153
|
+
intent: Intent
|
|
154
|
+
confidence: float = Field(..., ge=0.0, le=1.0)
|
|
155
|
+
reasoning: str
|
|
156
|
+
|
|
157
|
+
|
|
158
|
+
class TriageRequest(BaseModel):
|
|
159
|
+
message: str = Field(..., min_length=1)
|
|
160
|
+
user_id: str = Field(default="anonymous")
|
|
161
|
+
|
|
162
|
+
|
|
163
|
+
class TriageResponse(BaseModel):
|
|
164
|
+
conversation_id: str
|
|
165
|
+
intent: str
|
|
166
|
+
specialist_response: str
|
|
167
|
+
escalated: bool
|
|
168
|
+
trace_id: str
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### TypeScript (Zod)
|
|
172
|
+
|
|
173
|
+
```typescript
|
|
174
|
+
import { z } from "zod";
|
|
175
|
+
|
|
176
|
+
export const Intent = z.enum(["billing", "technical", "account", "general"]);
|
|
177
|
+
export const ClassificationResult = z.object({
|
|
178
|
+
intent: Intent,
|
|
179
|
+
confidence: z.number().min(0).max(1),
|
|
180
|
+
reasoning: z.string(),
|
|
181
|
+
});
|
|
182
|
+
export const TriageRequest = z.object({
|
|
183
|
+
message: z.string().min(1),
|
|
184
|
+
user_id: z.string().min(1),
|
|
185
|
+
});
|
|
186
|
+
export const TriageResponse = z.object({
|
|
187
|
+
conversation_id: z.string(),
|
|
188
|
+
intent: z.string(),
|
|
189
|
+
specialist_response: z.string(),
|
|
190
|
+
escalated: z.boolean(),
|
|
191
|
+
trace_id: z.string(),
|
|
192
|
+
});
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
## API Contract
|
|
196
|
+
|
|
197
|
+
### `POST /triage`
|
|
198
|
+
|
|
199
|
+
Classify a customer message and route to a specialist.
|
|
200
|
+
|
|
201
|
+
**Request:**
|
|
202
|
+
|
|
203
|
+
```json
|
|
204
|
+
{
|
|
205
|
+
"message": "I was charged twice for my subscription last month",
|
|
206
|
+
"user_id": "user-456"
|
|
207
|
+
}
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
**Response (200):**
|
|
211
|
+
|
|
212
|
+
```json
|
|
213
|
+
{
|
|
214
|
+
"conversation_id": "c1d2e3f4-...",
|
|
215
|
+
"intent": "billing",
|
|
216
|
+
"specialist_response": "I understand you're concerned about a duplicate charge. Let me look into your billing history...",
|
|
217
|
+
"escalated": false,
|
|
218
|
+
"trace_id": "a1b2c3d4-..."
|
|
219
|
+
}
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
**Escalation (200) — low confidence:**
|
|
223
|
+
|
|
224
|
+
```json
|
|
225
|
+
{
|
|
226
|
+
"conversation_id": "c1d2e3f4-...",
|
|
227
|
+
"intent": "general",
|
|
228
|
+
"specialist_response": "Escalated to human agent due to low classification confidence.",
|
|
229
|
+
"escalated": true,
|
|
230
|
+
"trace_id": "a1b2c3d4-..."
|
|
231
|
+
}
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
**Errors:**
|
|
235
|
+
|
|
236
|
+
| Status | Body | When |
|
|
237
|
+
|--------|------|------|
|
|
238
|
+
| 400 | `{"error": "Invalid request", "details": [...]}` | Empty message |
|
|
239
|
+
| 500 | `{"error": "Internal error"}` | LLM or service failure |
|
|
240
|
+
|
|
241
|
+
### `GET /health`
|
|
242
|
+
|
|
243
|
+
Returns `{"status": "ok"}`.
|
|
244
|
+
|
|
245
|
+
## Tool Specifications
|
|
246
|
+
|
|
247
|
+
### `lookup_billing`
|
|
248
|
+
|
|
249
|
+
| Field | Value |
|
|
250
|
+
|-------|-------|
|
|
251
|
+
| **Description** | Look up billing information for a customer using Stripe. Returns subscription details, recent charges, and payment method info. |
|
|
252
|
+
| **Parameter** | `query` (string, required) — Billing-related query (e.g., "duplicate charge", "refund", "subscription"). |
|
|
253
|
+
| **Return type** | `string` — Formatted billing summary with customer info and relevant details. |
|
|
254
|
+
|
|
255
|
+
### `search_knowledge_base`
|
|
256
|
+
|
|
257
|
+
| Field | Value |
|
|
258
|
+
|-------|-------|
|
|
259
|
+
| **Description** | Search the knowledge base for articles relevant to the customer's question. Used by technical and account specialists. |
|
|
260
|
+
| **Parameter** | `query` (string, required) — The search query. |
|
|
261
|
+
| **Return type** | `string` — Top matching KB articles formatted with titles and content, separated by dividers. |
|
|
262
|
+
|
|
263
|
+
## Prompt Specifications
|
|
264
|
+
|
|
265
|
+
### Classifier
|
|
266
|
+
|
|
267
|
+
```
|
|
268
|
+
You are a customer support intent classifier.
|
|
269
|
+
Given a customer message, classify it into exactly one of these intents:
|
|
270
|
+
- billing: payment issues, subscription changes, invoices, charges, refunds
|
|
271
|
+
- technical: bugs, errors, API issues, integration problems, performance
|
|
272
|
+
- account: password resets, profile updates, access issues, account settings
|
|
273
|
+
- general: everything else, general questions, feedback, feature requests
|
|
274
|
+
|
|
275
|
+
Return the intent, your confidence (0.0 to 1.0), and brief reasoning.
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
**Design rationale:** The classifier uses structured output (`result_type=ClassificationResult` / `generateObject`) to guarantee a valid intent enum. The confidence score enables downstream gating — below `ESCALATION_THRESHOLD` (default 0.7), the request is escalated to a human.
|
|
279
|
+
|
|
280
|
+
### Specialist prompts
|
|
281
|
+
|
|
282
|
+
Each specialist has a focused system prompt:
|
|
283
|
+
|
|
284
|
+
- **Billing:** `"You are a billing support specialist. Help customers with payment issues, subscription changes, invoices, and charges. You have access to the Stripe tool to look up billing information."`
|
|
285
|
+
- **Technical:** `"You are a technical support specialist. Help customers with bugs, errors, API issues, and integration problems. You have access to a knowledge base search tool."`
|
|
286
|
+
- **Account:** `"You are an account support specialist. Help customers with password resets, profile updates, and account settings. You have access to a knowledge base search tool."`
|
|
287
|
+
- **General:** `"You are a general support specialist. Help customers with general questions, feedback, and feature requests."`
|
|
288
|
+
|
|
289
|
+
**Design rationale:** Each specialist only sees the tools relevant to its domain. The billing agent can call Stripe but not search the KB. This prevents tool misuse and keeps each specialist focused.
|
|
290
|
+
|
|
291
|
+
## Implementation Roadmap
|
|
292
|
+
|
|
293
|
+
| Step | Task | Key deliverables |
|
|
294
|
+
|------|------|-----------------|
|
|
295
|
+
| 1 | **Project scaffolding** | FastAPI/Hono app with `/health`, settings, structured logging |
|
|
296
|
+
| 2 | **Data models** | Pydantic + Zod schemas for Intent, ClassificationResult, request/response |
|
|
297
|
+
| 3 | **Database models** | Conversation and Message tables for logging |
|
|
298
|
+
| 4 | **Classifier agent** | Pydantic AI agent with `result_type=ClassificationResult` |
|
|
299
|
+
| 5 | **Tool implementations** | `lookup_billing` (mock Stripe), `search_knowledge_base` (mock KB) |
|
|
300
|
+
| 6 | **Specialist agents** | 4 specialists, each with isolated tool set |
|
|
301
|
+
| 7 | **Triage router** | Classify → check confidence → route to specialist or escalate |
|
|
302
|
+
| 8 | **API endpoint** | `POST /triage` with conversation logging |
|
|
303
|
+
| 9 | **Cross-cutting** | JWT auth, rate limiting, Langfuse tracing |
|
|
304
|
+
| 10 | **Unit tests** | Classification schema, routing logic, tool mocks |
|
|
305
|
+
| 11 | **Integration + eval** | End-to-end triage with real LLM, promptfoo security scan |
|
|
306
|
+
|
|
307
|
+
## Environment & Deployment
|
|
308
|
+
|
|
309
|
+
### Environment variables
|
|
310
|
+
|
|
311
|
+
| Variable | Required | Default | Description |
|
|
312
|
+
|----------|----------|---------|-------------|
|
|
313
|
+
| `ANTHROPIC_API_KEY` | Yes | — | Anthropic API key |
|
|
314
|
+
| `CLASSIFIER_MODEL` | No | `claude-haiku-4-5-20251001` | Model for classification (cheaper) |
|
|
315
|
+
| `SPECIALIST_MODEL` | No | `claude-sonnet-4-6-20250514` | Model for specialist responses |
|
|
316
|
+
| `ESCALATION_THRESHOLD` | No | `0.7` | Min confidence to auto-route (below = escalate) |
|
|
317
|
+
| `DATABASE_URL` | No | `postgresql+asyncpg://agent:agent@localhost:5432/agent_db` | Postgres connection |
|
|
318
|
+
| `REDIS_URL` | No | `redis://localhost:6379` | Redis for rate limiting |
|
|
319
|
+
| `QDRANT_URL` | No | `http://localhost:6333` | Qdrant for KB search |
|
|
320
|
+
| `LANGFUSE_PUBLIC_KEY` | No | `pk-lf-local` | Langfuse public key |
|
|
321
|
+
| `LANGFUSE_SECRET_KEY` | No | `sk-lf-local` | Langfuse secret key |
|
|
322
|
+
| `LANGFUSE_HOST` | No | `http://localhost:3000` | Langfuse server URL |
|
|
323
|
+
| `JWT_SECRET` | No | `change-me-in-production` | JWT signing secret |
|
|
324
|
+
| `APP_ENV` | No | `development` | Environment name |
|
|
325
|
+
| `LOG_LEVEL` | No | `INFO` | Log level |
|
|
326
|
+
|
|
327
|
+
### Docker Compose
|
|
328
|
+
|
|
329
|
+
See [Docker Compose template](../reference/docker-compose-template.md) for base infrastructure. This agent needs: Postgres, Redis, Qdrant, Langfuse.
|
|
330
|
+
|
|
331
|
+
### Infrastructure dependencies
|
|
332
|
+
|
|
333
|
+
| Component | Required? | Why |
|
|
334
|
+
|-----------|-----------|-----|
|
|
335
|
+
| Postgres | Yes | Conversation logging and triage history |
|
|
336
|
+
| Redis | Yes | Rate limiting backend |
|
|
337
|
+
| Qdrant | Optional | Knowledge base vector search (can start with in-memory keyword search) |
|
|
338
|
+
| Langfuse | Recommended | LLM + tool call tracing (skip for local dev) |
|
|
339
|
+
|
|
340
|
+
## Test Strategy
|
|
341
|
+
|
|
342
|
+
### Unit tests
|
|
343
|
+
|
|
344
|
+
```python
|
|
345
|
+
def test_classification_returns_valid_intent(mock_llm_client):
|
|
346
|
+
"""Classifier always returns a valid Intent enum value."""
|
|
347
|
+
result = await classify_intent("I need a refund")
|
|
348
|
+
assert result.intent in ["billing", "technical", "account", "general"]
|
|
349
|
+
assert 0 <= result.confidence <= 1
|
|
350
|
+
|
|
351
|
+
def test_low_confidence_triggers_escalation():
|
|
352
|
+
"""Below ESCALATION_THRESHOLD, request is escalated."""
|
|
353
|
+
# Mock classifier to return confidence=0.3
|
|
354
|
+
# Assert response.escalated == True
|
|
355
|
+
|
|
356
|
+
def test_billing_specialist_uses_stripe_tool(mock_llm_client):
|
|
357
|
+
"""Billing specialist calls lookup_billing, not search_knowledge_base."""
|
|
358
|
+
result = await run_specialist("billing", "I was charged twice")
|
|
359
|
+
tool_names = [tc.tool_name for tc in result.tool_calls]
|
|
360
|
+
assert "lookup_billing" in tool_names
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
### Eval assertions
|
|
364
|
+
|
|
365
|
+
- Classification accuracy ≥ 90% on the 51-example dataset
|
|
366
|
+
- Billing queries always trigger `lookup_billing` tool
|
|
367
|
+
- Technical queries always trigger `search_knowledge_base` tool
|
|
368
|
+
- No prompt injection bypasses the classifier (promptfoo red-team)
|
|
369
|
+
|
|
370
|
+
## Eval Dataset
|
|
371
|
+
|
|
372
|
+
See the inline `eval/dataset.jsonl` (51 examples) in the Reference Implementation section below. Covers all 4 intents with ~12 examples each.
|
|
373
|
+
|
|
374
|
+
## Design decisions
|
|
375
|
+
|
|
376
|
+
- **Separate classifier and specialist models:** The classifier can use a cheaper/faster model since it only needs to produce structured classification. Specialists use the full model for nuanced responses.
|
|
377
|
+
- **Pydantic AI `result_type` for classification:** Structured output validation ensures the classifier always returns a valid intent enum, not a free-text guess.
|
|
378
|
+
- **Isolated specialist agents:** Each specialist has only its own tools. The billing agent can't accidentally search the KB, and the technical agent can't call Stripe. This prevents tool misuse.
|
|
379
|
+
- **Confidence score:** The classifier returns a confidence score, enabling future gating (e.g., fall back to general handler below 0.7 confidence).
|
|
380
|
+
|
|
381
|
+
## Reference Implementation
|
|
382
|
+
|
|
383
|
+
### Python
|
|
384
|
+
|
|
385
|
+
<details>
|
|
386
|
+
<summary><code>app/main.py</code></summary>
|
|
387
|
+
|
|
388
|
+
```python
|
|
389
|
+
"""FastAPI entrypoint for customer-support-triage."""
|
|
390
|
+
|
|
391
|
+
from contextlib import asynccontextmanager
|
|
392
|
+
|
|
393
|
+
import structlog
|
|
394
|
+
from fastapi import FastAPI
|
|
395
|
+
|
|
396
|
+
from app.api.triage import router as triage_router
|
|
397
|
+
from app.db.models import Base
|
|
398
|
+
from app.db.session import engine
|
|
399
|
+
from app.settings import settings
|
|
400
|
+
|
|
401
|
+
|
|
402
|
+
@asynccontextmanager
|
|
403
|
+
async def lifespan(app: FastAPI):
|
|
404
|
+
from agent_common.logs import configure
|
|
405
|
+
|
|
406
|
+
configure(settings.app_name, env=settings.app_env, log_level=settings.log_level)
|
|
407
|
+
|
|
408
|
+
logger = structlog.get_logger()
|
|
409
|
+
logger.info("starting", app=settings.app_name)
|
|
410
|
+
|
|
411
|
+
# Create tables (use Alembic in production)
|
|
412
|
+
async with engine.begin() as conn:
|
|
413
|
+
await conn.run_sync(Base.metadata.create_all)
|
|
414
|
+
|
|
415
|
+
yield
|
|
416
|
+
|
|
417
|
+
await engine.dispose()
|
|
418
|
+
|
|
419
|
+
|
|
420
|
+
app = FastAPI(
|
|
421
|
+
title=settings.app_name,
|
|
422
|
+
lifespan=lifespan,
|
|
423
|
+
)
|
|
424
|
+
|
|
425
|
+
app.include_router(triage_router)
|
|
426
|
+
|
|
427
|
+
|
|
428
|
+
@app.get("/health")
|
|
429
|
+
async def health():
|
|
430
|
+
return {"status": "ok"}
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
</details>
|
|
434
|
+
|
|
435
|
+
<details>
|
|
436
|
+
<summary><code>app/settings.py</code></summary>
|
|
437
|
+
|
|
438
|
+
```python
|
|
439
|
+
"""Application settings via pydantic-settings."""
|
|
440
|
+
|
|
441
|
+
from pydantic_settings import BaseSettings
|
|
442
|
+
|
|
443
|
+
|
|
444
|
+
class Settings(BaseSettings):
|
|
445
|
+
app_name: str = "customer-support-triage"
|
|
446
|
+
app_env: str = "development"
|
|
447
|
+
log_level: str = "INFO"
|
|
448
|
+
|
|
449
|
+
# LLM
|
|
450
|
+
anthropic_api_key: str = ""
|
|
451
|
+
classifier_model: str = "claude-haiku-4-5-20251001"
|
|
452
|
+
specialist_model: str = "claude-sonnet-4-6-20250514"
|
|
453
|
+
|
|
454
|
+
# Routing
|
|
455
|
+
escalation_threshold: float = 0.7
|
|
456
|
+
|
|
457
|
+
# Database
|
|
458
|
+
database_url: str = "postgresql+asyncpg://agent:agent@localhost:5432/agent_db"
|
|
459
|
+
|
|
460
|
+
# Redis
|
|
461
|
+
redis_url: str = "redis://localhost:6379"
|
|
462
|
+
|
|
463
|
+
# Qdrant
|
|
464
|
+
qdrant_url: str = "http://localhost:6333"
|
|
465
|
+
qdrant_collection: str = "support_kb"
|
|
466
|
+
|
|
467
|
+
# Auth
|
|
468
|
+
jwt_secret: str = "change-me-in-production"
|
|
469
|
+
|
|
470
|
+
# Langfuse
|
|
471
|
+
langfuse_public_key: str = "pk-lf-local"
|
|
472
|
+
langfuse_secret_key: str = "sk-lf-local"
|
|
473
|
+
langfuse_host: str = "http://localhost:3000"
|
|
474
|
+
|
|
475
|
+
model_config = {"env_file": ".env", "extra": "ignore"}
|
|
476
|
+
|
|
477
|
+
|
|
478
|
+
settings = Settings()
|
|
479
|
+
```
|
|
480
|
+
|
|
481
|
+
</details>
|
|
482
|
+
|
|
483
|
+
<details>
|
|
484
|
+
<summary><code>app/models/schemas.py</code></summary>
|
|
485
|
+
|
|
486
|
+
```python
|
|
487
|
+
"""Request/response schemas and domain types."""
|
|
488
|
+
|
|
489
|
+
from enum import Enum
|
|
490
|
+
|
|
491
|
+
from pydantic import BaseModel
|
|
492
|
+
|
|
493
|
+
|
|
494
|
+
class Intent(str, Enum):
|
|
495
|
+
BILLING = "billing"
|
|
496
|
+
TECHNICAL = "technical"
|
|
497
|
+
ACCOUNT = "account"
|
|
498
|
+
GENERAL = "general"
|
|
499
|
+
|
|
500
|
+
|
|
501
|
+
class ClassificationResult(BaseModel):
|
|
502
|
+
intent: Intent
|
|
503
|
+
confidence: float
|
|
504
|
+
reasoning: str
|
|
505
|
+
|
|
506
|
+
|
|
507
|
+
class TriageRequest(BaseModel):
|
|
508
|
+
message: str
|
|
509
|
+
user_id: str
|
|
510
|
+
|
|
511
|
+
|
|
512
|
+
class TriageResponse(BaseModel):
|
|
513
|
+
conversation_id: str
|
|
514
|
+
intent: str
|
|
515
|
+
specialist_response: str
|
|
516
|
+
escalated: bool
|
|
517
|
+
trace_id: str
|
|
518
|
+
|
|
519
|
+
|
|
520
|
+
class ConversationOut(BaseModel):
|
|
521
|
+
id: str
|
|
522
|
+
user_id: str
|
|
523
|
+
created_at: str
|
|
524
|
+
resolved_at: str | None
|
|
525
|
+
escalated: bool
|
|
526
|
+
messages: list["MessageOut"]
|
|
527
|
+
|
|
528
|
+
|
|
529
|
+
class MessageOut(BaseModel):
|
|
530
|
+
id: str
|
|
531
|
+
role: str
|
|
532
|
+
content: str
|
|
533
|
+
intent: str | None
|
|
534
|
+
tool_calls: list[dict] | None
|
|
535
|
+
created_at: str
|
|
536
|
+
```
|
|
537
|
+
|
|
538
|
+
</details>
|
|
539
|
+
|
|
540
|
+
<details>
|
|
541
|
+
<summary><code>app/agent/classifier.py</code></summary>
|
|
542
|
+
|
|
543
|
+
```python
|
|
544
|
+
"""Intent classifier using Pydantic AI with structured output."""
|
|
545
|
+
|
|
546
|
+
from pydantic_ai import Agent
|
|
547
|
+
|
|
548
|
+
from app.models.schemas import ClassificationResult
|
|
549
|
+
from app.settings import settings
|
|
550
|
+
|
|
551
|
+
CLASSIFIER_SYSTEM_PROMPT = """You are a customer support intent classifier.
|
|
552
|
+
Given a customer message, classify it into exactly one of these intents:
|
|
553
|
+
- billing: payment issues, subscription changes, invoices, charges, refunds
|
|
554
|
+
- technical: bugs, errors, API issues, integration problems, performance
|
|
555
|
+
- account: password resets, profile updates, access issues, account settings
|
|
556
|
+
- general: everything else, general questions, feedback, feature requests
|
|
557
|
+
|
|
558
|
+
Return the intent, your confidence (0.0 to 1.0), and brief reasoning."""
|
|
559
|
+
|
|
560
|
+
_classifier_agent: Agent | None = None
|
|
561
|
+
|
|
562
|
+
|
|
563
|
+
def _get_classifier() -> Agent:
|
|
564
|
+
global _classifier_agent
|
|
565
|
+
if _classifier_agent is None:
|
|
566
|
+
_classifier_agent = Agent(
|
|
567
|
+
f"anthropic:{settings.classifier_model}",
|
|
568
|
+
result_type=ClassificationResult,
|
|
569
|
+
system_prompt=CLASSIFIER_SYSTEM_PROMPT,
|
|
570
|
+
)
|
|
571
|
+
return _classifier_agent
|
|
572
|
+
|
|
573
|
+
|
|
574
|
+
async def classify_intent(message: str) -> ClassificationResult:
|
|
575
|
+
"""Classify a customer message into an intent category."""
|
|
576
|
+
agent = _get_classifier()
|
|
577
|
+
result = await agent.run(message)
|
|
578
|
+
return result.data
|
|
579
|
+
```
|
|
580
|
+
|
|
581
|
+
</details>
|
|
582
|
+
|
|
583
|
+
<details>
|
|
584
|
+
<summary><code>app/agent/specialists.py</code></summary>
|
|
585
|
+
|
|
586
|
+
```python
|
|
587
|
+
"""Specialist agents for each intent category."""
|
|
588
|
+
|
|
589
|
+
from pydantic_ai import Agent
|
|
590
|
+
|
|
591
|
+
from app.models.schemas import Intent
|
|
592
|
+
from app.settings import settings
|
|
593
|
+
from app.tools.kb import kb_search
|
|
594
|
+
from app.tools.stripe import stripe_lookup
|
|
595
|
+
|
|
596
|
+
SPECIALIST_PROMPTS = {
|
|
597
|
+
Intent.BILLING: """You are a billing support specialist. Help customers with payment issues,
|
|
598
|
+
subscription changes, invoices, and charges. You have access to the Stripe tool to look up
|
|
599
|
+
billing information. Be helpful, concise, and professional.""",
|
|
600
|
+
Intent.TECHNICAL: """You are a technical support specialist. Help customers with bugs, errors,
|
|
601
|
+
API issues, and integration problems. You have access to a knowledge base search tool.
|
|
602
|
+
Provide clear, actionable guidance.""",
|
|
603
|
+
Intent.ACCOUNT: """You are an account support specialist. Help customers with password resets,
|
|
604
|
+
profile updates, and account settings. You have access to a knowledge base search tool.
|
|
605
|
+
Guide them step by step.""",
|
|
606
|
+
Intent.GENERAL: """You are a general support specialist. Help customers with general questions,
|
|
607
|
+
feedback, and feature requests. Be friendly and helpful.""",
|
|
608
|
+
}
|
|
609
|
+
|
|
610
|
+
|
|
611
|
+
def _make_billing_agent() -> Agent:
|
|
612
|
+
agent = Agent(
|
|
613
|
+
f"anthropic:{settings.specialist_model}",
|
|
614
|
+
system_prompt=SPECIALIST_PROMPTS[Intent.BILLING],
|
|
615
|
+
)
|
|
616
|
+
|
|
617
|
+
@agent.tool_plain
|
|
618
|
+
async def lookup_billing(query: str) -> str:
|
|
619
|
+
"""Look up billing information for a customer using Stripe."""
|
|
620
|
+
return await stripe_lookup(query)
|
|
621
|
+
|
|
622
|
+
return agent
|
|
623
|
+
|
|
624
|
+
|
|
625
|
+
def _make_technical_agent() -> Agent:
|
|
626
|
+
agent = Agent(
|
|
627
|
+
f"anthropic:{settings.specialist_model}",
|
|
628
|
+
system_prompt=SPECIALIST_PROMPTS[Intent.TECHNICAL],
|
|
629
|
+
)
|
|
630
|
+
|
|
631
|
+
@agent.tool_plain
|
|
632
|
+
async def search_knowledge_base(query: str) -> str:
|
|
633
|
+
"""Search the technical knowledge base for relevant articles."""
|
|
634
|
+
return await kb_search(query)
|
|
635
|
+
|
|
636
|
+
return agent
|
|
637
|
+
|
|
638
|
+
|
|
639
|
+
def _make_account_agent() -> Agent:
|
|
640
|
+
agent = Agent(
|
|
641
|
+
f"anthropic:{settings.specialist_model}",
|
|
642
|
+
system_prompt=SPECIALIST_PROMPTS[Intent.ACCOUNT],
|
|
643
|
+
)
|
|
644
|
+
|
|
645
|
+
@agent.tool_plain
|
|
646
|
+
async def search_knowledge_base(query: str) -> str:
|
|
647
|
+
"""Search the account knowledge base for relevant articles."""
|
|
648
|
+
return await kb_search(query)
|
|
649
|
+
|
|
650
|
+
return agent
|
|
651
|
+
|
|
652
|
+
|
|
653
|
+
def _make_general_agent() -> Agent:
|
|
654
|
+
return Agent(
|
|
655
|
+
f"anthropic:{settings.specialist_model}",
|
|
656
|
+
system_prompt=SPECIALIST_PROMPTS[Intent.GENERAL],
|
|
657
|
+
)
|
|
658
|
+
|
|
659
|
+
|
|
660
|
+
_agents: dict[Intent, Agent] = {}
|
|
661
|
+
|
|
662
|
+
|
|
663
|
+
def get_specialist(intent: Intent) -> Agent:
|
|
664
|
+
"""Get the specialist agent for a given intent (lazy-initialized)."""
|
|
665
|
+
if intent not in _agents:
|
|
666
|
+
factories = {
|
|
667
|
+
Intent.BILLING: _make_billing_agent,
|
|
668
|
+
Intent.TECHNICAL: _make_technical_agent,
|
|
669
|
+
Intent.ACCOUNT: _make_account_agent,
|
|
670
|
+
Intent.GENERAL: _make_general_agent,
|
|
671
|
+
}
|
|
672
|
+
_agents[intent] = factories[intent]()
|
|
673
|
+
return _agents[intent]
|
|
674
|
+
|
|
675
|
+
|
|
676
|
+
async def run_specialist(intent: Intent, message: str) -> tuple[str, list[dict]]:
|
|
677
|
+
"""Run the specialist agent and return (response_text, tool_calls)."""
|
|
678
|
+
agent = get_specialist(intent)
|
|
679
|
+
result = await agent.run(message)
|
|
680
|
+
tool_calls = [
|
|
681
|
+
{"tool_name": call.tool_name, "args": call.args}
|
|
682
|
+
for call in result.all_messages()
|
|
683
|
+
if hasattr(call, "tool_name")
|
|
684
|
+
]
|
|
685
|
+
return result.data, tool_calls
|
|
686
|
+
```
|
|
687
|
+
|
|
688
|
+
</details>
|
|
689
|
+
|
|
690
|
+
<details>
|
|
691
|
+
<summary><code>app/api/triage.py</code></summary>
|
|
692
|
+
|
|
693
|
+
```python
|
|
694
|
+
"""Triage and conversation route handlers."""
|
|
695
|
+
|
|
696
|
+
import uuid
|
|
697
|
+
from datetime import UTC, datetime
|
|
698
|
+
|
|
699
|
+
import structlog
|
|
700
|
+
from fastapi import APIRouter, Depends, HTTPException
|
|
701
|
+
from sqlalchemy import select
|
|
702
|
+
from sqlalchemy.ext.asyncio import AsyncSession
|
|
703
|
+
from sqlalchemy.orm import selectinload
|
|
704
|
+
|
|
705
|
+
from app.agent.classifier import classify_intent
|
|
706
|
+
from app.agent.specialists import run_specialist
|
|
707
|
+
from app.db.models import Conversation, Message
|
|
708
|
+
from app.db.session import get_session
|
|
709
|
+
from app.models.schemas import (
|
|
710
|
+
ConversationOut,
|
|
711
|
+
MessageOut,
|
|
712
|
+
TriageRequest,
|
|
713
|
+
TriageResponse,
|
|
714
|
+
)
|
|
715
|
+
from app.settings import settings
|
|
716
|
+
|
|
717
|
+
logger = structlog.get_logger()
|
|
718
|
+
|
|
719
|
+
router = APIRouter()
|
|
720
|
+
|
|
721
|
+
|
|
722
|
+
@router.post("/triage", response_model=TriageResponse)
|
|
723
|
+
async def triage(
|
|
724
|
+
request: TriageRequest,
|
|
725
|
+
session: AsyncSession = Depends(get_session),
|
|
726
|
+
):
|
|
727
|
+
"""Classify intent, route to specialist, return resolution."""
|
|
728
|
+
trace_id = str(uuid.uuid4())
|
|
729
|
+
log = logger.bind(trace_id=trace_id, user_id=request.user_id)
|
|
730
|
+
|
|
731
|
+
# Create conversation
|
|
732
|
+
conversation = Conversation(user_id=request.user_id)
|
|
733
|
+
session.add(conversation)
|
|
734
|
+
|
|
735
|
+
# Store user message
|
|
736
|
+
user_msg = Message(
|
|
737
|
+
conversation_id=conversation.id,
|
|
738
|
+
role="user",
|
|
739
|
+
content=request.message,
|
|
740
|
+
)
|
|
741
|
+
session.add(user_msg)
|
|
742
|
+
|
|
743
|
+
# Classify intent
|
|
744
|
+
log.info("classifying_intent")
|
|
745
|
+
classification = await classify_intent(request.message)
|
|
746
|
+
log.info("intent_classified", intent=classification.intent, confidence=classification.confidence)
|
|
747
|
+
|
|
748
|
+
user_msg.intent = classification.intent.value
|
|
749
|
+
|
|
750
|
+
# Check escalation threshold
|
|
751
|
+
if classification.confidence < settings.escalation_threshold:
|
|
752
|
+
log.info("escalating", reason="low_confidence", confidence=classification.confidence)
|
|
753
|
+
conversation.escalated = True
|
|
754
|
+
assistant_msg = Message(
|
|
755
|
+
conversation_id=conversation.id,
|
|
756
|
+
role="assistant",
|
|
757
|
+
content=(
|
|
758
|
+
"I'm not fully confident in my assessment. "
|
|
759
|
+
"Let me connect you with a human agent who can better assist you."
|
|
760
|
+
),
|
|
761
|
+
intent=classification.intent.value,
|
|
762
|
+
)
|
|
763
|
+
session.add(assistant_msg)
|
|
764
|
+
await session.commit()
|
|
765
|
+
|
|
766
|
+
return TriageResponse(
|
|
767
|
+
conversation_id=conversation.id,
|
|
768
|
+
intent=classification.intent.value,
|
|
769
|
+
specialist_response="Escalated to human agent due to low classification confidence.",
|
|
770
|
+
escalated=True,
|
|
771
|
+
trace_id=trace_id,
|
|
772
|
+
)
|
|
773
|
+
|
|
774
|
+
# Route to specialist
|
|
775
|
+
log.info("routing_to_specialist", intent=classification.intent)
|
|
776
|
+
response_text, tool_calls = await run_specialist(classification.intent, request.message)
|
|
777
|
+
log.info("specialist_responded", tool_calls_count=len(tool_calls))
|
|
778
|
+
|
|
779
|
+
# Store assistant response
|
|
780
|
+
assistant_msg = Message(
|
|
781
|
+
conversation_id=conversation.id,
|
|
782
|
+
role="assistant",
|
|
783
|
+
content=response_text,
|
|
784
|
+
intent=classification.intent.value,
|
|
785
|
+
tool_calls_json=tool_calls if tool_calls else None,
|
|
786
|
+
)
|
|
787
|
+
session.add(assistant_msg)
|
|
788
|
+
|
|
789
|
+
conversation.resolved_at = datetime.now(UTC)
|
|
790
|
+
await session.commit()
|
|
791
|
+
|
|
792
|
+
return TriageResponse(
|
|
793
|
+
conversation_id=conversation.id,
|
|
794
|
+
intent=classification.intent.value,
|
|
795
|
+
specialist_response=response_text,
|
|
796
|
+
escalated=False,
|
|
797
|
+
trace_id=trace_id,
|
|
798
|
+
)
|
|
799
|
+
|
|
800
|
+
|
|
801
|
+
@router.get("/conversations/{conversation_id}", response_model=ConversationOut)
|
|
802
|
+
async def get_conversation(
|
|
803
|
+
conversation_id: str,
|
|
804
|
+
session: AsyncSession = Depends(get_session),
|
|
805
|
+
):
|
|
806
|
+
"""Fetch a conversation with its messages."""
|
|
807
|
+
result = await session.execute(
|
|
808
|
+
select(Conversation)
|
|
809
|
+
.where(Conversation.id == conversation_id)
|
|
810
|
+
.options(selectinload(Conversation.messages))
|
|
811
|
+
)
|
|
812
|
+
conversation = result.scalar_one_or_none()
|
|
813
|
+
|
|
814
|
+
if not conversation:
|
|
815
|
+
raise HTTPException(status_code=404, detail="Conversation not found")
|
|
816
|
+
|
|
817
|
+
return ConversationOut(
|
|
818
|
+
id=conversation.id,
|
|
819
|
+
user_id=conversation.user_id,
|
|
820
|
+
created_at=conversation.created_at.isoformat(),
|
|
821
|
+
resolved_at=conversation.resolved_at.isoformat() if conversation.resolved_at else None,
|
|
822
|
+
escalated=conversation.escalated,
|
|
823
|
+
messages=[
|
|
824
|
+
MessageOut(
|
|
825
|
+
id=m.id,
|
|
826
|
+
role=m.role,
|
|
827
|
+
content=m.content,
|
|
828
|
+
intent=m.intent,
|
|
829
|
+
tool_calls=m.tool_calls_json,
|
|
830
|
+
created_at=m.created_at.isoformat(),
|
|
831
|
+
)
|
|
832
|
+
for m in conversation.messages
|
|
833
|
+
],
|
|
834
|
+
)
|
|
835
|
+
```
|
|
836
|
+
|
|
837
|
+
</details>
|
|
838
|
+
|
|
839
|
+
<details>
|
|
840
|
+
<summary><code>app/tools/kb.py</code></summary>
|
|
841
|
+
|
|
842
|
+
```python
|
|
843
|
+
"""Knowledge base search tool using Qdrant.
|
|
844
|
+
|
|
845
|
+
Falls back to mock data when Qdrant is unavailable, making the prototype
|
|
846
|
+
runnable without the full infra stack for development and testing.
|
|
847
|
+
"""
|
|
848
|
+
|
|
849
|
+
import structlog
|
|
850
|
+
|
|
851
|
+
logger = structlog.get_logger()
|
|
852
|
+
|
|
853
|
+
# Mock KB articles for when Qdrant is unavailable
|
|
854
|
+
_MOCK_KB = [
|
|
855
|
+
{
|
|
856
|
+
"id": "kb-001",
|
|
857
|
+
"title": "How to reset your password",
|
|
858
|
+
"content": (
|
|
859
|
+
"Go to Settings > Security > Reset Password. Enter your current"
|
|
860
|
+
" password, then your new password twice. Click Save. You'll"
|
|
861
|
+
" receive a confirmation email."
|
|
862
|
+
),
|
|
863
|
+
"category": "account",
|
|
864
|
+
},
|
|
865
|
+
{
|
|
866
|
+
"id": "kb-002",
|
|
867
|
+
"title": "API rate limits and error codes",
|
|
868
|
+
"content": (
|
|
869
|
+
"Rate limits: 100 requests/minute for free tier, 1000/minute"
|
|
870
|
+
" for Pro. When exceeded, you'll get a 429 status code."
|
|
871
|
+
" Implement exponential backoff. Common errors: 400 (bad"
|
|
872
|
+
" request), 401 (unauthorized), 500 (server error —"
|
|
873
|
+
" contact support)."
|
|
874
|
+
),
|
|
875
|
+
"category": "technical",
|
|
876
|
+
},
|
|
877
|
+
{
|
|
878
|
+
"id": "kb-003",
|
|
879
|
+
"title": "Updating your billing information",
|
|
880
|
+
"content": (
|
|
881
|
+
"Navigate to Account > Billing > Payment Methods. Click"
|
|
882
|
+
" 'Update' next to your current method. Enter new card"
|
|
883
|
+
" details. Changes take effect on your next billing cycle."
|
|
884
|
+
),
|
|
885
|
+
"category": "billing",
|
|
886
|
+
},
|
|
887
|
+
{
|
|
888
|
+
"id": "kb-004",
|
|
889
|
+
"title": "Troubleshooting large payload errors",
|
|
890
|
+
"content": (
|
|
891
|
+
"Maximum payload size is 10MB for the standard API. For"
|
|
892
|
+
" larger payloads, use the streaming endpoint or split your"
|
|
893
|
+
" request. If you're getting 500 errors, check that your"
|
|
894
|
+
" Content-Type header is set correctly and the JSON is valid."
|
|
895
|
+
),
|
|
896
|
+
"category": "technical",
|
|
897
|
+
},
|
|
898
|
+
{
|
|
899
|
+
"id": "kb-005",
|
|
900
|
+
"title": "Two-factor authentication setup",
|
|
901
|
+
"content": (
|
|
902
|
+
"Go to Settings > Security > 2FA. Choose your method:"
|
|
903
|
+
" authenticator app (recommended) or SMS. Scan the QR code"
|
|
904
|
+
" with your authenticator app. Enter the 6-digit code to"
|
|
905
|
+
" verify. Save your backup codes in a secure location."
|
|
906
|
+
),
|
|
907
|
+
"category": "account",
|
|
908
|
+
},
|
|
909
|
+
{
|
|
910
|
+
"id": "kb-006",
|
|
911
|
+
"title": "Integration webhook configuration",
|
|
912
|
+
"content": (
|
|
913
|
+
"Set up webhooks at Settings > Integrations > Webhooks. Add"
|
|
914
|
+
" your endpoint URL, select events to subscribe to, and save."
|
|
915
|
+
" We'll send a test ping. Webhook payloads are signed with"
|
|
916
|
+
" your webhook secret for verification."
|
|
917
|
+
),
|
|
918
|
+
"category": "technical",
|
|
919
|
+
},
|
|
920
|
+
]
|
|
921
|
+
|
|
922
|
+
|
|
923
|
+
async def kb_search(query: str, top_k: int = 3) -> str:
|
|
924
|
+
"""Search the knowledge base. Falls back to keyword matching if Qdrant is unavailable."""
|
|
925
|
+
try:
|
|
926
|
+
return await _qdrant_search(query, top_k)
|
|
927
|
+
except Exception:
|
|
928
|
+
logger.info("qdrant_unavailable_using_mock", query=query)
|
|
929
|
+
return _mock_search(query, top_k)
|
|
930
|
+
|
|
931
|
+
|
|
932
|
+
async def _qdrant_search(query: str, top_k: int) -> str:
|
|
933
|
+
"""Search using Qdrant vector DB."""
|
|
934
|
+
from qdrant_client import AsyncQdrantClient
|
|
935
|
+
|
|
936
|
+
from app.settings import settings
|
|
937
|
+
|
|
938
|
+
client = AsyncQdrantClient(url=settings.qdrant_url)
|
|
939
|
+
|
|
940
|
+
collections = await client.get_collections()
|
|
941
|
+
collection_names = [c.name for c in collections.collections]
|
|
942
|
+
if settings.qdrant_collection not in collection_names:
|
|
943
|
+
raise RuntimeError("Collection not found")
|
|
944
|
+
|
|
945
|
+
results = await client.query(
|
|
946
|
+
collection_name=settings.qdrant_collection,
|
|
947
|
+
query_text=query,
|
|
948
|
+
limit=top_k,
|
|
949
|
+
)
|
|
950
|
+
|
|
951
|
+
if not results:
|
|
952
|
+
return "No relevant articles found in the knowledge base."
|
|
953
|
+
|
|
954
|
+
articles = []
|
|
955
|
+
for point in results:
|
|
956
|
+
payload = point.metadata
|
|
957
|
+
articles.append(f"**{payload.get('title', 'Untitled')}**\n{payload.get('content', '')}")
|
|
958
|
+
|
|
959
|
+
return "\n\n---\n\n".join(articles)
|
|
960
|
+
|
|
961
|
+
|
|
962
|
+
def _mock_search(query: str, top_k: int) -> str:
|
|
963
|
+
"""Simple keyword-based fallback search."""
|
|
964
|
+
query_lower = query.lower()
|
|
965
|
+
scored = []
|
|
966
|
+
for article in _MOCK_KB:
|
|
967
|
+
score = sum(
|
|
968
|
+
1
|
|
969
|
+
for word in query_lower.split()
|
|
970
|
+
if word in article["title"].lower() or word in article["content"].lower()
|
|
971
|
+
)
|
|
972
|
+
if score > 0:
|
|
973
|
+
scored.append((score, article))
|
|
974
|
+
|
|
975
|
+
scored.sort(key=lambda x: x[0], reverse=True)
|
|
976
|
+
top = scored[:top_k]
|
|
977
|
+
|
|
978
|
+
if not top:
|
|
979
|
+
return "No relevant articles found in the knowledge base."
|
|
980
|
+
|
|
981
|
+
articles = [f"**{a['title']}**\n{a['content']}" for _, a in top]
|
|
982
|
+
return "\n\n---\n\n".join(articles)
|
|
983
|
+
```
|
|
984
|
+
|
|
985
|
+
</details>
|
|
986
|
+
|
|
987
|
+
<details>
|
|
988
|
+
<summary><code>app/tools/stripe.py</code></summary>
|
|
989
|
+
|
|
990
|
+
```python
|
|
991
|
+
"""Mock Stripe MCP tool for billing lookups.
|
|
992
|
+
|
|
993
|
+
In production, this would connect to a real Stripe MCP server.
|
|
994
|
+
The mock returns realistic responses for demo and eval purposes.
|
|
995
|
+
"""
|
|
996
|
+
|
|
997
|
+
import json
|
|
998
|
+
|
|
999
|
+
_MOCK_CUSTOMERS = {
|
|
1000
|
+
"default": {
|
|
1001
|
+
"customer_id": "cus_demo123",
|
|
1002
|
+
"email": "customer@example.com",
|
|
1003
|
+
"plan": "Pro",
|
|
1004
|
+
"monthly_amount": "$49.00",
|
|
1005
|
+
"status": "active",
|
|
1006
|
+
"last_payment": "2026-04-01",
|
|
1007
|
+
"next_billing": "2026-05-01",
|
|
1008
|
+
"payment_method": "Visa ending in 4242",
|
|
1009
|
+
}
|
|
1010
|
+
}
|
|
1011
|
+
|
|
1012
|
+
_MOCK_RESPONSES = {
|
|
1013
|
+
"charge": (
|
|
1014
|
+
"Found recent charge of $49.00 on 2026-04-01 for Pro plan"
|
|
1015
|
+
" subscription. Status: succeeded. No duplicate charges detected."
|
|
1016
|
+
),
|
|
1017
|
+
"refund": (
|
|
1018
|
+
"Refund policy: Full refunds available within 30 days of charge."
|
|
1019
|
+
" To process a refund, the customer should confirm the charge"
|
|
1020
|
+
" date and amount."
|
|
1021
|
+
),
|
|
1022
|
+
"subscription": (
|
|
1023
|
+
"Current subscription: Pro plan at $49.00/month. Next billing"
|
|
1024
|
+
" date: 2026-05-01. Plan can be changed or cancelled from"
|
|
1025
|
+
" account settings."
|
|
1026
|
+
),
|
|
1027
|
+
"invoice": (
|
|
1028
|
+
"Most recent invoice #INV-2026-0401 for $49.00 issued on"
|
|
1029
|
+
" 2026-04-01. Status: paid. PDF available at billing portal."
|
|
1030
|
+
),
|
|
1031
|
+
"payment_method": (
|
|
1032
|
+
"Current payment method: Visa ending in 4242, expiring 12/2027."
|
|
1033
|
+
" To update, customer can visit the billing portal."
|
|
1034
|
+
),
|
|
1035
|
+
}
|
|
1036
|
+
|
|
1037
|
+
|
|
1038
|
+
async def stripe_lookup(query: str) -> str:
|
|
1039
|
+
"""Look up billing information. Returns a structured response string."""
|
|
1040
|
+
query_lower = query.lower()
|
|
1041
|
+
|
|
1042
|
+
for keyword, response in _MOCK_RESPONSES.items():
|
|
1043
|
+
if keyword in query_lower:
|
|
1044
|
+
customer = _MOCK_CUSTOMERS["default"]
|
|
1045
|
+
return f"Customer: {customer['email']} ({customer['customer_id']})\n{response}"
|
|
1046
|
+
|
|
1047
|
+
customer = _MOCK_CUSTOMERS["default"]
|
|
1048
|
+
return f"Customer billing summary:\n{json.dumps(customer, indent=2)}"
|
|
1049
|
+
```
|
|
1050
|
+
|
|
1051
|
+
</details>
|
|
1052
|
+
|
|
1053
|
+
<details>
|
|
1054
|
+
<summary><code>app/db/models.py</code></summary>
|
|
1055
|
+
|
|
1056
|
+
```python
|
|
1057
|
+
"""SQLAlchemy models for conversations and messages."""
|
|
1058
|
+
|
|
1059
|
+
import uuid
|
|
1060
|
+
from datetime import UTC, datetime
|
|
1061
|
+
|
|
1062
|
+
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, String, Text
|
|
1063
|
+
from sqlalchemy.dialects.postgresql import JSON
|
|
1064
|
+
from sqlalchemy.orm import DeclarativeBase, relationship
|
|
1065
|
+
|
|
1066
|
+
|
|
1067
|
+
class Base(DeclarativeBase):
|
|
1068
|
+
pass
|
|
1069
|
+
|
|
1070
|
+
|
|
1071
|
+
class Conversation(Base):
|
|
1072
|
+
__tablename__ = "conversations"
|
|
1073
|
+
|
|
1074
|
+
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
|
1075
|
+
user_id = Column(String, nullable=False, index=True)
|
|
1076
|
+
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
|
|
1077
|
+
resolved_at = Column(DateTime(timezone=True), nullable=True)
|
|
1078
|
+
escalated = Column(Boolean, default=False)
|
|
1079
|
+
|
|
1080
|
+
messages = relationship("Message", back_populates="conversation", order_by="Message.created_at")
|
|
1081
|
+
|
|
1082
|
+
|
|
1083
|
+
class Message(Base):
|
|
1084
|
+
__tablename__ = "messages"
|
|
1085
|
+
|
|
1086
|
+
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
|
1087
|
+
conversation_id = Column(String, ForeignKey("conversations.id"), nullable=False, index=True)
|
|
1088
|
+
role = Column(String, nullable=False)
|
|
1089
|
+
content = Column(Text, nullable=False)
|
|
1090
|
+
intent = Column(String, nullable=True)
|
|
1091
|
+
tool_calls_json = Column(JSON, nullable=True)
|
|
1092
|
+
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
|
|
1093
|
+
|
|
1094
|
+
conversation = relationship("Conversation", back_populates="messages")
|
|
1095
|
+
```
|
|
1096
|
+
|
|
1097
|
+
</details>
|
|
1098
|
+
|
|
1099
|
+
<details>
|
|
1100
|
+
<summary><code>app/db/session.py</code></summary>
|
|
1101
|
+
|
|
1102
|
+
```python
|
|
1103
|
+
"""Database session management."""
|
|
1104
|
+
|
|
1105
|
+
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
|
1106
|
+
|
|
1107
|
+
from app.settings import settings
|
|
1108
|
+
|
|
1109
|
+
engine = create_async_engine(settings.database_url, echo=settings.app_env == "development")
|
|
1110
|
+
async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
|
1111
|
+
|
|
1112
|
+
|
|
1113
|
+
async def get_session() -> AsyncSession:
|
|
1114
|
+
async with async_session() as session:
|
|
1115
|
+
yield session
|
|
1116
|
+
```
|
|
1117
|
+
|
|
1118
|
+
</details>
|
|
1119
|
+
|
|
1120
|
+
### TypeScript
|
|
1121
|
+
|
|
1122
|
+
<details>
|
|
1123
|
+
<summary><code>src/index.ts</code></summary>
|
|
1124
|
+
|
|
1125
|
+
```typescript
|
|
1126
|
+
import { serve } from "@hono/node-server";
|
|
1127
|
+
import { Hono } from "hono";
|
|
1128
|
+
import { triageRouter } from "./api/triage.js";
|
|
1129
|
+
|
|
1130
|
+
const app = new Hono();
|
|
1131
|
+
|
|
1132
|
+
app.get("/health", (c) => c.json({ status: "ok" }));
|
|
1133
|
+
app.route("/", triageRouter);
|
|
1134
|
+
|
|
1135
|
+
const port = Number(process.env.PORT ?? 8000);
|
|
1136
|
+
|
|
1137
|
+
serve({ fetch: app.fetch, port }, (info) => {
|
|
1138
|
+
console.log(
|
|
1139
|
+
`customer-support-triage running at http://localhost:${info.port}`,
|
|
1140
|
+
);
|
|
1141
|
+
});
|
|
1142
|
+
|
|
1143
|
+
export default app;
|
|
1144
|
+
```
|
|
1145
|
+
|
|
1146
|
+
</details>
|
|
1147
|
+
|
|
1148
|
+
<details>
|
|
1149
|
+
<summary><code>src/config.ts</code></summary>
|
|
1150
|
+
|
|
1151
|
+
```typescript
|
|
1152
|
+
import { z } from "zod";
|
|
1153
|
+
|
|
1154
|
+
const configSchema = z.object({
|
|
1155
|
+
appName: z.string().default("customer-support-triage"),
|
|
1156
|
+
appEnv: z.string().default("development"),
|
|
1157
|
+
logLevel: z.string().default("info"),
|
|
1158
|
+
|
|
1159
|
+
anthropicApiKey: z.string().default(""),
|
|
1160
|
+
classifierModel: z.string().default("claude-haiku-4-5-20251001"),
|
|
1161
|
+
specialistModel: z.string().default("claude-sonnet-4-6-20250514"),
|
|
1162
|
+
|
|
1163
|
+
escalationThreshold: z.coerce.number().default(0.7),
|
|
1164
|
+
|
|
1165
|
+
databaseUrl: z
|
|
1166
|
+
.string()
|
|
1167
|
+
.default("postgresql://agent:agent@localhost:5432/agent_db"),
|
|
1168
|
+
redisUrl: z.string().default("redis://localhost:6379"),
|
|
1169
|
+
qdrantUrl: z.string().default("http://localhost:6333"),
|
|
1170
|
+
|
|
1171
|
+
jwtSecret: z.string().default("change-me-in-production"),
|
|
1172
|
+
|
|
1173
|
+
langfusePublicKey: z.string().default("pk-lf-local"),
|
|
1174
|
+
langfuseSecretKey: z.string().default("sk-lf-local"),
|
|
1175
|
+
langfuseHost: z.string().default("http://localhost:3000"),
|
|
1176
|
+
});
|
|
1177
|
+
|
|
1178
|
+
export const config = configSchema.parse({
|
|
1179
|
+
appName: process.env.APP_NAME,
|
|
1180
|
+
appEnv: process.env.APP_ENV,
|
|
1181
|
+
logLevel: process.env.LOG_LEVEL,
|
|
1182
|
+
anthropicApiKey: process.env.ANTHROPIC_API_KEY,
|
|
1183
|
+
classifierModel: process.env.CLASSIFIER_MODEL,
|
|
1184
|
+
specialistModel: process.env.SPECIALIST_MODEL,
|
|
1185
|
+
escalationThreshold: process.env.ESCALATION_THRESHOLD,
|
|
1186
|
+
databaseUrl: process.env.DATABASE_URL,
|
|
1187
|
+
redisUrl: process.env.REDIS_URL,
|
|
1188
|
+
qdrantUrl: process.env.QDRANT_URL,
|
|
1189
|
+
jwtSecret: process.env.JWT_SECRET,
|
|
1190
|
+
langfusePublicKey: process.env.LANGFUSE_PUBLIC_KEY,
|
|
1191
|
+
langfuseSecretKey: process.env.LANGFUSE_SECRET_KEY,
|
|
1192
|
+
langfuseHost: process.env.LANGFUSE_HOST,
|
|
1193
|
+
});
|
|
1194
|
+
|
|
1195
|
+
export type Config = z.infer<typeof configSchema>;
|
|
1196
|
+
```
|
|
1197
|
+
|
|
1198
|
+
</details>
|
|
1199
|
+
|
|
1200
|
+
<details>
|
|
1201
|
+
<summary><code>src/schemas/index.ts</code></summary>
|
|
1202
|
+
|
|
1203
|
+
```typescript
|
|
1204
|
+
import { z } from "zod";
|
|
1205
|
+
|
|
1206
|
+
export const Intent = z.enum(["billing", "technical", "account", "general"]);
|
|
1207
|
+
export type Intent = z.infer<typeof Intent>;
|
|
1208
|
+
|
|
1209
|
+
export const ClassificationResult = z.object({
|
|
1210
|
+
intent: Intent,
|
|
1211
|
+
confidence: z.number().min(0).max(1),
|
|
1212
|
+
reasoning: z.string(),
|
|
1213
|
+
});
|
|
1214
|
+
export type ClassificationResult = z.infer<typeof ClassificationResult>;
|
|
1215
|
+
|
|
1216
|
+
export const TriageRequest = z.object({
|
|
1217
|
+
message: z.string().min(1),
|
|
1218
|
+
user_id: z.string().min(1),
|
|
1219
|
+
});
|
|
1220
|
+
export type TriageRequest = z.infer<typeof TriageRequest>;
|
|
1221
|
+
|
|
1222
|
+
export const TriageResponse = z.object({
|
|
1223
|
+
conversation_id: z.string(),
|
|
1224
|
+
intent: z.string(),
|
|
1225
|
+
specialist_response: z.string(),
|
|
1226
|
+
escalated: z.boolean(),
|
|
1227
|
+
trace_id: z.string(),
|
|
1228
|
+
});
|
|
1229
|
+
export type TriageResponse = z.infer<typeof TriageResponse>;
|
|
1230
|
+
```
|
|
1231
|
+
|
|
1232
|
+
</details>
|
|
1233
|
+
|
|
1234
|
+
<details>
|
|
1235
|
+
<summary><code>src/agent/classifier.ts</code></summary>
|
|
1236
|
+
|
|
1237
|
+
```typescript
|
|
1238
|
+
/**
|
|
1239
|
+
* Intent classifier using Vercel AI SDK with structured output.
|
|
1240
|
+
*/
|
|
1241
|
+
|
|
1242
|
+
import { anthropic } from "@ai-sdk/anthropic";
|
|
1243
|
+
import { generateObject } from "ai";
|
|
1244
|
+
import { config } from "../config.js";
|
|
1245
|
+
import { ClassificationResult } from "../schemas/index.js";
|
|
1246
|
+
|
|
1247
|
+
const CLASSIFIER_SYSTEM_PROMPT = `You are a customer support intent classifier.
|
|
1248
|
+
Given a customer message, classify it into exactly one of these intents:
|
|
1249
|
+
- billing: payment issues, subscription changes, invoices, charges, refunds
|
|
1250
|
+
- technical: bugs, errors, API issues, integration problems, performance
|
|
1251
|
+
- account: password resets, profile updates, access issues, account settings
|
|
1252
|
+
- general: everything else, general questions, feedback, feature requests
|
|
1253
|
+
|
|
1254
|
+
Return the intent, your confidence (0.0 to 1.0), and brief reasoning.`;
|
|
1255
|
+
|
|
1256
|
+
export async function classifyIntent(
|
|
1257
|
+
message: string,
|
|
1258
|
+
): Promise<{ intent: string; confidence: number; reasoning: string }> {
|
|
1259
|
+
const result = await generateObject({
|
|
1260
|
+
model: anthropic(config.classifierModel),
|
|
1261
|
+
schema: ClassificationResult,
|
|
1262
|
+
system: CLASSIFIER_SYSTEM_PROMPT,
|
|
1263
|
+
prompt: message,
|
|
1264
|
+
});
|
|
1265
|
+
|
|
1266
|
+
return result.object;
|
|
1267
|
+
}
|
|
1268
|
+
```
|
|
1269
|
+
|
|
1270
|
+
</details>
|
|
1271
|
+
|
|
1272
|
+
<details>
|
|
1273
|
+
<summary><code>src/agent/specialists.ts</code></summary>
|
|
1274
|
+
|
|
1275
|
+
```typescript
|
|
1276
|
+
/**
|
|
1277
|
+
* Specialist agents for each intent category.
|
|
1278
|
+
*/
|
|
1279
|
+
|
|
1280
|
+
import { anthropic } from "@ai-sdk/anthropic";
|
|
1281
|
+
import { generateText, tool } from "ai";
|
|
1282
|
+
import { z } from "zod";
|
|
1283
|
+
import { config } from "../config.js";
|
|
1284
|
+
import { kbSearch } from "../tools/kb.js";
|
|
1285
|
+
import { stripeLookup } from "../tools/stripe.js";
|
|
1286
|
+
|
|
1287
|
+
const SPECIALIST_PROMPTS: Record<string, string> = {
|
|
1288
|
+
billing: `You are a billing support specialist. Help customers with payment issues,
|
|
1289
|
+
subscription changes, invoices, and charges. You have access to the Stripe tool to look up
|
|
1290
|
+
billing information. Be helpful, concise, and professional.`,
|
|
1291
|
+
technical: `You are a technical support specialist. Help customers with bugs, errors,
|
|
1292
|
+
API issues, and integration problems. You have access to a knowledge base search tool.
|
|
1293
|
+
Provide clear, actionable guidance.`,
|
|
1294
|
+
account: `You are an account support specialist. Help customers with password resets,
|
|
1295
|
+
profile updates, and account settings. You have access to a knowledge base search tool.
|
|
1296
|
+
Guide them step by step.`,
|
|
1297
|
+
general: `You are a general support specialist. Help customers with general questions,
|
|
1298
|
+
feedback, and feature requests. Be friendly and helpful.`,
|
|
1299
|
+
};
|
|
1300
|
+
|
|
1301
|
+
const billingTools = {
|
|
1302
|
+
lookup_billing: tool({
|
|
1303
|
+
description: "Look up billing information for a customer using Stripe",
|
|
1304
|
+
parameters: z.object({ query: z.string() }),
|
|
1305
|
+
execute: async ({ query }) => stripeLookup(query),
|
|
1306
|
+
}),
|
|
1307
|
+
};
|
|
1308
|
+
|
|
1309
|
+
const kbTools = {
|
|
1310
|
+
search_knowledge_base: tool({
|
|
1311
|
+
description: "Search the knowledge base for relevant articles",
|
|
1312
|
+
parameters: z.object({ query: z.string() }),
|
|
1313
|
+
execute: async ({ query }) => kbSearch(query),
|
|
1314
|
+
}),
|
|
1315
|
+
};
|
|
1316
|
+
|
|
1317
|
+
const SPECIALIST_TOOLS: Record<string, typeof billingTools | typeof kbTools> = {
|
|
1318
|
+
billing: billingTools,
|
|
1319
|
+
technical: kbTools,
|
|
1320
|
+
account: kbTools,
|
|
1321
|
+
};
|
|
1322
|
+
|
|
1323
|
+
export async function runSpecialist(
|
|
1324
|
+
intent: string,
|
|
1325
|
+
message: string,
|
|
1326
|
+
): Promise<{
|
|
1327
|
+
text: string;
|
|
1328
|
+
toolCalls: Array<{ toolName: string; args: Record<string, unknown> }>;
|
|
1329
|
+
}> {
|
|
1330
|
+
const systemPrompt = SPECIALIST_PROMPTS[intent] ?? SPECIALIST_PROMPTS.general;
|
|
1331
|
+
const tools = SPECIALIST_TOOLS[intent] ?? undefined;
|
|
1332
|
+
|
|
1333
|
+
const result = await generateText({
|
|
1334
|
+
model: anthropic(config.specialistModel),
|
|
1335
|
+
system: systemPrompt,
|
|
1336
|
+
prompt: message,
|
|
1337
|
+
tools,
|
|
1338
|
+
maxSteps: 3,
|
|
1339
|
+
});
|
|
1340
|
+
|
|
1341
|
+
const toolCalls = result.steps
|
|
1342
|
+
.flatMap((s) => s.toolCalls)
|
|
1343
|
+
.map((tc: { toolName: string; args: unknown }) => ({
|
|
1344
|
+
toolName: tc.toolName,
|
|
1345
|
+
args: tc.args as Record<string, unknown>,
|
|
1346
|
+
}));
|
|
1347
|
+
|
|
1348
|
+
return { text: result.text, toolCalls };
|
|
1349
|
+
}
|
|
1350
|
+
```
|
|
1351
|
+
|
|
1352
|
+
</details>
|
|
1353
|
+
|
|
1354
|
+
<details>
|
|
1355
|
+
<summary><code>src/api/triage.ts</code></summary>
|
|
1356
|
+
|
|
1357
|
+
```typescript
|
|
1358
|
+
/**
|
|
1359
|
+
* Triage route handler.
|
|
1360
|
+
*/
|
|
1361
|
+
|
|
1362
|
+
import { Hono } from "hono";
|
|
1363
|
+
import { classifyIntent } from "../agent/classifier.js";
|
|
1364
|
+
import { runSpecialist } from "../agent/specialists.js";
|
|
1365
|
+
import { config } from "../config.js";
|
|
1366
|
+
import { TriageRequest } from "../schemas/index.js";
|
|
1367
|
+
|
|
1368
|
+
export const triageRouter = new Hono();
|
|
1369
|
+
|
|
1370
|
+
triageRouter.post("/triage", async (c) => {
|
|
1371
|
+
const body = await c.req.json();
|
|
1372
|
+
const parsed = TriageRequest.safeParse(body);
|
|
1373
|
+
|
|
1374
|
+
if (!parsed.success) {
|
|
1375
|
+
return c.json(
|
|
1376
|
+
{ error: "Invalid request", details: parsed.error.issues },
|
|
1377
|
+
400,
|
|
1378
|
+
);
|
|
1379
|
+
}
|
|
1380
|
+
|
|
1381
|
+
const { message, user_id } = parsed.data;
|
|
1382
|
+
const traceId = crypto.randomUUID();
|
|
1383
|
+
const conversationId = crypto.randomUUID();
|
|
1384
|
+
|
|
1385
|
+
// Classify intent
|
|
1386
|
+
const classification = await classifyIntent(message);
|
|
1387
|
+
|
|
1388
|
+
// Check escalation threshold
|
|
1389
|
+
if (classification.confidence < config.escalationThreshold) {
|
|
1390
|
+
return c.json({
|
|
1391
|
+
conversation_id: conversationId,
|
|
1392
|
+
intent: classification.intent,
|
|
1393
|
+
specialist_response:
|
|
1394
|
+
"Escalated to human agent due to low classification confidence.",
|
|
1395
|
+
escalated: true,
|
|
1396
|
+
trace_id: traceId,
|
|
1397
|
+
});
|
|
1398
|
+
}
|
|
1399
|
+
|
|
1400
|
+
// Route to specialist
|
|
1401
|
+
const { text, toolCalls } = await runSpecialist(
|
|
1402
|
+
classification.intent,
|
|
1403
|
+
message,
|
|
1404
|
+
);
|
|
1405
|
+
|
|
1406
|
+
return c.json({
|
|
1407
|
+
conversation_id: conversationId,
|
|
1408
|
+
intent: classification.intent,
|
|
1409
|
+
specialist_response: text,
|
|
1410
|
+
escalated: false,
|
|
1411
|
+
trace_id: traceId,
|
|
1412
|
+
});
|
|
1413
|
+
});
|
|
1414
|
+
```
|
|
1415
|
+
|
|
1416
|
+
</details>
|
|
1417
|
+
|
|
1418
|
+
<details>
|
|
1419
|
+
<summary><code>src/tools/kb.ts</code></summary>
|
|
1420
|
+
|
|
1421
|
+
```typescript
|
|
1422
|
+
/**
|
|
1423
|
+
* Knowledge base search tool. Falls back to mock data when Qdrant is unavailable.
|
|
1424
|
+
*/
|
|
1425
|
+
|
|
1426
|
+
interface KBArticle {
|
|
1427
|
+
id: string;
|
|
1428
|
+
title: string;
|
|
1429
|
+
content: string;
|
|
1430
|
+
category: string;
|
|
1431
|
+
}
|
|
1432
|
+
|
|
1433
|
+
const MOCK_KB: KBArticle[] = [
|
|
1434
|
+
{
|
|
1435
|
+
id: "kb-001",
|
|
1436
|
+
title: "How to reset your password",
|
|
1437
|
+
content:
|
|
1438
|
+
"Go to Settings > Security > Reset Password. Enter your current password, then your new password twice. Click Save.",
|
|
1439
|
+
category: "account",
|
|
1440
|
+
},
|
|
1441
|
+
{
|
|
1442
|
+
id: "kb-002",
|
|
1443
|
+
title: "API rate limits and error codes",
|
|
1444
|
+
content:
|
|
1445
|
+
"Rate limits: 100 req/min for free, 1000/min for Pro. 429 = rate limited. Common: 400 (bad request), 401 (unauthorized), 500 (server error).",
|
|
1446
|
+
category: "technical",
|
|
1447
|
+
},
|
|
1448
|
+
{
|
|
1449
|
+
id: "kb-003",
|
|
1450
|
+
title: "Updating your billing information",
|
|
1451
|
+
content:
|
|
1452
|
+
"Navigate to Account > Billing > Payment Methods. Click Update next to your current method. Enter new card details.",
|
|
1453
|
+
category: "billing",
|
|
1454
|
+
},
|
|
1455
|
+
{
|
|
1456
|
+
id: "kb-004",
|
|
1457
|
+
title: "Troubleshooting large payload errors",
|
|
1458
|
+
content:
|
|
1459
|
+
"Maximum payload size is 10MB. For larger payloads, use streaming or split requests. Check Content-Type header and JSON validity.",
|
|
1460
|
+
category: "technical",
|
|
1461
|
+
},
|
|
1462
|
+
{
|
|
1463
|
+
id: "kb-005",
|
|
1464
|
+
title: "Two-factor authentication setup",
|
|
1465
|
+
content:
|
|
1466
|
+
"Go to Settings > Security > 2FA. Choose authenticator app or SMS. Scan QR code. Enter 6-digit code to verify. Save backup codes.",
|
|
1467
|
+
category: "account",
|
|
1468
|
+
},
|
|
1469
|
+
{
|
|
1470
|
+
id: "kb-006",
|
|
1471
|
+
title: "Integration webhook configuration",
|
|
1472
|
+
content:
|
|
1473
|
+
"Set up at Settings > Integrations > Webhooks. Add endpoint URL, select events, save. Payloads are signed with your webhook secret.",
|
|
1474
|
+
category: "technical",
|
|
1475
|
+
},
|
|
1476
|
+
];
|
|
1477
|
+
|
|
1478
|
+
export function mockSearch(query: string, topK = 3): string {
|
|
1479
|
+
const words = query.toLowerCase().split(/\s+/);
|
|
1480
|
+
const scored: Array<[number, KBArticle]> = [];
|
|
1481
|
+
|
|
1482
|
+
for (const article of MOCK_KB) {
|
|
1483
|
+
const text = `${article.title} ${article.content}`.toLowerCase();
|
|
1484
|
+
const score = words.filter((w) => text.includes(w)).length;
|
|
1485
|
+
if (score > 0) scored.push([score, article]);
|
|
1486
|
+
}
|
|
1487
|
+
|
|
1488
|
+
scored.sort((a, b) => b[0] - a[0]);
|
|
1489
|
+
const top = scored.slice(0, topK);
|
|
1490
|
+
|
|
1491
|
+
if (top.length === 0)
|
|
1492
|
+
return "No relevant articles found in the knowledge base.";
|
|
1493
|
+
|
|
1494
|
+
return top.map(([, a]) => `**${a.title}**\n${a.content}`).join("\n\n---\n\n");
|
|
1495
|
+
}
|
|
1496
|
+
|
|
1497
|
+
export async function kbSearch(query: string, topK = 3): Promise<string> {
|
|
1498
|
+
return mockSearch(query, topK);
|
|
1499
|
+
}
|
|
1500
|
+
```
|
|
1501
|
+
|
|
1502
|
+
</details>
|
|
1503
|
+
|
|
1504
|
+
<details>
|
|
1505
|
+
<summary><code>src/tools/stripe.ts</code></summary>
|
|
1506
|
+
|
|
1507
|
+
```typescript
|
|
1508
|
+
/**
|
|
1509
|
+
* Mock Stripe tool for billing lookups.
|
|
1510
|
+
*/
|
|
1511
|
+
|
|
1512
|
+
const MOCK_CUSTOMER = {
|
|
1513
|
+
customer_id: "cus_demo123",
|
|
1514
|
+
email: "customer@example.com",
|
|
1515
|
+
plan: "Pro",
|
|
1516
|
+
monthly_amount: "$49.00",
|
|
1517
|
+
status: "active",
|
|
1518
|
+
last_payment: "2026-04-01",
|
|
1519
|
+
next_billing: "2026-05-01",
|
|
1520
|
+
payment_method: "Visa ending in 4242",
|
|
1521
|
+
};
|
|
1522
|
+
|
|
1523
|
+
const MOCK_RESPONSES: Record<string, string> = {
|
|
1524
|
+
charge:
|
|
1525
|
+
"Found recent charge of $49.00 on 2026-04-01 for Pro plan subscription. Status: succeeded. No duplicate charges detected.",
|
|
1526
|
+
refund:
|
|
1527
|
+
"Refund policy: Full refunds available within 30 days of charge. To process a refund, the customer should confirm the charge date and amount.",
|
|
1528
|
+
subscription:
|
|
1529
|
+
"Current subscription: Pro plan at $49.00/month. Next billing date: 2026-05-01. Plan can be changed or cancelled from account settings.",
|
|
1530
|
+
invoice:
|
|
1531
|
+
"Most recent invoice #INV-2026-0401 for $49.00 issued on 2026-04-01. Status: paid. PDF available at billing portal.",
|
|
1532
|
+
payment_method:
|
|
1533
|
+
"Current payment method: Visa ending in 4242, expiring 12/2027. To update, customer can visit the billing portal.",
|
|
1534
|
+
};
|
|
1535
|
+
|
|
1536
|
+
export async function stripeLookup(query: string): Promise<string> {
|
|
1537
|
+
const q = query.toLowerCase();
|
|
1538
|
+
|
|
1539
|
+
for (const [keyword, response] of Object.entries(MOCK_RESPONSES)) {
|
|
1540
|
+
if (q.includes(keyword)) {
|
|
1541
|
+
return `Customer: ${MOCK_CUSTOMER.email} (${MOCK_CUSTOMER.customer_id})\n${response}`;
|
|
1542
|
+
}
|
|
1543
|
+
}
|
|
1544
|
+
|
|
1545
|
+
return `Customer billing summary:\n${JSON.stringify(MOCK_CUSTOMER, null, 2)}`;
|
|
1546
|
+
}
|
|
1547
|
+
```
|
|
1548
|
+
|
|
1549
|
+
</details>
|
|
1550
|
+
|
|
1551
|
+
### Configuration & Eval
|
|
1552
|
+
|
|
1553
|
+
<details>
|
|
1554
|
+
<summary><code>.env.example</code></summary>
|
|
1555
|
+
|
|
1556
|
+
```bash
|
|
1557
|
+
# Required
|
|
1558
|
+
ANTHROPIC_API_KEY=sk-ant-...
|
|
1559
|
+
|
|
1560
|
+
# LLM models (defaults are fine for local dev)
|
|
1561
|
+
# CLASSIFIER_MODEL=claude-haiku-4-5-20251001
|
|
1562
|
+
# SPECIALIST_MODEL=claude-sonnet-4-6-20250514
|
|
1563
|
+
|
|
1564
|
+
# Routing
|
|
1565
|
+
# ESCALATION_THRESHOLD=0.7
|
|
1566
|
+
|
|
1567
|
+
# Postgres
|
|
1568
|
+
POSTGRES_USER=agent
|
|
1569
|
+
POSTGRES_PASSWORD=agent
|
|
1570
|
+
POSTGRES_DB=agent_db
|
|
1571
|
+
DATABASE_URL=postgresql+asyncpg://agent:agent@postgres:5432/agent_db
|
|
1572
|
+
|
|
1573
|
+
# Redis
|
|
1574
|
+
REDIS_URL=redis://redis:6379
|
|
1575
|
+
|
|
1576
|
+
# Qdrant
|
|
1577
|
+
QDRANT_URL=http://qdrant:6333
|
|
1578
|
+
|
|
1579
|
+
# Langfuse
|
|
1580
|
+
LANGFUSE_PUBLIC_KEY=pk-lf-local
|
|
1581
|
+
LANGFUSE_SECRET_KEY=sk-lf-local
|
|
1582
|
+
LANGFUSE_HOST=http://langfuse:3000
|
|
1583
|
+
|
|
1584
|
+
# Auth
|
|
1585
|
+
JWT_SECRET=change-me-in-production
|
|
1586
|
+
|
|
1587
|
+
# App
|
|
1588
|
+
APP_ENV=development
|
|
1589
|
+
LOG_LEVEL=INFO
|
|
1590
|
+
```
|
|
1591
|
+
|
|
1592
|
+
</details>
|
|
1593
|
+
|
|
1594
|
+
<details>
|
|
1595
|
+
<summary><code>eval/dataset.jsonl</code> (51 examples)</summary>
|
|
1596
|
+
|
|
1597
|
+
```jsonl
|
|
1598
|
+
{"input": "I was charged twice for my subscription this month", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1599
|
+
{"input": "Can you help me get a refund for last month?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1600
|
+
{"input": "What payment methods do you accept?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1601
|
+
{"input": "My invoice shows the wrong amount", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1602
|
+
{"input": "I want to upgrade my subscription plan", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1603
|
+
{"input": "When is my next billing date?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1604
|
+
{"input": "I need to update my credit card on file", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1605
|
+
{"input": "Why was I charged $99 instead of $49?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1606
|
+
{"input": "Can I get a receipt for my last payment?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1607
|
+
{"input": "I want to cancel my subscription", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1608
|
+
{"input": "How do I downgrade to the free plan?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1609
|
+
{"input": "Is there a discount for annual billing?", "expected_intent": "billing", "expected_tool": "lookup_billing"}
|
|
1610
|
+
{"input": "The API returns 500 errors when I send large payloads", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1611
|
+
{"input": "I'm getting a 429 rate limit error", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1612
|
+
{"input": "How do I authenticate with the API?", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1613
|
+
{"input": "The webhook integration isn't working", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1614
|
+
{"input": "What's the maximum payload size for the API?", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1615
|
+
{"input": "I'm getting CORS errors in the browser", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1616
|
+
{"input": "The SDK throws a timeout exception after 30 seconds", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1617
|
+
{"input": "How do I set up webhooks for my integration?", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1618
|
+
{"input": "My API key stopped working suddenly", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1619
|
+
{"input": "Is there a sandbox environment for testing?", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1620
|
+
{"input": "The response format changed and broke my parser", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1621
|
+
{"input": "How do I handle pagination in the list endpoint?", "expected_intent": "technical", "expected_tool": "search_knowledge_base"}
|
|
1622
|
+
{"input": "I need to reset my password", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1623
|
+
{"input": "How do I enable two-factor authentication?", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1624
|
+
{"input": "I want to change my email address", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1625
|
+
{"input": "How do I delete my account?", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1626
|
+
{"input": "I can't log in to my account", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1627
|
+
{"input": "How do I add a team member to my organization?", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1628
|
+
{"input": "I want to update my profile information", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1629
|
+
{"input": "How do I change my notification preferences?", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1630
|
+
{"input": "I forgot my username", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1631
|
+
{"input": "How do I revoke API keys from my account?", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1632
|
+
{"input": "Can I transfer ownership of my account?", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1633
|
+
{"input": "My account was locked after too many login attempts", "expected_intent": "account", "expected_tool": "search_knowledge_base"}
|
|
1634
|
+
{"input": "What features are included in the Pro plan?", "expected_intent": "general", "expected_tool": null}
|
|
1635
|
+
{"input": "Do you have a mobile app?", "expected_intent": "general", "expected_tool": null}
|
|
1636
|
+
{"input": "I'd like to request a new feature", "expected_intent": "general", "expected_tool": null}
|
|
1637
|
+
{"input": "What's on your product roadmap?", "expected_intent": "general", "expected_tool": null}
|
|
1638
|
+
{"input": "How does your product compare to competitors?", "expected_intent": "general", "expected_tool": null}
|
|
1639
|
+
{"input": "Can I schedule a demo?", "expected_intent": "general", "expected_tool": null}
|
|
1640
|
+
{"input": "Where can I find your documentation?", "expected_intent": "general", "expected_tool": null}
|
|
1641
|
+
{"input": "Do you offer enterprise plans?", "expected_intent": "general", "expected_tool": null}
|
|
1642
|
+
{"input": "What's your uptime SLA?", "expected_intent": "general", "expected_tool": null}
|
|
1643
|
+
{"input": "I love your product, keep up the great work!", "expected_intent": "general", "expected_tool": null}
|
|
1644
|
+
{"input": "Do you have a partner program?", "expected_intent": "general", "expected_tool": null}
|
|
1645
|
+
{"input": "Is there a community forum?", "expected_intent": "general", "expected_tool": null}
|
|
1646
|
+
{"input": "I need help but I'm not sure who to talk to", "expected_intent": "general", "expected_tool": null}
|
|
1647
|
+
{"input": "Can you tell me about your data privacy practices?", "expected_intent": "general", "expected_tool": null}
|
|
1648
|
+
```
|
|
1649
|
+
|
|
1650
|
+
</details>
|
|
1651
|
+
|
|
1652
|
+
<details>
|
|
1653
|
+
<summary><code>eval/promptfoo.yaml</code></summary>
|
|
1654
|
+
|
|
1655
|
+
```yaml
|
|
1656
|
+
description: "Security scan for customer-support-triage"
|
|
1657
|
+
|
|
1658
|
+
prompts:
|
|
1659
|
+
- "{{message}}"
|
|
1660
|
+
|
|
1661
|
+
providers:
|
|
1662
|
+
- id: http
|
|
1663
|
+
config:
|
|
1664
|
+
url: http://localhost:8000/triage
|
|
1665
|
+
method: POST
|
|
1666
|
+
headers:
|
|
1667
|
+
Content-Type: application/json
|
|
1668
|
+
body:
|
|
1669
|
+
message: "{{message}}"
|
|
1670
|
+
user_id: "eval-user"
|
|
1671
|
+
|
|
1672
|
+
redteam:
|
|
1673
|
+
plugins:
|
|
1674
|
+
- prompt-injection
|
|
1675
|
+
- jailbreak
|
|
1676
|
+
- pii
|
|
1677
|
+
```
|
|
1678
|
+
|
|
1679
|
+
</details>
|