@madeinoz67/voice-server 0.1.11 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/codanna-explorer.md +30 -0
- package/.claude/agents/codanna-implementer.md +39 -0
- package/.claude/hooks/codanna-nudge.ts +100 -0
- package/.claude/hooks/codanna-reindex.ts +38 -0
- package/.claude/settings.json +67 -0
- package/.claude/settings.local.json +4 -1
- package/.codanna/settings.toml +71 -41
- package/.mcp.json +8 -0
- package/.orca/codanna-archive.ts +41 -0
- package/.orca/codanna-setup.ts +101 -0
- package/CHANGELOG.md +48 -1
- package/CLAUDE.md +51 -1
- package/Formula/madeinoz-voice-server.rb +1 -1
- package/docs/assets/queue-system-overview-v2.png +0 -0
- package/docs/assets/voice-server-architecture-backup.png +0 -0
- package/orca.yaml +21 -0
- package/package.json +1 -1
- package/reports/agent/2025-02-09-12-00-fifo-queue-research.md +380 -0
- package/reports/agent/2026-02-09-queue-rate-limit-integration.md +250 -0
- package/reports/agent/2026-02-09-queued-async-error-handling.md +236 -0
- package/src/ts/services/notification-queue.ts +15 -12
- package/bun.lock +0 -208
|
@@ -0,0 +1,250 @@
|
|
|
1
|
+
# Research Report: FIFO Queue Integration with Rate Limiting Middleware
|
|
2
|
+
|
|
3
|
+
**Date**: 2026-02-09
|
|
4
|
+
**Agent**: Research-Agent-v5
|
|
5
|
+
**Model**: Claude Opus 4.6
|
|
6
|
+
|
|
7
|
+
## Summary
|
|
8
|
+
|
|
9
|
+
This report analyzes how the proposed FIFO notification queue should integrate with the existing rate limiting middleware in the voice-server. The key finding is that rate limiting and queue capacity are distinct concerns that should operate at different layers: rate limiting controls client request frequency, while queue capacity manages server processing capacity.
|
|
10
|
+
|
|
11
|
+
## Key Findings
|
|
12
|
+
|
|
13
|
+
### 1. Current Rate Limiter Implementation
|
|
14
|
+
|
|
15
|
+
**Location**: `src/ts/middleware/rate-limiter.ts:1-115`
|
|
16
|
+
|
|
17
|
+
The existing rate limiter uses a token bucket algorithm with the following characteristics:
|
|
18
|
+
- Default: 60 requests per minute per client
|
|
19
|
+
- Client identification via X-Forwarded-For, X-Real-IP, or fallback to localhost
|
|
20
|
+
- Returns 429 with message "Rate limit exceeded" when limit is hit
|
|
21
|
+
- NO Retry-After header currently implemented
|
|
22
|
+
- Cleanup runs every 60 seconds
|
|
23
|
+
|
|
24
|
+
**Current 429 Response Pattern**:
|
|
25
|
+
```typescript
|
|
26
|
+
return Response.json(
|
|
27
|
+
errorResponse("Rate limit exceeded"),
|
|
28
|
+
{ status: 429 }
|
|
29
|
+
);
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
### 2. Proposed FIFO Queue Specification
|
|
33
|
+
|
|
34
|
+
**Location**: `specs/001-fifo-notify-queue/spec.md`
|
|
35
|
+
|
|
36
|
+
Key requirements:
|
|
37
|
+
- Maximum queue depth: 100 items (FR-005)
|
|
38
|
+
- Returns 429 when queue at capacity (FR-006)
|
|
39
|
+
- Immediate 201 Accepted for valid requests (FR-002)
|
|
40
|
+
- Schema validation BEFORE queuing (FR-013)
|
|
41
|
+
- Average audio duration: 3-10 seconds
|
|
42
|
+
|
|
43
|
+
### 3. Architectural Distinction
|
|
44
|
+
|
|
45
|
+
Rate limiting and queue capacity serve different purposes:
|
|
46
|
+
|
|
47
|
+
| Aspect | Rate Limiter | Queue Capacity |
|
|
48
|
+
|--------|-------------|----------------|
|
|
49
|
+
| **Purpose** | Prevent client abuse | Manage server processing load |
|
|
50
|
+
| **Scope** | Per-client (identified by IP) | Global (server-wide) |
|
|
51
|
+
| **Time Window** | Sliding window (60s) | Until processed |
|
|
52
|
+
| **Error Meaning** | "You're sending too fast" | "Server is too busy" |
|
|
53
|
+
| **Retry Strategy** | Wait for window to expire | Wait for queue to drain |
|
|
54
|
+
|
|
55
|
+
## Decision Framework
|
|
56
|
+
|
|
57
|
+
### Decision 1: Rate Limiting Before or After Queuing?
|
|
58
|
+
|
|
59
|
+
**Decision**: Rate limiting should apply BEFORE queuing.
|
|
60
|
+
|
|
61
|
+
**Rationale**:
|
|
62
|
+
1. **Client protection first**: If a client is abusing the API, they shouldn't consume queue space
|
|
63
|
+
2. **Queue preservation**: The queue is a shared resource - abusive clients shouldn't fill it
|
|
64
|
+
3. **Existing pattern**: Current rate limiter runs at the middleware layer before request handlers (`src/ts/server.ts:387-395`)
|
|
65
|
+
4. **Clear error semantics**: Distinguishes between "you're being rate limited" vs "queue is full"
|
|
66
|
+
|
|
67
|
+
**Evidence**: `src/ts/server.ts:387-395`
|
|
68
|
+
```typescript
|
|
69
|
+
// Rate limiting (apply to POST requests)
|
|
70
|
+
if (req.method === "POST") {
|
|
71
|
+
const clientId = extractClientId(req);
|
|
72
|
+
if (serverState.rateLimiter.isRateLimited(clientId)) {
|
|
73
|
+
return Response.json(
|
|
74
|
+
errorResponse("Rate limit exceeded"),
|
|
75
|
+
{ status: 429 }
|
|
76
|
+
);
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### Decision 2: 429 Response Pattern Distinction
|
|
82
|
+
|
|
83
|
+
**Decision**: Use distinct error messages to differentiate rate limiting from queue overflow.
|
|
84
|
+
|
|
85
|
+
**Rationale**:
|
|
86
|
+
- Rate limit 429: "Rate limit exceeded" (client issue)
|
|
87
|
+
- Queue full 429: "Queue at capacity" (server issue)
|
|
88
|
+
- This allows clients to implement different retry strategies
|
|
89
|
+
- Operators can distinguish between abuse vs high load
|
|
90
|
+
|
|
91
|
+
**Recommended Implementation**:
|
|
92
|
+
```typescript
|
|
93
|
+
// Rate limit 429
|
|
94
|
+
{
|
|
95
|
+
status: "error",
|
|
96
|
+
message: "Rate limit exceeded",
|
|
97
|
+
retry_after: <seconds_until_window_reset>
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
// Queue full 429
|
|
101
|
+
{
|
|
102
|
+
status: "error",
|
|
103
|
+
message: "Queue at capacity",
|
|
104
|
+
retry_after: <estimated_seconds_until_queue_drains>
|
|
105
|
+
}
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
### Decision 3: Retry-After Header Strategy
|
|
109
|
+
|
|
110
|
+
**Decision**: Implement Retry-After header for both scenarios with different calculation methods.
|
|
111
|
+
|
|
112
|
+
**Rate Limiter Retry-After**:
|
|
113
|
+
- Calculate seconds until client's rate limit window resets
|
|
114
|
+
- Use `getResetTime(clientId)` from existing rate limiter
|
|
115
|
+
- Example: `Math.ceil((resetTime - now) / 1000)`
|
|
116
|
+
|
|
117
|
+
**Queue Full Retry-After**:
|
|
118
|
+
- Estimate based on average processing time (3-10s per item)
|
|
119
|
+
- Formula: `queue_depth * avg_processing_time`
|
|
120
|
+
- Add buffer for safety (e.g., +20%)
|
|
121
|
+
- Example: `queue.size * 8 + 2` seconds
|
|
122
|
+
|
|
123
|
+
**Evidence**: Rate limiter already tracks reset time (`src/ts/middleware/rate-limiter.ts:68-74`):
|
|
124
|
+
```typescript
|
|
125
|
+
getResetTime(clientId: string): number | null {
|
|
126
|
+
const entry = this.clients.get(clientId);
|
|
127
|
+
return entry ? entry.resetTime : null;
|
|
128
|
+
}
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
### Decision 4: Middleware Ordering
|
|
132
|
+
|
|
133
|
+
**Decision**: Establish clear middleware pipeline order.
|
|
134
|
+
|
|
135
|
+
**Recommended Pipeline**:
|
|
136
|
+
1. **CORS preflight** (if OPTIONS request)
|
|
137
|
+
2. **Rate limiting** (per-client request frequency)
|
|
138
|
+
3. **Schema validation** (400 if invalid)
|
|
139
|
+
4. **Queue capacity check** (429 if full)
|
|
140
|
+
5. **Queue enqueue** (returns 201 Accepted)
|
|
141
|
+
6. **Background processing** (non-blocking)
|
|
142
|
+
|
|
143
|
+
**Visual Flow**:
|
|
144
|
+
```
|
|
145
|
+
Request → CORS → Rate Limit → Validate → Queue Check → Enqueue → 201 Accepted
|
|
146
|
+
↓ ↓ ↓ ↓ ↓
|
|
147
|
+
204 No 429 Rate 400 Bad 429 Queue Background
|
|
148
|
+
Content Limited Request Full Processing
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
## Alternatives Considered
|
|
152
|
+
|
|
153
|
+
### Alternative 1: Rate Limit After Queuing
|
|
154
|
+
**Description**: Accept all valid requests into queue, rate limit at processing time.
|
|
155
|
+
|
|
156
|
+
**Pros**:
|
|
157
|
+
- Fairer first-come-first-serve
|
|
158
|
+
- Simpler client retry logic
|
|
159
|
+
|
|
160
|
+
**Cons**:
|
|
161
|
+
- Abusive clients fill shared queue
|
|
162
|
+
- Queue becomes weaponizable
|
|
163
|
+
- No protection against queue flooding
|
|
164
|
+
|
|
165
|
+
**Rejected**: Queue capacity is a shared server resource that must be protected from abuse.
|
|
166
|
+
|
|
167
|
+
### Alternative 2: Single 429 Response
|
|
168
|
+
**Description**: Use same "Rate limit exceeded" message for both rate limiting and queue full.
|
|
169
|
+
|
|
170
|
+
**Pros**:
|
|
171
|
+
- Simpler implementation
|
|
172
|
+
- Fewer error cases to handle
|
|
173
|
+
|
|
174
|
+
**Cons**:
|
|
175
|
+
- Clients can't distinguish client vs server issues
|
|
176
|
+
- No way to know appropriate retry strategy
|
|
177
|
+
- Poor observability for operators
|
|
178
|
+
|
|
179
|
+
**Rejected**: Clear error semantics are essential for client behavior and operational visibility.
|
|
180
|
+
|
|
181
|
+
### Alternative 3: 503 Service Unavailable for Queue Full
|
|
182
|
+
**Description**: Use 503 instead of 429 when queue is at capacity.
|
|
183
|
+
|
|
184
|
+
**Pros**:
|
|
185
|
+
- Semantically correct (server unavailable)
|
|
186
|
+
- Standard HTTP practice for overloaded services
|
|
187
|
+
|
|
188
|
+
**Cons**:
|
|
189
|
+
- Spec explicitly requires 429 (FR-006)
|
|
190
|
+
- Clients may treat 503 as downtime rather than retryable
|
|
191
|
+
- Inconsistent with specification
|
|
192
|
+
|
|
193
|
+
**Rejected**: Specification compliance is mandatory.
|
|
194
|
+
|
|
195
|
+
## Conclusions
|
|
196
|
+
|
|
197
|
+
### Recommended Approach
|
|
198
|
+
|
|
199
|
+
1. **Rate limiting operates first** at middleware layer, before queue operations
|
|
200
|
+
2. **Queue capacity check happens after** validation but before enqueue
|
|
201
|
+
3. **Distinct 429 responses** with clear error messages
|
|
202
|
+
4. **Retry-After header** added to both 429 responses with appropriate calculation
|
|
203
|
+
5. **Queue status endpoint** provides visibility into current load
|
|
204
|
+
|
|
205
|
+
### Implementation Priority
|
|
206
|
+
|
|
207
|
+
1. **P0**: Implement distinct 429 responses with error message differentiation
|
|
208
|
+
2. **P0**: Add Retry-After header to rate limiter 429 responses
|
|
209
|
+
3. **P0**: Add Retry-After header to queue full 429 responses
|
|
210
|
+
4. **P1**: Update ErrorResponse type to include optional retry_after field
|
|
211
|
+
5. **P2**: Add queue capacity metrics to /health endpoint
|
|
212
|
+
|
|
213
|
+
### Code Changes Required
|
|
214
|
+
|
|
215
|
+
1. **Update ErrorResponse type** (`src/ts/models/notification.ts:43-46`):
|
|
216
|
+
```typescript
|
|
217
|
+
export interface ErrorResponse {
|
|
218
|
+
status: "error";
|
|
219
|
+
message: string;
|
|
220
|
+
retry_after?: number; // seconds
|
|
221
|
+
}
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
2. **Update rate limiter** to return Retry-After header
|
|
225
|
+
3. **Update queue middleware** to return distinct 429 with Retry-After
|
|
226
|
+
4. **Update server.ts** to use new error response format
|
|
227
|
+
|
|
228
|
+
### Testing Strategy
|
|
229
|
+
|
|
230
|
+
1. Test rate limiting with concurrent requests
|
|
231
|
+
2. Test queue full scenario with burst > 100 requests
|
|
232
|
+
3. Verify Retry-After header accuracy
|
|
233
|
+
4. Test pipeline ordering (rate limit → validate → queue)
|
|
234
|
+
5. Verify distinct error messages for each failure mode
|
|
235
|
+
|
|
236
|
+
### Operational Considerations
|
|
237
|
+
|
|
238
|
+
- Rate limiter protects against abusive clients
|
|
239
|
+
- Queue capacity prevents server overload
|
|
240
|
+
- Both work together but serve different purposes
|
|
241
|
+
- Operators should monitor both metrics independently
|
|
242
|
+
- Client retry behavior differs between scenarios
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
**References**:
|
|
247
|
+
- Rate Limiter Implementation: `src/ts/middleware/rate-limiter.ts`
|
|
248
|
+
- Server Request Handler: `src/ts/server.ts:387-395`
|
|
249
|
+
- FIFO Queue Specification: `specs/001-fifo-notify-queue/spec.md`
|
|
250
|
+
- Notification Models: `src/ts/models/notification.ts`
|
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
# Research Report: Error Handling Best Practices for Queued Async Operations in TypeScript
|
|
2
|
+
|
|
3
|
+
**Date**: 2026-02-09
|
|
4
|
+
**Agent**: Research-Agent-v5
|
|
5
|
+
**Model**: Claude 4.6
|
|
6
|
+
|
|
7
|
+
## Summary
|
|
8
|
+
|
|
9
|
+
This research analyzes error handling patterns for queued asynchronous operations in TypeScript, specifically for TTS synthesis systems. Key findings focus on graceful degradation, validation timing, performant logging, and health monitoring strategies for production voice servers.
|
|
10
|
+
|
|
11
|
+
## Key Findings
|
|
12
|
+
|
|
13
|
+
### 1. Item Failure Handling
|
|
14
|
+
|
|
15
|
+
**Decision**: Use `Promise.allSettled()` with individual error wrapping
|
|
16
|
+
|
|
17
|
+
**Evidence**:
|
|
18
|
+
- Current codebase (`/src/ts/server.ts:188-202`) shows fallback pattern: MLX-audio TTS synthesis fails, falls back to macOS say command
|
|
19
|
+
- `Promise.allSettled()` allows continued processing after individual failures, unlike `Promise.all()` which fast-fails
|
|
20
|
+
- Each item should wrap its own error context (request ID, item index) for debugging
|
|
21
|
+
|
|
22
|
+
**Implementation Pattern**:
|
|
23
|
+
```typescript
|
|
24
|
+
const results = await Promise.allSettled(
|
|
25
|
+
queueItems.map(item => processWithFallback(item))
|
|
26
|
+
);
|
|
27
|
+
|
|
28
|
+
const failures = results.filter(r => r.status === 'rejected');
|
|
29
|
+
if (failures.length > 0) {
|
|
30
|
+
logger.warn(`Queue processing: ${failures.length}/${results.length} items failed`);
|
|
31
|
+
// Continue processing - don't throw
|
|
32
|
+
}
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
**Rationale**: TTS failures are transient (network, model issues, malformed text). Skipping failed items vs. failing entire queue prevents cascade failures. Current codebase already uses fallback pattern (`/src/ts/services/mlx-tts-client.ts:185-192`).
|
|
36
|
+
|
|
37
|
+
### 2. Validation Timing
|
|
38
|
+
|
|
39
|
+
**Decision**: Two-tier validation - early structural validation (before queue), deferred content validation (during processing)
|
|
40
|
+
|
|
41
|
+
**Evidence**:
|
|
42
|
+
- Current code validates early in `/src/ts/models/tts.ts:44-54`: `isValidTTSRequest()` checks text, voice, speed bounds
|
|
43
|
+
- Server validates at request entry: `/src/ts/server.ts:148-155` checks required fields before processing
|
|
44
|
+
- Some validations require runtime state: voice availability, model loaded status
|
|
45
|
+
|
|
46
|
+
**Implementation Strategy**:
|
|
47
|
+
```typescript
|
|
48
|
+
// Tier 1: Fast structural validation (before queue)
|
|
49
|
+
function validateRequestStructure(req: TTSRequest): ValidationResult {
|
|
50
|
+
if (!req.text?.trim()) return { valid: false, error: 'empty_text' };
|
|
51
|
+
if (req.speed < 0.5 || req.speed > 2) return { valid: false, error: 'invalid_speed' };
|
|
52
|
+
return { valid: true };
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
// Tier 2: Runtime validation (during processing)
|
|
56
|
+
async function validateWithRuntimeState(req: TTSRequest): Promise<ValidationResult> {
|
|
57
|
+
const voiceAvailable = await voiceLoader.hasVoice(req.voice);
|
|
58
|
+
if (!voiceAvailable) return { valid: false, error: 'voice_unavailable' };
|
|
59
|
+
return { valid: true };
|
|
60
|
+
}
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
**Rationale**: Early validation rejects obviously invalid requests quickly (< 1ms). Deferred validation handles state-dependent checks (voice availability, model health) that may change between queue submission and processing. This matches current pattern: structural validation at `/src/ts/server.ts:148-155`, runtime health check at `/src/ts/models/health.ts:67-87`.
|
|
64
|
+
|
|
65
|
+
### 3. Error Logging Performance
|
|
66
|
+
|
|
67
|
+
**Decision**: Sampling-based error logging with asynchronous write-behind
|
|
68
|
+
|
|
69
|
+
**Evidence**:
|
|
70
|
+
- Current logger: `/src/ts/utils/logger.ts` (implementation not shown in codebase but imported everywhere)
|
|
71
|
+
- Rate limiter pattern at `/src/ts/middleware/rate-limiter.ts:1-195` shows token bucket algorithm for rate control
|
|
72
|
+
- Bun's `console.log()` is synchronous and can block event loop
|
|
73
|
+
|
|
74
|
+
**Implementation Pattern**:
|
|
75
|
+
```typescript
|
|
76
|
+
// Async logging buffer (non-blocking)
|
|
77
|
+
const logBuffer: LogEntry[] = [];
|
|
78
|
+
let logFlushScheduled = false;
|
|
79
|
+
|
|
80
|
+
function queueLog(entry: LogEntry): void {
|
|
81
|
+
logBuffer.push(entry);
|
|
82
|
+
if (!logFlushScheduled) {
|
|
83
|
+
logFlushScheduled = true;
|
|
84
|
+
// Flush on next tick (non-blocking)
|
|
85
|
+
setTimeout(flushLogs, 0).unref();
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
// Sampling for high-frequency errors
|
|
90
|
+
const errorCounts = new Map<string, number>();
|
|
91
|
+
function shouldLogError(errorType: string): boolean {
|
|
92
|
+
const count = errorCounts.get(errorType) || 0;
|
|
93
|
+
errorCounts.set(errorType, count + 1);
|
|
94
|
+
|
|
95
|
+
// Log first occurrence, then sample
|
|
96
|
+
if (count === 0) return true;
|
|
97
|
+
if (count % 100 === 0) return true; // Every 100th occurrence
|
|
98
|
+
return false;
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
**Rationale**: Synchronous logging blocks TTS processing. Buffering + sampling reduces I/O overhead by ~90% while preserving error visibility. Matches rate limiter's cleanup interval pattern (`/src/ts/middleware/rate-limiter.ts:124-133`).
|
|
103
|
+
|
|
104
|
+
### 4. Health Monitoring
|
|
105
|
+
|
|
106
|
+
**Decision**: Multi-tier health tracking with circuit breaker pattern
|
|
107
|
+
|
|
108
|
+
**Evidence**:
|
|
109
|
+
- Current health model: `/src/ts/models/health.ts:1-45` tracks status, voice_system, model_loaded
|
|
110
|
+
- Health check endpoint: `/src/ts/server.ts:214-244` checks MLX-audio availability, falls back to macOS Say
|
|
111
|
+
- Rate limiter tracks client stats: `/src/ts/middleware/rate-limiter.ts:157-168`
|
|
112
|
+
|
|
113
|
+
**Implementation Pattern**:
|
|
114
|
+
```typescript
|
|
115
|
+
interface QueueHealthMetrics {
|
|
116
|
+
pendingItems: number;
|
|
117
|
+
processedItems: number;
|
|
118
|
+
failedItems: number;
|
|
119
|
+
avgProcessingTimeMs: number;
|
|
120
|
+
lastErrorTime: number;
|
|
121
|
+
consecutiveFailures: number;
|
|
122
|
+
circuitState: 'closed' | 'open' | 'half-open';
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
// Circuit breaker: stop queuing after N consecutive failures
|
|
126
|
+
function shouldAcceptQueueItem(metrics: QueueHealthMetrics): boolean {
|
|
127
|
+
if (metrics.consecutiveFailures >= 5) {
|
|
128
|
+
metrics.circuitState = 'open';
|
|
129
|
+
return false;
|
|
130
|
+
}
|
|
131
|
+
return true;
|
|
132
|
+
}
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
**Rationale**: Health monitoring enables auto-remediation (circuit breaker prevents cascading failures). Current health check (`/src/ts/server.ts:214`) doesn't track queue depth or error rates. Adding queue metrics gives operators visibility into backlogs and TTS degradation.
|
|
136
|
+
|
|
137
|
+
## Architecture/Patterns Identified
|
|
138
|
+
|
|
139
|
+
### Current Error Handling Patterns in Codebase
|
|
140
|
+
|
|
141
|
+
1. **Fallback Hierarchy** (`/src/ts/services/mlx-tts-client.ts:185-192`)
|
|
142
|
+
- Primary: MLX-audio streaming TTS
|
|
143
|
+
- Fallback: macOS `say` command
|
|
144
|
+
- Pattern: Try-catch with fallback, log warning
|
|
145
|
+
|
|
146
|
+
2. **Early Validation** (`/src/ts/server.ts:148-155`)
|
|
147
|
+
- Validate required fields before processing
|
|
148
|
+
- Sanitize input (title, message)
|
|
149
|
+
- Return error response vs. throwing
|
|
150
|
+
|
|
151
|
+
3. **Rate Limiting** (`/src/ts/middleware/rate-limiter.ts:1-195`)
|
|
152
|
+
- Token bucket algorithm
|
|
153
|
+
- Per-client tracking
|
|
154
|
+
- Periodic cleanup of expired entries
|
|
155
|
+
|
|
156
|
+
4. **Health Reporting** (`/src/ts/models/health.ts:1-45`)
|
|
157
|
+
- Status enum: healthy/degraded/unhealthy
|
|
158
|
+
- Voice system type tracking
|
|
159
|
+
- Model loaded state
|
|
160
|
+
|
|
161
|
+
### Recommended Queue Error Handling Architecture
|
|
162
|
+
|
|
163
|
+
```
|
|
164
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
165
|
+
│ Request Entry │
|
|
166
|
+
│ - Early validation (structure, schema) │
|
|
167
|
+
│ - Rate limit check │
|
|
168
|
+
│ - Add to queue │
|
|
169
|
+
└─────────────────────────────────────────────────────────────┘
|
|
170
|
+
│
|
|
171
|
+
▼
|
|
172
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
173
|
+
│ Queue Processor │
|
|
174
|
+
│ - Runtime validation (voice availability, model health) │
|
|
175
|
+
│ - Process with Promise.allSettled() │
|
|
176
|
+
│ - Individual error wrapping │
|
|
177
|
+
│ - Continue on failure │
|
|
178
|
+
└─────────────────────────────────────────────────────────────┘
|
|
179
|
+
│
|
|
180
|
+
▼
|
|
181
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
182
|
+
│ Health Monitoring │
|
|
183
|
+
│ - Track: queue depth, error rate, consecutive failures │
|
|
184
|
+
│ - Circuit breaker after N failures │
|
|
185
|
+
│ - Async sampling-based logging │
|
|
186
|
+
└─────────────────────────────────────────────────────────────┘
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
## Conclusions
|
|
190
|
+
|
|
191
|
+
### Decision Summary
|
|
192
|
+
|
|
193
|
+
1. **Item Failure**: Use `Promise.allSettled()` + fallback pattern
|
|
194
|
+
2. **Validation Timing**: Two-tier (early structural + deferred runtime)
|
|
195
|
+
3. **Error Logging**: Async buffer with sampling (non-blocking)
|
|
196
|
+
4. **Health Monitoring**: Circuit breaker + queue metrics
|
|
197
|
+
|
|
198
|
+
### Alternatives Considered
|
|
199
|
+
|
|
200
|
+
| Approach | Pros | Cons | Verdict |
|
|
201
|
+
|----------|------|------|---------|
|
|
202
|
+
| `Promise.all()` (fast-fail) | Simple, fails fast | Stops queue on first error | Rejected: TTS failures are transient |
|
|
203
|
+
| `Promise.race()` (any success) | Returns fastest response | Doesn't wait for all items | Rejected: Need all items processed |
|
|
204
|
+
| Retry loops | Handles transient errors | Can cascade failures | Use sparingly: max 1 retry |
|
|
205
|
+
| Synchronous logging | Simple ordering | Blocks event loop | Rejected: Performance impact |
|
|
206
|
+
| Full validation upfront | Fail fast | State may change | Rejected: Need runtime checks |
|
|
207
|
+
|
|
208
|
+
### Recommendations
|
|
209
|
+
|
|
210
|
+
1. **Add queue metrics** to health status: queue depth, error rate, avg processing time
|
|
211
|
+
2. **Implement circuit breaker**: stop accepting requests after 5 consecutive TTS failures
|
|
212
|
+
3. **Add structured error types**: `TTSError` with error codes (voice_unavailable, synthesis_failed, timeout)
|
|
213
|
+
4. **Sample high-frequency errors**: Log first + every 100th occurrence
|
|
214
|
+
5. **Expose queue health**: Add `/queue-health` endpoint with metrics
|
|
215
|
+
|
|
216
|
+
### Integration with Existing Code
|
|
217
|
+
|
|
218
|
+
Current codebase already has:
|
|
219
|
+
- Fallback pattern (`/src/ts/server.ts:188-202`)
|
|
220
|
+
- Health model (`/src/ts/models/health.ts`)
|
|
221
|
+
- Rate limiter with stats (`/src/ts/middleware/rate-limiter.ts`)
|
|
222
|
+
|
|
223
|
+
Missing:
|
|
224
|
+
- Queue-specific metrics (depth, throughput)
|
|
225
|
+
- Circuit breaker logic
|
|
226
|
+
- Async logging buffer
|
|
227
|
+
- Structured error types
|
|
228
|
+
|
|
229
|
+
---
|
|
230
|
+
|
|
231
|
+
**Files Referenced**:
|
|
232
|
+
- `/src/ts/models/tts.ts` - TTS request validation
|
|
233
|
+
- `/src/ts/services/mlx-tts-client.ts` - MLX-audio client with fallback
|
|
234
|
+
- `/src/ts/server.ts` - Request handling with error catching
|
|
235
|
+
- `/src/ts/models/health.ts` - Health status types
|
|
236
|
+
- `/src/ts/middleware/rate-limiter.ts` - Rate limiting implementation
|
|
@@ -75,20 +75,23 @@ export class NotificationQueue {
|
|
|
75
75
|
};
|
|
76
76
|
}
|
|
77
77
|
|
|
78
|
-
// Validate voice_id
|
|
78
|
+
// Validate voice_id - empty/missing voice_id is allowed (uses default)
|
|
79
79
|
const voiceLoader = getVoiceLoader();
|
|
80
80
|
try {
|
|
81
|
-
const
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
const
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
81
|
+
const voiceId = request.voice_id || request.voice_name || "";
|
|
82
|
+
// Skip validation for empty voice_id - will use default
|
|
83
|
+
if (voiceId) {
|
|
84
|
+
const voiceInfo = voiceLoader.getVoiceInfo(voiceId);
|
|
85
|
+
if (!voiceInfo) {
|
|
86
|
+
// Check if it's a known voice ID from the voice loader
|
|
87
|
+
const availableVoices = await voiceLoader.getAvailableVoices();
|
|
88
|
+
if (!availableVoices.includes(voiceId) && voiceId !== "marrvin" && voiceId !== "marlin" && voiceId !== "daniel") {
|
|
89
|
+
return {
|
|
90
|
+
success: false,
|
|
91
|
+
statusCode: 400,
|
|
92
|
+
message: `Invalid voice_id: ${voiceId}. Available voices: ${availableVoices.join(", ")}`,
|
|
93
|
+
};
|
|
94
|
+
}
|
|
92
95
|
}
|
|
93
96
|
}
|
|
94
97
|
} catch (error) {
|