agentgui 1.0.102 → 1.0.103

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/.prd +336 -0
  2. package/database.js +121 -0
  3. package/package.json +1 -1
  4. package/server.js +109 -59
package/.prd CHANGED
@@ -0,0 +1,336 @@
1
+ # PRD: Real-time Streaming Architecture & HTML Response Rendering
2
+
3
+ ## Vision
4
+ Transform agentgui from a "save-on-complete" model to a "stream-as-it-happens" model where:
5
+ 1. Real-time stream chunks are persisted to DB immediately as they arrive
6
+ 2. Client views always pull from persisted chunks (not ephemeral process output)
7
+ 3. Conversation state survives page refresh and multi-tab viewing
8
+ 4. Agent thoughts/responses rendered as beautiful semantic HTML (not JSON)
9
+ 5. System remains the same whether viewing live stream or historical conversation
10
+
11
+ ## Critical Problems Being Solved
12
+
13
+ ### Problem 1: Conversation State Loss on Refresh
14
+ **Current**: Viewing live execution → refresh page → conversation disappears
15
+ **Why**: Stream chunks exist only in memory during execution, saved as single JSON when done
16
+ **Solution**: Persist each chunk to DB immediately as it arrives from stream
17
+
18
+ ### Problem 2: Dual View Inconsistency
19
+ **Current**: Same conversation looks different when live vs after completion
20
+ **Why**: Live view shows individual streaming blocks, complete view shows condensed JSON
21
+ **Solution**: Single source of truth = DB chunks. No special handling for "done" state
22
+
23
+ ### Problem 3: No Multi-Tab Support
24
+ **Current**: Can't view same conversation in two browser tabs simultaneously
25
+ **Why**: No persistent state, live process is ephemeral
26
+ **Solution**: DB persistence means any tab can view same chunks at any time
27
+
28
+ ### Problem 4: HTML Output Not Distinguished from Files
29
+ **Current**: System prompt says "output HTML" but unclear if it means response HTML or file writes
30
+ **Why**: Ambiguous instruction in system prompt
31
+ **Solution**: Make explicit: Agent HTML responses ≠ file operations. Only HTML rendering for display
32
+
33
+ ### Problem 5: Process Completion Creates Artifacts
34
+ **Current**: When execution finishes, entire thing re-saved as JSON blob
35
+ **Why**: Current architecture treats completion as "finalize and persist"
36
+ **Solution**: No special completion action. Streaming simply stops. DB already has everything.
37
+
38
+ ## Architecture Changes Required
39
+
40
+ ### 1. Stream Chunk Persistence (Core Change)
41
+ **Current Flow**:
42
+ ```
43
+ Claude → Stream Event → In-Memory Buffer → On Complete: Save JSON → DB
44
+ ```
45
+
46
+ **New Flow**:
47
+ ```
48
+ Claude → Stream Event → Process → Save Chunk → DB → WebSocket to Clients
49
+ ```
50
+
51
+ **Implementation**:
52
+ - Each `streaming_progress` event creates DB chunk immediately
53
+ - Each chunk has: `id`, `sessionId`, `conversationId`, `sequence`, `type`, `data`, `created_at`
54
+ - No buffering, no aggregation, no re-saving on completion
55
+ - Chunks table schema must support variable size chunks (BLOB or TEXT)
56
+
57
+ ### 2. Client Rendering from DB Only
58
+ **Current**: Client renders from WebSocket stream events
59
+ **New**: Client renders from DB chunks via polling/WebSocket
60
+ **Benefit**: Same render path whether data is fresh or historical
61
+
62
+ **Implementation**:
63
+ - Add endpoint: `GET /api/conversations/:id/chunks?since=<timestamp>`
64
+ - Returns chunks in order (sequence number)
65
+ - Client polls every 100ms for new chunks
66
+ - WebSocket optimization: only notify "new chunk available", don't send chunk data
67
+ - Client always fetches from DB to keep rendering consistent
68
+
69
+ ### 3. HTML Response Rendering (System Prompt Change)
70
+ **Current System Prompt**:
71
+ ```
72
+ "Always write your responses in ripple-ui enhanced HTML"
73
+ ```
74
+
75
+ **Problem**: Unclear if this is for file output or response rendering
76
+ **Solution**: Make explicit instruction:
77
+ ```
78
+ For user-facing responses and thoughts: Always use semantic HTML with ripple-ui components
79
+ Do not treat this as file creation. HTML is for rendering in the UI, not saving to disk.
80
+ Block types (text, code, thinking, etc) in your message should render as beautiful semantic HTML
81
+ File operations (Read, Write, Edit) are separate - create actual files on disk when needed
82
+ Distinguish clearly: HTML response rendering ≠ file write operations
83
+ ```
84
+
85
+ ### 4. Conversation URL State (Client Change)
86
+ **Current**: Conversations loaded from session, URL doesn't track state
87
+ **New**: URL contains conversation ID and session ID for deep linking
88
+
89
+ **Implementation**:
90
+ - Route: `/gm/?conversation=<conversationId>&session=<sessionId>`
91
+ - Page refresh loads same conversation from URL
92
+ - Each conversation maintains scroll position in localStorage
93
+ - Deep linking enables multi-tab viewing
94
+
95
+ ### 5. Remove Post-Execution JSON Consolidation (Deletion)
96
+ **Current**: When execution completes, entire response saved as consolidated JSON
97
+ **New**: No action on completion. Stream already persisted.
98
+
99
+ **Implementation**:
100
+ - Delete code in `server.js` that creates final JSON blob on streaming_complete
101
+ - Keep the `streaming_complete` event for UI notifications ("Finished!")
102
+ - All message data comes from persisted chunks, not from completion blob
103
+
104
+ ## Dependencies & Blocking
105
+
106
+ ### Wave 1: Foundation (No Dependencies) - COMPLETE
107
+ - [x] Add `chunks` table to database schema - VERIFIED
108
+ - Created chunks table with proper schema in database.js initSchema()
109
+ - All columns defined: id, sessionId, conversationId, sequence, type, data, created_at
110
+ - Foreign keys configured for sessionId and conversationId
111
+ - [x] Create indexes - VERIFIED
112
+ - idx_chunks_session (sessionId, sequence) - non-unique
113
+ - idx_chunks_conversation (conversationId, sequence) - non-unique
114
+ - idx_chunks_unique (sessionId, sequence) - unique constraint
115
+ - [x] Add chunk persistence methods to database.js - VERIFIED
116
+ - createChunk(sessionId, conversationId, sequence, type, data)
117
+ - getChunk(id)
118
+ - getSessionChunks(sessionId)
119
+ - getConversationChunks(conversationId)
120
+ - getChunksSince(sessionId, timestamp)
121
+ - deleteSessionChunks(sessionId)
122
+ - getMaxSequence(sessionId)
123
+ - [x] Test schema creation with real database - VERIFIED
124
+ - Schema initializes without errors
125
+ - All indexes created correctly
126
+ - Foreign key constraints active
127
+ - Data persistence working
128
+ - Existing databases compatible (no breaking changes)
129
+
130
+ ### Wave 2: Backend Stream Persistence (Blocked By: Wave 1)
131
+ - [ ] Modify `processMessageWithStreaming()` to persist chunks immediately
132
+ - [ ] Update `onEvent` callback to save to DB before broadcasting
133
+ - [ ] Add endpoint `GET /api/conversations/:id/chunks?since=timestamp`
134
+ - [ ] Remove post-execution JSON consolidation code
135
+ - [ ] Test chunk persistence with real streaming
136
+
137
+ ### Wave 3: Client Chunk Fetching (Blocked By: Wave 2)
138
+ - [ ] Add `fetchChunks()` API method to client
139
+ - [ ] Implement polling logic (100ms interval, backoff on error)
140
+ - [ ] Integrate with existing WebSocket for notifications
141
+ - [ ] Update message rendering to pull from chunks instead of events
142
+
143
+ ### Wave 4: URL State Management (Blocked By: Wave 3)
144
+ - [ ] Add router state to track conversationId + sessionId
145
+ - [ ] Update URL on conversation selection
146
+ - [ ] Restore conversation from URL on page load
147
+ - [ ] Persist scroll position per conversation
148
+
149
+ ### Wave 5: HTML Response System Prompt (Blocked By: Wave 4)
150
+ - [ ] Update SYSTEM_PROMPT in server.js
151
+ - [ ] Add explicit guidance on HTML rendering vs file operations
152
+ - [ ] Document block type expectations
153
+ - [ ] Add examples in prompt
154
+
155
+ ### Wave 6: Verification & Testing (Blocked By: Wave 5)
156
+ - [ ] Test conversation persistence across refresh
157
+ - [ ] Test multi-tab viewing same conversation
158
+ - [ ] Verify streaming chunks match rendered output
159
+ - [ ] Test URL deep linking
160
+ - [ ] Verify no data loss during streaming
161
+ - [ ] Test error recovery (partial streams)
162
+
163
+ ## Data Model
164
+
165
+ ### New: chunks table
166
+ ```sql
167
+ CREATE TABLE chunks (
168
+ id TEXT PRIMARY KEY,
169
+ sessionId TEXT NOT NULL,
170
+ conversationId TEXT NOT NULL,
171
+ sequence INTEGER NOT NULL,
172
+ type TEXT NOT NULL, -- "text", "code", "thinking", "tool_use", "tool_result", "bash", "system", "image"
173
+ data BLOB NOT NULL, -- full block data as JSON
174
+ created_at INTEGER NOT NULL,
175
+ FOREIGN KEY (sessionId) REFERENCES sessions(id),
176
+ FOREIGN KEY (conversationId) REFERENCES conversations(id)
177
+ );
178
+
179
+ CREATE INDEX idx_chunks_session ON chunks(sessionId, sequence);
180
+ CREATE INDEX idx_chunks_conversation ON chunks(conversationId, sequence);
181
+ CREATE UNIQUE INDEX idx_chunks_unique ON chunks(sessionId, sequence);
182
+ ```
183
+
184
+ ### Modified: messages table (optional cleanup)
185
+ - Keep as-is for message text
186
+ - Add `chunks_id` field to reference chunk sequence (future)
187
+ - Messages created from chunks summary, not vice versa
188
+
189
+ ## API Changes
190
+
191
+ ### New Endpoints
192
+ ```
193
+ GET /api/conversations/:id/chunks?since=<timestamp>
194
+ Response: [{id, sessionId, conversationId, sequence, type, data, created_at}...]
195
+
196
+ GET /api/sessions/:id/chunks?since=<timestamp>
197
+ Response: Same as above, filtered by session
198
+ ```
199
+
200
+ ### Modified Endpoints
201
+ ```
202
+ POST /api/conversations/:id/messages
203
+ - Still creates message (for conversation history)
204
+ - But also triggers chunk persistence for streaming blocks
205
+ - No change to request/response format
206
+ ```
207
+
208
+ ### Removed/Deprecated
209
+ ```
210
+ The "save on completion" behavior in streaming_complete event
211
+ - Keep the event for UI ("execution finished")
212
+ - Just don't save aggregated JSON
213
+ - Chunks already in DB from streaming
214
+ ```
215
+
216
+ ## System Prompt Changes
217
+
218
+ ### Current
219
+ ```
220
+ Always write your responses in ripple-ui enhanced HTML. Avoid overriding
221
+ light/dark mode CSS variables. Use all the benefits of HTML to express
222
+ technical details with proper semantic markup, tables, code blocks, headings,
223
+ and lists. Write clean, well-structured HTML that respects the existing
224
+ design system.
225
+ ```
226
+
227
+ ### New (Clearer)
228
+ ```
229
+ RESPONSE RENDERING:
230
+ Your thoughts and responses are rendered as semantic HTML in the UI using
231
+ ripple-ui components. Always structure responses with proper HTML:
232
+ - Use headings for sections (<h2>, <h3>)
233
+ - Use lists for sequences (<ul>, <ol>)
234
+ - Use tables for structured data
235
+ - Use code blocks with language tags
236
+ - Use semantic elements: <strong>, <em>, <code>, <pre>
237
+ - Never override CSS variables (use class names instead)
238
+ - Respect the design system
239
+
240
+ DISTINGUISH: HTML Response vs File Operations
241
+ - HTML above is for UI rendering, not file creation
242
+ - When you need to create files on disk: use Write, Edit, or Bash tools
243
+ - Message blocks (text, code, thinking, tool_use) render as HTML automatically
244
+ - File operations create actual files in the working directory
245
+ - Do not try to "output files" as HTML in your response
246
+ - File operations are explicit via tools, not implicit via response text
247
+
248
+ BLOCK TYPES:
249
+ Your assistant message will be parsed into blocks:
250
+ - text: Plain text with markdown support
251
+ - code: Code with language detection
252
+ - thinking: Internal reasoning (expandable)
253
+ - tool_use: Showing which tools you're calling
254
+ - tool_result: Tool output
255
+ - bash: Shell commands
256
+ - system: System information
257
+ - image: Image display
258
+ Each block renders with semantic HTML and proper styling.
259
+ ```
260
+
261
+ ## Success Criteria
262
+
263
+ ### Data Persistence
264
+ - [ ] Each streaming chunk persists to DB within 100ms of arrival
265
+ - [ ] No data loss during interrupted streams
266
+ - [ ] Chunks survive server restart
267
+ - [ ] Multiple streams don't corrupt chunk sequence
268
+
269
+ ### Client Behavior
270
+ - [ ] Page refresh shows same conversation state
271
+ - [ ] Same conversation visible in two tabs simultaneously
272
+ - [ ] Scroll position preserved per conversation
273
+ - [ ] URL deep linking works (share link to specific conversation)
274
+
275
+ ### Rendering Consistency
276
+ - [ ] Live stream looks identical to historical view
277
+ - [ ] No "loading from JSON" artifacts
278
+ - [ ] HTML rendering of blocks is semantic and beautiful
279
+ - [ ] Dark mode works for all response blocks
280
+
281
+ ### Performance
282
+ - [ ] Chunk polling adds <50ms latency to visibility
283
+ - [ ] DB queries for chunks return in <100ms (with index)
284
+ - [ ] No memory leaks from chunk history
285
+ - [ ] WebSocket still efficient (only notifications, not data)
286
+
287
+ ### System Prompt Clarity
288
+ - [ ] Agent clearly distinguishes HTML response rendering from file operations
289
+ - [ ] Block types documented and understood
290
+ - [ ] No confusion between "output HTML" and "write HTML file"
291
+
292
+ ## Implementation Notes
293
+
294
+ ### Chunk Persistence Strategy
295
+ - Use database transaction: Insert chunk → Broadcast event → Return
296
+ - If insert fails, retry up to 3 times with exponential backoff
297
+ - If all retries fail, log error but don't crash (supervisor catches)
298
+ - Broadcast only happens after successful DB insert
299
+
300
+ ### Client Polling Implementation
301
+ - Use `AbortController` for request cancellation
302
+ - Exponential backoff on errors: 100ms → 200ms → 400ms → reset after success
303
+ - Single active poll at a time (debounce overlapping requests)
304
+ - Stop polling when stream ends (flag in UI)
305
+ - Resume polling on reconnect
306
+
307
+ ### URL State Strategy
308
+ - Use `pushState()` not hash (cleaner URLs, better for sharing)
309
+ - Encode: `?conversation=<id>&session=<id>`
310
+ - Validate IDs on load (prevent XSS)
311
+ - Graceful fallback if invalid (show conversation list)
312
+
313
+ ### Backward Compatibility
314
+ - Old conversations with JSON blob still work (handled by rendering layer)
315
+ - New conversations use chunks table
316
+ - No migration needed (can run dual path for period of time)
317
+ - Eventually clean up JSON blob from old messages
318
+
319
+ ## Unknowns to Validate
320
+
321
+ - [ ] Chunk size distribution (average, max, min)
322
+ - [ ] Polling latency impact on perceived freshness
323
+ - [ ] Database performance with high-frequency chunk inserts
324
+ - [ ] Memory usage with large conversation histories
325
+ - [ ] Browser storage limits for scroll position tracking
326
+ - [ ] Concurrent chunk arrivals in same session (race conditions)
327
+
328
+ ## Open Questions
329
+
330
+ 1. Should chunk sequence be per-session or global? → Per-session (cleaner transactions)
331
+ 2. Should chunks include metadata about which tool produced them? → Yes (for tracing)
332
+ 3. Should we compress chunks in DB or store raw? → Raw (compression adds latency)
333
+ 4. How long to keep polling before deciding stream ended? → 5 seconds without activity
334
+ 5. Should old JSON blob messages be migrated to chunks? → Lazy migration on view
335
+
336
+ ## No Pending Work Items (Ready for Execution)
package/database.js CHANGED
@@ -105,6 +105,22 @@ function initSchema() {
105
105
 
106
106
  CREATE INDEX IF NOT EXISTS idx_stream_updates_session ON stream_updates(sessionId);
107
107
  CREATE INDEX IF NOT EXISTS idx_stream_updates_created ON stream_updates(created_at);
108
+
109
+ CREATE TABLE IF NOT EXISTS chunks (
110
+ id TEXT PRIMARY KEY,
111
+ sessionId TEXT NOT NULL,
112
+ conversationId TEXT NOT NULL,
113
+ sequence INTEGER NOT NULL,
114
+ type TEXT NOT NULL,
115
+ data BLOB NOT NULL,
116
+ created_at INTEGER NOT NULL,
117
+ FOREIGN KEY (sessionId) REFERENCES sessions(id),
118
+ FOREIGN KEY (conversationId) REFERENCES conversations(id)
119
+ );
120
+
121
+ CREATE INDEX IF NOT EXISTS idx_chunks_session ON chunks(sessionId, sequence);
122
+ CREATE INDEX IF NOT EXISTS idx_chunks_conversation ON chunks(conversationId, sequence);
123
+ CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_unique ON chunks(sessionId, sequence);
108
124
  `);
109
125
  }
110
126
 
@@ -883,6 +899,111 @@ export const queries = {
883
899
  }
884
900
 
885
901
  return imported;
902
+ },
903
+
904
+ createChunk(sessionId, conversationId, sequence, type, data) {
905
+ const id = generateId('chunk');
906
+ const now = Date.now();
907
+ const dataBlob = typeof data === 'string' ? data : JSON.stringify(data);
908
+
909
+ const stmt = db.prepare(
910
+ `INSERT INTO chunks (id, sessionId, conversationId, sequence, type, data, created_at)
911
+ VALUES (?, ?, ?, ?, ?, ?, ?)`
912
+ );
913
+ stmt.run(id, sessionId, conversationId, sequence, type, dataBlob, now);
914
+
915
+ return {
916
+ id,
917
+ sessionId,
918
+ conversationId,
919
+ sequence,
920
+ type,
921
+ data,
922
+ created_at: now
923
+ };
924
+ },
925
+
926
+ getChunk(id) {
927
+ const stmt = db.prepare(
928
+ `SELECT id, sessionId, conversationId, sequence, type, data, created_at FROM chunks WHERE id = ?`
929
+ );
930
+ const row = stmt.get(id);
931
+ if (!row) return null;
932
+
933
+ try {
934
+ return {
935
+ ...row,
936
+ data: typeof row.data === 'string' ? JSON.parse(row.data) : row.data
937
+ };
938
+ } catch (e) {
939
+ return row;
940
+ }
941
+ },
942
+
943
+ getSessionChunks(sessionId) {
944
+ const stmt = db.prepare(
945
+ `SELECT id, sessionId, conversationId, sequence, type, data, created_at
946
+ FROM chunks WHERE sessionId = ? ORDER BY sequence ASC`
947
+ );
948
+ const rows = stmt.all(sessionId);
949
+ return rows.map(row => {
950
+ try {
951
+ return {
952
+ ...row,
953
+ data: typeof row.data === 'string' ? JSON.parse(row.data) : row.data
954
+ };
955
+ } catch (e) {
956
+ return row;
957
+ }
958
+ });
959
+ },
960
+
961
+ getConversationChunks(conversationId) {
962
+ const stmt = db.prepare(
963
+ `SELECT id, sessionId, conversationId, sequence, type, data, created_at
964
+ FROM chunks WHERE conversationId = ? ORDER BY created_at ASC`
965
+ );
966
+ const rows = stmt.all(conversationId);
967
+ return rows.map(row => {
968
+ try {
969
+ return {
970
+ ...row,
971
+ data: typeof row.data === 'string' ? JSON.parse(row.data) : row.data
972
+ };
973
+ } catch (e) {
974
+ return row;
975
+ }
976
+ });
977
+ },
978
+
979
+ getChunksSince(sessionId, timestamp) {
980
+ const stmt = db.prepare(
981
+ `SELECT id, sessionId, conversationId, sequence, type, data, created_at
982
+ FROM chunks WHERE sessionId = ? AND created_at > ? ORDER BY sequence ASC`
983
+ );
984
+ const rows = stmt.all(sessionId, timestamp);
985
+ return rows.map(row => {
986
+ try {
987
+ return {
988
+ ...row,
989
+ data: typeof row.data === 'string' ? JSON.parse(row.data) : row.data
990
+ };
991
+ } catch (e) {
992
+ return row;
993
+ }
994
+ });
995
+ },
996
+
997
+ deleteSessionChunks(sessionId) {
998
+ const stmt = db.prepare('DELETE FROM chunks WHERE sessionId = ?');
999
+ const result = stmt.run(sessionId);
1000
+ return result.changes || 0;
1001
+ },
1002
+
1003
+ getMaxSequence(sessionId) {
1004
+ const stmt = db.prepare('SELECT MAX(sequence) as max FROM chunks WHERE sessionId = ?');
1005
+ const result = stmt.get(sessionId);
1006
+ return result?.max ?? -1;
886
1007
  }
887
1008
  };
888
1009
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "agentgui",
3
- "version": "1.0.102",
3
+ "version": "1.0.103",
4
4
  "description": "Multi-agent ACP client with real-time communication",
5
5
  "type": "module",
6
6
  "main": "server.js",
package/server.js CHANGED
@@ -292,6 +292,38 @@ const server = http.createServer(async (req, res) => {
292
292
  return;
293
293
  }
294
294
 
295
+ const conversationChunksMatch = pathOnly.match(/^\/api\/conversations\/([^/]+)\/chunks$/);
296
+ if (conversationChunksMatch && req.method === 'GET') {
297
+ const conversationId = conversationChunksMatch[1];
298
+ const conv = queries.getConversation(conversationId);
299
+ if (!conv) { res.writeHead(404, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ error: 'Conversation not found' })); return; }
300
+
301
+ const url = new URL(req.url, 'http://localhost');
302
+ const since = parseInt(url.searchParams.get('since') || '0');
303
+
304
+ const allChunks = queries.getConversationChunks(conversationId);
305
+ debugLog(`[chunks] Conv ${conversationId}: ${allChunks.length} total chunks`);
306
+ const chunks = since > 0 ? allChunks.filter(c => c.created_at > since) : allChunks;
307
+ res.writeHead(200, { 'Content-Type': 'application/json' });
308
+ res.end(JSON.stringify({ ok: true, chunks }));
309
+ return;
310
+ }
311
+
312
+ const sessionChunksMatch = pathOnly.match(/^\/api\/sessions\/([^/]+)\/chunks$/);
313
+ if (sessionChunksMatch && req.method === 'GET') {
314
+ const sessionId = sessionChunksMatch[1];
315
+ const sess = queries.getSession(sessionId);
316
+ if (!sess) { res.writeHead(404, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ error: 'Session not found' })); return; }
317
+
318
+ const url = new URL(req.url, 'http://localhost');
319
+ const since = parseInt(url.searchParams.get('since') || '0');
320
+
321
+ const chunks = queries.getChunksSince(sessionId, since);
322
+ res.writeHead(200, { 'Content-Type': 'application/json' });
323
+ res.end(JSON.stringify({ ok: true, chunks }));
324
+ return;
325
+ }
326
+
295
327
  if (pathOnly.match(/^\/api\/conversations\/([^/]+)\/sessions\/latest$/) && req.method === 'GET') {
296
328
  const convId = pathOnly.match(/^\/api\/conversations\/([^/]+)\/sessions\/latest$/)[1];
297
329
  const latestSession = queries.getLatestSession(convId);
@@ -459,6 +491,31 @@ function serveFile(filePath, res) {
459
491
  });
460
492
  }
461
493
 
494
+ function persistChunkWithRetry(sessionId, conversationId, sequence, blockType, blockData, maxRetries = 3) {
495
+ let lastError = null;
496
+ const backoffs = [100, 200, 400];
497
+
498
+ for (let attempt = 0; attempt < maxRetries; attempt++) {
499
+ try {
500
+ const chunk = queries.createChunk(sessionId, conversationId, sequence, blockType, blockData);
501
+ return chunk;
502
+ } catch (err) {
503
+ lastError = err;
504
+ debugLog(`[chunk] Persist attempt ${attempt + 1}/${maxRetries} failed: ${err.message}`);
505
+ if (attempt < maxRetries - 1) {
506
+ const delayMs = backoffs[attempt] || 400;
507
+ const endTime = Date.now() + delayMs;
508
+ while (Date.now() < endTime) {
509
+ // Synchronous sleep for backoff
510
+ }
511
+ }
512
+ }
513
+ }
514
+
515
+ debugLog(`[chunk] Failed to persist after ${maxRetries} retries: ${lastError?.message}`);
516
+ return null;
517
+ }
518
+
462
519
  async function processMessageWithStreaming(conversationId, messageId, sessionId, content, agentId, skipPermissions = false) {
463
520
  const startTime = Date.now();
464
521
  activeExecutions.set(conversationId, true);
@@ -473,29 +530,40 @@ async function processMessageWithStreaming(conversationId, messageId, sessionId,
473
530
 
474
531
  let allBlocks = [];
475
532
  let eventCount = 0;
533
+ let currentSequence = queries.getMaxSequence(sessionId) ?? -1;
476
534
 
477
535
  const onEvent = (parsed) => {
478
536
  eventCount++;
537
+ debugLog(`[stream] Event ${eventCount}: type=${parsed.type}`);
479
538
 
480
539
  if (parsed.type === 'system') {
540
+ const systemBlock = {
541
+ type: 'system',
542
+ subtype: parsed.subtype,
543
+ model: parsed.model,
544
+ cwd: parsed.cwd,
545
+ tools: parsed.tools,
546
+ session_id: parsed.session_id
547
+ };
548
+
549
+ currentSequence++;
550
+ persistChunkWithRetry(sessionId, conversationId, currentSequence, 'system', systemBlock);
551
+
481
552
  broadcastSync({
482
553
  type: 'streaming_progress',
483
554
  sessionId,
484
555
  conversationId,
485
- block: {
486
- type: 'system',
487
- subtype: parsed.subtype,
488
- model: parsed.model,
489
- cwd: parsed.cwd,
490
- tools: parsed.tools,
491
- session_id: parsed.session_id
492
- },
556
+ block: systemBlock,
493
557
  blockIndex: allBlocks.length,
494
558
  timestamp: Date.now()
495
559
  });
496
560
  } else if (parsed.type === 'assistant' && parsed.message?.content) {
497
561
  for (const block of parsed.message.content) {
498
562
  allBlocks.push(block);
563
+
564
+ currentSequence++;
565
+ persistChunkWithRetry(sessionId, conversationId, currentSequence, block.type || 'assistant', block);
566
+
499
567
  broadcastSync({
500
568
  type: 'streaming_progress',
501
569
  sessionId,
@@ -508,39 +576,50 @@ async function processMessageWithStreaming(conversationId, messageId, sessionId,
508
576
  } else if (parsed.type === 'user' && parsed.message?.content) {
509
577
  for (const block of parsed.message.content) {
510
578
  if (block.type === 'tool_result') {
579
+ const toolResultBlock = {
580
+ type: 'tool_result',
581
+ tool_use_id: block.tool_use_id,
582
+ content: typeof block.content === 'string' ? block.content : JSON.stringify(block.content),
583
+ is_error: block.is_error || false
584
+ };
585
+
586
+ currentSequence++;
587
+ persistChunkWithRetry(sessionId, conversationId, currentSequence, 'tool_result', toolResultBlock);
588
+
511
589
  broadcastSync({
512
590
  type: 'streaming_progress',
513
591
  sessionId,
514
592
  conversationId,
515
- block: {
516
- type: 'tool_result',
517
- tool_use_id: block.tool_use_id,
518
- content: typeof block.content === 'string' ? block.content : JSON.stringify(block.content),
519
- is_error: block.is_error || false
520
- },
593
+ block: toolResultBlock,
521
594
  blockIndex: allBlocks.length,
522
595
  timestamp: Date.now()
523
596
  });
524
597
  }
525
598
  }
526
599
  } else if (parsed.type === 'result') {
600
+ const resultBlock = {
601
+ type: 'result',
602
+ subtype: parsed.subtype,
603
+ duration_ms: parsed.duration_ms,
604
+ total_cost_usd: parsed.total_cost_usd,
605
+ num_turns: parsed.num_turns,
606
+ is_error: parsed.is_error || false,
607
+ result: parsed.result
608
+ };
609
+
610
+ currentSequence++;
611
+ persistChunkWithRetry(sessionId, conversationId, currentSequence, 'result', resultBlock);
612
+
527
613
  broadcastSync({
528
614
  type: 'streaming_progress',
529
615
  sessionId,
530
616
  conversationId,
531
- block: {
532
- type: 'result',
533
- subtype: parsed.subtype,
534
- duration_ms: parsed.duration_ms,
535
- total_cost_usd: parsed.total_cost_usd,
536
- num_turns: parsed.num_turns,
537
- is_error: parsed.is_error || false,
538
- result: parsed.result
539
- },
617
+ block: resultBlock,
540
618
  blockIndex: allBlocks.length,
541
619
  isResult: true,
542
620
  timestamp: Date.now()
543
621
  });
622
+
544
623
  if (parsed.result && allBlocks.length === 0) {
545
624
  allBlocks.push({ type: 'text', text: String(parsed.result) });
546
625
  }
@@ -566,42 +645,13 @@ async function processMessageWithStreaming(conversationId, messageId, sessionId,
566
645
  debugLog(`[stream] Stored claudeSessionId=${claudeSessionId}`);
567
646
  }
568
647
 
569
- let messageContent = null;
570
- if (allBlocks.length > 0) {
571
- messageContent = JSON.stringify({
572
- type: 'claude_execution',
573
- blocks: allBlocks,
574
- timestamp: Date.now()
575
- });
576
- } else {
577
- let textParts = [];
578
- for (const output of outputs) {
579
- if (output.type === 'result' && output.result) {
580
- textParts.push(String(output.result));
581
- } else if (typeof output === 'string') {
582
- textParts.push(output);
583
- }
584
- }
585
- messageContent = textParts.join('\n').trim();
586
- }
587
-
588
- if (messageContent) {
589
- const assistantMessage = queries.createMessage(conversationId, 'assistant', messageContent);
590
- broadcastSync({
591
- type: 'streaming_complete',
592
- sessionId,
593
- conversationId,
594
- messageId: assistantMessage.id,
595
- eventCount,
596
- timestamp: Date.now()
597
- });
598
- broadcastSync({
599
- type: 'message_created',
600
- conversationId,
601
- message: assistantMessage,
602
- timestamp: Date.now()
603
- });
604
- }
648
+ broadcastSync({
649
+ type: 'streaming_complete',
650
+ sessionId,
651
+ conversationId,
652
+ eventCount,
653
+ timestamp: Date.now()
654
+ });
605
655
 
606
656
  debugLog(`[stream] Completed: ${outputs.length} outputs, ${eventCount} events`);
607
657
  } catch (error) {