free-coding-models 0.5.55 → 0.5.56

package/changelog/v0.5.56.md

@@ -0,0 +1,28 @@
+# Changelog v0.5.56 - 2026-07-25
+
+### Fixed
+- 🔁 **Router stream-stall failover after partial response** ([#137](https://github.com/vava-nessa/free-coding-models/issues/137)) — previously, if a model streamed some data then stalled (per-chunk timeout), the client received a truncated response with no automatic retry. Now the router:
+  1. Emits a synthetic SSE error event in OpenAI format (`code: fcm_stream_failover`) so the client knows the stream was truncated and that failover is happening.
+  2. Returns `failoverToNext: true` to the outer retry loop, which tries the next model on a fresh upstream connection.
+  3. The new model's chunks are appended to the same response object — the client sees one continuous stream with a clear mid-stream error marker.
+  4. On a mid-stream failover the router skips `res.writeHead()` (headers were already sent by the previous model) and emits an SSE comment `: fcm-router-failover-from=<oldKey>` so debuggers can see which model served which segment.
+
+  **Scope of the fix:** only `stream_stall_timeout` and `timeout` errors trigger mid-stream failover. Generic upstream errors (malformed JSON, connection reset, destroy) still close cleanly with no failover — their partial data is more likely to be invalid, so re-streaming from another model wouldn't help.
+
+### Maintenance
+- 🧪 **+1 regression test** (`test/test.js`) locking in the new failover behavior with a stalled-stream mock provider. The existing "does not retry after partial output" test still passes — it uses `res.destroy()` which is *not* a stall, so it correctly keeps the old behaviour.
+- 🧪 **591/591 tests pass** (`pnpm test`).
+- 🧹 `vite build` succeeds.
+
+### How clients see this
+A streaming request that stalls now produces something like:
+```
+data: {"choices":[{"delta":{"content":"partial"}}]}
+
+data: {"error":{"message":"Stream truncated by router due to upstream stream_stall_timeout; failing over to next model.","type":"stream_error","code":"fcm_stream_failover","reason":"stream_stall_timeout"}}
+
+data: {"choices":[{"delta":{"content":"fresh answer from fallback"}}]}
+
+data: [DONE]
+```
+Plus an SSE comment marker on each failover segment.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "free-coding-models",
-  "version": "0.5.55",
+  "version": "0.5.56",
   "description": "Find the fastest coding LLM models in seconds — ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
   "keywords": [
     "nvidia",

package/src/core/router-daemon.js

@@ -2194,11 +2194,23 @@ class RouterRuntime {
       }
 
       if (res.writableEnded) return { done: true }
-      res.writeHead(response.status, {
-        ...headerEntries(response.headers),
-        'x-fcm-router-model': key,
-        'x-request-id': requestId,
-      })
+      // 📖 Issue #137: when the previous model sent partial data, headers are
+      // 📖 already on the wire — re-calling writeHead throws ERR_HTTP_HEADERS_SENT.
+      // 📖 On a mid-stream failover we just append chunks to the existing response.
+      if (!res.headersSent) {
+        res.writeHead(response.status, {
+          ...headerEntries(response.headers),
+          'x-fcm-router-model': key,
+          'x-request-id': requestId,
+        })
+      } else {
+        // 📖 Reflect the new model in trailer-ish debug headers. Node won't let
+        // 📖 us add new headers after send, but we still update x-fcm-router-model
+        // 📖 semantics via a leading SSE comment so clients can see the switch.
+        try {
+          res.write(`: fcm-router-failover-from=${key}\n\n`)
+        } catch { /* best-effort */ }
+      }
       sentToClient = true
       res.write(firstChunkBuffer)
 
@@ -2230,6 +2242,12 @@ class RouterRuntime {
         return { done: true }
       }
       const reason = error.name === 'AbortError' ? 'timeout' : (error.message || String(error))
+      // 📖 Issue #137: stream-stall timeouts get a special tag so we can
+      // 📖 distinguish them from generic upstream errors below. Only stalls
+      // 📖 should trigger failover after a partial response — generic errors
+      // 📖 (malformed JSON, network reset, etc.) usually mean the partial
+      // 📖 data is invalid anyway, so closing cleanly is safer.
+      const isStall = reason === 'stream_stall_timeout' || reason === 'timeout'
       this.markFailure(key, reason)
       if (reason !== 'timeout') {
         this.recordRouterError('upstream_stream_error', requestId, { model: key, reason, partial: sentToClient })
@@ -2238,6 +2256,31 @@ class RouterRuntime {
       }
       this.addRequestLog({ request_id: requestId, model: key, status: 'ERR', latency_ms: null, tokens: 0, failover: attemptIndex > 0, error: reason, stream: true })
       if (sentToClient) {
+        if (isStall) {
+          // 📖 Issue #137: failover even after a partial response. Emit a
+          // 📖 synthetic SSE error event in OpenAI format so clients know the
+          // 📖 stream was truncated and that the router is failing over. The
+          // 📖 outer retry loop will then try the next model on a fresh
+          // 📖 upstream connection; its chunks are appended to the same
+          // 📖 response object so the client sees one continuous stream.
+          this.logger.warn(`Stream stall after partial response from ${key}, attempting failover`, { request_id: requestId, reason })
+          if (!res.writableEnded) {
+            try {
+              const errorPayload = JSON.stringify({
+                error: {
+                  message: `Stream truncated by router due to upstream ${reason}; failing over to next model.`,
+                  type: 'stream_error',
+                  code: 'fcm_stream_failover',
+                  reason,
+                },
+              })
+              res.write(`data: ${errorPayload}\n\n`)
+            } catch { /* best-effort */ }
+          }
+          return { done: false, failoverToNext: true, reason: `stream_stall_${reason}` }
+        }
+        // 📖 Non-stall errors after partial output: keep existing behaviour
+        // 📖 (close cleanly, no failover) to avoid sending malformed data.
         this.logger.warn(`Streaming failure after partial response from ${key}`, { request_id: requestId, reason })
         try { if (!res.writableEnded) res.end() } catch {}
         return { done: true }