@luckydraw/cumulus 0.30.14 → 0.30.16

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,17 @@
1
1
  # Changelog
2
2
 
3
+ ## v0.30.15
4
+ - Bug fixes and stability improvements
5
+
6
+
7
+ ## v0.30.14
8
+
9
+ - Cross-gateway agent messaging: agents on different gateways can now talk to each other
10
+ - Voice conversation mode with server-side narration for hands-free use
11
+ - First-run setup wizard, in-app update checker, and per-thread model selection
12
+ - Inline annotations: highlight chat text to comment like Google Docs
13
+ - Bug fixes and stability improvements (Windows spawn, blex resize, streaming reliability)
14
+
3
15
  ## v0.30.14
4
16
 
5
17
  - Windows 11 compatibility: fixed `spawn EINVAL` when the gateway tries to run the `claude.cmd` shim. All Claude CLI spawn sites (gateway, TUI, MCP handler, content-detector, summaries) now route through cmd.exe on Windows.
package/README.md CHANGED
@@ -1,147 +1,320 @@
1
1
  # Cumulus
2
2
 
3
- A CLI chat wrapper for Claude that implements the **Recursive Language Model (RLM)** pattern for unlimited conversation context.
3
+ A self-hosted multi-channel AI gateway built around Claude and other LLMs. Runs as a long-lived daemon, speaks to you through a web chat widget, Slack, Discord, iOS push, email webhooks, and more — with unlimited conversation context via the **Recursive Language Model (RLM)** pattern.
4
4
 
5
- ## The Problem
5
+ Originally a CLI wrapper for Claude, Cumulus has grown into a full gateway platform that coordinates agents, channels, models, and federated deployments.
6
6
 
7
- Standard LLM chat interfaces accumulate conversation history in the context window. As conversations grow:
7
+ ## What you get
8
8
 
9
- - **Context rot**: Model performance degrades as context fills up
10
- - **Compaction**: Old messages get summarized or dropped, losing detail
11
- - **Token limits**: Eventually you hit the wall, no matter how large the window
9
+ - **Gateway daemon** (`cumulus-gateway`) HTTP + WebSocket server with per-thread conversations, streaming responses, and an admin API.
10
+ - **Web chat widget** embeddable `/chat` interface with voice mode, push notifications, media uploads, and rich [blex block](https://www.npmjs.com/package/@luckydraw/blex) rendering (tables, forms, charts, kanban, diagrams).
11
+ - **Channel adapters** Slack and Discord bots, inbound email webhooks (Resend), and generic HTTP webhooks — all injecting into the same thread model.
12
+ - **Inter-agent messaging** — threads can talk to each other via `send_to_agent`, with support for CC/BCC visibility.
13
+ - **Federation** — hub-and-spoke mesh so agents on different machines can message each other across NATs.
14
+ - **Per-thread model selection** — Claude (via CLI) or any HuggingFace model (GLM-5, Kimi-K2.5, Qwen3, etc.) with tool calling.
15
+ - **Scheduled triggers, email, push, media serving** — built-in MCP tools so agents can send emails, schedule themselves, notify you, and upload files.
16
+ - **Unlimited history** — JSONL per thread + vector store + adaptive context budget. Conversations never truncate.
17
+ - **Classic CLI** (`cumulus`) — terminal chat for individual threads, backed by the same history store.
12
18
 
13
- ## The Solution
14
-
15
- Cumulus treats conversation history as an **external environment** that Claude queries programmatically, rather than stuffing it all into context.
19
+ ## Architecture
16
20
 
17
21
  ```
18
- Traditional: [msg1][msg2][msg3]...[msg847] → context rot, compaction, limits
22
+ ┌──────────────────────────────────────────────────────────────┐
23
+ │ Clients │
24
+ │ │
25
+ │ /chat (web) Slack Discord Email CLI Push (PWA) │
26
+ │ │ │ │ │ │ │ │
27
+ │ └─────────┴───────┴────────┴───────┴──────┘ │
28
+ │ │ │
29
+ │ ▼ │
30
+ │ ┌──────────────────────┐ │
31
+ │ │ cumulus-gateway │ │
32
+ │ │ (HTTP / WS daemon) │ │
33
+ │ └──────────┬───────────┘ │
34
+ │ │ │
35
+ │ ┌───────────────────┼──────────────────────┐ │
36
+ │ ▼ ▼ ▼ │
37
+ │ ┌─────────┐ ┌──────────────┐ ┌───────────────┐ │
38
+ │ │ Thread │ │ Model router │ │ Federation │ │
39
+ │ │ store │ │ Claude / HF │ │ hub/spoke │ │
40
+ │ │ (JSONL) │ │ MCP tools │ │ (WSS mesh) │ │
41
+ │ └────┬────┘ └──────┬───────┘ └───────────────┘ │
42
+ │ │ │ │
43
+ │ ▼ ▼ │
44
+ │ ~/.cumulus/ Claude CLI │
45
+ │ threads/ HuggingFace API │
46
+ │ content/ MCP stdio + in-process │
47
+ │ media/ │
48
+ └──────────────────────────────────────────────────────────────┘
49
+ ```
50
+
51
+ Every turn is a fresh model invocation. The gateway assembles a context budget from recent messages + RAG retrieval against the thread's history and content store, then streams the response back to the originating channel.
52
+
53
+ ## Installation
54
+
55
+ Requires Node 20+.
19
56
 
20
- Cumulus: [fresh context] + tools to query [external history]
21
-
22
- Claude retrieves what it needs, when it needs it
57
+ ```bash
58
+ npm install -g @luckydraw/cumulus
23
59
  ```
24
60
 
25
- Each message is a **fresh Claude invocation**. Claude uses MCP tools to retrieve relevant context from the complete conversation history stored in local files.
61
+ This installs three binaries:
62
+
63
+ | Command | Purpose |
64
+ | ----------------- | ------------------------------------------------- |
65
+ | `cumulus` | Terminal chat client for a single thread |
66
+ | `cumulus-mcp` | MCP server exposing history/content tools (stdio) |
67
+ | `cumulus-gateway` | Long-running daemon (HTTP + WebSocket + adapters) |
68
+
69
+ ## Quick start — gateway
26
70
 
27
- ## How It Works
71
+ ```bash
72
+ # Interactive setup: detects project directories, installs a service, generates keys.
73
+ cumulus-gateway setup
74
+
75
+ # Or non-interactive:
76
+ cumulus-gateway setup --project-root ~/projects --port 8080
28
77
 
78
+ # Start / stop / reload (if you skip the service install):
79
+ cumulus-gateway start
80
+ cumulus-gateway stop
81
+ cumulus-gateway reload # SIGHUP — drains active streams before restart
29
82
  ```
30
- ┌─────────────────────────────────────────────────────────────┐
31
- │ $ cumulus my-project │
32
- │ > Help me refactor the auth module we discussed yesterday │
33
- └─────────────────────────────────────────────────────────────┘
34
-
35
-
36
- ┌─────────────────────────────────────────────────────────────┐
37
- │ Cumulus │
38
- │ 1. Append user message to history │
39
- │ 2. Spawn fresh: claude --print --mcp-config history.json │
40
- │ 3. Claude uses search_history("auth module") tool │
41
- 4. Claude retrieves relevant messages from history │
42
- 5. Claude responds with full context awareness │
43
- 6. Append response to history │
44
- 7. Next message: completely fresh context │
45
- └─────────────────────────────────────────────────────────────┘
46
-
47
-
48
- ┌─────────────────────────────────────────────────────────────┐
49
- │ ~/.cumulus/threads/my-project.jsonl │
50
- ──────────────────────────────────────── │
51
- Full conversation history, never truncated │
52
- Claude queries via MCP tools as needed │
53
- └─────────────────────────────────────────────────────────────┘
83
+
84
+ Setup writes `~/.cumulus/gateway.config.json`, generates VAPID keys for push, scaffolds a systemd (Linux) or LaunchAgent (macOS) unit, and prints the generated API key.
85
+
86
+ Open `http://localhost:8080/chat`, paste the API key, and start talking. Messages hit your thread; responses stream back token-by-token.
87
+
88
+ ### Configuration
89
+
90
+ `~/.cumulus/gateway.config.json` — adjust any field with `cumulus-gateway config set <key> <value>` or edit directly:
91
+
92
+ ```jsonc
93
+ {
94
+ "apiKeys": ["sk-cumulus-…"],
95
+ "port": 8080,
96
+ "projectRoot": "/home/you/projects",
97
+ "model": "claude", // default per-thread model
98
+ "models": [
99
+ // available models for thread picker
100
+ { "id": "claude", "label": "Claude (CLI)", "provider": "claude-cli" },
101
+ { "id": "zai-org/GLM-5", "label": "GLM-5", "provider": "huggingface" },
102
+ { "id": "moonshotai/Kimi-K2.5", "label": "Kimi-K2.5", "provider": "huggingface" },
103
+ ],
104
+ "hfApiKey": "hf_…", // optional, for HuggingFace models
105
+ "channels": {
106
+ "slack": { "token": "xoxb-…", "signingSecret": "…", "appToken": "xapp-…" },
107
+ "discord": { "token": "…", "clientId": "…" },
108
+ },
109
+ "resend": { "apiKey": "re_…", "defaultFrom": "you@example.com" },
110
+ "vapid": { "publicKey": "…", "privateKey": "…", "subject": "mailto:you@example.com" },
111
+ "federation": {
112
+ "enabled": true,
113
+ "role": "hub", // "hub" or "spoke"
114
+ "allowedSpokes": ["mac-karl"], // hub only
115
+ // spoke config: { role:"spoke", hub:"wss://host/federation", apiKey:"…", name:"mac-…" }
116
+ },
117
+ }
54
118
  ```
55
119
 
56
- ## Key Features
120
+ Reload the daemon (`cumulus-gateway reload`) after editing. It waits for active streams to finish before restarting, so in-flight responses aren't dropped.
57
121
 
58
- - **Unlimited history**: Conversations can span thousands of messages without degradation
59
- - **Full Claude Code power**: All tools work (Bash, Edit, Read, etc.) - cumulus just manages context
60
- - **Persistent threads**: Pick up any conversation where you left off
61
- - **Selective retrieval**: Claude decides what context it needs, reducing noise
62
- - **Cost efficient**: Only retrieve relevant context, not everything
122
+ ## Gateway features
63
123
 
64
- ## Installation
124
+ ### Per-thread model selection
125
+
126
+ Each thread can run on a different model. Use the dropdown in the widget header, or the REST API:
65
127
 
66
128
  ```bash
67
- npm install -g cumulus-cli
129
+ curl -X PUT http://localhost:8080/api/thread/my-thread/config \
130
+ -H "X-API-Key: sk-…" \
131
+ -d '{"model": "zai-org/GLM-5"}'
68
132
  ```
69
133
 
70
- ## Usage
134
+ - `claude` — spawns `claude --print` per turn. Gets the full Claude Code tool surface.
135
+ - HuggingFace models — routed through an OpenAI-compatible endpoint with a built-in agentic loop that handles tool use, truncation recovery, and error retry.
71
136
 
72
- ```bash
73
- # Start or continue a conversation thread
74
- cumulus my-project
137
+ ### Web chat widget
75
138
 
76
- # List all threads
77
- cumulus --list
139
+ At `/chat`. Features:
78
140
 
79
- # Delete a thread
80
- cumulus --delete old-project
141
+ - **Streaming** responses over WebSocket, with interjection support (type while streaming to interrupt and redirect).
142
+ - **[Blex blocks](https://www.npmjs.com/package/@luckydraw/blex)** — `~~~blex:table`, `~~~blex:poll`, `~~~blex:kanban`, `~~~blex:mermaid`, and 18 other block types for rich interactive content.
143
+ - **Voice mode** — hands-free conversation using browser STT + TTS (optionally server-side Piper).
144
+ - **Push notifications** — PWA install + VAPID subscriptions. Agents call `notify_user` to alert you while you're away.
145
+ - **Media uploads** — drag files in; `upload_media` tool returns a public URL backed by `~/.cumulus/media/`.
146
+ - **Annotations** — highlight text, attach comments, send back as chips.
147
+ - **Texitool integration** — edit Unicode-art diagrams in-place via an embedded canvas.
148
+
149
+ ### Channel adapters
150
+
151
+ - **Slack** (`channels.slack`) — Socket Mode bot. Thread naming: `slack-{userId}-{channelId}`.
152
+ - **Discord** (`channels.discord`) — Gateway WebSocket. Thread naming: `discord-{userId}-{channelId}`.
153
+ - **Inbound webhooks** — `POST /api/hooks/:type` for email (Resend), forms, and generic events. Config-driven thread routing with HMAC signature verification.
154
+
155
+ ### Inter-agent messaging
156
+
157
+ Any thread can message another thread using the `send_to_agent` MCP tool:
158
+
159
+ ```
160
+ send_to_agent(target="devops", message="Deploy the new build", visibility="cc")
81
161
  ```
82
162
 
83
- ## MCP Tools Available to Claude
163
+ - `cc` (default) all recipients see each other.
164
+ - `blind` — each recipient thinks it's a direct message.
165
+ - `{hidden: […]}` — selective (observer pattern, hidden agents invisible to visible recipients).
84
166
 
85
- When you chat through Cumulus, Claude has access to:
167
+ If the target is busy, the message is queued and delivered as a batch when that thread is idle ("while you were busy, 3 messages arrived…").
86
168
 
87
- | Tool | Description |
88
- | ------------------- | ---------------------------------------------------- |
89
- | `search_history` | Find messages by keyword, semantic, or hybrid search |
90
- | `read_messages` | Read a specific range of messages |
91
- | `peek_recent` | See the last few messages |
92
- | `get_history_stats` | Get count and token estimate |
93
- | `sub_query` | Ask focused questions about history snippets |
94
- | `get_summary` | Get auto-generated summaries of conversation history |
169
+ ### Federation
95
170
 
96
- ## Architecture
171
+ Two gateways can be linked in a hub-and-spoke topology. The hub runs at a stable URL; spokes connect outbound via WSS, so NAT doesn't matter.
97
172
 
98
173
  ```
99
- src/
100
- ├── lib/
101
- │ └── history.ts # JSONL storage for conversation history
102
- ├── mcp/
103
- │ ├── server.ts # MCP server exposing history tools
104
- │ └── index.ts # MCP entry point
105
- └── cli/
106
- └── cumulus.ts # Main CLI, chat loop, Claude invocation
174
+ send_to_agent("thundercat:cumulus", "…") # cross-gateway addressing
175
+ list_agents() # aggregates threads across all spokes
107
176
  ```
108
177
 
109
- ## Background
178
+ Heartbeats every 25s with bidirectional WebSocket pings; dead connections are torn down within 75s.
110
179
 
111
- This project implements ideas from the paper ["Recursive Language Models"](https://arxiv.org/abs/...) which demonstrates that treating long prompts as external environment objects—rather than feeding them directly into the context window—dramatically improves performance on long-context tasks while maintaining reasonable costs.
180
+ ### Scheduled triggers
112
181
 
113
- Key insight: LLMs can programmatically examine, decompose, and query their context rather than processing it all at once. This enables effective reasoning over contexts 2+ orders of magnitude beyond the model's context window.
182
+ Agents can schedule themselves:
114
183
 
115
- ## Development
184
+ ```
185
+ schedule_trigger(at="2026-05-01T09:00:00Z", message="Follow up with lead")
186
+ schedule_trigger(cron="0 9 * * MON", message="Weekly check-in")
187
+ cancel_schedule(id="…")
188
+ ```
189
+
190
+ Schedules are per-thread, persisted in `{thread}.config.json`, and fire as message injections into the thread.
191
+
192
+ ### Email (Resend)
193
+
194
+ With `resend.apiKey` configured:
195
+
196
+ ```
197
+ send_email(to="person@example.com", subject="Hello", body="…")
198
+ list_emails(limit=10)
199
+ ```
200
+
201
+ Rate-limited per thread (default 10/hour). All sends are logged to thread history. First email from a new thread triggers a `notify_user` ping.
202
+
203
+ ### Self-update
116
204
 
117
205
  ```bash
118
- # Install dependencies
119
- npm install
206
+ cumulus-gateway check-update # compares running version to npm
207
+ cumulus-gateway update # bumps to latest, saves previous for rollback
208
+ cumulus-gateway rollback # restores the previous version
209
+ ```
120
210
 
121
- # Build
122
- npm run build
211
+ The widget's top bar also shows an "Update available" indicator when a new version lands on npm.
123
212
 
124
- # Run tests
125
- npm test
213
+ ## Classic CLI mode
126
214
 
127
- # Lint
128
- npm run lint
215
+ The original RLM chat loop still works. Great for quick terminal work without running the gateway.
216
+
217
+ ```bash
218
+ cumulus my-project # open or create a thread
219
+ cumulus --list # list threads
220
+ cumulus --delete old-project
129
221
  ```
130
222
 
131
- ## Roadmap
223
+ Each turn:
224
+
225
+ 1. Append your message to `~/.cumulus/threads/my-project.jsonl`.
226
+ 2. Spawn `claude --print` with `--mcp-config` pointing to the cumulus MCP server.
227
+ 3. Claude pulls whatever history it needs via `search_history`, `peek_recent`, etc.
228
+ 4. Append the response to the JSONL.
229
+ 5. Next turn starts from a fresh context.
230
+
231
+ ## MCP tools
232
+
233
+ The `cumulus-mcp` server exposes history and content tools. Usable from any MCP-compatible client.
234
+
235
+ **History:**
236
+
237
+ | Tool | Purpose |
238
+ | ------------------- | ---------------------------------------------------------------- |
239
+ | `search_history` | Keyword / semantic / hybrid search over a thread |
240
+ | `peek_recent` | Last N messages |
241
+ | `read_messages` | Message range by index |
242
+ | `get_history_stats` | Count, token estimate, time range |
243
+ | `get_summary` | Auto-generated summaries (recent chunk, full, or specific range) |
244
+ | `sub_query` | Recursive sub-LLM call over retrieved messages |
245
+
246
+ **Content store** (file reads, bash output, web fetches):
247
+
248
+ | Tool | Purpose |
249
+ | --------------------- | ---------------------------------------- |
250
+ | `read_file` | Read text/PDF, chunk + embed + store |
251
+ | `store_content` | Store arbitrary text for later retrieval |
252
+ | `search_content` | Search across stored content |
253
+ | `retrieve_content` | Get full content by `[STORED:xxx]` id |
254
+ | `read_content_chunk` | Read a specific chunk index |
255
+ | `list_stored_content` | List all stored items |
256
+ | `detect_anomalies` | Find out-of-place content in a store |
257
+ | `forget_content` | Remove a stored item |
258
+
259
+ **Gateway-only tools** (available to agents running inside the daemon):
260
+
261
+ `send_to_agent`, `list_agents`, `notify_user`, `schedule_trigger`, `cancel_schedule`, `list_schedules`, `send_email`, `list_emails`, `upload_media`, `create_plastic_app`, `update_pipeline`.
262
+
263
+ ## RAG & context management
264
+
265
+ - **JSONL history** per thread — every message, tool call, and tool result.
266
+ - **Content store** — chunked file reads, embedded with local HuggingFace transformers, stored as binary Float32.
267
+ - **Segment summaries** — LLM-generated per topic boundary, separately embedded for vocabulary-gap retrieval.
268
+ - **Adaptive context budget** — self-tuning per thread based on TTFT. Shrinks when slow, grows when fast + near-capacity. Default 300k, floor 100k, ceiling 1M.
269
+ - **Query-type-aware retrieval** — classifies queries (recall / synthesis / recent / decision) and adjusts scoring weights accordingly.
270
+
271
+ ## REST API (gateway)
272
+
273
+ | Method | Path | Purpose |
274
+ | ------ | --------------------------- | --------------------------------------- |
275
+ | GET | `/health` | Gateway status |
276
+ | POST | `/api/thread/:name/message` | Send a message (SSE stream in response) |
277
+ | GET | `/api/thread/:name/history` | Paginated thread history |
278
+ | GET | `/api/thread/:name/config` | Thread config |
279
+ | PUT | `/api/thread/:name/config` | Update thread config (model, etc.) |
280
+ | DELETE | `/api/thread/:name` | Delete a thread |
281
+ | GET | `/api/threads` | List threads |
282
+ | GET | `/api/agents` | List threads + streaming status |
283
+ | POST | `/api/agents/inject` | Inject message into a thread |
284
+ | GET | `/api/models` | Available models |
285
+ | POST | `/api/media/upload` | Upload a file |
286
+ | GET | `/media/:filename` | Serve uploaded file |
287
+ | POST | `/api/hooks/:type` | Inbound webhook |
288
+ | GET | `/api/push/vapid-key` | Public VAPID key |
289
+ | POST | `/api/push/subscribe` | Register a push subscription |
290
+ | GET | `/api/version` | Running version + update availability |
291
+ | POST | `/api/admin/update` | Trigger self-update (admin key) |
292
+
293
+ All `/api/*` routes require `X-API-Key: <key>` (from `apiKeys[]`).
294
+
295
+ WebSocket (`/chat/ws`) carries the same semantics with streaming, interjection, `inject`, `worker_cancel`, and voice-mode audio frames.
132
296
 
133
- **Phase 1 (MVP)**
297
+ ## Development
134
298
 
135
- - [x] Project setup with TypeScript/ESLint
136
- - [x] HistoryStore for JSONL persistence
137
- - [x] MCP server with history tools
138
- - [x] CLI wrapper with pure RLM chat loop
139
- - [x] Basic test suite
299
+ ```bash
300
+ git clone https://github.com/soapko/cumulus
301
+ cd cumulus
302
+ npm install
303
+ npm run build
304
+ npm test # vitest
305
+ npm run lint
306
+ npm run type-check
307
+ ```
308
+
309
+ - **Build:** `dist/` compiled TypeScript plus static assets (widget HTML/CSS/JS, blex bundles).
310
+ - **Tests:** vitest, 700+ tests covering agentic loop, retriever, adapters, federation, scheduler, push.
311
+ - **Deploy workflow:** bump version, `npm publish`, then `cumulus-gateway reload` on the host (SIGHUP drains active streams — see `docs/tasks/050-graceful-restart.md`).
312
+
313
+ ## Background
140
314
 
141
- **Phase 2 (Enhancements)**
315
+ Cumulus implements the **Recursive Language Model** pattern: treat conversation history as an external environment the model queries programmatically, rather than stuffing everything into context. This enables reasoning over contexts 2+ orders of magnitude beyond the model's window, with graceful cost scaling.
142
316
 
143
- - [x] Semantic search with embeddings
144
- - [x] Periodic summarization for very long histories
317
+ See `docs/` for task documents, ADRs, and implementation notes.
145
318
 
146
319
  ## License
147
320
 
@@ -46,13 +46,7 @@ export interface FederationInjectMessage {
46
46
  hidden: string[];
47
47
  };
48
48
  }
49
- export interface FederationPing {
50
- type: 'ping';
51
- }
52
- export interface FederationPong {
53
- type: 'pong';
54
- }
55
- export type FederationMessage = FederationAuthMessage | FederationAuthOk | FederationAuthError | FederationThreadSync | FederationRouteMessage | FederationInjectMessage | FederationPing | FederationPong;
49
+ export type FederationMessage = FederationAuthMessage | FederationAuthOk | FederationAuthError | FederationThreadSync | FederationRouteMessage | FederationInjectMessage;
56
50
  export interface FederationRouterOptions {
57
51
  hubName: string;
58
52
  apiKeys: string[];
@@ -124,6 +118,8 @@ export declare class FederationClient {
124
118
  private closed;
125
119
  private hubName;
126
120
  private lastSyncedThreads;
121
+ private pingTimer;
122
+ private lastSeen;
127
123
  constructor(options: FederationClientOptions);
128
124
  /** Connect to the hub */
129
125
  connect(): void;
@@ -135,6 +131,8 @@ export declare class FederationClient {
135
131
  syncThreads(): void;
136
132
  /** Handle a message from the hub destined for a local thread */
137
133
  private handleInject;
134
+ private startHeartbeat;
135
+ private stopHeartbeat;
138
136
  private scheduleReconnect;
139
137
  private send;
140
138
  get isConnected(): boolean;
@@ -1 +1 @@
1
- {"version":3,"file":"federation.d.ts","sourceRoot":"","sources":["../../src/gateway/federation.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,MAAM,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,QAAQ,CAAC;AAOrC,MAAM,WAAW,qBAAqB;IACpC,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,SAAS,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,mBAAmB;IAClC,IAAI,EAAE,YAAY,CAAC;IACnB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,oBAAoB;IACnC,IAAI,EAAE,aAAa,CAAC;IACpB,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAED,MAAM,WAAW,sBAAsB;IACrC,IAAI,EAAE,OAAO,CAAC;IACd,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,MAAM,CAAC;IACX,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,CAAC;CAC5C;AAED,MAAM,WAAW,uBAAuB;IACtC,IAAI,EAAE,QAAQ,CAAC;IACf,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,MAAM,CAAC;IACX,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,CAAC;CAC5C;AAED,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;CACd;AAED,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;CACd;AAED,MAAM,MAAM,iBAAiB,GACzB,qBAAqB,GACrB,gBAAgB,GAChB,mBAAmB,GACnB,oBAAoB,GACpB,sBAAsB,GACtB,uBAAuB,GACvB,cAAc,GACd,cAAc,CAAC;AAcnB,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,aAAa,CAAC,EAAE,MAAM,EAAE,CAAC;IACzB,uDAAuD;IACvD,WAAW,EAAE,CACX,MAAM,EAAE,MAAM,EACd,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,KACvC,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACjC,qCAAqC;IACrC,gBAAgB,EAAE,MAAM,MAAM,EAAE,CAAC;CAClC;AAED,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,GAAG,CAAkB;IAC7B,OAAO,CAAC,MAAM,CAAiC;IAC/C,OAAO,CAAC,OAAO,CAA0B;gBAE7B,OAAO,EAAE,uBAAuB;IAK5C,4DAA4D;IAC5D,aAAa,CAAC,GAAG,EAAE,eAAe,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,IAAI;IAMvE,OAAO,CAAC,gBAAgB;IAkHxB,sDAAsD;YACxC,WAAW;IA+CzB,qEAAqE;IAC/D,YAAY,CAChB,aAAa,EAAE,MAAM,EACrB,YAAY,EAAE,MAAM,EACpB,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,GACzC,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IAwB9B,uGAAuG;IACvG,aAAa,CAAC,UAAU,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IAShD,6DAA6D;IAC7D,kBAAkB,IAAI,KAAK,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IAU9E,kCAAkC;IAClC,SAAS,IAAI,KAAK,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,EAAE,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,OAAO,CAAA;KAAE,CAAC;IAc7F,OAAO,CAAC,YAAY;IAQpB,OAAO,CAAC,IAAI;IAMZ,KAAK,IAAI,IAAI;CAMd;AAID,MAAM,WAAW,uBAAuB;IACtC,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;IAClB,uDAAuD;IACvD,WAAW,EAAE,CACX,MAAM,EAAE,MAAM,EACd,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,KACvC,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACjC,0CAA0C;IAC1C,gBAAgB,EAAE,MAAM,MAAM,EAAE,CAAC;CAClC;AAED,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,EAAE,CAA0B;IACpC,OAAO,CAAC,OAAO,CAA0B;IACzC,OAAO,CAAC,cAAc,CAAQ;IAC9B,OAAO,CAAC,iBAAiB,CAAU;IACnC,OAAO,CAAC,cAAc,CAA8C;IACpE,OAAO,CAAC,aAAa,CAAS;IAC9B,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,OAAO,CAAM;IACrB,OAAO,CAAC,iBAAiB,CAAgB;gBAE7B,OAAO,EAAE,uBAAuB;IAI5C,yBAAyB;IACzB,OAAO,IAAI,IAAI;IAyEf,gEAAgE;IAChE,YAAY,CACV,aAAa,EAAE,MAAM,EACrB,YAAY,EAAE,MAAM,EACpB,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,GACzC,OAAO;IAcV,4CAA4C;IAC5C,WAAW,IAAI,IAAI;IAcnB,gEAAgE;YAClD,YAAY;IAiB1B,OAAO,CAAC,iBAAiB;IAYzB,OAAO,CAAC,IAAI;IAMZ,IAAI,WAAW,IAAI,OAAO,CAEzB;IAED,IAAI,gBAAgB,IAAI,MAAM,CAE7B;IAED,KAAK,IAAI,IAAI;CAKd;AAID,kEAAkE;AAClE,wBAAgB,YAAY,CAAC,OAAO,EAAE,MAAM,GAAG;IAAE,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CASxF"}
1
+ {"version":3,"file":"federation.d.ts","sourceRoot":"","sources":["../../src/gateway/federation.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,MAAM,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,QAAQ,CAAC;AAOrC,MAAM,WAAW,qBAAqB;IACpC,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,SAAS,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,mBAAmB;IAClC,IAAI,EAAE,YAAY,CAAC;IACnB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,oBAAoB;IACnC,IAAI,EAAE,aAAa,CAAC;IACpB,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAED,MAAM,WAAW,sBAAsB;IACrC,IAAI,EAAE,OAAO,CAAC;IACd,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,MAAM,CAAC;IACX,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,CAAC;CAC5C;AAED,MAAM,WAAW,uBAAuB;IACtC,IAAI,EAAE,QAAQ,CAAC;IACf,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,MAAM,CAAC;IACX,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,CAAC;CAC5C;AAED,MAAM,MAAM,iBAAiB,GACzB,qBAAqB,GACrB,gBAAgB,GAChB,mBAAmB,GACnB,oBAAoB,GACpB,sBAAsB,GACtB,uBAAuB,CAAC;AAkB5B,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,aAAa,CAAC,EAAE,MAAM,EAAE,CAAC;IACzB,uDAAuD;IACvD,WAAW,EAAE,CACX,MAAM,EAAE,MAAM,EACd,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,KACvC,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACjC,qCAAqC;IACrC,gBAAgB,EAAE,MAAM,MAAM,EAAE,CAAC;CAClC;AAED,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,GAAG,CAAkB;IAC7B,OAAO,CAAC,MAAM,CAAiC;IAC/C,OAAO,CAAC,OAAO,CAA0B;gBAE7B,OAAO,EAAE,uBAAuB;IAK5C,4DAA4D;IAC5D,aAAa,CAAC,GAAG,EAAE,eAAe,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,IAAI;IAMvE,OAAO,CAAC,gBAAgB;IA0HxB,sDAAsD;YACxC,WAAW;IA+CzB,qEAAqE;IAC/D,YAAY,CAChB,aAAa,EAAE,MAAM,EACrB,YAAY,EAAE,MAAM,EACpB,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,GACzC,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IAwB9B,uGAAuG;IACvG,aAAa,CAAC,UAAU,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IAShD,6DAA6D;IAC7D,kBAAkB,IAAI,KAAK,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IAU9E,kCAAkC;IAClC,SAAS,IAAI,KAAK,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,EAAE,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,OAAO,CAAA;KAAE,CAAC;IAc7F,OAAO,CAAC,YAAY;IAQpB,OAAO,CAAC,IAAI;IAMZ,KAAK,IAAI,IAAI;CAMd;AAID,MAAM,WAAW,uBAAuB;IACtC,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;IAClB,uDAAuD;IACvD,WAAW,EAAE,CACX,MAAM,EAAE,MAAM,EACd,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,KACvC,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACjC,0CAA0C;IAC1C,gBAAgB,EAAE,MAAM,MAAM,EAAE,CAAC;CAClC;AAED,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,EAAE,CAA0B;IACpC,OAAO,CAAC,OAAO,CAA0B;IACzC,OAAO,CAAC,cAAc,CAAQ;IAC9B,OAAO,CAAC,iBAAiB,CAAU;IACnC,OAAO,CAAC,cAAc,CAA8C;IACpE,OAAO,CAAC,aAAa,CAAS;IAC9B,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,OAAO,CAAM;IACrB,OAAO,CAAC,iBAAiB,CAAgB;IACzC,OAAO,CAAC,SAAS,CAA+C;IAChE,OAAO,CAAC,QAAQ,CAAK;gBAET,OAAO,EAAE,uBAAuB;IAI5C,yBAAyB;IACzB,OAAO,IAAI,IAAI;IAqFf,gEAAgE;IAChE,YAAY,CACV,aAAa,EAAE,MAAM,EACrB,YAAY,EAAE,MAAM,EACpB,OAAO,EAAE,MAAM,EACf,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,MAAM,GAAG;QAAE,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,GACzC,OAAO;IAcV,4CAA4C;IAC5C,WAAW,IAAI,IAAI;IAcnB,gEAAgE;YAClD,YAAY;IAiB1B,OAAO,CAAC,cAAc;IActB,OAAO,CAAC,aAAa;IAOrB,OAAO,CAAC,iBAAiB;IAYzB,OAAO,CAAC,IAAI;IAMZ,IAAI,WAAW,IAAI,OAAO,CAEzB;IAED,IAAI,gBAAgB,IAAI,MAAM,CAE7B;IAED,KAAK,IAAI,IAAI;CAMd;AAID,kEAAkE;AAClE,wBAAgB,YAAY,CAAC,OAAO,EAAE,MAAM,GAAG;IAAE,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CASxF"}
@@ -8,6 +8,9 @@
8
8
  * relays inbound messages to local threads.
9
9
  */
10
10
  import { WebSocket, WebSocketServer } from 'ws';
11
+ // Heartbeat: WebSocket-level ping frames (not JSON). 25s interval, 75s timeout.
12
+ const HEARTBEAT_INTERVAL_MS = 25_000;
13
+ const HEARTBEAT_TIMEOUT_MS = 75_000;
11
14
  export class FederationRouter {
12
15
  wss;
13
16
  spokes = new Map();
@@ -76,16 +79,27 @@ export class FederationRouter {
76
79
  threads: [],
77
80
  lastSeen: Date.now(),
78
81
  };
79
- // Start ping/pong heartbeat (30s interval)
82
+ // WS-level heartbeat: ping frames every 25s, timeout at 75s of silence.
83
+ // Spoke also pings us; either side's `pong` event updates lastSeen.
80
84
  entry.pingTimer = setInterval(() => {
81
- if (Date.now() - entry.lastSeen > 60_000) {
85
+ if (Date.now() - entry.lastSeen > HEARTBEAT_TIMEOUT_MS) {
82
86
  console.log(`[Federation] Spoke "${spokeName}" timed out, disconnecting`);
83
- ws.close(1000, 'Heartbeat timeout');
87
+ ws.terminate(); // Force-close: TCP may be dead even if readyState says OPEN
84
88
  this.cleanupSpoke(spokeName);
85
89
  return;
86
90
  }
87
- this.send(ws, { type: 'ping' });
88
- }, 30_000);
91
+ if (ws.readyState === WebSocket.OPEN) {
92
+ ws.ping();
93
+ }
94
+ }, HEARTBEAT_INTERVAL_MS);
95
+ // Any pong (from our ping) or ping (from spoke) updates liveness.
96
+ ws.on('pong', () => {
97
+ entry.lastSeen = Date.now();
98
+ });
99
+ ws.on('ping', () => {
100
+ entry.lastSeen = Date.now();
101
+ // ws library auto-responds with pong, no manual send needed
102
+ });
89
103
  this.spokes.set(spokeName, entry);
90
104
  this.send(ws, { type: 'auth_ok', hubName: this.options.hubName });
91
105
  console.log(`[Federation] Spoke "${spokeName}" connected`);
@@ -104,9 +118,6 @@ export class FederationRouter {
104
118
  case 'route':
105
119
  void this.handleRoute(spokeName, msg);
106
120
  break;
107
- case 'pong':
108
- // lastSeen already updated above
109
- break;
110
121
  default:
111
122
  console.log(`[Federation] Unknown message type from "${spokeName}": ${msg.type}`);
112
123
  }
@@ -244,6 +255,8 @@ export class FederationClient {
244
255
  closed = false;
245
256
  hubName = '';
246
257
  lastSyncedThreads = [];
258
+ pingTimer = null;
259
+ lastSeen = 0;
247
260
  constructor(options) {
248
261
  this.options = options;
249
262
  }
@@ -262,14 +275,25 @@ export class FederationClient {
262
275
  this.ws.on('open', () => {
263
276
  console.log(`[Federation] Connected to hub at ${this.options.hubUrl}`);
264
277
  this.reconnectDelay = 1000; // Reset backoff on successful connect
278
+ this.lastSeen = Date.now();
265
279
  // Send auth
266
280
  this.send({
267
281
  type: 'auth',
268
282
  name: this.options.spokeName,
269
283
  apiKey: this.options.apiKey,
270
284
  });
285
+ // WS-level heartbeat: ping hub every 25s, reconnect if no traffic for 75s.
286
+ this.startHeartbeat();
287
+ });
288
+ this.ws.on('pong', () => {
289
+ this.lastSeen = Date.now();
290
+ });
291
+ this.ws.on('ping', () => {
292
+ this.lastSeen = Date.now();
293
+ // ws library auto-responds with pong
271
294
  });
272
295
  this.ws.on('message', (raw) => {
296
+ this.lastSeen = Date.now();
273
297
  let msg;
274
298
  try {
275
299
  msg = JSON.parse(raw.toString());
@@ -294,15 +318,13 @@ export class FederationClient {
294
318
  case 'inject':
295
319
  void this.handleInject(msg);
296
320
  break;
297
- case 'ping':
298
- this.send({ type: 'pong' });
299
- break;
300
321
  default:
301
322
  break;
302
323
  }
303
324
  });
304
325
  this.ws.on('close', () => {
305
326
  this.authenticated = false;
327
+ this.stopHeartbeat();
306
328
  if (!this.closed) {
307
329
  console.log(`[Federation] Disconnected from hub, reconnecting in ${this.reconnectDelay}ms`);
308
330
  this.scheduleReconnect();
@@ -350,6 +372,25 @@ export class FederationClient {
350
372
  console.error(`[Federation] Failed to inject into local thread "${msg.to}":`, err.message);
351
373
  }
352
374
  }
375
+ startHeartbeat() {
376
+ this.stopHeartbeat();
377
+ this.pingTimer = setInterval(() => {
378
+ if (Date.now() - this.lastSeen > HEARTBEAT_TIMEOUT_MS) {
379
+ console.log('[Federation] Hub silent, forcing reconnect');
380
+ this.ws?.terminate(); // Force-close: triggers 'close' → reconnect
381
+ return;
382
+ }
383
+ if (this.ws?.readyState === WebSocket.OPEN) {
384
+ this.ws.ping();
385
+ }
386
+ }, HEARTBEAT_INTERVAL_MS);
387
+ }
388
+ stopHeartbeat() {
389
+ if (this.pingTimer) {
390
+ clearInterval(this.pingTimer);
391
+ this.pingTimer = null;
392
+ }
393
+ }
353
394
  scheduleReconnect() {
354
395
  if (this.closed)
355
396
  return;
@@ -374,6 +415,7 @@ export class FederationClient {
374
415
  }
375
416
  close() {
376
417
  this.closed = true;
418
+ this.stopHeartbeat();
377
419
  if (this.reconnectTimer)
378
420
  clearTimeout(this.reconnectTimer);
379
421
  this.ws?.close();