clew-code 0.2.11 → 0.2.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. package/dist/main.js +2596 -2527
  2. package/docs/architecture.html +148 -148
  3. package/docs/architecture.th.html +79 -79
  4. package/docs/clew-code-architecture.html +6 -6
  5. package/docs/commands.html +225 -224
  6. package/docs/commands.th.html +132 -131
  7. package/docs/configuration.html +147 -147
  8. package/docs/configuration.th.html +108 -108
  9. package/docs/daemon.html +129 -129
  10. package/docs/daemon.th.html +73 -73
  11. package/docs/features/bridge-mode.html +99 -99
  12. package/docs/features/bridge-mode.th.html +90 -90
  13. package/docs/features/evals.html +182 -182
  14. package/docs/features/evals.th.html +90 -90
  15. package/docs/features/peer.html +178 -178
  16. package/docs/features/searxng-search.html +151 -151
  17. package/docs/features/searxng-search.th.html +95 -95
  18. package/docs/features/sentry-setup.html +157 -157
  19. package/docs/features/sentry-setup.th.html +97 -97
  20. package/docs/index.html +298 -299
  21. package/docs/index.th.html +292 -292
  22. package/docs/installation.html +105 -105
  23. package/docs/installation.th.html +105 -105
  24. package/docs/internals/growthbook-ab-testing.html +113 -113
  25. package/docs/internals/growthbook-ab-testing.th.html +81 -81
  26. package/docs/internals/hidden-features.html +175 -149
  27. package/docs/internals/hidden-features.th.html +135 -109
  28. package/docs/loop.html +181 -181
  29. package/docs/loop.th.html +227 -227
  30. package/docs/mcp.html +247 -247
  31. package/docs/mcp.th.html +207 -207
  32. package/docs/models.html +110 -111
  33. package/docs/models.th.html +61 -61
  34. package/docs/peer.html +236 -236
  35. package/docs/peer.th.html +280 -280
  36. package/docs/permission-model.html +102 -102
  37. package/docs/permission-model.th.html +67 -67
  38. package/docs/plugins.html +102 -102
  39. package/docs/plugins.th.html +79 -79
  40. package/docs/providers.html +126 -126
  41. package/docs/providers.th.html +80 -80
  42. package/docs/quick-start.html +93 -93
  43. package/docs/quick-start.th.html +1 -1
  44. package/docs/research-memory.html +82 -82
  45. package/docs/research-memory.th.html +72 -72
  46. package/docs/skills.html +117 -117
  47. package/docs/skills.th.html +90 -90
  48. package/docs/tools.html +170 -170
  49. package/docs/tools.th.html +84 -84
  50. package/docs/troubleshooting.html +106 -106
  51. package/docs/troubleshooting.th.html +85 -85
  52. package/package.json +162 -164
package/docs/loop.html CHANGED
@@ -1,181 +1,181 @@
1
- <!DOCTYPE html>
2
- <html lang="en">
3
- <head>
4
- <meta charset="UTF-8">
5
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
- <title>Agent Loop — Clew Code</title>
7
- <meta name="description" content="Autonomous agent loop — task queue consumer, worker lifecycle, retry logic, dead-letter queue, lease-based concurrency">
8
- <link rel="preconnect" href="https://fonts.googleapis.com">
9
- <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
10
- <link href="https://fonts.googleapis.com/css2?family=DM+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet">
11
- <link rel="stylesheet" href="css/styles.css">
12
- <link rel="icon" type="image/svg+xml" href="./assets/clew.svg">
13
- </head>
14
- <body>
15
- <header class="header"></header>
16
- <div class="app">
17
- <aside class="sidebar" id="sidebar"></aside>
18
- <div class="sidebar-overlay" id="sidebarOverlay"></div>
19
- <div class="content-wrap">
20
- <main class="content">
21
- <div class="breadcrumbs"><a href="index.html">Home</a><span class="sep">/</span><span>Agent Loop</span></div>
22
- <h1>Autonomous Agent Loop</h1>
23
- <p class="section-subtitle">24/7 background execution loop — pulls tasks from queue, spawns workers, monitors status, retries on failure, dead-letters when exhausted.</p>
24
-
25
- <p>The agent loop lives in <code>src/services/autonomous/agentLoop.ts</code> — the single consumer that reads the task queue and manages the lifecycle of all worker processes.</p>
26
-
27
- <h2>Techniques &amp; Principles</h2>
28
-
29
- <h3>Why a Loop instead of a Message Queue?</h3>
30
- <p>Message queue systems like RabbitMQ/Kafka carry setup and maintenance overhead. The agent loop uses a <strong>JSON file as a persistent queue</strong> — zero external dependencies:</p>
31
- <ul>
32
- <li><strong>Zero external deps</strong> — uses <code>fs.watch</code> to detect queue file changes</li>
33
- <li><strong>File-based atomicity</strong> — single-file read/write under lock allows multiple processes to compete for consumption</li>
34
- <li><strong>Self-debouncing</strong> — <code>ourWriteInProgress</code> flag prevents self-triggered re-reads</li>
35
- <li><strong>Cross-platform</strong> — JSON files work everywhere, no broker installation needed</li>
36
- </ul>
37
-
38
- <h3>Lease-Based Concurrency</h3>
39
- <p>The classic distributed worker problem: two workers see the same task and duplicate work. Leases solve this without a distributed lock:</p>
40
- <pre><code>Main Loop Queue File (.json) Worker Process
41
- │ │ │
42
- │ getNextTask() │ │
43
- │─────────────────────────────►│ │
44
- │◄─── task {id, status:pending}│ │
45
- │ │ │
46
- │ leaseTask(id, agentId) │ │
47
- │─────────────────────────────►│ │
48
- │ (atomic: check no lease │ │
49
- │ → write leaseOwner + │ │
50
- │ leaseExpiresAt) │ │
51
- │◄─────── true (leased) ──────│ │
52
- │ │ │
53
- │ spawnWorker(prompt) │ │
54
- │─────────────────────────────────────────────────────────────►│
55
- │◄──── WorkerSession {id, pid} │ │
56
- │ │ │
57
- │ ═══ LOOP ═══ │ │
58
- │ while running: │ │
59
- │ checkWorker(sessionId) │ │
60
- │ ────────────────────────────────────────────────────────►│
61
- │ ◄─── "running" / "completed" / "failed" │
62
- │ │ │
63
- │ [completed] → releaseLease │ │
64
- │ → markCompleted │ │
65
- │ │ │
66
- │ [failed] → markFailed │ │
67
- │ → retryTask() │ │
68
- │ → stopWorker() │ │
69
- │ │ │
70
- │ [timeout 30m] → stopWorker │ │
71
- │ → releaseLease│ │
72
- │ → retryTask() │ │</code></pre>
73
-
74
- <h3>Retry with Exponential Backoff</h3>
75
- <pre><code>Retry attempt Backoff delay Cumulative wait
76
- ───────────── ───────────── ──────────────
77
- 1 base × 2¹ = 30s 30s
78
- 2 base × 2² = 60s 90s
79
- 3 base × 2³ = 120s 210s
80
- 4 base × 2⁴ = 240s 450s
81
- 5 (max) base × 2⁵ = 480s 930s (~15 min)
82
-
83
- After max retries → dead_letter queue
84
- Dead-letter preserves: title, description, lastError, errorLog, retryCount</code></pre>
85
- <ul>
86
- <li><strong>Exponential backoff</strong> — base = 15s, factor = 2</li>
87
- <li><strong>Max retries</strong> — 5 per task (default)</li>
88
- <li><strong>Dead-letter queue</strong> — exhausted tasks are moved to <code>dead_letter</code> status with reason + error log</li>
89
- </ul>
90
-
91
- <h3>Worker Lifecycle + Concurrent Cap</h3>
92
- <ul>
93
- <li><strong>MAX_CONCURRENT_WORKERS = 3</strong> — prevents resource exhaustion</li>
94
- <li><strong>Worker timeout = 30 min</strong> — long-running tasks are killed to free resources</li>
95
- <li><strong>Worker poll = 10s</strong> — status checks via supervisor IPC</li>
96
- <li><strong>Loop sleep = 5s</strong> — idle interval when no tasks or workers full</li>
97
- </ul>
98
-
99
- <h3>Supervisor Integration</h3>
100
- <p>Workers are spawned through a <strong>Supervisor process</strong> (child_process) for crash isolation:</p>
101
- <ul>
102
- <li><strong>Crash isolation</strong> — worker crash doesn't crash the loop</li>
103
- <li><strong>Output capture</strong> — stdout/stderr saved to <code>~/.claude/daemon/jobs/{sessionId}/output.log</code></li>
104
- <li><strong>Health via IPC</strong> — loop checks status through supervisor, not raw PID (avoids PID reuse bugs)</li>
105
- </ul>
106
-
107
- <h3>Integration Points</h3>
108
- <ul>
109
- <li><strong>Peer todo listener</strong> — receives tasks from remote peers via <code>/peer-todo</code> HTTP endpoint → adds to queue</li>
110
- <li><strong>Cron scheduler</strong> — fires scheduled tasks → adds to queue</li>
111
- <li><strong>File watcher</strong> — <code>fs.watch</code> on queue file detects new tasks from other processes (e.g. CLI <code>/task add</code>)</li>
112
- </ul>
113
-
114
- <h3>Crash Recovery</h3>
115
- <pre><code>Previous Run Crash
116
- │
117
- â–¼
118
- startLoop() called
119
- │
120
- ├── loadQueue() — restore queue from disk
121
- ├── sleep(2000ms) — ensure old process is dead
122
- ├── expireLeases() — clear all stale leases
123
- ├── start heartbeat (every 60s)
124
- ├── start cron scheduler
125
- ├── start peer sharing
126
- ├── start file watcher
127
- └── MAIN LOOP ──► getNextTask() → processTask() → loop</code></pre>
128
-
129
- <h2>Task Lifecycle (State Machine)</h2>
130
- <pre><code> ┌──────────┐
131
- │ pending │◄──────────────────────────────┐
132
- └────┬─────┘ │
133
- │ leaseTask() │
134
- ▼ │
135
- ┌──────────┐ │
136
- │in_progress│ │
137
- └────┬─────┘ │
138
- ┌───────────┼───────────┐ │
139
- ▼ ▼ ▼ │
140
- ┌──────────┐ ┌──────────┐ ┌──────────┐ │
141
- │completed │ │ failed │ │cancelled │ │
142
- └──────────┘ └────┬─────┘ └──────────┘ │
143
- │ retryTask() │
144
- ├── retryCount < max → backoff → ────┘
145
- │
146
- └── retryCount ≥ max
147
- │
148
- â–¼
149
- ┌──────────────┐
150
- │ dead_letter │
151
- │ (preserved │
152
- │ for review) │
153
- └──────────────┘</code></pre>
154
-
155
- <h2>Related Files</h2>
156
- <table>
157
- <tr><th>File</th><th>Role</th></tr>
158
- <tr><td><code>src/services/autonomous/agentLoop.ts</code></td><td>Main loop — start, stop, processTask, worker lifecycle</td></tr>
159
- <tr><td><code>src/services/autonomous/taskQueue.ts</code></td><td>Queue CRUD, lease management, retry, dead-letter, file watcher</td></tr>
160
- <tr><td><code>src/services/autonomous/daemonMode.ts</code></td><td>Daemon entry point — calls startLoop/stopLoop</td></tr>
161
- <tr><td><code>src/Task.ts</code></td><td>Task type definitions, state machine, task ID generation</td></tr>
162
- <tr><td><code>src/tasks/LocalAgentTask/</code></td><td>Local worker task — UI + lifecycle</td></tr>
163
- <tr><td><code>src/tasks/RemoteAgentTask/</code></td><td>Remote worker task — UI + lifecycle</td></tr>
164
- <tr><td><code>src/components/AutonomousExecutionAccordion.tsx</code></td><td>UI component for task queue display in REPL</td></tr>
165
- </table>
166
-
167
- <footer class="footer">
168
- <span>Clew Code — Open Source</span>
169
- <div class="footer-links">
170
- <a href="https://github.com/ClewCode/ClewCode">GitHub</a>
171
- <a href="https://github.com/ClewCode/ClewCode/issues">Issues</a>
172
- </div>
173
- </footer>
174
- </main>
175
- <nav class="toc-sidebar"></nav>
176
- </div>
177
- </div>
178
- <script src="js/main.js"></script>
179
- </body>
180
- </html>
181
-
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>Agent Loop — Clew Code</title>
7
+ <meta name="description" content="Autonomous agent loop — task queue consumer, worker lifecycle, retry logic, dead-letter queue, lease-based concurrency">
8
+ <link rel="preconnect" href="https://fonts.googleapis.com">
9
+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
10
+ <link href="https://fonts.googleapis.com/css2?family=DM+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet">
11
+ <link rel="stylesheet" href="css/styles.css">
12
+ <link rel="icon" type="image/svg+xml" href="./assets/clew.svg">
13
+ </head>
14
+ <body>
15
+ <header class="header"></header>
16
+ <div class="app">
17
+ <aside class="sidebar" id="sidebar"></aside>
18
+ <div class="sidebar-overlay" id="sidebarOverlay"></div>
19
+ <div class="content-wrap">
20
+ <main class="content">
21
+ <div class="breadcrumbs"><a href="index.html">Home</a><span class="sep">/</span><span>Agent Loop</span></div>
22
+ <h1>Autonomous Agent Loop</h1>
23
+ <p class="section-subtitle">24/7 background execution loop — pulls tasks from queue, spawns workers, monitors status, retries on failure, dead-letters when exhausted.</p>
24
+
25
+ <p>The agent loop lives in <code>src/services/autonomous/agentLoop.ts</code> — the single consumer that reads the task queue and manages the lifecycle of all worker processes.</p>
26
+
27
+ <h2>Techniques &amp; Principles</h2>
28
+
29
+ <h3>Why a Loop instead of a Message Queue?</h3>
30
+ <p>Message queue systems like RabbitMQ/Kafka carry setup and maintenance overhead. The agent loop uses a <strong>JSON file as a persistent queue</strong> — zero external dependencies:</p>
31
+ <ul>
32
+ <li><strong>Zero external deps</strong> — uses <code>fs.watch</code> to detect queue file changes</li>
33
+ <li><strong>File-based atomicity</strong> — single-file read/write under lock allows multiple processes to compete for consumption</li>
34
+ <li><strong>Self-debouncing</strong> — <code>ourWriteInProgress</code> flag prevents self-triggered re-reads</li>
35
+ <li><strong>Cross-platform</strong> — JSON files work everywhere, no broker installation needed</li>
36
+ </ul>
37
+
38
+ <h3>Lease-Based Concurrency</h3>
39
+ <p>The classic distributed worker problem: two workers see the same task and duplicate work. Leases solve this without a distributed lock:</p>
40
+ <pre><code>Main Loop Queue File (.json) Worker Process
41
+ │ │ │
42
+ │ getNextTask() │ │
43
+ │─────────────────────────────►│ │
44
+ │◄─── task {id, status:pending}│ │
45
+ │ │ │
46
+ │ leaseTask(id, agentId) │ │
47
+ │─────────────────────────────►│ │
48
+ │ (atomic: check no lease │ │
49
+ │ → write leaseOwner + │ │
50
+ │ leaseExpiresAt) │ │
51
+ │◄─────── true (leased) ──────│ │
52
+ │ │ │
53
+ │ spawnWorker(prompt) │ │
54
+ │─────────────────────────────────────────────────────────────►│
55
+ │◄──── WorkerSession {id, pid} │ │
56
+ │ │ │
57
+ │ ═══ LOOP ═══ │ │
58
+ │ while running: │ │
59
+ │ checkWorker(sessionId) │ │
60
+ │ ────────────────────────────────────────────────────────►│
61
+ │ ◄─── "running" / "completed" / "failed" │
62
+ │ │ │
63
+ │ [completed] → releaseLease │ │
64
+ │ → markCompleted │ │
65
+ │ │ │
66
+ │ [failed] → markFailed │ │
67
+ │ → retryTask() │ │
68
+ │ → stopWorker() │ │
69
+ │ │ │
70
+ │ [timeout 30m] → stopWorker │ │
71
+ │ → releaseLease│ │
72
+ │ → retryTask() │ │</code></pre>
73
+
74
+ <h3>Retry with Exponential Backoff</h3>
75
+ <pre><code>Retry attempt Backoff delay Cumulative wait
76
+ ───────────── ───────────── ──────────────
77
+ 1 base × 2¹ = 30s 30s
78
+ 2 base × 2² = 60s 90s
79
+ 3 base × 2³ = 120s 210s
80
+ 4 base × 2⁴ = 240s 450s
81
+ 5 (max) base × 2⁵ = 480s 930s (~15 min)
82
+
83
+ After max retries → dead_letter queue
84
+ Dead-letter preserves: title, description, lastError, errorLog, retryCount</code></pre>
85
+ <ul>
86
+ <li><strong>Exponential backoff</strong> — base = 15s, factor = 2</li>
87
+ <li><strong>Max retries</strong> — 5 per task (default)</li>
88
+ <li><strong>Dead-letter queue</strong> — exhausted tasks are moved to <code>dead_letter</code> status with reason + error log</li>
89
+ </ul>
90
+
91
+ <h3>Worker Lifecycle + Concurrent Cap</h3>
92
+ <ul>
93
+ <li><strong>MAX_CONCURRENT_WORKERS = 3</strong> — prevents resource exhaustion</li>
94
+ <li><strong>Worker timeout = 30 min</strong> — long-running tasks are killed to free resources</li>
95
+ <li><strong>Worker poll = 10s</strong> — status checks via supervisor IPC</li>
96
+ <li><strong>Loop sleep = 5s</strong> — idle interval when no tasks or workers full</li>
97
+ </ul>
98
+
99
+ <h3>Supervisor Integration</h3>
100
+ <p>Workers are spawned through a <strong>Supervisor process</strong> (child_process) for crash isolation:</p>
101
+ <ul>
102
+ <li><strong>Crash isolation</strong> — worker crash doesn't crash the loop</li>
103
+ <li><strong>Output capture</strong> — stdout/stderr saved to <code>~/.claude/daemon/jobs/{sessionId}/output.log</code></li>
104
+ <li><strong>Health via IPC</strong> — loop checks status through supervisor, not raw PID (avoids PID reuse bugs)</li>
105
+ </ul>
106
+
107
+ <h3>Integration Points</h3>
108
+ <ul>
109
+ <li><strong>Peer todo listener</strong> — receives tasks from remote peers via <code>/peer-todo</code> HTTP endpoint → adds to queue</li>
110
+ <li><strong>Cron scheduler</strong> — fires scheduled tasks → adds to queue</li>
111
+ <li><strong>File watcher</strong> — <code>fs.watch</code> on queue file detects new tasks from other processes (e.g. CLI <code>/task add</code>)</li>
112
+ </ul>
113
+
114
+ <h3>Crash Recovery</h3>
115
+ <pre><code>Previous Run Crash
116
+ │
117
+ â–¼
118
+ startLoop() called
119
+ │
120
+ ├── loadQueue() — restore queue from disk
121
+ ├── sleep(2000ms) — ensure old process is dead
122
+ ├── expireLeases() — clear all stale leases
123
+ ├── start heartbeat (every 60s)
124
+ ├── start cron scheduler
125
+ ├── start peer sharing
126
+ ├── start file watcher
127
+ └── MAIN LOOP ──► getNextTask() → processTask() → loop</code></pre>
128
+
129
+ <h2>Task Lifecycle (State Machine)</h2>
130
+ <pre><code> ┌──────────┐
131
+ │ pending │◄──────────────────────────────┐
132
+ └────┬─────┘ │
133
+ │ leaseTask() │
134
+ ▼ │
135
+ ┌──────────┐ │
136
+ │in_progress│ │
137
+ └────┬─────┘ │
138
+ ┌───────────┼───────────┐ │
139
+ ▼ ▼ ▼ │
140
+ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
141
+ │completed │ │ failed │ │cancelled │ │
142
+ └──────────┘ └────┬─────┘ └──────────┘ │
143
+ │ retryTask() │
144
+ ├── retryCount < max → backoff → ────┘
145
+ │
146
+ └── retryCount ≥ max
147
+ │
148
+ â–¼
149
+ ┌──────────────┐
150
+ │ dead_letter │
151
+ │ (preserved │
152
+ │ for review) │
153
+ └──────────────┘</code></pre>
154
+
155
+ <h2>Related Files</h2>
156
+ <table>
157
+ <tr><th>File</th><th>Role</th></tr>
158
+ <tr><td><code>src/services/autonomous/agentLoop.ts</code></td><td>Main loop — start, stop, processTask, worker lifecycle</td></tr>
159
+ <tr><td><code>src/services/autonomous/taskQueue.ts</code></td><td>Queue CRUD, lease management, retry, dead-letter, file watcher</td></tr>
160
+ <tr><td><code>src/services/autonomous/daemonMode.ts</code></td><td>Daemon entry point — calls startLoop/stopLoop</td></tr>
161
+ <tr><td><code>src/Task.ts</code></td><td>Task type definitions, state machine, task ID generation</td></tr>
162
+ <tr><td><code>src/tasks/LocalAgentTask/</code></td><td>Local worker task — UI + lifecycle</td></tr>
163
+ <tr><td><code>src/tasks/RemoteAgentTask/</code></td><td>Remote worker task — UI + lifecycle</td></tr>
164
+ <tr><td><code>src/components/AutonomousExecutionAccordion.tsx</code></td><td>UI component for task queue display in REPL</td></tr>
165
+ </table>
166
+
167
+ <footer class="footer">
168
+ <span>Clew Code — Open Source</span>
169
+ <div class="footer-links">
170
+ <a href="https://github.com/ClewCode/ClewCode">GitHub</a>
171
+ <a href="https://github.com/ClewCode/ClewCode/issues">Issues</a>
172
+ </div>
173
+ </footer>
174
+ </main>
175
+ <nav class="toc-sidebar"></nav>
176
+ </div>
177
+ </div>
178
+ <script src="js/main.js"></script>
179
+ </body>
180
+ </html>
181
+