wolverine-ai 6.0.0 → 6.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +771 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -101,6 +101,777 @@ The middleware handles everything: 402 responses, wallet signing, CDP facilitato
|
|
|
101
101
|
|
|
102
102
|
---
|
|
103
103
|
|
|
104
|
+
## Technical Paper: Autonomous AI Process Management for Node.js Servers
|
|
105
|
+
|
|
106
|
+
### 1. Abstract
|
|
107
|
+
|
|
108
|
+
Server downtime in production environments has direct costs: lost revenue, degraded user trust, and engineer hours spent on diagnosis and repair. The traditional incident response cycle -- detect, page an engineer, read logs, diagnose, write a fix, deploy, verify -- takes anywhere from 15 minutes to several hours. Much of this time is spent on routine errors that follow predictable patterns: missing dependencies, syntax mistakes, malformed configuration files, and port conflicts.
|
|
109
|
+
|
|
110
|
+
This paper describes Wolverine, a self-healing framework for Node.js servers that converts error output into structured AI prompts, routes errors through a tiered repair pipeline, and verifies fixes before restarting the process. The system uses a structured error-to-prompt conversion pipeline that enables autonomous recovery while keeping AI token costs low. Most errors are resolved for $0.00 to $0.01, with complex multi-file repairs typically under $0.10. The framework reduces mean time to recovery from minutes or hours (human intervention) to 3-60 seconds (autonomous AI repair), while maintaining security through defense-in-depth: secret redaction, prompt injection detection, file system sandboxing, and encrypted key storage.
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
### 2. The Case for AI-Driven Error Recovery
|
|
115
|
+
|
|
116
|
+
#### 2.1 The Traditional Incident Response Cycle
|
|
117
|
+
|
|
118
|
+
When a production server encounters an error, the typical response follows a well-known pattern:
|
|
119
|
+
|
|
120
|
+
```
|
|
121
|
+
1. Server crashes or returns 500 errors
|
|
122
|
+
2. Monitoring system detects the failure (30s - 5min)
|
|
123
|
+
3. Alert fires, pages on-call engineer (1-5min)
|
|
124
|
+
4. Engineer reads logs, identifies the error (5-15min)
|
|
125
|
+
5. Engineer diagnoses root cause (10-60min)
|
|
126
|
+
6. Engineer writes and tests a fix (15-120min)
|
|
127
|
+
7. Fix deployed to production (5-30min)
|
|
128
|
+
8. Engineer monitors for recurrence (15-30min)
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Total elapsed time: **30 minutes to 4+ hours**. During this entire window, the service is degraded or unavailable.
|
|
132
|
+
|
|
133
|
+
The financial impact scales with the nature of the service. An e-commerce API returning 500 errors loses transactions directly. A SaaS platform experiencing downtime erodes customer confidence. Even internal services have indirect costs in blocked engineering work and cascading failures.
|
|
134
|
+
|
|
135
|
+
#### 2.2 Error Messages as Natural Language Prompts
|
|
136
|
+
|
|
137
|
+
The key insight behind AI-driven error recovery is that error messages and stack traces are already structured natural language descriptions of what went wrong. Consider a typical Node.js error:
|
|
138
|
+
|
|
139
|
+
```
|
|
140
|
+
TypeError: Cannot read properties of undefined (reading 'map')
|
|
141
|
+
at /server/routes/users.js:47:23
|
|
142
|
+
at Array.forEach (<anonymous>)
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
This contains everything a competent developer needs to diagnose the problem:
|
|
146
|
+
|
|
147
|
+
- **Error type**: `TypeError` -- a value is the wrong type
|
|
148
|
+
- **Specific cause**: Something is `undefined` when it should have a `.map` method
|
|
149
|
+
- **Location**: `server/routes/users.js`, line 47, column 23
|
|
150
|
+
- **Context**: Inside an `Array.forEach` callback
|
|
151
|
+
|
|
152
|
+
Converting this into an AI repair prompt is a zero-cost transformation. The error message does not need to be rewritten or reformatted -- it needs to be combined with the relevant source file and directed to an LLM with instructions to produce a fix. The conversion from "error text a human would read" to "error text an AI will read" is effectively free because they are the same text.
|
|
153
|
+
|
|
154
|
+
This observation has a practical consequence: the bottleneck in autonomous error recovery is not understanding the error -- it is (a) routing the error to the cheapest effective repair path, (b) preventing the AI from doing something harmful, and (c) verifying the fix actually works before restarting.
|
|
155
|
+
|
|
156
|
+
#### 2.3 Why Autonomous Recovery, Not Just Monitoring
|
|
157
|
+
|
|
158
|
+
Existing tools like PM2, nodemon, and systemd handle process restarts but not diagnosis or repair. They detect that a process died and restart it -- which simply triggers the same crash again if the underlying bug persists. APM tools (Datadog, New Relic, Sentry) excel at aggregation and alerting but still require a human to write the fix.
|
|
159
|
+
|
|
160
|
+
Wolverine fills the gap between "we know the server crashed" and "someone needs to fix it." For errors that follow patterns the AI can handle -- which in practice covers the majority of runtime crashes in Node.js applications -- the system eliminates the human from the loop entirely. For errors it cannot handle (external service outages, expired API keys, disk full), it detects that no code fix is possible and routes to human notification instead of wasting AI tokens.
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
164
|
+
### 3. Token Efficiency Through Structured Diagnosis
|
|
165
|
+
|
|
166
|
+
#### 3.1 The Problem with Naive AI Repair
|
|
167
|
+
|
|
168
|
+
A naive approach to AI-driven error recovery would be: send the entire error output and the full source code to an LLM with the prompt "fix this." This approach has several problems:
|
|
169
|
+
|
|
170
|
+
- **Token waste**: Sending 50KB of source files when the error is in one 20-line function
|
|
171
|
+
- **Ambiguity**: Without classification, the AI may attempt code fixes for problems that require operational actions (like `npm install`)
|
|
172
|
+
- **Cost**: At $3-15 per million input tokens (depending on the model), unstructured prompts can cost $0.15-$0.75 per repair attempt
|
|
173
|
+
- **Latency**: More tokens means slower response times
|
|
174
|
+
|
|
175
|
+
#### 3.2 Wolverine's Tiered Repair Pipeline
|
|
176
|
+
|
|
177
|
+
Wolverine addresses this with a four-tier approach that routes each error to the cheapest effective repair path. The tiers are arranged by cost, with the cheapest options tried first:
|
|
178
|
+
|
|
179
|
+
```
|
|
180
|
+
Tier 0: Operational Fix — $0.00 (zero AI tokens)
|
|
181
|
+
Tier 1: Fast Path — $0.001-$0.01 (single AI call, focused context)
|
|
182
|
+
Tier 2: Agent — $0.01-$0.10 (multi-turn, 32 tools)
|
|
183
|
+
Tier 3: Sub-Agents — $0.05-$0.15 (explore -> plan -> fix with model escalation)
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
**Tier 0: Operational Fixes (Zero Tokens)**
|
|
187
|
+
|
|
188
|
+
Many production errors have deterministic solutions that require no AI involvement:
|
|
189
|
+
|
|
190
|
+
| Error Pattern | Detection | Resolution |
|
|
191
|
+
|---------------|-----------|------------|
|
|
192
|
+
| `Cannot find module 'cors'` | `missing_module` classification | `npm install cors` |
|
|
193
|
+
| `EADDRINUSE :3000` | `port_conflict` classification | Find and kill stale process |
|
|
194
|
+
| `ENOENT: no such file or directory 'config.json'` | `missing_file` classification | Read source code, infer expected fields, create file |
|
|
195
|
+
| `EACCES: permission denied` | `permission` classification | `chmod 755` on the target file |
|
|
196
|
+
|
|
197
|
+
These are handled by the `diagnoseDeps()` function and operational fix logic in the heal pipeline. The error parser classifies the error type, and the fix is applied directly -- no AI call, no token cost, no network latency. In practice, dependency errors (`Cannot find module`) are among the most common production crashes, and resolving them for $0.00 in under 2 seconds is a significant cost advantage.
|
|
198
|
+
|
|
199
|
+
For `ENOENT` (missing file) errors specifically, the system reads the source code that references the missing file, infers the expected structure (for JSON configs, it deduces the expected fields from how the config is accessed in code), and creates the file with the correct content. This is more sophisticated than simply creating an empty file, but still requires zero AI tokens.
|
|
200
|
+
|
|
201
|
+
**Tier 1: Fast Path (Minimal Tokens)**
|
|
202
|
+
|
|
203
|
+
For errors that require code changes but are localized to a single file:
|
|
204
|
+
|
|
205
|
+
```
|
|
206
|
+
Input to AI:
|
|
207
|
+
- Error message (redacted)
|
|
208
|
+
- Stack trace (redacted)
|
|
209
|
+
- Source file contents (the file where the error occurred)
|
|
210
|
+
- Last known good version (backup diff context)
|
|
211
|
+
|
|
212
|
+
Output from AI:
|
|
213
|
+
- JSON with code patches AND/OR shell commands
|
|
214
|
+
- Both are applied: commands first (npm install, mkdir), then patches
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
The fast path uses the `CODING_MODEL` (typically Claude Sonnet or GPT-4o) with a focused context window. By sending only the error, the relevant file, and the backup diff (so the AI can see what changed), the input is typically 1,000-3,000 tokens. The output is a structured JSON response with specific file edits.
|
|
218
|
+
|
|
219
|
+
Token budget for fast path: **20,000 tokens** for simple errors.
|
|
220
|
+
|
|
221
|
+
**Tier 2: Agent (Moderate Tokens)**
|
|
222
|
+
|
|
223
|
+
When the fast path fails verification (the fix did not resolve the error), the system escalates to a multi-turn agent with access to 32 tools:
|
|
224
|
+
|
|
225
|
+
| Category | Tools | Purpose |
|
|
226
|
+
|----------|-------|---------|
|
|
227
|
+
| File | `read_file`, `write_file`, `edit_file`, `glob_files`, `grep_code`, `list_dir`, `move_file` | Navigate and modify the codebase |
|
|
228
|
+
| Shell | `bash_exec`, `git_log`, `git_diff` | Run commands, inspect history |
|
|
229
|
+
| Database | `inspect_db`, `run_db_fix` | Examine and repair SQLite databases |
|
|
230
|
+
| Diagnostics | `check_port`, `check_env` | Investigate runtime environment |
|
|
231
|
+
| Dependencies | `audit_deps`, `check_migration` | Dependency health and upgrade paths |
|
|
232
|
+
| Research | `web_fetch` | Look up documentation and solutions |
|
|
233
|
+
| Control | `done` | Signal task completion |
|
|
234
|
+
|
|
235
|
+
The agent receives a dynamic system prompt sized to the error complexity:
|
|
236
|
+
|
|
237
|
+
- **Simple errors** (TypeError, ReferenceError): 400-token prompt with 7 essential tools
|
|
238
|
+
- **Complex errors** (multi-file, database, configuration): 1,200-token prompt with all 32 tools plus a fix strategy table mapping error types to recommended approaches
|
|
239
|
+
|
|
240
|
+
Token budget: **simple = 20K, moderate = 50K, complex = 100K**.
|
|
241
|
+
|
|
242
|
+
The agent also benefits from context compaction: every 3 turns, the conversation history is structurally compressed (extracting tool calls, file modifications, and error signals) without making an additional AI call. This zero-cost compaction prevents token usage from growing linearly with turn count.
|
|
243
|
+
|
|
244
|
+
**Tier 3: Sub-Agents (Maximum Capability)**
|
|
245
|
+
|
|
246
|
+
For errors that resist single-agent repair, the system spawns specialized sub-agents in sequence:
|
|
247
|
+
|
|
248
|
+
```
|
|
249
|
+
1. EXPLORE agent (read-only, REASONING_MODEL)
|
|
250
|
+
- Investigates the codebase: reads files, checks ports, inspects databases
|
|
251
|
+
- Produces a structured understanding of the problem
|
|
252
|
+
|
|
253
|
+
2. PLAN agent (read-only, REASONING_MODEL)
|
|
254
|
+
- Receives the explorer's findings
|
|
255
|
+
- Produces a specific fix plan with file paths and changes
|
|
256
|
+
|
|
257
|
+
3. FIX agent (read+write, CODING_MODEL)
|
|
258
|
+
- Receives the plan
|
|
259
|
+
- Executes the fix: file edits, shell commands, database operations
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
Sub-agents use model escalation: the explore and plan agents use cheaper models (Haiku-tier for triage), and only the fix agent uses the more capable (and expensive) Sonnet or Opus model. This reduces sub-agent costs by approximately 90% compared to running all phases on the most expensive model.
|
|
263
|
+
|
|
264
|
+
#### 3.3 Real-World Token Efficiency
|
|
265
|
+
|
|
266
|
+
The tiered approach produces measurable savings. Consider a `TypeError: Cannot read properties of undefined (reading 'map')` in a route handler:
|
|
267
|
+
|
|
268
|
+
| Approach | Tokens Used | Cost | Time |
|
|
269
|
+
|----------|-------------|------|------|
|
|
270
|
+
| Naive ("fix my code" + full source) | ~25,000 | ~$0.08-$0.31 | 8-15s |
|
|
271
|
+
| Wolverine fast path | ~2,000-5,000 | ~$0.01-$0.02 | 3-8s |
|
|
272
|
+
| If fast path fails, agent | ~8,000-20,000 | ~$0.03-$0.10 | 15-40s |
|
|
273
|
+
|
|
274
|
+
The framework also prevents token waste through several mechanisms:
|
|
275
|
+
|
|
276
|
+
- **Empty stderr guard**: Signal kills and clean shutdowns produce empty or near-empty stderr. The system detects `stderr.trim().length < 10` and skips the entire heal pipeline, saving 100% of tokens that would be wasted on non-errors.
|
|
277
|
+
- **Loop guard**: If the same error (identified by a normalized signature of error message + file path) fails to heal 3 times within 10 minutes, the system stops healing that error, files a bug report, and moves on. This prevents infinite loops where the AI keeps producing the same incorrect fix.
|
|
278
|
+
- **Rate limiter**: A global cap of 5 heals per 5 minutes prevents runaway spending regardless of how many errors occur.
|
|
279
|
+
- **Prior attempt summaries**: When escalating between tiers, the system passes concise "do NOT repeat" directives from failed attempts rather than the full conversation history. This reduces baseline token count while preserving the critical information about what has already been tried.
|
|
280
|
+
|
|
281
|
+
---
|
|
282
|
+
|
|
283
|
+
### 4. The Heal Pipeline
|
|
284
|
+
|
|
285
|
+
#### 4.1 Error Detection
|
|
286
|
+
|
|
287
|
+
Wolverine detects errors through three channels:
|
|
288
|
+
|
|
289
|
+
**Channel 1: Process Crashes (stderr)**
|
|
290
|
+
|
|
291
|
+
The primary channel. When the child server process exits with a non-zero code, its stderr output is captured and fed into the heal pipeline. The runner spawns the child with `stdio: ["inherit", "inherit", "pipe", "ipc"]` -- stdout passes through to the terminal, stderr is piped to the parent for analysis, and an IPC channel enables in-process communication.
|
|
292
|
+
|
|
293
|
+
**Channel 2: Caught 500 Errors (IPC)**
|
|
294
|
+
|
|
295
|
+
Most production bugs in Fastify/Express applications do not crash the process. The framework catches the error and returns a 500 response. Wolverine detects these through an error hook (`error-hook.js`) that is preloaded via Node.js `--require` flag:
|
|
296
|
+
|
|
297
|
+
```
|
|
298
|
+
Route handler throws → Fastify/Express error handler catches
|
|
299
|
+
→ error-hook.js reports error to parent via IPC
|
|
300
|
+
→ ErrorMonitor tracks errors per normalized route
|
|
301
|
+
→ Threshold reached (default: 1) → triggers heal pipeline
|
|
302
|
+
→ 60-second cooldown per route prevents duplicate heals
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
Route normalization collapses dynamic segments: `/api/users/123` and `/api/users/456` both map to `/api/users/:id`. This prevents the same underlying bug from triggering multiple independent heal attempts for different parameter values.
|
|
306
|
+
|
|
307
|
+
The error hook uses a WeakSet for deduplication, ensuring the same error object is never reported twice. It also auto-registers a default error handler if the user's server code never calls `setErrorHandler`, catching async route throws that would otherwise be silently swallowed.
|
|
308
|
+
|
|
309
|
+
**Channel 3: Health Check Failures**
|
|
310
|
+
|
|
311
|
+
Configurable health probes (`/health`, `/healthz`, `/ready`) run on a regular interval. If the health check fails (timeout, non-200 response, connection refused), the system treats this as a frozen process -- it force-kills the child and triggers a heal cycle.
|
|
312
|
+
|
|
313
|
+
#### 4.2 Error Processing
|
|
314
|
+
|
|
315
|
+
Once an error is detected, it passes through a processing pipeline before any AI is invoked:
|
|
316
|
+
|
|
317
|
+
```
|
|
318
|
+
Raw stderr/error
|
|
319
|
+
│
|
|
320
|
+
├─ 1. Empty check: stderr.trim().length < 10 → just restart, no AI ($0.00)
|
|
321
|
+
│
|
|
322
|
+
├─ 2. Secret redaction: all .env.local values replaced with key names
|
|
323
|
+
│ "Connection failed: sk-abc123..." → "Connection failed: process.env.OPENAI_API_KEY..."
|
|
324
|
+
│
|
|
325
|
+
├─ 3. Error parsing: extract file path, line number, error type, message, stack trace
|
|
326
|
+
│
|
|
327
|
+
├─ 4. Injection scan: ~50 regex patterns check for prompt injection attempts
|
|
328
|
+
│ "ignore all previous instructions" → BLOCKED
|
|
329
|
+
│ "0x" + 64 hex chars (private key) → BLOCKED (key_leak_critical)
|
|
330
|
+
│ "rm -rf /" in error message → BLOCKED (destructive_bash)
|
|
331
|
+
│
|
|
332
|
+
├─ 5. Loop guard: same error failed 3+ times in 10min?
|
|
333
|
+
│ → Yes: file bug report, stop healing this error
|
|
334
|
+
│ → No: continue
|
|
335
|
+
│
|
|
336
|
+
├─ 6. Rate limit: 5 heals per 5 minutes exceeded?
|
|
337
|
+
│ → Yes: wait for window to clear
|
|
338
|
+
│ → No: continue
|
|
339
|
+
│
|
|
340
|
+
└─ 7. Enter goal loop (Tier 0 → 1 → 2 → 3)
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
Secret redaction happens before any other processing, including logging. This ensures that API keys, database passwords, and other sensitive values from `.env.local` never appear in AI prompts, brain memory, event logs, dashboard displays, or telemetry payloads. The redactor reads all values from `.env.local` and replaces each occurrence with its key name (e.g., `sk-abc123def` becomes `process.env.OPENAI_API_KEY`).
|
|
344
|
+
|
|
345
|
+
#### 4.3 The Goal Loop
|
|
346
|
+
|
|
347
|
+
The goal loop orchestrates the tiered repair attempts with verification between each tier:
|
|
348
|
+
|
|
349
|
+
```
|
|
350
|
+
Goal Loop (max 3 iterations):
|
|
351
|
+
|
|
352
|
+
Iteration 1: Fast Path
|
|
353
|
+
→ CODING_MODEL, single file + error context
|
|
354
|
+
→ AI returns JSON: { changes: [...], commands: [...] }
|
|
355
|
+
→ Execute commands (npm install, mkdir, etc.)
|
|
356
|
+
→ Apply file patches
|
|
357
|
+
→ Backup created before any file modification
|
|
358
|
+
→ VERIFY: syntax check → boot probe
|
|
359
|
+
→ Pass? → Record success to brain, done
|
|
360
|
+
→ Fail? → Rollback changes, continue to iteration 2
|
|
361
|
+
|
|
362
|
+
Iteration 2: Agent
|
|
363
|
+
→ REASONING_MODEL with full tool harness (32 tools)
|
|
364
|
+
→ Dynamic prompt based on error complexity
|
|
365
|
+
→ Agent investigates, modifies files, runs commands
|
|
366
|
+
→ Turn budget: simple=4, config=5, complex=8 turns
|
|
367
|
+
→ VERIFY: syntax check → boot probe
|
|
368
|
+
→ Pass? → Record success to brain, done
|
|
369
|
+
→ Fail? → Rollback changes, continue to iteration 3
|
|
370
|
+
|
|
371
|
+
Iteration 3: Sub-Agents
|
|
372
|
+
→ explore (read-only) → plan (read-only) → fix (read+write)
|
|
373
|
+
→ Haiku-tier triage, Sonnet/Opus-tier fix only
|
|
374
|
+
→ Each failure from prior iterations fed as context
|
|
375
|
+
→ VERIFY: syntax check → boot probe
|
|
376
|
+
→ Pass? → Record success to brain, done
|
|
377
|
+
→ Fail? → Rollback changes, report failure
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
#### 4.4 Verification
|
|
381
|
+
|
|
382
|
+
Every fix attempt is verified before the server restarts:
|
|
383
|
+
|
|
384
|
+
1. **Syntax check**: The modified files are parsed by Node.js to detect syntax errors. A fix that introduces a new syntax error is rejected immediately.
|
|
385
|
+
|
|
386
|
+
2. **Boot probe**: The server is started in a temporary process to verify it boots without crashing. The probe listens for either a successful startup signal or an error within a timeout window.
|
|
387
|
+
|
|
388
|
+
3. **Error classification comparison**: The verifier compares the error class of the original crash with any error produced during the boot probe. If the boot probe produces the same error class, the fix is considered unsuccessful.
|
|
389
|
+
|
|
390
|
+
For simple errors (TypeError, ReferenceError), the route probe step is skipped. The rationale is that the ErrorMonitor is already active and will catch any remaining 500 errors on the affected route after restart. This optimization avoids an additional $0.29 route-probe cost on errors where the syntax check and boot probe provide sufficient confidence.
|
|
391
|
+
|
|
392
|
+
#### 4.5 Learning
|
|
393
|
+
|
|
394
|
+
After a successful heal:
|
|
395
|
+
|
|
396
|
+
1. The fix is recorded in **repair history** with full metadata: error message, error type, file path, resolution description, tokens consumed, cost, repair mode (fast path/agent/sub-agent), and duration.
|
|
397
|
+
|
|
398
|
+
2. The fix is stored in the **brain** (vector store) with the error context, enabling future heals to find relevant past fixes through semantic search.
|
|
399
|
+
|
|
400
|
+
3. The backup created before the fix is promoted through the lifecycle: UNSTABLE (just created) to VERIFIED (fix passed verification) to STABLE (30 minutes of uptime without crashes).
|
|
401
|
+
|
|
402
|
+
After a failed heal:
|
|
403
|
+
|
|
404
|
+
1. The brain records the failed attempt with a "DO NOT REPEAT" tag, ensuring future attempts on similar errors do not try the same approach.
|
|
405
|
+
|
|
406
|
+
2. All file changes are rolled back to the pre-heal backup.
|
|
407
|
+
|
|
408
|
+
3. The loop guard increments its counter for this error signature.
|
|
409
|
+
|
|
410
|
+
---
|
|
411
|
+
|
|
412
|
+
### 5. Security Architecture
|
|
413
|
+
|
|
414
|
+
AI-driven code modification introduces a new class of security concerns. An attacker who can control error messages (through crafted requests, poisoned dependencies, or malicious input) could potentially use prompt injection to make the AI write malicious code. Wolverine addresses this through defense-in-depth across six layers.
|
|
415
|
+
|
|
416
|
+
#### 5.1 Secret Redaction
|
|
417
|
+
|
|
418
|
+
All values from `.env.local` are loaded at startup and automatically replaced in every string that passes through the system. This includes:
|
|
419
|
+
|
|
420
|
+
- AI prompts (error messages, file contents, system prompts)
|
|
421
|
+
- Brain memory (stored fixes, learnings, function maps)
|
|
422
|
+
- Event logs (JSONL persistence)
|
|
423
|
+
- Dashboard displays (real-time event stream)
|
|
424
|
+
- Telemetry payloads (heartbeat data)
|
|
425
|
+
|
|
426
|
+
The replacement is value-to-key: the actual secret value `sk-abc123...` is replaced with the string `process.env.OPENAI_API_KEY`. This preserves the semantic meaning (the AI knows an API key is involved) while preventing the actual credential from leaking.
|
|
427
|
+
|
|
428
|
+
#### 5.2 Prompt Injection Detection
|
|
429
|
+
|
|
430
|
+
Every error message is scanned before being sent to any AI model. The detection operates in two layers that both always run:
|
|
431
|
+
|
|
432
|
+
**Layer 1: Pattern Matching (Free)**
|
|
433
|
+
|
|
434
|
+
Approximately 50 regex patterns detect known injection techniques:
|
|
435
|
+
|
|
436
|
+
| Category | Example Patterns | Label |
|
|
437
|
+
|----------|------------------|-------|
|
|
438
|
+
| Prompt override | `ignore all previous instructions`, `forget previous` | `prompt-override` |
|
|
439
|
+
| Role hijack | `you are now a`, `pretend to be` | `role-hijack` |
|
|
440
|
+
| Code execution | `eval(`, `require('child_process')`, `Function(` | `code-exec` |
|
|
441
|
+
| Data exfiltration | `process.env`, `curl`, `fetch('http...` | `exfiltration` |
|
|
442
|
+
| Destructive operations | `rm -rf`, `rimraf`, `fs.unlinkSync` | `destructive-fs` |
|
|
443
|
+
| Key material leak | 64-character hex strings (private keys) | `key-leak-critical` |
|
|
444
|
+
| Vault path references | `master.key`, `eth.vault`, `.wolverine/vault` | `vault-path-leak` |
|
|
445
|
+
| Bash sandbox escape | `rm -rf /`, `rmdir /`, `shutdown`, `reboot` | `destructive-bash` |
|
|
446
|
+
|
|
447
|
+
If any pattern matches, the heal is blocked entirely. The error is logged with the detected label, and no AI model is invoked.
|
|
448
|
+
|
|
449
|
+
**Layer 2: AI-Powered Deep Scan**
|
|
450
|
+
|
|
451
|
+
Even if no regex pattern matches, every error message is also analyzed by the `AUDIT_MODEL` (typically the cheapest available model, such as GPT-4o-mini or Claude Haiku). This catches novel injection attempts that do not match known patterns. The audit model is specifically prompted to identify attempts to manipulate AI behavior through error messages.
|
|
452
|
+
|
|
453
|
+
Both layers must pass for the error to proceed to the repair pipeline. If either layer flags the error, it is blocked.
|
|
454
|
+
|
|
455
|
+
#### 5.3 File System Sandbox
|
|
456
|
+
|
|
457
|
+
The agent operates within a strict sandbox that limits which files it can read and modify:
|
|
458
|
+
|
|
459
|
+
**Writable paths**: Only files within `server/` are writable by the agent. This is the user's application code -- the only code the agent should ever need to modify.
|
|
460
|
+
|
|
461
|
+
**Read-only paths**: The agent can read files outside `server/` for investigation purposes (understanding imports, checking configurations) but cannot modify them.
|
|
462
|
+
|
|
463
|
+
**Blocked paths** (neither read nor write by the agent's modification tools):
|
|
464
|
+
- `src/` -- Wolverine framework source
|
|
465
|
+
- `bin/` -- CLI entry points
|
|
466
|
+
- `tests/` -- Test suite
|
|
467
|
+
- `node_modules/` -- Dependencies
|
|
468
|
+
- `.env` and `.env.local` -- Secrets
|
|
469
|
+
- `package.json` -- Dependency manifest
|
|
470
|
+
|
|
471
|
+
Symlink escape detection prevents the agent from creating symbolic links that point outside the sandbox. The sandbox resolves all paths to their real location before applying access checks.
|
|
472
|
+
|
|
473
|
+
#### 5.4 Command Blocking
|
|
474
|
+
|
|
475
|
+
The agent's `bash_exec` tool filters all commands through 18+ blocked patterns before execution:
|
|
476
|
+
|
|
477
|
+
```
|
|
478
|
+
Blocked command patterns:
|
|
479
|
+
rm -rf / — recursive deletion of root
|
|
480
|
+
rmdir / — directory deletion at root
|
|
481
|
+
format, mkfs, dd — disk formatting
|
|
482
|
+
shutdown, reboot — system control
|
|
483
|
+
git push --force — destructive git operations
|
|
484
|
+
git reset --hard — history destruction
|
|
485
|
+
npm publish — accidental package publication
|
|
486
|
+
curl | bash — remote code execution
|
|
487
|
+
wget | sh — remote code execution
|
|
488
|
+
eval( — dynamic code execution
|
|
489
|
+
cat .env — secret file reading via shell
|
|
490
|
+
> src/ — redirect write into framework source
|
|
491
|
+
cp ... src/ — copy into framework source
|
|
492
|
+
mv ... src/ — move into framework source
|
|
493
|
+
tee ... src/ — tee into framework source
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
Additionally, a sandbox escape detector blocks commands that operate on paths outside the project directory. Commands containing `../` traversals, absolute paths outside the project root, or references to system directories are rejected.
|
|
497
|
+
|
|
498
|
+
#### 5.5 Adaptive Rate Limiting
|
|
499
|
+
|
|
500
|
+
The adaptive rate limiter monitors system resources in real-time and throttles incoming requests when the server is under pressure:
|
|
501
|
+
|
|
502
|
+
| Zone | CPU/Memory | Behavior |
|
|
503
|
+
|------|-----------|----------|
|
|
504
|
+
| GREEN | < 70% | Full throughput, no limiting |
|
|
505
|
+
| YELLOW | 70-85% | Gradual connection shedding |
|
|
506
|
+
| RED | > 85% | Reject non-essential requests with 503 |
|
|
507
|
+
|
|
508
|
+
The limiter samples CPU and memory every 5 seconds, maintaining a 1-minute rolling history (12 samples). It reserves approximately 200MB of memory headroom for Wolverine's heal tools -- ensuring the AI repair pipeline can operate even when the application server is under heavy load.
|
|
509
|
+
|
|
510
|
+
Exempt paths that are never rate-limited: `/health`, `/healthz`, `/ready`, and Wolverine internal routes (`/api/v1/heartbeat`, `/api/v1/register`). Requests with the `X-Wolverine-Internal` header also bypass the limiter.
|
|
511
|
+
|
|
512
|
+
#### 5.6 Vault Encryption
|
|
513
|
+
|
|
514
|
+
Private keys (Ethereum wallet keys for x402 payments) are stored using AES-256-GCM encryption:
|
|
515
|
+
|
|
516
|
+
```
|
|
517
|
+
.wolverine/vault/
|
|
518
|
+
master.key — 32 bytes raw AES-256 key (chmod 0600)
|
|
519
|
+
eth.vault — JSON with AES-256-GCM encrypted private key
|
|
520
|
+
```
|
|
521
|
+
|
|
522
|
+
Design principles enforced by the vault:
|
|
523
|
+
|
|
524
|
+
- **Buffer-only key handling**: Private keys never exist as JavaScript strings. They are stored in `Buffer` objects and explicitly zeroed (`buffer.fill(0)`) after use. JavaScript strings are immutable and garbage-collected nondeterministically, making them unsuitable for secret storage.
|
|
525
|
+
- **Generic error messages**: The wallet operations layer (`wallet-ops.js`) catches and swallows all vault error details before they can reach the AI. If the vault fails, the AI sees "Vault operation failed" rather than any information about the key material.
|
|
526
|
+
- **Single secret on disk**: The `master.key` file is the only unencrypted secret. Everything else (Ethereum private key, future secrets) is encrypted with this master key.
|
|
527
|
+
- **Agent isolation**: The sandbox blocks the agent from reading any path containing `.wolverine/vault`. Even if an injection attack succeeded in bypassing other protections, the agent's tools physically cannot access the vault directory.
|
|
528
|
+
- **Persistence**: The vault lives in `.wolverine/`, which survives `git pull`, `npm install`, and auto-updates. Vault files are also included in every backup snapshot and are on the protected list for rollback (never overwritten during restore).
|
|
529
|
+
|
|
530
|
+
#### 5.7 SSRF Protection
|
|
531
|
+
|
|
532
|
+
The agent's `web_fetch` tool blocks requests to private IP ranges (10.x.x.x, 172.16-31.x.x, 192.168.x.x, 127.x.x.x, and IPv6 equivalents). This prevents server-side request forgery attacks where a crafted error message tricks the agent into fetching internal services or metadata endpoints (such as cloud provider instance metadata at 169.254.169.254).
|
|
533
|
+
|
|
534
|
+
---
|
|
535
|
+
|
|
536
|
+
### 6. The Brain: Vector-Indexed Memory
|
|
537
|
+
|
|
538
|
+
#### 6.1 Architecture
|
|
539
|
+
|
|
540
|
+
The brain is a hybrid retrieval system combining semantic vector search with keyword-based BM25 search. It serves as the framework's persistent memory, storing past fixes, error patterns, tool documentation, and learned behaviors.
|
|
541
|
+
|
|
542
|
+
The vector store uses five optimization techniques for performance at scale:
|
|
543
|
+
|
|
544
|
+
1. **Pre-normalized vectors**: All embedding vectors are L2-normalized at insertion time. This converts cosine similarity computation into a simple dot product (eliminating the square root operations that dominate naive implementations).
|
|
545
|
+
|
|
546
|
+
2. **IVF (Inverted File Index)**: Vectors are clustered into sqrt(N) buckets using k-means++ initialization. At query time, only the nearest 20% of clusters are probed, reducing search from O(N) to O(sqrt(N)).
|
|
547
|
+
|
|
548
|
+
3. **BM25 inverted index**: A proper term frequency-inverse document frequency index for keyword search. Lookup is O(query_tokens) instead of O(N) linear scan.
|
|
549
|
+
|
|
550
|
+
4. **Binary persistence**: Vectors are stored as Float32Array buffers in a binary file format, providing 10x faster load times and 4x smaller file sizes compared to JSON serialization.
|
|
551
|
+
|
|
552
|
+
5. **Incremental indexing**: New entries are added without rebuilding the entire index. The IVF index is only rebuilt when cluster balance degrades beyond a threshold.
|
|
553
|
+
|
|
554
|
+
**Search performance benchmarks:**
|
|
555
|
+
|
|
556
|
+
| Entries | Semantic Search | BM25 Keyword | IVF Clusters |
|
|
557
|
+
|---------|----------------|--------------|--------------|
|
|
558
|
+
| 100 | 0.2ms | 0.005ms | 10 |
|
|
559
|
+
| 1,000 | 0.4ms | 0.01ms | 32 |
|
|
560
|
+
| 10,000 | 4.4ms | 0.1ms | 100 |
|
|
561
|
+
| 50,000 | 23.7ms | 0.5ms | 224 |
|
|
562
|
+
|
|
563
|
+
These benchmarks demonstrate sub-linear scaling. At 50,000 entries, a semantic search completes in under 24ms -- well within the latency budget for real-time repair decisions.
|
|
564
|
+
|
|
565
|
+
#### 6.2 Namespace Isolation
|
|
566
|
+
|
|
567
|
+
The brain organizes content into namespaces:
|
|
568
|
+
|
|
569
|
+
| Namespace | Contents | Searchable During Heals |
|
|
570
|
+
|-----------|----------|------------------------|
|
|
571
|
+
| `errors` | Past error messages with context | Yes |
|
|
572
|
+
| `fixes` | Successful repairs with full metadata | Yes |
|
|
573
|
+
| `learnings` | Research findings, discovered patterns | Yes |
|
|
574
|
+
| `functions` | Function map (routes, classes, exports) | Yes |
|
|
575
|
+
| `seeds` | 60+ framework documentation entries | No (unless query is about Wolverine itself) |
|
|
576
|
+
|
|
577
|
+
Namespace isolation is critical for token efficiency. The 60+ seed documents contain approximately 20,000 tokens of framework documentation (tool descriptions, security patterns, best practices). If these were included in every error heal search, they would consume context window space without contributing to the repair. By isolating seeds to a separate namespace that is only searched when the query explicitly concerns Wolverine's own behavior, the system saves approximately 50% of context space during normal heal operations.
|
|
578
|
+
|
|
579
|
+
#### 6.3 Function Map
|
|
580
|
+
|
|
581
|
+
On startup and periodically during operation, the brain scans the `server/` directory and indexes:
|
|
582
|
+
|
|
583
|
+
- **Routes**: HTTP method, path, handler function, file location
|
|
584
|
+
- **Functions**: Named functions with parameters and approximate line ranges
|
|
585
|
+
- **Classes**: Class definitions with method lists
|
|
586
|
+
- **Exports**: Module export signatures
|
|
587
|
+
|
|
588
|
+
This function map serves two purposes. First, it provides the AI agent with a structural understanding of the codebase without reading every file (saving tokens). Second, it enables the route prober to auto-discover and monitor all endpoints.
|
|
589
|
+
|
|
590
|
+
The function map uses a content hash to detect changes. If the hash matches the previously scanned state, the re-embedding step is skipped -- avoiding unnecessary API calls to the embedding model.
|
|
591
|
+
|
|
592
|
+
#### 6.4 Learning Loop
|
|
593
|
+
|
|
594
|
+
Every successful heal feeds back into the brain:
|
|
595
|
+
|
|
596
|
+
```
|
|
597
|
+
Error occurs → heal pipeline fixes it → verification passes
|
|
598
|
+
→ Store in brain:
|
|
599
|
+
- Namespace: "fixes"
|
|
600
|
+
- Content: error message + classification + fix description
|
|
601
|
+
- Metadata: file path, tokens used, cost, repair mode, duration
|
|
602
|
+
- Embedding: vector representation for semantic search
|
|
603
|
+
|
|
604
|
+
Future similar error occurs → brain search finds past fix
|
|
605
|
+
→ Past fix context injected into AI prompt
|
|
606
|
+
→ AI sees what worked before, avoids what failed
|
|
607
|
+
→ Faster repair, fewer iterations, lower cost
|
|
608
|
+
```
|
|
609
|
+
|
|
610
|
+
Failed heals are also stored, tagged with "DO NOT REPEAT" metadata. When the brain search returns a failed past fix, the AI prompt explicitly instructs the model not to attempt the same approach.
|
|
611
|
+
|
|
612
|
+
---
|
|
613
|
+
|
|
614
|
+
### 7. x402 Paid APIs
|
|
615
|
+
|
|
616
|
+
#### 7.1 Protocol Overview
|
|
617
|
+
|
|
618
|
+
The x402 protocol extends HTTP with a payment layer using the `402 Payment Required` status code. When a client requests a paid endpoint without a payment header, the server responds with 402 and a JSON body describing the payment requirements (amount, asset, network, recipient address). The client constructs a payment authorization, signs it, and retransmits the request with an `X-Payment` header containing the signed payment.
|
|
619
|
+
|
|
620
|
+
This enables machine-to-machine payments without user interaction -- an AI agent calling a paid API can programmatically construct and sign payments.
|
|
621
|
+
|
|
622
|
+
#### 7.2 Integration
|
|
623
|
+
|
|
624
|
+
Wolverine's x402 implementation is a Fastify plugin that converts any route into a paid endpoint with a single configuration object:
|
|
625
|
+
|
|
626
|
+
```javascript
|
|
627
|
+
// Fixed price: $0.01 per call
|
|
628
|
+
fastify.get("/api/premium", {
|
|
629
|
+
config: { x402: { price: "$0.01" } }
|
|
630
|
+
}, handler);
|
|
631
|
+
|
|
632
|
+
// Variable price: caller specifies amount
|
|
633
|
+
fastify.post("/api/credits", {
|
|
634
|
+
config: { x402: { variable: true, min: "$1", max: "$10000", priceField: "dollars" } }
|
|
635
|
+
}, handler);
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
No additional middleware, no payment processing code in the handler. The plugin handles the entire flow in the `preHandler` hook:
|
|
639
|
+
|
|
640
|
+
```
|
|
641
|
+
Request arrives
|
|
642
|
+
│
|
|
643
|
+
├─ No x402 config on route → pass through (free route)
|
|
644
|
+
│
|
|
645
|
+
├─ Has x402 config, no payment header
|
|
646
|
+
│ → Return 402 with payment requirements
|
|
647
|
+
│
|
|
648
|
+
├─ Has x402 config + payment header
|
|
649
|
+
│ ├─ Decode base64 payment payload
|
|
650
|
+
│ ├─ Verify via x402 facilitator (CDP)
|
|
651
|
+
│ │ ├─ Invalid → Return 402 with reason
|
|
652
|
+
│ │ └─ Valid → Continue
|
|
653
|
+
│ ├─ Settle via facilitator (USDC moves on-chain)
|
|
654
|
+
│ │ ├─ Failed → Return 402
|
|
655
|
+
│ │ └─ Success → Continue
|
|
656
|
+
│ ├─ Attach payment info to request: req.x402 = { paid, amount, from, txHash }
|
|
657
|
+
│ └─ Execute route handler (payment confirmed)
|
|
658
|
+
```
|
|
659
|
+
|
|
660
|
+
Settlement uses EIP-3009 `TransferWithAuthorization` via the Coinbase CDP facilitator. The USDC transfer is confirmed on Base L2 before the route handler executes. The handler only runs after money has moved on-chain.
|
|
661
|
+
|
|
662
|
+
#### 7.3 Wallet Management
|
|
663
|
+
|
|
664
|
+
The payment recipient address is auto-detected from the encrypted vault:
|
|
665
|
+
|
|
666
|
+
1. `wolverine --init-vault` generates an Ethereum private key, encrypts it with AES-256-GCM, and stores it in `.wolverine/vault/eth.vault`
|
|
667
|
+
2. On startup, the x402 plugin reads the wallet address (public key derivation only -- the private key is never loaded into the plugin)
|
|
668
|
+
3. The `payTo` address can also be set manually in `settings.json` or passed as a plugin option
|
|
669
|
+
|
|
670
|
+
Payment logs are maintained in `.wolverine/x402-payments.json` (capped at 1,000 entries) with route, method, amount, payer address, transaction hash, and timestamp.
|
|
671
|
+
|
|
672
|
+
---
|
|
673
|
+
|
|
674
|
+
### 8. Operational Features
|
|
675
|
+
|
|
676
|
+
#### 8.1 Backup System
|
|
677
|
+
|
|
678
|
+
All backups are stored in `~/.wolverine-safe-backups/` -- a location outside the project directory that survives `git pull`, `npm install`, project deletion, and framework updates.
|
|
679
|
+
|
|
680
|
+
**Lifecycle states:**
|
|
681
|
+
|
|
682
|
+
| State | Meaning | Transition |
|
|
683
|
+
|-------|---------|------------|
|
|
684
|
+
| UNSTABLE | Just created, fix not yet verified | → VERIFIED when fix passes verification |
|
|
685
|
+
| VERIFIED | Fix passed syntax + boot probe | → STABLE after 30 minutes of uptime |
|
|
686
|
+
| STABLE | Server has been running without crashes for 30+ minutes | Terminal state |
|
|
687
|
+
|
|
688
|
+
**Creation triggers:**
|
|
689
|
+
- Before every heal attempt (automatic)
|
|
690
|
+
- Before every framework update (automatic)
|
|
691
|
+
- On graceful shutdown (automatic)
|
|
692
|
+
- On manual request via CLI or dashboard
|
|
693
|
+
|
|
694
|
+
**Retention policy:**
|
|
695
|
+
- UNSTABLE and VERIFIED backups: pruned after 7 days
|
|
696
|
+
- STABLE backups older than 7 days: keep 1 per day (most recent that day)
|
|
697
|
+
|
|
698
|
+
**Protected files** (never overwritten during rollback):
|
|
699
|
+
- `server/config/settings.json` -- configuration
|
|
700
|
+
- `server/lib/db.js` -- database connection setup
|
|
701
|
+
- `server/lib/redis.js` -- Redis connection setup
|
|
702
|
+
- `.env` and `.env.local` -- secrets
|
|
703
|
+
- `.wolverine/vault/master.key` -- vault encryption key
|
|
704
|
+
- `.wolverine/vault/eth.vault` -- encrypted wallet
|
|
705
|
+
|
|
706
|
+
The rollback system creates a pre-rollback safety backup before restoring, enabling "undo rollback" if the restored state is worse than the current state.
|
|
707
|
+
|
|
708
|
+
#### 8.2 Auto-Update
|
|
709
|
+
|
|
710
|
+
The framework checks for new versions on npm and upgrades itself using a selective update strategy:
|
|
711
|
+
|
|
712
|
+
```
|
|
713
|
+
Update process:
|
|
714
|
+
1. Create emergency backup in ~/.wolverine-safe-backups/
|
|
715
|
+
2. Back up server/, .wolverine/, .env to memory
|
|
716
|
+
3. Update ONLY src/ and bin/ (framework code)
|
|
717
|
+
4. Update package.json dependencies
|
|
718
|
+
5. Restore all user files (server code, brain, backups, config)
|
|
719
|
+
6. Signal brain to merge new seed documents (append, not replace)
|
|
720
|
+
7. Verify boot
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
The selective approach is critical. A naive `git pull` or `npm install wolverine-ai` overwrites the `server/` directory, which contains user application code, routes, database configurations, and settings. The auto-updater explicitly avoids touching `server/`, `.wolverine/`, and `.env` files.
|
|
724
|
+
|
|
725
|
+
Update checks run 30 seconds after startup and then at a configurable interval (default: every 5 minutes, configurable via `settings.json` `autoUpdate.intervalMs`). A version lock ensures only one update attempt per boot cycle.
|
|
726
|
+
|
|
727
|
+
#### 8.3 Dashboard
|
|
728
|
+
|
|
729
|
+
The dashboard runs on `PORT+1` (default: 3001) and provides a real-time web interface:
|
|
730
|
+
|
|
731
|
+
| Panel | Data Source | Update Method |
|
|
732
|
+
|-------|------------|---------------|
|
|
733
|
+
| Overview | Event logger, repair history | SSE (real-time) |
|
|
734
|
+
| Events | Event log (JSONL) | SSE stream |
|
|
735
|
+
| Performance | Perf monitor, route prober | SSE + polling |
|
|
736
|
+
| Command | AI client (chat/tools/agent routes) | Request/response |
|
|
737
|
+
| Analytics | Token tracker, process monitor | Polling |
|
|
738
|
+
| Backups | Backup manager | On-demand |
|
|
739
|
+
| Brain | Vector store stats | On-demand |
|
|
740
|
+
| Repairs | Repair history | On-demand |
|
|
741
|
+
| Tools | Agent engine tool definitions | Static |
|
|
742
|
+
| Usage | Token tracker (by model/category/tool) | Polling |
|
|
743
|
+
|
|
744
|
+
The command interface classifies each user message into one of three routes:
|
|
745
|
+
|
|
746
|
+
- **SIMPLE** (CHAT_MODEL, no tools): Knowledge questions, explanations
|
|
747
|
+
- **TOOLS** (TOOL_MODEL, limited tools): Live data queries, file reads, brain searches
|
|
748
|
+
- **AGENT** (CODING_MODEL, full 32-tool harness): Build features, fix code, modify server
|
|
749
|
+
|
|
750
|
+
Secured with `WOLVERINE_ADMIN_KEY` plus IP allowlist (localhost always allowed, additional IPs via `WOLVERINE_ADMIN_IPS` environment variable or runtime API).
|
|
751
|
+
|
|
752
|
+
#### 8.4 Process Management
|
|
753
|
+
|
|
754
|
+
Wolverine functions as a PM2-like process manager with AI-augmented diagnostics:
|
|
755
|
+
|
|
756
|
+
- **Heartbeat monitoring**: Checks process liveness every 10 seconds
|
|
757
|
+
- **Memory tracking**: RSS and heap size monitoring with leak detection (N consecutive growth samples trigger restart)
|
|
758
|
+
- **Memory limit**: Auto-restart when RSS exceeds configurable threshold (default: 512MB)
|
|
759
|
+
- **CPU sampling**: Per-process CPU percentage with trend detection
|
|
760
|
+
- **SIGKILL/OOM detection**: Detects when the OS kills the process (out of memory, signal 9)
|
|
761
|
+
- **Spawn retry**: Configurable retry logic for process startup failures
|
|
762
|
+
- **Graceful shutdown**: SIGTERM with configurable grace period, escalating to SIGKILL
|
|
763
|
+
- **SIGTERM startup grace**: 3-second window after spawning where SIGTERM is ignored, preventing restart scripts from killing a newly spawned process
|
|
764
|
+
- **Process dedup**: PID file ensures only one Wolverine instance runs per project. On startup, any existing process with a stale PID file is killed. The exit handler only deletes the PID file if it still belongs to the current process, preventing race conditions during rapid restarts.
|
|
765
|
+
|
|
766
|
+
#### 8.5 Health Monitoring
|
|
767
|
+
|
|
768
|
+
Configurable health probes run on a regular interval:
|
|
769
|
+
|
|
770
|
+
- Default endpoints: `/health`, `/healthz`, `/ready`
|
|
771
|
+
- Timeout: configurable per-probe
|
|
772
|
+
- Failure behavior: force-kill the process and trigger a heal cycle
|
|
773
|
+
- Integration: health check failures feed into the same heal pipeline as crashes
|
|
774
|
+
|
|
775
|
+
#### 8.6 Cluster Support
|
|
776
|
+
|
|
777
|
+
The server handles its own clustering internally. Wolverine remains a single process manager that spawns the server entry point. If `WOLVERINE_CLUSTER=true`, the server's entry point forks worker processes:
|
|
778
|
+
|
|
779
|
+
```
|
|
780
|
+
Wolverine (single process)
|
|
781
|
+
└─ server/index.js (master)
|
|
782
|
+
├─ Worker 1 (port 3000, reusePort)
|
|
783
|
+
├─ Worker 2 (port 3000, reusePort)
|
|
784
|
+
└─ Worker N (port 3000, reusePort)
|
|
785
|
+
```
|
|
786
|
+
|
|
787
|
+
Workers share port 3000 via `reusePort` with OS-level load balancing. Dead workers auto-respawn. Wolverine kills the entire process tree on restart to prevent orphaned workers. The `WOLVERINE_RECOMMENDED_WORKERS` environment variable is auto-set based on detected CPU cores and available RAM.
|
|
788
|
+
|
|
789
|
+
---
|
|
790
|
+
|
|
791
|
+
### 9. Benchmarks and Real-World Performance
|
|
792
|
+
|
|
793
|
+
#### 9.1 Heal Timing
|
|
794
|
+
|
|
795
|
+
Measured from error detection to verified fix applied and server restarted:
|
|
796
|
+
|
|
797
|
+
| Error Type | Typical Heal Time | Repair Tier |
|
|
798
|
+
|------------|-------------------|-------------|
|
|
799
|
+
| Missing module (`Cannot find module`) | 2-5 seconds | Tier 0 (operational) |
|
|
800
|
+
| Port conflict (`EADDRINUSE`) | 1-3 seconds | Tier 0 (operational) |
|
|
801
|
+
| Missing config file (`ENOENT`) | 2-4 seconds | Tier 0 (operational) |
|
|
802
|
+
| Simple TypeError/ReferenceError | 3-8 seconds | Tier 1 (fast path) |
|
|
803
|
+
| Syntax error | 3-10 seconds | Tier 1 (fast path) |
|
|
804
|
+
| Multi-file import mismatch | 15-40 seconds | Tier 2 (agent) |
|
|
805
|
+
| Database schema error | 20-60 seconds | Tier 2 (agent) |
|
|
806
|
+
| Complex multi-component bug | 30-90 seconds | Tier 3 (sub-agents) |
|
|
807
|
+
|
|
808
|
+
Operational fixes (Tier 0) complete in under 5 seconds because they involve no network calls to AI providers. The bottleneck is the `npm install` or `kill` command itself.
|
|
809
|
+
|
|
810
|
+
The 5-minute heal timeout (configurable via `WOLVERINE_HEAL_TIMEOUT_MS`) acts as a hard upper bound. If a heal exceeds this limit, partially applied changes are rolled back and the system reports failure.
|
|
811
|
+
|
|
812
|
+
#### 9.2 Token Usage
|
|
813
|
+
|
|
814
|
+
Distribution of token consumption across heal operations:
|
|
815
|
+
|
|
816
|
+
| Metric | Value |
|
|
817
|
+
|--------|-------|
|
|
818
|
+
| Heals using zero tokens (Tier 0) | ~30-40% of all heals |
|
|
819
|
+
| Heals using < 5,000 tokens (Tier 1) | ~40-50% of all heals |
|
|
820
|
+
| Heals using 5,000-20,000 tokens (Tier 2) | ~15-25% of all heals |
|
|
821
|
+
| Heals using > 20,000 tokens (Tier 3) | ~5-10% of all heals |
|
|
822
|
+
|
|
823
|
+
The distribution is heavily skewed toward cheap operations. The majority of production errors fall into categories that either require no AI (missing dependencies, port conflicts) or are simple enough for a single focused AI call (typos, null reference errors, missing null checks).
|
|
824
|
+
|
|
825
|
+
#### 9.3 Cost Per Heal
|
|
826
|
+
|
|
827
|
+
Estimated costs using typical model pricing (Claude Sonnet at ~$3/$15 per million input/output tokens):
|
|
828
|
+
|
|
829
|
+
| Repair Tier | Token Range | Estimated Cost |
|
|
830
|
+
|-------------|-------------|---------------|
|
|
831
|
+
| Tier 0: Operational | 0 | $0.00 |
|
|
832
|
+
| Tier 1: Fast Path | 1,000-5,000 | $0.001-$0.02 |
|
|
833
|
+
| Tier 2: Agent | 5,000-50,000 | $0.02-$0.10 |
|
|
834
|
+
| Tier 3: Sub-Agents | 20,000-100,000 | $0.05-$0.15 |
|
|
835
|
+
|
|
836
|
+
Cost optimizations that contribute to these numbers:
|
|
837
|
+
|
|
838
|
+
- **Prompt caching** (Anthropic): System prompts are marked with `cache_control: ephemeral`. On repeat calls (which are common during multi-turn agent sessions), the cached system prompt is 90% cheaper. For a typical 12,000-16,000 token system prompt, this saves $0.03-$0.04 per heal.
|
|
839
|
+
- **Haiku triage for sub-agents**: Explorer, planner, and verifier sub-agents use the cheapest model tier. Only the fixer uses the more expensive model.
|
|
840
|
+
- **Dynamic prompt sizing**: Simple errors get a 400-token system prompt with 7 tools instead of the full 1,200-token prompt with 32 tools.
|
|
841
|
+
- **Token budget caps**: Hard limits prevent any single heal from exceeding its allocated budget (simple=20K, moderate=50K, complex=100K).
|
|
842
|
+
|
|
843
|
+
#### 9.4 Limitations
|
|
844
|
+
|
|
845
|
+
Honest accounting of what the system does not handle well:
|
|
846
|
+
|
|
847
|
+
- **Logic errors with no runtime symptoms**: If the code runs without errors but produces wrong results, Wolverine has no signal to trigger a heal. It responds to crashes and 500 errors, not incorrect business logic.
|
|
848
|
+
- **External service outages**: When the root cause is an external API being down, no code change will fix the problem. The system detects these (ECONNREFUSED, ETIMEDOUT, 503 responses) and routes to human notification rather than attempting futile code repairs.
|
|
849
|
+
- **Performance regressions**: The system does not currently detect or repair performance degradation that does not result in errors or crashes.
|
|
850
|
+
- **Complex architectural problems**: Multi-service coordination issues, race conditions, and distributed system failures typically exceed what a single-project agent can diagnose and repair.
|
|
851
|
+
- **Novel attack vectors**: While the injection detector covers approximately 50 known patterns plus AI-powered deep scanning, sufficiently sophisticated prompt injection attempts in error messages could theoretically bypass both layers. The sandbox and protected paths provide defense-in-depth, but no injection detection system is provably complete.
|
|
852
|
+
|
|
853
|
+
---
|
|
854
|
+
|
|
855
|
+
### 10. Conclusion
|
|
856
|
+
|
|
857
|
+
Wolverine demonstrates that the combination of structured error-to-prompt conversion with tiered AI escalation creates a practical autonomous recovery system for Node.js servers. The key technical contributions are:
|
|
858
|
+
|
|
859
|
+
1. **Error classification as a routing mechanism**: By parsing and classifying errors before invoking AI, the system routes the majority of crashes to zero-cost operational fixes. This transforms AI-driven error recovery from an expensive novelty into a cost-effective production tool.
|
|
860
|
+
|
|
861
|
+
2. **Tiered escalation with verification gates**: Each repair tier is more capable and more expensive than the last. Verification between tiers prevents unnecessary escalation and ensures fixes are validated before deployment. Failed fixes are automatically rolled back.
|
|
862
|
+
|
|
863
|
+
3. **Token efficiency through structured prompts**: Dynamic prompt sizing, context compaction, namespace isolation, and prompt caching reduce per-heal costs by 10-15x compared to naive approaches. The median heal costs under $0.01.
|
|
864
|
+
|
|
865
|
+
4. **Defense-in-depth security**: Six overlapping security layers (secret redaction, injection detection, file system sandbox, command blocking, adaptive rate limiting, and encrypted vault) address the unique risks of autonomous code modification. No single layer is sufficient on its own; the combination provides practical security against the most likely attack vectors.
|
|
866
|
+
|
|
867
|
+
5. **Persistent learning**: The vector-indexed brain enables the system to improve over time. Successful fixes accelerate future repairs of similar errors. Failed fixes are remembered and avoided. The function map provides structural awareness without requiring the AI to read every file on every heal.
|
|
868
|
+
|
|
869
|
+
The practical result is a reduction in mean time to recovery from minutes or hours (human response) to seconds (autonomous repair) for the class of errors the system can handle -- which, based on the error classification distribution, covers the majority of routine production crashes in Node.js applications. For errors outside this class, the system degrades gracefully: detecting that no code fix is viable and routing to human notification rather than consuming tokens on futile repair attempts.
|
|
870
|
+
|
|
871
|
+
The framework is open-source, published as `wolverine-ai` on npm, and designed to wrap any existing Node.js server with zero code changes required in the target application.
|
|
872
|
+
|
|
873
|
+
---
|
|
874
|
+
|
|
104
875
|
## Architecture
|
|
105
876
|
|
|
106
877
|
```
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "wolverine-ai",
|
|
3
|
-
"version": "6.0.
|
|
3
|
+
"version": "6.0.1",
|
|
4
4
|
"description": "Self-healing Node.js server framework powered by AI. Catches crashes, diagnoses errors, generates fixes, verifies, and restarts — automatically.",
|
|
5
5
|
"main": "src/index.js",
|
|
6
6
|
"bin": {
|