@ruvector/ruvllm 2.4.0 → 2.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +262 -298
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,6 +1,42 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
# @ruvector/ruvllm
|
|
4
|
+
|
|
5
|
+
### The First Purpose-Built LLM Runtime for Claude Code Agent Orchestration
|
|
6
|
+
|
|
7
|
+
**100% Routing Accuracy | Sub-Millisecond Inference | Self-Learning**
|
|
8
|
+
|
|
9
|
+
[](https://www.npmjs.com/package/@ruvector/ruvllm)
|
|
10
|
+
[](https://www.npmjs.com/package/@ruvector/ruvllm)
|
|
11
|
+
[](LICENSE)
|
|
12
|
+
[](./test)
|
|
13
|
+
|
|
14
|
+
[Quick Start](#quick-start) | [RLM](#rlm-recursive-language-model) | [Training](#training) | [Models](#models) | [API](#api-reference)
|
|
15
|
+
|
|
16
|
+
</div>
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## What is @ruvector/ruvllm?
|
|
21
|
+
|
|
22
|
+
**@ruvector/ruvllm** is a TypeScript/JavaScript SDK for intelligent LLM orchestration, specifically designed for **Claude Code** and multi-agent systems. It provides:
|
|
23
|
+
|
|
24
|
+
- **RLM (Recursive Language Model)** - Break complex queries into sub-queries, synthesize coherent answers
|
|
25
|
+
- **100% Routing Accuracy** - Hybrid keyword + embedding strategy for perfect agent selection
|
|
26
|
+
- **SONA Self-Learning** - Model improves with every successful interaction
|
|
27
|
+
- **SIMD Acceleration** - AVX2/NEON optimized inference
|
|
28
|
+
|
|
29
|
+
### Why @ruvector/ruvllm?
|
|
30
|
+
|
|
31
|
+
| Challenge | Traditional Approach | @ruvector/ruvllm Solution |
|
|
32
|
+
|-----------|---------------------|---------------------------|
|
|
33
|
+
| Agent selection | Manual or keyword-based | Semantic + keyword hybrid = **100%** |
|
|
34
|
+
| Complex queries | Single-shot RAG | Recursive decomposition + synthesis |
|
|
35
|
+
| Response latency | 2-5 seconds | **<1ms** cache, 50-200ms full |
|
|
36
|
+
| Learning | Static models | **Self-improving** (SONA) |
|
|
37
|
+
| Cost per route | $0.01+ (API call) | **$0** (local inference) |
|
|
38
|
+
|
|
39
|
+
---
|
|
4
40
|
|
|
5
41
|
## Installation
|
|
6
42
|
|
|
@@ -11,43 +47,99 @@ npm install @ruvector/ruvllm
|
|
|
11
47
|
## Quick Start
|
|
12
48
|
|
|
13
49
|
```typescript
|
|
14
|
-
import { RuvLLM,
|
|
15
|
-
|
|
16
|
-
// Initialize with default configuration
|
|
17
|
-
const llm = new RuvLLM();
|
|
50
|
+
import { RuvLLM, RlmController } from '@ruvector/ruvllm';
|
|
18
51
|
|
|
19
|
-
//
|
|
52
|
+
// Simple LLM inference
|
|
20
53
|
const llm = new RuvLLM({
|
|
21
|
-
modelPath: '
|
|
54
|
+
modelPath: '~/.ruvllm/models/ruvltra-claude-code-0.5b-q4_k_m.gguf',
|
|
22
55
|
sonaEnabled: true,
|
|
23
|
-
flashAttention: true,
|
|
24
|
-
maxTokens: 256,
|
|
25
56
|
});
|
|
26
57
|
|
|
27
|
-
// Generate text
|
|
28
58
|
const response = await llm.query('Explain quantum computing');
|
|
29
59
|
console.log(response.text);
|
|
30
60
|
|
|
31
|
-
//
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
61
|
+
// Recursive Language Model for complex queries
|
|
62
|
+
const rlm = new RlmController({ maxDepth: 5 });
|
|
63
|
+
const answer = await rlm.query('What are the causes AND solutions for slow API responses?');
|
|
64
|
+
// Automatically decomposes into sub-queries, retrieves context, synthesizes answer
|
|
35
65
|
```
|
|
36
66
|
|
|
37
|
-
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## Core Features
|
|
70
|
+
|
|
71
|
+
### 1. Claude Code Native Routing
|
|
72
|
+
|
|
73
|
+
Built **by** Claude Code, **for** Claude Code. Routes tasks to 60+ agent types:
|
|
74
|
+
|
|
75
|
+
```typescript
|
|
76
|
+
import { RuvLLM } from '@ruvector/ruvllm';
|
|
38
77
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
78
|
+
const llm = new RuvLLM({ model: 'ruv/ruvltra' });
|
|
79
|
+
|
|
80
|
+
// Intelligent routing
|
|
81
|
+
const route = await llm.route('implement OAuth2 authentication');
|
|
82
|
+
console.log(route.agent); // 'security-architect'
|
|
83
|
+
console.log(route.confidence); // 0.98
|
|
84
|
+
console.log(route.tier); // 2 (Haiku-level complexity)
|
|
85
|
+
|
|
86
|
+
// Multi-agent teams for complex tasks
|
|
87
|
+
const team = await llm.routeComplex('build full-stack app with auth');
|
|
88
|
+
// Returns: [system-architect, backend-dev, coder, security-architect, tester]
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### 2. 3-Tier Intelligent Routing
|
|
92
|
+
|
|
93
|
+
```
|
|
94
|
+
┌─────────────────────────────────────────────────────────┐
|
|
95
|
+
│ User Request │
|
|
96
|
+
└─────────────────────┬───────────────────────────────────┘
|
|
97
|
+
↓
|
|
98
|
+
[RuvLTRA Routing]
|
|
99
|
+
↓
|
|
100
|
+
┌─────────────┼─────────────┐
|
|
101
|
+
↓ ↓ ↓
|
|
102
|
+
┌───────────┐ ┌───────────┐ ┌───────────┐
|
|
103
|
+
│ Tier 1 │ │ Tier 2 │ │ Tier 3 │
|
|
104
|
+
│ Booster │ │ Haiku │ │ Opus │
|
|
105
|
+
│ <1ms │ │ ~500ms │ │ 2-5s │
|
|
106
|
+
│ $0 │ │ $0.0002 │ │ $0.015 │
|
|
107
|
+
└───────────┘ └───────────┘ └───────────┘
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
### 3. Self-Learning (SONA)
|
|
111
|
+
|
|
112
|
+
Every successful interaction improves the model:
|
|
113
|
+
|
|
114
|
+
```typescript
|
|
115
|
+
// First routing: Full inference
|
|
116
|
+
llm.route('implement OAuth2') → security-architect (97%)
|
|
117
|
+
|
|
118
|
+
// Later: Pattern hit in <25μs (learned from success)
|
|
119
|
+
llm.route('add OAuth2 flow') → security-architect (99%, cached pattern)
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
---
|
|
47
123
|
|
|
48
124
|
## RLM (Recursive Language Model)
|
|
49
125
|
|
|
50
|
-
RLM provides recursive
|
|
126
|
+
RLM provides **recursive query decomposition** - unlike traditional RAG that retrieves once, RLM breaks complex questions into sub-queries and synthesizes coherent answers.
|
|
127
|
+
|
|
128
|
+
### How It Works
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
Query: "What are the causes AND solutions for slow API responses?"
|
|
132
|
+
↓
|
|
133
|
+
[Decomposition]
|
|
134
|
+
/ \
|
|
135
|
+
"Causes of slow API?" "Solutions for slow API?"
|
|
136
|
+
↓ ↓
|
|
137
|
+
[Sub-answers] [Sub-answers]
|
|
138
|
+
\ /
|
|
139
|
+
[Synthesis]
|
|
140
|
+
↓
|
|
141
|
+
Coherent combined answer with sources
|
|
142
|
+
```
|
|
51
143
|
|
|
52
144
|
### Basic Usage
|
|
53
145
|
|
|
@@ -66,19 +158,15 @@ await rlm.addMemory('React is a library for building user interfaces.');
|
|
|
66
158
|
|
|
67
159
|
// Query with recursive retrieval
|
|
68
160
|
const answer = await rlm.query('What are causes and solutions for type errors in React?');
|
|
69
|
-
console.log(answer.text);
|
|
70
|
-
console.log(
|
|
71
|
-
console.log(
|
|
72
|
-
console.log(
|
|
161
|
+
console.log(answer.text); // Comprehensive synthesized answer
|
|
162
|
+
console.log(answer.sources); // Source attributions
|
|
163
|
+
console.log(answer.qualityScore); // 0.0-1.0
|
|
164
|
+
console.log(answer.confidence); // Routing confidence
|
|
73
165
|
```
|
|
74
166
|
|
|
75
167
|
### Streaming
|
|
76
168
|
|
|
77
169
|
```typescript
|
|
78
|
-
import { RlmController } from '@ruvector/ruvllm';
|
|
79
|
-
|
|
80
|
-
const rlm = new RlmController();
|
|
81
|
-
|
|
82
170
|
for await (const event of rlm.queryStream('Explain machine learning')) {
|
|
83
171
|
if (event.type === 'token') {
|
|
84
172
|
process.stdout.write(event.text);
|
|
@@ -88,18 +176,16 @@ for await (const event of rlm.queryStream('Explain machine learning')) {
|
|
|
88
176
|
}
|
|
89
177
|
```
|
|
90
178
|
|
|
91
|
-
### With Reflection
|
|
179
|
+
### With Self-Reflection
|
|
92
180
|
|
|
93
181
|
```typescript
|
|
94
|
-
import { RlmController } from '@ruvector/ruvllm';
|
|
95
|
-
|
|
96
182
|
const rlm = new RlmController({
|
|
97
183
|
enableReflection: true,
|
|
98
184
|
maxReflectionIterations: 2,
|
|
99
185
|
minQualityScore: 0.8,
|
|
100
186
|
});
|
|
101
187
|
|
|
102
|
-
// Answers
|
|
188
|
+
// Answers are iteratively refined until quality >= 0.8
|
|
103
189
|
const answer = await rlm.query('Complex multi-part technical question...');
|
|
104
190
|
```
|
|
105
191
|
|
|
@@ -119,58 +205,68 @@ interface RlmConfig {
|
|
|
119
205
|
}
|
|
120
206
|
```
|
|
121
207
|
|
|
122
|
-
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
## Unique Capabilities
|
|
211
|
+
|
|
212
|
+
### 1. Memory-Augmented Routing
|
|
213
|
+
|
|
214
|
+
Every successful routing is stored in HNSW-indexed memory for instant recall:
|
|
123
215
|
|
|
124
216
|
```typescript
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
RuvLLM,
|
|
128
|
-
RuvLLMConfig,
|
|
129
|
-
|
|
130
|
-
// RLM - Recursive Language Model
|
|
131
|
-
RlmController,
|
|
132
|
-
RlmConfig,
|
|
133
|
-
RlmAnswer,
|
|
134
|
-
MemorySpan,
|
|
135
|
-
StreamToken,
|
|
136
|
-
|
|
137
|
-
// RLM Training
|
|
138
|
-
RlmTrainer,
|
|
139
|
-
RlmTrainingConfig,
|
|
140
|
-
RlmTrainingExample,
|
|
141
|
-
createRlmTrainer,
|
|
142
|
-
DEFAULT_RLM_CONFIG,
|
|
143
|
-
FAST_RLM_CONFIG,
|
|
144
|
-
THOROUGH_RLM_CONFIG,
|
|
145
|
-
ROUTING_FOCUSED_CONFIG,
|
|
217
|
+
// First time: Full inference (~50ms)
|
|
218
|
+
route("implement OAuth2") → security-architect (97% confidence)
|
|
146
219
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
220
|
+
// Later: Memory hit (<25μs)
|
|
221
|
+
route("add OAuth2 flow") → security-architect (99% confidence, cached)
|
|
222
|
+
```
|
|
150
223
|
|
|
151
|
-
|
|
152
|
-
EphemeralAgent,
|
|
153
|
-
FederatedCoordinator,
|
|
224
|
+
### 2. Confidence-Aware Escalation
|
|
154
225
|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
226
|
+
```typescript
|
|
227
|
+
// Low confidence automatically escalates
|
|
228
|
+
Confidence > 0.9 → Use recommended agent
|
|
229
|
+
Confidence 0.7-0.9 → Use with human confirmation
|
|
230
|
+
Confidence < 0.7 → Escalate to higher tier
|
|
231
|
+
```
|
|
158
232
|
|
|
159
|
-
|
|
160
|
-
SessionManager,
|
|
233
|
+
### 3. Batch SIMD Operations
|
|
161
234
|
|
|
162
|
-
|
|
163
|
-
|
|
235
|
+
```typescript
|
|
236
|
+
import { simd } from '@ruvector/ruvllm/simd';
|
|
164
237
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
EmbeddingBenchmark,
|
|
169
|
-
} from '@ruvector/ruvllm';
|
|
238
|
+
// 4x faster vector operations with AVX2/NEON
|
|
239
|
+
const similarity = simd.batchCosineSimilarity(query, targets);
|
|
240
|
+
const attended = simd.flashAttention(q, k, v, scale);
|
|
170
241
|
```
|
|
171
242
|
|
|
243
|
+
### 4. Zero-Copy Caching
|
|
244
|
+
|
|
245
|
+
Arc-based string interning for 100-1000x faster cache hits on large responses.
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
172
249
|
## Performance
|
|
173
250
|
|
|
251
|
+
### Benchmarks (M4 Pro)
|
|
252
|
+
|
|
253
|
+
| Operation | Latency | Throughput |
|
|
254
|
+
|-----------|---------|------------|
|
|
255
|
+
| Query decomposition | 340 ns | 2.9M/s |
|
|
256
|
+
| Cache lookup | 23.5 ns | 42.5M/s |
|
|
257
|
+
| Embedding (384d) | 293 ns | 3.4M/s |
|
|
258
|
+
| Memory search (10k) | 0.4 ms | 2.5K/s |
|
|
259
|
+
| End-to-end routing | <1 ms | 1K+/s |
|
|
260
|
+
| Full RLM query | 50-200 ms | 5-20/s |
|
|
261
|
+
|
|
262
|
+
### Routing Accuracy
|
|
263
|
+
|
|
264
|
+
| Strategy | RuvLTRA | Qwen Base | OpenAI |
|
|
265
|
+
|----------|---------|-----------|--------|
|
|
266
|
+
| Embedding Only | 45% | 40% | 52% |
|
|
267
|
+
| Keyword Only | 78% | 78% | N/A |
|
|
268
|
+
| **Hybrid** | **100%** | 95% | N/A |
|
|
269
|
+
|
|
174
270
|
### Test Results
|
|
175
271
|
|
|
176
272
|
```
|
|
@@ -184,67 +280,57 @@ import {
|
|
|
184
280
|
- Benchmarks: 26 tests
|
|
185
281
|
```
|
|
186
282
|
|
|
187
|
-
|
|
283
|
+
---
|
|
188
284
|
|
|
189
|
-
|
|
190
|
-
|----------|---------|-----------|
|
|
191
|
-
| Embedding Only | 45% | 40% |
|
|
192
|
-
| Keyword-First (Hybrid) | **100%** | 95% |
|
|
285
|
+
## Models
|
|
193
286
|
|
|
194
|
-
###
|
|
287
|
+
### HuggingFace Repository
|
|
195
288
|
|
|
196
|
-
|
|
197
|
-
|-----------|-------------|
|
|
198
|
-
| Inference | 88-135 tok/s |
|
|
199
|
-
| Flash Attention | 320us (seq=2048) |
|
|
200
|
-
| HNSW Search | 17-62us |
|
|
201
|
-
| SONA Adapt | <1ms |
|
|
202
|
-
| RLM Query | 50-200ms |
|
|
289
|
+
**URL**: [https://huggingface.co/ruv/ruvltra](https://huggingface.co/ruv/ruvltra)
|
|
203
290
|
|
|
204
|
-
###
|
|
291
|
+
### Available Models
|
|
205
292
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
293
|
+
| Model | Size | Purpose | Accuracy |
|
|
294
|
+
|-------|------|---------|----------|
|
|
295
|
+
| **ruvltra-claude-code-0.5b-q4_k_m** | 398 MB | Agent routing | **100%** (hybrid) |
|
|
296
|
+
| ruvltra-small-0.5b-q4_k_m | ~400 MB | Embeddings | - |
|
|
297
|
+
| ruvltra-medium-1.1b-q4_k_m | ~1 GB | Full inference | - |
|
|
209
298
|
|
|
210
|
-
|
|
299
|
+
### Download Models
|
|
211
300
|
|
|
212
|
-
```
|
|
213
|
-
|
|
214
|
-
|
|
301
|
+
```typescript
|
|
302
|
+
// Programmatic
|
|
303
|
+
import { downloadModel } from '@ruvector/ruvllm';
|
|
304
|
+
await downloadModel('ruv/ruvltra', { quantization: 'q4_k_m' });
|
|
215
305
|
|
|
216
|
-
|
|
217
|
-
ruvllm
|
|
306
|
+
// CLI
|
|
307
|
+
ruvllm download ruv/ruvltra
|
|
308
|
+
```
|
|
218
309
|
|
|
219
|
-
|
|
220
|
-
ruvllm download ruvector/ruvltra-small-q4km
|
|
310
|
+
### Auto-Download
|
|
221
311
|
|
|
222
|
-
|
|
223
|
-
ruvllm bench ./models/model.gguf
|
|
312
|
+
Models are automatically downloaded on first use:
|
|
224
313
|
|
|
225
|
-
|
|
226
|
-
|
|
314
|
+
```typescript
|
|
315
|
+
const llm = new RuvLLM({ model: 'ruv/ruvltra' });
|
|
316
|
+
// Downloads to ~/.ruvllm/models/ if not present
|
|
227
317
|
```
|
|
228
318
|
|
|
229
|
-
|
|
319
|
+
---
|
|
230
320
|
|
|
231
|
-
|
|
321
|
+
## Training
|
|
232
322
|
|
|
233
|
-
Generate
|
|
323
|
+
### Generate Routing Dataset
|
|
234
324
|
|
|
235
325
|
```bash
|
|
236
|
-
# Generate routing dataset
|
|
237
326
|
node scripts/training/routing-dataset.js
|
|
238
|
-
|
|
239
|
-
# Output: 381 examples, 793 contrastive pairs
|
|
327
|
+
# Output: 381 examples, 793 contrastive pairs, 156 hard negatives
|
|
240
328
|
```
|
|
241
329
|
|
|
242
330
|
### Contrastive Fine-tuning
|
|
243
331
|
|
|
244
|
-
Fine-tune models with LoRA adapters:
|
|
245
|
-
|
|
246
332
|
```typescript
|
|
247
|
-
import { ContrastiveTrainer
|
|
333
|
+
import { ContrastiveTrainer } from '@ruvector/ruvllm';
|
|
248
334
|
|
|
249
335
|
const trainer = new ContrastiveTrainer({
|
|
250
336
|
modelPath: './models/base.gguf',
|
|
@@ -253,73 +339,25 @@ const trainer = new ContrastiveTrainer({
|
|
|
253
339
|
learningRate: 1e-4,
|
|
254
340
|
});
|
|
255
341
|
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
{
|
|
259
|
-
anchor: 'Fix the authentication bug in login.ts',
|
|
260
|
-
positive: 'coder',
|
|
261
|
-
negative: 'researcher',
|
|
262
|
-
},
|
|
342
|
+
const pairs = [
|
|
343
|
+
{ anchor: 'Fix auth bug', positive: 'coder', negative: 'researcher' },
|
|
263
344
|
// ... more pairs
|
|
264
345
|
];
|
|
265
346
|
|
|
266
|
-
await trainer.train(pairs, { epochs: 10
|
|
347
|
+
await trainer.train(pairs, { epochs: 10 });
|
|
267
348
|
await trainer.save('./adapters/routing-lora');
|
|
268
349
|
```
|
|
269
350
|
|
|
270
351
|
### Training Scripts
|
|
271
352
|
|
|
272
|
-
Located in `scripts/training/`:
|
|
273
|
-
|
|
274
353
|
| Script | Description |
|
|
275
354
|
|--------|-------------|
|
|
276
355
|
| `routing-dataset.js` | Generate 381 routing examples |
|
|
277
356
|
| `claude-code-synth.js` | Synthetic data generation |
|
|
278
357
|
| `contrastive-finetune.js` | LoRA fine-tuning pipeline |
|
|
358
|
+
| `rlm-dataset.js` | RLM training data (500 examples) |
|
|
279
359
|
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
### HuggingFace Repository
|
|
283
|
-
|
|
284
|
-
**URL**: [https://huggingface.co/ruv/ruvltra](https://huggingface.co/ruv/ruvltra)
|
|
285
|
-
|
|
286
|
-
### Available Models
|
|
287
|
-
|
|
288
|
-
| Model | File | Size | Purpose |
|
|
289
|
-
|-------|------|------|---------|
|
|
290
|
-
| RuvLTRA Claude Code 0.5B | `ruvltra-claude-code-0.5b-q4_k_m.gguf` | ~400MB | Agent routing (100% with hybrid) |
|
|
291
|
-
| RuvLTRA Small 0.5B | `ruvltra-0.5b-q4_k_m.gguf` | ~400MB | General embeddings |
|
|
292
|
-
| RuvLTRA Medium 3B | `ruvltra-3b-q4_k_m.gguf` | ~2GB | Full LLM inference |
|
|
293
|
-
|
|
294
|
-
### Download Models
|
|
295
|
-
|
|
296
|
-
```bash
|
|
297
|
-
# Using CLI
|
|
298
|
-
ruvllm download ruv/ruvltra
|
|
299
|
-
|
|
300
|
-
# Using HuggingFace CLI
|
|
301
|
-
huggingface-cli download ruv/ruvltra ruvltra-claude-code-0.5b-q4_k_m.gguf
|
|
302
|
-
|
|
303
|
-
# Programmatic download
|
|
304
|
-
import { downloadModel } from '@ruvector/ruvllm';
|
|
305
|
-
await downloadModel('ruv/ruvltra', { quantization: 'q4_k_m' });
|
|
306
|
-
```
|
|
307
|
-
|
|
308
|
-
### Environment Variables
|
|
309
|
-
|
|
310
|
-
```bash
|
|
311
|
-
# HuggingFace authentication (any of these)
|
|
312
|
-
HF_TOKEN=hf_xxx
|
|
313
|
-
HUGGING_FACE_HUB_TOKEN=hf_xxx
|
|
314
|
-
HUGGINGFACE_API_KEY=hf_xxx
|
|
315
|
-
```
|
|
316
|
-
|
|
317
|
-
### Local Model Storage
|
|
318
|
-
|
|
319
|
-
```bash
|
|
320
|
-
~/.ruvllm/models/ # Downloaded GGUF models
|
|
321
|
-
~/.ruvllm/training/ # Training data and configs
|
|
322
|
-
```
|
|
360
|
+
---
|
|
323
361
|
|
|
324
362
|
## API Reference
|
|
325
363
|
|
|
@@ -329,23 +367,16 @@ HUGGINGFACE_API_KEY=hf_xxx
|
|
|
329
367
|
class RuvLLM {
|
|
330
368
|
constructor(config?: RuvLLMConfig);
|
|
331
369
|
|
|
332
|
-
// Generate text
|
|
333
370
|
query(prompt: string, params?: GenerateParams): Promise<Response>;
|
|
334
|
-
|
|
335
|
-
// Stream generation
|
|
336
371
|
stream(prompt: string, params?: GenerateParams): AsyncIterable<string>;
|
|
372
|
+
route(task: string): Promise<RoutingResult>;
|
|
373
|
+
routeComplex(task: string): Promise<AgentTeam[]>;
|
|
337
374
|
|
|
338
|
-
// Load a model
|
|
339
375
|
loadModel(path: string): Promise<void>;
|
|
340
|
-
|
|
341
|
-
// Memory operations
|
|
342
|
-
addMemory(text: string, metadata?: Record<string, unknown>): number;
|
|
376
|
+
addMemory(text: string, metadata?: object): number;
|
|
343
377
|
searchMemory(query: string, topK?: number): MemoryResult[];
|
|
344
378
|
|
|
345
|
-
// Get SONA learning stats
|
|
346
379
|
sonaStats(): SonaStats | null;
|
|
347
|
-
|
|
348
|
-
// Adapt on feedback
|
|
349
380
|
adapt(input: Float32Array, quality: number): void;
|
|
350
381
|
}
|
|
351
382
|
```
|
|
@@ -356,147 +387,68 @@ class RuvLLM {
|
|
|
356
387
|
class RlmController {
|
|
357
388
|
constructor(config?: RlmConfig, engine?: RuvLLM);
|
|
358
389
|
|
|
359
|
-
// Query with recursive retrieval
|
|
360
390
|
query(input: string): Promise<RlmAnswer>;
|
|
361
|
-
|
|
362
|
-
// Stream query
|
|
363
391
|
queryStream(input: string): AsyncGenerator<StreamToken>;
|
|
364
392
|
|
|
365
|
-
|
|
366
|
-
addMemory(text: string, metadata?: Record<string, unknown>): Promise<string>;
|
|
393
|
+
addMemory(text: string, metadata?: object): Promise<string>;
|
|
367
394
|
searchMemory(query: string, topK?: number): Promise<MemorySpan[]>;
|
|
368
395
|
|
|
369
|
-
// Cache management
|
|
370
396
|
clearCache(): void;
|
|
371
397
|
getCacheStats(): { size: number; entries: number };
|
|
372
398
|
|
|
373
|
-
// Configuration
|
|
374
399
|
updateConfig(config: Partial<RlmConfig>): void;
|
|
375
400
|
getConfig(): Required<RlmConfig>;
|
|
376
401
|
}
|
|
377
402
|
```
|
|
378
403
|
|
|
379
|
-
###
|
|
404
|
+
### All Exports
|
|
380
405
|
|
|
381
406
|
```typescript
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
flashAttention?: boolean; // Use Flash Attention 2 (default: true)
|
|
386
|
-
maxTokens?: number; // Max generation tokens (default: 256)
|
|
387
|
-
temperature?: number; // Sampling temperature (default: 0.7)
|
|
388
|
-
topP?: number; // Top-p sampling (default: 0.9)
|
|
389
|
-
}
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
### Generate Parameters
|
|
393
|
-
|
|
394
|
-
```typescript
|
|
395
|
-
interface GenerateParams {
|
|
396
|
-
maxTokens?: number;
|
|
397
|
-
temperature?: number;
|
|
398
|
-
topP?: number;
|
|
399
|
-
topK?: number;
|
|
400
|
-
repetitionPenalty?: number;
|
|
401
|
-
stopSequences?: string[];
|
|
402
|
-
}
|
|
403
|
-
```
|
|
404
|
-
|
|
405
|
-
## SIMD Module
|
|
406
|
-
|
|
407
|
-
For direct access to optimized SIMD kernels:
|
|
407
|
+
import {
|
|
408
|
+
// Core
|
|
409
|
+
RuvLLM, RuvLLMConfig,
|
|
408
410
|
|
|
409
|
-
|
|
410
|
-
|
|
411
|
+
// RLM
|
|
412
|
+
RlmController, RlmConfig, RlmAnswer, MemorySpan, StreamToken,
|
|
411
413
|
|
|
412
|
-
//
|
|
413
|
-
|
|
414
|
+
// Training
|
|
415
|
+
RlmTrainer, ContrastiveTrainer, createRlmTrainer,
|
|
416
|
+
DEFAULT_RLM_CONFIG, FAST_RLM_CONFIG, THOROUGH_RLM_CONFIG,
|
|
414
417
|
|
|
415
|
-
//
|
|
416
|
-
|
|
418
|
+
// SONA Learning
|
|
419
|
+
SonaCoordinator, TrajectoryBuilder,
|
|
417
420
|
|
|
418
|
-
//
|
|
419
|
-
|
|
421
|
+
// LoRA
|
|
422
|
+
LoraAdapter, LoraManager,
|
|
420
423
|
|
|
421
|
-
//
|
|
422
|
-
|
|
424
|
+
// Benchmarks
|
|
425
|
+
ModelComparisonBenchmark, RoutingBenchmark, EmbeddingBenchmark,
|
|
426
|
+
} from '@ruvector/ruvllm';
|
|
423
427
|
```
|
|
424
428
|
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
Run model evaluations with SWE-Bench integration:
|
|
428
|
-
|
|
429
|
-
```typescript
|
|
430
|
-
import { RuvLLM, EvaluationHarness, AblationMode } from '@ruvector/ruvllm';
|
|
431
|
-
|
|
432
|
-
const harness = new EvaluationHarness({
|
|
433
|
-
modelPath: './models/model.gguf',
|
|
434
|
-
enableHnsw: true,
|
|
435
|
-
enableSona: true,
|
|
436
|
-
});
|
|
437
|
-
|
|
438
|
-
// Run single evaluation
|
|
439
|
-
const result = await harness.evaluate(
|
|
440
|
-
'Fix the null pointer exception',
|
|
441
|
-
'def process(data): return data.split()',
|
|
442
|
-
AblationMode.Full
|
|
443
|
-
);
|
|
429
|
+
---
|
|
444
430
|
|
|
445
|
-
|
|
431
|
+
## CLI
|
|
446
432
|
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
}
|
|
452
|
-
```
|
|
453
|
-
|
|
454
|
-
## mistral-rs Backend (Production Serving)
|
|
433
|
+
```bash
|
|
434
|
+
# Route a task
|
|
435
|
+
ruvllm route "add unit tests for auth module"
|
|
436
|
+
# → Agent: tester | Confidence: 0.96 | Tier: 2
|
|
455
437
|
|
|
456
|
-
|
|
438
|
+
# Query with streaming
|
|
439
|
+
ruvllm query --stream "Explain machine learning"
|
|
457
440
|
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
// Configure for production serving
|
|
462
|
-
const backend = new MistralBackend({
|
|
463
|
-
// PagedAttention: 5-10x more concurrent users
|
|
464
|
-
pagedAttention: {
|
|
465
|
-
blockSize: 16,
|
|
466
|
-
maxBlocks: 4096,
|
|
467
|
-
gpuMemoryFraction: 0.9,
|
|
468
|
-
prefixCaching: true,
|
|
469
|
-
},
|
|
470
|
-
// X-LoRA: Per-token adapter routing
|
|
471
|
-
xlora: {
|
|
472
|
-
adapters: ['./adapters/coder', './adapters/researcher'],
|
|
473
|
-
topK: 2,
|
|
474
|
-
},
|
|
475
|
-
// ISQ: Runtime quantization
|
|
476
|
-
isq: {
|
|
477
|
-
bits: 4,
|
|
478
|
-
method: 'awq',
|
|
479
|
-
},
|
|
480
|
-
});
|
|
441
|
+
# Download models
|
|
442
|
+
ruvllm download ruv/ruvltra
|
|
481
443
|
|
|
482
|
-
|
|
483
|
-
|
|
444
|
+
# Run benchmarks
|
|
445
|
+
ruvllm bench ./models/model.gguf
|
|
484
446
|
|
|
485
|
-
|
|
486
|
-
|
|
447
|
+
# Evaluate (SWE-Bench)
|
|
448
|
+
ruvllm eval --model ./models/model.gguf --subset lite
|
|
487
449
|
```
|
|
488
450
|
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
## Supported Models
|
|
492
|
-
|
|
493
|
-
- **RuvLTRA-Small** (494M) - Q4K, Q5K, Q8
|
|
494
|
-
- **RuvLTRA-Medium** (3B) - Q4K, Q5K, Q8
|
|
495
|
-
- **Qwen 2.5** (0.5B-72B)
|
|
496
|
-
- **Llama 3.x** (8B-70B)
|
|
497
|
-
- **Mistral** (7B-22B)
|
|
498
|
-
- **Phi-3** (3.8B-14B)
|
|
499
|
-
- **Gemma-2** (2B-27B)
|
|
451
|
+
---
|
|
500
452
|
|
|
501
453
|
## Platform Support
|
|
502
454
|
|
|
@@ -508,19 +460,31 @@ const response = await llm.query('Write production code');
|
|
|
508
460
|
| Linux | arm64 | Supported |
|
|
509
461
|
| Windows | x64 | Supported |
|
|
510
462
|
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
- [@ruvector/core](https://www.npmjs.com/package/@ruvector/core) - Vector operations
|
|
514
|
-
- [@ruvector/sona](https://www.npmjs.com/package/@ruvector/sona) - SONA learning engine
|
|
515
|
-
- [@ruvector/ruvector](https://www.npmjs.com/package/@ruvector/ruvector) - Full Ruvector SDK
|
|
463
|
+
---
|
|
516
464
|
|
|
517
465
|
## Links
|
|
518
466
|
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
467
|
+
| Resource | URL |
|
|
468
|
+
|----------|-----|
|
|
469
|
+
| **npm** | [npmjs.com/package/@ruvector/ruvllm](https://www.npmjs.com/package/@ruvector/ruvllm) |
|
|
470
|
+
| **HuggingFace** | [huggingface.co/ruv/ruvltra](https://huggingface.co/ruv/ruvltra) |
|
|
471
|
+
| **Crate (Rust)** | [crates.io/crates/ruvllm](https://crates.io/crates/ruvllm) |
|
|
472
|
+
| **Documentation** | [docs.rs/ruvllm](https://docs.rs/ruvllm) |
|
|
473
|
+
| **GitHub** | [github.com/ruvnet/ruvector](https://github.com/ruvnet/ruvector) |
|
|
474
|
+
| **Claude Flow** | [github.com/ruvnet/claude-flow](https://github.com/ruvnet/claude-flow) |
|
|
475
|
+
|
|
476
|
+
---
|
|
523
477
|
|
|
524
478
|
## License
|
|
525
479
|
|
|
526
480
|
MIT OR Apache-2.0
|
|
481
|
+
|
|
482
|
+
---
|
|
483
|
+
|
|
484
|
+
<div align="center">
|
|
485
|
+
|
|
486
|
+
**Built for Claude Code. Optimized for agents. Designed for speed.**
|
|
487
|
+
|
|
488
|
+
[Get Started](#quick-start) | [View on GitHub](https://github.com/ruvnet/ruvector)
|
|
489
|
+
|
|
490
|
+
</div>
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ruvector/ruvllm",
|
|
3
|
-
"version": "2.4.
|
|
3
|
+
"version": "2.4.1",
|
|
4
4
|
"description": "Self-learning LLM orchestration with SONA adaptive learning, HNSW memory, RLM recursive retrieval, FastGRNN routing, and SIMD inference",
|
|
5
5
|
"main": "dist/cjs/index.js",
|
|
6
6
|
"module": "dist/esm/index.js",
|