@hicaru/pi-rlm 0.3.15 → 0.3.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +96 -70
- package/README.ru.md +86 -59
- package/README.zh-CN.md +95 -65
- package/package.json +5 -5
- package/src/bridge/add-context.ts +1 -1
- package/src/bridge/handlers/await.ts +13 -22
- package/src/bridge/handlers/completion.ts +27 -5
- package/src/bridge/handlers/emitting.ts +2 -2
- package/src/bridge/handlers/llm-query.ts +46 -68
- package/src/bridge/handlers/rlm-query.ts +14 -84
- package/src/bridge/handlers/task-registry.ts +22 -17
- package/src/bridge/handlers/types.ts +8 -6
- package/src/bridge/model.ts +21 -4
- package/src/commands/rlm-llm.ts +1 -10
- package/src/commands/rlm-rlm.ts +1 -8
- package/src/config/defaults.ts +31 -12
- package/src/config/settings.ts +47 -33
- package/src/config/skillstate.ts +465 -0
- package/src/context/md-cache.ts +1 -1
- package/src/context/merge.ts +1 -1
- package/src/context/namespace.ts +2 -2
- package/src/context/refresh.ts +1 -1
- package/src/context/source-dir.ts +21 -11
- package/src/context/source-doc.ts +1 -1
- package/src/context/source-git.ts +3 -15
- package/src/context/source-text.ts +1 -1
- package/src/context/walk.ts +6 -14
- package/src/core/budget.ts +107 -21
- package/src/core/compaction.ts +44 -1
- package/src/core/engine.ts +192 -94
- package/src/core/iteration.ts +1 -1
- package/src/core/ledger.ts +10 -13
- package/src/core/limits.ts +1 -1
- package/src/core/model-registry.ts +1 -1
- package/src/core/resource-limits.ts +1 -1
- package/src/core/root-context.ts +126 -0
- package/src/core/root-digest.ts +213 -0
- package/src/core/root-state.ts +240 -0
- package/src/core/run-state.ts +577 -0
- package/src/core/types.ts +56 -12
- package/src/index.ts +167 -36
- package/src/mode/llm-model.ts +13 -1
- package/src/mode/native-guards.ts +0 -6
- package/src/mode/rlm-mode.ts +34 -11
- package/src/mode/subagent.ts +5 -5
- package/src/prompts/glossary.ts +41 -25
- package/src/prompts/native.ts +1 -3
- package/src/prompts/system.ts +12 -4
- package/src/prompts/user.ts +17 -0
- package/src/sandbox/context-file.ts +1 -1
- package/src/sandbox/interrupts.ts +25 -31
- package/src/sandbox/protocol.ts +14 -20
- package/src/sandbox/py/__pycache__/guards.cpython-314.pyc +0 -0
- package/src/sandbox/py/__pycache__/retrieval.cpython-314.pyc +0 -0
- package/src/sandbox/py/__pycache__/scaffold.cpython-314.pyc +0 -0
- package/src/sandbox/py/__pycache__/worker.cpython-314.pyc +0 -0
- package/src/sandbox/py/guards.py +1 -1
- package/src/sandbox/py/retrieval.py +4 -1
- package/src/sandbox/py/scaffold.py +24 -31
- package/src/sandbox/py/worker.py +3 -1
- package/src/sandbox/sandbox-manager.ts +2 -2
- package/src/sandbox/sandbox.ts +35 -5
- package/src/text/agent-text.ts +58 -0
- package/src/text/parsing.ts +35 -3
- package/src/text/preview.ts +3 -0
- package/src/text/repl-output.ts +1 -1
- package/src/tool/background-tasks.ts +1 -1
- package/src/tool/repl-render.ts +1 -1
- package/src/tool/repl-result.ts +1 -1
- package/src/tool/repl-tool.ts +50 -26
- package/src/tool/rlm-tool.ts +4 -5
- package/src/tool/subcall-render.ts +1 -1
- package/src/tool/subcall-store.ts +2 -2
- package/src/tool/tool-utils.ts +5 -5
- package/src/ui/config-panel.ts +39 -0
- package/src/ui/intro.ts +1 -1
- package/src/ui/modal/timeline-store.ts +1 -1
- package/src/ui/model-picker/drilldown.ts +1 -1
- package/src/ui/model-picker/levels.ts +1 -1
- package/src/ui/panel/run-registry.ts +1 -1
- package/src/ui/tree/tree-rows.ts +1 -1
- package/src/ui/tree/tree-widget.ts +1 -1
- package/src/util/bm25.ts +97 -0
- package/src/util/concurrency.ts +1 -1
- package/src/util/errors.ts +1 -1
- package/src/util/retry.ts +22 -7
- package/src/util/state-merge.ts +34 -0
- package/src/util/throttle.ts +1 -1
- package/src/util/type-guards.ts +6 -0
- package/src/core/memory.ts +0 -589
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
|
|
2
|
-
|
|
3
|
-
|
|
1
|
+
# rlm.pi PI plugin
|
|
2
|
+
|
|
3
|
+
> pi-rlm — Large contexts on cheap models: Recursive Language Model (RLM) for Pi
|
|
4
4
|
|
|
5
5
|
<p align="center">
|
|
6
6
|
<a href="https://www.npmjs.com/package/@hicaru/pi-rlm"><img src="https://img.shields.io/npm/v/@hicaru/pi-rlm?color=cb3837&logo=npm" alt="npm version"></a>
|
|
@@ -14,45 +14,74 @@
|
|
|
14
14
|
<a href="https://www.npmjs.com/package/@hicaru/pi-rlm">📦 npm</a>
|
|
15
15
|
</p>
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
## Install
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
pi install npm:@hicaru/pi-rlm
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Run `/reload` in Pi — `/rlm`, `/rlm-config`, `/rlm-stop` appear under **[Extensions]**.
|
|
24
|
+
Toggle with `Ctrl+Shift+R` or `/rlm`.
|
|
18
25
|
|
|
19
|
-
|
|
20
|
-
separate CLI, no YAML workflows — just `/rlm` and your existing Pi session becomes a
|
|
21
|
-
recursive orchestration engine — your best model orchestrates, cheap worker models
|
|
22
|
-
do the reading.
|
|
26
|
+
To remove it later:
|
|
23
27
|
|
|
24
|
-
|
|
28
|
+
```bash
|
|
29
|
+
pi uninstall npm:@hicaru/pi-rlm
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
<p align="center">
|
|
33
|
+
<img src="https://github.com/openzebra/rlm.pi/blob/master/assets/hero.png?raw=true" width="100%" alt="rlm.pi — OOLONG benchmark results">
|
|
34
|
+
</p>
|
|
25
35
|
|
|
26
|
-
##
|
|
36
|
+
## What is pi-rlm?
|
|
27
37
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
| 🧠 **Smartest model orchestrates, cheapest model researches** | Root uses your best model; workers auto-pick the cheapest. Recursive children inherit the full `context` for free. |
|
|
34
|
-
| ⏳ **Long-running with goals** | Toggle `/rlm` on, set a goal, let it loop. Runs survive across chat turns — go make coffee. |
|
|
35
|
-
| 🔒 **100% local, 100% private** | No servers. Your API keys never leave your machine. One `python3` subprocess — that's it. |
|
|
38
|
+
A Pi plugin that turns your session into a **Recursive Language Model (RLM)**: instead of
|
|
39
|
+
stuffing a huge document into the prompt, the context lives in a Python REPL and your best
|
|
40
|
+
model orchestrates it — searching, decomposing, and delegating leaf reads to cheap worker
|
|
41
|
+
models, recursively. Same Pi session, same tools, same keys: `/rlm` and go. Reads
|
|
42
|
+
`.pdf` `.docx` `.xlsx` `.epub` and more, works with any OpenRouter model, 100% local.
|
|
36
43
|
|
|
37
|
-
##
|
|
44
|
+
## Benchmarks
|
|
38
45
|
|
|
39
|
-
|
|
46
|
+
**OOLONG (oolong-synth)** — paper-tier long-context suite; latest journal per model,
|
|
47
|
+
cost per task from real `costUsd` (older journals estimated at OpenRouter list prices):
|
|
40
48
|
|
|
41
|
-
|
|
49
|
+
| Model | Score | Avg. cost/task |
|
|
50
|
+
|-------|-------|----------------|
|
|
51
|
+
| `qwen/qwen3.8-27b` | **100%** | $0.0127 |
|
|
52
|
+
| `google/gemma-3-27b-it` | 83.3% | $0.0013 |
|
|
53
|
+
| `qwen/qwen3-30b-a3b-instruct-2507` | 66.7% | $0.0009 |
|
|
54
|
+
| `mistralai/mistral-small-3.2-24b-instruct` | 66.7% | $0.0025 |
|
|
42
55
|
|
|
43
|
-
|
|
56
|
+
Lite suite — `needle` multi-needle recall, `codeqa` repo-QA, `coding` fix task
|
|
57
|
+
(7 tasks × 2 passes per model, deterministic graders, no LLM-as-judge):
|
|
44
58
|
|
|
45
|
-
|
|
59
|
+
| Model | Score | Accuracy |
|
|
60
|
+
|-------|-------|----------|
|
|
61
|
+
| `qwen/qwen3-30b-a3b-instruct-2507` | **14/14** | **100%** |
|
|
62
|
+
| `google/gemma-3-27b-it` | 12/14 | 86% |
|
|
63
|
+
| `mistralai/mistral-small-3.2-24b-instruct` | 12/14 | 86% |
|
|
64
|
+
|
|
65
|
+
Raw per-task rows (correct, recall, latency, tokens, cost) live in
|
|
66
|
+
`bench/runs/*.jsonl` — one JSONL row per task, committed as history.
|
|
67
|
+
|
|
68
|
+
### Run the benchmarks
|
|
46
69
|
|
|
47
70
|
```bash
|
|
48
|
-
|
|
49
|
-
```
|
|
71
|
+
export OPENROUTER_API_KEY=sk-or-... # required — env vars are the only key transport
|
|
50
72
|
|
|
51
|
-
|
|
73
|
+
bun run bench # lite suite: needle + codeqa + coding
|
|
74
|
+
bun run bench --suite needle --limit 1 # one suite, first task only
|
|
75
|
+
bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
|
|
76
|
+
bun run bench --list # print tasks, no engine / no key
|
|
77
|
+
bun run bench --suite paper # paper tier: s_niah, oolong, browsecomp, codeqa_lb (downloads datasets)
|
|
78
|
+
```
|
|
52
79
|
|
|
53
|
-
|
|
80
|
+
Suites: `all` (lite, default) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
|
|
81
|
+
`oolong` · `browsecomp` · `codeqa_lb`. Regenerate the hero chart:
|
|
82
|
+
`python3 bench/hero.py` (needs `matplotlib`).
|
|
54
83
|
|
|
55
|
-
##
|
|
84
|
+
## How it works
|
|
56
85
|
|
|
57
86
|
```
|
|
58
87
|
┌─────────────────────────┐
|
|
@@ -106,10 +135,6 @@ A **Recursive Language Model (RLM)** replaces `llm.completion(prompt)` with
|
|
|
106
135
|
launch sub-LLM and sub-RLM calls as ordinary Python functions — decomposing,
|
|
107
136
|
delegating, and synthesizing across a tree of models, not a single context window.
|
|
108
137
|
|
|
109
|
-
**This is the only plugin that brings true RLM recursion to Pi.** Prime Agent and
|
|
110
|
-
the reference Python library are separate agents you must switch to. pi-rlm lives
|
|
111
|
-
inside Pi — same session, same tools, same everything.
|
|
112
|
-
|
|
113
138
|
## Commands
|
|
114
139
|
|
|
115
140
|
| Command | Shortcut | What it does |
|
|
@@ -129,6 +154,45 @@ inside Pi — same session, same tools, same everything.
|
|
|
129
154
|
| REPL timeout | `120`s | Bump for slow computations |
|
|
130
155
|
| Max concurrent subs | `16` | More parallelism (costs RAM) |
|
|
131
156
|
|
|
157
|
+
## Sampling & reproducibility
|
|
158
|
+
|
|
159
|
+
The r3 bench showed the biggest capability lever is not the model — it is the sampling:
|
|
160
|
+
temperature 0 took OOLONG from 71% pooled / 40% flips to 91.7–100% all-stable for $1.74
|
|
161
|
+
total. Those knobs are first-class in `rlm.json` (`~/.pi/agent/rlm.json`) and on the
|
|
162
|
+
`/rlm-config` panel:
|
|
163
|
+
|
|
164
|
+
| Field | Where | Default | What it governs |
|
|
165
|
+
|-------|-------|---------|-----------------|
|
|
166
|
+
| `rootSampling.maxTokens` | rlm.json, panel | `16384` | Output cap per root-model turn (finalize included) |
|
|
167
|
+
| `rootSampling.temperature` | rlm.json, panel | provider default | Root + finalize sampling temperature; `0` = deterministic |
|
|
168
|
+
| `smartReasoning` | rlm.json, panel | none | Thinking effort for the root model |
|
|
169
|
+
| `subSampling.maxTokens` | rlm.json, panel | `8192` | Output cap per leaf sub-call (`llm_query`, `llm_batch`, `map_files`) |
|
|
170
|
+
| `subSampling.temperature` | rlm.json, panel | provider default | Leaf sampling temperature |
|
|
171
|
+
| `enableVerificationNudge` | rlm.json | off | One coached redo when the root finalizes early with a bare number / short label |
|
|
172
|
+
|
|
173
|
+
**Reproducibility recipe (validated by r3):**
|
|
174
|
+
|
|
175
|
+
```json
|
|
176
|
+
{
|
|
177
|
+
"config": {
|
|
178
|
+
"rootSampling": { "maxTokens": 8192, "temperature": 0, "reasoning": "high" }
|
|
179
|
+
}
|
|
180
|
+
}
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Reasoning tokens share the completion budget with the answer — with thinking on, keep
|
|
184
|
+
`maxTokens` generous (the bench doubles it to 8192; the engine warns once on turn 0 when it
|
|
185
|
+
is tight).
|
|
186
|
+
|
|
187
|
+
**Scope boundary:** `rlm.json` sampling applies to RLM-mode runs, `rlm()` delegation, and
|
|
188
|
+
child recursion at any depth (same engine function). The native Pi agent loop follows Pi's
|
|
189
|
+
own session settings — rlm.json never touches it.
|
|
190
|
+
|
|
191
|
+
**Model capability:** reasoning requires a model whose registry entry has `reasoning: true`.
|
|
192
|
+
Anything else has the level dropped before it reaches the provider (pi-ai clamps unsupported
|
|
193
|
+
levels to off); capability comes from the registry, so OpenRouter hybrids like
|
|
194
|
+
`qwen/qwen3.8-27b` just work.
|
|
195
|
+
|
|
132
196
|
## Prompt Architecture
|
|
133
197
|
|
|
134
198
|
The system prompt follows a **contract / routing / examples / rules** pattern
|
|
@@ -142,38 +206,6 @@ The system prompt follows a **contract / routing / examples / rules** pattern
|
|
|
142
206
|
**Key insight:** children see `Recursion depth: N` and calibrate ambition —
|
|
143
207
|
delegating only when their task genuinely decomposes further.
|
|
144
208
|
|
|
145
|
-
## Benchmarks
|
|
146
|
-
|
|
147
|
-
E2E runs of the real engine against OpenRouter chat models — the lite suite
|
|
148
|
-
(`needle` multi-needle recall, `codeqa` repo-QA, `coding` fix task; 7 tasks × 2 passes
|
|
149
|
-
per model). Deterministic graders (recall / gold containment / regex), no LLM-as-judge.
|
|
150
|
-
|
|
151
|
-
Latest results — small models (≤32B parameters, paid tier):
|
|
152
|
-
|
|
153
|
-
| Model | Params | Score | Accuracy | Latency/task |
|
|
154
|
-
|-------|--------|-------|----------|--------------|
|
|
155
|
-
| `qwen/qwen3-30b-a3b-instruct-2507` | MoE 30B / 3B active | **14/14** | **100%** | ~15s |
|
|
156
|
-
| `google/gemma-3-27b-it` | dense 27B | 12/14 | 86% | ~26s |
|
|
157
|
-
| `mistralai/mistral-small-3.2-24b-instruct` | dense 24B | 12/14 | 86% | ~28s |
|
|
158
|
-
|
|
159
|
-
Raw per-task rows (correct, recall, latency, tokens, cost) live in
|
|
160
|
-
`bench/runs/bench-<ts>.jsonl` — one JSONL row per task, committed as history.
|
|
161
|
-
|
|
162
|
-
### Run the benchmarks
|
|
163
|
-
|
|
164
|
-
```bash
|
|
165
|
-
export OPENROUTER_API_KEY=sk-or-... # required — env vars are the only key transport
|
|
166
|
-
|
|
167
|
-
bun run bench # lite suite: needle + codeqa + coding
|
|
168
|
-
bun run bench --suite needle --limit 1 # one suite, first task only
|
|
169
|
-
bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
|
|
170
|
-
bun run bench --list # print tasks, no engine / no key
|
|
171
|
-
bun run bench --suite paper # paper tier: s_niah, oolong, browsecomp, codeqa_lb (downloads datasets)
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
Suites: `all` (lite, default) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
|
|
175
|
-
`oolong` · `browsecomp` · `codeqa_lb`.
|
|
176
|
-
|
|
177
209
|
## Security
|
|
178
210
|
|
|
179
211
|
- **Key isolation** — provider keys live in TypeScript only; sandbox receives prompts, returns text.
|
|
@@ -182,12 +214,6 @@ Suites: `all` (lite, default) · `needle` · `codeqa` · `coding` · `paper` ·
|
|
|
182
214
|
- **Per-block timeout** — SIGALRM + parent watchdog (SIGKILL on hang).
|
|
183
215
|
- **Trust** — project-local install requires Pi project trust.
|
|
184
216
|
|
|
185
|
-
## Uninstall
|
|
186
|
-
|
|
187
|
-
```bash
|
|
188
|
-
pi uninstall npm:@hicaru/pi-rlm
|
|
189
|
-
```
|
|
190
|
-
|
|
191
217
|
## License
|
|
192
218
|
|
|
193
219
|
MIT — see [LICENSE](./LICENSE).
|
package/README.ru.md
CHANGED
|
@@ -1,11 +1,5 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
|
|
3
|
-
<img src="https://github.com/openzebra/rlm.pi/blob/master/assets/hero.png?raw=true" alt="pi-rlm">
|
|
4
|
-
|
|
5
|
-
</div>
|
|
6
|
-
|
|
7
|
-
<div align="center">
|
|
8
|
-
|
|
9
3
|
<sub>
|
|
10
4
|
<a href="README.md">English</a> · <a href="README.zh-CN.md">中文</a> · **Русский**
|
|
11
5
|
</sub>
|
|
@@ -14,33 +8,68 @@
|
|
|
14
8
|
|
|
15
9
|
---
|
|
16
10
|
|
|
17
|
-
# pi
|
|
11
|
+
# rlm.pi PI plugin
|
|
18
12
|
|
|
19
|
-
|
|
13
|
+
> pi-rlm — большие контексты на дешёвых моделях: рекурсивная языковая модель (RLM) для Pi
|
|
20
14
|
|
|
21
|
-
|
|
22
|
-
ПОЛНОСТЬЮ ЛОКАЛЬНО.
|
|
15
|
+
## Установка
|
|
23
16
|
|
|
24
|
-
|
|
17
|
+
```bash
|
|
18
|
+
pi install npm:@hicaru/pi-rlm
|
|
19
|
+
```
|
|
25
20
|
|
|
26
|
-
|
|
21
|
+
Чтобы удалить позже:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
pi uninstall npm:@hicaru/pi-rlm
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Затем выполните `/reload` или перезапустите Pi — `/rlm`, `/rlm-config` и `/rlm-stop` появятся в разделе **[Extensions]**. Переключение — `Ctrl+Shift+R` или `/rlm`.
|
|
27
28
|
|
|
28
|
-
|
|
29
|
+
<p align="center">
|
|
30
|
+
<img src="https://github.com/openzebra/rlm.pi/blob/master/assets/hero.png?raw=true" width="100%" alt="rlm.pi — результаты OOLONG">
|
|
31
|
+
</p>
|
|
29
32
|
|
|
30
|
-
|
|
33
|
+
## Что такое pi-rlm?
|
|
31
34
|
|
|
32
|
-
|
|
35
|
+
Плагин, который превращает сессию Pi в **рекурсивную языковую модель (RLM)**: вместо того чтобы заталкивать огромный документ в промпт, контекст живёт в Python REPL, а ваша лучшая модель им управляет — ищет, декомпозирует и делегирует чтения дешёвым worker-моделям, рекурсивно. Та же сессия Pi, те же инструменты, те же ключи — включите `/rlm` и вперёд. По методу [RLM paper](https://arxiv.org/abs/2512.24601); подробности — в разделе **Как это работает**.
|
|
33
36
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
-
|
|
37
|
+
## Бенчмарки
|
|
38
|
+
|
|
39
|
+
**OOLONG (oolong-synth)** — paper-tier сюит длинного контекста; последний журнал каждой модели, цена за задачу из реального `costUsd` (старые журналы оценены по прайс-листу OpenRouter):
|
|
40
|
+
|
|
41
|
+
| Модель | Счёт | Цена/задача |
|
|
42
|
+
|--------|------|-------------|
|
|
43
|
+
| `qwen/qwen3.8-27b` | **100%** | $0.0127 |
|
|
44
|
+
| `google/gemma-3-27b-it` | 83.3% | $0.0013 |
|
|
45
|
+
| `qwen/qwen3-30b-a3b-instruct-2507` | 66.7% | $0.0009 |
|
|
46
|
+
| `mistralai/mistral-small-3.2-24b-instruct` | 66.7% | $0.0025 |
|
|
47
|
+
|
|
48
|
+
Lite-сьют — `needle` (поиск игл в куче), `codeqa` (вопросы по коду), `coding` (задача-фикс; 7 задач × 2 прохода на модель, детерминированные грейдеры, без LLM-судьи):
|
|
41
49
|
|
|
42
|
-
|
|
43
|
-
|
|
50
|
+
| Модель | Счёт | Точность |
|
|
51
|
+
|--------|------|----------|
|
|
52
|
+
| `qwen/qwen3-30b-a3b-instruct-2507` | **14/14** | **100%** |
|
|
53
|
+
| `google/gemma-3-27b-it` | 12/14 | 86% |
|
|
54
|
+
| `mistralai/mistral-small-3.2-24b-instruct` | 12/14 | 86% |
|
|
55
|
+
|
|
56
|
+
Построчные результаты (correct, recall, latency, токены, цена) — в
|
|
57
|
+
`bench/runs/*.jsonl`, одна JSONL-строка на задачу.
|
|
58
|
+
|
|
59
|
+
### Как запустить
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
export OPENROUTER_API_KEY=sk-or-... # обязателен — ключи ходят только через env
|
|
63
|
+
|
|
64
|
+
bun run bench # lite-сьют: needle + codeqa + coding
|
|
65
|
+
bun run bench --suite needle --limit 1 # один сьют, первая задача
|
|
66
|
+
bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
|
|
67
|
+
bun run bench --list # показать задачи, без движка и ключа
|
|
68
|
+
bun run bench --suite paper # paper-сьют: s_niah, oolong, browsecomp, codeqa_lb (скачивает датасеты)
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Сьюты: `all` (lite, по умолчанию) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
|
|
72
|
+
`oolong` · `browsecomp` · `codeqa_lb`.
|
|
44
73
|
|
|
45
74
|
## Как это работает
|
|
46
75
|
|
|
@@ -60,24 +89,17 @@ pi process (TypeScript)
|
|
|
60
89
|
`rlm_query_batched`, `SHOW_VARS()`, `ask_user_question()` и словарь `answer`.
|
|
61
90
|
Модель отправляет окончательный результат, устанавливая `answer["ready"] = True`.
|
|
62
91
|
|
|
63
|
-
## Установка
|
|
92
|
+
## Установка из исходников (для разработки)
|
|
64
93
|
|
|
65
94
|
`pi-rlm` — это пакет Pi. Pi предоставляет peer-зависимости `@earendil-works/pi-*` и `typebox`; **не** устанавливайте их отдельную копию в этот пакет. Требуется `python3` в `PATH` (только стандартная библиотека).
|
|
66
95
|
|
|
67
|
-
|
|
96
|
+
Локальная установка при разработке:
|
|
68
97
|
|
|
69
98
|
```bash
|
|
70
99
|
pi install /path/to/this-repo/pi-plugin/rlm
|
|
71
100
|
```
|
|
72
101
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
```bash
|
|
76
|
-
npm publish # например, как @<you>/pi-rlm
|
|
77
|
-
pi install npm:@<you>/pi-rlm
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
> **Установка через Git** требует, чтобы манифест пакета находился в корне устанавливаемого репозитория. Для поддиректорий монорепозитория, таких как эта, предпочтительнее использовать локальный путь или npm, как указано выше.
|
|
102
|
+
> **Установка через Git** требует, чтобы манифест пакета находился в корне устанавливаемого репозитория. Для поддиректорий монорепозитория, таких как эта, предпочтительнее использовать локальный путь, как указано выше.
|
|
81
103
|
|
|
82
104
|
Если вы ранее копировали папку расширения напрямую, удалите ее, чтобы она не перекрывала пакет:
|
|
83
105
|
|
|
@@ -132,40 +154,45 @@ rm -rf ~/.pi/agent/extensions/rlm
|
|
|
132
154
|
|
|
133
155
|
> **Примечание по параллелизму:** каждый дочерний `rlm_query` запускает собственного worker `python3` (~50–150 мс «холодного старта»). В худшем случае количество параллельных интерпретаторов ≈ `maxConcurrentSubcalls`^(depth−1); при настройках по умолчанию (глубина 4, параллелизм 4) это 4³ = 64 в патологическом случае. Лимиты бюджета и ошибок (см. выше) ограничивают общие затраты независимо от степени разветвления.
|
|
134
156
|
|
|
135
|
-
##
|
|
157
|
+
## Сэмплинг и воспроизводимость
|
|
136
158
|
|
|
159
|
+
Бенчмарк r3 показал: самый большой рычаг качества — не модель, а сэмплинг. Temperature 0
|
|
160
|
+
перевела OOLONG с 71% (pooled) / 40% (flips) на 91.7–100% all-stable за $1.74 суммарно.
|
|
161
|
+
Эти ручки — полноправные поля `rlm.json` (`~/.pi/agent/rlm.json`) и панели `/rlm-config`:
|
|
137
162
|
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
163
|
+
| Поле | Где | По умолчанию | За что отвечает |
|
|
164
|
+
|---|---|---|---|
|
|
165
|
+
| `rootSampling.maxTokens` | rlm.json, панель | `16384` | Лимит вывода на ход корневой модели (включая finalize) |
|
|
166
|
+
| `rootSampling.temperature` | rlm.json, панель | провайдер | Температура корневых ходов и finalize; `0` = детерминированно |
|
|
167
|
+
| `smartReasoning` | rlm.json, панель | нет | Усилие «размышления» корневой модели |
|
|
168
|
+
| `subSampling.maxTokens` | rlm.json, панель | `8192` | Лимит вывода одного листового вызова (`llm_query`, `llm_batch`, `map_files`) |
|
|
169
|
+
| `subSampling.temperature` | rlm.json, панель | провайдер | Температура листовых вызовов |
|
|
170
|
+
| `enableVerificationNudge` | rlm.json | выкл | Один повтор с подсказкой, если корень рано финализирует «голым числом» |
|
|
143
171
|
|
|
144
|
-
|
|
172
|
+
**Рецепт воспроизводимости (проверен r3):**
|
|
145
173
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
174
|
+
```json
|
|
175
|
+
{
|
|
176
|
+
"config": {
|
|
177
|
+
"rootSampling": { "maxTokens": 8192, "temperature": 0, "reasoning": "high" }
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
```
|
|
151
181
|
|
|
152
|
-
|
|
153
|
-
`
|
|
182
|
+
Токены размышлений делят бюджет завершения с ответом — при включённом thinking держите
|
|
183
|
+
`maxTokens` щедрым (бенчмарк удваивает его до 8192; движок один раз предупредит на ходе 0,
|
|
184
|
+
если лимит тесный).
|
|
154
185
|
|
|
155
|
-
|
|
186
|
+
**Граница действия:** сэмплинг из `rlm.json` управляет запусками RLM-режима, делегированием
|
|
187
|
+
`rlm()` и рекурсией на любой глубине (тот же движок). Собственный цикл агента Pi следует
|
|
188
|
+
настройкам сессии Pi — rlm.json его не касается.
|
|
156
189
|
|
|
157
|
-
|
|
158
|
-
|
|
190
|
+
**Возможности модели:** reasoning требует модель с `reasoning: true` в записи реестра.
|
|
191
|
+
Для остальных уровень отбрасывается до отправки провайдеру (pi-ai приводит его к off);
|
|
192
|
+
возможность берётся из реестра, а не из конфига.
|
|
159
193
|
|
|
160
|
-
|
|
161
|
-
bun run bench --suite needle --limit 1 # один сьют, первая задача
|
|
162
|
-
bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
|
|
163
|
-
bun run bench --list # показать задачи, без движка и ключа
|
|
164
|
-
bun run bench --suite paper # paper-сьют: s_niah, oolong, browsecomp, codeqa_lb (скачивает датасеты)
|
|
165
|
-
```
|
|
194
|
+
## Логи запусков
|
|
166
195
|
|
|
167
|
-
Сьюты: `all` (lite, по умолчанию) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
|
|
168
|
-
`oolong` · `browsecomp` · `codeqa_lb`.
|
|
169
196
|
|
|
170
197
|
## Безопасность
|
|
171
198
|
|
package/README.zh-CN.md
CHANGED
|
@@ -1,11 +1,5 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
|
|
3
|
-
<img src="https://github.com/openzebra/rlm.pi/blob/master/assets/hero.png?raw=true" alt="pi-rlm">
|
|
4
|
-
|
|
5
|
-
</div>
|
|
6
|
-
|
|
7
|
-
<div align="center">
|
|
8
|
-
|
|
9
3
|
<sub>
|
|
10
4
|
<a href="README.md">English</a> · **中文** · <a href="README.ru.md">Русский</a>
|
|
11
5
|
</sub>
|
|
@@ -14,32 +8,73 @@
|
|
|
14
8
|
|
|
15
9
|
---
|
|
16
10
|
|
|
17
|
-
#
|
|
11
|
+
# rlm.pi PI plugin
|
|
18
12
|
|
|
19
|
-
|
|
13
|
+
> pi-rlm — 大上下文,廉价模型:为 Pi 提供的递归语言模型 (RLM)
|
|
20
14
|
|
|
21
|
-
|
|
22
|
-
完全本地。
|
|
15
|
+
## 安装
|
|
23
16
|
|
|
24
|
-
|
|
17
|
+
```bash
|
|
18
|
+
pi install npm:@hicaru/pi-rlm
|
|
19
|
+
```
|
|
25
20
|
|
|
26
|
-
|
|
21
|
+
以后要移除:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
pi uninstall npm:@hicaru/pi-rlm
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
然后在 Pi 中运行 `/reload` —— `/rlm`、`/rlm-config` 和 `/rlm-stop` 会出现在 **[Extensions]** 下。使用 `Ctrl+Shift+R` 或 `/rlm` 切换。
|
|
27
28
|
|
|
28
|
-
|
|
29
|
+
<p align="center">
|
|
30
|
+
<img src="https://github.com/openzebra/rlm.pi/blob/master/assets/hero.png?raw=true" width="100%" alt="rlm.pi — OOLONG 基准测试结果">
|
|
31
|
+
</p>
|
|
32
|
+
|
|
33
|
+
## 什么是 pi-rlm?
|
|
34
|
+
|
|
35
|
+
一个把 Pi 会话变成**递归语言模型 (RLM)** 的插件:不必把庞大的文档塞进提示词,上下文
|
|
36
|
+
驻留在 Python REPL 中,由你最好的模型编排 —— 检索、分解,并把叶子读取递归地委派给
|
|
37
|
+
廉价的工作模型。同一个 Pi 会话、同样的工具、同样的密钥 —— 打开 `/rlm` 即可。
|
|
38
|
+
方法基于 [RLM 论文](https://arxiv.org/abs/2512.24601);详见下方的**工作原理**。
|
|
39
|
+
|
|
40
|
+
## 基准测试
|
|
29
41
|
|
|
30
|
-
|
|
42
|
+
**OOLONG (oolong-synth)** —— paper 级长上下文套件;取每个模型最新的日志,每任务成本
|
|
43
|
+
来自真实 `costUsd`(旧日志按 OpenRouter 牌价估算):
|
|
31
44
|
|
|
32
|
-
|
|
45
|
+
| 模型 | 得分 | 每任务成本 |
|
|
46
|
+
|------|------|------------|
|
|
47
|
+
| `qwen/qwen3.8-27b` | **100%** | $0.0127 |
|
|
48
|
+
| `google/gemma-3-27b-it` | 83.3% | $0.0013 |
|
|
49
|
+
| `qwen/qwen3-30b-a3b-instruct-2507` | 66.7% | $0.0009 |
|
|
50
|
+
| `mistralai/mistral-small-3.2-24b-instruct` | 66.7% | $0.0025 |
|
|
33
51
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
- 困难的子问题通过 `rlm_query` **递归**到子 RLM 中(设有深度限制)。子 RLM 继承父级的 `context`
|
|
37
|
-
——已加载的文件以及通过 `add_context()` 追加的来源——因此可以在相同的路径上使用相同的检索原语。
|
|
38
|
-
继承不消耗额外的 token:内容存放在沙箱中,模型只看到一行大小信息。
|
|
39
|
-
- 所有内容均**在进程内**运行 —— 唯一的外部进程是一个本地的 `python3` worker。
|
|
52
|
+
精简套件 —— `needle`(多针召回)、`codeqa`(代码库问答)、`coding`(修复任务;
|
|
53
|
+
每模型 7 个任务 × 2 轮,确定性评分,无 LLM 评审):
|
|
40
54
|
|
|
41
|
-
|
|
42
|
-
|
|
55
|
+
| 模型 | 得分 | 准确率 |
|
|
56
|
+
|------|------|--------|
|
|
57
|
+
| `qwen/qwen3-30b-a3b-instruct-2507` | **14/14** | **100%** |
|
|
58
|
+
| `google/gemma-3-27b-it` | 12/14 | 86% |
|
|
59
|
+
| `mistralai/mistral-small-3.2-24b-instruct` | 12/14 | 86% |
|
|
60
|
+
|
|
61
|
+
逐任务原始数据(正确性、召回率、延迟、token、成本)位于 `bench/runs/*.jsonl`
|
|
62
|
+
—— 每个任务一行 JSONL,作为历史记录提交。
|
|
63
|
+
|
|
64
|
+
### 运行基准测试
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
export OPENROUTER_API_KEY=sk-or-... # 必需 —— 密钥仅通过环境变量传递
|
|
68
|
+
|
|
69
|
+
bun run bench # 精简套件:needle + codeqa + coding
|
|
70
|
+
bun run bench --suite needle --limit 1 # 单个套件,仅第一个任务
|
|
71
|
+
bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
|
|
72
|
+
bun run bench --list # 仅列出任务,无需引擎和密钥
|
|
73
|
+
bun run bench --suite paper # paper 套件:s_niah, oolong, browsecomp, codeqa_lb(需下载数据集)
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
套件:`all`(精简版,默认) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
|
|
77
|
+
`oolong` · `browsecomp` · `codeqa_lb`。
|
|
43
78
|
|
|
44
79
|
## 工作原理
|
|
45
80
|
|
|
@@ -61,26 +96,19 @@ pi 进程 (TypeScript)
|
|
|
61
96
|
`rlm_query_batched`, `SHOW_VARS()`, `ask_user_question()` 以及一个 `answer` 字典。
|
|
62
97
|
模型通过设置 `answer["ready"] = True` 来提交最终结果。
|
|
63
98
|
|
|
64
|
-
##
|
|
99
|
+
## 从源码安装(开发)
|
|
65
100
|
|
|
66
101
|
`pi-rlm` 是一个 Pi 包。Pi 提供了 `@earendil-works/pi-*` 和 `typebox` peer
|
|
67
102
|
依赖;请**不要**在该包中安装它们的独立副本。要求 `PATH` 中有 `python3` (仅限标准库)。
|
|
68
103
|
|
|
69
|
-
|
|
104
|
+
开发时的本地安装方式:
|
|
70
105
|
|
|
71
106
|
```bash
|
|
72
107
|
pi install /path/to/this-repo/pi-plugin/rlm
|
|
73
108
|
```
|
|
74
109
|
|
|
75
|
-
已发布的 npm 包安装方式:
|
|
76
|
-
|
|
77
|
-
```bash
|
|
78
|
-
npm publish # 例如 as @<you>/pi-rlm
|
|
79
|
-
pi install npm:@<you>/pi-rlm
|
|
80
|
-
```
|
|
81
|
-
|
|
82
110
|
> **Git 安装**要求包清单位于安装的仓库根目录下。
|
|
83
|
-
> 对于像这样一个 monorepo
|
|
111
|
+
> 对于像这样一个 monorepo 子目录,请优先使用上述的本地路径流程。
|
|
84
112
|
|
|
85
113
|
如果您之前直接复制了扩展文件夹,请将其删除,以免遮蔽 (shadow) 该包:
|
|
86
114
|
|
|
@@ -141,6 +169,40 @@ rm -rf ~/.pi/agent/extensions/rlm
|
|
|
141
169
|
> 默认设置下 (深度 4, 并发 4),极端情况下为 4³ = 64。预算和错误
|
|
142
170
|
> 上限 (见上文) 无论扇出 (fan-out) 如何都会限制总支出。
|
|
143
171
|
|
|
172
|
+
## 采样与可复现性
|
|
173
|
+
|
|
174
|
+
r3 基准测试表明:最大的能力杠杆不是模型,而是采样。仅 temperature 0 就让 OOLONG 从
|
|
175
|
+
71% (pooled) / 40% (flips) 提升到 91.7–100% all-stable,总计仅花费 $1.74。这些旋钮在
|
|
176
|
+
`rlm.json` (`~/.pi/agent/rlm.json`) 和 `/rlm-config` 面板中都是一等公民:
|
|
177
|
+
|
|
178
|
+
| 字段 | 位置 | 默认值 | 作用范围 |
|
|
179
|
+
|---|---|---|---|
|
|
180
|
+
| `rootSampling.maxTokens` | rlm.json, 面板 | `16384` | 根模型每轮的输出上限 (包括 finalize) |
|
|
181
|
+
| `rootSampling.temperature` | rlm.json, 面板 | 提供商默认 | 根轮次与 finalize 的采样温度;`0` = 确定性 |
|
|
182
|
+
| `smartReasoning` | rlm.json, 面板 | 无 | 根模型的思考强度 |
|
|
183
|
+
| `subSampling.maxTokens` | rlm.json, 面板 | `8192` | 每个叶子子调用 (`llm_query`, `llm_batch`, `map_files`) 的输出上限 |
|
|
184
|
+
| `subSampling.temperature` | rlm.json, 面板 | 提供商默认 | 叶子调用的采样温度 |
|
|
185
|
+
| `enableVerificationNudge` | rlm.json | 关 | 当根节点过早以“裸数字”定稿时,给予一次带提示的重做 |
|
|
186
|
+
|
|
187
|
+
**可复现性配方 (经 r3 验证):**
|
|
188
|
+
|
|
189
|
+
```json
|
|
190
|
+
{
|
|
191
|
+
"config": {
|
|
192
|
+
"rootSampling": { "maxTokens": 8192, "temperature": 0, "reasoning": "high" }
|
|
193
|
+
}
|
|
194
|
+
}
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
思考 token 与答案共享补全预算——开启 thinking 时请保持充裕的 `maxTokens`
|
|
198
|
+
(基准测试将其翻倍至 8192;若过紧,引擎会在第 0 轮警告一次)。
|
|
199
|
+
|
|
200
|
+
**作用边界:** `rlm.json` 的采样适用于 RLM 模式运行、`rlm()` 委托以及任意深度的子递归
|
|
201
|
+
(同一个引擎函数)。Pi 原生代理循环遵循 Pi 自身的会话设置——rlm.json 不会触及它。
|
|
202
|
+
|
|
203
|
+
**模型能力:** reasoning 要求模型的注册表条目为 `reasoning: true`。其他情况会在发送给
|
|
204
|
+
提供商之前丢弃该级别 (pi-ai 会将不支持的级别钳制为 off);能力来自注册表,而非配置。
|
|
205
|
+
|
|
144
206
|
## 运行日志
|
|
145
207
|
|
|
146
208
|
- **密钥隔离**:供应商密钥仅存在于 TypeScript (`AuthStorage`) 中;沙箱
|
|
@@ -157,38 +219,6 @@ rm -rf ~/.pi/agent/extensions/rlm
|
|
|
157
219
|
连续错误上限。
|
|
158
220
|
- **信任**:本地安装需要 Pi 项目信任。
|
|
159
221
|
|
|
160
|
-
## 基准测试
|
|
161
|
-
|
|
162
|
-
针对 OpenRouter 聊天模型对真实引擎进行端到端测试 —— 精简套件(`needle` 多针召回、
|
|
163
|
-
`codeqa` 代码库问答、`coding` 修复任务;每个模型 7 个任务 × 2 轮)。确定性评分
|
|
164
|
-
(召回率 / 标准答案包含 / 正则),无 LLM 评审。
|
|
165
|
-
|
|
166
|
-
最新结果 —— 小型模型(≤32B 参数,付费额度):
|
|
167
|
-
|
|
168
|
-
| 模型 | 参数 | 得分 | 准确率 | 每任务延迟 |
|
|
169
|
-
|------|------|------|--------|------------|
|
|
170
|
-
| `qwen/qwen3-30b-a3b-instruct-2507` | MoE 30B / 3B 激活 | **14/14** | **100%** | ~15s |
|
|
171
|
-
| `google/gemma-3-27b-it` | dense 27B | 12/14 | 86% | ~26s |
|
|
172
|
-
| `mistralai/mistral-small-3.2-24b-instruct` | dense 24B | 12/14 | 86% | ~28s |
|
|
173
|
-
|
|
174
|
-
逐任务原始数据(正确性、召回率、延迟、token、成本)位于 `bench/runs/bench-<ts>.jsonl`
|
|
175
|
-
—— 每个任务一行 JSONL,作为历史记录提交。
|
|
176
|
-
|
|
177
|
-
### 运行基准测试
|
|
178
|
-
|
|
179
|
-
```bash
|
|
180
|
-
export OPENROUTER_API_KEY=sk-or-... # 必需 —— 密钥仅通过环境变量传递
|
|
181
|
-
|
|
182
|
-
bun run bench # 精简套件:needle + codeqa + coding
|
|
183
|
-
bun run bench --suite needle --limit 1 # 单个套件,仅第一个任务
|
|
184
|
-
bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
|
|
185
|
-
bun run bench --list # 仅列出任务,无需引擎和密钥
|
|
186
|
-
bun run bench --suite paper # paper 套件:s_niah, oolong, browsecomp, codeqa_lb(需下载数据集)
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
套件:`all`(精简版,默认) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
|
|
190
|
-
`oolong` · `browsecomp` · `codeqa_lb`。
|
|
191
|
-
|
|
192
222
|
## 项目布局
|
|
193
223
|
|
|
194
224
|
```
|