@tagma/completion-llm-judge 0.1.4 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +99 -99
- package/package.json +2 -2
- package/src/index.ts +288 -288
package/README.md
CHANGED
|
@@ -1,99 +1,99 @@
|
|
|
1
|
-
# @tagma/completion-llm-judge
|
|
2
|
-
|
|
3
|
-
LLM-as-judge completion plugin for [@tagma/sdk](https://www.npmjs.com/package/@tagma/sdk).
|
|
4
|
-
|
|
5
|
-
Uses an OpenAI-compatible chat completions endpoint to verify whether a task's output satisfies a rubric. Complements the deterministic built-in completions (`exit_code`, `file_exists`, `output_check`) when task success is defined semantically rather than by a grep-able pattern.
|
|
6
|
-
|
|
7
|
-
**Default backend**: local [Ollama](https://ollama.com/) with `qwen3:4b` — a small reasoning model that runs on CPU with no API key. Swap `endpoint` + `model` to point at any OpenAI-compatible server (OpenAI, vLLM, llama.cpp, LM Studio, Groq, Together, OpenRouter, ...).
|
|
8
|
-
|
|
9
|
-
## Install
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
bun add @tagma/completion-llm-judge
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Then make sure Ollama is running with the default model pulled:
|
|
16
|
-
|
|
17
|
-
```bash
|
|
18
|
-
ollama pull qwen3:4b
|
|
19
|
-
ollama serve # usually auto-started
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
## Usage
|
|
23
|
-
|
|
24
|
-
```yaml
|
|
25
|
-
pipeline:
|
|
26
|
-
name: qa-loop
|
|
27
|
-
plugins:
|
|
28
|
-
- '@tagma/completion-llm-judge'
|
|
29
|
-
tracks:
|
|
30
|
-
- id: qa
|
|
31
|
-
name: QA
|
|
32
|
-
driver: claude-code
|
|
33
|
-
tasks:
|
|
34
|
-
- id: find-bugs
|
|
35
|
-
name: Find failing tests
|
|
36
|
-
prompt: 'List all failing tests in the current workspace with their file paths.'
|
|
37
|
-
completion:
|
|
38
|
-
type: llm_judge
|
|
39
|
-
rubric: |
|
|
40
|
-
The output must list at least 3 failing tests. Each entry must
|
|
41
|
-
include the test name, the file path, and the assertion that
|
|
42
|
-
failed. The output must not be an empty placeholder.
|
|
43
|
-
# endpoint / model / api_key_env all default to local Ollama + qwen3:4b
|
|
44
|
-
timeout: 120s
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
Swap to a hosted backend:
|
|
48
|
-
|
|
49
|
-
```yaml
|
|
50
|
-
completion:
|
|
51
|
-
type: llm_judge
|
|
52
|
-
rubric: '...'
|
|
53
|
-
endpoint: https://api.openai.com/v1/chat/completions
|
|
54
|
-
model: gpt-4o-mini
|
|
55
|
-
api_key_env: OPENAI_API_KEY
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
Or load it programmatically:
|
|
59
|
-
|
|
60
|
-
```ts
|
|
61
|
-
import { bootstrapBuiltins, loadPlugins } from '@tagma/sdk';
|
|
62
|
-
|
|
63
|
-
bootstrapBuiltins();
|
|
64
|
-
await loadPlugins(['@tagma/completion-llm-judge']);
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
## Config
|
|
68
|
-
|
|
69
|
-
| Field | Type | Default | Notes |
|
|
70
|
-
| ------------------ | -------- | -------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
71
|
-
| `rubric` | string | _(required)_ | Plain-language success criteria the judge should verify |
|
|
72
|
-
| `model` | string | `qwen3:4b` | Judge model name. Swap for `qwen3:8b`, `deepseek-r1:7b`, `gpt-4o-mini`, etc. |
|
|
73
|
-
| `endpoint` | string | `http://localhost:11434/v1/chat/completions` | OpenAI-compatible chat completions URL. Default points at local Ollama |
|
|
74
|
-
| `api_key_env` | string | _(none)_ | Env var holding the bearer token. Leave unset for local Ollama; set for hosted backends |
|
|
75
|
-
| `timeout` | duration | `120s` | Max time to wait for the judge response (reasoning models need more time than chat models) |
|
|
76
|
-
| `max_output_chars` | number | `8000` | Truncate task stdout before judging (head+tail preserved) |
|
|
77
|
-
|
|
78
|
-
## Behavior
|
|
79
|
-
|
|
80
|
-
- **Verdict format**: the judge is instructed to answer `PASS` or `FAIL` on the first line. Missing or ambiguous answers are treated as FAIL — a judge that errs open defeats the purpose of the gate.
|
|
81
|
-
- **Reasoning-model support**: `<think>...</think>` and `<thinking>...</thinking>` blocks are stripped from the response before verdict parsing, so qwen3, DeepSeek-R1, and other thinkers work without any extra config.
|
|
82
|
-
- **Truncation**: task stdout longer than `max_output_chars` is truncated head-and-tail (70/30 split) with a marker in the middle, so the judge still sees the task's intent and its final summary.
|
|
83
|
-
- **Error handling**: network errors, auth errors, timeout, or malformed responses all mark the task as not-complete and log a warning with the judge's verbatim response (if any).
|
|
84
|
-
- **Abort propagation**: the pipeline abort signal is wired into the judge fetch call, so cancelling a pipeline also cancels any in-flight judge request.
|
|
85
|
-
- **Determinism**: the call uses `temperature: 0` to keep verdicts as stable as the model allows.
|
|
86
|
-
|
|
87
|
-
## Alternative endpoints
|
|
88
|
-
|
|
89
|
-
Any OpenAI-compatible endpoint works — just point `endpoint` and `api_key_env` at it:
|
|
90
|
-
|
|
91
|
-
- **Local Ollama** (default): `http://localhost:11434/v1/chat/completions`, no API key
|
|
92
|
-
- **OpenAI**: `https://api.openai.com/v1/chat/completions`, `api_key_env: OPENAI_API_KEY`
|
|
93
|
-
- **Local models** via LM Studio, vLLM, llama.cpp OpenAI-compatible servers
|
|
94
|
-
- **Hosted**: Groq, Together, Fireworks, OpenRouter, DeepInfra, etc.
|
|
95
|
-
- **Anthropic** via an OpenAI-compat proxy (e.g. `anthropic-openai-proxy`)
|
|
96
|
-
|
|
97
|
-
## License
|
|
98
|
-
|
|
99
|
-
MIT
|
|
1
|
+
# @tagma/completion-llm-judge
|
|
2
|
+
|
|
3
|
+
LLM-as-judge completion plugin for [@tagma/sdk](https://www.npmjs.com/package/@tagma/sdk).
|
|
4
|
+
|
|
5
|
+
Uses an OpenAI-compatible chat completions endpoint to verify whether a task's output satisfies a rubric. Complements the deterministic built-in completions (`exit_code`, `file_exists`, `output_check`) when task success is defined semantically rather than by a grep-able pattern.
|
|
6
|
+
|
|
7
|
+
**Default backend**: local [Ollama](https://ollama.com/) with `qwen3:4b` — a small reasoning model that runs on CPU with no API key. Swap `endpoint` + `model` to point at any OpenAI-compatible server (OpenAI, vLLM, llama.cpp, LM Studio, Groq, Together, OpenRouter, ...).
|
|
8
|
+
|
|
9
|
+
## Install
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
bun add @tagma/completion-llm-judge
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Then make sure Ollama is running with the default model pulled:
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
ollama pull qwen3:4b
|
|
19
|
+
ollama serve # usually auto-started
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Usage
|
|
23
|
+
|
|
24
|
+
```yaml
|
|
25
|
+
pipeline:
|
|
26
|
+
name: qa-loop
|
|
27
|
+
plugins:
|
|
28
|
+
- '@tagma/completion-llm-judge'
|
|
29
|
+
tracks:
|
|
30
|
+
- id: qa
|
|
31
|
+
name: QA
|
|
32
|
+
driver: claude-code
|
|
33
|
+
tasks:
|
|
34
|
+
- id: find-bugs
|
|
35
|
+
name: Find failing tests
|
|
36
|
+
prompt: 'List all failing tests in the current workspace with their file paths.'
|
|
37
|
+
completion:
|
|
38
|
+
type: llm_judge
|
|
39
|
+
rubric: |
|
|
40
|
+
The output must list at least 3 failing tests. Each entry must
|
|
41
|
+
include the test name, the file path, and the assertion that
|
|
42
|
+
failed. The output must not be an empty placeholder.
|
|
43
|
+
# endpoint / model / api_key_env all default to local Ollama + qwen3:4b
|
|
44
|
+
timeout: 120s
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Swap to a hosted backend:
|
|
48
|
+
|
|
49
|
+
```yaml
|
|
50
|
+
completion:
|
|
51
|
+
type: llm_judge
|
|
52
|
+
rubric: '...'
|
|
53
|
+
endpoint: https://api.openai.com/v1/chat/completions
|
|
54
|
+
model: gpt-4o-mini
|
|
55
|
+
api_key_env: OPENAI_API_KEY
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Or load it programmatically:
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
import { bootstrapBuiltins, loadPlugins } from '@tagma/sdk';
|
|
62
|
+
|
|
63
|
+
bootstrapBuiltins();
|
|
64
|
+
await loadPlugins(['@tagma/completion-llm-judge']);
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Config
|
|
68
|
+
|
|
69
|
+
| Field | Type | Default | Notes |
|
|
70
|
+
| ------------------ | -------- | -------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
71
|
+
| `rubric` | string | _(required)_ | Plain-language success criteria the judge should verify |
|
|
72
|
+
| `model` | string | `qwen3:4b` | Judge model name. Swap for `qwen3:8b`, `deepseek-r1:7b`, `gpt-4o-mini`, etc. |
|
|
73
|
+
| `endpoint` | string | `http://localhost:11434/v1/chat/completions` | OpenAI-compatible chat completions URL. Default points at local Ollama |
|
|
74
|
+
| `api_key_env` | string | _(none)_ | Env var holding the bearer token. Leave unset for local Ollama; set for hosted backends |
|
|
75
|
+
| `timeout` | duration | `120s` | Max time to wait for the judge response (reasoning models need more time than chat models) |
|
|
76
|
+
| `max_output_chars` | number | `8000` | Truncate task stdout before judging (head+tail preserved) |
|
|
77
|
+
|
|
78
|
+
## Behavior
|
|
79
|
+
|
|
80
|
+
- **Verdict format**: the judge is instructed to answer `PASS` or `FAIL` on the first line. Missing or ambiguous answers are treated as FAIL — a judge that errs open defeats the purpose of the gate.
|
|
81
|
+
- **Reasoning-model support**: `<think>...</think>` and `<thinking>...</thinking>` blocks are stripped from the response before verdict parsing, so qwen3, DeepSeek-R1, and other thinkers work without any extra config.
|
|
82
|
+
- **Truncation**: task stdout longer than `max_output_chars` is truncated head-and-tail (70/30 split) with a marker in the middle, so the judge still sees the task's intent and its final summary.
|
|
83
|
+
- **Error handling**: network errors, auth errors, timeout, or malformed responses all mark the task as not-complete and log a warning with the judge's verbatim response (if any).
|
|
84
|
+
- **Abort propagation**: the pipeline abort signal is wired into the judge fetch call, so cancelling a pipeline also cancels any in-flight judge request.
|
|
85
|
+
- **Determinism**: the call uses `temperature: 0` to keep verdicts as stable as the model allows.
|
|
86
|
+
|
|
87
|
+
## Alternative endpoints
|
|
88
|
+
|
|
89
|
+
Any OpenAI-compatible endpoint works — just point `endpoint` and `api_key_env` at it:
|
|
90
|
+
|
|
91
|
+
- **Local Ollama** (default): `http://localhost:11434/v1/chat/completions`, no API key
|
|
92
|
+
- **OpenAI**: `https://api.openai.com/v1/chat/completions`, `api_key_env: OPENAI_API_KEY`
|
|
93
|
+
- **Local models** via LM Studio, vLLM, llama.cpp OpenAI-compatible servers
|
|
94
|
+
- **Hosted**: Groq, Together, Fireworks, OpenRouter, DeepInfra, etc.
|
|
95
|
+
- **Anthropic** via an OpenAI-compat proxy (e.g. `anthropic-openai-proxy`)
|
|
96
|
+
|
|
97
|
+
## License
|
|
98
|
+
|
|
99
|
+
MIT
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tagma/completion-llm-judge",
|
|
3
|
-
"version": "0.1
|
|
3
|
+
"version": "0.2.1",
|
|
4
4
|
"description": "LLM-as-judge completion plugin for tagma-sdk pipelines",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -49,7 +49,7 @@
|
|
|
49
49
|
"prepublishOnly": "bun run build"
|
|
50
50
|
},
|
|
51
51
|
"peerDependencies": {
|
|
52
|
-
"@tagma/types": "0.2
|
|
52
|
+
"@tagma/types": "0.4.2"
|
|
53
53
|
},
|
|
54
54
|
"devDependencies": {
|
|
55
55
|
"bun-types": "^1.3.11",
|
package/src/index.ts
CHANGED
|
@@ -1,288 +1,288 @@
|
|
|
1
|
-
// ═══ LLM-as-Judge Completion Plugin ═══
|
|
2
|
-
//
|
|
3
|
-
// Uses an OpenAI-compatible chat completions endpoint to verify whether a
|
|
4
|
-
// task's output satisfies a rubric. Complements the deterministic built-in
|
|
5
|
-
// completions (`exit_code`, `file_exists`, `output_check`) with AI-powered
|
|
6
|
-
// checks — useful when success is defined semantically rather than by a
|
|
7
|
-
// grep-able pattern or a file on disk.
|
|
8
|
-
//
|
|
9
|
-
// Default backend is a **local Ollama** server using its OpenAI-compatible
|
|
10
|
-
// route (`/v1/chat/completions`, available since Ollama 0.1.29+), with
|
|
11
|
-
// `qwen3:4b` as a small, cheap-to-run reasoning model. No API key is
|
|
12
|
-
// required for local Ollama; remote endpoints can set `api_key_env` to
|
|
13
|
-
// whatever header the server expects.
|
|
14
|
-
//
|
|
15
|
-
// The judge is instructed to answer PASS/FAIL on the first line. Reasoning
|
|
16
|
-
// models (qwen3, deepseek-r1, etc.) emit `<think>...</think>` blocks in
|
|
17
|
-
// their message content — we strip those before parsing, so the rubric
|
|
18
|
-
// works the same whether the judge model is a thinker or not.
|
|
19
|
-
//
|
|
20
|
-
// Usage in pipeline.yaml:
|
|
21
|
-
// plugins: ["@tagma/completion-llm-judge"]
|
|
22
|
-
// tracks:
|
|
23
|
-
// - tasks:
|
|
24
|
-
// - id: draft
|
|
25
|
-
// completion:
|
|
26
|
-
// type: llm_judge
|
|
27
|
-
// rubric: "Output must list at least 3 failing tests with file paths."
|
|
28
|
-
// # endpoint / model / api_key_env all default to local Ollama + qwen3:4b
|
|
29
|
-
|
|
30
|
-
import type { CompletionPlugin, CompletionContext, TaskResult } from '@tagma/types';
|
|
31
|
-
|
|
32
|
-
// Ollama exposes an OpenAI-compatible `/v1/chat/completions` route on port
|
|
33
|
-
// 11434 by default. Point this at any OpenAI-compatible server (OpenAI,
|
|
34
|
-
// vLLM, llama.cpp, LM Studio, Groq, Together, etc.) to swap backends.
|
|
35
|
-
const DEFAULT_ENDPOINT = 'http://localhost:11434/v1/chat/completions';
|
|
36
|
-
// qwen3:4b is a small reasoning model (~2.5 GB on disk, runs on CPU) that
|
|
37
|
-
// reliably follows the PASS/FAIL-on-first-line instruction. Swap to
|
|
38
|
-
// `qwen3:8b`, `deepseek-r1:7b`, or a hosted model for stricter judging.
|
|
39
|
-
const DEFAULT_MODEL = 'qwen3:4b';
|
|
40
|
-
const DEFAULT_TIMEOUT_MS = 120_000;
|
|
41
|
-
const DEFAULT_MAX_OUTPUT_CHARS = 8_000;
|
|
42
|
-
|
|
43
|
-
const SYSTEM_PROMPT =
|
|
44
|
-
'You are a strict quality judge for task outputs.\n' +
|
|
45
|
-
'Given the task rubric and actual output, answer on the FIRST LINE with exactly "PASS" or "FAIL".\n' +
|
|
46
|
-
'On subsequent lines you may provide a one-sentence justification.\n' +
|
|
47
|
-
'Do not use any other format. Do not wrap the answer in code fences.\n' +
|
|
48
|
-
'If you reason step-by-step internally, still put PASS or FAIL on the first line of your final answer.';
|
|
49
|
-
|
|
50
|
-
interface ChatMessage {
|
|
51
|
-
readonly role: 'system' | 'user';
|
|
52
|
-
readonly content: string;
|
|
53
|
-
}
|
|
54
|
-
|
|
55
|
-
interface ChatCompletionResponse {
|
|
56
|
-
readonly choices?: ReadonlyArray<{
|
|
57
|
-
readonly message?: { readonly content?: string };
|
|
58
|
-
}>;
|
|
59
|
-
}
|
|
60
|
-
|
|
61
|
-
function parseDurationSafe(raw: unknown, fallback: number): number {
|
|
62
|
-
if (raw == null) return fallback;
|
|
63
|
-
const str = String(raw).trim();
|
|
64
|
-
const m = str.match(/^(\d+(?:\.\d+)?)(ms|s|m|h)?$/);
|
|
65
|
-
if (!m) return fallback;
|
|
66
|
-
const n = Number(m[1]);
|
|
67
|
-
switch (m[2]) {
|
|
68
|
-
case 'ms':
|
|
69
|
-
return n;
|
|
70
|
-
case 'm':
|
|
71
|
-
return n * 60_000;
|
|
72
|
-
case 'h':
|
|
73
|
-
return n * 3_600_000;
|
|
74
|
-
case 's':
|
|
75
|
-
default:
|
|
76
|
-
return n * 1000;
|
|
77
|
-
}
|
|
78
|
-
}
|
|
79
|
-
|
|
80
|
-
// Head-and-tail truncation preserves the start of the output (where agents
|
|
81
|
-
// usually declare intent) and the end (where they summarize results),
|
|
82
|
-
// dropping the middle when the combined length exceeds the budget. This
|
|
83
|
-
// keeps the judge's view of the output meaningful even for very long runs.
|
|
84
|
-
function truncateForJudge(text: string, maxChars: number): string {
|
|
85
|
-
if (text.length <= maxChars) return text;
|
|
86
|
-
const marker = '\n...[truncated]...\n';
|
|
87
|
-
const budget = maxChars - marker.length;
|
|
88
|
-
if (budget <= 0) return text.slice(0, maxChars);
|
|
89
|
-
const head = Math.floor(budget * 0.7);
|
|
90
|
-
const tail = budget - head;
|
|
91
|
-
return text.slice(0, head) + marker + text.slice(-tail);
|
|
92
|
-
}
|
|
93
|
-
|
|
94
|
-
// Strip reasoning-model thinking blocks so verdict parsing sees the real
|
|
95
|
-
// answer. Qwen3 and DeepSeek-R1 both emit `<think>...</think>` inline in
|
|
96
|
-
// message content when served via Ollama's OpenAI-compat route. We also
|
|
97
|
-
// drop the legacy `<thinking>` variant some fine-tunes use. Applied
|
|
98
|
-
// before any trimming so leading whitespace from the stripped block
|
|
99
|
-
// doesn't leak into the first line.
|
|
100
|
-
function stripThinking(content: string): string {
|
|
101
|
-
return content
|
|
102
|
-
.replace(/<think>[\s\S]*?<\/think>/gi, '')
|
|
103
|
-
.replace(/<thinking>[\s\S]*?<\/thinking>/gi, '')
|
|
104
|
-
.trim();
|
|
105
|
-
}
|
|
106
|
-
|
|
107
|
-
async function callJudge(
|
|
108
|
-
endpoint: string,
|
|
109
|
-
model: string,
|
|
110
|
-
apiKey: string | undefined,
|
|
111
|
-
messages: readonly ChatMessage[],
|
|
112
|
-
timeoutMs: number,
|
|
113
|
-
externalSignal: AbortSignal | undefined,
|
|
114
|
-
): Promise<string> {
|
|
115
|
-
const controller = new AbortController();
|
|
116
|
-
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
|
117
|
-
|
|
118
|
-
const onExternalAbort = (): void => controller.abort();
|
|
119
|
-
if (externalSignal) {
|
|
120
|
-
if (externalSignal.aborted) {
|
|
121
|
-
controller.abort();
|
|
122
|
-
} else {
|
|
123
|
-
externalSignal.addEventListener('abort', onExternalAbort, { once: true });
|
|
124
|
-
}
|
|
125
|
-
}
|
|
126
|
-
|
|
127
|
-
try {
|
|
128
|
-
const headers: Record<string, string> = {
|
|
129
|
-
'content-type': 'application/json',
|
|
130
|
-
};
|
|
131
|
-
// Only send Authorization when we actually have a key — local Ollama
|
|
132
|
-
// doesn't require one, and some OpenAI-compat proxies reject bogus
|
|
133
|
-
// placeholder tokens like "ollama" with 401 instead of ignoring them.
|
|
134
|
-
if (apiKey) {
|
|
135
|
-
headers.authorization = `Bearer ${apiKey}`;
|
|
136
|
-
}
|
|
137
|
-
|
|
138
|
-
const res = await fetch(endpoint, {
|
|
139
|
-
method: 'POST',
|
|
140
|
-
headers,
|
|
141
|
-
body: JSON.stringify({
|
|
142
|
-
model,
|
|
143
|
-
messages,
|
|
144
|
-
temperature: 0,
|
|
145
|
-
// `stream: false` is already the default but we set it explicitly
|
|
146
|
-
// because Ollama's OpenAI-compat route streams by default in some
|
|
147
|
-
// older versions.
|
|
148
|
-
stream: false,
|
|
149
|
-
}),
|
|
150
|
-
signal: controller.signal,
|
|
151
|
-
});
|
|
152
|
-
if (!res.ok) {
|
|
153
|
-
const text = await res.text().catch(() => '');
|
|
154
|
-
throw new Error(`judge endpoint ${res.status}: ${text.slice(0, 200)}`);
|
|
155
|
-
}
|
|
156
|
-
const payload = (await res.json()) as ChatCompletionResponse;
|
|
157
|
-
const content = payload.choices?.[0]?.message?.content;
|
|
158
|
-
if (typeof content !== 'string') {
|
|
159
|
-
throw new Error('judge endpoint returned no message content');
|
|
160
|
-
}
|
|
161
|
-
return stripThinking(content);
|
|
162
|
-
} finally {
|
|
163
|
-
clearTimeout(timer);
|
|
164
|
-
if (externalSignal) externalSignal.removeEventListener('abort', onExternalAbort);
|
|
165
|
-
}
|
|
166
|
-
}
|
|
167
|
-
|
|
168
|
-
const LlmJudgeCompletion: CompletionPlugin = {
|
|
169
|
-
name: 'llm_judge',
|
|
170
|
-
schema: {
|
|
171
|
-
description:
|
|
172
|
-
'Use an LLM to judge whether the task output satisfies a rubric. Answers PASS/FAIL.',
|
|
173
|
-
fields: {
|
|
174
|
-
rubric: {
|
|
175
|
-
type: 'string',
|
|
176
|
-
required: true,
|
|
177
|
-
description: 'Criteria the judge should verify. Plain language.',
|
|
178
|
-
placeholder: 'Output must list at least 3 failing tests with file paths.',
|
|
179
|
-
},
|
|
180
|
-
model: {
|
|
181
|
-
type: 'string',
|
|
182
|
-
default: DEFAULT_MODEL,
|
|
183
|
-
description:
|
|
184
|
-
'Judge model name. Default is a small Ollama reasoning model (qwen3:4b). Swap for qwen3:8b, deepseek-r1:7b, or a hosted model for stricter judging.',
|
|
185
|
-
placeholder: DEFAULT_MODEL,
|
|
186
|
-
},
|
|
187
|
-
endpoint: {
|
|
188
|
-
type: 'string',
|
|
189
|
-
default: DEFAULT_ENDPOINT,
|
|
190
|
-
description:
|
|
191
|
-
'OpenAI-compatible chat completions endpoint. Defaults to local Ollama (http://localhost:11434/v1/chat/completions).',
|
|
192
|
-
placeholder: DEFAULT_ENDPOINT,
|
|
193
|
-
},
|
|
194
|
-
api_key_env: {
|
|
195
|
-
type: 'string',
|
|
196
|
-
description:
|
|
197
|
-
'Env var containing the bearer token for the judge endpoint. Leave unset for local Ollama; set to OPENAI_API_KEY etc. for hosted backends.',
|
|
198
|
-
placeholder: 'OPENAI_API_KEY',
|
|
199
|
-
},
|
|
200
|
-
timeout: {
|
|
201
|
-
type: 'duration',
|
|
202
|
-
default: '120s',
|
|
203
|
-
description:
|
|
204
|
-
'Maximum time to wait for the judge response. Reasoning models need more time than chat models.',
|
|
205
|
-
},
|
|
206
|
-
max_output_chars: {
|
|
207
|
-
type: 'number',
|
|
208
|
-
default: DEFAULT_MAX_OUTPUT_CHARS,
|
|
209
|
-
min: 500,
|
|
210
|
-
max: 200_000,
|
|
211
|
-
description: 'Truncate task stdout to this many chars before judging.',
|
|
212
|
-
},
|
|
213
|
-
},
|
|
214
|
-
},
|
|
215
|
-
|
|
216
|
-
async check(
|
|
217
|
-
config: Record<string, unknown>,
|
|
218
|
-
result: TaskResult,
|
|
219
|
-
ctx: CompletionContext,
|
|
220
|
-
): Promise<boolean> {
|
|
221
|
-
const rubric = config.rubric as string | undefined;
|
|
222
|
-
if (!rubric) throw new Error('llm_judge completion: "rubric" is required');
|
|
223
|
-
|
|
224
|
-
// api_key_env is optional — when unset we talk to the endpoint
|
|
225
|
-
// anonymously (correct for local Ollama). When the user names an env
|
|
226
|
-
// var, we require it to be populated so config errors fail loudly
|
|
227
|
-
// instead of silently stripping auth.
|
|
228
|
-
const apiKeyEnv = config.api_key_env as string | undefined;
|
|
229
|
-
let apiKey: string | undefined;
|
|
230
|
-
if (apiKeyEnv) {
|
|
231
|
-
apiKey = process.env[apiKeyEnv];
|
|
232
|
-
if (!apiKey) {
|
|
233
|
-
throw new Error(`llm_judge completion: env var ${apiKeyEnv} is not set`);
|
|
234
|
-
}
|
|
235
|
-
}
|
|
236
|
-
|
|
237
|
-
const model = (config.model as string | undefined) ?? DEFAULT_MODEL;
|
|
238
|
-
const endpoint = (config.endpoint as string | undefined) ?? DEFAULT_ENDPOINT;
|
|
239
|
-
const timeoutMs = parseDurationSafe(config.timeout, DEFAULT_TIMEOUT_MS);
|
|
240
|
-
const maxChars =
|
|
241
|
-
typeof config.max_output_chars === 'number' && config.max_output_chars > 0
|
|
242
|
-
? Math.floor(config.max_output_chars)
|
|
243
|
-
: DEFAULT_MAX_OUTPUT_CHARS;
|
|
244
|
-
|
|
245
|
-
// Prefer the driver-normalized text (e.g. concatenated message text
|
|
246
|
-
// from AI drivers that emit NDJSON). Feeding raw NDJSON to the judge
|
|
247
|
-
// wastes tokens and obscures the semantic output the judge is meant
|
|
248
|
-
// to grade. Command tasks and drivers without parseResult fall back
|
|
249
|
-
// to raw stdout, which for them IS the semantic output.
|
|
250
|
-
const taskOutput = result.normalizedOutput ?? result.stdout;
|
|
251
|
-
|
|
252
|
-
const userContent =
|
|
253
|
-
`[Rubric]\n${rubric}\n\n` +
|
|
254
|
-
`[Exit Code]\n${result.exitCode}\n\n` +
|
|
255
|
-
`[Task Output]\n${truncateForJudge(taskOutput, maxChars)}`;
|
|
256
|
-
|
|
257
|
-
const messages: ChatMessage[] = [
|
|
258
|
-
{ role: 'system', content: SYSTEM_PROMPT },
|
|
259
|
-
{ role: 'user', content: userContent },
|
|
260
|
-
];
|
|
261
|
-
|
|
262
|
-
try {
|
|
263
|
-
const content = await callJudge(endpoint, model, apiKey, messages, timeoutMs, ctx.signal);
|
|
264
|
-
const firstLine = (content.split(/\r?\n/, 1)[0] ?? '').trim().toUpperCase();
|
|
265
|
-
const passed = firstLine.startsWith('PASS');
|
|
266
|
-
if (!passed) {
|
|
267
|
-
// Surface the judge's reasoning in logs so pipeline operators can
|
|
268
|
-
// see why an output was rejected without re-running it themselves.
|
|
269
|
-
console.warn(
|
|
270
|
-
`[llm_judge] verdict=${firstLine || '<empty>'} — full judge response:\n${content}`,
|
|
271
|
-
);
|
|
272
|
-
}
|
|
273
|
-
return passed;
|
|
274
|
-
} catch (err) {
|
|
275
|
-
// Treat judge failures as FAIL: a completion gate that errors open
|
|
276
|
-
// is worse than one that errors closed. Operators can re-run the
|
|
277
|
-
// task once the judge endpoint is healthy again.
|
|
278
|
-
const msg = err instanceof Error ? err.message : String(err);
|
|
279
|
-
console.warn(`[llm_judge] judge call failed, marking task as not-complete: ${msg}`);
|
|
280
|
-
return false;
|
|
281
|
-
}
|
|
282
|
-
},
|
|
283
|
-
};
|
|
284
|
-
|
|
285
|
-
// ═══ Plugin self-description exports ═══
|
|
286
|
-
export const pluginCategory = 'completions';
|
|
287
|
-
export const pluginType = 'llm_judge';
|
|
288
|
-
export default LlmJudgeCompletion;
|
|
1
|
+
// ═══ LLM-as-Judge Completion Plugin ═══
|
|
2
|
+
//
|
|
3
|
+
// Uses an OpenAI-compatible chat completions endpoint to verify whether a
|
|
4
|
+
// task's output satisfies a rubric. Complements the deterministic built-in
|
|
5
|
+
// completions (`exit_code`, `file_exists`, `output_check`) with AI-powered
|
|
6
|
+
// checks — useful when success is defined semantically rather than by a
|
|
7
|
+
// grep-able pattern or a file on disk.
|
|
8
|
+
//
|
|
9
|
+
// Default backend is a **local Ollama** server using its OpenAI-compatible
|
|
10
|
+
// route (`/v1/chat/completions`, available since Ollama 0.1.29+), with
|
|
11
|
+
// `qwen3:4b` as a small, cheap-to-run reasoning model. No API key is
|
|
12
|
+
// required for local Ollama; remote endpoints can set `api_key_env` to
|
|
13
|
+
// whatever header the server expects.
|
|
14
|
+
//
|
|
15
|
+
// The judge is instructed to answer PASS/FAIL on the first line. Reasoning
|
|
16
|
+
// models (qwen3, deepseek-r1, etc.) emit `<think>...</think>` blocks in
|
|
17
|
+
// their message content — we strip those before parsing, so the rubric
|
|
18
|
+
// works the same whether the judge model is a thinker or not.
|
|
19
|
+
//
|
|
20
|
+
// Usage in pipeline.yaml:
|
|
21
|
+
// plugins: ["@tagma/completion-llm-judge"]
|
|
22
|
+
// tracks:
|
|
23
|
+
// - tasks:
|
|
24
|
+
// - id: draft
|
|
25
|
+
// completion:
|
|
26
|
+
// type: llm_judge
|
|
27
|
+
// rubric: "Output must list at least 3 failing tests with file paths."
|
|
28
|
+
// # endpoint / model / api_key_env all default to local Ollama + qwen3:4b
|
|
29
|
+
|
|
30
|
+
import type { CompletionPlugin, CompletionContext, TaskResult } from '@tagma/types';
|
|
31
|
+
|
|
32
|
+
// Ollama exposes an OpenAI-compatible `/v1/chat/completions` route on port
|
|
33
|
+
// 11434 by default. Point this at any OpenAI-compatible server (OpenAI,
|
|
34
|
+
// vLLM, llama.cpp, LM Studio, Groq, Together, etc.) to swap backends.
|
|
35
|
+
const DEFAULT_ENDPOINT = 'http://localhost:11434/v1/chat/completions';
|
|
36
|
+
// qwen3:4b is a small reasoning model (~2.5 GB on disk, runs on CPU) that
|
|
37
|
+
// reliably follows the PASS/FAIL-on-first-line instruction. Swap to
|
|
38
|
+
// `qwen3:8b`, `deepseek-r1:7b`, or a hosted model for stricter judging.
|
|
39
|
+
const DEFAULT_MODEL = 'qwen3:4b';
|
|
40
|
+
const DEFAULT_TIMEOUT_MS = 120_000;
|
|
41
|
+
const DEFAULT_MAX_OUTPUT_CHARS = 8_000;
|
|
42
|
+
|
|
43
|
+
const SYSTEM_PROMPT =
|
|
44
|
+
'You are a strict quality judge for task outputs.\n' +
|
|
45
|
+
'Given the task rubric and actual output, answer on the FIRST LINE with exactly "PASS" or "FAIL".\n' +
|
|
46
|
+
'On subsequent lines you may provide a one-sentence justification.\n' +
|
|
47
|
+
'Do not use any other format. Do not wrap the answer in code fences.\n' +
|
|
48
|
+
'If you reason step-by-step internally, still put PASS or FAIL on the first line of your final answer.';
|
|
49
|
+
|
|
50
|
+
interface ChatMessage {
|
|
51
|
+
readonly role: 'system' | 'user';
|
|
52
|
+
readonly content: string;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
interface ChatCompletionResponse {
|
|
56
|
+
readonly choices?: ReadonlyArray<{
|
|
57
|
+
readonly message?: { readonly content?: string };
|
|
58
|
+
}>;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
function parseDurationSafe(raw: unknown, fallback: number): number {
|
|
62
|
+
if (raw == null) return fallback;
|
|
63
|
+
const str = String(raw).trim();
|
|
64
|
+
const m = str.match(/^(\d+(?:\.\d+)?)(ms|s|m|h)?$/);
|
|
65
|
+
if (!m) return fallback;
|
|
66
|
+
const n = Number(m[1]);
|
|
67
|
+
switch (m[2]) {
|
|
68
|
+
case 'ms':
|
|
69
|
+
return n;
|
|
70
|
+
case 'm':
|
|
71
|
+
return n * 60_000;
|
|
72
|
+
case 'h':
|
|
73
|
+
return n * 3_600_000;
|
|
74
|
+
case 's':
|
|
75
|
+
default:
|
|
76
|
+
return n * 1000;
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
// Head-and-tail truncation preserves the start of the output (where agents
|
|
81
|
+
// usually declare intent) and the end (where they summarize results),
|
|
82
|
+
// dropping the middle when the combined length exceeds the budget. This
|
|
83
|
+
// keeps the judge's view of the output meaningful even for very long runs.
|
|
84
|
+
function truncateForJudge(text: string, maxChars: number): string {
|
|
85
|
+
if (text.length <= maxChars) return text;
|
|
86
|
+
const marker = '\n...[truncated]...\n';
|
|
87
|
+
const budget = maxChars - marker.length;
|
|
88
|
+
if (budget <= 0) return text.slice(0, maxChars);
|
|
89
|
+
const head = Math.floor(budget * 0.7);
|
|
90
|
+
const tail = budget - head;
|
|
91
|
+
return text.slice(0, head) + marker + text.slice(-tail);
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
// Strip reasoning-model thinking blocks so verdict parsing sees the real
|
|
95
|
+
// answer. Qwen3 and DeepSeek-R1 both emit `<think>...</think>` inline in
|
|
96
|
+
// message content when served via Ollama's OpenAI-compat route. We also
|
|
97
|
+
// drop the legacy `<thinking>` variant some fine-tunes use. Applied
|
|
98
|
+
// before any trimming so leading whitespace from the stripped block
|
|
99
|
+
// doesn't leak into the first line.
|
|
100
|
+
function stripThinking(content: string): string {
|
|
101
|
+
return content
|
|
102
|
+
.replace(/<think>[\s\S]*?<\/think>/gi, '')
|
|
103
|
+
.replace(/<thinking>[\s\S]*?<\/thinking>/gi, '')
|
|
104
|
+
.trim();
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
async function callJudge(
|
|
108
|
+
endpoint: string,
|
|
109
|
+
model: string,
|
|
110
|
+
apiKey: string | undefined,
|
|
111
|
+
messages: readonly ChatMessage[],
|
|
112
|
+
timeoutMs: number,
|
|
113
|
+
externalSignal: AbortSignal | undefined,
|
|
114
|
+
): Promise<string> {
|
|
115
|
+
const controller = new AbortController();
|
|
116
|
+
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
|
117
|
+
|
|
118
|
+
const onExternalAbort = (): void => controller.abort();
|
|
119
|
+
if (externalSignal) {
|
|
120
|
+
if (externalSignal.aborted) {
|
|
121
|
+
controller.abort();
|
|
122
|
+
} else {
|
|
123
|
+
externalSignal.addEventListener('abort', onExternalAbort, { once: true });
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
try {
|
|
128
|
+
const headers: Record<string, string> = {
|
|
129
|
+
'content-type': 'application/json',
|
|
130
|
+
};
|
|
131
|
+
// Only send Authorization when we actually have a key — local Ollama
|
|
132
|
+
// doesn't require one, and some OpenAI-compat proxies reject bogus
|
|
133
|
+
// placeholder tokens like "ollama" with 401 instead of ignoring them.
|
|
134
|
+
if (apiKey) {
|
|
135
|
+
headers.authorization = `Bearer ${apiKey}`;
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
const res = await fetch(endpoint, {
|
|
139
|
+
method: 'POST',
|
|
140
|
+
headers,
|
|
141
|
+
body: JSON.stringify({
|
|
142
|
+
model,
|
|
143
|
+
messages,
|
|
144
|
+
temperature: 0,
|
|
145
|
+
// `stream: false` is already the default but we set it explicitly
|
|
146
|
+
// because Ollama's OpenAI-compat route streams by default in some
|
|
147
|
+
// older versions.
|
|
148
|
+
stream: false,
|
|
149
|
+
}),
|
|
150
|
+
signal: controller.signal,
|
|
151
|
+
});
|
|
152
|
+
if (!res.ok) {
|
|
153
|
+
const text = await res.text().catch(() => '');
|
|
154
|
+
throw new Error(`judge endpoint ${res.status}: ${text.slice(0, 200)}`);
|
|
155
|
+
}
|
|
156
|
+
const payload = (await res.json()) as ChatCompletionResponse;
|
|
157
|
+
const content = payload.choices?.[0]?.message?.content;
|
|
158
|
+
if (typeof content !== 'string') {
|
|
159
|
+
throw new Error('judge endpoint returned no message content');
|
|
160
|
+
}
|
|
161
|
+
return stripThinking(content);
|
|
162
|
+
} finally {
|
|
163
|
+
clearTimeout(timer);
|
|
164
|
+
if (externalSignal) externalSignal.removeEventListener('abort', onExternalAbort);
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
|
|
168
|
+
const LlmJudgeCompletion: CompletionPlugin = {
|
|
169
|
+
name: 'llm_judge',
|
|
170
|
+
schema: {
|
|
171
|
+
description:
|
|
172
|
+
'Use an LLM to judge whether the task output satisfies a rubric. Answers PASS/FAIL.',
|
|
173
|
+
fields: {
|
|
174
|
+
rubric: {
|
|
175
|
+
type: 'string',
|
|
176
|
+
required: true,
|
|
177
|
+
description: 'Criteria the judge should verify. Plain language.',
|
|
178
|
+
placeholder: 'Output must list at least 3 failing tests with file paths.',
|
|
179
|
+
},
|
|
180
|
+
model: {
|
|
181
|
+
type: 'string',
|
|
182
|
+
default: DEFAULT_MODEL,
|
|
183
|
+
description:
|
|
184
|
+
'Judge model name. Default is a small Ollama reasoning model (qwen3:4b). Swap for qwen3:8b, deepseek-r1:7b, or a hosted model for stricter judging.',
|
|
185
|
+
placeholder: DEFAULT_MODEL,
|
|
186
|
+
},
|
|
187
|
+
endpoint: {
|
|
188
|
+
type: 'string',
|
|
189
|
+
default: DEFAULT_ENDPOINT,
|
|
190
|
+
description:
|
|
191
|
+
'OpenAI-compatible chat completions endpoint. Defaults to local Ollama (http://localhost:11434/v1/chat/completions).',
|
|
192
|
+
placeholder: DEFAULT_ENDPOINT,
|
|
193
|
+
},
|
|
194
|
+
api_key_env: {
|
|
195
|
+
type: 'string',
|
|
196
|
+
description:
|
|
197
|
+
'Env var containing the bearer token for the judge endpoint. Leave unset for local Ollama; set to OPENAI_API_KEY etc. for hosted backends.',
|
|
198
|
+
placeholder: 'OPENAI_API_KEY',
|
|
199
|
+
},
|
|
200
|
+
timeout: {
|
|
201
|
+
type: 'duration',
|
|
202
|
+
default: '120s',
|
|
203
|
+
description:
|
|
204
|
+
'Maximum time to wait for the judge response. Reasoning models need more time than chat models.',
|
|
205
|
+
},
|
|
206
|
+
max_output_chars: {
|
|
207
|
+
type: 'number',
|
|
208
|
+
default: DEFAULT_MAX_OUTPUT_CHARS,
|
|
209
|
+
min: 500,
|
|
210
|
+
max: 200_000,
|
|
211
|
+
description: 'Truncate task stdout to this many chars before judging.',
|
|
212
|
+
},
|
|
213
|
+
},
|
|
214
|
+
},
|
|
215
|
+
|
|
216
|
+
async check(
|
|
217
|
+
config: Record<string, unknown>,
|
|
218
|
+
result: TaskResult,
|
|
219
|
+
ctx: CompletionContext,
|
|
220
|
+
): Promise<boolean> {
|
|
221
|
+
const rubric = config.rubric as string | undefined;
|
|
222
|
+
if (!rubric) throw new Error('llm_judge completion: "rubric" is required');
|
|
223
|
+
|
|
224
|
+
// api_key_env is optional — when unset we talk to the endpoint
|
|
225
|
+
// anonymously (correct for local Ollama). When the user names an env
|
|
226
|
+
// var, we require it to be populated so config errors fail loudly
|
|
227
|
+
// instead of silently stripping auth.
|
|
228
|
+
const apiKeyEnv = config.api_key_env as string | undefined;
|
|
229
|
+
let apiKey: string | undefined;
|
|
230
|
+
if (apiKeyEnv) {
|
|
231
|
+
apiKey = process.env[apiKeyEnv];
|
|
232
|
+
if (!apiKey) {
|
|
233
|
+
throw new Error(`llm_judge completion: env var ${apiKeyEnv} is not set`);
|
|
234
|
+
}
|
|
235
|
+
}
|
|
236
|
+
|
|
237
|
+
const model = (config.model as string | undefined) ?? DEFAULT_MODEL;
|
|
238
|
+
const endpoint = (config.endpoint as string | undefined) ?? DEFAULT_ENDPOINT;
|
|
239
|
+
const timeoutMs = parseDurationSafe(config.timeout, DEFAULT_TIMEOUT_MS);
|
|
240
|
+
const maxChars =
|
|
241
|
+
typeof config.max_output_chars === 'number' && config.max_output_chars > 0
|
|
242
|
+
? Math.floor(config.max_output_chars)
|
|
243
|
+
: DEFAULT_MAX_OUTPUT_CHARS;
|
|
244
|
+
|
|
245
|
+
// Prefer the driver-normalized text (e.g. concatenated message text
|
|
246
|
+
// from AI drivers that emit NDJSON). Feeding raw NDJSON to the judge
|
|
247
|
+
// wastes tokens and obscures the semantic output the judge is meant
|
|
248
|
+
// to grade. Command tasks and drivers without parseResult fall back
|
|
249
|
+
// to raw stdout, which for them IS the semantic output.
|
|
250
|
+
const taskOutput = result.normalizedOutput ?? result.stdout;
|
|
251
|
+
|
|
252
|
+
const userContent =
|
|
253
|
+
`[Rubric]\n${rubric}\n\n` +
|
|
254
|
+
`[Exit Code]\n${result.exitCode}\n\n` +
|
|
255
|
+
`[Task Output]\n${truncateForJudge(taskOutput, maxChars)}`;
|
|
256
|
+
|
|
257
|
+
const messages: ChatMessage[] = [
|
|
258
|
+
{ role: 'system', content: SYSTEM_PROMPT },
|
|
259
|
+
{ role: 'user', content: userContent },
|
|
260
|
+
];
|
|
261
|
+
|
|
262
|
+
try {
|
|
263
|
+
const content = await callJudge(endpoint, model, apiKey, messages, timeoutMs, ctx.signal);
|
|
264
|
+
const firstLine = (content.split(/\r?\n/, 1)[0] ?? '').trim().toUpperCase();
|
|
265
|
+
const passed = firstLine.startsWith('PASS');
|
|
266
|
+
if (!passed) {
|
|
267
|
+
// Surface the judge's reasoning in logs so pipeline operators can
|
|
268
|
+
// see why an output was rejected without re-running it themselves.
|
|
269
|
+
console.warn(
|
|
270
|
+
`[llm_judge] verdict=${firstLine || '<empty>'} — full judge response:\n${content}`,
|
|
271
|
+
);
|
|
272
|
+
}
|
|
273
|
+
return passed;
|
|
274
|
+
} catch (err) {
|
|
275
|
+
// Treat judge failures as FAIL: a completion gate that errors open
|
|
276
|
+
// is worse than one that errors closed. Operators can re-run the
|
|
277
|
+
// task once the judge endpoint is healthy again.
|
|
278
|
+
const msg = err instanceof Error ? err.message : String(err);
|
|
279
|
+
console.warn(`[llm_judge] judge call failed, marking task as not-complete: ${msg}`);
|
|
280
|
+
return false;
|
|
281
|
+
}
|
|
282
|
+
},
|
|
283
|
+
};
|
|
284
|
+
|
|
285
|
+
// ═══ Plugin self-description exports ═══
|
|
286
|
+
export const pluginCategory = 'completions';
|
|
287
|
+
export const pluginType = 'llm_judge';
|
|
288
|
+
export default LlmJudgeCompletion;
|