vskill 0.5.84 → 0.5.86
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +51 -3
- package/dist/eval/env.d.ts +18 -0
- package/dist/eval/env.js +74 -0
- package/dist/eval/env.js.map +1 -0
- package/dist/eval/llm.d.ts +1 -1
- package/dist/eval/llm.js +87 -7
- package/dist/eval/llm.js.map +1 -1
- package/dist/eval-server/api-routes.d.ts +47 -0
- package/dist/eval-server/api-routes.js +256 -21
- package/dist/eval-server/api-routes.js.map +1 -1
- package/dist/eval-server/settings-store.d.ts +39 -0
- package/dist/eval-server/settings-store.js +195 -0
- package/dist/eval-server/settings-store.js.map +1 -0
- package/dist/eval-server/skill-create-routes.d.ts +60 -0
- package/dist/eval-server/skill-create-routes.js +272 -7
- package/dist/eval-server/skill-create-routes.js.map +1 -1
- package/dist/eval-server/studio-json.d.ts +9 -0
- package/dist/eval-server/studio-json.js +82 -0
- package/dist/eval-server/studio-json.js.map +1 -0
- package/dist/eval-ui/assets/{CommandPalette-eD24lhFu.js → CommandPalette-BYCtdLfL.js} +1 -1
- package/dist/eval-ui/assets/UpdateDropdown-E99dDGz-.js +1 -0
- package/dist/eval-ui/assets/index-C7orlG3B.css +1 -0
- package/dist/eval-ui/assets/index-CrHFymNR.js +79 -0
- package/dist/eval-ui/index.html +2 -2
- package/package.json +2 -1
- package/dist/eval-ui/assets/index-Bi2n8mTB.js +0 -65
- package/dist/eval-ui/assets/index-EnxOcJMr.css +0 -1
package/README.md
CHANGED
|
@@ -352,15 +352,22 @@ npx vskill eval generate-all # Batch-generate for all skills
|
|
|
352
352
|
|
|
353
353
|
Previous benchmark results are displayed on the skill detail page without re-running. Per-case pass/fail status, time, and token usage are shown inline.
|
|
354
354
|
|
|
355
|
+
## Claude Max/Pro subscription compliance
|
|
356
|
+
|
|
357
|
+
vSkill Studio does not consume your Max/Pro subscription quota directly. It delegates to the official [Claude Code CLI](https://docs.claude.com/en/docs/claude-code), the sanctioned consumer per Anthropic's April 2026 Terms of Service update. The Claude adapter never reads `~/.claude/credentials*`, `~/.claude/auth*`, or `~/.claude/token*` — a bundled unit test (`src/eval/__tests__/claude-cli-compliance.test.ts`) plus a dist-bundle grep gate (`scripts/check-bundle-compliance.sh`) enforce this on every build.
|
|
358
|
+
|
|
359
|
+
Issue API keys for direct access at [platform.claude.com/settings/keys](https://platform.claude.com/settings/keys) or [openrouter.ai/settings/keys](https://openrouter.ai/settings/keys). Keys entered in Studio's Settings modal are stored locally on-device only (in-memory + browser localStorage mirror, or macOS Keychain on opt-in) — never synced, never committed to git, never transmitted off-device except to the provider's own API.
|
|
360
|
+
|
|
355
361
|
### Model configuration
|
|
356
362
|
|
|
357
363
|
The eval system supports multiple LLM providers. Switch between them in the eval UI dropdown or via environment variables.
|
|
358
364
|
|
|
359
365
|
| Provider | Models | Requirements |
|
|
360
366
|
|:---------|:-------|:-------------|
|
|
361
|
-
| **Claude CLI** | Sonnet, Opus, Haiku |
|
|
367
|
+
| **Claude Code (CLI)** | Sonnet, Opus, Haiku | `@anthropic-ai/claude-code` installed on PATH — Studio delegates to your existing Claude Code session. |
|
|
362
368
|
| **Anthropic API** | Claude Sonnet 4.6, Opus 4.6, Haiku 4.5 | `ANTHROPIC_API_KEY` env var |
|
|
363
369
|
| **Ollama** | Any locally installed model | Ollama running at `localhost:11434` |
|
|
370
|
+
| **LM Studio** | Any model loaded in LM Studio | LM Studio running at `localhost:1234` (no API key needed) |
|
|
364
371
|
|
|
365
372
|
```bash
|
|
366
373
|
# Use Anthropic API with Opus
|
|
@@ -369,8 +376,16 @@ VSKILL_EVAL_PROVIDER=anthropic VSKILL_EVAL_MODEL=claude-opus-4-6 npx vskill eval
|
|
|
369
376
|
# Use Ollama with a local model
|
|
370
377
|
VSKILL_EVAL_PROVIDER=ollama VSKILL_EVAL_MODEL=qwen2.5:32b npx vskill eval run my-skill
|
|
371
378
|
|
|
372
|
-
# Custom Ollama server
|
|
373
|
-
OLLAMA_BASE_URL
|
|
379
|
+
# Custom Ollama server — OLLAMA_HOST is the primary env var (matches Ollama's own docs).
|
|
380
|
+
# OLLAMA_BASE_URL is preserved for backcompat but deprecated; Studio logs a one-shot
|
|
381
|
+
# warning if both are set.
|
|
382
|
+
OLLAMA_HOST=http://gpu-server:11434 VSKILL_EVAL_PROVIDER=ollama npx vskill eval run my-skill
|
|
383
|
+
|
|
384
|
+
# Use LM Studio with a locally loaded model (no API key required)
|
|
385
|
+
VSKILL_EVAL_PROVIDER=lm-studio VSKILL_EVAL_MODEL=qwen2.5-coder-7b npx vskill eval run my-skill
|
|
386
|
+
|
|
387
|
+
# Custom LM Studio endpoint
|
|
388
|
+
LM_STUDIO_BASE_URL=http://lan-box:1234/v1 VSKILL_EVAL_PROVIDER=lm-studio npx vskill eval run my-skill
|
|
374
389
|
```
|
|
375
390
|
|
|
376
391
|
**Which model for what?**
|
|
@@ -378,6 +393,7 @@ OLLAMA_BASE_URL=http://gpu-server:11434 VSKILL_EVAL_PROVIDER=ollama npx vskill e
|
|
|
378
393
|
- **Skill creation/improvement**: Claude (Sonnet or Opus) produces the best SKILL.md refinements. Other models like Gemini and Codex can create skills too — they understand the SKILL.md format — but output quality may vary. See Anthropic's [Skill Creator](https://github.com/anthropics/skills/tree/main/skills/skill-creator) for the reference methodology.
|
|
379
394
|
- **Benchmarks & A/B comparisons**: Use any model. Cross-model testing reveals whether your skill helps weaker models, and whether base model improvements have made a capability uplift skill unnecessary.
|
|
380
395
|
- **Ollama**: Free, local, no API key. Useful for rapid iteration and validating cross-model portability.
|
|
396
|
+
- **LM Studio**: Free, local, OpenAI-compatible server. Defaults to `http://localhost:1234/v1`; override with `LM_STUDIO_BASE_URL`. LM Studio ignores the API key, so vskill sends a dummy `Bearer lm-studio` token automatically.
|
|
381
397
|
|
|
382
398
|
### Platform integration
|
|
383
399
|
|
|
@@ -399,6 +415,38 @@ vskill info remotion-best-practices # skill details
|
|
|
399
415
|
|
|
400
416
|
<br/>
|
|
401
417
|
|
|
418
|
+
## SKILL.md Spec Compliance
|
|
419
|
+
|
|
420
|
+
vskill emits SKILL.md files that conform to the canonical specification at **[agentskills.io/specification](https://agentskills.io/specification)**.
|
|
421
|
+
|
|
422
|
+
The spec requires `tags` and `target-agents` to live **under a `metadata:` block**, not at the top level of the frontmatter:
|
|
423
|
+
|
|
424
|
+
```yaml
|
|
425
|
+
---
|
|
426
|
+
name: my-skill
|
|
427
|
+
description: "..."
|
|
428
|
+
version: 1.0.0
|
|
429
|
+
metadata:
|
|
430
|
+
tags:
|
|
431
|
+
- devtools
|
|
432
|
+
- cli
|
|
433
|
+
target-agents:
|
|
434
|
+
- claude-code
|
|
435
|
+
- cursor
|
|
436
|
+
---
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
### Validation
|
|
440
|
+
|
|
441
|
+
- **CI gate** — `npm run lint:skills-spec` walks the repo for every `SKILL.md` and blocks drift. Uses the external `skills-ref` CLI when available; otherwise falls back to a built-in check that enforces the `tags` / `target-agents` nesting rule.
|
|
442
|
+
- **Post-creation (Studio)** — the `interpretValidatorResult` / `formatValidatorReport` helpers in `src/eval-server/skill-create-routes.ts` wrap `skills-ref validate`. Warn-only by default (spec drift prints a `Validation warnings` block, skill file stays on disk); `strict: true` flips the outcome to a blocking error with exit code 1. A missing `skills-ref` binary is non-blocking by design — CI is the enforcement line.
|
|
443
|
+
|
|
444
|
+
### Migration note for downstream consumers
|
|
445
|
+
|
|
446
|
+
If you previously read `frontmatter.tags` or `frontmatter['target-agents']` at the top level, read them under `frontmatter.metadata.tags` / `frontmatter.metadata['target-agents']` instead. All emitters migrated in lockstep; there is no period where vskill emits mixed shapes.
|
|
447
|
+
|
|
448
|
+
<br/>
|
|
449
|
+
|
|
402
450
|
## Contributing
|
|
403
451
|
|
|
404
452
|
Submit your skill for verification:
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
export type WarnOnceLogger = (message: string) => void;
|
|
2
|
+
/**
|
|
3
|
+
* Emit `message` via `logger` (default: console.warn) exactly once per
|
|
4
|
+
* process. Subsequent calls with the same string are no-ops.
|
|
5
|
+
*
|
|
6
|
+
* Test-only helper `_resetWarnOnce()` below clears the dedup set so unit
|
|
7
|
+
* tests can observe the one-shot behaviour in isolation.
|
|
8
|
+
*/
|
|
9
|
+
export declare function warnOnce(message: string, logger?: WarnOnceLogger): void;
|
|
10
|
+
/** Test-only — clears the warn-once dedup set between tests. */
|
|
11
|
+
export declare function _resetWarnOnce(): void;
|
|
12
|
+
/**
|
|
13
|
+
* Resolve the Ollama base URL using the documented precedence rules.
|
|
14
|
+
*
|
|
15
|
+
* @param env - usually `process.env`; injected so tests can supply fixtures.
|
|
16
|
+
* @param logger - optional logger for the deprecation warn-once.
|
|
17
|
+
*/
|
|
18
|
+
export declare function resolveOllamaBaseUrl(env?: NodeJS.ProcessEnv, logger?: WarnOnceLogger): string;
|
package/dist/eval/env.js
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
// ---------------------------------------------------------------------------
|
|
2
|
+
// env.ts — Canonical environment-variable resolution helpers.
|
|
3
|
+
//
|
|
4
|
+
// Introduced for increment 0682 (Studio Agent + Model Picker) to:
|
|
5
|
+
// 1. Normalise Ollama base URL across `OLLAMA_HOST` (primary, matches
|
|
6
|
+
// Ollama's own docs) and the deprecated `OLLAMA_BASE_URL` (backcompat).
|
|
7
|
+
// 2. Centralise env reads so there is exactly one call site per variable;
|
|
8
|
+
// this keeps docs/ARCHITECTURE.md §5 trustworthy and makes voice-lint
|
|
9
|
+
// / compliance greps reliable.
|
|
10
|
+
// 3. Guarantee a single deprecation warning per process via `warnOnce()`
|
|
11
|
+
// so users with both vars set see one advisory line, not a storm.
|
|
12
|
+
//
|
|
13
|
+
// Precedence for `resolveOllamaBaseUrl`:
|
|
14
|
+
// (1) env.OLLAMA_HOST
|
|
15
|
+
// (2) env.OLLAMA_BASE_URL (deprecated, emits warn-once on conflict)
|
|
16
|
+
// (3) "http://localhost:11434"
|
|
17
|
+
//
|
|
18
|
+
// Bare host:port values (no scheme) get `http://` prepended. Values with
|
|
19
|
+
// any scheme (`http://`, `https://`, etc.) pass through untouched.
|
|
20
|
+
// ---------------------------------------------------------------------------
|
|
21
|
+
const DEFAULT_OLLAMA_BASE_URL = "http://localhost:11434";
|
|
22
|
+
// Module-scoped dedup set — one emission per unique message per process.
|
|
23
|
+
const warnedMessages = new Set();
|
|
24
|
+
/**
|
|
25
|
+
* Emit `message` via `logger` (default: console.warn) exactly once per
|
|
26
|
+
* process. Subsequent calls with the same string are no-ops.
|
|
27
|
+
*
|
|
28
|
+
* Test-only helper `_resetWarnOnce()` below clears the dedup set so unit
|
|
29
|
+
* tests can observe the one-shot behaviour in isolation.
|
|
30
|
+
*/
|
|
31
|
+
export function warnOnce(message, logger) {
|
|
32
|
+
if (warnedMessages.has(message))
|
|
33
|
+
return;
|
|
34
|
+
warnedMessages.add(message);
|
|
35
|
+
const write = logger ?? ((m) => console.warn(m));
|
|
36
|
+
write(message);
|
|
37
|
+
}
|
|
38
|
+
/** Test-only — clears the warn-once dedup set between tests. */
|
|
39
|
+
export function _resetWarnOnce() {
|
|
40
|
+
warnedMessages.clear();
|
|
41
|
+
}
|
|
42
|
+
/**
|
|
43
|
+
* Prepend `http://` to bare `host:port` values. Pass-through for anything
|
|
44
|
+
* that already has a scheme. Empty / whitespace values fall through to
|
|
45
|
+
* the caller's default.
|
|
46
|
+
*/
|
|
47
|
+
function ensureScheme(value) {
|
|
48
|
+
const trimmed = value.trim();
|
|
49
|
+
if (!trimmed)
|
|
50
|
+
return trimmed;
|
|
51
|
+
if (/^[a-z][a-z0-9+.-]*:\/\//i.test(trimmed))
|
|
52
|
+
return trimmed;
|
|
53
|
+
return `http://${trimmed}`;
|
|
54
|
+
}
|
|
55
|
+
/**
|
|
56
|
+
* Resolve the Ollama base URL using the documented precedence rules.
|
|
57
|
+
*
|
|
58
|
+
* @param env - usually `process.env`; injected so tests can supply fixtures.
|
|
59
|
+
* @param logger - optional logger for the deprecation warn-once.
|
|
60
|
+
*/
|
|
61
|
+
export function resolveOllamaBaseUrl(env = process.env, logger) {
|
|
62
|
+
const host = env.OLLAMA_HOST?.trim();
|
|
63
|
+
const baseUrl = env.OLLAMA_BASE_URL?.trim();
|
|
64
|
+
// Both set with different values → emit deprecation warning exactly once.
|
|
65
|
+
if (host && baseUrl && host !== baseUrl) {
|
|
66
|
+
warnOnce(`Both OLLAMA_HOST and OLLAMA_BASE_URL are set. Using OLLAMA_HOST (${host}). OLLAMA_BASE_URL is deprecated.`, logger);
|
|
67
|
+
}
|
|
68
|
+
if (host)
|
|
69
|
+
return ensureScheme(host);
|
|
70
|
+
if (baseUrl)
|
|
71
|
+
return ensureScheme(baseUrl);
|
|
72
|
+
return DEFAULT_OLLAMA_BASE_URL;
|
|
73
|
+
}
|
|
74
|
+
//# sourceMappingURL=env.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"env.js","sourceRoot":"","sources":["../../src/eval/env.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,8DAA8D;AAC9D,EAAE;AACF,kEAAkE;AAClE,wEAAwE;AACxE,6EAA6E;AAC7E,4EAA4E;AAC5E,2EAA2E;AAC3E,oCAAoC;AACpC,2EAA2E;AAC3E,uEAAuE;AACvE,EAAE;AACF,yCAAyC;AACzC,wBAAwB;AACxB,uEAAuE;AACvE,iCAAiC;AACjC,EAAE;AACF,yEAAyE;AACzE,mEAAmE;AACnE,8EAA8E;AAE9E,MAAM,uBAAuB,GAAG,wBAAwB,CAAC;AAEzD,yEAAyE;AACzE,MAAM,cAAc,GAAG,IAAI,GAAG,EAAU,CAAC;AAIzC;;;;;;GAMG;AACH,MAAM,UAAU,QAAQ,CAAC,OAAe,EAAE,MAAuB;IAC/D,IAAI,cAAc,CAAC,GAAG,CAAC,OAAO,CAAC;QAAE,OAAO;IACxC,cAAc,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IAC5B,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,CAAC,CAAS,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;IACzD,KAAK,CAAC,OAAO,CAAC,CAAC;AACjB,CAAC;AAED,gEAAgE;AAChE,MAAM,UAAU,cAAc;IAC5B,cAAc,CAAC,KAAK,EAAE,CAAC;AACzB,CAAC;AAED;;;;GAIG;AACH,SAAS,YAAY,CAAC,KAAa;IACjC,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC;IAC7B,IAAI,CAAC,OAAO;QAAE,OAAO,OAAO,CAAC;IAC7B,IAAI,0BAA0B,CAAC,IAAI,CAAC,OAAO,CAAC;QAAE,OAAO,OAAO,CAAC;IAC7D,OAAO,UAAU,OAAO,EAAE,CAAC;AAC7B,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,oBAAoB,CAClC,MAAyB,OAAO,CAAC,GAAG,EACpC,MAAuB;IAEvB,MAAM,IAAI,GAAG,GAAG,CAAC,WAAW,EAAE,IAAI,EAAE,CAAC;IACrC,MAAM,OAAO,GAAG,GAAG,CAAC,eAAe,EAAE,IAAI,EAAE,CAAC;IAE5C,0EAA0E;IAC1E,IAAI,IAAI,IAAI,OAAO,IAAI,IAAI,KAAK,OAAO,EAAE,CAAC;QACxC,QAAQ,CACN,oEAAoE,IAAI,mCAAmC,EAC3G,MAAM,CACP,CAAC;IACJ,CAAC;IAED,IAAI,IAAI;QAAE,OAAO,YAAY,CAAC,IAAI,CAAC,CAAC;IACpC,IAAI,OAAO;QAAE,OAAO,YAAY,CAAC,OAAO,CAAC,CAAC;IAC1C,OAAO,uBAAuB,CAAC;AACjC,CAAC"}
|
package/dist/eval/llm.d.ts
CHANGED
|
@@ -11,7 +11,7 @@ export interface LlmClient {
|
|
|
11
11
|
generate(systemPrompt: string, userPrompt: string): Promise<GenerateResult>;
|
|
12
12
|
readonly model: string;
|
|
13
13
|
}
|
|
14
|
-
export type ProviderName = "anthropic" | "claude-cli" | "codex-cli" | "gemini-cli" | "ollama" | "openrouter";
|
|
14
|
+
export type ProviderName = "anthropic" | "claude-cli" | "codex-cli" | "gemini-cli" | "lm-studio" | "ollama" | "openrouter";
|
|
15
15
|
export interface LlmOverrides {
|
|
16
16
|
provider?: ProviderName;
|
|
17
17
|
model?: string;
|
package/dist/eval/llm.js
CHANGED
|
@@ -2,8 +2,11 @@
|
|
|
2
2
|
// LLM client for eval commands — supports multiple CLI tools and API providers
|
|
3
3
|
//
|
|
4
4
|
// Provider selection via VSKILL_EVAL_PROVIDER env var:
|
|
5
|
-
// "claude-cli" — Claude Code CLI (
|
|
6
|
-
//
|
|
5
|
+
// "claude-cli" — Claude Code CLI (delegates to your existing Claude Code
|
|
6
|
+
// session; the CLI handles quota. No API key, no OAuth, and
|
|
7
|
+
// no reads of ~/.claude/credentials|auth|token. See the
|
|
8
|
+
// compliance block above createClaudeCliClient() below.)
|
|
9
|
+
// "codex-cli" — OpenAI Codex CLI (uses ChatGPT session or CODEX_API_KEY)
|
|
7
10
|
// "gemini-cli" — Google Gemini CLI (free tier or GOOGLE_API_KEY)
|
|
8
11
|
// "anthropic" — Anthropic API (requires ANTHROPIC_API_KEY)
|
|
9
12
|
// "ollama" — Local Ollama server (free, requires ollama running)
|
|
@@ -25,6 +28,7 @@
|
|
|
25
28
|
import { spawn } from "node:child_process";
|
|
26
29
|
import { resolveCliBinary, enhancedPath } from "../utils/resolve-binary.js";
|
|
27
30
|
import { calculateCost, getBillingMode } from "./pricing.js";
|
|
31
|
+
import { resolveOllamaBaseUrl } from "./env.js";
|
|
28
32
|
function detectProvider() {
|
|
29
33
|
return "claude-cli";
|
|
30
34
|
}
|
|
@@ -50,6 +54,7 @@ export function estimateDurationSec(provider, totalCases, totalAssertions) {
|
|
|
50
54
|
"codex-cli": [8, 20],
|
|
51
55
|
"gemini-cli": [8, 20],
|
|
52
56
|
"ollama": [5, 30],
|
|
57
|
+
"lm-studio": [5, 30],
|
|
53
58
|
"openrouter": [4, 15],
|
|
54
59
|
};
|
|
55
60
|
const [lo, hi] = perCall[provider] ?? [5, 20];
|
|
@@ -74,10 +79,12 @@ export function createLlmClient(overrides) {
|
|
|
74
79
|
return createGeminiCliClient(modelOverride);
|
|
75
80
|
case "ollama":
|
|
76
81
|
return createOllamaClient(modelOverride);
|
|
82
|
+
case "lm-studio":
|
|
83
|
+
return createLmStudioClient(modelOverride);
|
|
77
84
|
case "openrouter":
|
|
78
85
|
return createOpenRouterClient(modelOverride);
|
|
79
86
|
default:
|
|
80
|
-
throw new Error(`Unknown VSKILL_EVAL_PROVIDER: "${provider}". Use "claude-cli", "codex-cli", "gemini-cli", "anthropic", "ollama", or "openrouter".`);
|
|
87
|
+
throw new Error(`Unknown VSKILL_EVAL_PROVIDER: "${provider}". Use "claude-cli", "codex-cli", "gemini-cli", "anthropic", "ollama", "lm-studio", or "openrouter".`);
|
|
81
88
|
}
|
|
82
89
|
}
|
|
83
90
|
// ---------------------------------------------------------------------------
|
|
@@ -226,8 +233,20 @@ function createCliClient(config) {
|
|
|
226
233
|
};
|
|
227
234
|
}
|
|
228
235
|
// ---------------------------------------------------------------------------
|
|
229
|
-
// Provider: Claude CLI
|
|
230
|
-
//
|
|
236
|
+
// Provider: Claude CLI — delegates to the official `claude` binary.
|
|
237
|
+
//
|
|
238
|
+
// COMPLIANCE CONTRACT (ToS, April 2026):
|
|
239
|
+
// This adapter MUST NOT read any file under `~/.claude/credentials*`,
|
|
240
|
+
// `~/.claude/auth*`, or `~/.claude/token*`. All session auth is owned by
|
|
241
|
+
// the `claude` CLI child process; vSkill Studio delegates and consumes
|
|
242
|
+
// only stdout/stderr. See `src/eval/__tests__/claude-cli-compliance.test.ts`
|
|
243
|
+
// which asserts zero fs reads against those paths at the Node API level,
|
|
244
|
+
// and `scripts/check-bundle-compliance.sh` which greps the built bundle
|
|
245
|
+
// for the literal path strings.
|
|
246
|
+
//
|
|
247
|
+
// Env handling: `CLAUDE*` env vars are stripped from the child so the CLI
|
|
248
|
+
// does not detect a nested session. No env var is added to compensate —
|
|
249
|
+
// the CLI resolves its own auth state.
|
|
231
250
|
// ---------------------------------------------------------------------------
|
|
232
251
|
function createClaudeCliClient(modelOverride) {
|
|
233
252
|
const raw = modelOverride || process.env.VSKILL_EVAL_MODEL || "sonnet";
|
|
@@ -239,7 +258,7 @@ function createClaudeCliClient(modelOverride) {
|
|
|
239
258
|
displayModel: model,
|
|
240
259
|
stripEnvPrefix: "CLAUDE",
|
|
241
260
|
provider: "claude-cli",
|
|
242
|
-
notFoundMsg: "Claude
|
|
261
|
+
notFoundMsg: "Claude Code not found. Install it: `npm install -g @anthropic-ai/claude-code`. Or choose a provider with an API key.",
|
|
243
262
|
});
|
|
244
263
|
}
|
|
245
264
|
// ---------------------------------------------------------------------------
|
|
@@ -275,7 +294,7 @@ function createGeminiCliClient(modelOverride) {
|
|
|
275
294
|
// Provider: Ollama (local models — free, no API key)
|
|
276
295
|
// ---------------------------------------------------------------------------
|
|
277
296
|
function createOllamaClient(modelOverride) {
|
|
278
|
-
const baseUrl = process.env
|
|
297
|
+
const baseUrl = resolveOllamaBaseUrl(process.env);
|
|
279
298
|
const model = modelOverride || process.env.VSKILL_EVAL_MODEL || "llama3.1:8b";
|
|
280
299
|
return {
|
|
281
300
|
model,
|
|
@@ -366,4 +385,65 @@ function createOpenRouterClient(modelOverride) {
|
|
|
366
385
|
},
|
|
367
386
|
};
|
|
368
387
|
}
|
|
388
|
+
// ---------------------------------------------------------------------------
|
|
389
|
+
// Provider: LM Studio (local OpenAI-compatible server — free, no API key)
|
|
390
|
+
//
|
|
391
|
+
// Mirrors the OpenRouter adapter at the HTTP level because LM Studio exposes
|
|
392
|
+
// the OpenAI chat completions surface verbatim at `<base>/chat/completions`.
|
|
393
|
+
// Three constants differ from OpenRouter:
|
|
394
|
+
// - base URL: process.env.LM_STUDIO_BASE_URL ?? "http://localhost:1234/v1"
|
|
395
|
+
// - API key: the literal string "lm-studio" (LM Studio ignores the token)
|
|
396
|
+
// - billing: "free" (local inference)
|
|
397
|
+
//
|
|
398
|
+
// Env overrides:
|
|
399
|
+
// - LM_STUDIO_BASE_URL — custom endpoint (e.g. a LAN machine)
|
|
400
|
+
// - VSKILL_EVAL_MODEL — the model id loaded in LM Studio
|
|
401
|
+
//
|
|
402
|
+
// Note: LM Studio does not enforce the Bearer token, so the dummy value
|
|
403
|
+
// "lm-studio" is safe to hardcode. Forks that do enforce it can override via
|
|
404
|
+
// a future LM_STUDIO_API_KEY env var (not wired in this increment per
|
|
405
|
+
// ADR-0677-02).
|
|
406
|
+
// ---------------------------------------------------------------------------
|
|
407
|
+
function createLmStudioClient(modelOverride) {
|
|
408
|
+
const baseUrl = process.env.LM_STUDIO_BASE_URL || "http://localhost:1234/v1";
|
|
409
|
+
const model = modelOverride || process.env.VSKILL_EVAL_MODEL || "local-model";
|
|
410
|
+
return {
|
|
411
|
+
model,
|
|
412
|
+
async generate(systemPrompt, userPrompt) {
|
|
413
|
+
const start = Date.now();
|
|
414
|
+
const response = await fetch(`${baseUrl}/chat/completions`, {
|
|
415
|
+
method: "POST",
|
|
416
|
+
headers: {
|
|
417
|
+
"Content-Type": "application/json",
|
|
418
|
+
"Authorization": "Bearer lm-studio",
|
|
419
|
+
},
|
|
420
|
+
body: JSON.stringify({
|
|
421
|
+
model,
|
|
422
|
+
messages: [
|
|
423
|
+
{ role: "system", content: systemPrompt },
|
|
424
|
+
{ role: "user", content: userPrompt },
|
|
425
|
+
],
|
|
426
|
+
max_tokens: 4096,
|
|
427
|
+
}),
|
|
428
|
+
signal: AbortSignal.timeout(getTimeoutMs()),
|
|
429
|
+
});
|
|
430
|
+
if (!response.ok) {
|
|
431
|
+
const errorBody = await response.text();
|
|
432
|
+
throw new Error(`LM Studio request failed (${response.status}): ${errorBody.slice(0, 200)}`);
|
|
433
|
+
}
|
|
434
|
+
const data = (await response.json());
|
|
435
|
+
const text = data.choices?.[0]?.message?.content || "";
|
|
436
|
+
const inputTokens = data.usage?.prompt_tokens ?? null;
|
|
437
|
+
const outputTokens = data.usage?.completion_tokens ?? null;
|
|
438
|
+
return {
|
|
439
|
+
text,
|
|
440
|
+
durationMs: Date.now() - start,
|
|
441
|
+
inputTokens,
|
|
442
|
+
outputTokens,
|
|
443
|
+
cost: 0,
|
|
444
|
+
billingMode: "free",
|
|
445
|
+
};
|
|
446
|
+
},
|
|
447
|
+
};
|
|
448
|
+
}
|
|
369
449
|
//# sourceMappingURL=llm.js.map
|
package/dist/eval/llm.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"llm.js","sourceRoot":"","sources":["../../src/eval/llm.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,+EAA+E;AAC/E,EAAE;AACF,uDAAuD;AACvD,wEAAwE;AACxE,iFAAiF;AACjF,mEAAmE;AACnE,8DAA8D;AAC9D,uEAAuE;AACvE,EAAE;AACF,uDAAuD;AACvD,iFAAiF;AACjF,2EAA2E;AAC3E,EAAE;AACF,mEAAmE;AACnE,oEAAoE;AACpE,EAAE;AACF,iDAAiD;AACjD,+DAA+D;AAC/D,4EAA4E;AAC5E,iFAAiF;AACjF,4DAA4D;AAC5D,mDAAmD;AACnD,8EAA8E;AAE9E,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,gBAAgB,EAAE,YAAY,EAAE,MAAM,4BAA4B,CAAC;AAC5E,OAAO,EAAE,aAAa,EAAE,cAAc,EAAE,MAAM,cAAc,CAAC;AAoB7D,SAAS,cAAc;IACrB,OAAO,YAAY,CAAC;AACtB,CAAC;AAED,oFAAoF;AACpF,SAAS,YAAY;IACnB,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,mBAAmB,CAAC;IAC/C,IAAI,MAAM,EAAE,CAAC;QACX,MAAM,OAAO,GAAG,QAAQ,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QACrC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,OAAO,GAAG,CAAC;YAAE,OAAO,OAAO,GAAG,IAAI,CAAC;IAC5D,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC;AAOD;;;GAGG;AACH,MAAM,UAAU,mBAAmB,CACjC,QAAsB,EACtB,UAAkB,EAClB,eAAuB;IAEvB,+CAA+C;IAC/C,MAAM,OAAO,GAA2C;QACtD,YAAY,EAAG,CAAC,EAAE,EAAE,EAAE,CAAC;QACvB,WAAW,EAAI,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,WAAW,EAAI,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,YAAY,EAAG,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,QAAQ,EAAO,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,YAAY,EAAG,CAAC,CAAC,EAAE,EAAE,CAAC;KACvB,CAAC;IACF,MAAM,CAAC,EAAE,EAAE,EAAE,CAAC,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IAC9C,MAAM,UAAU,GAAG,UAAU,GAAG,eAAe,CAAC,CAAC,wBAAwB;IACzE,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,GAAG,EAAE,CAAC,CAAC;IAC3C,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,GAAG,EAAE,CAAC,CAAC;IAE3C,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC;IACxE,MAAM,KAAK,GAAG,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,MAAM,CAAC,IAAI,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;IAChF,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC;AACnC,CAAC;AAED,MAAM,UAAU,eAAe,CAAC,SAAwB;IACtD,MAAM,QAAQ,GAAG,CAAC,SAAS,EAAE,QAAQ,IAAI,OAAO,CAAC,GAAG,CAAC,oBAAoB,IAAI,cAAc,EAAE,CAAiB,CAAC;IAC/G,MAAM,aAAa,GAAG,SAAS,EAAE,KAAK,CAAC;IACvC,QAAQ,QAAQ,EAAE,CAAC;QACjB,KAAK,WAAW;YACd,OAAO,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC9C,KAAK,YAAY;YACf,OAAO,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC9C,KAAK,WAAW;YACd,OAAO,oBAAoB,CAAC,aAAa,CAAC,CAAC;QAC7C,KAAK,YAAY;YACf,OAAO,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC9C,KAAK,QAAQ;YACX,OAAO,kBAAkB,CAAC,aAAa,CAAC,CAAC;QAC3C,KAAK,YAAY;YACf,OAAO,sBAAsB,CAAC,aAAa,CAAC,CAAC;QAC/C;YACE,MAAM,IAAI,KAAK,CACb,kCAAkC,QAAQ,yFAAyF,CACpI,CAAC;IACN,CAAC;AACH,CAAC;AAED,8EAA8E;AAC9E,gEAAgE;AAChE,qEAAqE;AACrE,yEAAyE;AACzE,8EAA8E;AAC9E,MAAM,oBAAoB,GAA2B;IACnD,eAAe,EAAE,QAAQ;IACzB,aAAa,EAAE,MAAM;IACrB,cAAc,EAAE,OAAO;IACvB,mBAAmB,EAAE,QAAQ;IAC7B,0BAA0B,EAAE,QAAQ;IACpC,iBAAiB,EAAE,MAAM;IACzB,iBAAiB,EAAE,MAAM;IACzB,wBAAwB,EAAE,MAAM;IAChC,2BAA2B,EAAE,OAAO;CACrC,CAAC;AAEF,MAAM,mBAAmB,GAA2B;IAClD,QAAQ,EAAE,mBAAmB;IAC7B,MAAM,EAAE,iBAAiB;IACzB,OAAO,EAAE,2BAA2B;IACpC,eAAe,EAAE,mBAAmB;IACpC,aAAa,EAAE,iBAAiB;IAChC,cAAc,EAAE,2BAA2B;CAC5C,CAAC;AAEF,SAAS,uBAAuB,CAAC,KAAa;IAC5C,OAAO,oBAAoB,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC;AAC9C,CAAC;AAED,SAAS,uBAAuB,CAAC,KAAa;IAC5C,OAAO,mBAAmB,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC;AAC7C,CAAC;AAED,8EAA8E;AAC9E,0BAA0B;AAC1B,8EAA8E;AAC9E,SAAS,qBAAqB,CAAC,aAAsB;IACnD,MAAM,aAAa,GAAG,mBAAmB,CAAC;IAE1C,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,iBAAiB,CAAC;IAC7C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,sRAAsR,CACvR,CAAC;IACJ,CAAC;IAED,MAAM,GAAG,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,aAAa,CAAC;IAC5E,MAAM,KAAK,GAAG,uBAAuB,CAAC,GAAG,CAAC,CAAC;IAC3C,IAAI,cAAc,GAAQ,IAAI,CAAC;IAE/B,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,IAAI,CAAC,cAAc,EAAE,CAAC;gBACpB,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,CAAC;gBACjE,cAAc,GAAG,IAAI,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC;YAC7C,CAAC;YAED,MAAM,UAAU,GAAG,IAAI,eAAe,EAAE,CAAC;YACzC,MAAM,OAAO,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,UAAU,CAAC,KAAK,EAAE,EAAE,YAAY,EAAE,CAAC,CAAC;YACrE,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YACzB,IAAI,CAAC;gBACH,MAAM,QAAQ,GAAG,MAAM,cAAc,CAAC,QAAQ,CAAC,MAAM,CACnD;oBACE,KAAK;oBACL,MAAM,EAAE,YAAY;oBACpB,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE,CAAC;oBACjD,UAAU,EAAE,IAAI;iBACjB,EACD,EAAE,MAAM,EAAE,UAAU,CAAC,MAAM,EAAE,CAC9B,CAAC;gBACF,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC;gBAEtC,MAAM,SAAS,GAAG,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAM,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC;gBACvE,MAAM,IAAI,GAAG,SAAS,IAAI,MAAM,IAAI,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC;gBACpE,MAAM,WAAW,GAAG,QAAQ,CAAC,KAAK,EAAE,YAAY,IAAI,IAAI,CAAC;gBACzD,MAAM,YAAY,GAAG,QAAQ,CAAC,KAAK,EAAE,aAAa,IAAI,IAAI,CAAC;gBAC3D,OAAO;oBACL,IAAI;oBACJ,UAAU;oBACV,WAAW;oBACX,YAAY;oBACZ,IAAI,EAAE,aAAa,CAAC,WAAW,EAAE,KAAK,EAAE,WAAW,EAAE,YAAY,CAAC;oBAClE,WAAW,EAAE,cAAc,CAAC,WAAW,CAAC;iBACzC,CAAC;YACJ,CAAC;oBAAS,CAAC;gBACT,YAAY,CAAC,OAAO,CAAC,CAAC;YACxB,CAAC;QACH,CAAC;KACF,CAAC;AACJ,CAAC;AAkBD,SAAS,eAAe,CAAC,MAAiB;IACxC,uEAAuE;IACvE,MAAM,cAAc,GAAG,gBAAgB,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;IACvD,gFAAgF;IAChF,MAAM,kBAAkB,GAAG,YAAY,EAAE,CAAC;IAE1C,OAAO;QACL,KAAK,EAAE,MAAM,CAAC,YAAY;QAC1B,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,cAAc,GAAG,GAAG,YAAY,OAAO,UAAU,EAAE,CAAC;YAC1D,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,IAAI,GAAG,MAAM,IAAI,OAAO,CAAS,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;gBACzD,mEAAmE;gBACnE,IAAI,GAA2B,CAAC;gBAChC,IAAI,MAAM,CAAC,cAAc,EAAE,CAAC;oBAC1B,GAAG,GAAG,EAAE,CAAC;oBACT,MAAM,MAAM,GAAG,MAAM,CAAC,cAAc,CAAC;oBACrC,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;wBACjD,IAAI,CAAC,KAAK,SAAS,IAAI,CAAC,CAAC,CAAC,UAAU,CAAC,MAAM,CAAC;4BAAE,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;oBAC3D,CAAC;gBACH,CAAC;qBAAM,CAAC;oBACN,GAAG,GAAG,EAAE,GAAG,OAAO,CAAC,GAA6B,EAAE,CAAC;gBACrD,CAAC;gBACD,GAAG,CAAC,IAAI,GAAG,kBAAkB,CAAC;gBAE9B,sEAAsE;gBACtE,MAAM,UAAU,GAAG,OAAO,CAAC,QAAQ,KAAK,OAAO,IAAI,eAAe,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;gBAExF,MAAM,IAAI,GAAG,KAAK,CAAC,cAAc,EAAE,MAAM,CAAC,IAAI,EAAE;oBAC9C,KAAK,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC;oBAC/B,GAAG;oBACH,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;iBACvC,CAAC,CAAC;gBAEH,IAAI,MAAM,GAAG,EAAE,CAAC;gBAChB,IAAI,MAAM,GAAG,EAAE,CAAC;gBAChB,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,CAAS,EAAE,EAAE,GAAG,MAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;gBACnE,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,CAAS,EAAE,EAAE,GAAG,MAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;gBAEnE,MAAM,SAAS,GAAG,YAAY,EAAE,CAAC;gBACjC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE;oBAC5B,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;oBACrB,MAAM,CAAC,IAAI,KAAK,CAAC,GAAG,MAAM,CAAC,IAAI,wBAAwB,SAAS,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC;gBAC/E,CAAC,EAAE,SAAS,CAAC,CAAC;gBAEd,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,GAA0B,EAAE,EAAE;oBAC9C,YAAY,CAAC,KAAK,CAAC,CAAC;oBACpB,IAAI,GAAG,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;wBAC1B,MAAM,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC,CAAC;oBACxC,CAAC;yBAAM,CAAC;wBACN,MAAM,CAAC,IAAI,KAAK,CAAC,GAAG,MAAM,CAAC,IAAI,gBAAgB,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;oBACjE,CAAC;gBACH,CAAC,CAAC,CAAC;gBAEH,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,IAAI,EAAE,EAAE;oBACxB,YAAY,CAAC,KAAK,CAAC,CAAC;oBACpB,IAAI,IAAI,KAAK,CAAC,EAAE,CAAC;wBACf,OAAO,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;oBACzB,CAAC;yBAAM,CAAC;wBACN,MAAM,MAAM,GAAG,CAAC,MAAM,IAAI,MAAM,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;wBAChD,MAAM,CAAC,IAAI,KAAK,CACd,GAAG,MAAM,CAAC,IAAI,yBAAyB,IAAI,GAAG,MAAM,CAAC,CAAC,CAAC,IAAI,GAAG,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAC5E,CAAC,CAAC;oBACL,CAAC;gBACH,CAAC,CAAC,CAAC;gBAEH,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC;YACjC,CAAC,CAAC,CAAC;YAEH,OAAO,EAAE,IAAI,EAAE,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,EAAE,WAAW,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,WAAW,EAAE,cAAc,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;QACnJ,CAAC;KACF,CAAC;AACJ,CAAC;AAED,8EAA8E;AAC9E,4EAA4E;AAC5E,uEAAuE;AACvE,8EAA8E;AAC9E,SAAS,qBAAqB,CAAC,aAAsB;IACnD,MAAM,GAAG,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,QAAQ,CAAC;IACvE,MAAM,KAAK,GAAG,uBAAuB,CAAC,GAAG,CAAC,CAAC;IAC3C,OAAO,eAAe,CAAC;QACrB,MAAM,EAAE,QAAQ;QAChB,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,CAAC,IAAI,EAAE,SAAS,EAAE,KAAK,CAAC;QAC9B,YAAY,EAAE,KAAK;QACnB,cAAc,EAAE,QAAQ;QACxB,QAAQ,EAAE,YAAY;QACtB,WAAW,EACT,qJAAqJ;KACxJ,CAAC,CAAC;AACL,CAAC;AAED,8EAA8E;AAC9E,iFAAiF;AACjF,8EAA8E;AAC9E,SAAS,oBAAoB,CAAC,aAAsB;IAClD,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,SAAS,CAAC;IAC1E,OAAO,eAAe,CAAC;QACrB,MAAM,EAAE,OAAO;QACf,IAAI,EAAE,OAAO;QACb,IAAI,EAAE,CAAC,MAAM,EAAE,SAAS,EAAE,KAAK,CAAC;QAChC,YAAY,EAAE,SAAS,KAAK,EAAE;QAC9B,QAAQ,EAAE,WAAW;QACrB,WAAW,EACT,4IAA4I;KAC/I,CAAC,CAAC;AACL,CAAC;AAED,8EAA8E;AAC9E,iFAAiF;AACjF,kFAAkF;AAClF,8EAA8E;AAC9E,SAAS,qBAAqB,CAAC,aAAsB;IACnD,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,gBAAgB,CAAC;IACjF,OAAO,eAAe,CAAC;QACrB,MAAM,EAAE,QAAQ;QAChB,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,CAAC,IAAI,EAAE,SAAS,EAAE,KAAK,CAAC;QAC9B,YAAY,EAAE,KAAK;QACnB,QAAQ,EAAE,YAAY;QACtB,WAAW,EACT,kJAAkJ;KACrJ,CAAC,CAAC;AACL,CAAC;AAED,8EAA8E;AAC9E,qDAAqD;AACrD,8EAA8E;AAC9E,SAAS,kBAAkB,CAAC,aAAsB;IAChD,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,wBAAwB,CAAC;IACxE,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,aAAa,CAAC;IAE9E,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,UAAU,GAAG,GAAG,YAAY,OAAO,UAAU,EAAE,CAAC;YACtD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,eAAe,EAAE;gBACtD,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;gBAC/C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,KAAK;oBACL,MAAM,EAAE,UAAU;oBAClB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE;wBACP,WAAW,EAAE,IAAI;wBACjB,WAAW,EAAE,GAAG;qBACjB;iBACF,CAAC;gBACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC;aAC5C,CAAC,CAAC;YAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,KAAK,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;gBACpC,IAAI,KAAK,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,WAAW,CAAC,EAAE,CAAC;oBAC3D,MAAM,IAAI,KAAK,CACb,iBAAiB,KAAK,8CAA8C,KAAK,EAAE,CAC5E,CAAC;gBACJ,CAAC;gBACD,MAAM,IAAI,KAAK,CAAC,0BAA0B,KAAK,EAAE,CAAC,CAAC;YACrD,CAAC;YAED,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAIlC,CAAC;YACF,OAAO;gBACL,IAAI,EAAE,IAAI,CAAC,QAAQ,IAAI,EAAE;gBACzB,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK;gBAC9B,WAAW,EAAE,IAAI,CAAC,iBAAiB,IAAI,IAAI;gBAC3C,YAAY,EAAE,IAAI,CAAC,UAAU,IAAI,IAAI;gBACrC,IAAI,EAAE,CAAC;gBACP,WAAW,EAAE,MAAe;aAC7B,CAAC;QACJ,CAAC;KACF,CAAC;AACJ,CAAC;AAED,8EAA8E;AAC9E,wDAAwD;AACxD,8EAA8E;AAC9E,SAAS,sBAAsB,CAAC,aAAsB;IACpD,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;IAC9C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,oIAAoI,CACrI,CAAC;IACJ,CAAC;IAED,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,2BAA2B,CAAC;IAE5F,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,+CAA+C,EAAE;gBAC5E,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE;oBACP,cAAc,EAAE,kBAAkB;oBAClC,eAAe,EAAE,UAAU,MAAM,EAAE;oBACnC,cAAc,EAAE,4BAA4B;oBAC5C,SAAS,EAAE,eAAe;iBAC3B;gBACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,KAAK;oBACL,QAAQ,EAAE;wBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,YAAY,EAAE;wBACzC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE;qBACtC;oBACD,UAAU,EAAE,IAAI;iBACjB,CAAC;gBACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC;aAC5C,CAAC,CAAC;YAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,KAAK,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;gBACpC,MAAM,IAAI,KAAK,CAAC,8BAA8B,QAAQ,CAAC,MAAM,MAAM,KAAK,EAAE,CAAC,CAAC;YAC9E,CAAC;YAED,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAOlC,CAAC;YAEF,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,IAAI,EAAE,CAAC;YACvD,MAAM,WAAW,GAAG,IAAI,CAAC,KAAK,EAAE,aAAa,IAAI,IAAI,CAAC;YACtD,MAAM,YAAY,GAAG,IAAI,CAAC,KAAK,EAAE,iBAAiB,IAAI,IAAI,CAAC;YAC3D,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,EAAE,UAAU,IAAI,IAAI,CAAC;YAC/C,OAAO;gBACL,IAAI;gBACJ,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK;gBAC9B,WAAW;gBACX,YAAY;gBACZ,IAAI,EAAE,OAAO,IAAI,aAAa,CAAC,YAAY,EAAE,KAAK,EAAE,WAAW,EAAE,YAAY,CAAC;gBAC9E,WAAW,EAAE,WAAoB;aAClC,CAAC;QACJ,CAAC;KACF,CAAC;AACJ,CAAC"}
|
|
1
|
+
{"version":3,"file":"llm.js","sourceRoot":"","sources":["../../src/eval/llm.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,+EAA+E;AAC/E,EAAE;AACF,uDAAuD;AACvD,2EAA2E;AAC3E,6EAA6E;AAC7E,yEAAyE;AACzE,0EAA0E;AAC1E,4EAA4E;AAC5E,mEAAmE;AACnE,8DAA8D;AAC9D,uEAAuE;AACvE,EAAE;AACF,uDAAuD;AACvD,iFAAiF;AACjF,2EAA2E;AAC3E,EAAE;AACF,mEAAmE;AACnE,oEAAoE;AACpE,EAAE;AACF,iDAAiD;AACjD,+DAA+D;AAC/D,4EAA4E;AAC5E,iFAAiF;AACjF,4DAA4D;AAC5D,mDAAmD;AACnD,8EAA8E;AAE9E,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,gBAAgB,EAAE,YAAY,EAAE,MAAM,4BAA4B,CAAC;AAC5E,OAAO,EAAE,aAAa,EAAE,cAAc,EAAE,MAAM,cAAc,CAAC;AAC7D,OAAO,EAAE,oBAAoB,EAAE,MAAM,UAAU,CAAC;AAoBhD,SAAS,cAAc;IACrB,OAAO,YAAY,CAAC;AACtB,CAAC;AAED,oFAAoF;AACpF,SAAS,YAAY;IACnB,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,mBAAmB,CAAC;IAC/C,IAAI,MAAM,EAAE,CAAC;QACX,MAAM,OAAO,GAAG,QAAQ,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QACrC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,OAAO,GAAG,CAAC;YAAE,OAAO,OAAO,GAAG,IAAI,CAAC;IAC5D,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC;AAOD;;;GAGG;AACH,MAAM,UAAU,mBAAmB,CACjC,QAAsB,EACtB,UAAkB,EAClB,eAAuB;IAEvB,+CAA+C;IAC/C,MAAM,OAAO,GAA2C;QACtD,YAAY,EAAG,CAAC,EAAE,EAAE,EAAE,CAAC;QACvB,WAAW,EAAI,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,WAAW,EAAI,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,YAAY,EAAG,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,QAAQ,EAAO,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,WAAW,EAAI,CAAC,CAAC,EAAE,EAAE,CAAC;QACtB,YAAY,EAAG,CAAC,CAAC,EAAE,EAAE,CAAC;KACvB,CAAC;IACF,MAAM,CAAC,EAAE,EAAE,EAAE,CAAC,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IAC9C,MAAM,UAAU,GAAG,UAAU,GAAG,eAAe,CAAC,CAAC,wBAAwB;IACzE,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,GAAG,EAAE,CAAC,CAAC;IAC3C,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,GAAG,EAAE,CAAC,CAAC;IAE3C,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC;IACxE,MAAM,KAAK,GAAG,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,MAAM,CAAC,IAAI,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;IAChF,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC;AACnC,CAAC;AAED,MAAM,UAAU,eAAe,CAAC,SAAwB;IACtD,MAAM,QAAQ,GAAG,CAAC,SAAS,EAAE,QAAQ,IAAI,OAAO,CAAC,GAAG,CAAC,oBAAoB,IAAI,cAAc,EAAE,CAAiB,CAAC;IAC/G,MAAM,aAAa,GAAG,SAAS,EAAE,KAAK,CAAC;IACvC,QAAQ,QAAQ,EAAE,CAAC;QACjB,KAAK,WAAW;YACd,OAAO,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC9C,KAAK,YAAY;YACf,OAAO,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC9C,KAAK,WAAW;YACd,OAAO,oBAAoB,CAAC,aAAa,CAAC,CAAC;QAC7C,KAAK,YAAY;YACf,OAAO,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC9C,KAAK,QAAQ;YACX,OAAO,kBAAkB,CAAC,aAAa,CAAC,CAAC;QAC3C,KAAK,WAAW;YACd,OAAO,oBAAoB,CAAC,aAAa,CAAC,CAAC;QAC7C,KAAK,YAAY;YACf,OAAO,sBAAsB,CAAC,aAAa,CAAC,CAAC;QAC/C;YACE,MAAM,IAAI,KAAK,CACb,kCAAkC,QAAQ,sGAAsG,CACjJ,CAAC;IACN,CAAC;AACH,CAAC;AAED,8EAA8E;AAC9E,gEAAgE;AAChE,qEAAqE;AACrE,yEAAyE;AACzE,8EAA8E;AAC9E,MAAM,oBAAoB,GAA2B;IACnD,eAAe,EAAE,QAAQ;IACzB,aAAa,EAAE,MAAM;IACrB,cAAc,EAAE,OAAO;IACvB,mBAAmB,EAAE,QAAQ;IAC7B,0BAA0B,EAAE,QAAQ;IACpC,iBAAiB,EAAE,MAAM;IACzB,iBAAiB,EAAE,MAAM;IACzB,wBAAwB,EAAE,MAAM;IAChC,2BAA2B,EAAE,OAAO;CACrC,CAAC;AAEF,MAAM,mBAAmB,GAA2B;IAClD,QAAQ,EAAE,mBAAmB;IAC7B,MAAM,EAAE,iBAAiB;IACzB,OAAO,EAAE,2BAA2B;IACpC,eAAe,EAAE,mBAAmB;IACpC,aAAa,EAAE,iBAAiB;IAChC,cAAc,EAAE,2BAA2B;CAC5C,CAAC;AAEF,SAAS,uBAAuB,CAAC,KAAa;IAC5C,OAAO,oBAAoB,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC;AAC9C,CAAC;AAED,SAAS,uBAAuB,CAAC,KAAa;IAC5C,OAAO,mBAAmB,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC;AAC7C,CAAC;AAED,8EAA8E;AAC9E,0BAA0B;AAC1B,8EAA8E;AAC9E,SAAS,qBAAqB,CAAC,aAAsB;IACnD,MAAM,aAAa,GAAG,mBAAmB,CAAC;IAE1C,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,iBAAiB,CAAC;IAC7C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,sRAAsR,CACvR,CAAC;IACJ,CAAC;IAED,MAAM,GAAG,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,aAAa,CAAC;IAC5E,MAAM,KAAK,GAAG,uBAAuB,CAAC,GAAG,CAAC,CAAC;IAC3C,IAAI,cAAc,GAAQ,IAAI,CAAC;IAE/B,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,IAAI,CAAC,cAAc,EAAE,CAAC;gBACpB,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,CAAC;gBACjE,cAAc,GAAG,IAAI,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC;YAC7C,CAAC;YAED,MAAM,UAAU,GAAG,IAAI,eAAe,EAAE,CAAC;YACzC,MAAM,OAAO,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,UAAU,CAAC,KAAK,EAAE,EAAE,YAAY,EAAE,CAAC,CAAC;YACrE,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YACzB,IAAI,CAAC;gBACH,MAAM,QAAQ,GAAG,MAAM,cAAc,CAAC,QAAQ,CAAC,MAAM,CACnD;oBACE,KAAK;oBACL,MAAM,EAAE,YAAY;oBACpB,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE,CAAC;oBACjD,UAAU,EAAE,IAAI;iBACjB,EACD,EAAE,MAAM,EAAE,UAAU,CAAC,MAAM,EAAE,CAC9B,CAAC;gBACF,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC;gBAEtC,MAAM,SAAS,GAAG,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAM,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC;gBACvE,MAAM,IAAI,GAAG,SAAS,IAAI,MAAM,IAAI,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC;gBACpE,MAAM,WAAW,GAAG,QAAQ,CAAC,KAAK,EAAE,YAAY,IAAI,IAAI,CAAC;gBACzD,MAAM,YAAY,GAAG,QAAQ,CAAC,KAAK,EAAE,aAAa,IAAI,IAAI,CAAC;gBAC3D,OAAO;oBACL,IAAI;oBACJ,UAAU;oBACV,WAAW;oBACX,YAAY;oBACZ,IAAI,EAAE,aAAa,CAAC,WAAW,EAAE,KAAK,EAAE,WAAW,EAAE,YAAY,CAAC;oBAClE,WAAW,EAAE,cAAc,CAAC,WAAW,CAAC;iBACzC,CAAC;YACJ,CAAC;oBAAS,CAAC;gBACT,YAAY,CAAC,OAAO,CAAC,CAAC;YACxB,CAAC;QACH,CAAC;KACF,CAAC;AACJ,CAAC;AAkBD,SAAS,eAAe,CAAC,MAAiB;IACxC,uEAAuE;IACvE,MAAM,cAAc,GAAG,gBAAgB,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;IACvD,gFAAgF;IAChF,MAAM,kBAAkB,GAAG,YAAY,EAAE,CAAC;IAE1C,OAAO;QACL,KAAK,EAAE,MAAM,CAAC,YAAY;QAC1B,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,cAAc,GAAG,GAAG,YAAY,OAAO,UAAU,EAAE,CAAC;YAC1D,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,IAAI,GAAG,MAAM,IAAI,OAAO,CAAS,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;gBACzD,mEAAmE;gBACnE,IAAI,GAA2B,CAAC;gBAChC,IAAI,MAAM,CAAC,cAAc,EAAE,CAAC;oBAC1B,GAAG,GAAG,EAAE,CAAC;oBACT,MAAM,MAAM,GAAG,MAAM,CAAC,cAAc,CAAC;oBACrC,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;wBACjD,IAAI,CAAC,KAAK,SAAS,IAAI,CAAC,CAAC,CAAC,UAAU,CAAC,MAAM,CAAC;4BAAE,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;oBAC3D,CAAC;gBACH,CAAC;qBAAM,CAAC;oBACN,GAAG,GAAG,EAAE,GAAG,OAAO,CAAC,GAA6B,EAAE,CAAC;gBACrD,CAAC;gBACD,GAAG,CAAC,IAAI,GAAG,kBAAkB,CAAC;gBAE9B,sEAAsE;gBACtE,MAAM,UAAU,GAAG,OAAO,CAAC,QAAQ,KAAK,OAAO,IAAI,eAAe,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;gBAExF,MAAM,IAAI,GAAG,KAAK,CAAC,cAAc,EAAE,MAAM,CAAC,IAAI,EAAE;oBAC9C,KAAK,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC;oBAC/B,GAAG;oBACH,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;iBACvC,CAAC,CAAC;gBAEH,IAAI,MAAM,GAAG,EAAE,CAAC;gBAChB,IAAI,MAAM,GAAG,EAAE,CAAC;gBAChB,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,CAAS,EAAE,EAAE,GAAG,MAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;gBACnE,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,CAAS,EAAE,EAAE,GAAG,MAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;gBAEnE,MAAM,SAAS,GAAG,YAAY,EAAE,CAAC;gBACjC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE;oBAC5B,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;oBACrB,MAAM,CAAC,IAAI,KAAK,CAAC,GAAG,MAAM,CAAC,IAAI,wBAAwB,SAAS,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC;gBAC/E,CAAC,EAAE,SAAS,CAAC,CAAC;gBAEd,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,GAA0B,EAAE,EAAE;oBAC9C,YAAY,CAAC,KAAK,CAAC,CAAC;oBACpB,IAAI,GAAG,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;wBAC1B,MAAM,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC,CAAC;oBACxC,CAAC;yBAAM,CAAC;wBACN,MAAM,CAAC,IAAI,KAAK,CAAC,GAAG,MAAM,CAAC,IAAI,gBAAgB,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;oBACjE,CAAC;gBACH,CAAC,CAAC,CAAC;gBAEH,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,IAAI,EAAE,EAAE;oBACxB,YAAY,CAAC,KAAK,CAAC,CAAC;oBACpB,IAAI,IAAI,KAAK,CAAC,EAAE,CAAC;wBACf,OAAO,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;oBACzB,CAAC;yBAAM,CAAC;wBACN,MAAM,MAAM,GAAG,CAAC,MAAM,IAAI,MAAM,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;wBAChD,MAAM,CAAC,IAAI,KAAK,CACd,GAAG,MAAM,CAAC,IAAI,yBAAyB,IAAI,GAAG,MAAM,CAAC,CAAC,CAAC,IAAI,GAAG,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAC5E,CAAC,CAAC;oBACL,CAAC;gBACH,CAAC,CAAC,CAAC;gBAEH,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC;YACjC,CAAC,CAAC,CAAC;YAEH,OAAO,EAAE,IAAI,EAAE,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,EAAE,WAAW,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,WAAW,EAAE,cAAc,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;QACnJ,CAAC;KACF,CAAC;AACJ,CAAC;AAED,8EAA8E;AAC9E,oEAAoE;AACpE,EAAE;AACF,yCAAyC;AACzC,wEAAwE;AACxE,2EAA2E;AAC3E,yEAAyE;AACzE,+EAA+E;AAC/E,2EAA2E;AAC3E,0EAA0E;AAC1E,kCAAkC;AAClC,EAAE;AACF,4EAA4E;AAC5E,0EAA0E;AAC1E,yCAAyC;AACzC,8EAA8E;AAC9E,SAAS,qBAAqB,CAAC,aAAsB;IACnD,MAAM,GAAG,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,QAAQ,CAAC;IACvE,MAAM,KAAK,GAAG,uBAAuB,CAAC,GAAG,CAAC,CAAC;IAC3C,OAAO,eAAe,CAAC;QACrB,MAAM,EAAE,QAAQ;QAChB,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,CAAC,IAAI,EAAE,SAAS,EAAE,KAAK,CAAC;QAC9B,YAAY,EAAE,KAAK;QACnB,cAAc,EAAE,QAAQ;QACxB,QAAQ,EAAE,YAAY;QACtB,WAAW,EACT,sHAAsH;KACzH,CAAC,CAAC;AACL,CAAC;AAED,8EAA8E;AAC9E,iFAAiF;AACjF,8EAA8E;AAC9E,SAAS,oBAAoB,CAAC,aAAsB;IAClD,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,SAAS,CAAC;IAC1E,OAAO,eAAe,CAAC;QACrB,MAAM,EAAE,OAAO;QACf,IAAI,EAAE,OAAO;QACb,IAAI,EAAE,CAAC,MAAM,EAAE,SAAS,EAAE,KAAK,CAAC;QAChC,YAAY,EAAE,SAAS,KAAK,EAAE;QAC9B,QAAQ,EAAE,WAAW;QACrB,WAAW,EACT,4IAA4I;KAC/I,CAAC,CAAC;AACL,CAAC;AAED,8EAA8E;AAC9E,iFAAiF;AACjF,kFAAkF;AAClF,8EAA8E;AAC9E,SAAS,qBAAqB,CAAC,aAAsB;IACnD,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,gBAAgB,CAAC;IACjF,OAAO,eAAe,CAAC;QACrB,MAAM,EAAE,QAAQ;QAChB,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,CAAC,IAAI,EAAE,SAAS,EAAE,KAAK,CAAC;QAC9B,YAAY,EAAE,KAAK;QACnB,QAAQ,EAAE,YAAY;QACtB,WAAW,EACT,kJAAkJ;KACrJ,CAAC,CAAC;AACL,CAAC;AAED,8EAA8E;AAC9E,qDAAqD;AACrD,8EAA8E;AAC9E,SAAS,kBAAkB,CAAC,aAAsB;IAChD,MAAM,OAAO,GAAG,oBAAoB,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAClD,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,aAAa,CAAC;IAE9E,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,UAAU,GAAG,GAAG,YAAY,OAAO,UAAU,EAAE,CAAC;YACtD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,eAAe,EAAE;gBACtD,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;gBAC/C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,KAAK;oBACL,MAAM,EAAE,UAAU;oBAClB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE;wBACP,WAAW,EAAE,IAAI;wBACjB,WAAW,EAAE,GAAG;qBACjB;iBACF,CAAC;gBACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC;aAC5C,CAAC,CAAC;YAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,KAAK,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;gBACpC,IAAI,KAAK,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,WAAW,CAAC,EAAE,CAAC;oBAC3D,MAAM,IAAI,KAAK,CACb,iBAAiB,KAAK,8CAA8C,KAAK,EAAE,CAC5E,CAAC;gBACJ,CAAC;gBACD,MAAM,IAAI,KAAK,CAAC,0BAA0B,KAAK,EAAE,CAAC,CAAC;YACrD,CAAC;YAED,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAIlC,CAAC;YACF,OAAO;gBACL,IAAI,EAAE,IAAI,CAAC,QAAQ,IAAI,EAAE;gBACzB,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK;gBAC9B,WAAW,EAAE,IAAI,CAAC,iBAAiB,IAAI,IAAI;gBAC3C,YAAY,EAAE,IAAI,CAAC,UAAU,IAAI,IAAI;gBACrC,IAAI,EAAE,CAAC;gBACP,WAAW,EAAE,MAAe;aAC7B,CAAC;QACJ,CAAC;KACF,CAAC;AACJ,CAAC;AAED,8EAA8E;AAC9E,wDAAwD;AACxD,8EAA8E;AAC9E,SAAS,sBAAsB,CAAC,aAAsB;IACpD,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;IAC9C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,oIAAoI,CACrI,CAAC;IACJ,CAAC;IAED,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,2BAA2B,CAAC;IAE5F,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,+CAA+C,EAAE;gBAC5E,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE;oBACP,cAAc,EAAE,kBAAkB;oBAClC,eAAe,EAAE,UAAU,MAAM,EAAE;oBACnC,cAAc,EAAE,4BAA4B;oBAC5C,SAAS,EAAE,eAAe;iBAC3B;gBACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,KAAK;oBACL,QAAQ,EAAE;wBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,YAAY,EAAE;wBACzC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE;qBACtC;oBACD,UAAU,EAAE,IAAI;iBACjB,CAAC;gBACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC;aAC5C,CAAC,CAAC;YAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,KAAK,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;gBACpC,MAAM,IAAI,KAAK,CAAC,8BAA8B,QAAQ,CAAC,MAAM,MAAM,KAAK,EAAE,CAAC,CAAC;YAC9E,CAAC;YAED,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAOlC,CAAC;YAEF,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,IAAI,EAAE,CAAC;YACvD,MAAM,WAAW,GAAG,IAAI,CAAC,KAAK,EAAE,aAAa,IAAI,IAAI,CAAC;YACtD,MAAM,YAAY,GAAG,IAAI,CAAC,KAAK,EAAE,iBAAiB,IAAI,IAAI,CAAC;YAC3D,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,EAAE,UAAU,IAAI,IAAI,CAAC;YAC/C,OAAO;gBACL,IAAI;gBACJ,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK;gBAC9B,WAAW;gBACX,YAAY;gBACZ,IAAI,EAAE,OAAO,IAAI,aAAa,CAAC,YAAY,EAAE,KAAK,EAAE,WAAW,EAAE,YAAY,CAAC;gBAC9E,WAAW,EAAE,WAAoB;aAClC,CAAC;QACJ,CAAC;KACF,CAAC;AACJ,CAAC;AAED,8EAA8E;AAC9E,0EAA0E;AAC1E,EAAE;AACF,6EAA6E;AAC7E,6EAA6E;AAC7E,0CAA0C;AAC1C,6EAA6E;AAC7E,6EAA6E;AAC7E,yCAAyC;AACzC,EAAE;AACF,iBAAiB;AACjB,mEAAmE;AACnE,+DAA+D;AAC/D,EAAE;AACF,wEAAwE;AACxE,6EAA6E;AAC7E,sEAAsE;AACtE,gBAAgB;AAChB,8EAA8E;AAC9E,SAAS,oBAAoB,CAAC,aAAsB;IAClD,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,IAAI,0BAA0B,CAAC;IAC7E,MAAM,KAAK,GAAG,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,iBAAiB,IAAI,aAAa,CAAC;IAE9E,OAAO;QACL,KAAK;QACL,KAAK,CAAC,QAAQ,CAAC,YAAoB,EAAE,UAAkB;YACrD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAEzB,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,mBAAmB,EAAE;gBAC1D,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE;oBACP,cAAc,EAAE,kBAAkB;oBAClC,eAAe,EAAE,kBAAkB;iBACpC;gBACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;oBACnB,KAAK;oBACL,QAAQ,EAAE;wBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,YAAY,EAAE;wBACzC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE;qBACtC;oBACD,UAAU,EAAE,IAAI;iBACjB,CAAC;gBACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC;aAC5C,CAAC,CAAC;YAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,SAAS,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;gBACxC,MAAM,IAAI,KAAK,CACb,6BAA6B,QAAQ,CAAC,MAAM,MAAM,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAC5E,CAAC;YACJ,CAAC;YAED,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAMlC,CAAC;YAEF,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,IAAI,EAAE,CAAC;YACvD,MAAM,WAAW,GAAG,IAAI,CAAC,KAAK,EAAE,aAAa,IAAI,IAAI,CAAC;YACtD,MAAM,YAAY,GAAG,IAAI,CAAC,KAAK,EAAE,iBAAiB,IAAI,IAAI,CAAC;YAC3D,OAAO;gBACL,IAAI;gBACJ,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK;gBAC9B,WAAW;gBACX,YAAY;gBACZ,IAAI,EAAE,CAAC;gBACP,WAAW,EAAE,MAAe;aAC7B,CAAC;QACJ,CAAC;KACF,CAAC;AACJ,CAAC"}
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import type { Router } from "./router.js";
|
|
2
|
+
import type { ProviderName } from "../eval/llm.js";
|
|
2
3
|
import type { AgentDefinition } from "../agents/agents-registry.js";
|
|
3
4
|
export interface InstalledAgentEntry {
|
|
4
5
|
id: string;
|
|
@@ -50,4 +51,50 @@ export declare function deriveSourceAgent(skillDir: string, root: string, origin
|
|
|
50
51
|
* response never fails because of a single bad skill.
|
|
51
52
|
*/
|
|
52
53
|
export declare function buildSkillMetadata(skillDir: string, origin: "source" | "installed", root: string): SkillMetadataFields;
|
|
54
|
+
interface ModelOption {
|
|
55
|
+
id: string;
|
|
56
|
+
label: string;
|
|
57
|
+
}
|
|
58
|
+
type OpenRouterCacheEntry = {
|
|
59
|
+
value: Array<{
|
|
60
|
+
id: string;
|
|
61
|
+
name: string;
|
|
62
|
+
contextWindow?: number;
|
|
63
|
+
pricing: {
|
|
64
|
+
prompt: number;
|
|
65
|
+
completion: number;
|
|
66
|
+
};
|
|
67
|
+
}>;
|
|
68
|
+
fetchedAt: number;
|
|
69
|
+
};
|
|
70
|
+
export declare const OPENROUTER_CACHE: Map<string, OpenRouterCacheEntry>;
|
|
71
|
+
export declare function resetOpenRouterCache(): void;
|
|
72
|
+
/** Test hook: clear all probe caches so the next detectAvailableProviders() re-probes. */
|
|
73
|
+
export declare function resetDetectionCache(): void;
|
|
74
|
+
/**
|
|
75
|
+
* Detection block — surfaces wrapper-folder presence and binary availability
|
|
76
|
+
* so the UI can render accurate "installed" dots and "install me" CTAs.
|
|
77
|
+
*
|
|
78
|
+
* Shape is part of the /api/config response (the frontend types.ts is
|
|
79
|
+
* read-only per the 0682 ownership boundary, so the field is carried as
|
|
80
|
+
* opaque JSON and consumed by useAgentCatalog via its own typing).
|
|
81
|
+
*/
|
|
82
|
+
export interface DetectionInfo {
|
|
83
|
+
wrapperFolders: Record<string, boolean>;
|
|
84
|
+
binaries: Record<string, boolean>;
|
|
85
|
+
}
|
|
86
|
+
export declare function resetProjectDetectionCache(): void;
|
|
87
|
+
/**
|
|
88
|
+
* Scan the project root for known agent wrapper folders and the system
|
|
89
|
+
* PATH for known agent binaries. Cheap synchronous scan (`existsSync` +
|
|
90
|
+
* `which`) cached for 30 s so repeated `/api/config` polls don't burn CPU.
|
|
91
|
+
*/
|
|
92
|
+
export declare function detectProjectAgents(root: string): DetectionInfo;
|
|
93
|
+
export declare function detectAvailableProviders(): Promise<Array<{
|
|
94
|
+
id: ProviderName;
|
|
95
|
+
label: string;
|
|
96
|
+
available: boolean;
|
|
97
|
+
models: ModelOption[];
|
|
98
|
+
}>>;
|
|
53
99
|
export declare function registerRoutes(router: Router, root: string, projectName?: string): void;
|
|
100
|
+
export {};
|