@exercode/problem-utils 3.2.2 → 4.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -85
- package/dist/helpers/runHarnessProcess.cjs +1 -1
- package/dist/helpers/runHarnessProcess.cjs.map +1 -1
- package/dist/helpers/runHarnessProcess.js +1 -1
- package/dist/helpers/runHarnessProcess.js.map +1 -1
- package/package.json +5 -43
- package/dist/presets/llm.cjs +0 -2
- package/dist/presets/llm.cjs.map +0 -1
- package/dist/presets/llm.d.ts +0 -47
- package/dist/presets/llm.js +0 -2
- package/dist/presets/llm.js.map +0 -1
package/README.md
CHANGED
|
@@ -1,88 +1,7 @@
|
|
|
1
1
|
# @exercode/problem-utils
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
[](https://github.com/semantic-release/semantic-release)
|
|
6
|
-
[](https://github.com/WillBooster/shared/tree/main/packages/wbfy)
|
|
3
|
+
CLI, validation, result types, and stdio/command/GUI/evaluation judging presets for Exercode.
|
|
4
|
+
This package does not depend on a browser library or AI SDK providers.
|
|
7
5
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
## CLI
|
|
11
|
-
|
|
12
|
-
The package ships an `exercode-problem` command for problem authors (run it with `bun x` in a repository that depends on `@exercode/problem-utils`):
|
|
13
|
-
|
|
14
|
-
```bash
|
|
15
|
-
# Judge all model answers of all problems (directories containing problem.md or <id>.problem.md) under a directory.
|
|
16
|
-
# model_answers/* must be fully accepted; model_answers.fails/* must fail at least one test case.
|
|
17
|
-
bun x exercode-problem # everything under the current directory
|
|
18
|
-
bun x exercode-problem courses/foo # everything under courses/foo
|
|
19
|
-
bun x exercode-problem --only a_plus --skip gui_ --concurrency 2
|
|
20
|
-
|
|
21
|
-
# Judge one answer directory of the problem in the current directory.
|
|
22
|
-
bun x exercode-problem judge model_answers/python
|
|
23
|
-
bun x exercode-problem judge model_answers/python '{ "language": "python" }'
|
|
24
|
-
|
|
25
|
-
# Debug one answer directory of the problem in the current directory.
|
|
26
|
-
bun x exercode-problem debug model_answers/python '{ "stdin": "1 2" }'
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
`judge` and `debug` run a custom `judge.ts` / `debug.ts` when the problem has one, and apply `stdioJudgePreset` / `stdioDebugPreset` otherwise, mirroring the Exercode server. The debug fallback applies only to standard problems: a problem with a custom `judge.ts` needs its own `debug.ts`, and `exercode-problem debug` fails with a message otherwise (the server likewise reports debug as unsupported there).
|
|
30
|
-
|
|
31
|
-
The all-problem check judges serially by default because time limits are measured in wall-clock time; pass `--concurrency <n>` to parallelize when the checked problems are not timing-sensitive.
|
|
32
|
-
|
|
33
|
-
A standard stdin/stdout problem must NOT commit a `judge.ts` or `debug.ts` that is identical to the default stdio harness: the absence of `judge.ts` marks the problem as standard, and committed copies would drift from the server's defaults. The CLI rejects such files; a file kept intentionally (e.g. to demonstrate the default harness) can add an explanatory comment to be treated as custom.
|
|
34
|
-
|
|
35
|
-
## Validators
|
|
36
|
-
|
|
37
|
-
The CLI also validates learning-material files without running any program, mirroring the checks the Exercode importer applies:
|
|
38
|
-
|
|
39
|
-
```bash
|
|
40
|
-
# Validate problem directories (problem.md frontmatter, test_cases, model_answers, templates, judge.ts / debug.ts).
|
|
41
|
-
bun x exercode-problem validate-problem <problemDir>...
|
|
42
|
-
# Validate a course directory (course.yaml, lecture materials with embedded questions, problem references).
|
|
43
|
-
bun x exercode-problem validate-course <courseDir> [--problems-dir <dir>]
|
|
44
|
-
# Validate a contest (*.contest.yaml) file.
|
|
45
|
-
bun x exercode-problem validate-contest <contestYamlPath> [--problems-dir <dir>]
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
Each target prints `OK` or `NG` followed by its errors and warnings; the command exits 1 when any target has an error. `--problems-dir` points to the directory holding the referenced problems; a course is always searched at any depth, since Exercode links a material only to the problems inside its course, so for a course the option must name a directory inside it. The validators are also exported (`validateProblemDirectory`, `validateCourseDirectory`, `validateContestFile`, `validateMaterialFile`).
|
|
49
|
-
|
|
50
|
-
## Agent skills
|
|
51
|
-
|
|
52
|
-
The [`skills/`](skills/) directory holds skills for AI coding agents that author and review Exercode learning content: `generate-learning-content` (entry point), `generate-course-materials`, `generate-judge-problems`, `generate-judge-contest`, `review-learning-content`, and `setup-exercode-course-repository`. Agents working in this repository load them through the symlinks under `.claude/skills/`; install them elsewhere with the [skills](https://github.com/vercel-labs/skills) CLI:
|
|
53
|
-
|
|
54
|
-
```bash
|
|
55
|
-
bun x skills add WillBooster/exercode-problem-utils --agent claude-code --agent codex
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
## Test cases
|
|
59
|
-
|
|
60
|
-
A problem keeps its test cases under `test_cases/`. A test case id is the shared name of the following entries, and each entry is optional:
|
|
61
|
-
|
|
62
|
-
| Entry | Meaning |
|
|
63
|
-
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
64
|
-
| `<id>.in` | Standard input. Omit it (or leave it empty) when the program reads nothing. |
|
|
65
|
-
| `<id>.out` | Expected standard output. |
|
|
66
|
-
| `<id>.fin/` | Files copied into the working directory before the run (input files). |
|
|
67
|
-
| `<id>.fout/` | Expected output files, compared with the files of the same relative paths in the working directory. |
|
|
68
|
-
| `_shared.fin/` | Files copied into the working directory before every test case. |
|
|
69
|
-
| `<id>.json` | Configuration for a custom `judge.ts` that reads it itself; the presets ignore it (the Judge server lists it as a test case of the custom judge). |
|
|
70
|
-
|
|
71
|
-
A test case whose id contains `example` (the judge server's rule, e.g. `example_1` or `01_example_small`) is an example shown to learners; every other case is hidden.
|
|
72
|
-
|
|
73
|
-
Standard output and text files are compared as space-separated tokens: consecutive white spaces count as one separator, and an expected token that contains a decimal point and parses as a finite number (e.g. `3.14`, but not `1`, `1e-3` or `1.0e309`) accepts a value within an absolute or relative error of `1e-6`. A file is text when it is valid UTF-8 without NUL bytes; other files (e.g. images) must match byte for byte. A received file larger than 8 MiB counts as not produced, an expected file larger than 8 MiB is an authoring error (the case is reported as a runtime error), and a file larger than 1 MiB is left out of the reported pair. When a file differs, the result carries `<name>_expected.<ext>` and `<name>_received.<ext>` so Exercode can show both (Exercode decides per test case whether a learner may see them, as it does for expected stdout).
|
|
74
|
-
|
|
75
|
-
How a missing expectation is treated depends on the harness:
|
|
76
|
-
|
|
77
|
-
- `stdioJudgePreset` (the default for problems without `judge.ts`) requires `<id>.out` or a non-empty `<id>.fout/` for every test case, so a standard problem cannot accept a run without checking it. A problem whose `problem.md` declares `requiredOutputFilePaths` or `isManualScoringRequired` is exempt, because every test case is judged by those instead; code rules (`requiredRegExpsInCode` etc.) and `requiredSubmissionFilePaths` check the submission once, in addition to the output comparison, and do not exempt.
|
|
78
|
-
- `commandJudgePreset` without a `test` option checks whatever expectations exist, and a test case with neither only has to run within the limits. A `test` option replaces that comparison; it receives `testCase.output`, `testCase.fileOutputPath` and `cwd` and can call the exported `judgeAgainstExpectations` (or `compareStdoutAsSpaceSeparatedTokens` and `compareExpectedOutputFiles`). A custom `readTestCases` may return any test case type with `id` (plus optional `input` and `fileInputPath`); the default verdict judges any case that exposes a string `output` or `fileOutputPath`, which the default reader's `CommandTestCase` does.
|
|
79
|
-
- `guiCommandJudgePreset` passes the expectations to the problem's `test`, which decides everything.
|
|
80
|
-
- `llmJudgePreset` runs no program, so it copies no `.fin/`; it hands `<id>.in` as the prompt input and the whole entry (including `fileOutputPath`) to the problem's `test`.
|
|
81
|
-
- `stdioDebugPreset` by default copies the answer directory to a temporary directory and builds and runs it there together with `_shared.fin/` and the first example case's `.fin/` (the case's files win; hidden cases' inputs are never handed to learner code), so the answer directory is left untouched (its `node_modules`, and those of its ancestors, are linked into the copy rather than duplicated); files the program writes are reported only through `requiredOutputFilePaths`. Callers that already own a disposable answer directory can pass `{ disposableWorkingDirectory: true }` as the second argument to build and run directly there; they must remove it themselves, including when the harness is terminated.
|
|
82
|
-
- `evaluationJudgePreset` does not use `test_cases/`.
|
|
83
|
-
|
|
84
|
-
`readTestCases` is exported for harnesses that enumerate `test_cases/` themselves.
|
|
85
|
-
|
|
86
|
-
## Measurements
|
|
87
|
-
|
|
88
|
-
`stdioJudgePreset`, `stdioDebugPreset`, and `commandJudgePreset` (with its default runner; a custom `runCommand` decides whether its results carry `cpuTimeSeconds`) run a program under GNU time (`/usr/bin/time` on Linux, `gtime` on macOS) and report its wall time (`timeSeconds`), user plus system CPU time (`cpuTimeSeconds`), and peak resident set size (`memoryBytes`, at least the footprint of the GNU `timeout` wrapper the program runs under, about 1 MiB) in every test case result; `guiCommandJudgePreset` reports the wall time and the peak resident set size, and `llmJudgePreset` the wall time only. Time limits are judged by wall time. The CPU time is recorded even for a run that exceeded its limit, so a judge server sharing CPUs between programs can tell a program that used up its limit (its CPU time, summed over its threads, reaches the limit, so it would exceed it on any single CPU) from one that may only have waited for a CPU (its CPU time stays below the limit) and re-run just the latter alone.
|
|
6
|
+
See the [repository documentation](https://github.com/WillBooster/exercode-problem-utils)
|
|
7
|
+
for usage, problem formats, and the separate browser and LLM packages.
|
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
"use strict";const e=require("../_virtual/_rolldown/runtime.cjs"),t=require("./temporaryProblemDirCopy.cjs");let n=require("node:child_process");n=e.__toESM(n,1);const r=new Set;let i=!1;function a(e,t){return o(),new Promise(i=>{let a=n.default.spawn(process.execPath,e,{cwd:t.cwd,detached:process.platform!==`win32`,env:t.env,stdio:[`ignore`,`pipe`,`pipe`]}),o=a.pid===void 0?void 0:{pid:a.pid,tempRoot:t.tempRoot};o&&r.add(o);let c=[],l=[],u=0,d,f=!1,p=!1,m=e=>{if(d!==void 0)return!1;d=e,a.stdout?.destroy(),a.stderr?.destroy();try{a.pid===void 0?a.kill(`SIGKILL`):s(a.pid)}catch{a.kill(`SIGKILL`)}return!0},h=setTimeout(()=>{f=m(`timed out after ${t.timeoutMs/1e3} seconds`)},t.timeoutMs),g=(e,n)=>{if(u+=e.byteLength,u>t.maxOutputBytes){m(`the harness printed more than ${t.maxOutputBytes/1024/1024} MB of output`);return}n.push(e)};a.stdout?.on(`data`,e=>g(e,c)),a.stderr?.on(`data`,e=>g(e,l));let _=(e,t,n)=>{p||(p=!0,clearTimeout(h),o&&r.delete(o),i({stdout:Buffer.concat(c).toString(`utf8`),stderr:Buffer.concat(l).toString(`utf8`),exitCode:e,signal:t,failureReason:d??(n?`failed to run the harness: ${n.message}`:void 0),timedOut:f}))};a.on(`error`,e=>_(void 0,void 0,e)),a.on(`close`,(e,t)=>_(e??void 0,t??void 0))})}function o(){if(!i){i=!0;for(let e of[`SIGINT`,`SIGTERM`])process.once(e,()=>{for(let e of r){try{s(e.pid)}catch{}e.tempRoot&&t.forciblyRemoveDirectorySync(e.tempRoot)}process.kill(process.pid,e)})}}function s(e){process.platform===`win32`?n.default.spawnSync(`taskkill`,[`/PID`,String(e),`/T`,`/F`],{stdio:`ignore`}):process.kill(-e,`SIGKILL`)}exports.runHarnessProcess=a;
|
|
1
|
+
"use strict";const e=require("../_virtual/_rolldown/runtime.cjs"),t=require("./temporaryProblemDirCopy.cjs");let n=require("node:child_process");n=e.__toESM(n,1);const r=new Set;let i=!1;function a(e,t){return o(),new Promise(i=>{let a=n.default.spawn(process.execPath,e,{cwd:t.cwd,detached:process.platform!==`win32`,env:t.tempRoot?{...t.env,TMPDIR:t.tempRoot}:t.env,stdio:[`ignore`,`pipe`,`pipe`]}),o=a.pid===void 0?void 0:{pid:a.pid,tempRoot:t.tempRoot};o&&r.add(o);let c=[],l=[],u=0,d,f=!1,p=!1,m=e=>{if(d!==void 0)return!1;d=e,a.stdout?.destroy(),a.stderr?.destroy();try{a.pid===void 0?a.kill(`SIGKILL`):s(a.pid)}catch{a.kill(`SIGKILL`)}return!0},h=setTimeout(()=>{f=m(`timed out after ${t.timeoutMs/1e3} seconds`)},t.timeoutMs),g=(e,n)=>{if(u+=e.byteLength,u>t.maxOutputBytes){m(`the harness printed more than ${t.maxOutputBytes/1024/1024} MB of output`);return}n.push(e)};a.stdout?.on(`data`,e=>g(e,c)),a.stderr?.on(`data`,e=>g(e,l));let _=(e,t,n)=>{p||(p=!0,clearTimeout(h),o&&r.delete(o),i({stdout:Buffer.concat(c).toString(`utf8`),stderr:Buffer.concat(l).toString(`utf8`),exitCode:e,signal:t,failureReason:d??(n?`failed to run the harness: ${n.message}`:void 0),timedOut:f}))};a.on(`error`,e=>_(void 0,void 0,e)),a.on(`close`,(e,t)=>_(e??void 0,t??void 0))})}function o(){if(!i){i=!0;for(let e of[`SIGINT`,`SIGTERM`])process.once(e,()=>{for(let e of r){try{s(e.pid)}catch{}e.tempRoot&&t.forciblyRemoveDirectorySync(e.tempRoot)}process.kill(process.pid,e)})}}function s(e){process.platform===`win32`?n.default.spawnSync(`taskkill`,[`/PID`,String(e),`/T`,`/F`],{stdio:`ignore`}):process.kill(-e,`SIGKILL`)}exports.runHarnessProcess=a;
|
|
2
2
|
//# sourceMappingURL=runHarnessProcess.cjs.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runHarnessProcess.cjs","names":["child_process"],"sources":["../../src/helpers/runHarnessProcess.ts"],"sourcesContent":["import child_process from 'node:child_process';\n\nimport { forciblyRemoveDirectorySync } from './temporaryProblemDirCopy.js';\n\nexport interface HarnessProcessOptions {\n cwd: string;\n env: NodeJS.ProcessEnv;\n timeoutMs: number;\n maxOutputBytes: number;\n /** A temporary copy to remove when the check itself is interrupted by a signal. */\n tempRoot?: string;\n}\n\nexport interface HarnessProcessResult {\n stdout: string;\n stderr: string;\n exitCode: number | undefined;\n /** The signal that terminated the harness when it did not exit on its own. */\n signal: NodeJS.Signals | undefined;\n /** Set when the run was killed by this helper (timeout or output cap) or could not be spawned. */\n failureReason: string | undefined;\n timedOut: boolean;\n}\n\ninterface LiveHarnessRun {\n pid: number;\n tempRoot: string | undefined;\n}\n\n// Detached harness groups no longer receive the terminal's SIGINT, so an interrupted run must\n// tear them down (and remove their temp copies) itself before exiting. The handlers stay installed\n// for the process lifetime: once the last run has settled they only iterate an empty set and re-raise\n// the signal, so removing them would change nothing observable.\nconst liveHarnessRuns = new Set<LiveHarnessRun>();\nlet signalHandlersInstalled = false;\n\n/**\n * Run a judge harness with `process.execPath` in its own process group, killing the whole group on\n * timeout or when the output cap is exceeded. The submissions a harness judges through\n * `spawnWithLimits` run in sessions of their own, out of that kill's reach; each wrapper's process\n * group is bounded by its own GNU `timeout` instead, while a descendant that moved to yet another\n * session may survive without stalling the harness.\n */\nexport function runHarnessProcess(\n commandArgs: readonly string[],\n options: HarnessProcessOptions\n): Promise<HarnessProcessResult> {\n installSignalHandlers();\n return new Promise((resolve) => {\n // process.execPath keeps the harness on the same bun executable regardless of PATH.\n const child = child_process.spawn(process.execPath, commandArgs, {\n cwd: options.cwd,\n detached: process.platform !== 'win32',\n env: options.env,\n stdio: ['ignore', 'pipe', 'pipe'],\n });\n const liveRun: LiveHarnessRun | undefined =\n child.pid === undefined ? undefined : { pid: child.pid, tempRoot: options.tempRoot };\n if (liveRun) liveHarnessRuns.add(liveRun);\n const stdoutChunks: Buffer[] = [];\n const stderrChunks: Buffer[] = [];\n let totalOutputBytes = 0;\n let failureReason: string | undefined;\n let timedOut = false;\n let settled = false;\n\n // Returns false when an earlier kill already recorded its reason, so callers can tell whether theirs won.\n const killProcessGroup = (reason: string): boolean => {\n if (failureReason !== undefined) return false;\n failureReason = reason;\n // Stop buffering immediately: OS pipe buffers can keep emitting data after the kill, and\n // destroyed streams also let `close` fire even if a stray grandchild inherited the pipes.\n child.stdout?.destroy();\n child.stderr?.destroy();\n try {\n if (child.pid === undefined) {\n child.kill('SIGKILL');\n } else {\n killHarnessTree(child.pid);\n }\n } catch {\n child.kill('SIGKILL');\n }\n return true;\n };\n const timeoutId = setTimeout(() => {\n timedOut = killProcessGroup(`timed out after ${options.timeoutMs / 1000} seconds`);\n }, options.timeoutMs);\n\n const appendOutput = (chunk: Buffer, chunks: Buffer[]): void => {\n totalOutputBytes += chunk.byteLength;\n if (totalOutputBytes > options.maxOutputBytes) {\n killProcessGroup(`the harness printed more than ${options.maxOutputBytes / 1024 / 1024} MB of output`);\n return;\n }\n chunks.push(chunk);\n };\n child.stdout?.on('data', (chunk: Buffer) => appendOutput(chunk, stdoutChunks));\n child.stderr?.on('data', (chunk: Buffer) => appendOutput(chunk, stderrChunks));\n\n const settle = (exitCode: number | undefined, signal: NodeJS.Signals | undefined, spawnError?: Error): void => {\n if (settled) return;\n settled = true;\n clearTimeout(timeoutId);\n if (liveRun) liveHarnessRuns.delete(liveRun);\n resolve({\n stdout: Buffer.concat(stdoutChunks).toString('utf8'),\n stderr: Buffer.concat(stderrChunks).toString('utf8'),\n exitCode,\n signal,\n failureReason: failureReason ?? (spawnError ? `failed to run the harness: ${spawnError.message}` : undefined),\n timedOut,\n });\n };\n child.on('error', (error) => settle(undefined, undefined, error));\n child.on('close', (exitCode, signal) => settle(exitCode ?? undefined, signal ?? undefined));\n });\n}\n\nfunction installSignalHandlers(): void {\n if (signalHandlersInstalled) return;\n signalHandlersInstalled = true;\n for (const signal of ['SIGINT', 'SIGTERM'] as const) {\n process.once(signal, () => {\n for (const liveRun of liveHarnessRuns) {\n try {\n killHarnessTree(liveRun.pid);\n } catch {\n // The tree already exited.\n }\n // Best-effort cleanup while exiting; judged code may have left permission-locked entries.\n if (liveRun.tempRoot) forciblyRemoveDirectorySync(liveRun.tempRoot);\n }\n process.kill(process.pid, signal);\n });\n }\n}\n\n// On Windows the harness is not detached (process groups are unavailable), so killing only the\n// direct child would leave submission grandchildren running; taskkill terminates the whole tree.\nfunction killHarnessTree(pid: number): void {\n if (process.platform === 'win32') {\n child_process.spawnSync('taskkill', ['/PID', String(pid), '/T', '/F'], { stdio: 'ignore' });\n } else {\n process.kill(-pid, 'SIGKILL');\n }\n}\n"],"mappings":"kKAiCA,MAAM,EAAkB,IAAI,IAC5B,IAAI,EAA0B,GAS9B,SAAgB,EACd,EACA,EAC+B,CAE/B,OADA,EAAsB,EACf,IAAI,QAAS,GAAY,CAE9B,IAAM,EAAQA,EAAAA,QAAc,MAAM,QAAQ,SAAU,EAAa,CAC/D,IAAK,EAAQ,IACb,SAAU,QAAQ,WAAa,
|
|
1
|
+
{"version":3,"file":"runHarnessProcess.cjs","names":["child_process"],"sources":["../../src/helpers/runHarnessProcess.ts"],"sourcesContent":["import child_process from 'node:child_process';\n\nimport { forciblyRemoveDirectorySync } from './temporaryProblemDirCopy.js';\n\nexport interface HarnessProcessOptions {\n cwd: string;\n env: NodeJS.ProcessEnv;\n timeoutMs: number;\n maxOutputBytes: number;\n /** A temporary copy to remove when the check itself is interrupted by a signal. */\n tempRoot?: string;\n}\n\nexport interface HarnessProcessResult {\n stdout: string;\n stderr: string;\n exitCode: number | undefined;\n /** The signal that terminated the harness when it did not exit on its own. */\n signal: NodeJS.Signals | undefined;\n /** Set when the run was killed by this helper (timeout or output cap) or could not be spawned. */\n failureReason: string | undefined;\n timedOut: boolean;\n}\n\ninterface LiveHarnessRun {\n pid: number;\n tempRoot: string | undefined;\n}\n\n// Detached harness groups no longer receive the terminal's SIGINT, so an interrupted run must\n// tear them down (and remove their temp copies) itself before exiting. The handlers stay installed\n// for the process lifetime: once the last run has settled they only iterate an empty set and re-raise\n// the signal, so removing them would change nothing observable.\nconst liveHarnessRuns = new Set<LiveHarnessRun>();\nlet signalHandlersInstalled = false;\n\n/**\n * Run a judge harness with `process.execPath` in its own process group, killing the whole group on\n * timeout or when the output cap is exceeded. The submissions a harness judges through\n * `spawnWithLimits` run in sessions of their own, out of that kill's reach; each wrapper's process\n * group is bounded by its own GNU `timeout` instead, while a descendant that moved to yet another\n * session may survive without stalling the harness.\n */\nexport function runHarnessProcess(\n commandArgs: readonly string[],\n options: HarnessProcessOptions\n): Promise<HarnessProcessResult> {\n installSignalHandlers();\n return new Promise((resolve) => {\n // process.execPath keeps the harness on the same bun executable regardless of PATH.\n const child = child_process.spawn(process.execPath, commandArgs, {\n cwd: options.cwd,\n detached: process.platform !== 'win32',\n // The parent removes this root even when SIGKILL prevents child cleanup.\n env: options.tempRoot ? { ...options.env, TMPDIR: options.tempRoot } : options.env,\n stdio: ['ignore', 'pipe', 'pipe'],\n });\n const liveRun: LiveHarnessRun | undefined =\n child.pid === undefined ? undefined : { pid: child.pid, tempRoot: options.tempRoot };\n if (liveRun) liveHarnessRuns.add(liveRun);\n const stdoutChunks: Buffer[] = [];\n const stderrChunks: Buffer[] = [];\n let totalOutputBytes = 0;\n let failureReason: string | undefined;\n let timedOut = false;\n let settled = false;\n\n // Returns false when an earlier kill already recorded its reason, so callers can tell whether theirs won.\n const killProcessGroup = (reason: string): boolean => {\n if (failureReason !== undefined) return false;\n failureReason = reason;\n // Stop buffering immediately: OS pipe buffers can keep emitting data after the kill, and\n // destroyed streams also let `close` fire even if a stray grandchild inherited the pipes.\n child.stdout?.destroy();\n child.stderr?.destroy();\n try {\n if (child.pid === undefined) {\n child.kill('SIGKILL');\n } else {\n killHarnessTree(child.pid);\n }\n } catch {\n child.kill('SIGKILL');\n }\n return true;\n };\n const timeoutId = setTimeout(() => {\n timedOut = killProcessGroup(`timed out after ${options.timeoutMs / 1000} seconds`);\n }, options.timeoutMs);\n\n const appendOutput = (chunk: Buffer, chunks: Buffer[]): void => {\n totalOutputBytes += chunk.byteLength;\n if (totalOutputBytes > options.maxOutputBytes) {\n killProcessGroup(`the harness printed more than ${options.maxOutputBytes / 1024 / 1024} MB of output`);\n return;\n }\n chunks.push(chunk);\n };\n child.stdout?.on('data', (chunk: Buffer) => appendOutput(chunk, stdoutChunks));\n child.stderr?.on('data', (chunk: Buffer) => appendOutput(chunk, stderrChunks));\n\n const settle = (exitCode: number | undefined, signal: NodeJS.Signals | undefined, spawnError?: Error): void => {\n if (settled) return;\n settled = true;\n clearTimeout(timeoutId);\n if (liveRun) liveHarnessRuns.delete(liveRun);\n resolve({\n stdout: Buffer.concat(stdoutChunks).toString('utf8'),\n stderr: Buffer.concat(stderrChunks).toString('utf8'),\n exitCode,\n signal,\n failureReason: failureReason ?? (spawnError ? `failed to run the harness: ${spawnError.message}` : undefined),\n timedOut,\n });\n };\n child.on('error', (error) => settle(undefined, undefined, error));\n child.on('close', (exitCode, signal) => settle(exitCode ?? undefined, signal ?? undefined));\n });\n}\n\nfunction installSignalHandlers(): void {\n if (signalHandlersInstalled) return;\n signalHandlersInstalled = true;\n for (const signal of ['SIGINT', 'SIGTERM'] as const) {\n process.once(signal, () => {\n for (const liveRun of liveHarnessRuns) {\n try {\n killHarnessTree(liveRun.pid);\n } catch {\n // The tree already exited.\n }\n // Best-effort cleanup while exiting; judged code may have left permission-locked entries.\n if (liveRun.tempRoot) forciblyRemoveDirectorySync(liveRun.tempRoot);\n }\n process.kill(process.pid, signal);\n });\n }\n}\n\n// On Windows the harness is not detached (process groups are unavailable), so killing only the\n// direct child would leave submission grandchildren running; taskkill terminates the whole tree.\nfunction killHarnessTree(pid: number): void {\n if (process.platform === 'win32') {\n child_process.spawnSync('taskkill', ['/PID', String(pid), '/T', '/F'], { stdio: 'ignore' });\n } else {\n process.kill(-pid, 'SIGKILL');\n }\n}\n"],"mappings":"kKAiCA,MAAM,EAAkB,IAAI,IAC5B,IAAI,EAA0B,GAS9B,SAAgB,EACd,EACA,EAC+B,CAE/B,OADA,EAAsB,EACf,IAAI,QAAS,GAAY,CAE9B,IAAM,EAAQA,EAAAA,QAAc,MAAM,QAAQ,SAAU,EAAa,CAC/D,IAAK,EAAQ,IACb,SAAU,QAAQ,WAAa,QAE/B,IAAK,EAAQ,SAAW,CAAE,GAAG,EAAQ,IAAK,OAAQ,EAAQ,QAAS,EAAI,EAAQ,IAC/E,MAAO,CAAC,SAAU,OAAQ,MAAM,CAClC,CAAC,EACK,EACJ,EAAM,MAAQ,IAAA,GAAY,IAAA,GAAY,CAAE,IAAK,EAAM,IAAK,SAAU,EAAQ,QAAS,EACjF,GAAS,EAAgB,IAAI,CAAO,EACxC,IAAM,EAAyB,CAAC,EAC1B,EAAyB,CAAC,EAC5B,EAAmB,EACnB,EACA,EAAW,GACX,EAAU,GAGR,EAAoB,GAA4B,CACpD,GAAI,IAAkB,IAAA,GAAW,MAAO,GACxC,EAAgB,EAGhB,EAAM,QAAQ,QAAQ,EACtB,EAAM,QAAQ,QAAQ,EACtB,GAAI,CACE,EAAM,MAAQ,IAAA,GAChB,EAAM,KAAK,SAAS,EAEpB,EAAgB,EAAM,GAAG,CAE7B,MAAQ,CACN,EAAM,KAAK,SAAS,CACtB,CACA,MAAO,EACT,EACM,EAAY,eAAiB,CACjC,EAAW,EAAiB,mBAAmB,EAAQ,UAAY,IAAK,SAAS,CACnF,EAAG,EAAQ,SAAS,EAEd,GAAgB,EAAe,IAA2B,CAE9D,GADA,GAAoB,EAAM,WACtB,EAAmB,EAAQ,eAAgB,CAC7C,EAAiB,iCAAiC,EAAQ,eAAiB,KAAO,KAAK,cAAc,EACrG,MACF,CACA,EAAO,KAAK,CAAK,CACnB,EACA,EAAM,QAAQ,GAAG,OAAS,GAAkB,EAAa,EAAO,CAAY,CAAC,EAC7E,EAAM,QAAQ,GAAG,OAAS,GAAkB,EAAa,EAAO,CAAY,CAAC,EAE7E,IAAM,GAAU,EAA8B,EAAoC,IAA6B,CACzG,IACJ,EAAU,GACV,aAAa,CAAS,EAClB,GAAS,EAAgB,OAAO,CAAO,EAC3C,EAAQ,CACN,OAAQ,OAAO,OAAO,CAAY,CAAC,CAAC,SAAS,MAAM,EACnD,OAAQ,OAAO,OAAO,CAAY,CAAC,CAAC,SAAS,MAAM,EACnD,WACA,SACA,cAAe,IAAkB,EAAa,8BAA8B,EAAW,UAAY,IAAA,IACnG,UACF,CAAC,EACH,EACA,EAAM,GAAG,QAAU,GAAU,EAAO,IAAA,GAAW,IAAA,GAAW,CAAK,CAAC,EAChE,EAAM,GAAG,SAAU,EAAU,IAAW,EAAO,GAAY,IAAA,GAAW,GAAU,IAAA,EAAS,CAAC,CAC5F,CAAC,CACH,CAEA,SAAS,GAA8B,CACjC,MACJ,GAA0B,GAC1B,IAAK,IAAM,IAAU,CAAC,SAAU,SAAS,EACvC,QAAQ,KAAK,MAAc,CACzB,IAAK,IAAM,KAAW,EAAiB,CACrC,GAAI,CACF,EAAgB,EAAQ,GAAG,CAC7B,MAAQ,CAER,CAEI,EAAQ,UAAU,EAAA,4BAA4B,EAAQ,QAAQ,CACpE,CACA,QAAQ,KAAK,QAAQ,IAAK,CAAM,CAClC,CAAC,CAbuB,CAe5B,CAIA,SAAS,EAAgB,EAAmB,CACtC,QAAQ,WAAa,QACvB,EAAA,QAAc,UAAU,WAAY,CAAC,OAAQ,OAAO,CAAG,EAAG,KAAM,IAAI,EAAG,CAAE,MAAO,QAAS,CAAC,EAE1F,QAAQ,KAAK,CAAC,EAAK,SAAS,CAEhC"}
|
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
import{forciblyRemoveDirectorySync as e}from"./temporaryProblemDirCopy.js";import t from"node:child_process";const n=new Set;let r=!1;function i(e,r){return a(),new Promise(i=>{let a=t.spawn(process.execPath,e,{cwd:r.cwd,detached:process.platform!==`win32`,env:r.env,stdio:[`ignore`,`pipe`,`pipe`]}),s=a.pid===void 0?void 0:{pid:a.pid,tempRoot:r.tempRoot};s&&n.add(s);let c=[],l=[],u=0,d,f=!1,p=!1,m=e=>{if(d!==void 0)return!1;d=e,a.stdout?.destroy(),a.stderr?.destroy();try{a.pid===void 0?a.kill(`SIGKILL`):o(a.pid)}catch{a.kill(`SIGKILL`)}return!0},h=setTimeout(()=>{f=m(`timed out after ${r.timeoutMs/1e3} seconds`)},r.timeoutMs),g=(e,t)=>{if(u+=e.byteLength,u>r.maxOutputBytes){m(`the harness printed more than ${r.maxOutputBytes/1024/1024} MB of output`);return}t.push(e)};a.stdout?.on(`data`,e=>g(e,c)),a.stderr?.on(`data`,e=>g(e,l));let _=(e,t,r)=>{p||(p=!0,clearTimeout(h),s&&n.delete(s),i({stdout:Buffer.concat(c).toString(`utf8`),stderr:Buffer.concat(l).toString(`utf8`),exitCode:e,signal:t,failureReason:d??(r?`failed to run the harness: ${r.message}`:void 0),timedOut:f}))};a.on(`error`,e=>_(void 0,void 0,e)),a.on(`close`,(e,t)=>_(e??void 0,t??void 0))})}function a(){if(!r){r=!0;for(let t of[`SIGINT`,`SIGTERM`])process.once(t,()=>{for(let t of n){try{o(t.pid)}catch{}t.tempRoot&&e(t.tempRoot)}process.kill(process.pid,t)})}}function o(e){process.platform===`win32`?t.spawnSync(`taskkill`,[`/PID`,String(e),`/T`,`/F`],{stdio:`ignore`}):process.kill(-e,`SIGKILL`)}export{i as runHarnessProcess};
|
|
1
|
+
import{forciblyRemoveDirectorySync as e}from"./temporaryProblemDirCopy.js";import t from"node:child_process";const n=new Set;let r=!1;function i(e,r){return a(),new Promise(i=>{let a=t.spawn(process.execPath,e,{cwd:r.cwd,detached:process.platform!==`win32`,env:r.tempRoot?{...r.env,TMPDIR:r.tempRoot}:r.env,stdio:[`ignore`,`pipe`,`pipe`]}),s=a.pid===void 0?void 0:{pid:a.pid,tempRoot:r.tempRoot};s&&n.add(s);let c=[],l=[],u=0,d,f=!1,p=!1,m=e=>{if(d!==void 0)return!1;d=e,a.stdout?.destroy(),a.stderr?.destroy();try{a.pid===void 0?a.kill(`SIGKILL`):o(a.pid)}catch{a.kill(`SIGKILL`)}return!0},h=setTimeout(()=>{f=m(`timed out after ${r.timeoutMs/1e3} seconds`)},r.timeoutMs),g=(e,t)=>{if(u+=e.byteLength,u>r.maxOutputBytes){m(`the harness printed more than ${r.maxOutputBytes/1024/1024} MB of output`);return}t.push(e)};a.stdout?.on(`data`,e=>g(e,c)),a.stderr?.on(`data`,e=>g(e,l));let _=(e,t,r)=>{p||(p=!0,clearTimeout(h),s&&n.delete(s),i({stdout:Buffer.concat(c).toString(`utf8`),stderr:Buffer.concat(l).toString(`utf8`),exitCode:e,signal:t,failureReason:d??(r?`failed to run the harness: ${r.message}`:void 0),timedOut:f}))};a.on(`error`,e=>_(void 0,void 0,e)),a.on(`close`,(e,t)=>_(e??void 0,t??void 0))})}function a(){if(!r){r=!0;for(let t of[`SIGINT`,`SIGTERM`])process.once(t,()=>{for(let t of n){try{o(t.pid)}catch{}t.tempRoot&&e(t.tempRoot)}process.kill(process.pid,t)})}}function o(e){process.platform===`win32`?t.spawnSync(`taskkill`,[`/PID`,String(e),`/T`,`/F`],{stdio:`ignore`}):process.kill(-e,`SIGKILL`)}export{i as runHarnessProcess};
|
|
2
2
|
//# sourceMappingURL=runHarnessProcess.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runHarnessProcess.js","names":["child_process"],"sources":["../../src/helpers/runHarnessProcess.ts"],"sourcesContent":["import child_process from 'node:child_process';\n\nimport { forciblyRemoveDirectorySync } from './temporaryProblemDirCopy.js';\n\nexport interface HarnessProcessOptions {\n cwd: string;\n env: NodeJS.ProcessEnv;\n timeoutMs: number;\n maxOutputBytes: number;\n /** A temporary copy to remove when the check itself is interrupted by a signal. */\n tempRoot?: string;\n}\n\nexport interface HarnessProcessResult {\n stdout: string;\n stderr: string;\n exitCode: number | undefined;\n /** The signal that terminated the harness when it did not exit on its own. */\n signal: NodeJS.Signals | undefined;\n /** Set when the run was killed by this helper (timeout or output cap) or could not be spawned. */\n failureReason: string | undefined;\n timedOut: boolean;\n}\n\ninterface LiveHarnessRun {\n pid: number;\n tempRoot: string | undefined;\n}\n\n// Detached harness groups no longer receive the terminal's SIGINT, so an interrupted run must\n// tear them down (and remove their temp copies) itself before exiting. The handlers stay installed\n// for the process lifetime: once the last run has settled they only iterate an empty set and re-raise\n// the signal, so removing them would change nothing observable.\nconst liveHarnessRuns = new Set<LiveHarnessRun>();\nlet signalHandlersInstalled = false;\n\n/**\n * Run a judge harness with `process.execPath` in its own process group, killing the whole group on\n * timeout or when the output cap is exceeded. The submissions a harness judges through\n * `spawnWithLimits` run in sessions of their own, out of that kill's reach; each wrapper's process\n * group is bounded by its own GNU `timeout` instead, while a descendant that moved to yet another\n * session may survive without stalling the harness.\n */\nexport function runHarnessProcess(\n commandArgs: readonly string[],\n options: HarnessProcessOptions\n): Promise<HarnessProcessResult> {\n installSignalHandlers();\n return new Promise((resolve) => {\n // process.execPath keeps the harness on the same bun executable regardless of PATH.\n const child = child_process.spawn(process.execPath, commandArgs, {\n cwd: options.cwd,\n detached: process.platform !== 'win32',\n env: options.env,\n stdio: ['ignore', 'pipe', 'pipe'],\n });\n const liveRun: LiveHarnessRun | undefined =\n child.pid === undefined ? undefined : { pid: child.pid, tempRoot: options.tempRoot };\n if (liveRun) liveHarnessRuns.add(liveRun);\n const stdoutChunks: Buffer[] = [];\n const stderrChunks: Buffer[] = [];\n let totalOutputBytes = 0;\n let failureReason: string | undefined;\n let timedOut = false;\n let settled = false;\n\n // Returns false when an earlier kill already recorded its reason, so callers can tell whether theirs won.\n const killProcessGroup = (reason: string): boolean => {\n if (failureReason !== undefined) return false;\n failureReason = reason;\n // Stop buffering immediately: OS pipe buffers can keep emitting data after the kill, and\n // destroyed streams also let `close` fire even if a stray grandchild inherited the pipes.\n child.stdout?.destroy();\n child.stderr?.destroy();\n try {\n if (child.pid === undefined) {\n child.kill('SIGKILL');\n } else {\n killHarnessTree(child.pid);\n }\n } catch {\n child.kill('SIGKILL');\n }\n return true;\n };\n const timeoutId = setTimeout(() => {\n timedOut = killProcessGroup(`timed out after ${options.timeoutMs / 1000} seconds`);\n }, options.timeoutMs);\n\n const appendOutput = (chunk: Buffer, chunks: Buffer[]): void => {\n totalOutputBytes += chunk.byteLength;\n if (totalOutputBytes > options.maxOutputBytes) {\n killProcessGroup(`the harness printed more than ${options.maxOutputBytes / 1024 / 1024} MB of output`);\n return;\n }\n chunks.push(chunk);\n };\n child.stdout?.on('data', (chunk: Buffer) => appendOutput(chunk, stdoutChunks));\n child.stderr?.on('data', (chunk: Buffer) => appendOutput(chunk, stderrChunks));\n\n const settle = (exitCode: number | undefined, signal: NodeJS.Signals | undefined, spawnError?: Error): void => {\n if (settled) return;\n settled = true;\n clearTimeout(timeoutId);\n if (liveRun) liveHarnessRuns.delete(liveRun);\n resolve({\n stdout: Buffer.concat(stdoutChunks).toString('utf8'),\n stderr: Buffer.concat(stderrChunks).toString('utf8'),\n exitCode,\n signal,\n failureReason: failureReason ?? (spawnError ? `failed to run the harness: ${spawnError.message}` : undefined),\n timedOut,\n });\n };\n child.on('error', (error) => settle(undefined, undefined, error));\n child.on('close', (exitCode, signal) => settle(exitCode ?? undefined, signal ?? undefined));\n });\n}\n\nfunction installSignalHandlers(): void {\n if (signalHandlersInstalled) return;\n signalHandlersInstalled = true;\n for (const signal of ['SIGINT', 'SIGTERM'] as const) {\n process.once(signal, () => {\n for (const liveRun of liveHarnessRuns) {\n try {\n killHarnessTree(liveRun.pid);\n } catch {\n // The tree already exited.\n }\n // Best-effort cleanup while exiting; judged code may have left permission-locked entries.\n if (liveRun.tempRoot) forciblyRemoveDirectorySync(liveRun.tempRoot);\n }\n process.kill(process.pid, signal);\n });\n }\n}\n\n// On Windows the harness is not detached (process groups are unavailable), so killing only the\n// direct child would leave submission grandchildren running; taskkill terminates the whole tree.\nfunction killHarnessTree(pid: number): void {\n if (process.platform === 'win32') {\n child_process.spawnSync('taskkill', ['/PID', String(pid), '/T', '/F'], { stdio: 'ignore' });\n } else {\n process.kill(-pid, 'SIGKILL');\n }\n}\n"],"mappings":"6GAiCA,MAAM,EAAkB,IAAI,IAC5B,IAAI,EAA0B,GAS9B,SAAgB,EACd,EACA,EAC+B,CAE/B,OADA,EAAsB,EACf,IAAI,QAAS,GAAY,CAE9B,IAAM,EAAQA,EAAc,MAAM,QAAQ,SAAU,EAAa,CAC/D,IAAK,EAAQ,IACb,SAAU,QAAQ,WAAa,
|
|
1
|
+
{"version":3,"file":"runHarnessProcess.js","names":["child_process"],"sources":["../../src/helpers/runHarnessProcess.ts"],"sourcesContent":["import child_process from 'node:child_process';\n\nimport { forciblyRemoveDirectorySync } from './temporaryProblemDirCopy.js';\n\nexport interface HarnessProcessOptions {\n cwd: string;\n env: NodeJS.ProcessEnv;\n timeoutMs: number;\n maxOutputBytes: number;\n /** A temporary copy to remove when the check itself is interrupted by a signal. */\n tempRoot?: string;\n}\n\nexport interface HarnessProcessResult {\n stdout: string;\n stderr: string;\n exitCode: number | undefined;\n /** The signal that terminated the harness when it did not exit on its own. */\n signal: NodeJS.Signals | undefined;\n /** Set when the run was killed by this helper (timeout or output cap) or could not be spawned. */\n failureReason: string | undefined;\n timedOut: boolean;\n}\n\ninterface LiveHarnessRun {\n pid: number;\n tempRoot: string | undefined;\n}\n\n// Detached harness groups no longer receive the terminal's SIGINT, so an interrupted run must\n// tear them down (and remove their temp copies) itself before exiting. The handlers stay installed\n// for the process lifetime: once the last run has settled they only iterate an empty set and re-raise\n// the signal, so removing them would change nothing observable.\nconst liveHarnessRuns = new Set<LiveHarnessRun>();\nlet signalHandlersInstalled = false;\n\n/**\n * Run a judge harness with `process.execPath` in its own process group, killing the whole group on\n * timeout or when the output cap is exceeded. The submissions a harness judges through\n * `spawnWithLimits` run in sessions of their own, out of that kill's reach; each wrapper's process\n * group is bounded by its own GNU `timeout` instead, while a descendant that moved to yet another\n * session may survive without stalling the harness.\n */\nexport function runHarnessProcess(\n commandArgs: readonly string[],\n options: HarnessProcessOptions\n): Promise<HarnessProcessResult> {\n installSignalHandlers();\n return new Promise((resolve) => {\n // process.execPath keeps the harness on the same bun executable regardless of PATH.\n const child = child_process.spawn(process.execPath, commandArgs, {\n cwd: options.cwd,\n detached: process.platform !== 'win32',\n // The parent removes this root even when SIGKILL prevents child cleanup.\n env: options.tempRoot ? { ...options.env, TMPDIR: options.tempRoot } : options.env,\n stdio: ['ignore', 'pipe', 'pipe'],\n });\n const liveRun: LiveHarnessRun | undefined =\n child.pid === undefined ? undefined : { pid: child.pid, tempRoot: options.tempRoot };\n if (liveRun) liveHarnessRuns.add(liveRun);\n const stdoutChunks: Buffer[] = [];\n const stderrChunks: Buffer[] = [];\n let totalOutputBytes = 0;\n let failureReason: string | undefined;\n let timedOut = false;\n let settled = false;\n\n // Returns false when an earlier kill already recorded its reason, so callers can tell whether theirs won.\n const killProcessGroup = (reason: string): boolean => {\n if (failureReason !== undefined) return false;\n failureReason = reason;\n // Stop buffering immediately: OS pipe buffers can keep emitting data after the kill, and\n // destroyed streams also let `close` fire even if a stray grandchild inherited the pipes.\n child.stdout?.destroy();\n child.stderr?.destroy();\n try {\n if (child.pid === undefined) {\n child.kill('SIGKILL');\n } else {\n killHarnessTree(child.pid);\n }\n } catch {\n child.kill('SIGKILL');\n }\n return true;\n };\n const timeoutId = setTimeout(() => {\n timedOut = killProcessGroup(`timed out after ${options.timeoutMs / 1000} seconds`);\n }, options.timeoutMs);\n\n const appendOutput = (chunk: Buffer, chunks: Buffer[]): void => {\n totalOutputBytes += chunk.byteLength;\n if (totalOutputBytes > options.maxOutputBytes) {\n killProcessGroup(`the harness printed more than ${options.maxOutputBytes / 1024 / 1024} MB of output`);\n return;\n }\n chunks.push(chunk);\n };\n child.stdout?.on('data', (chunk: Buffer) => appendOutput(chunk, stdoutChunks));\n child.stderr?.on('data', (chunk: Buffer) => appendOutput(chunk, stderrChunks));\n\n const settle = (exitCode: number | undefined, signal: NodeJS.Signals | undefined, spawnError?: Error): void => {\n if (settled) return;\n settled = true;\n clearTimeout(timeoutId);\n if (liveRun) liveHarnessRuns.delete(liveRun);\n resolve({\n stdout: Buffer.concat(stdoutChunks).toString('utf8'),\n stderr: Buffer.concat(stderrChunks).toString('utf8'),\n exitCode,\n signal,\n failureReason: failureReason ?? (spawnError ? `failed to run the harness: ${spawnError.message}` : undefined),\n timedOut,\n });\n };\n child.on('error', (error) => settle(undefined, undefined, error));\n child.on('close', (exitCode, signal) => settle(exitCode ?? undefined, signal ?? undefined));\n });\n}\n\nfunction installSignalHandlers(): void {\n if (signalHandlersInstalled) return;\n signalHandlersInstalled = true;\n for (const signal of ['SIGINT', 'SIGTERM'] as const) {\n process.once(signal, () => {\n for (const liveRun of liveHarnessRuns) {\n try {\n killHarnessTree(liveRun.pid);\n } catch {\n // The tree already exited.\n }\n // Best-effort cleanup while exiting; judged code may have left permission-locked entries.\n if (liveRun.tempRoot) forciblyRemoveDirectorySync(liveRun.tempRoot);\n }\n process.kill(process.pid, signal);\n });\n }\n}\n\n// On Windows the harness is not detached (process groups are unavailable), so killing only the\n// direct child would leave submission grandchildren running; taskkill terminates the whole tree.\nfunction killHarnessTree(pid: number): void {\n if (process.platform === 'win32') {\n child_process.spawnSync('taskkill', ['/PID', String(pid), '/T', '/F'], { stdio: 'ignore' });\n } else {\n process.kill(-pid, 'SIGKILL');\n }\n}\n"],"mappings":"6GAiCA,MAAM,EAAkB,IAAI,IAC5B,IAAI,EAA0B,GAS9B,SAAgB,EACd,EACA,EAC+B,CAE/B,OADA,EAAsB,EACf,IAAI,QAAS,GAAY,CAE9B,IAAM,EAAQA,EAAc,MAAM,QAAQ,SAAU,EAAa,CAC/D,IAAK,EAAQ,IACb,SAAU,QAAQ,WAAa,QAE/B,IAAK,EAAQ,SAAW,CAAE,GAAG,EAAQ,IAAK,OAAQ,EAAQ,QAAS,EAAI,EAAQ,IAC/E,MAAO,CAAC,SAAU,OAAQ,MAAM,CAClC,CAAC,EACK,EACJ,EAAM,MAAQ,IAAA,GAAY,IAAA,GAAY,CAAE,IAAK,EAAM,IAAK,SAAU,EAAQ,QAAS,EACjF,GAAS,EAAgB,IAAI,CAAO,EACxC,IAAM,EAAyB,CAAC,EAC1B,EAAyB,CAAC,EAC5B,EAAmB,EACnB,EACA,EAAW,GACX,EAAU,GAGR,EAAoB,GAA4B,CACpD,GAAI,IAAkB,IAAA,GAAW,MAAO,GACxC,EAAgB,EAGhB,EAAM,QAAQ,QAAQ,EACtB,EAAM,QAAQ,QAAQ,EACtB,GAAI,CACE,EAAM,MAAQ,IAAA,GAChB,EAAM,KAAK,SAAS,EAEpB,EAAgB,EAAM,GAAG,CAE7B,MAAQ,CACN,EAAM,KAAK,SAAS,CACtB,CACA,MAAO,EACT,EACM,EAAY,eAAiB,CACjC,EAAW,EAAiB,mBAAmB,EAAQ,UAAY,IAAK,SAAS,CACnF,EAAG,EAAQ,SAAS,EAEd,GAAgB,EAAe,IAA2B,CAE9D,GADA,GAAoB,EAAM,WACtB,EAAmB,EAAQ,eAAgB,CAC7C,EAAiB,iCAAiC,EAAQ,eAAiB,KAAO,KAAK,cAAc,EACrG,MACF,CACA,EAAO,KAAK,CAAK,CACnB,EACA,EAAM,QAAQ,GAAG,OAAS,GAAkB,EAAa,EAAO,CAAY,CAAC,EAC7E,EAAM,QAAQ,GAAG,OAAS,GAAkB,EAAa,EAAO,CAAY,CAAC,EAE7E,IAAM,GAAU,EAA8B,EAAoC,IAA6B,CACzG,IACJ,EAAU,GACV,aAAa,CAAS,EAClB,GAAS,EAAgB,OAAO,CAAO,EAC3C,EAAQ,CACN,OAAQ,OAAO,OAAO,CAAY,CAAC,CAAC,SAAS,MAAM,EACnD,OAAQ,OAAO,OAAO,CAAY,CAAC,CAAC,SAAS,MAAM,EACnD,WACA,SACA,cAAe,IAAkB,EAAa,8BAA8B,EAAW,UAAY,IAAA,IACnG,UACF,CAAC,EACH,EACA,EAAM,GAAG,QAAU,GAAU,EAAO,IAAA,GAAW,IAAA,GAAW,CAAK,CAAC,EAChE,EAAM,GAAG,SAAU,EAAU,IAAW,EAAO,GAAY,IAAA,GAAW,GAAU,IAAA,EAAS,CAAC,CAC5F,CAAC,CACH,CAEA,SAAS,GAA8B,CACjC,MACJ,GAA0B,GAC1B,IAAK,IAAM,IAAU,CAAC,SAAU,SAAS,EACvC,QAAQ,KAAK,MAAc,CACzB,IAAK,IAAM,KAAW,EAAiB,CACrC,GAAI,CACF,EAAgB,EAAQ,GAAG,CAC7B,MAAQ,CAER,CAEI,EAAQ,UAAU,EAA4B,EAAQ,QAAQ,CACpE,CACA,QAAQ,KAAK,QAAQ,IAAK,CAAM,CAClC,CAAC,CAbuB,CAe5B,CAIA,SAAS,EAAgB,EAAmB,CACtC,QAAQ,WAAa,QACvB,EAAc,UAAU,WAAY,CAAC,OAAQ,OAAO,CAAG,EAAG,KAAM,IAAI,EAAG,CAAE,MAAO,QAAS,CAAC,EAE1F,QAAQ,KAAK,CAAC,EAAK,SAAS,CAEhC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@exercode/problem-utils",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "4.0.0",
|
|
4
4
|
"description": ":100: A set of utilities for judging programs on Exercode (https://exercode.willbooster.com/).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"exercode",
|
|
@@ -8,7 +8,8 @@
|
|
|
8
8
|
],
|
|
9
9
|
"repository": {
|
|
10
10
|
"type": "git",
|
|
11
|
-
"url": "git+https://github.com/WillBooster/exercode-problem-utils.git"
|
|
11
|
+
"url": "git+https://github.com/WillBooster/exercode-problem-utils.git",
|
|
12
|
+
"directory": "packages/problem-utils"
|
|
12
13
|
},
|
|
13
14
|
"license": "Apache-2.0",
|
|
14
15
|
"author": "WillBooster Inc.",
|
|
@@ -37,55 +38,16 @@
|
|
|
37
38
|
"dist/"
|
|
38
39
|
],
|
|
39
40
|
"scripts": {
|
|
40
|
-
"build": "build-ts lib --input src/index.ts src/presets/*.ts src/cli/*.ts"
|
|
41
|
-
"cleanup": "bun wb lint --fix --format",
|
|
42
|
-
"format": "bun wb lint --format",
|
|
43
|
-
"lint": "bun wb lint",
|
|
44
|
-
"lint-fix": "bun wb lint --fix",
|
|
45
|
-
"prepare": "(bun lefthook install || true) && (test ! -f .lefthook/normalize-bun-lockfile.sh || bash .lefthook/normalize-bun-lockfile.sh)",
|
|
46
|
-
"start": "bun src/index.ts",
|
|
47
|
-
"test": "bun wb test",
|
|
48
|
-
"test/ci": "bun wb test-on-ci",
|
|
49
|
-
"test/ci-setup": "bun run build && bun install --cwd example && cd example && bun run puppeteer browsers install chrome",
|
|
50
|
-
"typecheck": "bun wb typecheck",
|
|
51
|
-
"verify": "bun wb verify",
|
|
52
|
-
"verify-full": "bun wb verify --full"
|
|
41
|
+
"build": "build-ts lib --input src/index.ts src/presets/*.ts src/cli/*.ts"
|
|
53
42
|
},
|
|
54
43
|
"prettier": "@willbooster/prettier-config",
|
|
55
44
|
"dependencies": {
|
|
56
|
-
"@ai-sdk/amazon-bedrock": "5.0.67",
|
|
57
|
-
"@ai-sdk/google": "4.0.57",
|
|
58
|
-
"@ai-sdk/openai": "4.0.51",
|
|
59
|
-
"@ai-sdk/xai": "4.0.49",
|
|
60
|
-
"ai": "7.0.84",
|
|
61
45
|
"front-matter": "4.0.2",
|
|
62
46
|
"yaml": "2.9.0",
|
|
63
47
|
"zod": "4.5.2"
|
|
64
48
|
},
|
|
65
49
|
"devDependencies": {
|
|
66
|
-
"
|
|
67
|
-
"@tsconfig/node-lts": "24.0.1",
|
|
68
|
-
"@tsconfig/node-ts": "23.6.4",
|
|
69
|
-
"@tsconfig/node24": "24.0.5",
|
|
70
|
-
"@types/bun": "1.4.0",
|
|
71
|
-
"@types/node": "26.4.0",
|
|
72
|
-
"@willbooster/oxfmt-config": "1.2.2",
|
|
73
|
-
"@willbooster/oxlint-config": "1.4.8",
|
|
74
|
-
"@willbooster/prettier-config": "10.4.0",
|
|
75
|
-
"@willbooster/wb": "22.1.4",
|
|
76
|
-
"build-ts": "21.0.12",
|
|
77
|
-
"conventional-changelog-conventionalcommits": "10.4.0",
|
|
78
|
-
"lefthook": "2.1.12",
|
|
79
|
-
"oxfmt": "0.65.0",
|
|
80
|
-
"oxlint": "1.80.0",
|
|
81
|
-
"oxlint-tsgolint": "7.0.2001",
|
|
82
|
-
"prettier": "3.9.6",
|
|
83
|
-
"prettier-plugin-java": "2.10.3",
|
|
84
|
-
"puppeteer": "25.9.0",
|
|
85
|
-
"semantic-release": "25.0.9",
|
|
86
|
-
"sort-package-json": "4.0.0",
|
|
87
|
-
"typescript": "7.0.2",
|
|
88
|
-
"vitest": "4.1.11"
|
|
50
|
+
"build-ts": "21.0.12"
|
|
89
51
|
},
|
|
90
52
|
"engines": {
|
|
91
53
|
"node": ">=24"
|
package/dist/presets/llm.cjs
DELETED
|
@@ -1,2 +0,0 @@
|
|
|
1
|
-
"use strict";Object.defineProperty(exports,Symbol.toStringTag,{value:`Module`});const e=require("../_virtual/_rolldown/runtime.cjs"),t=require("../helpers/printTestCaseResult.cjs"),n=require("../helpers/parseArgs.cjs"),r=require("../helpers/readTestCases.cjs"),i=require("../types/decisionCode.cjs");let a=require("node:fs");a=e.__toESM(a,1);let o=require("node:path");o=e.__toESM(o,1);let s=require("zod"),c=require("@ai-sdk/amazon-bedrock"),l=require("@ai-sdk/google"),u=require("@ai-sdk/openai"),d=require("@ai-sdk/xai"),f=require("ai");const p={bedrock:c.bedrock,google:l.google,openai:u.openai,xai:d.xai},m=s.z.object({model:s.z.string().min(1)});async function h(e,s){let c=n.parseArgs(process.argv);if(!c.cwd)throw Error(`cwd argument required`);let l=m.parse(c.params),u=await r.readTestCases(o.default.join(e,`test_cases`)),d=await a.default.promises.readFile(o.default.join(c.cwd,`prompt.txt`),`utf8`);for(let e of u){let n=Date.now();try{let{text:r}=await(0,f.generateText)({model:_(l.model),...g(s.buildPrompt?.({prompt:d,testCase:e})??d.replaceAll(`{input}`,e.input??``))}),a=Date.now(),o={testCaseId:e.id,decisionCode:i.DecisionCode.ACCEPTED,stdin:e.input,stdout:r,timeSeconds:(a-n)/1e3,...await s.test({testCase:e,result:{output:r}})};if(t.printTestCaseResult(o),o.decisionCode!==i.DecisionCode.ACCEPTED)break}catch(r){let a=Date.now();t.printTestCaseResult({testCaseId:e.id,decisionCode:i.DecisionCode.RUNTIME_ERROR,stdin:e.input,stderr:r instanceof Error?r.message:String(r),timeSeconds:(a-n)/1e3});break}}}function g(e){return typeof e==`string`?{prompt:e}:{messages:e,allowSystemInMessages:!0}}function _(e){let[t,n]=e.split(`/`);if(!t||!n)throw Error(`bad model: ${e}`);let r=p[t]?.(n);if(!r)throw Error(`model not found: ${e}`);return r}exports.llmJudgePreset=h;
|
|
2
|
-
//# sourceMappingURL=llm.cjs.map
|
package/dist/presets/llm.cjs.map
DELETED
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"file":"llm.cjs","names":["z","parseArgs","readTestCases","path","fs","generateText","DecisionCode"],"sources":["../../src/presets/llm.ts"],"sourcesContent":["import fs from 'node:fs';\nimport path from 'node:path';\n\nimport { bedrock } from '@ai-sdk/amazon-bedrock';\nimport { google } from '@ai-sdk/google';\nimport { openai } from '@ai-sdk/openai';\nimport { xai } from '@ai-sdk/xai';\nimport type { LanguageModel, ModelMessage } from 'ai';\nimport { generateText } from 'ai';\nimport { z } from 'zod';\n\nimport { parseArgs } from '../helpers/parseArgs.js';\nimport { printTestCaseResult } from '../helpers/printTestCaseResult.js';\nimport { readTestCases } from '../helpers/readTestCases.js';\nimport { DecisionCode } from '../types/decisionCode.js';\nimport type { TestCaseResult } from '../types/testCaseResult.js';\n\nconst PROMPT_FILENAME = 'prompt.txt';\n\nconst providerByName: Record<string, typeof bedrock | typeof google | typeof openai | typeof xai> = {\n // requires `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`\n bedrock,\n // requires `GOOGLE_GENERATIVE_AI_API_KEY`\n google,\n // requires `OPENAI_API_KEY`\n openai,\n // requires `XAI_API_KEY`\n xai,\n} as const;\n\nconst judgeParamsSchema = z.object({\n model: z.string().min(1),\n});\n\n/** A `test_cases/` entry; the preset uses `input` for the prompt and hands everything to `test`. */\ninterface LlmTestCase {\n id: string;\n input?: string;\n output?: string;\n /** Present when `<id>.fin/` exists; the preset does not copy it anywhere (there is no program run). */\n fileInputPath?: string;\n /** Present when `<id>.fout/` exists, for `test` to read. */\n fileOutputPath?: string;\n}\n\ninterface LlmJudgePresetOptions {\n buildPrompt?: (context: { prompt: string; testCase: LlmTestCase }) => string | ModelMessage[];\n test: (context: {\n testCase: LlmTestCase;\n result: { output: string };\n }) => Partial<TestCaseResult> | Promise<Partial<TestCaseResult>>;\n}\n\n/**\n * A preset judge function for running and testing a user prompt in LLM.\n *\n * @example\n * Create `judge.ts`:\n * ```ts\n * import { llmJudgePreset } from '@exercode/problem-utils/presets/llm';\n * import { DecisionCode } from '@exercode/problem-utils';\n *\n * await llmJudgePreset(import.meta.dirname, {\n * test: (context) {\n * return { decisionCode: context.result.output ? DecisionCode.ACCEPTED : DecisionCode.WRONG_ANSWER };\n * }\n * });\n * ```\n *\n * Run with the required parameters:\n * ```bash\n * bun judge.ts model_answers/java '{ \"model\": \"gemini-2.5-flash-lite\" }'\n * ```\n */\nexport async function llmJudgePreset(problemDir: string, options: LlmJudgePresetOptions): Promise<void> {\n const args = parseArgs(process.argv);\n if (!args.cwd) throw new Error('cwd argument required');\n const params = judgeParamsSchema.parse(args.params);\n\n const testCases = await readTestCases(path.join(problemDir, 'test_cases'));\n\n const prompt = await fs.promises.readFile(path.join(args.cwd, PROMPT_FILENAME), 'utf8');\n\n for (const testCase of testCases) {\n const startTimeMilliseconds = Date.now();\n try {\n const { text } = await generateText({\n model: toLanguageModel(params.model),\n ...toPromptOptions(\n options.buildPrompt?.({ prompt, testCase }) ?? prompt.replaceAll('{input}', testCase.input ?? '')\n ),\n });\n\n const stopTimeMilliseconds = Date.now();\n\n const testCaseResult = {\n testCaseId: testCase.id,\n decisionCode: DecisionCode.ACCEPTED,\n stdin: testCase.input,\n stdout: text,\n timeSeconds: (stopTimeMilliseconds - startTimeMilliseconds) / 1000,\n ...(await options.test({ testCase, result: { output: text } })),\n };\n\n printTestCaseResult(testCaseResult);\n\n if (testCaseResult.decisionCode !== DecisionCode.ACCEPTED) break;\n } catch (error) {\n const stopTimeMilliseconds = Date.now();\n\n printTestCaseResult({\n testCaseId: testCase.id,\n decisionCode: DecisionCode.RUNTIME_ERROR,\n stdin: testCase.input,\n stderr: error instanceof Error ? error.message : String(error),\n timeSeconds: (stopTimeMilliseconds - startTimeMilliseconds) / 1000,\n });\n\n break;\n }\n }\n}\n\n/** `ai` rejects system messages inside `messages` unless told that a `buildPrompt` may place them there. */\nfunction toPromptOptions(\n built: string | ModelMessage[]\n): { prompt: string } | { messages: ModelMessage[]; allowSystemInMessages: true } {\n return typeof built === 'string' ? { prompt: built } : { messages: built, allowSystemInMessages: true };\n}\n\nfunction toLanguageModel(model: string): LanguageModel {\n const [providerId, modelId] = model.split('/');\n if (!providerId || !modelId) throw new Error(`bad model: ${model}`);\n const languageModel = providerByName[providerId]?.(modelId);\n if (!languageModel) throw new Error(`model not found: ${model}`);\n return languageModel;\n}\n"],"mappings":"4hBAiBA,MAEM,EAA8F,CAElG,QAAA,EAAA,QAEA,OAAA,EAAA,OAEA,OAAA,EAAA,OAEA,IAAA,EAAA,GACF,EAEM,EAAoBA,EAAAA,EAAE,OAAO,CACjC,MAAOA,EAAAA,EAAE,OAAO,CAAC,CAAC,IAAI,CAAC,CACzB,CAAC,EA0CD,eAAsB,EAAe,EAAoB,EAA+C,CACtG,IAAM,EAAOC,EAAAA,UAAU,QAAQ,IAAI,EACnC,GAAI,CAAC,EAAK,IAAK,MAAU,MAAM,uBAAuB,EACtD,IAAM,EAAS,EAAkB,MAAM,EAAK,MAAM,EAE5C,EAAY,MAAMC,EAAAA,cAAcC,EAAAA,QAAK,KAAK,EAAY,YAAY,CAAC,EAEnE,EAAS,MAAMC,EAAAA,QAAG,SAAS,SAASD,EAAAA,QAAK,KAAK,EAAK,IAAK,YAAe,EAAG,MAAM,EAEtF,IAAK,IAAM,KAAY,EAAW,CAChC,IAAM,EAAwB,KAAK,IAAI,EACvC,GAAI,CACF,GAAM,CAAE,QAAS,MAAA,EAAME,EAAAA,aAAAA,CAAa,CAClC,MAAO,EAAgB,EAAO,KAAK,EACnC,GAAG,EACD,EAAQ,cAAc,CAAE,SAAQ,UAAS,CAAC,GAAK,EAAO,WAAW,UAAW,EAAS,OAAS,EAAE,CAClG,CACF,CAAC,EAEK,EAAuB,KAAK,IAAI,EAEhC,EAAiB,CACrB,WAAY,EAAS,GACrB,aAAcC,EAAAA,aAAa,SAC3B,MAAO,EAAS,MAChB,OAAQ,EACR,aAAc,EAAuB,GAAyB,IAC9D,GAAI,MAAM,EAAQ,KAAK,CAAE,WAAU,OAAQ,CAAE,OAAQ,CAAK,CAAE,CAAC,CAC/D,EAIA,GAFA,EAAA,oBAAoB,CAAc,EAE9B,EAAe,eAAiBA,EAAAA,aAAa,SAAU,KAC7D,OAAS,EAAO,CACd,IAAM,EAAuB,KAAK,IAAI,EAEtC,EAAA,oBAAoB,CAClB,WAAY,EAAS,GACrB,aAAcA,EAAAA,aAAa,cAC3B,MAAO,EAAS,MAChB,OAAQ,aAAiB,MAAQ,EAAM,QAAU,OAAO,CAAK,EAC7D,aAAc,EAAuB,GAAyB,GAChE,CAAC,EAED,KACF,CACF,CACF,CAGA,SAAS,EACP,EACgF,CAChF,OAAO,OAAO,GAAU,SAAW,CAAE,OAAQ,CAAM,EAAI,CAAE,SAAU,EAAO,sBAAuB,EAAK,CACxG,CAEA,SAAS,EAAgB,EAA8B,CACrD,GAAM,CAAC,EAAY,GAAW,EAAM,MAAM,GAAG,EAC7C,GAAI,CAAC,GAAc,CAAC,EAAS,MAAU,MAAM,cAAc,GAAO,EAClE,IAAM,EAAgB,EAAe,EAAW,GAAG,CAAO,EAC1D,GAAI,CAAC,EAAe,MAAU,MAAM,oBAAoB,GAAO,EAC/D,OAAO,CACT"}
|
package/dist/presets/llm.d.ts
DELETED
|
@@ -1,47 +0,0 @@
|
|
|
1
|
-
import type { ModelMessage } from 'ai';
|
|
2
|
-
import type { TestCaseResult } from '../types/testCaseResult.js';
|
|
3
|
-
/** A `test_cases/` entry; the preset uses `input` for the prompt and hands everything to `test`. */
|
|
4
|
-
interface LlmTestCase {
|
|
5
|
-
id: string;
|
|
6
|
-
input?: string;
|
|
7
|
-
output?: string;
|
|
8
|
-
/** Present when `<id>.fin/` exists; the preset does not copy it anywhere (there is no program run). */
|
|
9
|
-
fileInputPath?: string;
|
|
10
|
-
/** Present when `<id>.fout/` exists, for `test` to read. */
|
|
11
|
-
fileOutputPath?: string;
|
|
12
|
-
}
|
|
13
|
-
interface LlmJudgePresetOptions {
|
|
14
|
-
buildPrompt?: (context: {
|
|
15
|
-
prompt: string;
|
|
16
|
-
testCase: LlmTestCase;
|
|
17
|
-
}) => string | ModelMessage[];
|
|
18
|
-
test: (context: {
|
|
19
|
-
testCase: LlmTestCase;
|
|
20
|
-
result: {
|
|
21
|
-
output: string;
|
|
22
|
-
};
|
|
23
|
-
}) => Partial<TestCaseResult> | Promise<Partial<TestCaseResult>>;
|
|
24
|
-
}
|
|
25
|
-
/**
|
|
26
|
-
* A preset judge function for running and testing a user prompt in LLM.
|
|
27
|
-
*
|
|
28
|
-
* @example
|
|
29
|
-
* Create `judge.ts`:
|
|
30
|
-
* ```ts
|
|
31
|
-
* import { llmJudgePreset } from '@exercode/problem-utils/presets/llm';
|
|
32
|
-
* import { DecisionCode } from '@exercode/problem-utils';
|
|
33
|
-
*
|
|
34
|
-
* await llmJudgePreset(import.meta.dirname, {
|
|
35
|
-
* test: (context) {
|
|
36
|
-
* return { decisionCode: context.result.output ? DecisionCode.ACCEPTED : DecisionCode.WRONG_ANSWER };
|
|
37
|
-
* }
|
|
38
|
-
* });
|
|
39
|
-
* ```
|
|
40
|
-
*
|
|
41
|
-
* Run with the required parameters:
|
|
42
|
-
* ```bash
|
|
43
|
-
* bun judge.ts model_answers/java '{ "model": "gemini-2.5-flash-lite" }'
|
|
44
|
-
* ```
|
|
45
|
-
*/
|
|
46
|
-
export declare function llmJudgePreset(problemDir: string, options: LlmJudgePresetOptions): Promise<void>;
|
|
47
|
-
export {};
|
package/dist/presets/llm.js
DELETED
|
@@ -1,2 +0,0 @@
|
|
|
1
|
-
import{printTestCaseResult as e}from"../helpers/printTestCaseResult.js";import{parseArgs as t}from"../helpers/parseArgs.js";import{readTestCases as n}from"../helpers/readTestCases.js";import{DecisionCode as r}from"../types/decisionCode.js";import i from"node:fs";import a from"node:path";import{z as o}from"zod";import{bedrock as s}from"@ai-sdk/amazon-bedrock";import{google as c}from"@ai-sdk/google";import{openai as l}from"@ai-sdk/openai";import{xai as u}from"@ai-sdk/xai";import{generateText as d}from"ai";const f={bedrock:s,google:c,openai:l,xai:u},p=o.object({model:o.string().min(1)});async function m(o,s){let c=t(process.argv);if(!c.cwd)throw Error(`cwd argument required`);let l=p.parse(c.params),u=await n(a.join(o,`test_cases`)),f=await i.promises.readFile(a.join(c.cwd,`prompt.txt`),`utf8`);for(let t of u){let n=Date.now();try{let{text:i}=await d({model:g(l.model),...h(s.buildPrompt?.({prompt:f,testCase:t})??f.replaceAll(`{input}`,t.input??``))}),a=Date.now(),o={testCaseId:t.id,decisionCode:r.ACCEPTED,stdin:t.input,stdout:i,timeSeconds:(a-n)/1e3,...await s.test({testCase:t,result:{output:i}})};if(e(o),o.decisionCode!==r.ACCEPTED)break}catch(i){let a=Date.now();e({testCaseId:t.id,decisionCode:r.RUNTIME_ERROR,stdin:t.input,stderr:i instanceof Error?i.message:String(i),timeSeconds:(a-n)/1e3});break}}}function h(e){return typeof e==`string`?{prompt:e}:{messages:e,allowSystemInMessages:!0}}function g(e){let[t,n]=e.split(`/`);if(!t||!n)throw Error(`bad model: ${e}`);let r=f[t]?.(n);if(!r)throw Error(`model not found: ${e}`);return r}export{m as llmJudgePreset};
|
|
2
|
-
//# sourceMappingURL=llm.js.map
|
package/dist/presets/llm.js.map
DELETED
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"file":"llm.js","names":[],"sources":["../../src/presets/llm.ts"],"sourcesContent":["import fs from 'node:fs';\nimport path from 'node:path';\n\nimport { bedrock } from '@ai-sdk/amazon-bedrock';\nimport { google } from '@ai-sdk/google';\nimport { openai } from '@ai-sdk/openai';\nimport { xai } from '@ai-sdk/xai';\nimport type { LanguageModel, ModelMessage } from 'ai';\nimport { generateText } from 'ai';\nimport { z } from 'zod';\n\nimport { parseArgs } from '../helpers/parseArgs.js';\nimport { printTestCaseResult } from '../helpers/printTestCaseResult.js';\nimport { readTestCases } from '../helpers/readTestCases.js';\nimport { DecisionCode } from '../types/decisionCode.js';\nimport type { TestCaseResult } from '../types/testCaseResult.js';\n\nconst PROMPT_FILENAME = 'prompt.txt';\n\nconst providerByName: Record<string, typeof bedrock | typeof google | typeof openai | typeof xai> = {\n // requires `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`\n bedrock,\n // requires `GOOGLE_GENERATIVE_AI_API_KEY`\n google,\n // requires `OPENAI_API_KEY`\n openai,\n // requires `XAI_API_KEY`\n xai,\n} as const;\n\nconst judgeParamsSchema = z.object({\n model: z.string().min(1),\n});\n\n/** A `test_cases/` entry; the preset uses `input` for the prompt and hands everything to `test`. */\ninterface LlmTestCase {\n id: string;\n input?: string;\n output?: string;\n /** Present when `<id>.fin/` exists; the preset does not copy it anywhere (there is no program run). */\n fileInputPath?: string;\n /** Present when `<id>.fout/` exists, for `test` to read. */\n fileOutputPath?: string;\n}\n\ninterface LlmJudgePresetOptions {\n buildPrompt?: (context: { prompt: string; testCase: LlmTestCase }) => string | ModelMessage[];\n test: (context: {\n testCase: LlmTestCase;\n result: { output: string };\n }) => Partial<TestCaseResult> | Promise<Partial<TestCaseResult>>;\n}\n\n/**\n * A preset judge function for running and testing a user prompt in LLM.\n *\n * @example\n * Create `judge.ts`:\n * ```ts\n * import { llmJudgePreset } from '@exercode/problem-utils/presets/llm';\n * import { DecisionCode } from '@exercode/problem-utils';\n *\n * await llmJudgePreset(import.meta.dirname, {\n * test: (context) {\n * return { decisionCode: context.result.output ? DecisionCode.ACCEPTED : DecisionCode.WRONG_ANSWER };\n * }\n * });\n * ```\n *\n * Run with the required parameters:\n * ```bash\n * bun judge.ts model_answers/java '{ \"model\": \"gemini-2.5-flash-lite\" }'\n * ```\n */\nexport async function llmJudgePreset(problemDir: string, options: LlmJudgePresetOptions): Promise<void> {\n const args = parseArgs(process.argv);\n if (!args.cwd) throw new Error('cwd argument required');\n const params = judgeParamsSchema.parse(args.params);\n\n const testCases = await readTestCases(path.join(problemDir, 'test_cases'));\n\n const prompt = await fs.promises.readFile(path.join(args.cwd, PROMPT_FILENAME), 'utf8');\n\n for (const testCase of testCases) {\n const startTimeMilliseconds = Date.now();\n try {\n const { text } = await generateText({\n model: toLanguageModel(params.model),\n ...toPromptOptions(\n options.buildPrompt?.({ prompt, testCase }) ?? prompt.replaceAll('{input}', testCase.input ?? '')\n ),\n });\n\n const stopTimeMilliseconds = Date.now();\n\n const testCaseResult = {\n testCaseId: testCase.id,\n decisionCode: DecisionCode.ACCEPTED,\n stdin: testCase.input,\n stdout: text,\n timeSeconds: (stopTimeMilliseconds - startTimeMilliseconds) / 1000,\n ...(await options.test({ testCase, result: { output: text } })),\n };\n\n printTestCaseResult(testCaseResult);\n\n if (testCaseResult.decisionCode !== DecisionCode.ACCEPTED) break;\n } catch (error) {\n const stopTimeMilliseconds = Date.now();\n\n printTestCaseResult({\n testCaseId: testCase.id,\n decisionCode: DecisionCode.RUNTIME_ERROR,\n stdin: testCase.input,\n stderr: error instanceof Error ? error.message : String(error),\n timeSeconds: (stopTimeMilliseconds - startTimeMilliseconds) / 1000,\n });\n\n break;\n }\n }\n}\n\n/** `ai` rejects system messages inside `messages` unless told that a `buildPrompt` may place them there. */\nfunction toPromptOptions(\n built: string | ModelMessage[]\n): { prompt: string } | { messages: ModelMessage[]; allowSystemInMessages: true } {\n return typeof built === 'string' ? { prompt: built } : { messages: built, allowSystemInMessages: true };\n}\n\nfunction toLanguageModel(model: string): LanguageModel {\n const [providerId, modelId] = model.split('/');\n if (!providerId || !modelId) throw new Error(`bad model: ${model}`);\n const languageModel = providerByName[providerId]?.(modelId);\n if (!languageModel) throw new Error(`model not found: ${model}`);\n return languageModel;\n}\n"],"mappings":"6fAiBA,MAEM,EAA8F,CAElG,UAEA,SAEA,SAEA,KACF,EAEM,EAAoB,EAAE,OAAO,CACjC,MAAO,EAAE,OAAO,CAAC,CAAC,IAAI,CAAC,CACzB,CAAC,EA0CD,eAAsB,EAAe,EAAoB,EAA+C,CACtG,IAAM,EAAO,EAAU,QAAQ,IAAI,EACnC,GAAI,CAAC,EAAK,IAAK,MAAU,MAAM,uBAAuB,EACtD,IAAM,EAAS,EAAkB,MAAM,EAAK,MAAM,EAE5C,EAAY,MAAM,EAAc,EAAK,KAAK,EAAY,YAAY,CAAC,EAEnE,EAAS,MAAM,EAAG,SAAS,SAAS,EAAK,KAAK,EAAK,IAAK,YAAe,EAAG,MAAM,EAEtF,IAAK,IAAM,KAAY,EAAW,CAChC,IAAM,EAAwB,KAAK,IAAI,EACvC,GAAI,CACF,GAAM,CAAE,QAAS,MAAM,EAAa,CAClC,MAAO,EAAgB,EAAO,KAAK,EACnC,GAAG,EACD,EAAQ,cAAc,CAAE,SAAQ,UAAS,CAAC,GAAK,EAAO,WAAW,UAAW,EAAS,OAAS,EAAE,CAClG,CACF,CAAC,EAEK,EAAuB,KAAK,IAAI,EAEhC,EAAiB,CACrB,WAAY,EAAS,GACrB,aAAc,EAAa,SAC3B,MAAO,EAAS,MAChB,OAAQ,EACR,aAAc,EAAuB,GAAyB,IAC9D,GAAI,MAAM,EAAQ,KAAK,CAAE,WAAU,OAAQ,CAAE,OAAQ,CAAK,CAAE,CAAC,CAC/D,EAIA,GAFA,EAAoB,CAAc,EAE9B,EAAe,eAAiB,EAAa,SAAU,KAC7D,OAAS,EAAO,CACd,IAAM,EAAuB,KAAK,IAAI,EAEtC,EAAoB,CAClB,WAAY,EAAS,GACrB,aAAc,EAAa,cAC3B,MAAO,EAAS,MAChB,OAAQ,aAAiB,MAAQ,EAAM,QAAU,OAAO,CAAK,EAC7D,aAAc,EAAuB,GAAyB,GAChE,CAAC,EAED,KACF,CACF,CACF,CAGA,SAAS,EACP,EACgF,CAChF,OAAO,OAAO,GAAU,SAAW,CAAE,OAAQ,CAAM,EAAI,CAAE,SAAU,EAAO,sBAAuB,EAAK,CACxG,CAEA,SAAS,EAAgB,EAA8B,CACrD,GAAM,CAAC,EAAY,GAAW,EAAM,MAAM,GAAG,EAC7C,GAAI,CAAC,GAAc,CAAC,EAAS,MAAU,MAAM,cAAc,GAAO,EAClE,IAAM,EAAgB,EAAe,EAAW,GAAG,CAAO,EAC1D,GAAI,CAAC,EAAe,MAAU,MAAM,oBAAoB,GAAO,EAC/D,OAAO,CACT"}
|