oh-my-knowledge 1.0.0-beta.12 → 1.0.0-beta.13

package/README.md

@@ -12,12 +12,14 @@
 
 OMK is a toolkit for observing, evaluating, and improving knowledge in AI applications. Inspect how real tasks ran, extract reviewable knowledge items from work logs, and measure changes to prompts, RAG, skills, agents, or workflows to inform adoption, rollback, and further improvement.
 
-**Observe** preserves real behavior and its sources. **Measure** compares knowledge artifacts with the same model, cases, and execution conditions. **Know** connects decisions back to evidence. If you already have two artifact versions, start directly with evaluation; logs are not a prerequisite.
+**Observe** preserves real behavior and its sources. **Measure** fixes models and cases, controls and records execution conditions, and compares knowledge artifacts. **Know** connects decisions back to evidence. If you already have two artifact versions, start directly with evaluation; logs are not a prerequisite.
 
 Version 1.0 is still **in Beta iteration**; APIs and storage contracts may change. Read the [migration guide](docs/guides/v1-preview-migration.md) before upgrading an older installation.
 
 ![OMK: from controlled evaluation to real-world feedback](./docs/public/omk-knowledge-flow-en-animated.gif)
 
+The diagram describes composable workflows: review knowledge and manually apply it to an artifact. Complete links across these stages and an automated end-to-end workflow remain partly unimplemented.
+
 ## Choose how to use OMK
 
 | Interface | Use it for |
@@ -35,6 +37,7 @@ Version 1.0 is still **in Beta iteration**; APIs and storage contracts may chang
 | Preserve knowledge from work | [Extract knowledge from logs](docs/guides/extract-knowledge.md) | Candidate knowledge with sources and conditions; revise, retain, or discard it |
 | Record a confirmed knowledge problem | [MCP feedback](docs/guides/mcp-integration.md) | An observation for review; draft a case after confirming a real issue |
 | Decide whether to adopt a change | [Evaluation quickstart](docs/quickstart-skill-eval.md) | A version decision, uncertainty, failed cases, and scoring evidence |
+| Verify that an agent completed a task | [Local task acceptance](docs/guides/local-tasks.md) | Modified files, independent checks, and diagnostics after failure, timeout, or cancellation |
 | Try improving a skill automatically | [Iterative improvement](docs/guides/auto-improve-skills.md) | Candidate versions screened through controlled comparisons |
 | Embed evaluation in a service or platform | [Node.js service](docs/guides/eval-runtime.md) · [Platform host](docs/guides/platform-host-integration.md) | Composable execution, scoring, and comparison interfaces |
 
@@ -46,6 +49,8 @@ Installation requires Node.js >=22. Model calls also require an authenticated ru
 npm i -g oh-my-knowledge@next
 ```
 
+Local task acceptance requires `1.0.0-beta.13` or later. Check `omk --version` after installation, then follow the [local-task guide](docs/guides/local-tasks.md) to run a trusted example.
+
 ### Inspect an existing task
 
 ```bash
@@ -74,6 +79,12 @@ The scaffold contains two skills and three cases. This small default set checks
 
 The [full walkthrough](docs/quickstart-skill-eval.md) covers runtime selection, your own skills, and interpreting results. The [example gallery](examples/README.md) offers more runnable scenarios.
 
+### Verify the actual file changes
+
+Local tasks run two knowledge variants from declared file snapshots, collect files, and check them with an independent verifier. Failures, timeouts, and cancellations also retain declared files and partial logs. This measures explicitly supplied knowledge content, not native skill discovery. Tasks and verifiers must be trusted; workspace copies are not security sandboxes.
+
+The [cart example with an npm dependency](examples/local-task/project/README.md) records one real Codex run: treatment changed the target source and passed six checks; the negative control, instructed not to edit, passed three. This establishes a working execution and acceptance path, not general knowledge effectiveness. Model settings, usage, and durations are recorded in the example. The [local-task guide](docs/guides/local-tasks.md) covers installation, configuration, and evidence access.
+
 ## Use inside agents and DSH
 
 ### Agent Skill and MCP
@@ -116,9 +127,9 @@ A knowledge item is a reviewable fact, experience, or method. An artifact is a p
 
 These paths compose; they are not a mandatory linear state machine. Passing an evaluation does not mean knowledge has been written or a version published.
 
-Conclusions depend on the cases, scoring criteria, and execution environment. Observation signals are not causal conclusions, and generated cases are not an independent release-validation set. See [statistical rigor](docs/explanation/statistical-rigor.md) and the [three-stage workflow](docs/explanation/three-stage-workflow.md) for the method and limits.
+Fixed configuration cannot guarantee identical external model services or rerun outcomes. Conclusions depend on the cases, scoring criteria, and execution environment. Observation signals are not causal conclusions, and generated cases are not an independent release-validation set. See [statistical rigor](docs/explanation/statistical-rigor.md) and the [three-stage workflow](docs/explanation/three-stage-workflow.md) for the method and limits.
 
-Project evaluation and observation evidence lives under `.omk/`; machine-level state lives under `~/.oh-my-knowledge/`. Extracted knowledge items live in the explicitly selected knowledge workspace. The current version neither reads nor migrates the old storage layout. Back up before upgrading and follow the [migration guide](docs/guides/v1-preview-migration.md) to establish new evidence.
+Project evaluation and observation evidence lives under `.omk/`; local tasks default to `.omk/eval`, with `--output` selecting another report and diagnostic directory. Machine-level state lives under `~/.oh-my-knowledge/`. Extracted knowledge items live in the explicitly selected knowledge workspace. The current version neither reads nor migrates the old storage layout. Back up before upgrading and follow the [migration guide](docs/guides/v1-preview-migration.md) to establish new evidence.
 
 ## Documentation
 
@@ -150,6 +161,7 @@ This tool is designed for **local trusted environments** (dev machines, CI pipel
 | Feature | Risk | Scope |
 |---|---|---|
 | **Custom assertions** (`custom`) | dynamically loads and executes user-specified `.mjs` files | only use assertion files you authored or reviewed |
+| **Local tasks and verifiers** | Executors and independent verifiers run local code; raw artifacts / logs may contain sensitive information | Use reviewed tasks, inspect file lists and output directories; copies are not security sandboxes |
 | **eval-samples.json** | assertion configs can reference external file paths | don't use sample files from untrusted sources |
 
 **Recommendations:**

package/README.zh.md

@@ -12,12 +12,14 @@
 
 OMK 是面向 AI 应用作者与维护者的知识观测、评测与改进工具体系。查看真实任务如何执行，从工作日志提炼可复核的知识内容，再测量 prompt、RAG、skill、agent 或 workflow 的载体改动是否有效，为采纳、回退和继续改进提供证据。
 
-**Observe** 保留真实表现与来源，**Measure** 在相同模型、用例与运行条件下比较知识载体，**Know** 让决定可以回到证据核对。已有两版载体时，可以直接开始评测；不必先收集日志。
+**Observe** 保留真实表现与来源，**Measure** 固定模型与用例，控制并记录运行条件，比较知识载体，**Know** 让决定可以回到证据核对。已有两版载体时，可以直接开始评测；不必先收集日志。
 
 当前 1.0 仍处于 **Beta 迭代期**，接口与存储契约可能继续变化。从旧版本升级前，请先阅读[迁移指南](docs/zh/guides/v1-preview-migration.md)。
 
 ![OMK：从受控评测到真实使用反馈](./docs/public/omk-knowledge-flow-animated.gif)
 
+图示描述可组合的工作路径：知识需复核并人工落实为载体改动，完整关联与自动闭环仍有待实现部分。
+
 ## 选择使用方式
 
 | 入口 | 适合什么场景 |
@@ -35,6 +37,7 @@ OMK 是面向 AI 应用作者与维护者的知识观测、评测与改进工具
 | 把工作经验沉淀为知识 | [从日志提炼知识](docs/zh/guides/extract-knowledge.md) | 带来源和适用条件的候选知识，支持修订、保留和舍弃 |
 | 记录已确认的知识问题 | [MCP 主动反馈](docs/zh/guides/mcp-integration.md) | 待复核 observation；确认真实问题后再草拟用例 |
 | 判断一版改动是否值得采用 | [评测快速上手](docs/zh/quickstart-skill-eval.md) | 版本判定、不确定性、失败用例和评分依据 |
+| 验证 Agent 是否真正完成任务 | [本地任务验收](docs/zh/guides/local-tasks.md) | 修改文件、独立验收结果，以及失败／超时／取消后的现场证据 |
 | 自动尝试改进 skill | [迭代改进](docs/zh/guides/auto-improve-skills.md) | 经受控比较筛选的候选版本 |
 | 将评测接入服务或平台 | [Node.js 服务](docs/zh/guides/eval-runtime.md) · [平台宿主](docs/zh/guides/platform-host-integration.md) | 可组合的执行、评分与比较接口 |
 
@@ -46,6 +49,8 @@ OMK 是面向 AI 应用作者与维护者的知识观测、评测与改进工具
 npm i -g oh-my-knowledge@next
 ```
 
+本地任务验收需要 `1.0.0-beta.13` 或更新版本。安装后用 `omk --version` 核对版本，再按[本地任务指南](docs/zh/guides/local-tasks.md)运行可信示例。
+
 ### 先看已有任务
 
 ```bash
@@ -74,6 +79,12 @@ omk eval --control code-review-v1 --treatment code-review-v2
 
 [完整教程](docs/zh/quickstart-skill-eval.md)介绍模型选择、换成自己的 skill 和结果解读；[示例画廊](examples/README.zh.md)提供更多可运行场景。
 
+### 验证文件是否真的改对了
+
+本地任务从声明的文件快照执行两版知识，采集文件并交给独立验收程序检查；失败、超时或取消也保存声明文件和部分日志。它测量明确提供知识内容后的效果，不测宿主原生 skill 发现。任务与验收程序必须可信，工作区副本不是安全沙箱。
+
+[带 npm 依赖的购物车案例](examples/local-task/project/README.zh.md)记录了一次真实 Codex 运行：treatment 修改目标源码并通过六项检查，要求不编辑的负对照通过三项。该案例证明执行与验收链路可用，不证明知识版本的普遍收益；模型、用量和耗时见案例记录。[本地任务指南](docs/zh/guides/local-tasks.md)提供安装、配置与证据读取入口。
+
 ## 在 Agent 与 DSH 中使用
 
 ### Agent Skill 与 MCP
@@ -116,9 +127,9 @@ dsh --profile web
 
 这是可组合的工作路径，不是强制的单向状态机。一次评测通过也不等于知识已写入或版本已发布。
 
-结论受用例、评分准则和执行环境约束。观测信号不等于因果结论，生成的样本也不能直接充当独立发布验证集。原理与限制见[统计严谨性](docs/zh/explanation/statistical-rigor.md)和[三阶段工作流](docs/zh/explanation/three-stage-workflow.md)。
+固定配置不能保证外部模型服务或重跑结果完全一致。结论受用例、评分准则和执行环境约束。观测信号不等于因果结论，生成的样本也不能直接充当独立发布验证集。原理与限制见[统计严谨性](docs/zh/explanation/statistical-rigor.md)和[三阶段工作流](docs/zh/explanation/three-stage-workflow.md)。
 
-项目评测与观测证据保存在 `.omk/`，机器级状态位于 `~/.oh-my-knowledge/`；提炼的知识内容保存在显式选择的知识工作区。当前版本不读取或迁移旧存储布局；升级前备份，按[迁移指南](docs/zh/guides/v1-preview-migration.md)重新建立证据。
+项目评测与观测证据保存在 `.omk/`，本地任务默认写入 `.omk/eval`，也可通过 `--output` 指定报告与诊断目录。机器级状态位于 `~/.oh-my-knowledge/`；提炼的知识内容保存在显式选择的知识工作区。当前版本不读取或迁移旧存储布局；升级前备份，按[迁移指南](docs/zh/guides/v1-preview-migration.md)重新建立证据。
 
 ## 文档
 
@@ -150,6 +161,7 @@ dsh --profile web
 | 功能 | 风险 | 适用场景 |
 |------|------|----------|
 | **自定义断言**（`custom`） | 动态加载并执行用户指定的 `.mjs` 文件 | 仅使用自己编写或审查过的断言文件 |
+| **本地任务与验收程序** | 执行器与独立验收会运行本地代码，原始产物／日志可能含敏感信息 | 仅运行已审查的任务，核对文件清单与输出目录；副本不提供安全沙箱 |
 | **eval-samples.json** | 断言配置可引用外部文件路径 | 不要使用来源不明的用例文件 |
 
 **建议：**

package/dist/assets/agent-skills/omk/references/commands.md

@@ -247,6 +247,26 @@ omk eval gold validate <dir> [flags]
 
 - `--lang` `option`:输出语言 zh|en，优先级 CLI > OMK_LANG env > 全局设置 > 系统 locale > zh。
 
+## omk eval task
+
+从固定快照执行可信本地任务 A/B，采集产物并独立验收。
+
+**用法:**
+
+```bash
+omk eval task <definition> [flags]
+```
+
+**参数:**
+
+- `definition`(必填):本地任务 YAML／JSON 定义。
+
+**Flags:**
+
+- `--dry-run` `boolean`:预览封存计划，不执行任务。
+- `--lang` `option`:输出语言 zh|en，优先级 CLI > OMK_LANG env > 全局设置 > 系统 locale > zh。
+- `--output` `option` (默认 `.omk/eval`):报告输出目录。
+
 ## omk evolve
 
 自动迭代改进 skill:多轮 eval + skill 重写，直到达到 --target 或耗尽 --rounds。

package/dist/cli/commands/eval/task.d.ts

@@ -0,0 +1,13 @@
+import { BaseCommand } from '../../oclif/base-command.js';
+export default class EvalTask extends BaseCommand {
+    static description: string;
+    static args: {
+        definition: import("@oclif/core/interfaces").Arg<string, Record<string, unknown>>;
+    };
+    static flags: {
+        lang: import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        'dry-run': import("@oclif/core/interfaces").BooleanFlag<boolean>;
+        output: import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
+    };
+    run(): Promise<void>;
+}

package/dist/cli/commands/eval/task.js

@@ -0,0 +1,42 @@
+import { Args, Flags } from '@oclif/core';
+import { mkdtemp, rm } from 'node:fs/promises';
+import { tmpdir, homedir } from 'node:os';
+import { join, resolve } from 'node:path';
+import { BaseCommand } from '../../oclif/base-command.js';
+import { LANG_FLAG, bilingual } from '../../oclif/i18n.js';
+import { prepareLocalTask } from '../../../eval-workflows/hosts/local-task/application.js';
+export default class EvalTask extends BaseCommand {
+    static description = bilingual({ zh: '从固定快照执行可信本地任务 A/B，采集产物并独立验收。', en: 'Run trusted local task A/B from a fixed snapshot, collect artifacts and verify outcomes independently.' });
+    static args = { definition: Args.string({ required: true, description: bilingual({ zh: '本地任务 YAML／JSON 定义。', en: 'Local task YAML/JSON definition.' }) }) };
+    static flags = {
+        lang: LANG_FLAG,
+        'dry-run': Flags.boolean({ description: bilingual({ zh: '预览封存计划，不执行任务。', en: 'Preview the sealed plan without executing tasks.' }) }),
+        output: Flags.string({ default: '.omk/eval', description: bilingual({ zh: '报告输出目录。', en: 'Report output directory.' }) }),
+    };
+    async run() {
+        const { args, flags } = await this.parse(EvalTask);
+        const temporaryRoot = await mkdtemp(join(tmpdir(), 'omk-local-task-'));
+        try {
+            await this.runWithCancellation(async (signal) => {
+                const application = await prepareLocalTask({ definitionPath: resolve(args.definition), temporaryRoot,
+                    outputDirectory: resolve(flags.output),
+                    authenticationFile: join(process.env.CODEX_HOME ?? join(homedir(), '.codex'), 'auth.json'),
+                });
+                if (flags['dry-run']) {
+                    this.log(JSON.stringify(application.preview, null, 2));
+                    return;
+                }
+                process.stderr.write(`${JSON.stringify(application.preview, null, 2)}\n`);
+                const { result, artifacts, reference, diagnosticsDirectory, diagnosticReferences } = await application.run(signal);
+                this.log(JSON.stringify({ runId: result.runId, outputDirectory: resolve(flags.output), reference, diagnosticsDirectory, diagnosticReferences, report: artifacts.report }, null, 2));
+                if (result.status !== 'completed' || artifacts.execution.records.some((record) => record.executionStatus !== 'completed')
+                    || artifacts.evaluation.records.some((record) => record.evaluationStatus !== 'completed'
+                        || record.observations.some((observation) => observation.observationStatus !== 'observed')))
+                    this.exit(1);
+            });
+        }
+        finally {
+            await rm(temporaryRoot, { recursive: true, force: true });
+        }
+    }
+}

package/dist/eval-workflows/hosts/local-task/acceptance.d.ts

@@ -0,0 +1,13 @@
+import type { CustomEvaluator } from '../../../eval-runtime/custom-evaluator.js';
+import { type LocalTaskDefinition, type TaskFile } from '../../inputs/contracts/local-task.js';
+import type { TaskAttemptDiagnostic } from './diagnostics.js';
+import { type CapturedIdentityFile } from '../adapters/shared/content-identity.js';
+export declare const TASK_METRIC_ID = "task-accepted";
+export declare function createTaskAcceptance(input: {
+    definition: LocalTaskDefinition['acceptance'];
+    files: readonly TaskFile[];
+    requiredArtifacts: readonly string[];
+    temporaryRoot: string;
+    runtimeFiles: readonly CapturedIdentityFile[];
+    captureDiagnostic: (diagnostic: TaskAttemptDiagnostic) => Promise<void>;
+}): CustomEvaluator;

package/dist/eval-workflows/hosts/local-task/acceptance.js

@@ -0,0 +1,101 @@
+import { mkdtemp, mkdir, rm } from 'node:fs/promises';
+import { join } from 'node:path';
+import { z } from 'zod';
+import { ExecutionPortFailure } from '../../../eval-core/execution/index.js';
+import { spawnWithSigintPropagation } from '../../../executors/core/subprocess.js';
+import { LocalTaskAcceptanceSchema, LocalTaskOutputSchema } from '../../inputs/contracts/local-task.js';
+import { materializeTaskFiles, taskTreeDigest } from './files.js';
+import { assertIdentityFilesUnchanged } from '../adapters/shared/content-identity.js';
+export const TASK_METRIC_ID = 'task-accepted';
+export function createTaskAcceptance(input) {
+    const acceptanceDigest = taskTreeDigest(input.files);
+    const evidence = (value) => ({ value, classification: 'sensitive' });
+    const missing = (reasonCode) => ({
+        resultKind: 'completed', results: [{ metricId: TASK_METRIC_ID, resultKind: 'missing', reasonCode,
+                evidence: evidence({ acceptanceDigest, status: reasonCode }) }],
+    });
+    return {
+        evaluatorKind: 'custom', evaluatorId: 'local-task-acceptance', instrumentId: 'local-task-acceptance-v1',
+        metrics: [{ metricId: TASK_METRIC_ID, valueType: 'boolean', direction: 'higher-is-better', missingPolicyId: 'exclude/v1' }],
+        bindings: [{ bindingId: 'actual', sourceKind: 'output', pointer: '' }],
+        implementation: {
+            implementationId: 'omk.local-task-acceptance/v1', version: '1.1.0',
+            schemas: {
+                bindings: z.strictObject({ actual: LocalTaskOutputSchema }),
+                values: { [TASK_METRIC_ID]: z.boolean() },
+                fingerprintFacets: { output: 'omk.local-task-output/v1', acceptance: 'omk.local-task-acceptance/v1' },
+            },
+            fingerprintFacets: { acceptanceDigest, nodeVersion: process.version, entrypoint: input.definition.entrypoint,
+                runtimeFiles: input.runtimeFiles.map(({ facetId, digest, size }) => ({ facetId, digest, size })),
+                platform: process.platform, architecture: process.arch,
+                environment: { LANG: 'C', TZ: 'UTC', inheritance: 'none', home: 'acceptance-private' },
+                timeoutMs: input.definition.timeoutMs, requiredArtifacts: [...input.requiredArtifacts].sort() },
+            async evaluate({ bindings, signal }) {
+                const actual = LocalTaskOutputSchema.parse(bindings.actual);
+                if (actual.collectionErrors.length > 0 || actual.missing.length > 0
+                    || input.requiredArtifacts.some((path) => !actual.files.some((file) => file.path === path))) {
+                    return missing('TASK_ARTIFACTS_MISSING');
+                }
+                signal.throwIfAborted();
+                const root = await mkdtemp(join(input.temporaryRoot, 'acceptance-'));
+                let result;
+                let stdout = '';
+                let stderr = '';
+                let processOutcome = null;
+                let failureCode;
+                try {
+                    await assertIdentityFilesUnchanged(input.runtimeFiles, { adapterLabel: 'Local task acceptance',
+                        cancellationCode: 'TASK_ACCEPTANCE_CANCELLED', identityChangedCode: 'TASK_ACCEPTANCE_RUNTIME_CHANGED', signal });
+                    const artifacts = join(root, 'artifacts');
+                    const verifier = join(root, 'verifier');
+                    await mkdir(artifacts, { mode: 0o700 });
+                    await mkdir(verifier, { mode: 0o700 });
+                    await materializeTaskFiles(artifacts, actual.files);
+                    await materializeTaskFiles(verifier, input.files);
+                    const { child, done } = spawnWithSigintPropagation(process.execPath, [join(verifier, input.definition.entrypoint)], {
+                        cwd: artifacts, env: { HOME: root, TMPDIR: root, LANG: 'C', TZ: 'UTC' }, timeoutMs: input.definition.timeoutMs,
+                        abortSignal: signal, maxBuffer: 256 * 1024,
+                    });
+                    child.stdin?.on('error', () => { });
+                    child.stdin?.end(JSON.stringify({ sampleId: actual.source.sampleId }));
+                    const completed = await done;
+                    stdout = completed.stdout;
+                    stderr = completed.stderr;
+                    processOutcome = { exitCode: completed.code, signal: completed.signal, failureKind: null };
+                    const acceptance = LocalTaskAcceptanceSchema.safeParse(JSON.parse(completed.stdout));
+                    if (!acceptance.success) {
+                        failureCode = 'TASK_ACCEPTANCE_INVALID';
+                        result = missing(failureCode);
+                    }
+                    else
+                        result = { resultKind: 'completed', results: [{
+                                    metricId: TASK_METRIC_ID, resultKind: 'score', value: acceptance.data.passed,
+                                    evidence: evidence({ acceptanceDigest, ...acceptance.data }),
+                                }] };
+                }
+                catch (error) {
+                    const failure = error;
+                    stdout = failure.stdout ?? stdout;
+                    stderr = failure.stderr ?? stderr;
+                    if (failure.failureKind)
+                        processOutcome = { exitCode: failure.code ?? null, signal: failure.signal ?? null, failureKind: failure.failureKind };
+                    failureCode = signal.aborted ? 'TASK_ACCEPTANCE_CANCELLED'
+                        : failure.killedByTimeout ? 'TASK_ACCEPTANCE_TIMEOUT'
+                            : error instanceof ExecutionPortFailure ? 'TASK_ACCEPTANCE_RUNTIME_CHANGED'
+                                : error instanceof SyntaxError ? 'TASK_ACCEPTANCE_INVALID' : 'TASK_ACCEPTANCE_FAILED';
+                    result = missing(failureCode);
+                }
+                try {
+                    await rm(root, { recursive: true, force: true });
+                }
+                catch {
+                    failureCode = 'TASK_ACCEPTANCE_CLEANUP_FAILED';
+                    result = missing(failureCode);
+                }
+                if (failureCode)
+                    await input.captureDiagnostic({ stage: 'acceptance', status: failureCode, output: actual, stdout, stderr, processOutcome });
+                return result;
+            },
+        },
+    };
+}

package/dist/eval-workflows/hosts/local-task/application.d.ts

@@ -0,0 +1,99 @@
+import { type JsonValue } from '../../../eval-core/contracts/index.js';
+import type { EvaluationResult } from '../../../eval-runtime/evaluation/contracts.js';
+import { type StoredCoreRunArtifacts } from '../../artifact-store/index.js';
+import { type LocalTaskDefinition } from '../../inputs/contracts/local-task.js';
+export interface LocalTaskOptions {
+    readonly definitionPath: string;
+    readonly temporaryRoot: string;
+    readonly outputDirectory: string;
+    /** Read only for explicit live runs; copied into an attempt-private home, never persisted. */
+    readonly authenticationFile?: string;
+}
+/** Host application shared by CLI and programmatic users. All locators remain outside identities. */
+export declare function prepareLocalTask(options: LocalTaskOptions): Promise<Readonly<{
+    preview: {
+        projectionKind: string;
+        taskId: string;
+        knowledgeMode: string;
+        runtimeKind: "codex" | "fixture";
+        runtimeVersion: string;
+        model: string;
+        effort: string;
+        executionTimeoutMs: number;
+        acceptanceTimeoutMs: number;
+        toolPolicy: string;
+        strongIsolation: boolean;
+        diagnosticsPolicy: string;
+        maxStreamBytes: number;
+        runtimeFiles: {
+            facetId: string;
+            digest: `sha256:${string}`;
+            size: number;
+        }[];
+        snapshotDigest: `sha256:${string}`;
+        acceptanceDigest: `sha256:${string}`;
+        snapshotFiles: string[];
+        artifactFiles: string[];
+        maxArtifactBytes: number;
+        samples: {
+            annotations?: JsonValue | undefined;
+            sampleId: string;
+        }[];
+        trust: "trusted-local";
+        writeBoundary: string;
+        retryState: string;
+        estimatedCostUSD: null;
+        estimatedWork: {
+            uncertain: ("provider-cost" | "active-duration" | "wall-clock" | "retries" | "early-termination")[];
+            sampleCount: number;
+            variantCount: number;
+            trialCount: number;
+            executionCoordinates: number;
+            evaluationCoordinates: number;
+            plannedInvocations: number;
+        };
+        limitations: string[];
+    };
+    run(signal?: AbortSignal): Promise<{
+        result: {
+            status: "completed" | "cancelled" | "budget-exhausted";
+            artifacts: import("../../../eval-core/facade.js").EvaluationRunArtifacts;
+            report: import("../../../eval-core/contracts/artifacts.js").EvaluationReport;
+        } & Readonly<{
+            runId: string;
+            definition: import("../../../eval-runtime/evaluation/contracts.js").PreparedEvaluationPlan["definition"];
+            policy: import("../../../eval-runtime/evaluation/contracts.js").PreparedEvaluationPlan["measurementPolicy"];
+            analysisResults: Readonly<Record<string, import("../../../eval-core/contracts/artifacts.js").AnalysisRecord>>;
+        }>;
+        artifacts: StoredCoreRunArtifacts;
+        reference: {
+            mediaType: string;
+            digest: string;
+            size?: number | undefined;
+            uri?: string | undefined;
+        };
+        diagnosticsDirectory: string;
+        diagnosticReferences: {
+            mediaType: string;
+            digest: string;
+            size?: number | undefined;
+            uri?: string | undefined;
+        }[];
+    }>;
+    rescore(source: EvaluationResult, acceptance: LocalTaskDefinition["acceptance"], root: string, signal?: AbortSignal): Promise<{
+        result: EvaluationResult;
+        reference: {
+            mediaType: string;
+            digest: string;
+            size?: number | undefined;
+            uri?: string | undefined;
+        };
+        diagnosticsDirectory: string;
+        diagnosticReferences: {
+            mediaType: string;
+            digest: string;
+            size?: number | undefined;
+            uri?: string | undefined;
+        }[];
+    }>;
+}>>;

package/dist/eval-workflows/hosts/local-task/application.js

@@ -0,0 +1,177 @@
+import { randomUUID } from 'node:crypto';
+import { mkdtemp, readFile, realpath, rm } from 'node:fs/promises';
+import { delimiter, dirname, join, resolve } from 'node:path';
+import yaml from 'js-yaml';
+import { deepFreezeCanonicalJson, digestCanonicalJson } from '../../../eval-core/contracts/index.js';
+import { prepareEvaluation } from '../../../eval-runtime/evaluation/prepare.js';
+import { rescore } from '../../../eval-runtime/evaluation/reuse.js';
+import { saveEvaluationResult } from '../../../eval-runtime/result-store.js';
+import { createNodeCoreContentStore, createNodeCoreRunArtifactStore } from '../../artifact-store/index.js';
+import { persistCoreArtifactSidecars } from '../../orchestration/artifact-graph-persistence.js';
+import { LocalTaskDefinitionSchema, LocalTaskOutputSchema } from '../../inputs/contracts/local-task.js';
+import { captureTaskFiles, fileDigest, taskTreeDigest } from './files.js';
+import { createTaskDiagnostics } from './diagnostics.js';
+import { createTaskWorkspace } from './workspace.js';
+import { createTaskExecutor } from './executor.js';
+import { createTaskAcceptance, TASK_METRIC_ID } from './acceptance.js';
+import { probeCodexCliVersion } from '../adapters/codex/version.js';
+import { captureIdentityFiles } from '../adapters/shared/content-identity.js';
+function assertSupported(task) {
+    if (task.samples.requires !== undefined)
+        throw new TypeError('TASK_REQUIREMENTS_UNSUPPORTED');
+    for (const sample of task.samples.samples) {
+        if (sample.input.inputKind !== 'text' || sample.executionContext !== undefined
+            || sample.evaluationContext !== undefined || sample.expected !== undefined) {
+            throw new TypeError('Local tasks currently accept text sample v3 input and annotations; acceptance belongs in the independent verifier.');
+        }
+    }
+}
+/** Host application shared by CLI and programmatic users. All locators remain outside identities. */
+export async function prepareLocalTask(options) {
+    const base = dirname(resolve(options.definitionPath));
+    const task = LocalTaskDefinitionSchema.parse(yaml.load(await readFile(options.definitionPath, 'utf8')));
+    assertSupported(task);
+    const snapshotRoot = await realpath(resolve(base, task.snapshot.root));
+    const acceptanceRoot = await realpath(resolve(base, task.acceptance.root));
+    const snapshotPaths = new Set(task.snapshot.files.map((file) => join(snapshotRoot, file)));
+    if (task.acceptance.files.some((file) => snapshotPaths.has(join(acceptanceRoot, file)))) {
+        throw new TypeError('TASK_ACCEPTANCE_EXPOSED_IN_SNAPSHOT');
+    }
+    const [snapshot, acceptanceFiles, control, treatment] = await Promise.all([
+        captureTaskFiles(snapshotRoot, task.snapshot.files, task.artifacts.maxBytes),
+        captureTaskFiles(acceptanceRoot, task.acceptance.files, 16 * 1024 * 1024),
+        readFile(resolve(base, task.variants.control), 'utf8'),
+        readFile(resolve(base, task.variants.treatment), 'utf8'),
+    ]);
+    if (control.trim() === '' || treatment.trim() === '')
+        throw new TypeError('TASK_KNOWLEDGE_EMPTY');
+    const execution = task.execution;
+    const executable = execution.runtimeKind === 'fixture' ? process.execPath : await realpath(resolve(base, execution.executable));
+    const fixtureSource = execution.runtimeKind === 'fixture' ? await readFile(resolve(base, execution.script), 'utf8') : undefined;
+    const identityFiles = await captureIdentityFiles([
+        { facetId: 'node-runtime', path: process.execPath },
+        ...(execution.runtimeKind === 'codex' ? [{ facetId: 'codex-executable', path: executable },
+            ...execution.identityFiles.map((file) => ({ ...file, path: resolve(base, file.path) }))] : []),
+    ], 'Local task');
+    const executableDigest = fixtureSource === undefined
+        ? identityFiles.find((file) => file.facetId === 'codex-executable').digest : fileDigest(Buffer.from(fixtureSource));
+    let version = process.version;
+    if (execution.runtimeKind === 'codex') {
+        const probeHome = await mkdtemp(join(options.temporaryRoot, 'probe-'));
+        try {
+            version = await probeCodexCliVersion({ executablePath: executable,
+                environment: { PATH: [dirname(process.execPath), '/usr/bin', '/bin'].join(delimiter),
+                    HOME: probeHome, CODEX_HOME: probeHome, TMPDIR: probeHome, LANG: 'C', TZ: 'UTC' },
+                maxOutputBytes: 65536, timeoutMs: 5000 });
+        }
+        finally {
+            await rm(probeHome, { recursive: true, force: true });
+        }
+    }
+    const workspace = createTaskWorkspace(snapshot, options.temporaryRoot);
+    const contentStore = createNodeCoreContentStore(join(options.outputDirectory, 'content'));
+    const diagnostics = createTaskDiagnostics(contentStore, options.outputDirectory);
+    const executor = createTaskExecutor({ captureDiagnostic: diagnostics.capture, definition: task, snapshot, provider: workspace.provider,
+        temporaryRoot: options.temporaryRoot, executable, executableDigest, version, identityFiles,
+        ...(fixtureSource === undefined ? {} : { fixtureSource }),
+        ...(options.authenticationFile === undefined ? {} : { authenticationFile: options.authenticationFile }),
+    });
+    const store = createNodeCoreRunArtifactStore(options.outputDirectory, { contentResolver: contentStore });
+    const declaration = {
+        dataset: { datasetId: task.taskId, samples: task.samples.samples.map((sample) => ({
+                sampleId: sample.sampleId, input: sample.input,
+                ...(sample.annotations === undefined ? {} : { annotations: sample.annotations }),
+            })) },
+        variants: ['control', 'treatment'].map((variantId) => ({
+            variantId, artifact: { name: variantId, kind: 'skill', source: 'inline', content: variantId === 'control' ? control : treatment },
+            execution: { executor, workspace: workspace.descriptor },
+        })),
+        evaluators: [createTaskAcceptance({ captureDiagnostic: diagnostics.capture, definition: task.acceptance, files: acceptanceFiles,
+                requiredArtifacts: task.artifacts.files, temporaryRoot: options.temporaryRoot,
+                runtimeFiles: identityFiles.filter((file) => file.facetId === 'node-runtime') })],
+        comparisons: [{ comparisonId: 'task-ab', controlVariantId: 'control', treatmentVariantIds: ['treatment'], metricIds: [TASK_METRIC_ID] }],
+        analyses: [{ analysisId: 'task-difference', analysisKind: 'comparison-interval', statistic: 'mean-difference',
+                comparisonId: 'task-ab', treatmentVariantId: 'treatment', metricId: TASK_METRIC_ID,
+                confidence: { method: 'percentile-bootstrap', level: 0.95, resamples: 1000 } }],
+        decision: { decisionKind: 'analysis', analysisId: 'task-difference', minimumEvidenceStatus: 'complete' },
+        experiment: { seed: task.seed, trials: task.trials, sampling: { samplingKind: 'paired', seedCoupling: 'uncontrolled' } },
+        policy: { execution: { maxConcurrency: 1, timeoutMs: execution.timeoutMs }, evaluation: { maxConcurrency: 1 },
+            evidence: { output: 'full', trace: 'full', evaluatorEvidence: 'full', maximumClassification: 'sensitive' } },
+        infrastructure: { contentStore, contentResolver: contentStore },
+    };
+    const prepared = await prepareEvaluation(declaration);
+    async function archiveFiles(files, mediaType) {
+        return contentStore.put({ value: [...files], digest: taskTreeDigest(files), mediaType, classification: 'sensitive' });
+    }
+    async function persist(result, plan = prepared.plan) {
+        const { execution: executed, evaluation, analysis } = result.artifacts ?? {};
+        if (!executed || !evaluation || !analysis || !result.report)
+            throw new Error(`TASK_RUN_INCOMPLETE:${result.status}`);
+        const artifacts = await store.save({ runId: result.runId, createdAt: new Date().toISOString(),
+            plan, execution: executed, evaluation, analysis, report: result.report });
+        await persistCoreArtifactSidecars({ source: artifacts, outputDirectory: options.outputDirectory, cwd: base });
+        return artifacts;
+    }
+    const preview = deepFreezeCanonicalJson({
+        projectionKind: 'local-task-preview', taskId: task.taskId, knowledgeMode: 'provided-content',
+        runtimeKind: execution.runtimeKind, runtimeVersion: version,
+        model: execution.runtimeKind === 'codex' ? execution.model : 'offline-fixture',
+        effort: execution.runtimeKind === 'codex' ? execution.effort : 'none',
+        executionTimeoutMs: execution.timeoutMs, acceptanceTimeoutMs: task.acceptance.timeoutMs,
+        toolPolicy: 'runtime-default', strongIsolation: false,
+        diagnosticsPolicy: 'attempt-snapshot/v1', maxStreamBytes: 10 * 1024 * 1024,
+        runtimeFiles: identityFiles.map(({ facetId, digest, size }) => ({ facetId, digest, size })),
+        snapshotDigest: taskTreeDigest(snapshot), acceptanceDigest: taskTreeDigest(acceptanceFiles),
+        snapshotFiles: task.snapshot.files, artifactFiles: task.artifacts.files, maxArtifactBytes: task.artifacts.maxBytes,
+        samples: task.samples.samples.map(({ sampleId, annotations }) => ({ sampleId, ...(annotations === undefined ? {} : { annotations: annotations }) })),
+        trust: task.trust, writeBoundary: 'trial-private-workspace', retryState: 'shared-within-trial',
+        estimatedCostUSD: null, estimatedWork: { ...prepared.estimatedWork, uncertain: [...prepared.estimatedWork.uncertain] },
+        limitations: ['trusted-local-not-hostile-code-sandbox', 'native-skill-discovery-not-measured',
+            'external-provider-conditions-uncontrolled', 'small-example-is-not-release-evidence'],
+    });
+    return Object.freeze({
+        preview,
+        async run(signal) {
+            const snapshotReference = await archiveFiles(snapshot, workspace.descriptor.mediaType);
+            const acceptanceReference = await archiveFiles(acceptanceFiles, 'application/vnd.omk.local-task-verifier');
+            const runId = `run-${randomUUID()}`;
+            const diagnosticsDirectory = `task-diagnostics/${runId}`;
+            const { value: result, references: diagnosticReferences } = await diagnostics.run(runId, () => prepared.run({ runId, signal, annotations: { ...preview, snapshotReference, acceptanceReference, diagnosticsDirectory } }));
+            if (result.status === 'failed') {
+                const value = JSON.parse(JSON.stringify(result));
+                const failureReference = await contentStore.put({ value, digest: digestCanonicalJson(value), mediaType: 'application/json', classification: 'sensitive' });
+                throw new Error(`TASK_RUN_FAILED:${result.error.code}:${failureReference.uri}`);
+            }
+            const reference = await saveEvaluationResult({ result, store: contentStore });
+            return { result, artifacts: await persist(result), reference, diagnosticsDirectory, diagnosticReferences };
+        },
+        async rescore(source, acceptance, root, signal) {
+            const verifiedAcceptance = LocalTaskDefinitionSchema.shape.acceptance.parse(acceptance);
+            const records = source.artifacts?.execution?.records;
+            if (!records || records.length === 0)
+                throw new TypeError('TASK_RESCORE_EVIDENCE_MISSING');
+            for (const record of records) {
+                if (record.executionStatus !== 'completed' || record.output?.contentKind !== 'inline')
+                    throw new TypeError('TASK_RESCORE_EVIDENCE_MISSING');
+                const output = LocalTaskOutputSchema.parse(record.output.value);
+                if (output.missing.length || output.collectionErrors.length
+                    || task.artifacts.files.some((path) => !output.files.some((file) => file.path === path)))
+                    throw new TypeError('TASK_RESCORE_EVIDENCE_MISSING');
+            }
+            const files = await captureTaskFiles(resolve(root, verifiedAcceptance.root), verifiedAcceptance.files, 16 * 1024 * 1024);
+            const nextInput = { ...declaration,
+                evaluators: [createTaskAcceptance({ captureDiagnostic: diagnostics.capture, definition: verifiedAcceptance, files, requiredArtifacts: task.artifacts.files,
+                        temporaryRoot: options.temporaryRoot, runtimeFiles: identityFiles.filter((file) => file.facetId === 'node-runtime') })] };
+            const acceptanceReference = await archiveFiles(files, 'application/vnd.omk.local-task-verifier');
+            const runId = `run-${randomUUID()}`;
+            const diagnosticsDirectory = `task-diagnostics/${runId}`;
+            const { value: result, references: diagnosticReferences } = await diagnostics.run(runId, () => rescore(nextInput, source, { runId, signal, annotations: { ...preview, diagnosticsDirectory,
+                    acceptanceDigest: taskTreeDigest(files), acceptanceTimeoutMs: verifiedAcceptance.timeoutMs,
+                    acceptanceReference, postHocRescore: true, sourceRunId: source.runId } }));
+            // The existing run-artifact store requires a single run contract. A reused Execution
+            // retains its original contract, so persist suffix runs through Runtime's result store.
+            const reference = await saveEvaluationResult({ result, store: contentStore });
+            return { result, reference, diagnosticsDirectory, diagnosticReferences };
+        },
+    });
+}